Example:  A large organization has had a DR office and program in place for many years, multiple backup sites in place and holds recovery exercises on a regular basis throughout the year.  Internal audit regularly gives them their seal of approval.  Sounds good, right?  On the surface, the Tier 1 metrics will paint an average or above average picture of the program and its adherence to best practices.  Now, once we used Tier 2 metrics to assess the real recovery capability of the organization, it was quickly determined that the true IT disaster recovery capability did not exist and that the DR program was failing miserably in its ability to recover the critical systems and applications of the organization. Senior management in IT had lived under the belief that all was well in the realm of DR when in reality, could not even begin to recover the systems and applications needed to run the business.

Our metrics tool analyzes the level of compliance, weights them and produces speedometer graphs that quickly show management the state of the component.  Additionally, our tool ensures that if key components in an area are below average, the entire component cannot be raised even if its score dictates it.  Using metrics is painful and can be eye-opening but also can lead to increased management support, allocation of resources and most importantly, the multi-year budget needed to implement what you need to succeed.  Use them, they work.

We must change our mindset and manage our Crisis Management Team by those with the capability and authority to strategically respond and recover from an unplanned event.  A client of ours experienced a highly publicized event regarding their CEO that required their Incident Management team to be assembled for a long period of time and quickly found that  a number of team members placed on the team by title, needed to be replaced by others who could execute.  The members were released and the team members who could execute, continued their efforts.  The event forever changed how this client will manage their Crisis Management Team.  Managing your department well on a day to day basis and being able to manage the team through a crisis are two totally different events.

We must look at our teams like a professional sports team and identify the strengths and weaknesses from the Leader to the individual role players.  In my consulting firm, we rank our consultants by capability in client interaction,  ability to lead engagements, work behind the scenes, etc.  By doing this, we are able to place the best consultant fit for the particular engagement.

This is not an easy task to complete as Crisis Management Teams are typically filled with politics and alliances across the senior management team who typically fills these roles.  To assess team performance, we recommend that you have primary and alternate team members participate in mock disaster exercises, have different team leaders run the exercises and stress test individuals on the team by requiring specific tasks to be completed.  By doing this, you can assess team member performance, strength and weaknesses.

On a regular basis, sit down with your senior management and ask them to rank the Crisis Management team and its players in its ability to execute.  Discuss the strengths and opportunities of the team and remind senior management that its the ability to execute in a crisis is what a team member must be able to do.  Lets look beyond the title and bring in the people who can lead the organization.

Here is why.  Comparing what you spend against other organizations is not indicative that the dollars being spent are prudent, appropriate and  supporting a recovery environment that will meet the needs of your organization and customers:

• Example 1:  Large multi-billion organization is spending 2% more than their industry annually on multiple backup sites, offsite storage, network, hardware maintenance, staff, etc.  Looks good, right?  The in-depth assessment reveals the budget simply maintains multiple backup sites and strategies that can simply provide for standalone application recovery and not recover the  business.  The spend comparison, even though looking like they spent more than their peers, actually showed dollars were being poorly spent on strategies and technologies that had returned little to no value.
• Example 2:  Large regional hospital wants to compare themselves against the healthcare industry.  In our experience, hospitals have experienced severe budget constraints over many years and often do not spend the appropriate dollars needed to protect their highly complex computing environments.  So, a comparison against other hospitals may not always be a good guideline.  Lastly, we view hospitals as important as the financial industry.   Hospitals should compare themselves against the leaders in Disaster Recovery, the financial industry.  I always say their “human transactions” are more valuable than “money transactions”.
• Example 3: Private college in the northeast wants to develop a working recovery environment based on what the industry is spending.  Well, for one, colleges like hospitals, have not been big spenders and two, creating the initial environment will take significant investments over time before spending is somewhat stabilized.

In the many cases we see, the Disaster Recovery environment requires significant investments over time to even reach a stable point of operation before an annual budget stabilizes.  And even then, you could expect a 15% or more budget increase annually with maintenance, new hardware, faster network, storage, etc.  This happens in new and existing recovery programs.  Comparing yourself from a spend perspective can be helpful when you have a stable, working recovery environment that can recover your business.  This allows you to compare your spend more evenly.

Last, don’t always compare to your industry; compare yourself to those who do it best to give you a goal in mind.

Additionally, the lack of education at the senior management level regarding current recovery capabilities is beyond comprehension.  We do not expect senior executives to have a “boots on the ground” perspective but at least have a high level understanding of how, when, where and why.  This lack of knowledge is highlighted when we display current capabilities versus needs and you see the deer in the headlights look!    You must educate the executives today, tomorrow and in the future.

To  present the best possible picture for alignment we do the following:

1. Trust the “best educated” guess of the business unit leaders interviewed.
2. Use weightings in our BIAs for each impact category to ensure most important categories (loss of revenue, customer service) to the organization receive the appropriate weightings.
3. Identify the core applications in each BIA to eliminate non-critical supporting applications being deemed critical.
4.  Include IT representation in all key meetings.
5. Remind IT this is about the business process and not worry immediately about the servers, network, etc.  needed for recovery.
6. Have an objective team review the results before presentation.
7. Remind everyone that manual workarounds have little to no shelf life in today’s highly automated world.

• Supported by Senior Management
• Proactive in Activation – Not Afraid to Activate
• Use a Competency Based Leadership Model to Develop Team Members
• Do Not Manage by Title but by Ability to Lead
• Roles are Well Understood
• Team is Scalable Depending on Need
• Exercises and Training are Regularly Held
• Depth Across Primary and Backups
• Team Members Empowered to Act
• Followed Established Incident Principles and Priorities
• Follow a Comprehensive Crisis Management Plan
• Use a Standardized Decision Making Process

The size of the organization will not dictate whether or not a good IT change management process is in place. We have seen small IT organizations that have stringent processes in place and Fortune 100 with gaping holes. Or the process is highly dependent on a number of key people who manage the changes like a hawk but when absent changes are made with little resistance.

Without an IT change management process that has teeth when it comes to DR planning, your recovery capabilities stand a good chance of failing when you need them most.

Your iT change management process must address the DR process and ensure changes are reflected in recovery plans, strategies and configurations in parallel with the change being made to production. The simplest of changes could lead to a catastrophic failure in recovery.

It is important to remember that every crisis specifically influences living, behavior and decision making of people. It directly influences all managing processes:

Even with the best pre-planning and documented exercise task list in-hand, if you can’t ensure your exercise participants will cooperate, work together, think outside the box you  and mesh as a unit, you are headed for trouble.  If you are the Exercise Coordinator, you must assess the personalities and eccentricities of those attending the exercise.  Know ahead of time what to expect from all of those involved.  Get to know each person and how you can maximize their performance.   This is your team and you are the leader.

A good example is a recent exercise we attended.  A staff member of ours, who is very insightful, knew a couple of the team members were more introverted and needed specialized attention to make them comfortable and maximize their potential for success.  Others needed to be reined in and clearly told to follow the rules of the game.  Make sure there are no silos and people are stepping up to help each other.

Thus recent exercise was one of the best examples of people working together, cooperating, stepping in to help others when needed and taking accountability for success of the exercise.  Yes, we expect success, in each and every exercise and when you set this as the mark to meet at the beginning, you can see a difference in performance.

Know your team, who they are, what makes them tick and how they all fill the right seats of your recovery machine.  Maximize your people and you will maximize the process.

• Business Impact Analysis (BIA) Recovery Time Objective (RTO)
• Recovery Demand (RD)

Exercising applications with a critical BIA RTO and high RD score are easily defensible to management. It is the less obvious values that will require management decisions as to what levels they deem acceptable.

“We believe in educating our colleagues, we hope that each of them had one takeaway they could use immediately”, said Michael Herrera, CEO MHA Consulting.