All posts by hmpeace

BC/DR and the Corporate SDLC

Last year I had the opportunity to meet a senior member of a former client’s executive management team while we both spoke at a conference in Miami. I began by asking him if they still did a Peace Business Continuity BIA before any new system was put in place. He smiled at me and said, “So you’re the guy who worked with so and so back in 2000.”

One of my priorities then and now was to insert BC/DR considerations into the System Development Life Cycle (SDLC) so they would be addressed up front. There are some very important reasons for doing this.

Every major change should be preceded by a focused BIA.   A properly conducted BIA for the change will identify what new pressures will be brought to bear on the current state recovery strategy. New applications, business functions, and infrastructure changes can have a discrete and, perhaps, debilitating impact on disaster recovery strategies and business continuity planning. The BIA should be used to measure outage impacts for the new change and be used to measure the overall impact on recovery requirements. If the SDLC includes a BIA hook, it’ll happen.

Each BIA Impact Category should have a list of recovery requirements. For example, at the client mentioned above, I used the categories we developed in our Y2K work to separate the most critical (AAA) from the more deferrable (D). I then used the categories to delineate what recovery requirements each new application needed to include in their planning. For the AAA through B ratings, the developers were required to include local failover (multi-processor environment), remote failover at the recovery instance, written recovery plans for the application and business units supported, and annual failover recovery testing. Putting those requirements early in the SDLC made for far better upfront planning and trying to retrofit them later would have been impossible. It eliminates the ‘but nobody told us” excuse.

BC/DR considerations must be included in the budget. The final price has to include provisions for recovery from the very beginning because there will be no money available later. For example, a new AAA application will be required to run in a multi-processor environment for local failover, not a single-threaded server, and another one is needed in the recovery environment. Now we’re talking about two specialized servers, not one and some possible impact on the network response time. The project budget should include this additional capability. Pricing things right in the beginning will mean an accurate assessment of the true costs.

A Gap Analysis should be performed to determine the overall impact of the change. The current state recovery capabilities should be examined to determine if the change will open any gaps in the company’s ability to recover critical business functions in the timeframe required. The change can impact the data center, data and voice networks, and business area recovery, or all of the above. Any gaps discovered can affect budgets and project delivery timeframes and priorities.

Here is what you’re trying to accomplish: At some opportune point between the time some executive says, “That’s a great idea” and the time they reach for their wallet, you want BC/DR considerations to be addressed so the real price is known and the implementation plan is written accurately. If there are BC/DR hooks in the SDLC, all the recovery issues are addressed in a timely fashion and the fixes won’t need to be bolted on later, if at all. As always, leave some comments or contact me directly if you’d like.

© Copyright and All Rights Reserved Howard M. Peace

 

Decision Making At Time of Disaster

Every organization has their own structure and process for making decisions, especially those concerning the spending of money. However, when faced with an unexpected outage or disaster, things can change, sometimes unpredictably. Normal management processes can be completely disrupted depending on the nature of the event and the availability of the normal decision makers. For example, on 9-11, I had the exalted title of Director, Disaster Recovery and Business Continuance for a major technology company. Soon after the event began to unfold I found out that the CEO and CFO were on vacation together with their families and no one knew where. With air travel stopped dead and cell coverage overloaded, they were completely out of touch and unable to direct efforts to respond to the crisis. Additionally, almost all of the other senior management team were visiting a potential acquisition in another state and they were off the table also. In effect, the Facilities Manager and I were the only senior managers available to lead the crisis management efforts, and we were huddled around a small black and white TV trying to find out what was going on in the world. We soldiered on and made financial and management decisions normally above our pay grades and received confirmation that they were the correct decisions several days later when management came back on line. Afterward, I thought of a couple of items that should be included in our planning efforts. They are:

Management Succession Plans. In order to provide for the unavailability of key decision makers in a crisis, either personal or corporate, every manager should have a written management succession plan. Key leaders should formally identify who exactly should take over for them in their absence and what special authorities should be granted to them. (Remember Al Haig declared he was in charge when President Reagan was shot until someone reminded him that’s not what the Constitution says.) I developed my methodology when I was called in to consult with a family-owned Southern California Bank to write a plan for their IT Director. The Board had only recently found out his hobby was flying small acrobatic planes and they were worried that, God forbid, he crashed, no one knew what exactly he was working on or how to manage his department in his absence. After assuring him the Board wasn’t looking to replace him, I was able to work with him to develop his first written job description, identify key staff who could assume most of his time-critical responsibilities, develop a written status report format and schedule, and improve his future planning documentation. I also interviewed him about his decision making methods and war gamed some scenarios to expose how he would likely respond if he were on duty. Lastly, I arranged for another senior manager to spend two days per month sitting with him getting up to speed on current state developments. In the end, the Board felt they had closed a major gap in their recovery planning efforts and used the methodology to create plans for other key roles as well.

Refined Tabletop Exercises. I am a firm believer in the usefulness of tabletop exercises in testing all facets of disaster recovery and business continuity plans. The exercise is a conference room based walkthrough of how a company will respond to an unexpected outage scenario revealed at the start of the exercise. The one wrinkle I always include is the unavailability of at least one key decision maker (or key technical guru). If the scenario is imagined correctly, their absence will be used to identify areas of expertise, tribal knowledge, or key decisions which are not documented or delegated and are sorely missed. In my experience, the following often comes up if the key decision maker is not available:

  • Who has approval authority now for the emergency expenditures? (If the backup has a much lower limit, how does that get bumped up to cover the spend?)
  • Who has authority to increase corporate credit card limits for the traveling recovery staff? (Do they even have corporate cards? If not, how do we expedite this?)
  • Who else can authorize increased system access authority levels? (Who thinks through the security risks?)
  • Who approves the PR statement that goes out? (Think about a canned statement with blanks to fill in with actual details so the word smiths don’t take 2 days to invent the wheel.)
  • What do we say about casualties and injuries?
  • Can we send the hourly staff home and still pay them for the day? (On 9-11 I used HR’s snow day policy to authorize this. I put on my Pirates of Penszance Admiral’s hat as the guy in charge and declared it was snowing outside!)
  • Do we still pay people who can’t come in because of the disaster? (What is our normal policy and if we want to make an exception, do we have the authority to do that?)
  • Who can authorize unexpected overtime?
  • What are our legal and contractual obligations and can we get some mercy?
  • Do we notify our customers yet, and do we have to report this to a regulating authority?

If the people who make these decisions, and more, aren’t available, is there a management succession plan in place which delegates the authority to make the call in their absence? Don’t worry if you uncover missing elements, that’s what the exercise is meant to do: expose gaps that can be remediated.

Conclusion. Without good management succession plans in place the chances are that good decisions will not be made or even thought about until it’s too late. As a father, I have seen the truth in the old saying, “One child, one brain. Two children, half a brain. Three children, no brain at all.” (In extreme pressure situations, some managers tend to revert to this in the middle of “groupthink”.) In order to avoid confusion, conflicting agendas, and the Al Haig Syndrome, make sure everyone knows how decisions will be made in an unexpected event with key decision makers absent. As usual, leave comments here or contact me directly.  Happy decision making!

 

© Copyright and All Rights Reserved Howard M. Peace

 

Business Continuity & Information Security: They Need Each Other

Having played in both of these fields, I have become aware of the symbiotic relationship that many overlook. Both are important aspects of corporate risk management and should be natural allies. In the early eighties I managed both as the Data Security Officer for Manufacturers Hanover Trust in New York City. As one of the first CISO’s in the country, I came to understand the importance of both disciplines and had a ring side seat as they grew to maturity. In those early days the emphasis in Infosec was password changes (remember RACF on the mainframe?) and violation reports (Al Gore had not yet invented the Internet and PC’s were an oddity. I still have one arm longer than the other from carrying an Osborne around). Business Continuity barely existed and primarily the focus was limited to disaster recovery of the data center. In their current states, they influence and help control all aspects of the business and their responsibilities are now 24/7 and reach around the world. As they have matured, the playing field has changed, but their need for mutual cooperation has remained. Let me show you what I mean.

Infosec depends on good business continuity. Infosec needs a reliable platform that includes continuous availability of the tools they need to monitor and control access to the systems.

  1. Without a reliable recovery program in place, a denial of service attack (DOS) brings a company to their knees in a hurry. In effect, the DOS triggers a business continuity event. Infosec needs quickly restored access to determine the cause of the outage and repair whatever damage has been done.
  2. If the backup procedures are not correctly implemented, rebuilding the systems becomes problematic and can take much longer as they search through previous generations to find an uninfected version.
  3. If telecommunications can’t be quickly restored, the external (and internal) threats can’t be efficiently managed and the Infosec team will have trouble coordinating their efforts. (That’s why when planning recovery efforts I always want to know how the techies communicate with each other (phone, cell, email, IM, etc.) because those tools become critical to recover as quickly as possible.)
  4. A good BCP will identify any new access requirements that are necessary to accomplish recovery efforts. For example, new or increased VPN access, higher or increased level access authority, emergency contact information, increased approval authority, etc.

Business Continuity depends on good Infosec. Remember the old John Wayne flick The War Wagon? The plan of attack against the armored stagecoach was to force it off the safest path onto a trail where the security efforts could more easily be thwarted. Exercising a recovery plan can open the organization to increased security threats.

  1. During the recovery process user passwords are sometimes shared, defeating individual responsibility controls. “Joe’s not here, but I know his password” can sometimes lead to unauthorized access that exposes the company to fraudulent activities and the disclosure of information thought to be secure. (Threats include disclosure of Personal Identification Information (PII), credit card numbers, access to approval levels to release payments, etc.)
  2. Admin passwords can be used to bypass normal change management control procedures, leading to mistakes. “It’s an emergency, so we don’t have time to test this first, just go ahead and move it into production.” Famous last words as the system crashes, delaying recovery.
  3. System techs will sometimes leave remote access ports open to make their job easier during recovery, thereby exposing the systems to outside threats. Hackers love open windows.
  4. There can be pressures brought to bear to circumvent normal access approval procedures that require written, and sometimes dual, signatures. “The boss said it was alright, and it’s only temporary.”
  5. There’s never enough time to review access violation reports. Many strange things can go bump in the night when the watchers aren’t watching.
  6. Physical access at the damaged facility can get loose allowing unauthorized parties to wander around to see what they can find and what they can do. The card key access system is down, so doors get propped open.

As you can see, there is a great need for BCP and Infosec to work closely together to address the threats posed by security incidents and unexpected outages. They may be separate silos but there had better be a good bridge between them. As always, your comments are welcome and feel free to contact me directly. Be safe out there.

© Copyright and All Rights Reserved Howard M. Peace

The 5 Best Reasons for Using A BCP Consultant

Of course my favorite reason is because that’s how I make enough money for my family to eat and sleep indoors. But really, there are some very good reasons to reach outside your organization for subject matter expertise and experienced help in accomplishing your BC projects. The best reasons I can think of are:

  1. A lack of internal expertise. It may very well be that no one at your company has ever done a BCP project and there isn’t a clear understanding of how to go about accomplishing the assigned mandate. A directive has come down from on high to “go forth and business contingify, whatever that is, and, by the way, you have six months.” An experienced consultant can develop project focus, detail all the steps required, develop a reasonable schedule, and accomplish the tasks in the allotted time frame. A good consultant will also major on knowledge transfer rather than building dependent step children.
  2. A lack of internal resources. It’s especially true in smaller organizations, but most companies operate on a rather lean staff budget and they don’t have a pool of technical and business people sitting around with large open areas in their job description just waiting to take on a project of this magnitude. The leader of the project usually has an already crowded plate and nothing gets removed to make time for this new responsibility. A good consultant is a dedicated resource who can focus entirely on the project and should bring not just direction but strong shoulders to carry the bulk of the weight. I tell prospective clients, “I hire people to do things like wallpaper the kitchen because I have too many Howard things to do. Assign me part of your to do list so you can focus on the stuff that only you can do. Give me this project and I’ll get it done, and make you look smart for hiring me.”
  3. People tend to cooperate with outside consultants. The staff usually realizes the company is paying for this consultant and so they should probably cooperate so whoever brought them in doesn’t get a report that they have become a roadblock. Like many families, they treat outsiders with a little more deference than they would one of their own. This is not always the case, but a good consultant recognizes the obstacles and uses his powers of persuasion to get on calendars, run meetings efficiently, gather information as painlessly as possible, and honor everyone’s time pressures.
  4. A good consultant brings a deep well of experience. This BCP project is not their first rodeo, or least it shouldn’t be or you’ve got the wrong consultant. They may have even done similar projects at similar companies in the same industry. They should be able to provide insight on how your peers are doing BCP and be able to cross pollinate solutions and recovery methods from other industries as well. The consultant should be able to enrich the solution set with lessons learned on successful efforts performed elsewhere. They should have a clear understanding of the technology required to support the client’s recovery needs and priorities, a handle on realistic expectations for the time required for recovery, an ability to outline effective recovery strategies, and the ability to build the business case for BC/DR expenditures. He should also be able to identify a roadmap for the way forward to improved recovery capabilities (not just upsell opportunities for his firm). And, perhaps it goes without saying, he should have a good handle on project management metrics including adequate status reporting on the project’s progress and unresolved issues.
  5. Company budgets often have restrictions on hiring new staff. Even though the BCP project has high level support and has been deemed an important requirement for this year, the bean counters have convinced management that the people costs are the easiest and most important to control. The edict comes down, “No more hires this year”, and the door for adding skilled staff slams shut. You’d like to go out and find an experienced BC person to bring onboard, but that is not possible. However, consultant dollars often come from a different budget and since the project has a defined cost and duration, money becomes available to bring in an outsider to handle a short term project. When the project is over, he goes away and you are not saddled with the cost of an ongoing head count.

In a future blog I will discuss how to find and use a good consultant, but for now let me just encourage you to make use of the wealth of experience a good consultant has learned while someone else was paying for acquiring that knowledge. I realize mine is not an entirely objective opinion (my professional motto is, after all, ”Consulteo ergo sum” I consult therefore I am), but I believe the right consultant can be a tremendous resource for the success of your BC project. As always, I welcome your comments and please feel free to contact me. Good projects to you.

 

© Copyright and All Rights Reserved Howard M. Peace

 

5 Important BC Questions to Ask For Mergers & Acquisitions

It seems every day there is a new announcement about one company buying another one. Sometimes the target company is a competitor, other times it will be a company that will enable the buyer to complement or expand their portfolio of services or products. (For example, Verizon bought CyberTrust to acquire our extensive portfolio of professional services.) An early stage of the M&A courtship is the Due Diligence phase, where the buyer sizes up the target, checks the books, reviews production, and details assets. If everything looks good and the price is right, they walk the aisle together.

What does BC have to do with M&A? I’m glad you asked. Here are what I believe are 5 important BC-related questions that should a part of every M&A effort and should be asked of the target company:

    1. Do they have an active contract with a BC/DR vendor, and is it assumable? If the target company has a valid contract with a vendor it may represent an ongoing legal agreement and financial obligation that may not end with the purchase, or maybe can’t be assumed by the buyer if it so desires. Depending on the integration plans the buyer intends to implement, the arrangement may still be necessary for a length of time (and may need to be extended) or it could be unnecessary or redundant. Often times this is overlooked during the Due Diligence Phase and could turn out to be an unpleasant surprise later, effecting the actual cost of integration. (Always remember, management doesn’t like M&A surprises!)
    2. Do they have trained BC/DR staff? It has been my experience that those companies that do poor due diligence often don’t recognize the value of staff they are acquiring. By not addressing personnel issues right up front they find the cream of the crop walking out the door, taking their valuable skills and tribal knowledge with them, and are left with the less nimble folks who have fewer options, and maybe less skill and knowledge. Finding a trained staff who can implement a successful BC/DR program can be a valuable asset when evaluating the talent about to join the mothership.
    3. Are there any assets that can be used to augment the buyer’s recovery capabilities? Many M&A methodologies overlook technology and physical assets that would be a wonderful addition to the resources available to improve recoverability for the buyer. Instead of sending everything to a landfill in New Jersey or selling it for pennies on the dollar, the M&A team should be encouraged to keep an eye out for what seems to be redundant resources that could be used to upgrade the buyer’s current state of recovery and integrate new capabilities that will be needed for the newly acquired operations. Equipment I always look for includes telephone switches, servers, disc farms, tape management systems, routers, generators, UPS, and HVAC equipment. I also want to take a look at their facilities in case I find a great place to create a new cold/warm site, or find useful alternate telecommunications and power pathways, or provide an offsite business area recovery capability. Like sorting through a flea market or attic, you may find the technology equivalent of a Mickey Mantle rookie year baseball card.
    4. Have they done a recent Business Impact Analysis? A good BIA will tell the buyer what the most critical business processes are at the target and will help set expectations for priorities for the integration phase and ongoing recovery requirements and capabilities. I have on a number of occasions also used the ISO standards to review security and recovery concerns at acquisition targets to get a better feel for what problems might come along before the buyer’s name goes on the wall.
    5. How will the acquisition change recovery requirements at the buyer? If the acquisition goes through, it will inevitably change the scope and perhaps the way in which the buyer plans for disaster recovery and business continuity. The new addition to the company fold will bring with it new equipment, connectivity needs, business areas, and perhaps facilities. For example, the target may bring with them 100 new servers. Some of those servers may support critical operations that require immediate failover where none currently exists in the buyer’s backup location. Getting out in front of these costs will go a long way to determine just how great a deal this was. Including these expenditures during the Due Diligence Phase can help to effect a number of decisions regarding price, data center expansion, and additional recovery costs.

 

 

With these questions in mind, I always encourage the BC/DR manager to sit down with the M&A team to present these questions for inclusion in their due diligence methodology. Getting them to keep an eye out for this information you need, along with your potential shopping list, will increase the BC Program’s visibility and will perform a valuable contribution to a successful merger. Involvement in the M&A process will prevent a merger diluting the current state recovery capability and can serve to enhance and expand it. At the very least, BC/DR will get to look over the loot before it disappears out the door. Please feel free to make comments and offer your own suggestions and war stories.  And, as always, feel free to contact me directly. Happy merging and acquiring.

 

© Copyright and All Rights Reserved Howard M. Peace

5 Ways Consulting Resembles Curling

Okay, so for those who didn’t grow up with 3 channel black and white TV in the fifties, Curling is a Canadian sport involving ice, brooms, and large tubs. (For those from NYC, think of ice Bocce.) When I was a kid a thousand years ago, Curling used to come on one of the snowy channels we got and the season started after football season in the old days, before the playoffs extended to Valentine’s Day. In fact, Curling is now an Olympic sport, if you can believe it, and can be interesting to watch if you understand the rules. It occurred to me several years ago that Curling and consulting are the same sport. Keep reading. I promise to make sense of my analogy.

Curling begins behind a start line where a player moves forward and lowers the moving tub onto the ice with the goal of hitting a target 100 feet away. If the tub moves too fast, or if it strikes the target too hard, it will careen outside the scoring circle and the fans will shout obscene comments on the players’ ability and family heritage. The pace and direction of the tub down the ice is influenced by two players with brooms who will sweep the ice ahead of it in order to change direction and speed to enable it to hit the target and remain in scoring position, unless it is pushed aside by the opposing team. Got it? Now let’s compare the game to consulting.

  1. Every BCP consulting engagement has a target, whether it is a Business Impact Analysis, or a Business Continuity Plan, or a Disaster Recovery Test. It is conducted often on a rather slippery field with outside influences, some beyond your control, which will attempt to effect the outcome. Some of the players will have separate agendas entirely and some of your players will not be very adept at keeping things on target. The difference between curling and consulting is, unfortunately, that the target can move if it isn’t written in stone in the Statement of Work (SOW). Mission creep outside the scoring zone is a real possibility if you’re not careful.

2.  BCP Consulting engagements are a team sport. Any consultant who thinks he can accomplish the target without help will have no supporting players to keep the goal firmly in mind and the project on track. Cultivate cooperation toward your shared goal. Remember you are there to help your team navigate the mine fields and produce a mutually beneficial result. You are there to serve, not rule.

3.  As in curling, consultants are not allowed to smack the tub with the broom to keep it on track. This is sometimes very tempting, but don’t do it. It’s against the rules. Treat every opinion with respect, no matter how idiotic, and work hard at gaining consensus. Resist pontificating on BCP dogma and project management methodology and you’ll stand a better chance of hitting the target. Always remember, you can drywall with a sledgehammer but the cleanup is messy. You can make your point without leaving a gaping divot.

4. Use influence by getting out in front of the project. That’s what the brooms are for, influencing the project by smoothing the way toward the goal. You can use influence by subtly arranging the way for things to go at the speed and direction which will accomplish your goals. Recognize the staff power structure and the important players and enlist their help in keeping the project on target. Inexperienced consultants need to learn how to convince already busy people to do one more damned thing, for them. Consultants who don’t do “subtle” are rarely successful and don’t get return engagements.

5.  The game is over when the official (client) says so. Sure, I have had clients that were like Lt. Colombo (“Ah, one more thing…”) and they’ve tried to drag the project out before final approval (and payment). But always remember your goal is to deliver an end product that’s sits squarely in scoring position. The client should be confident that he got what he paid for and that his goals, not just yours, were reached. For me, the most efficient way to accomplish this is through consistent feedback throughout the project and vetting both the data and the conclusions. Don’t drop your final report off like a drive by newspaper delivery. Use drafts and rewrites until you both know your data collection is complete and accurate and that any conclusions and recommendations make sense and are on target. No surprises. I learned this while doing bank risk reviews for Lloyds of London in Latin America. At the end of the week, I would have all my findings and recommendations written down and would meet with the board of directors to lay it all out for them. Any changes that needed to be made were made with them so there would be no surprises when London took my report and made the recommendations requirements for renewed insurance coverage. If you take the same approach, you will have much happier clients and they will consider you for future engagements.

Hopefully, this advice makes sense and will help you to leave the clients happy and cheering as you hit the target and scored a successful project conclusion. And remember, the same analogy works for internal projects if you are an employee. Use persuasion and don’t just rely on policy to gain cooperation. Leave me any comments you might have, including any correction of my insight into a fine Canadian sport. Happy consulting!

© Copyright and All Rights Reserved Howard M. Peace

Where Should BCP Report and to Whom?

As a consultant, I get asked this question a lot. Either when a client is in the process of establishing a new function, or they have become disenchanted with the results they are getting, or they are having problems getting cooperation or visibility for the Program. Usually, however, they ask the question wrong. They ask where it should report, or who it should report to, but miss the fact that it is really a two part question. As with any business function, where BCP reports can be as problematic as having the wrong management overseeing it, so it’s both where and to whom. Allow me to discuss some of the pros and cons and offer some general guidance on the proper placement of the BCP function which will help ensure its best opportunity for success.

Where should BCP report? I’ve encountered successful BCP functions reporting at numerous places in the organization. Frequently, it resides in the technology department, but quite often it sits under other more general business functions, such as administration, treasury, project management, physical security, or facilities.   Wherever it sits, it must have the following to be successful:

  • It should report somewhere where it won’t get lost. In highly complex organizations where there a multitude of foci, it is more difficult to bubble up the important BCP issues to the top for proper attention to detail. So much noise is generated with everyone else’s issues that the voice of one crying in the wilderness receives little or no attention. Meeting agendas are so clogged that BCP issues are saved for last and often get short shrift, or they run out of time before they can be even addressed.
  • It should not report too high in any organization. I’ve seen it report to the CIO, the CFO, or the COO and watched as those busy managers just had too much on their plates to give BCP any but the slightest attention. Better to have a reasonable chain of command that can get on their calendar when necessary with an agenda properly focused on what BCP needs.
  • It should not report too low in any organization. If BCP sits too far down the chain the relevant issues won’t ever bubble up and the decision process will become overly onerous and long. Sitting too low says to the rest of the organization that BCP isn’t very important and the BC manager doesn’t rate as someone they should listen to. What you’re looking for is “just right”, like in the story of the three bears. Close enough to decision makers to get the right amount of attention but not so low that BCP gets drowned out by everything else. And “just right” is different in every organization. Like Goldilocks, you’ll figure it out.
  • It should not report to the audit department. Audit’s role is to ensure an organization’s processes and documentation comply with internal and external policies and requirements. In effect, they check the work of others. Implementing the BC Program themselves removes their ability to objectively review the content and procedures of the BCP. While they might have some great ideas and tribal knowledge to offer, they should refrain from direct responsibility for implementation, freeing them to fulfill their charter. Any smart BC Manager will maintain open communication and cooperation with audit, but it is my opinion that the reporting of BCP should be placed elsewhere in the organization.
  • The reporting structure should include BCP accountability. Most organizations use a management by objective (MBO) accountability for managers that will be used to measure personal success for the year. Having a personal MBO for the success of the BCP function reporting to them does wonders to focus attention and provides incentive to ensure BCP gets the cooperation it needs both inside their realm and across the company.
  • Reporting to Technology is sometimes problematic. Sometimes locating BCP in Technology sends the message to the rest of the organization that BCP is primarily, maybe completely, a technology problem, in effect just disaster recovery. Get the systems up and running and everything will be fine. Not. Years ago, getting the lights blinking at the recovery center in Philadelphia was all we in disaster recovery had to worry about. In the years since, especially after 9/11, the mission has expanded to include everything else.   A correctly implemented BCP must address outage issues residing in the business functions, facilities, remote locations, assembly lines, and on and on. Locating the BCP in Technology sometimes clouds organization thinking about extending disaster recovery beyond the data center to include true business continuity.
  • Not reporting to Technology is sometimes problematic. Contrarily, reporting to a non-technology area can lead to barriers in communicating with the technical staff and management. (I’ve had sys admins and DBA’s talk slow to me like English was my second language until I explained I was one of them once, before I went over to the dark side.) BCP can be viewed as an outside organization that really couldn’t understand the massive undertaking of recovery and is there to lay more burdens on an already over worked and underfunded technology group. If it reports elsewhere, the BCP group must work hard to build relationships and earn respect to gain cooperation. They should acknowledge what they don’t know, ask good questions and listen to the answers, and show appreciation when they ask already busy people to do one more damned thing, for them.

 

To whom should BCP report? Now we’re getting to the other side of the equation. In all honesty, I’d rather have BCP report to the right person in almost any area in the company than the “right” organization with the “wrong” person in charge. Why? Because the right manager can accomplish BCP initiatives wherever he or she sits in the organization. He knows how to get things done, has a solid reputation, knows and is known by the right people, and knows how to manage for success. Here are some things to consider when choosing who should manage BCP reporting (I’ll be using “he” instead of he/she for convenience, but I have seen and worked for great managers of both sexes):

  • He should understand the importance of BCP. Having had the additional function plopped on his plate, he should endeavor to educate himself in the BCP mission and issues. He must learn to recognize the critical importance of BCP and be prepared to devote as much time and effort as necessary to accomplish success for his new reports. The “wrong” manager will think little of the importance of his new function and won’t even learn how to spell BCP, much less acquire the knowledge to speak intelligently to senior management about it.
  • He should be well versed in the corporate tribal knowledge. Knowing the inner workings of the organization as a whole will enable him to guide the BCP staff in accomplishing their mission. He will know how things are supposed to work, how they really work, and understands the difference. Each company is unique and speaks their own language (I once was corrected at MBNA that they referred to employees as their “people”, not “staff”). He will also know who the right decision makers are, how to approach them, how and when to report status to them, etc.
  • He must be able to open the right doors for BCP. A successful BCP requires access to highly placed decision makers and the new reporting manager must be able to open doors to the executive suite at appropriate times and with actionable information they need to see. He should command their respect and use that influence to garner a spot on their calendar. He will also know what BCP needs to attain visibility, such as which meetings to be invited to, which email lists to be included on, what memos to read, and how to acquire a place on other managers’ agendas. He should be able to smooth the way for his new staff to interact with senior leaders of other areas that are important to the success of their mission. Sort of virtue by association.
  • He should be a good manager and coach. He will have to evaluate the strengths and weaknesses of his new staff and determine how best to deploy them. In some, he will recognize their ability to interface successfully with senior management. Others he will determine are better suited to background roles. He should also evaluate their technology knowledge, writing ability, and company business awareness and make plans to fill in the blanks, perhaps using his education budget. As a coach, he should be an encourager to keep spirits up in the face of the adversity they will face and keep the focus sharp on the objectives that will make BCP successful.

 

Placing a BCP function in the right reporting structure and under the right manager is worth the time and effort and will go a long way toward enabling their success. Both where, and to whom, should be equally considered, and every company is different. I wish you success as you make your determination and welcome any comments on this blog. My contact information is also on this site and I welcome any opportunity to help create world class BCP organizations.

 

BCP function: staff from within or hire from outside?

As usual, my answer is: it depends. Are you looking for a manager to build a new BCP function from the ground up, or are you looking for someone to take an existing function to the next level? If it is a new function, prior BCP experience can be very important. A good outside candidate will have a proven track record you can verify. He should have professional credentials, like a CBCP (Certified Business Continuity Planner), and solid references from their experience at building and managing a successful program. If, however, you are looking for someone to raise the existing function’s game to a higher effectiveness, I would first want to know what management thinks are the shortcomings they want to improve. So let’s noodle this thing through and see what we can come up with.

The benefits of hiring from within. First on the list of pros is the fact that the candidate is a known quantity with a proven track record. Check out my previous blog on where and to whom BCP should report for guidance, but a good start is finding a good manager. Frederick Drucker, my all time favorite management consulting expert, once opined, “A manager is a manager is a manager”. His point was an excellent manager can manage brain surgeons or engineers or human resources. They may be different fields, but the mechanics of good management are transferrable. A good manager will figure out the skill sets and knowledge required and create an atmosphere where his staff can shine. His job is to always keep in mind the big picture and understand the underlying requirements for a successful implementation. Second on my list of pros is someone who possesses deep company tribal knowledge. As outlined in my previous blog, every company operates differently and has their own tribal language. An experienced manager will understand how decisions are made and by whom, how things really get done, and can make sure the BCP vision is properly communicated at all levels of the organization. All this can be hard to discern for an outsider, valuable time can be lost, and mistakes that hinder success will be made. Lastly, I would look for someone who is a quick study. BCP is not rocket science, but there are many things to learn concerning methodology, priorities, tools, and raising awareness of the importance of BCP to the company’s overall risk management activities. Finding the right candidate internally can go a long way toward starting on the right foot and propelling the BCP function toward success. These points are especially true when taking over an existing, underperforming function. However, if BCP is a new effort, hiring inside has some downsides.

The cons of hiring from within. For a new function, a major hindrance would be if the candidate lacks any BCP background. At the risk of negating what I said above, being thrown into the BCP pool without prior experience can be daunting, however quick a study the candidate is. Creating a new BCP function from the ground up requires vision and a deep understanding of how to address the recovery requirements of the organization in the right priority and with the right solution set. Navigating the mine fields without significant BCP knowledge can be difficult, at best. In this situation, I would suggest buying a block of hours from an experienced consultant to help develop the Program. (Big surprise, coming from someone who makes a living doing just that, huh?) For existing functions, the cons are a little different. If the internal candidate continues to maintain significant responsibilities for other functions, as is too often the case, his efforts will be diluted and success will be harder to achieve. BCP is a big hat to wear and unless he is given the freedom to focus on the issues, the improvements of the Program’s effectiveness that management is looking for may be long in coming, if they show up at all.   Also, if the candidate has spent all his time in only one silo at the company, such as the data center, he may have trouble expanding the BCP function corporate-wide, taking it from mere disaster recovery to true business continuity.

The benefits of hiring from outside. If there are no suitable internal candidates readily available, then going to the marketplace to obtain outside talent can be the right move. Obviously, the first benefit is you are able to buy experience that someone else has paid for. You have resumes and references to check out and thorough interviews to conduct, but if you do your vetting correctly, you can hire someone who has exactly the right background and ability to be successful, whether it is a new function or for a BCP Program that is in trouble. It’s important to set up mentoring to teach the new employee the lay of the land, but the right candidate will find his way. A second important benefit with an outside hire is the new set of eyes he brings to the situation. The outsider is exactly that, someone who should not be limited to “We’ve always done it that way.” A fresh approach may be just what the doctor ordered. Lastly, a new hire starts with a clean slate. Having no prior history with the company, he will benefit from being given a chance to prove himself without the limitations of past performance. Most employees will give the new guy or gal the chance to be successful and some will actually be rooting for them.

The cons of hiring from the outside. All the benefits of hiring from within, as discussed above, are flipped with an outside hire. The new hire has no tribal knowledge, no proven in house connections, and no broad understanding of corporate functions and priorities. It’s all new to them and they may have problems navigating toward success. Also, despite all your efforts to properly vet the candidate, some surprises may be in store. Unfortunately, you may soon figure out why he was looking for a new job in the first place, and for reasons that didn’t come up in the interview process. However, if you’ve done your job vetting the candidate, any problems should be easy to correct and the new hire can be a great boon to the organization. Access to a good mentor should make for a smooth transition and produce the right environment for success, but it doesn’t always work out the way it was intended. You pay your money and you take your chances.

Hiring the right candidate for any position is always an involved and sometimes difficult process. Finding the right person to be responsible for creating or improving the corporate BCP Program is definitely worth the effort. Hopefully, this blog has been helpful. As always, feel free to leave comments or contact me directly. Happy hiring.

 

 

© Copyright and All Rights Reserved Howard M. Peace

 

The Business Impact Analysis (BIA): Building the Business Case for BC/DR

I admit it, I’m a BIA bigot. I’ve been doing them since 1980 and I’ve never found a more effective tool in building the business case that convinces senior management that BC/DR is important, worth the investment, and essential to corporate survival. A properly conducted BIA will provide the following business case data:

  • A more thorough understanding of the business and all of its components. Think of it as stepladder in the cubicle farm. Most organizations are fairly complex and consist of silos of business and technical activity. The BIA should include interviews with each component and will provide an overview of essential activities that will need to continue in the face of an unexpected outage. It will also gather in one place, sometimes for the first time ever, the identification of the complete technical resources required to support ongoing business processes, the interrelationships between business functions, and an understanding of the progressive degradation of critical functionality over time.

 

  • The impacts of outages on critical business functions. The BIA should be used to measure how well a business area can continue operating with the loss of some or all of its critical resources. The impacts should be estimated using parameters on operations, revenue, costs, obligations, customer impact, and corporate reputation. The effects should be measured over time so the impacts can be graphed and presented to management.

 

  • The identification of critical business processes. Every department has a mix of critical and non-essential or deferrable functions. The BIA interviews should be used to separate out those activities which support the most critical activities from those that a delay would have little or no impact. Once the most critical activities have bubbled to the surface, a recovery strategy to address those in a timely fashion can be developed. These critical activities are usually the ones which have the highest level of impact, whether it be financial, operations, or negative customer effects.

 

  • Recovery time objectives (RTO). A general rule of thumb is the more immediate the recovery, the more it costs and the more complicated recovery becomes. The BIA should identify the timeframes in which the outage increases the impact on critical functions to an unacceptable level. In the BIA methodology I have developed over the years I use an impact scale that rises from a score of zero (no impact at all) to a five (OMG, hair on fire, this is really, really bad). A good interviewer tries to talk the business representative off the ledge (Is it really a 5 at a half day of outage, or maybe only a 3?), but there will be activities that rise to the highest level of impact in a very short time. This information allows me to draw a pie chart that shows which functions require, in effect, immediate failover recovery and those that need to be recovered in the 24, 48, 72, and 96 plus hours timeframe. The chart becomes a great tool to use in the development of recovery strategies to meet the timeframe requirements, perhaps using different options for each slice.

 

  • Identification of gaps in the current state recovery capabilities. By now we know how fast the various critical functions need to recovered to reduce the effect of an outage. It’s time now to examine the infrastructure’s ability to meet those demands. For example, let’s say you discover a critical function must be recovered at an offsite location in 4 hours or less, but there is no server available there to support such a quick recovery. Or you uncover the fact that the database needed to support the application will take 96 hours to recover from tape, blowing your RTO out of the water if the most current data is not already there. Or you discover that failing over critical calls to another call center would extend wait times to an unacceptable level, thereby violating contract agreements. The good news is the BIA has identified gaps in the recovery capability that can be addressed. The Gap Analysis section of the BIA should include the costs and implementation efforts for the remedy and will provide the business case for doing so in a timely fashion.

 

  • Input for the development of future state recovery. This is the part where a good BIA marries the critical business activities and outage information, the RTO’s, and the Gap Analysis to create a high level roadmap of where the organization should be heading if they want to increase their recovery capabilities to an acceptable level. The data from the BIA can be used to influence both infrastructure and business planning. For example, the shortfalls in the Gap Analysis might be addressed by an upgrade in the offsite capabilities, a change in the timing and content of backups of critical data, or a move to virtual servers. It’s possible an organization could now justify splitting the data center between two locations with the capability of mutual recovery for critical resources. Or they could decide to use the pie chart to plan to provide for immediate failover needs in a co-location space while using a cloud arrangement for the more deferrable recovery requirements, thereby saving cost without increasing risks. Likewise, the BIA data can influence business planning to support decisions to outsource certain functions, split physical locations to reduce risk, or to simplify certain business processes.

With all the data collected, analyzed, and digested, a business case can be developed that outlines the threat, presents the business impacts over time, details the timeframes for recovery of critical functionality, identifies the gaps in recovery capabilities, and provides the makings of a high level roadmap that includes the effort and expense of increasing recovery capabilities to an acceptable risk posture. If you’ve done an effective BIA, the way forward should be much clearer and you’ve been able to build a persuasive business case that gives senior management the data it needs to make a good technical, business, and financial decisions.

Please feel free to leave comments on this blog or contact me at the number and email address above. Enjoy doing a BIA!