Monthly Archives: July 2015

BC/DR and the Corporate SDLC

Last year I had the opportunity to meet a senior member of a former client’s executive management team while we both spoke at a conference in Miami. I began by asking him if they still did a Peace Business Continuity BIA before any new system was put in place. He smiled at me and said, “So you’re the guy who worked with so and so back in 2000.”

One of my priorities then and now was to insert BC/DR considerations into the System Development Life Cycle (SDLC) so they would be addressed up front. There are some very important reasons for doing this.

Every major change should be preceded by a focused BIA.   A properly conducted BIA for the change will identify what new pressures will be brought to bear on the current state recovery strategy. New applications, business functions, and infrastructure changes can have a discrete and, perhaps, debilitating impact on disaster recovery strategies and business continuity planning. The BIA should be used to measure outage impacts for the new change and be used to measure the overall impact on recovery requirements. If the SDLC includes a BIA hook, it’ll happen.

Each BIA Impact Category should have a list of recovery requirements. For example, at the client mentioned above, I used the categories we developed in our Y2K work to separate the most critical (AAA) from the more deferrable (D). I then used the categories to delineate what recovery requirements each new application needed to include in their planning. For the AAA through B ratings, the developers were required to include local failover (multi-processor environment), remote failover at the recovery instance, written recovery plans for the application and business units supported, and annual failover recovery testing. Putting those requirements early in the SDLC made for far better upfront planning and trying to retrofit them later would have been impossible. It eliminates the ‘but nobody told us” excuse.

BC/DR considerations must be included in the budget. The final price has to include provisions for recovery from the very beginning because there will be no money available later. For example, a new AAA application will be required to run in a multi-processor environment for local failover, not a single-threaded server, and another one is needed in the recovery environment. Now we’re talking about two specialized servers, not one and some possible impact on the network response time. The project budget should include this additional capability. Pricing things right in the beginning will mean an accurate assessment of the true costs.

A Gap Analysis should be performed to determine the overall impact of the change. The current state recovery capabilities should be examined to determine if the change will open any gaps in the company’s ability to recover critical business functions in the timeframe required. The change can impact the data center, data and voice networks, and business area recovery, or all of the above. Any gaps discovered can affect budgets and project delivery timeframes and priorities.

Here is what you’re trying to accomplish: At some opportune point between the time some executive says, “That’s a great idea” and the time they reach for their wallet, you want BC/DR considerations to be addressed so the real price is known and the implementation plan is written accurately. If there are BC/DR hooks in the SDLC, all the recovery issues are addressed in a timely fashion and the fixes won’t need to be bolted on later, if at all. As always, leave some comments or contact me directly if you’d like.

© Copyright and All Rights Reserved Howard M. Peace

 

Decision Making At Time of Disaster

Every organization has their own structure and process for making decisions, especially those concerning the spending of money. However, when faced with an unexpected outage or disaster, things can change, sometimes unpredictably. Normal management processes can be completely disrupted depending on the nature of the event and the availability of the normal decision makers. For example, on 9-11, I had the exalted title of Director, Disaster Recovery and Business Continuance for a major technology company. Soon after the event began to unfold I found out that the CEO and CFO were on vacation together with their families and no one knew where. With air travel stopped dead and cell coverage overloaded, they were completely out of touch and unable to direct efforts to respond to the crisis. Additionally, almost all of the other senior management team were visiting a potential acquisition in another state and they were off the table also. In effect, the Facilities Manager and I were the only senior managers available to lead the crisis management efforts, and we were huddled around a small black and white TV trying to find out what was going on in the world. We soldiered on and made financial and management decisions normally above our pay grades and received confirmation that they were the correct decisions several days later when management came back on line. Afterward, I thought of a couple of items that should be included in our planning efforts. They are:

Management Succession Plans. In order to provide for the unavailability of key decision makers in a crisis, either personal or corporate, every manager should have a written management succession plan. Key leaders should formally identify who exactly should take over for them in their absence and what special authorities should be granted to them. (Remember Al Haig declared he was in charge when President Reagan was shot until someone reminded him that’s not what the Constitution says.) I developed my methodology when I was called in to consult with a family-owned Southern California Bank to write a plan for their IT Director. The Board had only recently found out his hobby was flying small acrobatic planes and they were worried that, God forbid, he crashed, no one knew what exactly he was working on or how to manage his department in his absence. After assuring him the Board wasn’t looking to replace him, I was able to work with him to develop his first written job description, identify key staff who could assume most of his time-critical responsibilities, develop a written status report format and schedule, and improve his future planning documentation. I also interviewed him about his decision making methods and war gamed some scenarios to expose how he would likely respond if he were on duty. Lastly, I arranged for another senior manager to spend two days per month sitting with him getting up to speed on current state developments. In the end, the Board felt they had closed a major gap in their recovery planning efforts and used the methodology to create plans for other key roles as well.

Refined Tabletop Exercises. I am a firm believer in the usefulness of tabletop exercises in testing all facets of disaster recovery and business continuity plans. The exercise is a conference room based walkthrough of how a company will respond to an unexpected outage scenario revealed at the start of the exercise. The one wrinkle I always include is the unavailability of at least one key decision maker (or key technical guru). If the scenario is imagined correctly, their absence will be used to identify areas of expertise, tribal knowledge, or key decisions which are not documented or delegated and are sorely missed. In my experience, the following often comes up if the key decision maker is not available:

  • Who has approval authority now for the emergency expenditures? (If the backup has a much lower limit, how does that get bumped up to cover the spend?)
  • Who has authority to increase corporate credit card limits for the traveling recovery staff? (Do they even have corporate cards? If not, how do we expedite this?)
  • Who else can authorize increased system access authority levels? (Who thinks through the security risks?)
  • Who approves the PR statement that goes out? (Think about a canned statement with blanks to fill in with actual details so the word smiths don’t take 2 days to invent the wheel.)
  • What do we say about casualties and injuries?
  • Can we send the hourly staff home and still pay them for the day? (On 9-11 I used HR’s snow day policy to authorize this. I put on my Pirates of Penszance Admiral’s hat as the guy in charge and declared it was snowing outside!)
  • Do we still pay people who can’t come in because of the disaster? (What is our normal policy and if we want to make an exception, do we have the authority to do that?)
  • Who can authorize unexpected overtime?
  • What are our legal and contractual obligations and can we get some mercy?
  • Do we notify our customers yet, and do we have to report this to a regulating authority?

If the people who make these decisions, and more, aren’t available, is there a management succession plan in place which delegates the authority to make the call in their absence? Don’t worry if you uncover missing elements, that’s what the exercise is meant to do: expose gaps that can be remediated.

Conclusion. Without good management succession plans in place the chances are that good decisions will not be made or even thought about until it’s too late. As a father, I have seen the truth in the old saying, “One child, one brain. Two children, half a brain. Three children, no brain at all.” (In extreme pressure situations, some managers tend to revert to this in the middle of “groupthink”.) In order to avoid confusion, conflicting agendas, and the Al Haig Syndrome, make sure everyone knows how decisions will be made in an unexpected event with key decision makers absent. As usual, leave comments here or contact me directly.  Happy decision making!

 

© Copyright and All Rights Reserved Howard M. Peace

 

Business Continuity & Information Security: They Need Each Other

Having played in both of these fields, I have become aware of the symbiotic relationship that many overlook. Both are important aspects of corporate risk management and should be natural allies. In the early eighties I managed both as the Data Security Officer for Manufacturers Hanover Trust in New York City. As one of the first CISO’s in the country, I came to understand the importance of both disciplines and had a ring side seat as they grew to maturity. In those early days the emphasis in Infosec was password changes (remember RACF on the mainframe?) and violation reports (Al Gore had not yet invented the Internet and PC’s were an oddity. I still have one arm longer than the other from carrying an Osborne around). Business Continuity barely existed and primarily the focus was limited to disaster recovery of the data center. In their current states, they influence and help control all aspects of the business and their responsibilities are now 24/7 and reach around the world. As they have matured, the playing field has changed, but their need for mutual cooperation has remained. Let me show you what I mean.

Infosec depends on good business continuity. Infosec needs a reliable platform that includes continuous availability of the tools they need to monitor and control access to the systems.

  1. Without a reliable recovery program in place, a denial of service attack (DOS) brings a company to their knees in a hurry. In effect, the DOS triggers a business continuity event. Infosec needs quickly restored access to determine the cause of the outage and repair whatever damage has been done.
  2. If the backup procedures are not correctly implemented, rebuilding the systems becomes problematic and can take much longer as they search through previous generations to find an uninfected version.
  3. If telecommunications can’t be quickly restored, the external (and internal) threats can’t be efficiently managed and the Infosec team will have trouble coordinating their efforts. (That’s why when planning recovery efforts I always want to know how the techies communicate with each other (phone, cell, email, IM, etc.) because those tools become critical to recover as quickly as possible.)
  4. A good BCP will identify any new access requirements that are necessary to accomplish recovery efforts. For example, new or increased VPN access, higher or increased level access authority, emergency contact information, increased approval authority, etc.

Business Continuity depends on good Infosec. Remember the old John Wayne flick The War Wagon? The plan of attack against the armored stagecoach was to force it off the safest path onto a trail where the security efforts could more easily be thwarted. Exercising a recovery plan can open the organization to increased security threats.

  1. During the recovery process user passwords are sometimes shared, defeating individual responsibility controls. “Joe’s not here, but I know his password” can sometimes lead to unauthorized access that exposes the company to fraudulent activities and the disclosure of information thought to be secure. (Threats include disclosure of Personal Identification Information (PII), credit card numbers, access to approval levels to release payments, etc.)
  2. Admin passwords can be used to bypass normal change management control procedures, leading to mistakes. “It’s an emergency, so we don’t have time to test this first, just go ahead and move it into production.” Famous last words as the system crashes, delaying recovery.
  3. System techs will sometimes leave remote access ports open to make their job easier during recovery, thereby exposing the systems to outside threats. Hackers love open windows.
  4. There can be pressures brought to bear to circumvent normal access approval procedures that require written, and sometimes dual, signatures. “The boss said it was alright, and it’s only temporary.”
  5. There’s never enough time to review access violation reports. Many strange things can go bump in the night when the watchers aren’t watching.
  6. Physical access at the damaged facility can get loose allowing unauthorized parties to wander around to see what they can find and what they can do. The card key access system is down, so doors get propped open.

As you can see, there is a great need for BCP and Infosec to work closely together to address the threats posed by security incidents and unexpected outages. They may be separate silos but there had better be a good bridge between them. As always, your comments are welcome and feel free to contact me directly. Be safe out there.

© Copyright and All Rights Reserved Howard M. Peace

The 5 Best Reasons for Using A BCP Consultant

Of course my favorite reason is because that’s how I make enough money for my family to eat and sleep indoors. But really, there are some very good reasons to reach outside your organization for subject matter expertise and experienced help in accomplishing your BC projects. The best reasons I can think of are:

  1. A lack of internal expertise. It may very well be that no one at your company has ever done a BCP project and there isn’t a clear understanding of how to go about accomplishing the assigned mandate. A directive has come down from on high to “go forth and business contingify, whatever that is, and, by the way, you have six months.” An experienced consultant can develop project focus, detail all the steps required, develop a reasonable schedule, and accomplish the tasks in the allotted time frame. A good consultant will also major on knowledge transfer rather than building dependent step children.
  2. A lack of internal resources. It’s especially true in smaller organizations, but most companies operate on a rather lean staff budget and they don’t have a pool of technical and business people sitting around with large open areas in their job description just waiting to take on a project of this magnitude. The leader of the project usually has an already crowded plate and nothing gets removed to make time for this new responsibility. A good consultant is a dedicated resource who can focus entirely on the project and should bring not just direction but strong shoulders to carry the bulk of the weight. I tell prospective clients, “I hire people to do things like wallpaper the kitchen because I have too many Howard things to do. Assign me part of your to do list so you can focus on the stuff that only you can do. Give me this project and I’ll get it done, and make you look smart for hiring me.”
  3. People tend to cooperate with outside consultants. The staff usually realizes the company is paying for this consultant and so they should probably cooperate so whoever brought them in doesn’t get a report that they have become a roadblock. Like many families, they treat outsiders with a little more deference than they would one of their own. This is not always the case, but a good consultant recognizes the obstacles and uses his powers of persuasion to get on calendars, run meetings efficiently, gather information as painlessly as possible, and honor everyone’s time pressures.
  4. A good consultant brings a deep well of experience. This BCP project is not their first rodeo, or least it shouldn’t be or you’ve got the wrong consultant. They may have even done similar projects at similar companies in the same industry. They should be able to provide insight on how your peers are doing BCP and be able to cross pollinate solutions and recovery methods from other industries as well. The consultant should be able to enrich the solution set with lessons learned on successful efforts performed elsewhere. They should have a clear understanding of the technology required to support the client’s recovery needs and priorities, a handle on realistic expectations for the time required for recovery, an ability to outline effective recovery strategies, and the ability to build the business case for BC/DR expenditures. He should also be able to identify a roadmap for the way forward to improved recovery capabilities (not just upsell opportunities for his firm). And, perhaps it goes without saying, he should have a good handle on project management metrics including adequate status reporting on the project’s progress and unresolved issues.
  5. Company budgets often have restrictions on hiring new staff. Even though the BCP project has high level support and has been deemed an important requirement for this year, the bean counters have convinced management that the people costs are the easiest and most important to control. The edict comes down, “No more hires this year”, and the door for adding skilled staff slams shut. You’d like to go out and find an experienced BC person to bring onboard, but that is not possible. However, consultant dollars often come from a different budget and since the project has a defined cost and duration, money becomes available to bring in an outsider to handle a short term project. When the project is over, he goes away and you are not saddled with the cost of an ongoing head count.

In a future blog I will discuss how to find and use a good consultant, but for now let me just encourage you to make use of the wealth of experience a good consultant has learned while someone else was paying for acquiring that knowledge. I realize mine is not an entirely objective opinion (my professional motto is, after all, ”Consulteo ergo sum” I consult therefore I am), but I believe the right consultant can be a tremendous resource for the success of your BC project. As always, I welcome your comments and please feel free to contact me. Good projects to you.

 

© Copyright and All Rights Reserved Howard M. Peace