“It isn’t easy being thought of as Chicken Little and having to be constantly reminding people that they need to be prepared for emergencies before they happen,” Jim Grogan, vice-president of alliances at SunGard, a disaster recovery and continuity services provider based in Wayne, Pa., explained in a recent interview with HSToday
.“There’s a group of us specialists who haveto pay attention all the time to what might go wrong. We get paid tothink about the unthinkable, about catastrophes either natural ormanmade,” he added. “But let’s face it, we have to understand we’regoing against the grain of some fundamentals of human nature. Unlessthere’s a real emergency, most people’s natural inclination is to worryabout what’s number one and number two in their in-box that day, andplanning to maintain critical processes isn’t top of mind.”
Such is the conundrum faced by continuityplanners. People and organizations know abstractly how crucialcontinuity planning is in the post-Sept. 11, 2001, environment. Yetmany barriers—economic, organizational and psychological, as much astechnical—still make it tough to translate that knowledge intoeffective action.
“The challenge,” Grogan said, “is to makecontinuity training and techniques part of a rational routine, ratherthan a harried response to fear and panic.”
The West Virginia experience
The stakes are certainly huge. “Disruptiontraditionally meant lost revenue and market share,” observed BrianTurley, president of Strohl Systems, a developer of continuitytechnologies located in King of Prussia, Pa. “But now you’re alsotalking about potential cut-off of crucial public services, andendangering public health and safety. The difference between poor andgood preparation is felt in terms of human suffering and lives, as wellas dollars.”
In July 2001 the state of West Virginiapublic-health system, for instance, found itself in a chaotic situationwhen the most serious flooding in a generation hit the state. Asfloodwaters mounted, dozens of state and local health departmentsresponded in piecemeal fashion, with each pursuing its own proprietarydisaster-response plan. Though the Division of Public Health (DPH) atthe state Department of Health (DoH) was charged with coordinatingcommunication with each local health department in affected areas, theylacked the tools.
“First we found ourselves without emergencycontact numbers for many local health staff members,” recalled AmyAtkins, transitions coordinator for the West Virginia DPH’s Nursing andAdministration department. “Then we realized there was no provision forthe supply, distribution process, roles or responsibility for providingadequate tetanus vaccines, or other medicines and first aid.”
In this chaotic environment, at least sixWest Virginians died and many more were injured or exposed to potentialdisease due to water contamination.
“Though the floods were the worst to hit ourstate in decades, there was a terrible realization on the part of manystate health professionals that better planning would have preventedmuch of the suffering and dislocation,” Atkins said.
In the aftermath of the disaster, a specialstatewide group called the Invitational Roundtable on Public HealthPartnerships took up the task of making sure the experience would notbe repeated. “Over the next year, we met regularly, determined to poolevery available resource in the state to better coordinate planning foremergency response between state and local public health departments,”not only for natural disasters but for new threats like bioterrorism,said Atkins.
Two years later, when Hurricane Isabelthreatened West Virginia, the state system was much more prepared. Thedisaster network the roundtable had put in place was immediatelyactivated. DoH staff began calling and emailing local agencies withspecific instructions. The day before landfall, local healthdepartments distributed prepared emergency response procedures to thelocal media, moved vaccines to facilities with backup generator powerand conducted emergency planning meetings with key personnel in eachlocal department in the state via teleconference.
“This time,” said Atkins, “the entire heathapparatus of the state jelled together as a single organism. All localagencies in the state had ready supplies of vaccine handy, and astatewide database enabled relevant authorities to monitor supply, aswell as inform locales as needed where they could obtain backupsupplies.
Despite its literally life-and-deathimportance, focus on continuity has been erratic, ebbing and flowingwith fluctuations in the public mood.
Last spring, for instance, a widelypublicized report by the US General Accounting Office (GAO) found that,although 20 of 23 of the largest civilian departments and agencies haddeveloped and documented elements of a post-disaster continuity plan,none were fully compliant with Federal Emergency Management Agency(FEMA) mandates. These mandates, published in the most recent versionof Federal Preparedness Circular (FPC) 65, call for each public agencyto have well developed continuity plans for delegation of authority,provisioning for re-allocating resources or setting up alternativefacilities, interoperability of emergency communications and protectionof vital records and databases.
Upon release of the report, there was anoutcry in Congress, particularly from Rep. Tom Davis (R-Va.), chairmanof the House Government Reform Committee, who claimed the reportdramatized how major gaps in continuity planning made essentialgovernment functions unacceptably vulnerable nearly three years after9/11.
Despite such attention, funds for continuitytraining and technology for federal, state and local agencies remainrelatively hard to come by as investments go to higher- profilecounterterror pursuits.
“Continuity of operations has thus far beenessentially an unfunded mandate,” declared Chris Alvord, CEO of McLean,Va.-based COOP Systems and veteran consultant, teacher and writer ontopics of continuity and disaster-recovery planning. “There have beenlots of demands for rapid progress and lots of well-intentioned, andoften accurate, complaints about how unprepared our public and manyprivate institutions are, but until very recently the funding was goingelsewhere. It’s only been in the past few months that words have beguntranslating into money.”
Anita Crease, vice president of marketing forOpWatch, a business continuity services provider based in Rockville,Md., offered an explanation. “The first stage of post-9/11 fundingunderstandably involved buying things that obviously made a differencethat people could literally see right away, things like HAZMAT suits.Compared to that, the need to invest in critical processes that aren’tso easy to see is not so obvious. But it’s equally important to realsecurity.”
Grogan believes politicians and bureaucrats underestimate the nature of the problems involved in achieving these goals.
“DHS itself, whom we work with, has a totalof 350 critical systems within the department,” he said. “According toa recent audit, less than 40 percent of those had been assessed forrisk or prioritized for overall importance. And that’s just one exampleof the tremendous complexity you get in a large agency. Beyondidentifying critical systems, there then needs to be a coherent set ofmethodologies and procedures that cross departments within agencies.
“Federal mandates are useful,” he said, “inthat they are a way of expressing a sense of urgency and priority thatexists at the highest executive levels. But just giving broad goalsdoesn’t provide enough granularity for project managers in anorganization or agency to gauge exactly where they are.”
Alvord agrees. “FEMA has done a nice job attelling everyone where they should be aiming,” he said. “But that skipsthe trickiest part, which is the methods of getting there and tellinghow much progress you’ve really made.”
One area where progress has definitely been made is in continuity-planning support technology.
“The technology has changed dramatically,”said Grogan. “Five years ago, there were very few commerciallyavailable hardware and software tools and most of those werecost-prohibitive. Today, there are a variety of solutions around thatcan be tailored to need.”
The leading technologies attempt to helpautomate key areas of strategic continuity planning, including businessimpact analysis (BIA), business continuity planning (BCP), crisiscommunications and data protection/recovery.
BIA involves identifying the financial,operational and service impacts that may result from a catastrophicevent. This is done through development of a checklist and inventory ofkey functions and processes the organization performs and a critical-functions assessment survey determining critical processes in terms ofpriority to an organization’s core services.
Once organizations know what their criticalsystems and processes are, they can conduct a risk analysis. Thisinvolves documenting potential threats, internal vulnerabilities andthe likely consequences of various failure scenarios, includingestimates of their probability.
A business continuity plan documents themembership of the emergency-response team and delineates in detailtheir roles and responsibilities in relation to each key task to beperformed. It also involves setting up practice “games” in order toexplore a variety of disaster scenarios and to test the teams’responsiveness and effectiveness.
Related to, and generally included in, acomprehensive BCP are emergency incident management and crisiscommunications. Incident management involves tracking the status ofresponse to a declared emergency by each component of the emergencytask team on an ongoing basis. Crisis communication involvesdisseminating relevant information about a crisis throughout anorganization and/or between organizations to appropriate task teammembers according to the BCP model.
BCP software automates the process ofplanning by providing a single place to collect and organize continuityplans and templates or “scripts” designed to walk users through theirroles and tasks. The most comprehensive packages allow the BCPcoordinator to link assigned tasks to a central library.
As important as technology is to continuityplanning, technology is only as useful as the organizational knowledgeand commitment that supports it.
“Continuity planning is about analyzing,analyzing and analyzing some more and documenting,” said COOP’s Alvord.“When you’re dealing with large complex systems, learning curves areslow,” he added. “It’s not just a question of buying a certain productand then you’re done. It’s going to be a three- to five-year processuntil all the pieces of an effective continuity system are in place.And most of that work is very unsexy, undramatic trial-and-errortesting and retesting. The labor is in developing really detailed,explicit plans based on painstaking audits and assessments ofeverything that goes on in an organization. Without that groundwork,technology won’t help much.”
“The most successful continuity projects mayuse different technologies, but they always have one thing incommon—strong executive sponsorship from the highest level,” saidTurley. “That’s the only way real coordination between departments orunits can develop. Most organizations, especially government agencies,operate in solos. The IT department has its emergency plans, and allthe individual departments have theirs, whether good or bad. And it’samazing how uncoordinated they still are. In one test we ran involvinga disaster scenario, we found much to everyone’s embarrassed surprisethat multiple agencies had contingency plans to relocate theiroperations to the same off-site location. There were over 300 peopleplanning to go to a site which only holds about 50 people!”
Uncoordinated planning also costs money,according to Turley. “When seven or eight individual agencies in agovernment department, for instance, each do their own planning foroff-site facilities,” he said, “each of them will pay a much largerprice for the spaces than if they pooled together and made a bigger buyat a discount rate.”
Heroism is usually associated with highlyvisible demonstrations of courage and valor performed in situations ofintense drama. The irony of continuity planning is that the moreheroically the job of planning is done the less visible it is. The goalof continuity planners is precisely to avoid drama, and the better theyperform their job the more routine and unobtrusive their handiwork willappear.
Not too surprisingly, continuity planning hastended to be misunderstood and mistreated in most discussions ofhomeland security. In a culture accustomed to looking for quicktechnological fixes to problems and “show-me-now” results, thepainstaking nature of progress in the field tries people’s patience andshort attention spans.
This mentality must be overcome, for trueprogress in continuity planning requires levels of long-termcommitment, persistence and focus not traditionally maintained withinmany American public and private organizations. HST
Continuity planning tools and technology
The newest generation of collaborativesoftware enables continuity planners to meld their overall plans,emergency-management procedures and notification processes into asingle web-based environment that can be easily shared between agenciesand departments within an agency.
One such tool is the “Ops Planner” softwaresuite from OpWatch, Rockville, Md., which delivers business continuityPlanning (BCP), emergency-response management and communications overthe web and can be integrated with common Microsoft Office applications.
“We view the big challenge in state andregional homeland security as being cooperation between smallerfirst-response units,” explained Anita Crease, OpWatch’s vicepresident. “It’s not only not necessary, it’s counterproductive foreach county or town to pursue its continuity plans in isolation. Eachlocale has its own needs, but procedures need to be coordinated. Onebreakthrough we think we’ve made is to provide an easy-to-usecollaborative tool, which can be shared by many locales. Not only candifferent agencies or locales communicate during a crisis but they canbe developing, testing and updating their emergency plans andprocedures, and tracking progress, before an emergency.”
Another new solution, “MyCoop” software fromCOOP Systems, McLean, Va., offers emergency planners an expert planning“wizard” leading them step by step through the creation of up to eighttypes of disaster-recovery plans. An optional feature also enablesinstant wireless email notification via Blackberry or a similar deviceto recovery teams in a disaster.
The Living Disaster Recovery Planning System(LDRPS) suite from Strohl Systems in King of Prussia, Pa., integratesBIA, BCP templates, emergency management and notification applications.It’s also designed to store all continuity planning within anorganization (linking, for example, individual plans from multipledepartments) in a single database, a major factor in effectivecoordination, according to Deb Stafford, business communicationconsultant for the State of Minnesota.
“Working out of the state Department ofAdministration, I’m charged with coordinating the continuity plans ofseveral large state agencies,” said Stafford. “State agencies takepride in their autonomy for good reason, as they do very differentthings and have different critical processes to protect. So the naturaltendency of each is to develop unique, customized, internal continuityplans. Using a centralized suite like LDRPS, we can bring coherence toall those diverse plans in a way that wasn’t feasible a few years ago.Everybody now knows what everyone else is doing in an emergency, whichhelps prevent conflict, confusion and duplication of effort.”
New technology also has been developed for insuring critical data and data systems are not lost during a catastrophic event.
Software-based data replication offers aneconomical alternative to hardware-based data mirroring. For example,Veritas of Mountain View, Calif., offers storage-replicator softwareenabling organizations to centralize data backup without disruptingnormal server operations. The software automatically replicates remoteoffice data over an IP connection, either continuously or on ascheduled basis, to a central location where it can be reliably backedup and stored.
The “Windowless Data” Replication servicefrom SunGard, Wayne, Pa., allows organizations to continuously captureand replicate changes at the byte level. Using standard networkconnections the server replicates selected files or entire volumes fromone or more Windows servers.
Another task most government and privateorganizations are only now tackling is to educate and formally train ageneration of continuity experts.
To this end, Chris Alvord has developed andteaches a course at the US Department of Agriculture Graduate School asa possible model of professional education for continuity-of-operationsplanners. Students learn such fundamental skills as how to involvemanagers in launching COOP efforts, and are rigorously coached in theABCs of COOP, including methodologies for collecting information toconduct impact analysis, risk assessment, planning and task teambuilding, as well as choosing software and hardware.
New York University’s School of Continuingand Professional Studies has also launched a new professionalcertificate program in continuity planning. Program curriculum coversdeveloping and implementing BCP, emergency response and operations,planning exercises, measuring results and technology.
“In law, when you pass the bar you’vepresumably mastered a body of knowledge and unique set of proceduresthat is the same as that of your professional peers,” observed Alvord.“But that hasn’t been true of continuity planning as a discipline. Itneeds to be. We need to develop a similar transparency where everyonewith planning responsibility speaks the same professional language.”