Dark Lessons: Learning from the Blackout of August ’03

And it certainly played out like an act ofterrorism: Guests at posh hotels on Times Square flooded the sidewalksas backup generators failed. Streets became jammed as traffic lightswent dead. Airplanes were grounded. Frightened families flocked togrocery stores to buy all remaining bottles of water.

Before long, however, the word got out: Thiswas no act of terror, even though reports surfaced that Al Qaeda wastaking credit via chatrooms and other rumor-mill conduits.

The blackout was linked to a computer bug andold-fashioned human error, according to a subsequent investigationconducted by the US-Canada Power System Outage Task Force. AtFirstEnergy in Ohio, the nation’s fourth-largest, investor-ownedutility, an undetected software flaw caused the company’s computeralarm system to fail, the task force concluded. By November 2003,federal investigators found FirstEnergy and theMidwestern IndependentSystem Operator—the industry entity that oversees the regional powergrid—in violation of voluntary industry reliability standards. Amongthe investigation’s findings: Control-room operators should have beenbetter aware of computer tools monitoring the network, and FirstEnergyshould have done a better job cutting heavy trees near power lines. TheU.S. price tag: as much as $10 billion.

The questions

In the aftermath, ominous questions remainabout homeland security and the grid: What if terrorists target ourpower supply? What if our enemies wanted to knock power out to enhancea biological or physical attack? After all, such an outage would surelythwart emergency responders and health-care providers. It’s a scenariowith disastrous implications. And one not easily remedied, experts say.

The task force findings resulted in urgentpleas for Congress to get tougher on the security and reliability ofthe grid. But momentum for any action stalled amid legislativesquabbling over tax breaks and write-offs for the energy industry.Still, the Princeton, NJ-based North American Electric ReliabilityCouncil (NERC)—a group that develops reliability standards for theelectric industry—is implementing change.

Most recently, NERC sent a letter to EnergySecretary Spencer Abraham outlining steps that NERC and the electricindustry have taken to improve security on the grid in the wake of theAug. 14 blackout. In October 2003, NERC asked electric companies toreview their reliability practices to avoid future blackouts. Amongother things, companies were asked to ensure sufficient voltagesupport; strengthen communications among control area operators andreliability coordinators; and put emergency action plans in place to“arrest disturbances and prevent cascading.” NERC further reported toAbraham that the response rate among these companies was 100 percent.

When it comes to terrorism, threats to the grid break down into two simple areas: cyber attacks and physical assaults.

The network IT/Internet/Intranet concerns arevalid. In the six months after the 9/11 attacks, energy companies werecyberattacked at twice the rate of other industries surveyed, accordingto Cupertino, Calif.-based Symantec Corp., an information securitycompany.

According to a March 2004 report fromSymantec, energy was the fourth most-likely industry to suffer severecyberattacks, lagging only behind financial services, business servicesand health care. The energy industry’s rate of attack, in fact, remainstwice that of the high-tech industry (with 5.4 attacks per 10,000online interactions for the energy industry, vs. 2.4 attacks per 10,000interactions for high tech).

“While the blackout was not related to a cyberattack, it brought attention to the vulnerability of the system and

has encouraged companies to re-evaluate theircybersecurity,” saidGary Sevounts, director of industry solutions atSymantec. While industry is paying more attention to network security,more can be done, he said. Even though no major cyber attack has hitthe grid network since the blackout, there is a history of computerthreats to the energy industry. In January 2003, the Slammer wormcrippled two computer systems monitoring pressure and temperaturewithin a U.S. nuclear power plant for nearly five hours.

The energy industry has responded.Ironically, NERC adopted its first-ever, mandatory, industry-widecybersecurity standard on Aug. 13, 2003—the day before the blackout.(However, while software-related, the blackout itself was not caused bya systematic, network-based failure.) These cyberstandards requirecompanies to submit their network operations to annual reviews fromNERC; document all cyberassets and electronic entry points and theemployees who are allowed access; and establish minimum training forthose working with the grid networks.

“The blackout was largely caused by a failureof people to follow the rules,” said Lynn Costantini, chief informationofficer for NERC. “But we were working on cybersecurity issues foryears before the blackout. We were raising awareness starting with thebuildup to the Y2K software situation. Then, the attacks of Sept. 11only accelerated our efforts. We are determined to secure our datainfrastructure as an industry. We will continue to conduct constantrisk assessments and protect sensitive information. The threat ofterror adds a new dimension to all of this planning. We’ve always dealtwith hurricanes, tornados and even acts of vandalism. Terrorism, now,is an addition to that spectrum that we will always have to considerand always will consider.”

NERC has gathered compliance information forthe electric industry regarding its adherence to the mandatorycybersecurity standards, but declines to release them, citing securityconsiderations. It is not required by federal code to release them, butCostantini says she is “very encouraged” by the results. NERC is nowworking on an update of the cybersecurity standards that should beapproved by summer 2005. (Updates regarding public hearings on the newstandards will be posted at www.nerc.com.)

Ramnath Chellappa, an assistant professor ofinformation and operations management at the University of SouthernCalifornia who speaks regularly about IT and the grid, said thenation’s power operations would be more reliable if energy networkswere run in ways similar to the Internet.

 “With the Internet, even if part of thenetwork goes down because of an attack or an accident, packets may bere-routed through other nodes and will be very much alive,” Chellappasaid. “If our energy chain operated in a similar way—if one part of thegrid went down—demand would be served from another part of the gridautomatically. Theoretically, this is very possible. But are theeconomic incentives out there to work it out? This we have not seenyet.”

Getting physical

From a physical threat perspective, gridoperators still have their work cut out for them. In a recent analysisof site visits to power companies, four of five involved partiesreceived recommendations to repair, replace or enhance their cooling orbackup power equipment, according to San Diego-based InfrastructureDevelopment Corp., the consulting company that conducted the analysis.

Grid operators do a good job of protectingtheir core centers of operations, but the peripheral points are toovulnerable, said Steven Kuhr, who was working as an emergencymanagement consultant to the Port Authority of New York and New Jerseyat the time of the 2003 blackout. When it comes to the transmissionpower lines and the substations—those fenced-in, metal-fronted placeswhere power from thegrid is doled out to businesses and homes—moreoversight is needed.

“Look, you don’t even need an act ofterrorism to do some major disruption there,” said Kuhr, who is nowchief operating officer for Criterion Strategies Inc., a New York-basedthreat-assessment firm. “One time, in Rockaway, a Mylar balloon landedon a transmission line. Being that it was aluminum, it caused an outageand a local hospital lost power for a couple hours. So these systemsare vulnerable—terrorism or no terrorism. On the positive side, morecompanies are using closed-circuit TV cameras to monitor key powerlines and substations, but more can be done here.”

As an aside, Kuhr also strongly urged clientsto boot up their backup generators with a full load every month, tomake sure they’re working in case of an attack or disruption.

Room for improvement

Whether the threat stems from technology,physical assault or a combination of the two, experts say there’s stillmuch that can be improved. In recent years, energy companies have takenproactive steps with regard to improving employee background checks,inspecting lines from aircraft and tightening access control, saidJoseph Malatesta Jr., former chief counsel of the Pennsylvania PublicUtility Commission. But they’ve also been reluctant to impose strictsecurity standards industry-wide.

“They’ve instead opted for the ‘bestpractices’ approach,” said Malatesta, who is now a utility practicepartner at Saul Ewing LLP, a mid-Atlantic regional law firm based inPhiladelphia. “ This approach recognizes that ‘one size does not fitall.’ However, it also allows a lot of wiggle room for companies tospend resources on other priorities. Regulators must find the happymedium.”

He added that it’s clear that utilitycompanies are doing their part, as they realize it is simply goodbusiness. “They want to provide safe and reliable service to theircustomers,” Malatesta said. “They are very community-minded. They alsohave business interests to protect. Every blackout is lost money tothem. They have every incentive to make sure it doesn’t happen.”


Ultimately, any long-term impact on securityand reliability will depend on cost. But, too often, the immediateshock of a major blackout fades over time and, with that, the nationalsentiment for an overhaul fades.

“In terms of infrastructure improvements,it’s important to observe that, in the U.S., transmission enhancementshave lagged the generation growth,” said national energy authorityVijay Vittal, a professor of engineering at Iowa State University whodirects the Electric Power Research Center there. “As a result,investment has to be made to reinforce the grid. The electric grid isthe most complex machine engineered. Its sheer geographical expanse isimmense. As a result, no guarantees can be given that another blackoutof this magnitude will not occur. The question is: ‘What is one willingto pay for a certain level of reliability?’”

The problem is that history has taughtregulators and industry virtually nothing, critics say. A majorblackout in the northeast in 1965 was supposed to trigger a completereorganization of the grid to prevent future outages. But that didn’tstop the infamous 1977 blackout in New York State that lasted more thana day. No less than 10 western states lost power in 1984. Sabotagecaused a major blackout in San Francisco in 1997. Then Chicago lostthree of four transformers in 1999.

“We are told there have been improvements, sothis won’t happen again,” said Peter Neumann, a principal scientistwith SRI International, a Menlo Park, Calif.-based research outfit, andauthor of the book Computer-Related Risks (Addison-Wesley). “We havenot learned from the past. We are not improving significantly, and manyof the non-trivial risks are being ignored. The responses to date aremore or less like banning plastic knives on airplanes after Sept. 11.Many outages have resulted from accidents, but future ones could verywell be caused intentionally—by insiders with computer access oroutsiders. The physical plant is seriously at risk, and those risks arebeing largely ignored.” HST

(Visited 5 times, 1 visits today)

The Government Technology & Services Coalition's Homeland Security Today (HSToday) is the premier news and information resource for the homeland security community, dedicated to elevating the discussions and insights that can support a safe and secure nation. A non-profit magazine and media platform, HSToday provides readers with the whole story, placing facts and comments in context to inform debate and drive realistic solutions to some of the nation’s most vexing security challenges.

Leave a Reply