(Editor’s Note: The dark photo up top is the Toronto skyline on Aug. 14, 2003; there’s a reason you can’t see much. At the bottom of the story is a timeline of how the blackout unfolded and, after that, a satellite photo showing the dark swath cut by the outages.)
One line, one tree. Eventually, one grid in peril.
That was the simple, mathematical momentum of the Northeast Blackout of August 2003, It did not start out as a spectacular, immediate failure of power and transmission. Yet it stealthily rolled on, wave by wave, into perhaps the second most widespread blackout in history. About 55 million people in eight states and wide swaths of Canada lost electric power for anywhere from a few hours to several days. Eleven people died directly or indirectly from impacts by the massive outage.
Silence is the ultimate weapon of power, Charles de Gaulle reportedly said. And the Northeast Blackout, by quietly taking power away, was lethal and resounding. It was a stunning, sobering wakeup call to those overseeing the U.S. and North American electrical infrastructure and the political powers guiding it.
“We are a superpower with a third-world electrical grid,” New Mexico Gov. Bill Richardson, who had been energy security during the Clinton Administration said at the time.
Thirteen years ago this month, the historic blackout started humbly enough. On a warm day a FirstEnergy 345-kV transmission line in Ohio dipped and made contact with some overgrown trees and faulted. An alarm system which should have alerted operators did not do so, so the high-voltage bottleneck rerouted itself and caused other lines to sag with the heat and snag on trees, leading to other faults.
But the FirstEnergy mishap was only one of nearly 800 events which contributed to the blackout that day. Multiple utilities and facilities were part of it, while grid operators PJM and MISO were also cited in the final report by the North American Electric Reliability Corporation.
In the moment, many officials weren’t sure what caused the blackout. One report blamed it on a fire at Con Edison plant, while others had ill-founded worries that it was a victim of the Blaster computer worm. Still another incorrectly pinned it on a presumed nuclear plant outage in Pennsylvania.
The cascading outages jumped into Michigan, Pennsylvania, New Jersey, New York, Toronto and other portions of Ontario. It caught many public officials by surprise—although Richardson said he had been warning about this threat for years—and forced the U.S. into a major re-valuation of its grid protection system.
The industry “had put in technology (by that time) to give them some visibility and understanding what was happening” on the grid, Brad Williams, Oracle Utilities’ vice president of product management, said in an interview with Electric Light & Power. But “the technology failed to update and gave no visibility to the operator.”
The U.S.-Canada Power System Outage Task Force, in its report nearly eight months after the blackout, was critical of FirstEnergy’s “little reactive margin” for error and cited “inadequate system planning” and operations staff poorly trained to deal with emergency situations.
“Analysis of Cleveland-Akron area voltages and reactive margins shows that FirstEnergy was operating that system on the very edge of NERC operational reliability standards, and that it could have been compromised by a number of potentially disruptive scenarios that were foreseeable by thorough planning and operations studies,” the Task Force wrote.
The electric power industry surely has tried to learn its lesson and avoid another Northeast Blackout scenario in the future. Federal regulators and utilities have worked together to birth a whole family of interconnected rules and security layers throughout the generation, transmission and distribution sectors. Sensors, synchophasers and customer communication efforts were all expanded in response to the blackout’s aftermath.
The U.S. Congress’ Energy Policy Act of 2005 codified mandatory standards for utilities and levying fines as much as $1 million per day on those who fail to keep their systems at top-level reliability. The federal apparatus and its counterpart in Canada also certified the North American Electric Reliability Corporation (NERC) as the standard-bearer to ensure safe and steady power against all threats, whether they be superstorms, computer hackers or overgrown trees.
Utilities got serious about vegetation management, cybersecurity and hardening the system.
“When companies saw what happened in 2003, they started to beef up their vegetation management programs. They knew that now if you spent more on vegetation management that (regulators) would allow costs to be passed through their rates,” Jim Fama, who works with the Edison Electric Institute on these issues, told EL&P. “Reliability coordinators beefed up their situational awareness capability”.They would do analysis of what would happen if you lost a big generator”.How would power flow? How would power reroute?”
The U.S. Department of Energy worked with NERC and the industry to develop the North American Synchophaser Initiative, vowing to bring the “grid into the 21st Century” with billions’ worth of investment. The synchophasers monitor and measure instantaneous voltage, current and frequency at specific locations on the grid, according to the DOE. This is part of what EEI’s Fama and others call “visibility.”
“Operators need to move quickly to shed load,” he said. “They have to black out small portions of the system to prevent disturbances cascading from one system to another.”
Three years ago, FirstEnergy built a $45 million transmission control center in Akron, Ohio. The Akron Control Center has massive video screens and advanced computer systems to provide a deeper look at what’s going on system-wide, company spokesman Mark Durbin pointed out.
Load-shedding equipment also has been installed, giving FirstEnergy the potential to quickly reduce about 1,300 MW off the lines. Retired generating units at the Eastlake Plant were converted into synchronous condensers which can help stabilize voltage.
“Over the past 13 years, FirstEnergy and the transmission industry as a whole have added new facilities and equipment, enhanced operator training, and implemented new NERC operational standards that are mandatory and enforceable in order to help enhance the reliability of the transmission system,” Durbin said.
Yet, despite the moves made to enhance protection continent-wide, many warning shots have been fired across the bow of the industry in the intervening 13 years saying it needs to do more. A 2008 report by the NextGen Energy Council cautioned that an even more crippling blackout was possible, due to lower baseload generation capacity reserve margins and lagging reinvestment in transmission infrastructure.
Infrastructural wakeup calls are the norm in past decades: the 9/11 attacks, the California electricity supply crisis, the Northeast Blackout, Hurricane Katrina and, more recently, Superstorm Sandy and the Ukraine grid cyberattack. Those who oversee our grid security have been forced to think about doing battle on all fronts to protect the system.
Meanwhile, there is a revolution going at the “edge” of the power grid that interacts with customers. Thousands, perhaps millions of residents are adopting distributed generation sources such as rooftop solar which force the power to flow bi-directionally, while the Internet of Things connectivity and smart meter technology expand the communication and potential for cyber threats.
“More than ever, the systems have to be able to scale up to support major events,” Oracle Utilities’ Williams said. “They cannot lock up and they cannot fall on their face.”
In addition to NERC and its CIPs (Critical Infrastructure Protocols), federal regulators and the industry have added numerable layers of oversight over the visible grid. This includes bodies such as the Electricity Information Sharing and Analysis Center and the Mutual Assistance Network (MAN) program within EEI.
Mutual Assistance Networks help coordinate the flow of workers from unaffected utilities to help those in need during a weather or blackout crisis. Industry insiders also are working to develop a similar pact on the cyber front.
Nearly four dozen companies are involved in the cyber MAN, including investor owned utilities, Regional Transmission Organizations and coops. These would make available a pool of potential cyber workers to come in and help a utility or region hit by a computer meltdown or hacking incident.
“We’re trying to do on the cyber side part what we do on the physical side,” Fama said.
Williams stressed that the industry needs to step up its exercise—as in game-playing to simulate crises such as blackouts, devastating storms or cyberattacks.
“Utilities need to practice major events, particularly if they haven’t in a while,” he said. “We need drills hitting operators with a large number of events”such as if a critical system fails to update”.What are the possible failure modes in a major event?”
All in all, North American power industry leaders learned much from the Northeast Blackout of 2003. NERC CIPs, sensors, synchophasers and grid security exercises are all evidence of that. “I think we’re headed in the right direction,” Fama said.
Could a Northeast Blackout happen again? Of course it could, judging from smaller, but still impact subsequent meltdowns in Florida and the southwest U.S. since then. But industry insiders are more confident that visibility and protection schemes have improved dramatically.
“I think we are in a better position. Utilities are more prepared; if they’re not prepared they will get fined,” Williams said. “We’re more dependent on the technology so make sure it’s going to be there when you need it.”
Timeline of 2003 Northeast Blackout
Time Date State Event Capacity Impact
12:05-1:31 p.m. Aug. 14 Ohio Three generation units trip More than 1,700 MW
2:02 p.m. Ohio Stuart-Atlanta line disconnects 345-kV
3:05-3:31 p.m. Ohio Three more 345-kV lines disconnect
3:45-4:08 p.m. Ohio Two more 345-kV and 138-kV lines disconnect
4:08-4:10 p.m. Michigan Kinder Morgan generation trips 200 MW
4:10 p.m.: Twenty generators along Lake Erie shut down 2,174 MW
4:10 p.m.: Four transmission lines disconnect between Pennsylvania and New York
4:10 p.m.: New York-New England lines disconnect (New England islands, stays operational)
4:11 p.m.: Ontario system separates from New York west of Niagara Falls 2,500 MW
4:12 p.m.: Remaining lines between Michigan and Ontario separate
4:13 p.m.: Cascading sequence essentially complete