By Randall W. Helmick and John H. Zemanek, Entergy Corp.
Planning, preparation and performance are the cornerstones of any response plan-a rather generic hypothesis that Entergy proved true after two hurricanes, Katrina and Rita, hit its service area in late summer.
The company’s deployment of their emergency response plan and recovery strategy reconnected critical portions of the nation’s energy infrastructure and 1.9 million customers by Oct. 15, a first step in saving much of the Gulf Coast region from an economic death spiral.
Planning for the Worst
For seven consecutive years, the Edison Electric Institute has recognized Entergy’s track record in dealing with disasters. During Katrina and Rita restoration, Entergy’s emergency response plan directed the activities of thousands of people-spelling out what had to be done. The emphasis was on dealing with the unforeseen, with both public and employee safety paramount.
There were no cobwebs on the plan when Katrina and Rita came. In April, the company had drilled for a Category 4 storm striking New Orleans with 20-foot flooding of the city. Additionally, in May, Entergy briefed local officials and media on its preparations for the coming hurricane season.
Click here to enlarge image
Improvements to the plan were made later in the spring. Entergy upgraded its employee evacuation plan, developed a web-based employee information site called StormNET and upgraded the external website-incorporating lessons learned from participating in the 2004 Florida hurricane recovery effort into the plan. The upgraded plan was tested in July when hurricane Cindy struck, knocking out service to 270,000 Entergy customers in Louisiana, the largest single event in terms of outages Entergy had ever experienced.
Following a Proven Plan
“There’s certainly a chance [Katrina] can weaken a bit before it gets to the coast, but unfortunately this is so large and so powerful that it’s a little bit like the difference between being run over by an 18-wheeler or a freight train. Neither prospect is good.”
-Max Mayfield, National Hurricane Center director
When it became likely that Katrina would strike Entergy’s territory, the company activated its system command center in Jackson, Miss., and moved a multi-disciplinary team to that facility. Command structure included additional transmission command centers in Jackson and New Orleans and distribution operations command centers at utility operations headquarters in Louisiana, Mississippi, Texas and Arkansas.
Hurricane Katrina took out 263 substations and 1,550 feeders. Shown here is damage to the Grand Isle substation. Photo copyright Entergy 2005. Click here to enlarge image
Command center personnel tracked the storm, coordinated preparation efforts, began recruiting outside restoration help through mutual assistance agreements with other utilities, and lined up safe staging areas for materials and crews. Additionally, notice to the public in the storm’s strike area began, and local crews in the strike area were put on alert. Logistics (like lodging outside the impact area) were arranged to support Entergy crews and crews from outside the company.
“A monumental task lies before us. But, day by day, we will recover. Many of us are not only members of the recovery team, but also victims of the storm.”
-Rick Smith, Entergy group president, utility operations
Katrina struck on Aug. 29 near Buras, La., as a Category 4 hurricane with sustained winds of 140 mph. The storm tracked to the north, just east of New Orleans, before final landfall on the Mississippi coast.
In its wake, Katrina left massive damage throughout southeast Louisiana, much of Mississippi and parts of Alabama. Katrina affected 37,000 square miles in Louisiana and Mississippi, roughly one third of Entergy’s service territory. Approximately 3,000 miles of the transmission system were knocked out of service, about one fifth of the 15,000-mile system. More than 28,000 miles of distribution lines were impacted. The massive storm also took out 263 substations and 1,550 feeders.
Flooding in New Orleans presented a daunting challenge to Entergy’s restoration crews, who had to be inoculated against certain diseases before entering flooded areas. Photo copyright Entergy 2005. Click here to enlarge image
Along with damage to its transmission and distribution networks, Katrina presented unique challenges. The company’s corporate headquarters in New Orleans was evacuated and subsequently suffered massive damage. Katrina’s flood waters immersed Entergy’s gas distribution facilities in New Orleans and effectively took out natural gas service to about 145,000 customers.
“Everything was on the ground.”
-Mike Murray, Entergy senior lineman
After Katrina had devastated Entergy’s transmission and distribution systems in Louisiana and Mississippi, the company put its recovery plan into action. Finding out how badly the system had been hit was the first priority. Damage assessment scouts, positioned in advance, began their appraisals on Aug. 30. Among areas most quickly identified: places where flooding or debris prevented access.
Katrina created the largest number of power disruptions from a single event in Entergy’s history. Following the response plan, crews began restoring essential services such as hospitals, law enforcement, fire departments, sewer and water utilities. An outside-in strategy focused on restoring power on the perimeters of affected areas, then working into more severely impacted areas.
Backbone feeders with major trunk lines that support large electrical loads were first priority. The initial assessment helped managers estimate optimal crew size, the resources needed and the time required to complete restoration.
“Fueling the trucks was an elaborate process. It involved 20 people, and was really something to see.”
-Joe Catalanotto, Entergy New Orleans manager, business & operations support
Katrina didn’t just leave Entergy debris to pick up; it also created a number of operations challenges, including issues of fleet management, crew dispersal and crew communications.
Providing diesel fuel and gasoline for restoration vehicles and equipment was a daunting logistical task. Flooding and outages complicated the procurement of fuel for restoration vehicles. But, when the supplies came through, it was common for staging areas to receive 20,000 gallons of fuel in a single day.
“It was like running a hotel,” said Matt Flott, Entergy’s staging manager at the Belle Promenade Mall staging area in Louisiana near New Orleans. “A lot of people there worked long hours to make sure the workers had good meals and a nice, safe place to sleep and shower. We tried to keep them as comfortable as possible.”
To safely house the thousands of workers, Entergy arranged for several “tent cities” to be built. Site managers estimate it took hundreds of support personnel to keep the numerous tent cities operating. Caterers were up at 4 a.m. cooking breakfast for crews to eat by 6 a.m. and preparing box lunches for restoration crews.
Personnel safety and security were also major considerations in planning restoration in the New Orleans area. Restoration crews that would be working in flooded, mosquito-infested areas had to be inoculated against certain diseases. Security personnel had to be assigned to crews entering areas where safety was a concern.
On top of these challenges, much of the area’s communications systems were out of service. Telephone service was non-existent; cell phone service was spotty. This made internal communication-as well as communication with public sector agencies-difficult. Communicating with vendors and processing invoices presented an additional challenge.
Employees manning facilities in the devastated areas performed under extremely difficult circumstances and showed resourcefulness in coping with the storm’s aftermath. Portable generators were brought in to provide power to command centers when necessary, and senior lineman Mike Murray even brought along his own personal chain saw to cut away trees blocking access to his work site. Often, restoration crews slept in their trucks.
Click here to enlarge image
Despite these extraordinary challenges, by Sept. 1, Entergy had restored power to a number of critical facilities in the New Orleans area. These included East and West Jefferson Parish hospitals, the Louis Armstrong Airport, Gretna’s water facilities, the Emergency Operations Center and the Jefferson Parish Sewage Plant. Gretna’s police and fire departments also had power.
By restoration day 12, crews had restored electricity to nearly 750,000 customers in Louisiana and Mississippi. More than 340,000 Louisiana customers remained without power, with the bulk of those concentrated in the New Orleans Metro area and Chalmette, Hammond, Bogalusa and Amite.
Another challenge involved assuring contracted restoration companies of timely payment for their services, in light of Entergy New Orleans’ bankruptcy filing on Sept. 23. In the midst of the devastation, business had to go on. This involved Entergy’s business continuity team reviewing, reconciling and processing invoices while it relocated office workers to facilities where they could do their jobs.
A Milestone Reached
By Sept. 15, service had been restored to 861,000 of the 1.1 million customers who had lost power due to Katrina. In addition, service had been restored to six of eight refineries between Baton Rouge and New Orleans. Power was subsequently restored to the other two. Of the company’s 25 large industrial customers impacted, 20 had been returned to service by mid-September and were running at some level of capacity. All transmission substations that were accessible and not flooded were back in service and restoration of distribution service was moving on schedule.
Lines serving large blocks of customers were restored next. Distribution lines serving neighborhoods followed, thereby restoring service to multiple customers. Individual services were restored last because fewer customers were involved, and, in the case of scattered outages, it often took more time to restore power to them.
Crews then moved into the more difficult phase of restoring service to severely damaged areas, especially those in flooded areas around New Orleans and south of the city. The work was physically, mentally and emotionally exhausting. Restoration crews dealing with flooding in Louisiana, especially near New Orleans, rerouted circuits around the flooded substations, and the extensive flooding in south Louisiana sometimes required trucks and other equipment to be barged in. Even when trucks finally began rolling through this devastated area, they had to carry extensive stocks of vehicle maintenance and repair items. It was common for restoration trucks to keep rolling with a dozen or more plugs in their tires-the holes punctured by storm debris. Perhaps the most telling aspect of Katrina restoration was that crews going into flooded areas in south Louisiana carried telephone numbers of local morgues, in anticipation of finding those killed by the storm’s fury.
Then Came Rita
“Hurricane Rita has devastated our electrical system.”
-Joe Domino, president, Entergy Texas
Just 26 days after Katrina struck, Rita made landfall as a Category 3 hurricane near the mouth of the Sabine River. This second hurricane created even greater damage to Entergy’s transmission system, severing the ties between generation and load east-to-west, and plunging the area from Conroe, Texas, to Jennings, La., into darkness. As the restoration workforce shifted gears to deal with Rita’s ravages, the “watchword” phrase quickly became: “Safety Trumps Speed.”
Hurricane Rita’s widespread destruction required the first-ever use of helicopters to assess damage to the distribution network. Helicopters were also used to help rebuild. Photo copyright Entergy 2005. Click here to enlarge image
Rita hit southeast Texas and southwest Louisiana with a vengeance that impacted every Entergy jurisdiction. With 800,000 new outages, restoration crews set about replacing almost 29,000 distribution poles and bringing 522 transmission lines and 715 substations that had been affected by both storms back into service. Creating 810,000 outages, Rita was the second worst storm in company’s history. It did massive damage to the transmission system, and damaged and isolated generation plants from the grid, impacting all four states Entergy serves.
The need to balance the transmission system’s load with demand necessitated several days of rolling outages for 142,000 Texas customers. Large numbers of the exhausted Katrina workforce mobilized to restore Rita’s damage, while restoration management teams faced additional huge logistical challenge that included dealing with material shortages Katrina restoration had created. Houston’s mass evacuation-and its impact on lodging for available restoration crews moving in behind Rita to repair the damage-complicated the restoration process.
Within 42 days following Katrina, power was being restored to customers at the rate of nearly 26,000 each day. Photo copyright Entergy 2005. Click here to enlarge image
Rita presented challenges that required innovative solutions in both processes and uses of equipment. When Rita hit, the two storms’ areas of devastation were essentially in competition for personnel, materials and other necessary resources. Rita’s widespread destruction required the first-ever use of helicopters to assess damage to the distribution network, although helicopters have been used previously to assess transmission damage.
“I don’t know what type of corporate award, Entergy, we can give you at CNBC, but we are giving you a symbolic one….While we were sitting around enjoying our three-day weekend I don’t think anybody at that company did. Talk about a Labor Day weekend, a lot of labor going on there.”
Power had been restored to 1.9 million customers and to virtually all industrial and commercial customers in the wake of back-to-back hurricanes. Within 42 days following Katrina, power was being restored to customers in Mississippi, Louisiana and in the New Orleans area at the rate of nearly 26,000 each day. In the 21 days after Rita made landfall, each day power was restored at an average rate of 38,000 customers in Arkansas, Louisiana, Mississippi, New Orleans and Texas. Many factors made that possible: being prepared, having in place a tested and proven disaster recovery plan, and having that plan executed by men and women with the experience and skills to make it succeed. That foundation let Entergy safely evacuate 1,900 employees and contractors from New Orleans area and extricate from its damaged facilities the equipment needed to manage the restoration processes.
The company had recruited the necessary manpower and managed the repair of the worst damage ever incurred on its system, restoring more than six times the previous highest number of customer outages. In fact, it was the largest restoration workforce in the company’s history. Additionally, the restoration management team met the daunting logistical needs of this army of 16,000 workers-in some cases competing for resources with the general public and government relief agencies. This was done at the same time the team was managing orderly shutdown and recovery of major parts of its generation and transmission systems as necessary to expedite restoration.
While juggling the logistics of this crisis, the company maintained business continuity following the loss of its corporate headquarters building and major damage to its IT infrastructure and maintained security for work forces in areas of civil unrest and environmental hazards. On the financial side, the company managed the bankruptcy of Entergy New Orleans and the potential impact on its suppliers. Throughout, there was never a compromise of company core values, especially safety and customer focus. By any measure of disaster recovery, these accomplishments can be viewed as unprecedented in energy industry history.<<
Randall W. Helmick has served as Entergy Corp.’s vice president of customer service support-utility operations since December 2003. He assumed duties of vice president transmission Jan. 2.
John H. Zemanek has served as Entergy Corp.’s vice president of transmission since 1998. After guiding Entergy transmission through significant changes brought about by the deregulation of the wholesale power market, he announced his retirement from Entergy in mid-December 2005.