“Calamity is man’s true touchstone.”
-Fletcher & Beaumont,
The playwriting team was right; disasters are a benchmark for people and their organiza-tions. Disaster reveals the true mettle of someone or something by forcing them to respond to the challenges they face.
In late August 2005, Hurricane Katrina formed in the Atlantic basin and for six days battered the Gulf Coast. Shortly thereafter, in early September 2005, Hurricane Rita blasted the same area. As if the Gulf Coast hadn’t seen enough, the weather rose up again in October 2005 and Hurricane Wilma blasted Florida, Mexico and Cuba. All three hurricanes were rated as Category 5 hurricanes, the most powerful rating a hurricane can get.
In a report compiled for Electric Light & Power by Dave Hurst of the Joint Information Center, (which works with FEMA) in Austin, Texas, the destructive force of Hurricane Rita is laid bare. The report lists approximately 1 million customers in Texas alone as being without power. Entergy’s transmission system in Texas had 279 lines and 284 substations out of service after the hurricane hit.
During Hurricane Katrina, Entergy saw massive damage, with 3,000 miles of the company’s transmission system out of commission and 1.1 million of its customers in the dark. The National Hurricane Center set the damage cost of Katrina at $75 billion.
Barry Moline, executive director of the Florida Municipal Electric Association, told EL&P that the municipal utilities that reported damage totals suffered almost $5 million worth of damage to their T&D infrastructure during Hurricane Wilma. “If you want to total damage from all the municipals in Florida for the hurricane season, you could go with $7.5 million and be in the ballpark,” Moline said via e-mail.
responding to disaster
For a utility, an outage management system (OMS) is one key to restoring power. Though they can be costly, an OMS “pays for itself” when hurricanes like Katrina, Rita or Wilma hit, said Terry Nielsen, an outage management expert at SPL WorldGroup.
In a recent interview with EL&P, Nielsen explained the general features of an OMS and how they can make a utility’s response to disaster one that will see it through the challenge.
An OMS consists of 90 percent software with some needs for hardware, according to Nielsen. But, the most important feature of an OMS for Nielsen is integration. A utility must connect its OMS to its customer information systems (CIS), geographic information systems (GIS), voice response systems, and automatic meter reading (AMR) systems for effective integration.
“An outage management system doesn’t work unless it’s very well integrated,” said Nielsen. “The integration aspect needs to be thought about first and foremost… It can lead to some grief if they [the utilities] don’t want to spend the time and the money to do the integration right.”
Charles Sentell, customer operations manager for Mississippi Power Co. (MPC, a Southern Company), and Janell Ross, senior system analyst for MPC, told EL&P that many of Southern Co.’s utilities use an OMS from SPL WorldGroup they commonly refer to as TCMS (Trouble Call Management System). With regard to integration, they learned after Hurricane Ivan in 2004 they needed better prediction rules as well as a modified GIS and better OMS updating in the field.
“We modified our GIS distribution system data to uniquely identify automated sectionalizing devices vs. non-automated sectionalizing devices and went through our GIS distribution data to correct any solid blade sectionalizing devices identified as fused devices,” said Sentell and Ross via e-mail. “MPC created printable outage reports and customer call reports by distribution feeder to assist distribution feeder coordinators in the field in accessing and reporting back what was actually out so the OMS system could accurately reflect the state of the distribution system… MPC also discussed and planned with the other TCMS Southern Company Operating Company users how to assist each other with updating OMS from their respective locations. This gave additional help without having to send employees in harm’s way,” Sentell and Ross said.
Additionally, Sentell and Ross said MPC shut off its automated customer call back because power can go out several times in the same area before it is fully restored. Also, MPC used a feature of its OMS to automatically fill out certain data fields for dispatchers, added Sentell and Ross. For example, during a hurricane the cause of an outage is automatically reported as wind/rain. Having data fields automatically filled during a major event saves the dispatchers time on the forms they fill out for the OMS when outages are reported.
a flood of data
As multiple outages occur during a large outage event, an OMS is flooded with data that, in the past, may have caused system crashes or slow response times, said Nielsen. But over the years “the hardware’s gotten better, the software’s gotten better, the functionality within the OMS to deal with those large events has become more prevalent,” Nielsen said of modern OMS offerings.
Sentell and Ross confirmed this, saying their OMS was “running at maximum capacity without any problems” and that their “OMS system was the only computer application still working” during Hurricanes Katrina and Rita.
The data that fills the OMS during an event comes from multiple sources. One of the largest sources of data will come from customers calling to report a power outage. “Literally, it [the OMS] might take hundreds of thousands of calls that it will keep track of,” said Nielsen. However, said Nielsen, this is not the most important data the OMS needs.
“Damage assessment is really one of the biggest pieces of the first phase of a restoration effort,” said Nielsen. “The utility will start… systematically going from every substation and every feeder and surveying the damage, noting how many poles are out, how many wires are down. They are basically inventorying how much damage there is.”
After the damage assessments are finished Nielsen said the next phase of a restoration is to prioritize the work. “This is another big piece that the OMS can help them with,” Nielsen said. “Because essentially, you’ve got way more work than you can do in any given day – if you think about some of the most recent hurricanes, in Florida or New Orleans – they had weeks and weeks and weeks of work to do. The OMS keeps track of all of the critical locations that need to be restored first. So 911 call centers, hospitals, fire departments, public shelters, when schools become public shelters – all of that information is entered into the OMS and then it will give the managers in the control centers a prioritized list.” With the prioritized list, Nielsen said, the managers can deploy work crews to get the various jobs done.
In the case of the recent hurricanes, priorities could change in ways perhaps never imagined. Some areas were so devastated, full-scale evacuations occurred and destruction was nearly 100 percent. What then is the priority when there is nothing left?
Nielsen observed that today’s OMS solutions can make logistical conclusions about the affected areas, directing priority away from areas where there are no meters, homes or customers to restore. Nielsen went on to say that some OMS solutions now have the ability to “dynamically adjust” the priority of customers. This means a utility can manually enter certain variables that will change an areas’ priority. In major events like a hurricane, a local school has a low static priority, but, if it is declared a public shelter, its priority level hits the top of the list. This is a variable a utility can enter into the OMS so the solution keeps its priorities straight.
Then the work begins. Crews work to repair the most crucial sectors and constantly communicate through wireless means (or otherwise) to the OMS, said Nielsen. In this way, the OMS continues to prioritize and organize the restoration effort.
Sentell and Ross saw their OMS perform such duties firsthand during Hurricanes Katrina and Rita. “The results of fine tuning our prediction rules in TCMS and modifications in our GIS data made a significant difference in the way our OMS ran,” said Sentell and Ross. “We restored power to every customer that could take power in 12 days. Our reports (generated by the OMS) assisted with this effort.”
MPC did encounter problems with its OMS. In one instance, said Sentell and Ross, the OMS gave customer outage counts that exceeded the actual amount of customers they served. The problem was solved, Sentell and Ross said, by providing data to the OMS with accurate customer counts, in effect, further integrating the OMS during the actual event.
Also, 100 percent of MPC customers lost power. As a result, said Sentell and Ross, their OMS retrieved so much data its storage space was filled to capacity. Sentell and Ross stated that the OMS was retrieving the data every two minutes, so MPC extended the retrieval time to every five minutes, thus freeing more space.
Despite those challenges, Sentell and Ross thought their OMS performed “better than expected. When you lose power to every customer you serve and your OMS system never stops working, all you can say is “˜exceptional!'”
Nielsen said most large utilities that serve several hundred thousand customers have a commercially produced OMS, but some utilities have an “in-house” OMS built by “an army of IT people.” Nielsen cautioned against a homegrown approach to OMS saying that a benefit of a commercial solution is the ability to communicate with other utilities’ OMS solutions. After the devastating effects of Katrina, Rita and Wilma, many utilities had to rely on each other for support, yet found blocks when their OMS solutions could not communicate between different companies.
Sentell and Ross concurred saying, “The single most important feature of our OMS was our ability to have a seamless operation between operating companies.” They also said they would have “given up a lot of benefits” without an OMS. “Without an OMS, we would not have been able to accurately report customer counts or efficiently restore service to our customers. Instead of being able to restore power in 12 days, it probably would have taken weeks, if not months,” Sentell and Ross commented.
But, some utilities try it, said Nielsen. “They (utilities without an OMS) have ways of dealing with it, but it’s amazingly archaic actually. They will keep lots of note pads, and binder clips, and staples, big wall boards with push pins and post-it notes,” said Nielsen. “They manage to cope, but it’s always unique as to how they do it. It will usually be a combination of off-the-shelf office tools like Word documents along with paper and pencil and maps.”
After such devastating storms there are lessons to be learned. Sentell and Ross found that Southern Co. made the right choice when opting to have uniform OMS solutions throughout its various utilities. Also, Sentell and Ross pointed out the need to manage customer accounts in a unique way during such a catastrophic event.
“A lesson learned was that during outages where you have customers that will never return, we need to turn off our Customer Load File routing. With an existing outage that lasts several days where customers are being deleted from our accounting system, the Customer Load File was changing our numerator and denominator in calculating our reliability numbers. We were also keeping track of the customers we lost through our OMS. The customer’s account was being deleted in our Customer Accounting Information, removing it from our OMS. We corrected the customer counts in our OMS and turned off our Customer Load File until we had our counts and information we needed for reporting,” Sentell and Ross commented.
Nielsen, who spends much of his time traveling among utilities and learning their OMS needs, said the larger lessons will be slow to come. Lesson learning is a protracted process for utilities as they have a day-to-day challenge even after the storm is over. The big lessons, said Nielsen, will come after utilities hold an annual meeting dubbed the “storm summit” to discuss their OMS experiences.
All of the Gulf Coast utilities in the affected areas were tested to the limits of their abilities in the hurricane season of 2005. With the help of OMS solutions and dedicated support staff, at least one Gulf Coast utility company was able to restore power to customers in less than two weeks. And though getting back to normal for many people will be a long road, they have the power to hasten that normalcy.