Cybersecurity, Outage Management, T&D

D-Day at the Substation- Historic Challenges and Lessons

Issue 3 and Volume 93.

by Peter Manos, McDonnell Group

Substation engineers are among the utility industry’s unsung heroes.

Substation-related challenges faced by utility executives, substation engineering personnel and operations and maintenance (O&M) managers were the focus of “D-Day at the Substation,” a panel session at DistribuTECH Conference & Exhibition in San Diego in February.

Selection of the four panelists and proposal of the well-attended session was an outgrowth of important trends analyzed during a McDonnell Group research project in mid-2014.

Major shifts in priorities of substation engineers, project managers and utility decision-makers are underway.

Why is “D-Day” in the session title? Answering a question with a question, can you tell who made the following statement? First, here are two hints:

1. The statement was made by someone in a leading executive role;

2. The topic was a project involving enormous engineering challenges.

Ready? So who do you think said the following, “Let me have the best solution worked out. Don’t argue the matter. The difficulties will argue for themselves.” If you are familiar with utility challenges regarding operation of high-voltage substations, you easily could think the statement was made recently by an electric utility executive. It was actually in an important memo written by Winston Churchill.

Churchill’s memo led to the successful execution of what many consider one of the greatest engineering feats in military history. But more on that later-here it is to be noted that Churchill’s engineering project turns out to have a lot in common with the challenges utilities have to meet regarding substation assets.

Figure 1:

Although each panelist discussed aspects of substation hardening, grid resiliency, reliability and cybersecurity, all four highlighted most of the same key lessons in reviewing their recent substation-related work:

1. The highest possible safety levels still must be upheld;

2. Engineers must address a more challenging day-to-day operational environment than before;

3. Assets must be hardened to withstand unprecedented increases in frequency and severity of major storms and other major weather events;

4. Designers must anticipate and neutralize new potential failure modes; and

Mulberry Harbor

Many of the engineering and logistics lessons learned from Mulberry Harbor are instructive for engineering design work being done to harden substations and ensure their ongoing, reliable operation.

The quote, “Let me have the best solution worked out. Don’t argue the matter. The difficulties will argue for themselves,” was part of a famously angry memo British Prime Minister Winston Churchill sent to Lord Mountbatten in 1942.

Churchill’s demanding memo changed the course of history. It led to the successful construction of Mulberry Harbor, an enormous man-made harbor consisting of two prefabricated ports off the coast of Normandy, France, that many consider the greatest military engineering feat in human history.

It has been 71 years since this photo was taken-10 days after the June 6, 1944, D-Day landing.

A key part of the preparations for the D-Day landing involved the rapid assembly of Mulberry Harbor. The strong defenses around existing ports necessitated this ingenious approach, which employed floating ramps and enormous artificial break harbors across more than 1 square mile (750 acres).

As a result, Allied ships could unload tanks and armaments despite the area’s very rough waters.

5. Planners must optimize large projects, critical projects or both-key parts of which much be done secretly because a hostile enemy is ready to strike at any potential vulnerabilities.

D-Day-June 6, 1944-obviously required secrecy, and it is no less obvious that aspects and details of our substation-related defenses must be undertaken with secrecy, as well. This is evident, given the coordinated sniper attack on the Metcalf substation, as well as other exploits against vulnerabilities that substations encounter.

Figure 2:

So unfortunately, our substation heroes likely will remain unsung. To inform our look ahead, what are the common lessons to be learned by looking back?

With intentional quote marks here, “expert” panelists presented interesting process-optimization insights at our “D-Day at the Substation” session, along with practical engineering design and project trends for addressing our storm hardening and physical hardening, perimeter shielding and cybersecurity needs.

Why does the word “expert” have quote marks around it? They are a good symbol for something that came across loud and clear during the panel session: the idea that an expert is someone who has come to see the complexities of a subject. A result of seeing complexities more deeply is that the “ex” pert is no longer a “pert” (which is an old word for a know-it-all).

An ex-pert in an area usually ends up trying to learn more about his or her area of expertise in a never-ending process.

The panel’s common theme was how such expertise is gained. A common lesson learned in different ways during every presentation of the panel session was the need for an ongoing process for demonstrably improving reliability. Every step, the four panelists had examples of this. The first panelist, Richard Wernsing of PSE&G, first highlighted this.

Although the diagram Wernsing presented was specific to critical infrastructure security goals, one can view it as a diagram for how to develop expertise in any area (see Figure 1).

Wernsing said the industry is employing this security risk framework to build resilience and redundancy into the electric grid and reduce risks while ensuring solutions are cost-effective and prudent.

As a result of many requirements, the physical, cyber and human layers span across diverse improvement processes for critical assets, which must be adjusted continually. Adjustments are done by employing feedback for continual improvement.

The need for a framework for continual improvement must be based on a people-centric, interactive approach. Whether related to electrical engineering designs, physical security, cybersecurity or human risk factors, rigorous feedback and practical approaches for cost-effective solutions are key elements for success.

Panelist Neal Rich of Mississippi-based Asset Engineering addressed design issues in substation projects (see Figure 2).

The Asset Engineering projects and issues presented, along with additional discussions at the panel session, made clear there is no replacement for having highly experienced designers. The type of work required will include much designing to optimize reliability and resiliency in unique situations. It will be a situation leveraging specialized expertise of seasoned engineers.

Whether work is done in-house or with third-party engineering design team involvement, personnel must have excellent communication within project teams. Rich discussed virtual teams that his company promotes with its utility clients. Such teamwork enables better project outcomes by creating a culture of trust and openness, which help uncover potential needs and issues quickly to address them best.

The types of projects Rich described during the panel varied and showed the need to balance technical and people skills. They included substation asset hardening against attacks and severe weather plus a complex mix of projects to address wide-ranging needs.

Examples included design optimization after performance of special studies, as well as other types of electrical design optimization work. Projects often involve a mix of considerations. For example, asset replacement and upgrade work increasingly is being done in a hybrid fashion to get the most out of existing assets and control and protection systems while designing to address higher contingencies.

Along with Wernsing and Rich, the panel’s ballistic and acoustic shielding participant, Bob Wiles of Valmont, and cybersecurity panelist, Charles White of Charles A. White & Associates, also provided examples that clarified the need to respect one another’s “ex-pert-ise” here.

How important is it that the work for addressing these needs be done 100 percent in-house at a utility vs. using outside contractors?

Panelists and audience members confirmed something that McDonnell Group also saw in its research: There is a well-established, highly mixed set of relationships among utilities and their contractors and subcontractors that continues as it had in the past, before our substation’s D-Day.

The size of third-party engineering firms is less of a consideration than the trusted relationships and skills they bring to their utility clients.

What is different post-D-Day vs. pre-D-Day is the even greater need now for excellent performance and rigorous use of these best practices and trusted relationships.

Conclusion

Now that we are at another anniversary of D-Day, it is clear that Churchill had it right. It is through the experience of addressing these problems and adjusting approaches in our responses to them that we arrive at the best solution. He said it better than that when he noted in his memo starting the D-Day engineering project that “the difficulties will argue for themselves.” What project was Churchill’s D-Day memo about? The memo was actually about Mulberry Harbor (see sidebar), an amazing logistics achievement. May our industry’s achievements in substation hardening and grid resiliency show the same level of mettle.


Author

Peter Manos is principal strategy consultant at McDonnell Group. He previously worked as an engineer at Con Edison, where he held positions across the generation, transmission and distribution systems. He has an MBA in marketing and finance from NYU, a Bachelor of Science in Electrical Engineering from Polytechnic University (NYU) and a Bachelor of Arts in Philosophy from Vassar College.