If you attended DistribuTECH 2019 and joined any of the “defending the grid” sessions, you saw that intense focus continues in all things cybersecurity. It seems everyone realizes cybersecurity has rapidly become one of the greatest operational risk to a utility. Due to the extreme technical nature of threats and preventative measures, if you are not a cybersecurity expert these can be very difficult to discuss, let alone understand.
If the extent of your cybersecurity questions have been answered by your cybersecurity staff with a statement similar to this: “˜`We have a defense in-depth approach to protecting the utility,’ I offer several thoughts on how to drive the conversation to get the details you need and ensure you have sufficient protection in place. After all, when a cyber incident occurs, it is you, the leaders of the utility, who will be questioned on what happened, how the breach could have possibly occurred, why it wasn’t prevented, and how you are responding and recovering. Don’t wait until it is too late to learn how to answer these questions.
The following are suggestions on how non-technical leaders can have a meaningful discussion with their cyber team, drive cross team collaboration, and ensure you are well educated in the threat and defense of your utility. The goal is for you to gain knowledge to allow you to understand the real risks, how you are protected, where you should be funding additional work, and how you should be driving your security programs forward.
Three key questions
It is key to ask questions that promote detailed discussions. If your cyber team continues to use only highly technical terms, you should bring in someone who can translate the techno-speak into operational language. Do not leave the conversation until you understand and are able to comfortably discuss on your own any of the NIST framework functions (Identify, Protect, Detect, Respond, and Recover), how they are leveraged, and how often risks are reviewed in each function.
What are the threats we have seen on our utility, what has been found that you have protected us from? This is a great two-part question because so often we are told about “˜threats’ in general terms, without really understanding what they are, and often we are not given any details. There is so much focus on email threats, and they tend to be easier to understand, but what about the rest of the systems? If your utility has prevented specific intrusion attacks, you should understand what they were. If a threat was successful in penetrating a layer of defense, it should be explained. As a leader of the utility, you need to be given details on the total operational risk to your utility, so you can manage the risk appropriately, allocate funding to ensure the appropriate actions are taken to reduce risk.
Different systems have different risks, how are we protected across the variety of systems? An enterprise system like email is protected differently from an operational technology (OT) system like Supervisory Control and Data Acquisition (SCADA). There are different protections in cyber functions such as monitoring systems, access control methods, incident response processes. Consider how physical threats and risk may play into the overall risk assessment and mitigation strategy. You need to make sure you understand these differences and how the mitigations, protections and responses operate for all the different systems.
How do we execute penetration testing and vulnerability assessments, what[CB2] are the results? It is critical to perform Penetration Testing, Vulnerability Assessments, Drills and Exercises to validate your systems are secure, risks are understood, and you can respond and recover to incidents as they occur. There must be strong collaboration between your system support teams and cybersecurity teams. Gaps in collaboration create cracks in a strong cyber defense.
There should be ongoing testing and assessments across all systems, including test and production environments. Production penetration testing is necessary to understand the actual risk, because test environments rarely mimic end-to-end production configurations. There is a certain amount of risk that exists with this type of testing, but executing controlled tests allows you to have a controlled recovery and gain a solid understanding of true risk. Results should be clearly documented.
There should be ongoing drills (tabletop) and exercises (actually performed) to test cybersecurity incident response across all systems. Performing exercises is critical; drills are good preparation but until you perform exercises, you have not proven out the ability to react to events and recover systems. Do you understand how the loss of critical systems will impact your ability to manage the grid?
Scenarios should be executed thinking about both systems recovery and personnel response to incidents. These types of exercises can include isolating critical systems, rebuilding critical systems, identifying and removing threats. The benefits of performing real exercises is that you get to validate expected responses, see how both people and systems react in demanding and challenging situations, and demonstrate that you can perform grid management and rebuild systems as needed.
The difference in performing a disaster recovery (DR) exercise and a cybersecurity exercise is that you have control over your environment during a DR exercise, and operations personnel are typically not included in recovery effort; during a cyber event exercise, you don’t know what the threat is or what it will take to recover from it. Results and improvement opportunities should be documented.
Understanding, exercising and perfecting cross team incident response may be the most critical of all preparations. Today, many utility companies are comprised of several smaller utilities. It is common to have shared systems where you have one system used by all utilities. Understanding responsibilities is key to incident response. How will you handle a threat that impacts a shared asset? Who has authority to determine the actions to be taken, who will make the recovery decisions that impact you?
Exercises are the way to prove out your response and recovery procedures. During DistribuTECH, I spoke with Stan Pietrowicz from Perspecta Labs, to discuss the critical intrusions that he is seeing across the utility sector, and to understand the measures that need to be taken to keep the power flowing. Stan agreed that utilities need to invest resources into exercising recovery procedures that will allow them to recover from strategic cyberattacks.
“Conducting cyber exercises and vetting recovery procedures are core components of a modern utility’s cybersecurity and disaster recovery programs. Intelligent, highly-interconnected energy systems are the present and future of this industry. Just look around at the vendors on the exhibit floor and the technology at hand. Having attended DTECH over the past 10 years, the shift in focus from traditional energy delivery equipment to automation, smart systems, analytics and customer enablement is striking. As a result, every utility’s security program is undergoing significant change to protect an increasingly intelligent infrastructure against adversaries who are well versed in targeting highly interconnected systems. The attacks that occurred in the Ukraine are a wake-up call, and maybe the first of increasingly sophisticated attacks that may be attempted against critical infrastructure. Cyber exercises, like DARPA’s recent exercise on Plum Island, can provide critical insight into whether plans, policies and procedures, and implementation practices are built to handle a real situation and can help uncover unexpected obstacles that may stand in the way. The absolute worst time to begin thinking about how you are going to respond, contain and regain control is in the middle of an active cyber-attack. Utilities prepare for natural disasters and other kinetic events–it’s time to take the same proactive approach to cyber preparedness.”
Once you have performed testing, drills and exercises, the result and findings must be documented along with clear operational impacts, to make sure the true risk is understood. While most of the remediations will be internally owned, some will be owned by your vendor partners. Contract terms should include security requirements and remediation expectation language to ensure the vendors are accountable for fixing any issues. All results should be prioritized with a risk-based heat map; remediation should be funded and executed. You should understand the worst-case scenario – in terms of operational impact – for all the findings. You should receive ongoing updates on remediation.
Non-traditional methods of detection
So many of the current protection efforts involve internal assessment and threat identification. However, with additional threats coming from the dark web, detection methods need to expand to the deep web. I stopped by the Terrago Booth at DistribuTECH to talk about this type of detection with Chip Hathaway. He stated
“Traditional defenses aren’t enough. Every utility with a breach had firewalls, anti-virus and tactical security feeds. Today, we have to go further, not just to the edge of the network, but well beyond. Proactive defense must include awareness of who is targeting your cyber assets, how these assets can be targeted and what are the risks associated with your cyber infrastructure. This type of defense is only viable by actively scanning both the open and dark web to find the threats before they penetrate your networks.”
You should also understand how your cyber team researches and implements new protections, for instance, how they leverage new technology like blockchain or Mitre’s ATT&CK framework.
Closing paragraph: Protecting a utility from cybersecurity risks is a challenging and ongoing effort. Threats become more sophisticated as each day passes, and keeping up with these new techniques and technologies is daunting. These suggestions can drive critical conversations on how you can gain the understanding you need to discuss and evaluate risks, understand how your utility is protected, and ensure recovery plans are in place and exercised. As a leader, driving a culture of cyber risk awareness, cross team collaboration and preparedness is essential to defeating the ongoing threats to utilities.
About the author: Carol Bartucci is president and founder of Crisp Consulting, LLC. She is a Strategically focused Information Technology executive with a proven track record on transformational project execution, operational excellence and creating multi-year roadmaps to enable business strategy. Responsibilities have included developing the strategy and implementing the utility industry’s largest analytics program; running a telecommunication and Smart Grid/IoT department, as well as extensive experience with real time systems. Carol worked for over 30 years at Exelon Corporation and has a degree in Electrical Engineering. She is currently serving on the board of trustees at Mother McAuley Liberal Arts High School and is a strong supporter of careers in the stem fields.