By Joe Weiss, PE, CISM, Applied Control Solutions, LLC
Much has been written lately about cyber security and the NERC Critical Infrastructure Protection (CIP) cyber security standards. There has been a prevailing view that meeting the NERC CIP standards will make the utility compliant and secure. The CIP standards have not yet been accepted by FERC, which means meeting the CIP standards as written may not make the utility compliant. More importantly, it may not make the utility more secure.
The goals of a prudent control system cyber security program should be to help make the utility more secure, maintain and when possible, improve system reliability and availability, and meet regulatory requirements. Let’s address five areas that may be overlooked in establishing or maintaining a prudent cyber security program. (Many of these issues were identified in the FERC Technical Staff Assessment of the NERC CIPs.)
The five overlooked areas that a comprehensive control system cyber security program should include are:
- Including control system-specific cyber security policies and procedures;
- Identifying all relevant stakeholders;
- Performing vulnerability assessments to identify all electronic connections;
- Performing risk assessments from both NERC and business perspectives; and,
- Addressing interconnections.
The basis of this list is derived from a number of factors including: knowledge of control systems published publicly by the hacking community, experience with utility cyber security programs including performing vulnerability and risk assessments, knowledge of control systems and how the newest control system technologies may be deployed over the next three to five years, and knowledge of control system cyber security incidents that could occur and have occurred.
1.) Implementing Control System-Specific Cyber Security Policies and Procedures.
The most common and critical opportunity in electric utility (and other industry) control system security programs is to create and enact comprehensive control system cyber security policies and procedures and associated control system cyber security training. To ensure they are taken seriously, the adherence to these policies and procedures should be one of the performance goals of senior management (per the NERC CIPs). Almost all utilities have cyber security policies and associated training, but many are based on traditional IT policies and technologies. This is a problem for the control system’s environment and needs to be treated as such. While some components of an IT security program can be applied to control systems, many of these policies are not relevant to the real-time control system environment and inappropriate when addressing legacy field devices. For example, there have been numerous cases where inappropriately applying traditional IT security technologies such as certificates, block encryption, or even anti-virus have impacted or completely obstructed control system operation. Traditional IT security testing can be even more problematic for legacy systems. Many legacy control systems have been designed without a complete IP communication stack. Scanning legacy control system devices and/or networks utilizing traditional IP scanning tools can lead to broadcast storms as the scanning tool attempts to locate devices that cannot adequately respond. A broadcast storm is a state in which a message that has been broadcast across a network results in even more responses, and each response results in still more responses in a snowball effect. A severe broadcast storm can block all other network traffic, resulting in a network meltdown.
I am aware of several actual control system cases including a presentation at the 2005 KEMA Control System Cyber Security Workshop where scanning control system networks and/or devices resulted in broadcast storms significantly impacting control system performance. In at least one case, scanning resulted in damage to control system equipment requiring replacement before the equipment (in this case variable speed drives) could be reused. Consequently, it cannot be stressed enough how dangerous scanning can be to legacy systems if not performed knowledgeably and with caution.
Another common problem is security of dial-up modems. Many users feel that all modems have been identified and disconnected when not needed. When visiting users (not just utilities), I have yet to meet a user that hasn’t told me they know where all of their modems are and they are disconnected when not in use. Conversely, after detailed discussions and walk-downs, I have yet to find a user that hasn’t found at least one modem they didn’t know they had or at least one modem that was connected they thought was disconnected. Any port that is not secured is a cyber security vulnerability. Without appropriate control system policies, procedures, and training, this is the surest way to fail a “real” control system audit or the quickest path to unintentional control system problems. I know of only a few technologies that have been tested on user modems and protocols (e.g., DNP3, Conitel, etc) and only one that can perform modem protection in real-time, while potentially improving system performance.
2.) Identifying all Relevant Stakeholders.
In the past, identifying relevant stakeholders for a SCADA or plant control system was easy: It was limited to facility and corporate operations and engineering. Today, it is much more complex and tomorrow will be even worse. Part of what makes control systems more productive is also what makes them more insecure-system integration. More and more organizations are finding their most valuable and useful data is the real-time control system data. This is leading to many internal organizations establishing, or wanting to establish, connections to a SCADA, plant control system, programmable logic controller, or control system database without the corporate or facility operations and engineering organizations even being aware. Depending on how the networks are configured, this may result in unintended cyber issues.
3.) Performing Vulnerability Assessment to Prudently Identify all Electronic Connections.
Utility organizations are beginning the process of assessing cyber vulnerabilities of their control systems to meet the NERC CIPs. The creation and execution of these assessments needs to be done carefully as there are several significant and frequently conflicting issues at play. The first is scope. NERC is focused on grid reliability. There are many specific scope exclusions in CIP-002 such as telecom, market functions, distribution, and non-routable protocols. Many utilities have excluded these systems in their vulnerability assessments since they have been excluded by the NERC CIP. Many of these excluded systems are cyber vulnerable and directly communicate with systems that are in the CIP-002 scope. Consequently, it is not possible to comprehensively identify the cyber vulnerabilities that can impact these critical cyber assets. Implicitly, there is another exception-small facilities. The NERC CIP implies that traditional reliability criteria can be followed in defining what equipment need be identified and addressed as critical assets, which implies large facilities. This makes sense from NERC’s traditional reliability perspective. However, we are now addressing a cyber security issue. From a cyber security issue, it is irrelevant how large or critical the system is to normal reliability considerations. From a cyber perspective, what matters is if the equipment is electronically connected. Even the smallest facility, if electronically connected to a control center, can be a pathway to compromise the control center. Conversely, a very large facility that is critical for reliability considerations but has no electronic connections is irrelevant from a cyber perspective. When addressing cyber security, it is not the size of the device or facility, but the connections that matter. Another issue that must be considered is the exclusion of telecom. One of the most probable causes or paths for cyber intrusions are the inherent vulnerabilities within the telecommunications environment. This is particularly true for leased lines, microwave and multiple address radios. It has been demonstrated by one of the national laboratories that 900 MHz spread spectrum, frequency hopping radios can be hacked. These radio systems provide the critical communications within the substation and provide input directly to SCADA. Compromise of these radio systems can lead to compromise of the devices within the substation. If the current exclusions in the NERC CIP are followed, these devices using non-routable protocols will be excluded from the assessment process.
Often the distribution system is excluded from the assessment. However, because they often have undergone the most upgrades, it is the distribution systems that have now become arguably the T&D system’s most cyber vulnerable parts. As distribution systems are electronically connected with transmission systems, they should not be ignored. The market function of an EMS system receives data from insecure meters and also electronically connects with SCADA. As with distribution, the market functions are often excluded by the NERC CIP. These vulnerabilities could lead to very significant economic impacts if meter or billing data is compromised.
Therefore, it should be evident that by excluding systems from NERC CIP programs, it is not possible to identify all the critical cyber assets much less the vulnerabilities that can impact critical cyber assets. Remember: It’s all about the connections where the real cyber vulnerability exists.
4.) Performing Risk Assessments from both NERC and Business Perspectives.
Cyber risk needs to be addressed for grid reliability to meet NERC CIP requirements. However, cyber risk also affects systems that can significantly affect the business, but not necessarily affect grid reliability. Many systems that are critical to the utility’s economic health may not be critical to grid operations and are consequently excluded from the NERC CIP. Facilities such as small power plants, low- to medium-voltage distribution substations, and automated metering infrastructure are examples of facilities and systems that are “business critical,” but not “grid critical.” There is a significant potential liability to a company for ignoring cyber risks to the business even though these systems are excluded by NERC CIP.
5.) Addressing Interconnections and Interdependencies.
The last issue is possibly the most subtle, but certainly not the least important. That is the impact of interconnections between transmission systems. Electric utilities often share equipment such as RTUs. Utilities also interconnect with one another. There is an old saying in the cyber community that you are only as secure as your weakest link. In this case, your weakest link could be your neighbor. How this is addressed impacts not only you, but also your interconnection partner. These interconnections need to be addressed comprehensively. This was also recognized in the FERC Staff Assessment.
The issues addressed in the NERC CIP have done the utility industry a great service by beginning the process of requiring cyber security to be specifically addressed. However, it has done so in a limited manner. Many of the identified limitations have already led to cyber events. To minimize risk to the utility infrastructure and business operations, it is incumbent on the utility to utilize due care and diligence in establishing and maintaining their cyber security programs. Cyber issues can materially affect the utility industry’s bottom line from a positive direction (improving system reliability and availability) or from a negative direction (cyber impacts). The positive direction takes a comprehensive program beyond “just meeting the NERC CIP requirements.” The negative direction can occur because the program was not sufficiently comprehensive and can lead to punitive damages as suggested by NERC.
The choice is up to you.
Weiss, a well-known cyber security expert, works as consultant and partner at Applied Control Solutions, LLC. You can contact him at email@example.com.