Utilities Need Test Bed to Evaluate Legacy Industrial Control System Cybersecurity Technologies

BY JOE WEISS, APPLIED CONTROL SOLUTIONS, AND STEVEN BRUNASSO, AN ELECTRIC UTILITY

Utilities Need Test Bed

Industrial control systems (ICSs) are technologically and administratively different than information technology (IT) systems. ICSs are purpose-built systems engineered for highly reliable operations in harsh settings 24/7/365 and operate like a well-engineered clock. They often use older, resource-constrained systems and devices. They are deterministic systems that must function precisely within well-defined and regulated periods often within tens to hundreds of milliseconds. Drifting beyond that precise cycle time through variations of latency can lead to denial-of-service or worse.

Many IT cybersecurity technologies, however, were designed to operate with unlimited resources that could delay operation a few seconds, such as many anti-virus solutions, yet still be considered within IT design requirements. Encryption forms the core of many IT security solutions, although it might be unnecessary or harmful to ICS operation. Introducing a pair of encryption or decryption devices can introduce latency of 2-3 milliseconds. This delay on every loop might cause the ICS to fail because of a cascading slippage in the timing of the ICS program loops. Significant latency, however, cannot be tolerated in ICS applications without affecting their core mission: reliable, deterministic performance. Delaying a power relay one cycle (0.017 second) potentially could expose some 4,996 kilometers of your interconnected neighbors to that fault over interconnections. Most ICS and supervisory control and data acquisition (SCADA) security solutions, however, are recycled IT solutions and do not address the ICS-unique issues. This was confirmed during the 2013 RSA Conference in San Francisco.

Whereas IT applications follow the conventional CIA triad-confidentiality, integrity, availability-ICSs follow the inverse: IAC. Because integrity and authentication are critical, but not confidentiality, encryption might be unnecessary for data in motion applications. ICSs generally consist of any combination of networks, some with routable Internet Protocol (IP) and some direct links such as point-to-point serial. These point-to-point links provide the smallest possible latency for a communication path but have no external visibility beyond the two systems connected. The benefit of encrypting this information is unclear. This might be why they are excluded from the North American Electric Reliability Corp. (NERC) critical infrastructure protection (CIP) requirements.

Security information and event management (SIEM) systems were designed to monitor all the traffic on an IP-addressed network and do not address cyberinformation from non-IP networks or dedicated local serial buses. ICS protocols generally are insecure, particularly in older legacy ICSs where they might connect only instruments or actuators on a local ICS serial loop rather than the wide-area network. ICSs include forensics and logging for physical parameters such as pressure, temperature and voltage to perform root cause analyses after an upset condition. Few ICSs, especially legacy ICSs, however, have traditional cyberforensics and logging capabilities, which makes it difficult to recognize cyberincidents that affect field devices such as controllers. Most modern systems can add a large historian solution on the chassis; however, most sites use the existing controller memory for this function, but it limits one’s ability to perform forensics in the IT sense.

IT testing and technologies have affected the performance of ICSs. Even a simple scan of the basic network connectivity has disrupted normally stable ICS networks. ICSs require periodic updating, which generally is done remotely or through portable devices. It is impossible to exclude remote access to these systems without an unacceptable impact on performance.

Currently, there is no objective and standardized criteria for how much security is enough. Performance must win. ICSs are systems of systems where the security vulnerabilities might not be in the individual boxes but in the interconnections between the boxes. Consequently, legacy ICSs need different security approaches than traditional IT systems. They need a systems engineering approach with specialists that know the range of technologies from process control, computer engineering and awareness of what traditional IT system security might already have addressed.

State of Electric industry for securing legacy control systems

Electric utilities have developed and focused on NERC CIP compliance. The NERC CIPs do not require ICSs to be secured. They make specific requirements for some solutions to be implemented despite their effect on ICS performance. Another unintended consequence of the NERC CIPs to the protection of critical infrastructure is inhibited innovation that could improve security and reliability. If a utility has a better solution that doesn’t meet the mandate and implements the solution, it may be considered a potential audit finding and result in a large financial impact. Without sharing good solutions and engineering practices, there is less innovation. Consequently, there is not a thriving market for ICS vendors to provide security solutions for legacy ICSs when there is little market demand. This silence also creates an unintended consequence through the lack of data on the resources required to secure legacy ICSs. And often, ICS domain experts either are excluded or see no reason to participate in ICS security evaluations that are in the purview of the NERC CIP compliance specialists, further minimizing the potential that reliability and robust control will be considered.

An unintended consequence of this compliance approach is the removal of engineering from security.

a utility test bed is needed

Most utility (large industrial) assets have a life span measured in decades, and reliability is most important. This generates a tendency to wait until technology has been demonstrated in an environment with equipment similar to what has been installed. In other words, utilities often are in a rush to be No. 2 after No. 1 has had a positive outcome and clear benefit. This is what drove the establishment of demonstration test beds such as the Electric Power Research Institute’s (EPRI’s) monitoring and diagnostic center at what was Philadelphia Electric Co.’s Eddystone Power Plant to evaluate and demonstrate the value of predictive maintenance technologies. The Instrumentation and Control (I&C) Center at the Tennessee Valley Authority’s Kingston Steam Plant provided a similar venue to evaluate and demonstrate the value of new, generally digital I&C technologies. Neither center considered cybersecurity because cybersecurity was not considered an engineering design requirement yet. When the Idaho National Laboratory was being established in 2003-2004, I helped obtain vendor support for the National SCADA Test Bed. The same approach of having vendors participate is being used for the utility test bed to evaluate technologies for securing legacy ICSs. There is also a need for the utility test bed to demonstrate that well-engineered solutions that include security technologies can help maintain or improve operational reliability, that is, improve ICS robustness. Security for security’s sake is poor insurance with an almost guaranteed negative return on investment; however, if the security technology can provide an operational benefit, it will be used and maintained. Through sharing good systems engineering solutions with security as a design constraint, we might achieve protection of all the critical infrastructures, which was the original intent of the NERC CIPs.

Characteristics necessary for utility to stand up test bed

Why isn’t there a utility-not a national lab-cybersecurity test bed? The answer is because the necessary ingredients haven’t been there until now. These ingredients include

  • The utility does not need to meet NERC CIP requirements so it can concentrate on reliability, not compliance, and the attendant audit risks of not being compliant.
  • The utility has visionary senior management-engineers who focus on reliability, not compliance.
  • The utility needs ICSs representative of the industry.
  • The utility needs to understand why this is important; that is, understand the impacts of intentional and unintentional ICS cyberincidents.
  • The utility needs to be willing to discuss results openly within reasonable constraints.

The utility needs to be willing to be a test bed without government support so government strings that concern what can be tested, what can be disclosed, vendor interactions, etc., are eliminated.

What is happening now

A domestic utility is willing to be a test bed and discuss the results. (This same model is being discussed with an international utility.) The utility is representative of most utilities with power plants, electric distribution, electric subtransmission, SCADA and other common systems supplied by a mix of major ICS suppliers with a mix of vintages-some old, some new. The basic thesis of this project is to consider security as a discipline to increase reliability (robustness), not compliance. Consequently, a major part of this effort will be to quantify the impact (plus or minus) on reliability. The test bed approach is viewed as a long-term effort to evaluate new security and reliability technologies as they are developed and share solutions that have been implemented and evaluated.

This utility is also the initial test bed for implementing the Aurora hardware mitigation solution. As the Aurora test and mitigation were performed in the 2007 time frame, the project is also evaluating how newer technologies such as phaser measurement units might support the Aurora mitigation effort and address other grid vulnerabilities.

The first step for the test bed is to make ICS and security solution providers aware of this opportunity. Interested solution providers are asked to provide a one-page description of the technology and where it has been used so the utility can decide if the technology might have promise. If so, the utility will work with solution providers by first evaluating the technology in a test bench situation to validate the solution will do no harm before moving to field-testing. Concurrently, the utility is working with several ICS suppliers to address insecure legacy ICSs already installed. As these systems continue to operate reliably, they will not be replaced simply because they are not secure. The utility and its vendors are working on technologies that can have a security retrofit and with technologies that are incapable of a security technology retrofit (unfortunately, this is very common).

For this approach, the vendors provide the technologies and support for specific applications, and the utility evaluates the technology and discusses the results. The result is helping develop demonstrated solutions and making a market for implementing security technologies that can have a positive ROI.

The utility and selected vendors will discuss the project status at the October ICS Cybersecurity Conference in Atlanta.

Joe Weiss is an industry expert on cybersecurity of industrial control systems who provides subject matter expert support to government, end users and equipment suppliers.

Steven Brunasso is a security systems manager at an undisclosed U.S. electric utility.

More PowerGrid International Issue Articles
PowerGrid International Articles Archives
View Power Generation Articles on PennEnergy.com
Previous articleUtilities, Vendors Can Help Fill Power Industry Jobs
Next articleGPS Fleet-tracking Software
The Clarion Energy Content Team is made up of editors from various publications, including POWERGRID International, Power Engineering, Renewable Energy World, Hydro Review, Smart Energy International, and Power Engineering International. Contact the content lead for this publication at Jennifer.Runyon@ClarionEvents.com.

No posts to display