By John Shaw, GarrettCom
Substation cyber security is getting a lot of attention these days. Until recently, one might imagine a utility manager saying something like, “I have too many greater worries; it would take an act of Congress to get me to invest time and money in substation cyber security.”
Well, here we are.
With the mandates of the Energy Policy Act of 2005 coming into reality in 2008–with hefty fines for those not in compliance in 2009–many utilities are aggressively pursuing cyber security, due to both cyber threats and, now, regulatory oversight.
Despite the NERC CIP Cyber Security standards (CIP-002 — CIP-009) that FERC endorsed earlier this year, there is still some debate over “how much cyber security is enough.” There is a wide range of different approaches in play among utilities–depending both on their risk assessment, and on the human and financial assets that the utilities are prepared to put forth during this initial compliance period. In addition, there is a growing realization that the task is large and will not end with a single initiative. Even the FERC acceptance of the current standards requires that NERC continue to study and refine the standards, with ongoing assistance from cyber security experts. Indeed, new forms of attack emerge over time. As the industry grows more comfortable with baseline measures for the most critical assets, commonly accepted “best practices” will likely emerge, with utilities voluntarily spreading these technologies and processes to other assets deemed less critical, but still valuable.
In approaching this current phase of cyber security deployments, there are several choices that utilities may make to either minimize or expand the scope of their initial activities, potentially deferring some ultimate target technologies until after initial compliance is achieved. Choices include the selection of critical assets, additional data networking decisions that define “critical cyber assets” (CCAs), the sophistication of initial electronic security perimeter technologies, the scope of permitted remote access, and the use of remote substation servers that may require even more sophisticated threat remediation.
Limiting Critical Cyber Assets
The CIP-002 standard describes how utilities define critical assets, as well as critical “cyber” assets. Essentially all bulk transmission assets are deemed critical, and utilities may designate additional assets as critical based on other factors. Many utilities are working to minimize the initial project scope by designating as few critical substations as possible.
An additional limiting technique used by some utilities is to avoid the use of dial-up or routable protocols–i.e., networks that involve IP protocol, as is generally used with Ethernet-based substation devices (RTUS and IEDs) and even with some serial devices (with serial-IP terminal servers). If dial-up or routable protocols are not used to communicate to critical assets at the substation, then the substation does not contain critical cyber assets and the other aspects of CIP standards do not apply to that substation. Since NERC CIP includes a requirement to establish relatively extensive physical security measures at remote substations, such as enhanced fencing, gate access systems and video monitoring–as well as advanced network and system security measures–eliminating CCAs to limit project scope can significantly reduce the initial cost and workload of the first phase of compliance.
The “non-routable” network exemption in NERC CIP was primarily put in place to effectively grandfather existing serial-protocol SCADA systems that use dedicated analog leased lines to communicate from control centers (SCADA masters) to remote RTUs. The non-routable exemption is not limited to analog leased lines, however. Some fiber multiplexer systems and some frame relay-based networking systems can also be used to connect control centers and substations without use of routable protocols (e.g., IP). For example, as shown in Figure 1 (to the left, top) serial SCADA messages can be encapsulated directly into frame relay protocol (considered a non-routable protocol) using a technique called SCADA Frame Forwarding. This would not include use of the TCP/IP headers that are common with serial-IP terminal servers and routers. SCADA Frame Forwarding can combine several serial data streams over a single digital circuit or across a frame relay network without creating CCAs at the substation.
No utility would consider avoidance of routable or dial-up communications to be a viable long-term approach to cyber security, since IP networking is becoming the mainstay of automation systems. However, some utilities are even moving backward in their use of IP networks, disconnecting some Ethernet devices and reverting to completely serial communications in order to defer the upgrading of some substations and enable initial activities to focus more narrowly.
Sophistication of the Electronic Security Perimeter
For those substations that are deemed to contain critical cyber assets many choices remain that can affect the complexity of CIP implementation. A primary requirement is that a substation with critical cyber assets must establish both a physical security perimeter (CIP-006) and an electronic security perimeter, or ESP (CIP-005). The main technology for establishing an ESP is a firewall. NERC CIP defines the firewall requirement somewhat loosely, allowing utilities to choose the level of firewall sophistication that is most appropriate for their environment.
In the simplest case, a firewall can be implemented as a software function in a typical IP router that is used to terminate wide area network (WAN) connections at the substation. A firewall-enabled router such as in the following diagram filters out and rejects all data packets that do not conform to specified rules as to source and destination IP addresses and TCP port numbers. The router would log and report important information for use in auditing and network forensics.
Filter-based firewalls are an appropriate, sufficient ESP when the WAN is a private network dedicated to substation communications, and where other mechanisms are in place to log on-demand remote access, such as engineering access to remote IEDs. In a more typical IT environment, however, ESP technologies include virtual private networks (VPNs) that include encrypted communications between secured sites. While most utilities use private networks, some may consider Internet-related services such as DSL service, newer carrier provided MPLS-based VPN services, or wireless Ethernet networking, all of which provide some greater accessibility to attackers than do private facility networks, dedicated leased lines or frame relay services. If a utility is using a public IP service or sharing the substation WAN with enterprise IT communications, then the substation connections should have VPN protection in addition to basic firewall filters.
The most common VPN technology is tunnel-mode IPsec (IP security protocol). IPsec provides an encrypted “tunnel” from one point to another. As in the next diagram, IPsec VPN tunnels may be provided by the IP WAN router at the substation, as an embedded function. In other cases, a utility could decide to deploy a new dedicated security appliance. Typically, the VPN tunnels will connect substations across a WAN to one or more control centers (see Figure 2, pg. 32, bottom).
Note that NERC CIP does not explicitly require IPsec VPN technology, especially with private substation WANs. Since VPNs are relatively common in the IT environment, there is some speculation that future versions of NERC CIP standards may both eliminate the “non-routable exception” described above and also mandate the use of VPN technology for critical substations. Network planners deploying new devices for either non-routable networks or more basic filters-based private-network firewalls should consider implementing devices today that can also meet future firewall and VPN requirements.
Remote Access vs. Remote Servers
Another major challenge in NERC CIP, touching several of the standards but especially CIP-007, is control over access to systems in the control center and RTUs, IEDs and other critical cyber assets at the substation such as servers, routers and other network devices. CIP standards require individual user profiles and access controls such that personnel can only get at those systems and devices for which they are authorized–and only for those specific operational functions that they require. Strong access controls include two-factor authentication (such as a both a password and an RSA SecurID token), strong form passwords and archived logging of all sessions. Ideally, logs will include actual session activity. For example, some control systems allow archival of all key strokes used during the session, enabling forensic analysis of user activities should the need arise.
There are some differing approaches to access control. Some are more oriented toward dial-up network arrangements and others to more IP-based WANs, while still others can combine both dial-up and IP environments in a common architecture. Some are based on central control servers, while others require new access control servers at each substation.
An effective way to put an access control system in place quickly is to use an architecture that can work with many existing network arrangements, rather than require new technology at every substation. One such architecture uses one or more secure access manager servers as central gateways for all on-demand communications to substation devices. The central server provides authentication of users, maintains individual profiles and logs all communications. All communication to remote substation devices is from the central (typically redundant) gateways. This means several different WAN technologies (e.g. IP, frame relay, dial-up, fiber Ethernet) and remote substation local communications technologies (e.g. port switch, Ethernet LAN, communications gateways, terminals servers) can be supported, with these differences all essentially transparent to the actual end users (see Figure 3, pg. 34).
An alternative approach is a less centralized architecture that places secure gateways at each substation and requires that the local devices at the substation are all locally re-connected to the new device. This new device is typically a server itself and thus requires its own “patch management” and ongoing system administration as a complex critical cyber asset. In the long run, such distributed gateway/servers may provide greater overall flexibility, but planners can consider the more centralized architecture to enable a quick start, followed by selective, gradual deployment of remote servers.
The impact on end user productivity is another consideration in implementing NERC CIP access controls. Access controls can be implemented with generic IT-oriented technologies that add overhead to the process of end users connecting and getting their jobs done. There are also a few solutions that are specifically designed for use in a substation environment. GarrettCom’s CrossBow Secure Access Manager, for example, provides an electronic security perimeter and auditable secure remote access to intelligent electric devices (IEDs) and other industrial devices.
An appropriate access control system can be a major productivity boost for engineers and operators requiring remote substation access. Authorized IEDs and RTUs can be easily organized in graphical directories; access across complex networks can be simplified to a basic click-through operation, and vendor or device-specific client applications can be preset and automatically launched to facilitate IED interaction.
Considering the potential productivity boost, some utilities are implementing access control systems as a more universal remote access tool, not just for CIP-designated critical substations. In particular, the centralized approach of CrossBow-type systems enables use of a common user environment across all substations without having to put sophisticated electronic security perimeters or other CIP processes at substations that are not CIP-critical. This also positions the utility to more readily expand full CIP controls to more substations in the future.
Advanced Defense in Depth
While the topics above describe some ways that utilities can take measured steps into CIP compliance, other utilities are looking to push forward into advanced cyber technologies more aggressively. There are many measures that NERC CIP requires in the relatively IT-oriented world of control rooms. Servers with general purpose operating systems such as Windows and Linux are more vulnerable to various sophisticated attacks than individual legacy devices. NERC CIP requires “malware” (e.g., anti-virus protection) and careful “patch management” (control over software and configuration file changes). Control centers will typically have a “DMZ” (a so-called demilitarized zone) with sophisticated enterprise firewalls and Intrusion Detection Systems (IDS) to provide greater separation between the control center and the substation WAN on one side and the utilities enterprise “intranet” or any Internet connections or other connections to outside partners on the other. This concept of layer upon layer of threat remediation technology is sometimes referred to as Defense in Depth.
To the extent that some utilities have begun to move general server technologies out to substations, there is a requirement to implement these additional defensive measures including “malware” and patch management processes out to substations as well. Over time, some utilities will likely pursue IDS and other technologies at substations, although these are not required in the current CIP standards.
No Rest for the Weary
The NERC CIP campaign is now well under way. Many utilities are marching forward to major implementation milestones this summer, with preliminary compliance audits already scheduled. Others are still in more of an analysis mode, but realize the clock is ticking. As this article describes, there are still many choices to be considered, and there are approaches that enable some tradeoffs and gradual evolution from pragmatic compliance to ultimate best practices. In reality, cyber security implementation will likely never actually be completed. New threats will emerge. New regulations will emerge. Perhaps most importantly, as utilities proceed with cyber security implementation, they will become more comfortable with the technologies and related processes and begin to view heightened security measures as the normal way of doing business and protecting business assets, not only at CIP-mandated facilities, but throughout the utility network.
John M. Shaw is executive vice president of GarrettCom, Inc., a supplier of substation-hardened networking products. He has more than 25 years experience in telecommunications, including executive roles at network technology start-ups and large carrier-equipment suppliers. As director of data services at NYNEX (Verizon) he pioneered frame relay and fiber-based data services. He has extensive early career experience with design and implementation of large-scale utility-grade data networks. Contact him at email@example.com.