By Sharon Edwards and Greg Stout, Cinergy, and Jeff Gill, Bow Networks
Editor’s note: The following article was completed in mid-March 2006. At the time of this writing, the Cinergy-Duke Energy merger had not yet closed. The merger was expected to close around the beginning of April, after this issue went to print.
Like most utilities, Cinergy currently faces the challenge of improving security and operations while coming into compliance with new and emerging NERC cyber security standards. Those standards were designed to ensure that all entities responsible for North American bulk electric system reliability identify and protect the critical cyber assets that control or could impact reliability.
The permanent NERC standards consist of eight different CIP (critical infrastructure protection) sections, each of which deals with a different aspect of cyber security. Utilities will need to address their compliance with the NERC requirements for each section individually. (The NERC permanent cyber security CIP standard has a proposed effective date of June 1, 2006.)
This article’s focus is on the three NERC CIP sections that impact electronic security.
1. First, NERC CIP 002 provides specifics that the participant must consider when performing a risk assessment and listing their critical asset sites and critical systems, including cyber assets.
2. NERC requires that participants define their electronic security perimeter and all access points to the perimeter with CIP 005. (Participants must also implement electronic access controls and monitor electronic access for all critical cyber assets within the perimeter 24 hours a day, 7 days a week.)
3. CIP 007 sets forth requirements on a broad range of security topics including account management, system security status (log review), and disposal of assets.
Cinergy Gets in Line with CIP
Cinergy’s implementation of NERC’s Temporary Urgent Action Standard 1200 (as a first step to permanent CIP solutions and to enhance security) involved securing critical SCADA, energy management system (EMS), and control center assets with firewalls and intrusion detection systems, and ensuring that only authorized users had access to the critical systems. The monitoring and reporting of the critical assets involved numerous systems and several manual processes.
Cinergy turned to Bow Networks and TECSys Development (TDi) to provide an integrated solution that would bring together monitoring, management and reporting on both EMS/SCADA and substation infrastructure, and to Dymec for the substation hardware to create the electronic security perimeter.
The main components of Cinergy’s solution architecture are:
“- ConsoleWorks IT infrastructure monitoring and management server from TDi;
“- IED Anywhere 2 access server from Bow Networks, which provides remote access control to critical cyber assets in the substation;
“- DynaStar Network Integration Systems (NIS) from Dymec, which provide physical connectivity to IEDs in the substation;
“- Cinergy’s existing RSA ACE Authentication server, providing two-factor authentication for all access; and
“- Cinergy’s existing client PCs, running various native IED applications (e.g. AcSELerator, WinECP).
ConsoleWorks is a real-time monitoring, management and compliance solution. It monitors Cinergy’s critical infrastructure 24×7 and provides real-time notification of events taking place within the infrastructure via direct serial connection to the console ports of monitored devices (or across the network via syslog messages or SNMP traps sent from the device or application). It manages devices by allowing authorized users direct, secure access to the console ports of the monitored devices. In Cinergy’s implementation, ConsoleWorks monitors and manages EMS and SCADA servers, operating systems and applications; PC hardware and operating systems; network routers and switches; disk controllers; firewalls; and, intrusion detection systems.
The solution logs all messages from these monitored devices and applications and provides a centralized reporting engine within their GUIs to satisfy some of the NERC reporting requirements.
ConsoleWorks security features include SSL encryption between the server and client web browsers, and SSH encryption between the ConsoleWorks server and the monitored devices. In addition, TDi developed an interface to Cinergy’s existing RSA ACE server to provide secure two-factor authentication for accessing ConsoleWorks.
IED Anywhere 2 provides a management facility for secure remote connectivity to field devices such as relays, remote terminal units (RTUs), digital fault recorders (DFRs) and other IEDs. It provides productivity gains to users who need to maintain, configure and retrieve data from remote IEDs. In Cinergy’s implementation, IED Anywhere 2 manages access to RTUs, relays, DFRs, data concentrators, substation gateways, breaker monitors and transformer monitors.
IED Anywhere 2 security features include SSL encryption between the server and the client and an interface to our RSA ACE server.
The DynaStar NIS is a comprehensive network solution that provides async terminal server, ethernet switch, IP access router, frame relay access device and legacy protocol mediation functions in a single, hardened platform. DynaStar NIS security features include basic firewall services (IP and TCP address/port filtering), IPsec VPN, and link encryption facilities.
Figure 1 shows how these components fit together in Cinergy’s implementation.
Were Cinergy’s Objective’s Met?
Cinergy implemented NERC’s Temporary Urgent Action 1200 existing cyber security standard in its control area operations center and primary and backup transmission and distribution operations centers. The project’s objectives were to:
1. Provide a central monitoring and reporting facility for Cinergy cyber assets and users;
2. Achieve productivity gains by employing centrally managed systems;
3. Provide secure remote access to all cyber assets at substations for the purposes of monitoring, configuration and management;
4. Leverage existing corporate security systems;
5. Leverage existing and planned methods of substation connectivity;
6. Employ scalable systems;
7. Minimize training and configuration effort; and, most importantly,
8. Comply with all applicable NERC CIP standards.
The company believes all objectives have been met with a general level of success. Detailed results on each objective follows.
Provide a central reporting facility. ConsoleWorks and IED Anywhere provided reporting on all cyber asset accesses and security- and performance-related events. The consolidation of reports from IED Anywhere and ConsoleWorks occurs under the ConsoleWorks facility, providing a browser-based interface that anyone with privileges can access. This reporting mechanism, while integrated, still requires refinement. While the reporting is readily accessible in one place, greater flexibility in formatting and organizing information is required.
Achieve productivity gains by employing centrally managed systems. IED Anywhere 2 and ConsoleWorks each have a centralized database for managing users and devices. Only authorized administrators have access to these databases. One important feature of IED Anywhere 2 is that the users, not the administrators, manage the applications on their own PCs to access their approved devices.
Provide secure remote access. With the administration facility, individual user privileges and passwords can be assigned and controlled. All users are further authenticated using a two-factor authentication mechanism, and all client connections to IT infrastructure and field IEDs are encrypted. This allows users to monitor, configure and manage all devices from a desktop or remote setting, saving time and money. In seeking to secure remote access, Cinergy has derived the additional benefit of now having a global system for all substation device access. It is Cinergy’s intention to deploy the IED Anywhere application across all substations.
Leverage existing corporate security systems. As RSA SecurID is the corporate standard for user authentication, it was necessary that both ConsoleWorks and IED Anywhere support the same technology. This allows our existing users to use these tools as they would any other corporate application. Administration of user and revocation lists is also simplified as it remains in one place. In addition, the use of RSA authentication eliminates the need for password administration in both products.
Leverage existing and planned methods of substation connectivity. The overall solution is well-suited to the use of modern networking technology as well as legacy dial-up systems. The DynaStar NIS can be used on both our private WAN and our public Frame Relay WAN.
Employ systems that are scalable. Both IED Anywhere and ConsoleWorks are licensed by device count, and each are easily upgradeable as the number of monitored devices grows. Because each product needs only IP connectivity between the client and the server, we can easily accommodate users in multiple sites around Cinergy.
Minimize training and configuration effort. Both IED Anywhere and ConsoleWorks were implemented in a few weeks. Training took place during implementation, allowing for a hands-on learning experience. Subsequent additions of new device types have been straightforward and reliable.
Comply with applicable CIP standards. CIP OO2: The concept of tracking critical cyber assets at substations and generating plants, along with the associated detailed reporting obligations and log review, presented new requirements for Cinergy. Cinergy believed manual procedures would not be sufficient to assure compliance and that automated tracking tools would be required. ConsoleWorks and IED Anywhere can be used to print lists of cyber assets periodically for review and approval as well as more complex tasks associated with tracking the devices, which are now centrally managed.
CIP 005: Using the IED Anywhere 2 software-along with the Dymec Dynastar routing capabilities and RSA SecurID-Cinergy employs strong authentication for remote electronic access to substation devices 24/7.
CIP 007: Log review and the desire for automated alerts in response to security events created new challenges at Cinergy. Cinergy knew that an after-the-fact review of access logs would not create the timely security alerts needed. The volume of the EMS logs compounded the problem. Cinergy determined that manual log review was not a feasible alternative.
The project has enhanced Cinergy’s account management controls regarding remote access to substation IEDs. When IED Anywhere is installed, the responsible person configures various defined work groups-each with designated privileges according to work requirements and responsibilities. When new users are added they are assigned to these specific groups as appropriate. In addition, a relationship is created between the user and the associated devices, which are required by their work. In this manner the software ensures that the access privileges correspond with the appropriate personnel’s roles and responsibilities. Lists of users can be printed to perform the periodic reviews as required.
Since IED Anywhere 2 will be monitoring the Cinergy substation devices and ConsoleWorks will be monitoring Cinergy’s EMS devices, the software provides a comprehensive audit trail of user activity and device performance for an unlimited period. This allows for root cause analysis in certain failure situations. Because IED Anywhere and ConsoleWorks work together, pre-defined events identified in the IED Anywhere logs can be sent to the ConsoleWorks database. ConsoleWorks will provide appropriate Cinergy staff with alerts needed on pre-defined events, such as unauthorized access attempts, related to both IED Anywhere 2’s monitoring of substations and ConsoleWorks monitoring of critical EMS devices.
Field deployment of this architecture was completed at the time of this writing. Thus far, a number of considerations have been identified.
Cinergy will use IED Anywhere to provide global remote electronic access to substation devices for all Cinergy substation IED users, regardless of whether the substation falls under the NERC CIP classification or not.
While Bow Network’s IED Anywhere 2 provides the monitoring and controlling of remote electronic access to substations devices, NERC’s CIP standards also provide specific requirements governing physical access to these devices if they are on the utility’s list of critical cyber assets. In the future, Cinergy will have to study various alternatives to determine how to best provide the required physical monitoring and controlling for its substation critical cyber assets and implement the appropriate solutions.
To be “auditably compliant” with the NERC regulations, participants must produce 12 months of documentation to support compliance. This puts a good deal of importance on accurate reporting. Cinergy believes the ConsoleWorks and IED Anywhere products can be leveraged to automate much of the tracking and reporting requirements contained in the NERC CIP standards. In the future, Cinergy will continue to work closely with Bow Networks and TDi to enhance integrated reporting capabilities.
NERC provides specific instructions regarding the identification of critical cyber assets. Among other things, participants must include all cyber assets that use a routable protocol to communicate outside the electronic security perimeter at the critical asset site. Cinergy had previously implemented a frame relay project to enhance communication with its substation devices. New security restrictions may increase the cost of this type of project and cause utilities to rethink their desire to move from propriety protocols to routable protocols, such as IP. In the future, utilities and suppliers must work together to provide new methods of security as the industry moves from proprietary protocols to more advanced protocols. Cinergy will continue to work with its suppliers to identify new solutions.â®â®
Sharon Edwards holds a bachelor’s degree from Xavier University in Cincinnati and an M.B.A. from Thomas More College in Northern Kentucky. Sharon has been a Cinergy employee since 1989. She was the project manager for implementation of the NERC Urgent Action Standard 1200 at Cinergy and is currently the compliance monitor. Sharon has also been active in the NERC regional Critical Infrastructure Protection group.
Greg Stout is a senior engineer in Cinergy’s EMS department. Greg holds a Masters degree in Electrical Engineering from the University of Louisville and has more than 13 years experience in EMS, SCADA and substation communications.
Jeff Gill is president of Bow Networks, based in Calgary, Alberta. Jeff holds an engineering degree from Queens University in Kingston, Ontario, and has been active in the EMS, SCADA and substation integration market for the past 20 years.