By Michael Pesin, Joe Orth and Deryk Yuill
As technology has improved, there has been a clear trend toward deploying more intelligent electronic devices (IEDs) in substations. In many utilities, different departments working to solve specific problems have deployed these devices in a piecemeal fashion. Often, a unified vision for substation automation and data acquisition has been lacking.
With this proliferation of substation IEDs, there has also been a dramatic increase in the amount of data available. Currently, much of this data remains isolated in the substation.
At the control center, technology has had a significant impact as well. Traditionally, SCADA and other substation data has been accessed through a hierarchical system, with the EMS at its apex. In this topology, other enterprise-level applications that require this data must get it from the EMS. In many utilities, these other applications are increasing in both quantity and importance. The EMS is still a critical application for operating the utility, but it is no longer the “center of the universe.” Full-featured historians, such as PI, are now playing a key role in operating the power grid, and the use of third-party network and maintenance applications is becoming widespread.
These new applications are frequently interested in data that is produced by the new IEDs but that is not accessible through the EMS. A solution is needed for communicating this data to the applications that need it. Moving it up the traditional communications channels through the EMS is not only an inefficient use of valuable EMS resources, but in many cases the protocols used cannot handle the newer data types.
An architecture is needed that allows data from multiple sources (IEDs, RTUs) to be accessed by multiple hosts (control center or enterprise applications or human users). In other words, a “many-to-many” communications infrastructure is needed.
This architecture should also be flexible enough to support the ongoing addition of IEDs and new applications or queries that require the data. An increasing number of these applications come from an office or IT environment, and have no knowledge of SCADA or IED protocols.
Fortunately, the evolving IT world has been addressing many of these issues. There is a tremendous wealth of mature, mainstream technology available to address the networking, data access and security issues required by this new utility information infrastructure.
Over the past several years, Tacoma Power, a municipally owned electric utility in Tacoma, Wash., has implemented a network-based substation automation system. Ethernet LANs were deployed in substations, and connected to the control center via a SONET ring.
Although certain benefits were achieved through this system, a number of problems remained:
- It was observed that the addition of a direct Ethernet interface to an RTU resulted in noticeably increased latency over their serial counterparts. This was measured at between 100 and 750 mS for a DNP poll. In a round-robin polling scheme, this resulted in Tacoma Power’s two-second update requirement being exceeded.
- Many IEDs have communications deficiencies. Protocol implementations are inconsistent, and in many cases data is transmitted with no confirmation, retry, error checking or security measures.
- Other key operational systems (distribution automation, GIS, outage management, other SCADA systems) had difficulty in getting the data they needed.
- The traditional hierarchy requires considerable maintenance, particularly in data concentrator mapping tables.
- Difficulty in adapting to new standards and requirements, particularly in areas of data sharing and communications security.
The situation had reached a point were the “do nothing” option would not yield a satisfactory solution.
In response to these issues, Tacoma Power initiated a multi-departmental initiative dubbed SDI (Substation Data Integration).
Because the project scope spanned multiple departments, a series of meetings was convened to allow the stakeholders to learn about the SDI vision and comment on it. In the end, a system was defined which provided the capabilities Tacoma Power required, while minimizing the impact on current work practices (see Sidebar, next page, SDI Project Objectives).
The eLAN substation server from Bow Networks was selected as the platform for SDI. It is based on Linux, running on a diskless, substation hardened Pentium III platform, based on industrial PC hardware.
The initial project scope included one major substation (Lincoln) and 17 distribution automation sites (vaults and pole-tops).
The figure on Page 20 shows a (greatly) simplified version of the SDI architecture. Networking components such as routers, switches and firewalls have not been shown, nor has the overall corporate network topology.
Note that there are two SDI servers. Each performs roughly the same functions, but for a different set of field devices. Lincoln substation has many IEDs, which justified a dedicated SDI server in the substation. Connections to the SDI server from the IEDs are either direct serial connections or over the station LAN.
The second SDI server is located at the control center. It provides access to data in a number of IEDs and RTUs scattered throughout multiple smaller sites, such as pole top distribution automation (DA) RTUs and distribution vaults. A small, hardened terminal server provides a remote serial port at the field device, with all associated software located in the control center SDI server. Since the eLAN solution is a network-based architecture, distributed implementations such as this are possible, resulting in a much lower system cost.
A primary objective of the system is to provide multiple, simultaneous host applications accessing each IED. The following are the primary applications deployed. Note that in many cases, a single data point is served to more than one of these applications.
EMS System Interface
The EMS polls for data from each of the SDI servers. The server provides a single, virtual RTU for each physical device. Because of the SDI server’s processing power and the bandwidth available on the WAN, there is no need to concentrate multiple physical devices into a single virtual device. Maintaining a one-to-one relationship between physical and virtual devices allows point maps to be maintained from end to end through the system, resulting in an easy-to-understand system, and considerably reduced configuration effort.
Note that even though the network-based RTUs could be polled directly from the EMS, they are mirrored through the SDI server as well. This is to take advantage of the much lower poll latency of the SDI server. See http://www.bowsoft.com/downloads/dnp_poll_perf.pdf for further details.
Long term, only a small subset of the available data will be reported to the EMS. This traffic has priority over all other data.
PI Historian Interface
The PI historian is the repository for all IED data that is believed to be of interest, either now or in the future. The SDI servers maintain a separate connection with PI, based on the OPC (OLE for Process Control) standard.
Tacoma Power intends to deploy the necessary desktop application that will allow all interested staff to query the historical database for specific information, mine data and build their own custom displays.
DA Master Interface
The DA master interface functions similarly to the EMS interface. The control center SDI server collects data from the various DA RTUs and network protectors, and creates virtual RTUs for each, which the DA master polls for.
Fault Record Management
The SDI server is responsible for extracting fault records from IEDs as they are captured, and archiving them on a central file server at the control center. Interested staff may receive notification of record capture via e-mail. This central repository for all fault records provides a consistent access point for all staff that needs those records.
Remote Maintenance Access
Another goal of the SDI project was to create an easy-to-use mechanism for remotely accessing maintenance ports on all remote devices, from the vendor’s configuration/maintenance application. The eLAN IED Anywhere application provides this functionality, and provides a central administrative tool for this.
The IED Anywhere Administrator allows the creation of “sessions,” which are one-to-one associations between a specific IED and its maintenance application. The session is given a unique, unambiguous name so there is no confusion as to which device is being connected to. The Administrator also allows individual users to be defined, and specifies which users may access which sessions.
Network and application security is a broad issue, which should be addressed in a holistic manner across the utility. A unified security vision must exist that accommodates all networking, from SCADA data to remote e-mail access for corporate executives, to Internet access for employees.
Tacoma Power has viewed the SDI project’s security requirements as an extension of its existing, multi-tiered network architecture. The eLAN family offers a number of security options, compatible with the existing technology, and allows the system to follow the NSC recommended security practices.
Consistent with these practices, and with Tacoma Power policy, specific architectural details remain confidential.
Tacoma Power’s SDI project has illustrated the benefits of building a data communications infrastructure on powerful, mainstream technology. Based on the success of the pilot project, Tacoma Power plans to roll out the SDI concept throughout its network.
Michael Pesin has worked for Tacoma Power for the last 10 years in the areas of SCADA, EMS, substation automation, distribution automation (DA), substations and generation controls, communications, protection and metering. Since 1996, he has been a special project manager for Tacoma Power’s T&D operations.
Joe Orth held various positions at Tacoma Power from 1991 to 2003 in the generation, power management and T&D sections. From 1997, he was special project manager for the power system automation & IT group.
Deryk Yuill is Bow Networks’ vice president of business development. He serves on the IEEE substation committee, and on IEC TC57, working Group 15, which is defining security standards for substation communications.
Tacoma Power’s SDI Objectives
- Provide open, flexible access to all substation and distribution automation devices, from any authorized user or application.
- Decrease communications latency for individual devices to Provide timely, accurate delivery of time-stamped events and system data including targets from relays to improve the outage management and system event investigation processes.
- Leverage mainstream hardware and software technology as much as possible. Tacoma Power had a strong preference for the Linux operating system, due to its robustness in a networked environment. Additionally, there was a preference for products based on commercial, off-the-shelf (COTS) hardware, to ensure flexibility, and to allow future technological advancements to be easily accommodated.
- Support current and emerging networking standards, including a full range of security options.
- Reduce system integration and configuration complexity.
- Provide a simple, secure method of controlling user access.
- Preserve investment in legacy field devices and control center applications.
- Scalability, so SDI’s benefits could be realized by small vault and pole-top installations as well as major substations.
Specific functional objectives:
- Provide all desired SCADA data to the EMS, which would then provide data to PI for the pilot only.
- Provide all desired IED data to the PI historian. This will be the primary repository for field data, for user queries and reports.
- Provide intelligent management of event data from IEDs (filter duplicate events).
- Provide automated management of fault records.
- Provide remote maintenance access to all field devices.