By John M. Shaw, GarrettCom Inc.
A good substation network invites rather than inhibits growth in automation. Effective network strategies make each successive substation automation project easier to implement, building on an evolutionary plan and avoiding self-imposed barriers to functional or capacity expansion. Rarely, if ever, does a project engineer get to deploy an ideal long-term network infrastructure as part of an incremental substation automation project. The pressure is to meet the current project requirements most expediently. Even with deployment of brand new substations, projects are constrained by pre-existing processes and systems, limited budgets and timeframes, and training and experience. With upgrades to current substations, the weight of long-term investments on the ground makes these constraints more binding.
Poorly planned substation network infrastructure deployments can become a technological or economic barrier to the next project that comes along. With a thoughtful plan, however, the network can evolve, each step building on those before, and each step making the next project easier to accomplish.
What Can Go Wrong
Common mistakes in network planning include having too narrow a technology focus, missing still-emergent product requirements, introducing too many vendors with no clear strategy for management and interoperability, and making tactical compromises on requirements such as product hardening.
When technology domains are not viewed in the context of a single holistic architecture, problems arise. For example, some plans correctly identify Ethernet switching as the clear winner for long-term core substation infrastructure, but then deal with the world of non-Ethernet protocols and technologies (e.g., serial-based IED interfaces, cyber security and carrier-based WAN services) as separate planning decisions. Walls can arise between these worlds and reduce system flexibility and create duplicate costs.
Missing Future Features
Planners need to scan technology developments and project likely future requirements onto current product decisions. A good example is cyber security. There is a danger of paralysis as the emphasis on planning and “compliance” with industry standards is strong right now-but complete clarity on requirements is still missing. The recent NERC Critical Infrastructure Protection cyber security standards for substations (CIP-002 to CIP-009) present a foundation, but these requirements will surely evolve over time and become increasingly sophisticated.
Too Many Vendors
Different project champions often find different point solution, i.e., different specific vendor products which appear to offer the most expedient solutions to their short-term needs. But, introducing too many vendors into a network environment creates issues, such as higher training and administration costs, more potential for interoperability problems, and less consistent approaches to critical issues such as network resiliency, which needs to be coordinated across several different levels of network technology.
Narrow, project-specific product decisions may also lead to dropping standards for product performance, in particular, meeting environmental hardening requirements specific to substations. While such compromises may lead to quicker, lower-cost implementations, they also may lead to shorter-lived, less reliable investments that future planners will need to replace.
Key Principles for Future Success
A few key techniques can help substation networks evolve effectively. First is a comprehensive architectural vision that builds on an Ethernet technology core with multiple complementary technologies. Second is a strategic requirements forecast that looks into the future to anticipate emerging feature requirements. Finally is an analysis of “Total Cost of Ownership” that drives decisions toward long asset life and low operational cost.
The most fundamental requirement is to define a clear and comprehensive long-term architectural vision for the substation network. Ethernet technology belongs at the core of this architecture. This is consistent with dominant technology trends and is recognized by IEC 61850 and UCA architectures. Most utilities have already begun deploying Ethernet switching as the basis for new systems in larger substations. With extremely wide adoption, Ethernet has become a low-cost physical media, universally accepted across IT systems suppliers. In substation environments, it provides enormous data network capacity. In larger substations, the Ethernet core requires multi-Gigabit capacity switches with growth capacity for video, high-volume file transfers and high-priority process control traffic.
A holistic architecture includes several elements surrounding this Ethernet core, including the Ethernet edge, WAN access and the serial edge-all of which are inter-related, inter-operable and integrated with network-wide approaches to network resiliency and network security (see Figure 1).
The Ethernet edge network extends fiber media effectively throughout a substation and connects Ethernet-based IEDs back to the core network. The most basic edge network is point-to-point links with fiber-copper media conversion units. Many substations now have hierarchical edge networks using multi-port Ethernet collector switches. These come in a variety of compact form factors and can be effectively panel-mounted throughout a substation. Ideally, these distribution points have dual-homed fiber connectivity to the core network, now available in both managed and some unmanaged switches. While the collector switches may be unmanaged devices today, increasingly the Ethernet edge will use compact managed switches that provide additional resiliency, access security and network event monitoring capabilities throughout a distributed substation network.
A similar transition is under way at substations for connecting IEDs that have serial protocol interfaces. At present, many serial edge connections use static serial-over-fiber link/repeaters to extend IED connections from centralized data communications processors or terminal servers. In the case of some widely deployed SCADA protocols, the associated systems are not readily integrated with Ethernet-based core architecture and require special handling on wide area network connections back to SCADA masters. But, for full-time or occasional-use access to serial-based IED administrative ports and for many serial SCADA protocols (e.g., serial DNP-3), serial-to-TCP/IP protocol converters (device servers/terminal servers) are often placed at centralized substation hubs to provide integration with the IP/Ethernet core infrastructure. As with the Ethernet edge, this device/terminal server function will increasingly be distributed using compact devices deployed throughout a large substation. This more dynamic serial edge network will provide dual fiber connectivity and resilient networking features and extend security (e.g., SSH/SSL) to the connection point of remote serial IEDs. In some cases, multi-purpose devices will provide both serial and Ethernet edge connectivity.
Finally, the substation must be connected to the outside world, usually to redundant control centers via wide area networks (WANs). Many substations are still making the transition from per-application dedicated leased lines or dial-up connections to integrated WANs across all substation systems. The demands placed on WAN access are rapidly changing. For example, to integrate SCADA with other applications, WAN access must be able to effectively prioritize application traffic differently, giving SCADA preferred treatment. As discussed below, WAN service options from carriers are continually evolving and access devices should be flexible enough to support multiple options. Also, the WAN access layer increasingly must play the role of “electronic security perimeter” for substation cyber security. WAN access solutions must meet these varied external requirements while effectively integrating with Ethernet core and serial edge architectures.
Not every substation automation project will impact all elements of the substation architecture. What is important is to be able to fit any incremental element into the larger picture so that no architectural barriers are created that inhibit utilizing a more comprehensive network design in the future.
Strategic Features Forecasting
In setting requirements for near-term projects, planners need to also look long-term toward new requirements that may emerge for this project or future projects in the same substation environment. One clear area of concern globally is cyber security. In North America, most of the cyber security attention is focused on the recently approved NERC CIP (Critical Infrastructure Protection) standards, and for the purposes of substation data network infrastructure, most particularly on CIP-005 Electronic Perimeter Security and CIP-007 Systems Security. However, these standards are only a baseline. Most strategic planners can already project additional requirements that will emerge. NERC may act on its own or be influenced by initiatives sponsored by DOE, DHS or other Federal mandates. Even industry best practices will continue to evolve. For the planner, this means a high likelihood of additional requirements or more stringent definitions of requirements already proposed (see Figure 2).
Electronic Perimeter Security (CIP-005), for example, now requires “Firewall” capability on all routable network connections, meaning all IP based connections leaving the substation boundary. As a baseline, this requires IP address and related TCP port filters or Access Control Lists (ACLs) to prevent unauthorized IP users/sessions from gaining access to the network. Also, inside the substation, unused physical and logical ports must be disabled. Systems Security (CIP-007) requires network elements themselves (as “systems”) to have user access controls with strong passwords and differentiated user profiles, as well as extensive logging of security, management and network events and activities linked to centralized archival, auditing and compliance management systems.
Additional security features that are not yet specific requirements, but are recognized as best practices, include SSH for serial (e.g., CLI) console access, SSL/SHTTP for access to web-based management interfaces and SNMPv3 security for system-level management applications. Other “futures” that should be taken into account include Virtual Private Networks with IPsec, strong data encryption such as AES, and Intrusion Detection Systems (IDS) in substations, which are now mostly limited to control centers.
Changes in carrier WAN service offerings are another area where technology trends need to be taken into account. As an example, Frame Relay remains a popular and extremely effective service technology for connecting distributed substations to control centers, supporting resilient connectivity and providing rigorous traffic prioritization. However, new MPLS-based private IP network services, i.e., virtual private networks using Multi-Protocol Label Switching (MPLS), are being positioned by major carriers as strategic replacements for Frame Relay, as well as for dedicated leased analog or digital circuits.
Planners can never see all possibilities nor can they afford to cover all contingencies, but it is important to probe suppliers about their own awareness of industry trends and their current and planned accommodation of foreseeable future requirements.
Total Cost of Ownership
The keys to optimizing Total Cost of Ownership (TCO) are to build for a long-lived installation and to factor in both immediate equipment costs and longer-term costs of maintenance, replacement and ongoing systems integration.
A basic consideration for substation project longevity is network product hardening. IEC 61850-3 and IEEE 1613 standards define equipment requirements that increase product reliability in harsh substation environments. Meeting these standards involves higher product immunity on both power input and I/O interfaces to surges, fast transients and other electromagnetic events common in substations, as well as supporting extended high and low temperature ranges (-40 to +85° C). Many “industrial hardened” products claim some heightened electrical immunity relative to basic commercial grade products, but “substation” standards are considerably more stringent (see Figure 3). Also, while some products can tolerate short term exposure to harsh environmental factors such as high temperatures, sustained high temperatures above product ratings significantly diminish long-term product failure rates (MTBF). Planners should look at both third party “type test” results and more extensive product MTBF analysis before depending upon products in substation settings.
The number of different vendors introduced into a substation network environment significantly impacts maintenance costs, training costs and vendor administration costs. Having too many vendors sacrifices “economies of scale” in all aspects of vendor interaction. Perhaps most importantly, with multiple vendors, there is no focused responsibility for interoperability and systems level integration. A role of a network architecture is to identify key standards for interoperability among network elements and make this set of technology standards (e.g., RSTP, VLAN, SNMP, SSH) part of the baseline that all products must meet. In the end, nothing helps to prevent or resolve interoperability questions better than the handy “one neck to grab.” Suppliers who can provide all or most of the design elements needed both now and in the future add value far above the sum of their product parts.
An effective substation automation network plan begins with clear and comprehensive target architecture and then proceeds progressively forward without setting traps that lead to short asset lives and network redesign. The effective plan should enable, not impede, the ongoing expansion of automation applications into substations.
John M. Shaw is executive vice president of GarrettCom Inc., a supplier of substation-hardened networking products. He has more than 25 years experience in telecommunications including executive roles at network technology start-ups and large carrier-equipment suppliers. Contact him at firstname.lastname@example.org.