Hardened Infrastructure for Distribution Automation

By Lee House, GarrettCom

If the smart grid is to manage capital equipment, power network load, distribution efficiency and fault prevention and isolation efficiently, then the communication networks that support these goals must be designed for continuous uptime and long life.

While some of the back-end networking, computing and applications can use office-grade data processing equipment, much of the networking equipment needed to deploy a smart grid infrastructure is field gear subject to harsh environments and demands. Appropriately hardened products from the network edge, where intelligent electronic devices (IEDs) are first connected to the network, all the way through to the switches and routers that transmit data to the information technology department, provide the greatest level of reliability.

Reliability and Redundancy Inside and Out

One way to harden distribution equipment is physical: toughening up the components. Office-grade equipment is designed to operate in a controlled-temperature environment, typically 0-40 C. The communications equipment in a power distribution operation likely is to be installed in environments below freezing to more than 80 C. In addition, environmental factors such as dust, insects and moisture add to reliability challenges. Industrial-grade metal casings, often supplemented with conformal coatings to seal out moisture, are a key component of hardened electrical systems to ensure reliability. Electrical systems must be protected from electromagnetic interference (EMI), as well, when hardening communications systems for the smart grid. This happens at two levels: fiber media for cabling and the metal cases typically used for industrially hardened communications equipment, which also provide shielding.

Another way to harden distribution equipment is less physical: Reliability can be added to with redundant paths. Designing redundant network paths to every critical device through a ring topology is a simple, effective method to increase uptime in power substations and distribution systems at the edge of the communications network.

Redundant ring solutions provide two points of connectivity in a ring with one forwarding or operating port, and one backup or standby port that becomes the forwarding port when the primary port becomes inoperative because of a broken link. (Figure 1 shows a configuration in which the original primary port has been blocked and the second port has taken over.) With the latest version of Rapid Spanning Tree Protocol (RSTP 802.1D-2004), low, subsecond recovery times can meet the performance requirements of almost all mission-critical applications.

Dual homing also adds reliability by allowing a device to connect to the network by way of two independent connection points (see Figure 2). One access point is the operating connection. The other is a standby or backup connection that is activated during an operating connection failure.

Serial device routers (see Figure 3) integrate the functions of a terminal server, an Ethernet switch and an Internet Protocol (IP) router with a firewall, which is an additional help with that less physical reliability. New per-virtual local area network (VLAN) routing technology can allow a serial device router to operate as multiple virtual Ethernet switches, multiple virtual terminal servers or both at the same time.

Hardening With Security in Mind

Fiber traditionally has been employed as the backbone media and is the preferred medium for noise immunity and long-distance connectivity. It also has benefits in high-EMI environments. An added fiber benefit is its extra security protection within a complex such as a substation. Security is a growing concern, and, as opposed to copper, a typical media line tap will not work on fiber.

Like reliability, security is benefited by redundancy. One good defense is a layered approach that uses several forms of network security to protect against intrusion from physical and cyberborne attacks. The layers are set up to work in parallel, one technology overlapping, in many cases, with another. Together they form a significant safeguard against attack.

Figure 4 shows a simplified power utility network with multiple access points and network hops (private and public) that is wide open to abuse from cyber or physical attacks. A layered approach to hardening this area of distribution could include firewalls, virtual private networks (VPNs), VLANs, access management systems and secure access management systems, secure network management, video surveillance and physical access restrictions. All offer positives to help layer security:

  • Using firewalls at entry points to the core network and all remote facilities provides a gate to protect and ensure that nothing private goes out and nothing malicious comes in.
  • VPNs layer onto a more general network using specific protocols or methods to ensure private transmission of data, tunneling across the transport network in an encapsulated, encrypted and secure format.
  • VLANS make it possible to segregate the different traffic flows (such as voice-over-IP, video, management and control applications) into separate broadcast and multicast domains, keeping applications more secure by limiting where applications are visible.
  • Access management system servers obtain credentials from the end user and authenticate the user and target devices he or she is authorized to use after interrogating security or authentication systems. Secure access management systems enforce specific security policies, ensuring only authorized people can access systems and networks electronically.
  • Secure network management requires each network element to implement secure management interfaces requiring rigorous authentication and authorization, as well as local logging and remote event notification regarding status, configuration changes and network security events.¬†Video surveillance technologies and physical access restrictions offer a real-world security layer. Access control devices (such as electronic card readers, video cameras and pixel-based systems) and physical restrictions prohibit unauthorized devices from being plugged into an Ethernet switch or router or a terminal server using technologies such as VLAN.

Introducing simple security elements into a power utility substation and distribution network significantly will reduce risk of physical and cyberattacks. The resulting network topology should look much like Figure 5.

Smart grid hardening strategies include reliability. In a time of evolving standards and government regulations, identifying adaptable equipment, software and protocols provides for cost-effective, long-lived, secure solutions. The best hardened solutions take advantage of the best network solutions and are flexible enough to evolve.

Lee House is executive vice president and chief technology officer at GarrettCom. He has more than 20 years’ experience in research and development and product development with a focus on LANs, wide area networks, and IP at companies including 3Com, IBM, and Jetstream Communications. Reach him at lhouse@garrettcom.com.

Hardened Distribution Case Study: Blue Ridge EMC

When Blue Ridge Electric Membership Cooperative (EMC) wanted to plug into the smart grid, the first priority was to provide reliable, IP-based communications services in its demanding northwestern North Carolina service area. Much of the territory is in the Appalachians.

Blue Ridge had to provide communications to remote locations at a reasonable cost to enable its automatic meter-reading (AMR) system to read electric power meters remotely with a granularity of up to an hour. Fiber connectivity at substations is the logical choice for backhauling meter-reading and load analysis data to the corporate office. Where intelligent electronic devices (IEDs) have been installed, engineers can analyze fault data, and dispatchers in the operations center can ping individual meters to determine exactly where an outage has occurred.

To build out this project, the Blue Ridge team needed switching equipment that was hardened to withstand the electrical and environmental extremes found in substations and beyond in the distribution system. In addition, new equipment had to be compatible with the existing network equipment, had to meet North American Electric Reliability Council (NERC) Critical Infrastructure Protocol (CIP) requirements, as well as be flexible enough to support anticipated future directions, and had to be easily monitored and managed remotely.

Security gateways made by Astaro Corp. and Magnum 6K Managed Switches from GarrettCom formed the basis of the communications network. Where fiber has been deployed, it is connected directly to the Magnum switch at the substation. To securely transmit information over the DSLs, the security gateways act as a firewall between the substation network and Internet. The network switching equipment protects the substation network and transmits data over a separate DSL to corporate. All unused ports on the Magnum switches are disabled to further enhance security. Fiber was used to deploy multiple VLANs to segregate engineering applications and corporate Ethernet traffic. DSL does not support VLANs and, therefore, works best in distribution stations that have minimal transmission equipment. Figure 6 shows the new substation and distribution layout that is a combination of Ethernet-connected IEDs and serial links (see Figure 6 below).

More PowerGrid International Issue Articles
PowerGrid International Articles Archives
View Power Generation Articles on PennEnergy.com
Previous articleWhere the Standards are
Next articlePowering the Smart Meter Promise

No posts to display