Optimizing the Technology Migration Process
By Leo Verbeek, CIO, EZH, The Netherlands; President, EZH Systems, Hawthorne, NY
The process used by many utilities to evaluate and plan their investments in new automation and telecommunications technology is not only an arduous one, but also one where rationalizing the costs for migration labor, planning and supervision can pose even greater challenges. Moreover, it is not always possible to replace existing systems within a time frame that is acceptable to all of the parties involved, further complicating an already difficult task. EZH, a large electric utility in the Netherlands, was able to overcome many of the difficulties and constraints that are typically encountered in the technology migration process.
EZH provides electricity to the most industrialized and densely populated part of Holland, which includes the cities of Rotterdam and The Hague. One of the main problems faced by EZH was how to migrate from older real-time systems and networks to more modern technology within an acceptable time frame and at a reasonable cost. For example, replacing real-time equipment in high-voltage substations needed to be scheduled in phases when parts of the affected substations were out of operation for maintenance. Under this scenario, it could take as long as 8 to 15 years to migrate just 25 of the 150-kV or 380-kV substations.
After studying the cost problem, EZH decided to use its automation and telecommunications engineering group for in-house development of a new approach to solving the problem. The resulting technology is called Generic Real-Time Network (GRTN). These solutions have been brought to the North American market by EZH Systems, a U.S.-based subsidiary of EZH.
The purpose of GRTN is to extend the natural lifetime of existing real-time systems and telecommunication lines, intermixing old and new technology for incremental migration, as opposed to a complete replacement of all the existing equipment at one time. In addition, the money saved by extending the useful life of the older technologies could then be used to offset migration costs. This GRTN technology, developed and implemented by EZH, has been successfully installed and operating at the Dutch utility since 1990.
The solutions provided by GRTN are particularly well suited for operating in “electromagnetically polluted” environments, such as electric power generating plants and high-voltage substations. The primary market focus for GRTN is on applications such as telemetering systems, telecontrol systems, networks for accountable consumption measurement and supervisory control and data acquisition-like applications for substation and control-center automation.
Advantages of GRTN Technology
GRTN is a flexible technology for building real-time systems. Because it contains many built-in features, the equipment specifications are simplified, resulting in less equipment, reduced maintenance and improved system management. And, like the increasingly popular asynchronous-transfer mode technology, GRTN is also based on cell-relay technology providing numerous advantages–some of which are summarized below.
Almost any type of communication line or channel can be used in the same GRTN network with a mixture of synchronous and asynchronous channels operating at different bit rates. This helped EZH to extend the useful life of its existing networks, many of which were 10 to 25 years old, as well as migrating its communications facilities to a modern SONET network that has now been in operation for more than five years.
Virtually any degree of redundancy can be used. This allowed EZH to build extremely robust real-time networks. In fact, the network used for remote power generator control employs five levels of redundancy and has also been in use since 1990.
GRTN offers a “bandwidth on demand” feature that allows EZH to transmit different kinds of data over the same telecommunication channels while minimizing not only the total number of channels required, but also the amount of physical equipment needed to support varied communications disciplines.
GRTN process computers can be either general purpose computers or digital signal processor (DSP) boards. Initially, EZH used general purpose computers, but the need for routine operating system upgrades proved to be more costly than the relatively modest costs of the GRTN software upgrades themselves. For this reason EZH is now migrating to DSP boards, with programs that execute directly on the DSPs without the intervention of any operating system. Once installed, these boards do not require upgrades to support system growth and migration, a benefit that has allowed EZH to drastically reduce routine maintenance costs.
By offering self-contained or “embedded” networks, GRTN helps to streamline and facilitate the long-range planning process. For example, the difficult course of establishing a cohesive future vision of information technology that is acceptable to (and shared by) everyone throughout the enterprise on a long-term basis becomes far less critical. GRTN technology not only works well when implemented in the form of separate networks but also when integrated into a single enterprise-wide network.
The need to replace existing equipment when applying GRTN is virtually eliminated since the encapsulation properties of the embedded network readily accommodate and support the evolving combinations of old and new real-time equipment that are typical, and often necessary, in the utility environment.
Properties of Embedded Networks
Embedded networks are self-contained networks that form the cornerstones of GRTN technology. A typical embedded network contains access nodes, peripheral nodes and service nodes as depicted in Figure 1.
Embedded networks can only be connected to other embedded networks through access nodes. These interconnections may be redundant, but even in a physically redundant configuration they form a hierarchical structure. A set of interconnected embedded networks forms a larger network with the same properties. However, the specific technology used inside the embedded network is virtually unrestricted, thus solving many problems associated with the technology assessment and selection process, as will be discussed further on.
Different types of equipment, even mixtures of old and new devices, may be used as peripherals of an embedded network with access nodes tailored in such a way that permits almost any peripheral and/or transmission line to be connected, with the service nodes responsible for customized processing of the data exchanged with the peripherals.
A Practical Example
A high reliability GRTN wide-area network 125 miles long and five-fold redundant was built by EZH for the exchange of control data for regulating six electric power generators. This high reliability design was chosen to avoid the severe cost penalties that accrue when the network is not available. Built in 1992, it allows the six generators to be controlled by either the remote-control center or by the local-backup center and has since provided an unprecedented 100-percent availability.
Conventional switching systems could not be used in this case because variations in latencies caused by switching to redundant communication channels triggered electric transients in the control process. These latency variations subsequently distorted the wave forms of analog signals transmitted over the network. Also, existing telecommunication networks containing PCM-lines, modem-controlled lines, circuits routed through PBXs and channels in EZH`s synchronous optical network facilities all had to be part of the final solution (Figure 2).
Reduced Maintenance Costs
New software development unavoidably comes with software bugs. Eliminating these bugs is a part of long-term maintenance. Simply stated: Minimizing the number of bugs means reduced maintenance costs. To facilitate efficient software development while minimizing the number of bugs, GRTN technology provides a high component of reusable software. GRTN software designs follow a well-defined software architecture–termed “resource- oriented” design–that is an extension to object-oriented programming focused on large-scale software reusability. Moreover, access and service-node software is written in the Ada programming language, a language specifically chosen for its excellent long-term stability.
According to the resource-oriented design, software programs are divided into 22 categories of software modules. The internal parts of these software modules are built following object-oriented design techniques that allow the modules to fit into one another in such a way that the program is split into application-dependent and application-independent parts. The reusable modules are then used to implement the part of the program that is application-independent, making this part of the software very dependable since its modules are used and tested in many applications. Naturally, the balance of the program must still be written separately using conventional methods.
The program that is executed on a service node is termed an “embedded program.” The structure of the embedded program reflects the structure of the embedded network. It contains three parts, representing three sets of services:
1. signal services to support signal peripherals,
2. internal services to support the combined working of all service nodes inside the same embedded network, and
3. external services to represent the services offered through an access node to higher-level embedded networks and event handling to support all three parts.
This basic structure of the software architecture is straightforward and easily understood, resulting in a software maintenance, training and support scheme that is far less complicated and time-consuming than would typically be the case for such a powerful yet flexible system.
Simplified System Management
When using embedded networks, system management is constructed in layers. All device-specific support functions are placed on the lowest level of the embedded network hierarchy, but the higher levels tend to become more abstract. In general, this means that the lower-level functions are performed automatically when controlled by the higher levels. Conversely, these functions are performed manually on lower levels through the user peripherals.
This tendency to perform low-level network functions simplifies system management work. With a smaller number of system managers required to control the network, the cost of system management can be substantially reduced. However, this tendency toward more abstraction on higher levels does not imply that lower functions could not be controlled from a central point in the GRTN network. On the contrary, the embedded network is merely a functional unit that can be physically dispersed over a large area, allowing user peripherals belonging to embedded networks lower in the network hierarchy to be placed inside a control center. In addition, user peripherals can also be accessed via public cellular networks. Small embedded networks do not require all of the parts described above.
The use of GRTN technology, even in its most elementary form, yields measurable gains in planning, performance, efficiency, cost and support as characterized by the EZH experience described here. However, the ability to build on these gains through the smooth and unrestricted migration path provided by the GRTN approach is virtually assured, leading to better utilization of current assets and a much broader set of choices with regard to how technology is viewed, planned, selected and implemented in the future.