Software Maintenance–The Thorn on the Substation Automation Rose
By Lawrence Ricci, Hathaway Industrial Automation
Since the 1980s, the buzzword has been “substation automation.” In spite of the buzz, implementations to date have been limited. The question is why? Why do utilities put in one or two systems, often pronounce them successful, and then stop or slow down the program? The reason, articulated by the utility or not, is often an unforeseen high level of software maintenance.
Sometimes the problem is discovered after installation. For example, one utility wound up subcontracting maintenance, seven days a week, 24 hours a day, to the suppliers at what was considered a high cost. In other cases, problems have been discovered during project implementation. As the number of installed automated stations grows, maintenance, clean up and reengineering to spec the installed base occupies more and more engineering time, slowing down progress to a fraction of what was originally anticipated. By way of illustration, consider the model in Figure 1.
This model assumes a team growing to 10 men, applied full time to substation installation and maintenance. An estimate of 1,000 man-hours is used for each design and installation. Maintenance is assumed to be 300 hours the first year, 200 hours per year thereafter. These are low maintenance estimates, based on office/factory equipment norms. Even so, with such assumptions, the team begins to be absorbed by maintenance only two or three years into the program.
Software Maintenance Problems
Software maintenance is a whole new category of substation equipment maintenance, so cost increases are to be expected. Table 1 explicitly lists some items broadly covered under software maintenance. In a benign office environment, with a network administrator and tight company standards to limit maintenance cost, these items will typically absorb 10 to 60 percent of the capital cost every year. But, for a U.S. utility substation, the costs can be higher.
These tasks are more time-consuming for substations than for office LANs. For example, the simple software maintenance task of “control, alt, delete” can often restart a balky PC program. In an office or in an attended factory operation, this software maintenance can be performed by a non-specialist user, takes only a few minutes of hourglass watching, and never appears on a time sheet. However, in a substation, the same operation plus travel time can take a day or more, and, depending on union rules, may tie up two or more specialist technicians.
Database maintenance is the most pernicious of the software maintenance problems. In the real world of today`s substations, with a crudely adapted industrial LAN bridging between the network interface module (NIM), the intelligent electrical device (IED) and the PC, databases multiply at an alarming rate.
IEDs, NIMs, programmable logic controllers (PLCs), smart networks, PCs, human machine interface (HMI) packages, SCADA systems and remote terminal units (RTUs) all have their own databases containing everything from tag number, attending engineer name and date of calibration to device address, scan rates, scale ranges and alarm limits. Usually these databases are attended by another database in some sort of PC program containing ASCII strings or other data that cannot be stored in the IED. Instruction manuals and operating procedures are another form of database as well. Often the IED`s database can be changed on the faceplate of the device, with no automatic way to fully update the PC configuration database in the off-line PC configuration program.
Making a change in one database usually requires making a change in several, perhaps cutting across the traditional company boundaries of operations/dispatch, protection, maintenance and substation engineering. Even determining which databases are affected by a change can be quite a chore. Table 2 lists the various databases that might be affected by a current transformer (CT) change on a feeder.
A new CT is an easy database change; the variable stays the same and is still connected to the same places, only its range changes. Indeed, in a particular situation (a given IED, substation, etc.) not all of these databases would be changed or even exist.
However, the harder changes are the more common ones. These include adding extra points, new IEDs, and interlock and control program modifications. When this type of change is required, tough problems, like database size restrictions and communication bottlenecks are often encountered. In addition, some related problems, like reformatting reports and displays to fit the new rows and columns along with training required to read the new reports often crop up.
By and large, database maintenance is difficult and tedious work, requiring the comparison of entries on one list to entries on another list. The time required is not trivial, and there is no safety margin or tolerance for error. And finally, there is no easy way to know when the job is complete. Each database has to be identified, presumably by asking a lot of people if they have anything to do with a particular measurement, and then updated. But people`s memories are faulty, and some undiscovered fugitive database might still be lurking out there somewhere. Also, the people who tend this database may be in another department in another city, or they may even work for a different company like an independent system operator, large customer or co-generator.
Controlling the Problem
The need for substation automation/integration is real. The issue of how to meet the need without creating a maintenance monster is big. However, there are things that can be done to mitigate the problem.
The first issue is industry standards. Currently, a great deal of quality work is being done to develop MMS, GOMSFE, CASM, CCAPI CIM, UIB and other vendor-neutral industry wide standards. This however will take time. At the current U.S. electrical demand growth rate, virtually all capacity addition and substation automation will take place in existing substations. An engineer commencing a 20-year career in substation automation should recognize that most of the devices he will integrate are already installed. In addition, the devices that will be added will be chosen because they are similar to devices that are in substations now. While it is likely that the new crop of standards will replace some `du jour` standards like distributed network protocol (DNP) and some ad hoc vendor standards like PLC highways, they simply do not solve the problem for the vast majority of already installed equipment. Only practical engineering will effectively minimize software maintenance on these encompassing systems.
Never Take the Old Bus
The ad hoc, vendor-proprietary industrial bus found at the core of many substation automation projects is software maintenance villain number one. These buses were expedient, ad hoc solutions and should never have been in substations to begin with. Most of these buses do not directly handle SOE and waveform data, two critical data types for any utility operation. They do not meet typical substation specs for power, noise immunity or temperature range. Next, these buses demand nests of NIMs and baskets of bridge-muxes to reduce each IED to the bus` very low common denominator. In many cases these devices cost almost as much as the IED, and of course, they multiply databases with vigor. Finally, most of these “sub LAN” busses prevent or impede remote software maintenance and device diagnostics tied to the bus through NIMs. A relay or meter that once could be accessed via a dial-up line now becomes an island that demands a personal visit for just about all maintenance and diagnostics.
To control maintenance, an industry standard (MMS) bus should be used for devices that are MMS, for the remaining devices a simple radial network back to a RTU or PC should be used. A radial network with IED software drivers offers the simplest and cheapest solution. Such a radial network can scan real-time data as well as support “dial through direct connect” for remote maintenance. It can be provided in fiber or wire and can be delivered with substation environmental specs. Considering the cost of NIMs and rack space, a radial network is probably much less costly than most LAN-based systems. The exception to this rule is the specific device LANs offered by many IED suppliers. These suppliers often recommend, and the smart engineer will employ, an interface device that pulls together multiple relays or other IEDs from the same supplier.
Pick Your Applications
Before deciding what substation equipment to employ, questions should be asked. Such as: What needs to be done in the substation? Is an HMI really needed, and if so, how complete an HMI? It is important to remember that if something equivalent to the SCADA control center is built at each substation, the reasonable assumption is that the software maintenance cost will equal the software maintenance cost of the central SCADA system. Can a simpler man machine interface (MMI) be used, just to reduce maintenance costs? Each application and/or each data point that is integrated/automated brings with it software maintenance. This should be considered in the project cost analysis.
Reduce Software Suppliers
Overlaying software from different suppliers increases exposure to maintenance problems. When one supplier updates its package, will the other packages still run? Who will test compatibility and install the proper configuration? What if it does not run? In particular, is a full-function HMI package in a substation really needed, or will some simple visual basic displays or a real time link to a spreadsheet or browser do? Is it necessary to be dependent on network software apart from the operating system (OS), or can OS supplied features do the job? If the number of software suppliers can be kept down, then the supplier can be expected to manage, at least to some extent, the transition across versions of its software.
Declare Neutrality in the Desktop Wars
To the extent that appealing desktop technology is embraced in the substation, the owner is exposed to the cross-fire between software giants, each leveraging and modifying its specs to cripple its competitors. To declare neutrality, embracing POSIX and TCP/IP standards where they fit can help. These standards are so deeply embedded in the exploding Internet technology they are relatively immune from manipulation. The laser printer is testament to how well stable standards can work. It is a very complex multiprocessing system running software from many suppliers, yet it is still low maintenance and has a long useful life.
Comparative architectures for substation automation should be reviewed for one vital feature–how much of the system can be maintained remotely. In order to do this, list every device in the substation, and along side it, list each maintenance function it might require. Then simply make a score card for each configuration being considered and determine how much of the system can be maintained from remote locations. Ideally, it should be possible to completely configure the substation computer, including OS and network software updates, over the network. Beyond that, it should be possible to access various IEDs, change parameters and perform first level diagnostics. Ideally there should be a single maintenance tool for all this; but realistically, each IED supplier`s configuration/diagnostic software will still have to be accommodated.
Lawrence Ricci has 20 years professional business development and industrial management experience with control equipment and networks. His experience includes significant landmarks such as the formation of the first United States-Soviet Joint Venture in 1988 and the first sale of a U.S. industrial control system to the Peoples` Republic of China in 1977. Ricci has successfully implemented process improvement programs with time cycle and total quality management used to reduce maintenance and operations cost. Currently Ricci is associated with Hathaway Corp. where he is introducing software standards to the company`s various products.
If you would like to see more articles on this topic, circle R.S. 104.