by David Kreiss and Masoud Abaei, AMI Operations Consulting LLC
Most utilities that have deployed advanced metering infrastructure (AMI) systems have not created centralized operations centers that provide real-time, end-to-end monitoring, analysis and management of their systems. The operation of an AMI system in most cases is shared by a collection of disparate groups often not in the same location. Each group has management responsibilities for a component of the AMI system to include the meter data management (MDM), information technology (IT), communications, cybersecurity and metering hardware. These groups often work normal business hours and are not structured to provide near-time system monitoring and analytics.
A centralized operations center provides many benefits, including:
- Overall lower staffing and resource costs;
- Superior system performance (meter read rates);
- Higher reliability;
- Better security with respect to identifying potential cyberattacks; and
- Superior risk management.
Centralized AMI Operations Center
An AMI operations center is a formal structure for the management of an AMI system to allow the operator to manage performance, security, reliability and risk. It has many characteristics of a grid control operations center. AMI systems are an extension of the distribution system. Although the primary function of an AMI system is the reliable collection of energy usage data, it is expected that AMI systems will support transmission and distribution (T&D) and smart grid activities to include storm management, load balancing, voltage control and home-area network (HAN)/demand response (DR) activities. AMI operations center will need to be structured to address the growing collection of activities.
The operation of an AMI system requires many tasks that vary by utility, depending on the organization of departments–revenue services, meter services, etc.–but most operations centers address the following daily tasks:
- Identifying deviations in overall performance and analyzing root cause;
- Mitigating device issues to include noncommunicating meters, failed meters and connectivity issues;
- Provisioning and commissioning of newly installed devices;
- Metering false positive security alert analytics;
- Conducting over-the-air meter reprogramming and firmware downloads;
- Backhaul management that generally requires activation, deactivation and suspension of backhauls;
- Backhaul business management when using a public network;
- Process updates;
- Security alert analysis; and
- No lights support (load side voltage test).
The goal of an AMI operations center is to complete the daily tasks as defined in the center’s processes and to ensure system performance, maintain reliability and minimize risk. An AMI operations center has:
- A secure, dedicated facility;
- A single, secure and traceable point for ad hoc over-the-air communications;
- Tools or a system to provide situational awareness and real-time system analysis;
- Dedicated full-time staff to address routine and daily processes and triage incidents;
- A complete set of approved processes and service level agreements (SLAs); and
- Work management to ensure business-as-usual tasks and incidents are completed within required SLAs.
AMI Operations Center Benefits
An AMI operations center can improve management of performance, reliability, security and risk all at a lower cost.
Performance. The primary objective of an AMI operations center is to ensure the reliable, secure collection of interval data for billing and that daily read performance targets are met. This requires that read performance be monitored in real time and tools and processes are in place to mitigate identified issues. All aspects of the AMI system should be monitored to identify trends to address issues before they affect performance. Typically for a 1-million-meter deployment, a utility can expect 10,000 to 30,000 meters that may not read on a given day.
The operations center can perform the following to improve system performance:
- Intelligent scheduling of OTA daily jobs;
- Identifying clusters of low-responding meters that could be mitigated by the addition of backhaul devices;
- Identifying and mitigating communication bottlenecks;
- Identifying and alerting on performance degradation of IT components; and
- Rapidly identifying and mitigating backhaul devices.
Optimized read performance translates to fewer pick-up reads and delayed bills.
Reliability. AMI system reliability can be broken down into three categories: field device (meter and backhaul hardware), communications (radio frequency (RF) and IP) and IT (server and software) reliability. The AMI operations center is a key player in all three areas. The meter services organization might have the lead role in monitoring meter hardware issues, but the AMI operations center is the first line of defense in immediately identifying hardware incidents and meter issues that result from OTA programming activities such as firmware downloads. Utility IT operations and maintenance typically are responsible for the AMI IT system but generally are only involved when a server or service crashes. The AMI operations center’s end-to-end monitoring philosophy would facilitate the monitoring of IT system loading and capacity (if not CPU and memory usage) and allow for the early identification and prediction of IT-related issues. Finally, a responsibility of the AMI operations center would be to monitor the communications network, including obtaining RF network statistics, backhaul device loading and the public or private IP communications system.
Security. Cybersecurity attacks can occur directly over the communications system and via the IT system. Either way, obtaining and using meter data is essential to the analysis of a possible attack. Meters record events and issue exceptions that are associated with possible cyberattack. Unfortunately, a number of normal operations cause meters to generate these events. The result is false positive events. Their analysis is time-consuming, which could lead to utilities’ falling behind in performing false positive analytics. This would pose a risk of missing a true security breach. A centralized operations center would have the correlated data, staff and tools to be the logical solution for the rapid analysis of these events to remove obvious false positive and issue-validated security alerts.
Risk management. AMI vendors have a history of exceptional reliability. Although a collection of troubling incidents have occurred, no catastrophic events have been reported. Utilities have never before deployed this quantity of mission-critical new technology assets in their T&D systems. Some could argue the potential risk is not yet understood or there is a lack of ability to identify and mitigate a cascading catastrophic event. Such an event could be a large-scale, remote service switch disconnect, a collapsed communications system or even a cascading meter hardware failure. Whether the origination is a security breach, an IT system bug or a firmware bug exposed during firmware activation, the consequences could be the same.
The AMI operations center plays the key role in risk management, specifically to identify the issue quickly and immediately address the situation. The operations center would supply management quickly with the best information to ensure the best decisions are made and the most reliable, comprehensive information can be communicated to interested parties.
O&M savings. An AMI operations center results in lower operating costs because a centralized focused staff requires smaller head count. The improvement of performance and reliability reduces pickup reads and delayed billing issues. Improved over-the-air analytics reduces truck rolls and meter replacements. In addition, an operations center is naturally structured to provide additional services and analytics easily, which reduce the incremental costs of supporting T&D and smart grid activities. These activities include storm restoration, no lights, high impedance fault detection, load balancing and transformer overload detection projects.
AMI Operations Center Components
Processes. An important, resource-intensive component of an AMI operations center, which is generally underestimated, is the development, implementation and tracking of AMI-related processes. These include monitoring, analysis and mitigation of nonresponding meters, firmware download and responses to cybersecurity alerts. A typical AMI operations center might be responsible for more than 100 processes. A key to an effective operations center is to minimize swim lane activities of groups external to the center.
Staffing. The operations center should be staffed to minimize reliance on external groups to perform their core processes. This requires staff with expertise in communications, meter hardware, security, database and IT. Change management and process staff also are required.
Facility. The operations center facility should facilitate efficient communications among staff. It should include whiteboards, meeting tables and visualization technologies to allow teams to triage incidents and evaluate events. For most large AMI deployments, a large, central visualization display is useful to annunciate incidents and provide situational awareness. A single facility also allows the utility to manage over-the-air activities and provide security that otherwise would be unavailable. Finally, IT must provide the staff within the facility with access (firewall configurations) to data sources used in their work.
Operations center software tools. The operations center must have tools to analyze the huge quantities of meter data collected daily. Tools provided by the vendors have proved insufficient. Vendors generally supply tools that consist of the head-end system (or collection engine) that is the single point of communications with the meters, an operational reporting system and a backhaul (IP device) monitoring and reporting system. These tools are not meant to provide an integrated end-to-end view of the system. They neither consider the data necessary for monitoring and analysis of systems beyond the AMI system nor do they provide real-time situational awareness.
Utilities are addressing the need for AMI operational tools in two ways: through the acquisition of standalone application or through an integrated approach. An integrated system is a better approach that is less costly long term and enhances the benefit of the operations center. The following describes the functionality of an integrated set of tools–the AMI operations center software system–that satisfies the needs of a larger AMI operations center:
- Rules engine for automated incident identification. The quantities of data are too large and the analysis steps too complex for incidents to be identified manually. Many incidents are being ignored and others are not addressed in a timely manner, reducing read performance and increasing risk.
- Data access to the broad scope of data sources required by AMI processes. It is necessary to access from the source system such data as meter and backhaul RMA (returns), backhaul (cell relay) activation status and work orders, fieldwork orders and HAN registrations. In addition, the center’s meter incidents and work management information must be integrated.
- Real-time situational awareness. Operators must be able to identify issues in real time and have the tools to perform basic analytics to understand the emerging situation. Such issues include a cyber alert, cascading communications, field device or IT system failure, or poor read response during a broadcast job. In addition to issue identification, the system must provide information that would help mitigate the issue, as well as the information that would be communicated to management to allow escalated action to be taken. A geospatial application is essential.
- Automated work manage-ment. Work activities must be generated for each incident identified by the system. This ensures the mitigation is performed in an efficient and timely way. The work activity would include the tasks that must be performed to complete the activity and record the timing and person who completed the task for tracking.
- End-to-end monitoring. Inci-dents can be identified reliably only when viewing the system from end to end. This means operators must have insight into the IT infrastructure, including all subsystems, the IP communications system (public or private) and related “data pipes,” the RF backhaul devices, and finally the meters. In some cases, the ZigBee and HAN elements also should be in view.
The Push for an AMI Operations Center
AMI systems challenge the utility industry. Not only is AMI a relatively new technology; it is large and costly. The operation of an AMI system is more complex than anticipated. The analytics of analyzing field device issues are much more involved than expected, and AMI operational activities such as firmware download require many more steps than initially thought.
Utilities are in the early stages of defining the operational needs of an AMI system and formulating structures, processes and tools. A few utilities recently implemented AMI operations centers and developed the required tools. This provides the utility industry with working models that can be used to develop utilities’ own operations centers that meet their needs.
The benefits of an AMI operations center for a larger AMI system outweigh the costs. The benefits include optimizing system performance, operational efficiencies, improved system reliability and security, and the management of risk.
A large AMI system requires a dedicated facility with a skilled staff and documented processes to meet the existing expectations and support planned smart grid projects. Expectation can best be met with a well-structured AMI operations center that provides real-time end-to-end system monitoring, analysis and mitigation.
David Kreiss is a consultant for AMI Operations Consulting LLC. He has worked as general manager of Southern California Edison’s AMI operations center, SCE project manager of the SCMAS (SmartConnect monitoring and analysis system) and founder of Kreiss Johnson Technologies, a utility smart grid software development company. Reach him at email@example.com.
Masoud Abaei is a consultant for AMI Operations Consulting LLC. He has worked as the chief architect of Southern California Edison’s SmartConnect project that included the design and deployment of SCE’s MDM system and SCMAS. Abaei also was the chief architect for SCE’s Customer Service System. Reach him at firstname.lastname@example.org.
PowerGrid International Articles Archives
View Power Generation Articles on PennEnergy.com