Avoiding common AMI/MDM pitfalls

By Jeremy Bisset, Senior Project Engineer, Black & Veatch

Each advanced metering infrastructure / meter data management (AMI/MDM) project has a unique set of challenges resulting from the mix of components and features that the utility chooses to implement. Even relatively mature product offerings can have implementation issues related to new features, configuration of business rules, integration with new versions of products in the suite or unexpected data conditions.

The number of variables in a smart grid project can turn up a surprising number of issues.

The lab vs. the real world

AMI meters are expected to store consumption intervals for a month or more in order to reliably support time-of-use data for new billing rates and customer usage portals. This local storage mitigates the risk of data loss if communications to the meter is lost for an extended time.

The chosen vendor may have shown the ability to backfill missed reads in the lab, but in reality back-filling missed data may only work for a limited number of meters before saturating the network – manual processes may still be required to instigate data recovery.

In the real world, with flickering power conditions and multiple unreliable communications paths, meter data storage may also be insufficient to cope with events and alarms in addition to the consumption history. Be prepared for gaps in customer data, even with automated validation/editing/estimating (VEE) tools, through the development of data management strategies and appropriate business processes. Longer gaps may require manual intervention.

Prior to the advent of customer web portals and time-of-use rates, missed meter reads were not such a problem since the only data being collected was the register read. Bills can still be estimated as long as there is a good read every few months and the bill will reconcile.

If there is a desire to engage the customer by displaying their detailed consumption interval history, or apply time-of-use rates that rely on that history to calculate peak rate billing buckets, loss of data is now evident to the customer.

Mitigating timestamp corruption

Bad timestamps have been received from AMI systems, which can wreak havoc on MDM systems trying to validate consumption gaps, flag changes to previously-billed data, or handle unexpected consumption from the future. Ask each AMI vendor if there are certain conditions that can cause meters to become confused about what time it is, or otherwise lead to corrupt timestamp data. Scenarios to consider should include brown-outs and momentary outages.

Another example of timestamp corruption is incorrect handling of daylight savings spring/fall changes. Past bugs in AMI systems have led to local time jumping by two hours instead of one. These bugs may not be apparent until after the AMI system has been deployed and billing is determined from the collected data.

Be prepared to correct production data as well as resolve AMI system issues after go-live, given that testing may not cover 100 percent of the possible scenarios that will occur in production.

Adding up all the consumption intervals in a month and comparing the total consumption to the register read difference from one month to the next should match fairly closely. Within the MDM system one could expect minor discrepancies due to estimations of gaps, but discrepancies can also occur in the AMI meter itself due to timestamp issues.

Integrating systems successfully

An AMI head-end can transfer duplicate meter data in a single data file if a meter hops between communications nodes. Some MDM systems may mark this data as bad and reject the whole file. Ask the MDM vendor about receiving duplicates from the AMI system in the same batch file, or processes that the AMI vendor uses to prevent its occurrence.

If the customer information system sends transactions to the MDM asynchronously over an enterprise message bus, the order of transactions may matter – such as when an existing customer account needs to be finalized and a new customer moves in same day. Be aware that the CIS may send the related account changes in the correct order but the MDM can process them out of order. Ask the MDM vendor for ways to support asynchronous messaging with processing order preserved.

Provisioning of new meters through web services may work well in small quantities. However, when exchanging non-AMI meters with AMI meters in large numbers (thousands per day during mass deployment), the meter provisioning may start failing due to time-outs.

A common theme with web services is that large batch transactions are not successfully handled due to request/response time limits. The workaround is to break provisioning into smaller chunks — but make sure the provisioning system can do that.

Getting the configuration right

When a system is misconfigured it may not be obvious. Mysterious unrepeatable memory failures in the middle of batch processing can corrupt data and chew up weeks of fruitless investigative time trying to get to the bottom of the problem — potentially due to a single-letter typo in an install script.

Ask each vendor if their system is configured primarily through a forms interface (to minimize the risk of typos in scripts). Configuration mistakes that cause database deadlocks or random program crashes can delay projects significantly.

Reducing the risk

Expect the unexpected. Select a system integrator and/or test team with strong project experience. They have lived these conditions on other projects. Actively participate in the user groups of the selected vendors; other utilities have lived similar challenges.

Actively seek out and engage those that have gone before you – specifically those with the vendor set that you have selected. Allow time in the project plan to implement workarounds – the problems are fixable, but the business case often depends on keeping the AMI/MDM project moving in the meantime.

A surprising number of issues can turn up during a smart grid project. Avoid the common pitfalls associated with AMI/MDM deployment by asking the right questions of vendors, leveraging experienced and knowledgeable partners, and developing a realistic timeline.

Author: With 28 years of experience in the energy/utility arena, Mr. Bisset is a technical leader with extensive experience in project planning and execution, vendor coordination, technical architecture, software engineering, system integration, application design and development. His expertise focuses on smart grid, geographic information systems (GIS), and related technology solutions for utility IT initiatives. Reach him at BissetJW@BV.com

Previous articleDNV KEMA to develop power testing labs in Brazil
Next articleUtility commissions have tools to promote energy efficiency, progress lags

Avoiding common AMI/MDM pitfalls

By Jeremy Bisset, Senior Project Engineer, Black & Veatch

Each advanced metering infrastructure / meter data management (AMI/MDM) project has a unique set of challenges resulting from the mix of components and features that the utility chooses to implement. Even relatively mature product offerings can have implementation issues related to new features, configuration of business rules, integration with new versions of products in the suite or unexpected data conditions.

The number of variables in a smart grid project can turn up a surprising number of issues.

The lab vs. the real world

AMI meters are expected to store consumption intervals for a month or more in order to reliably support time-of-use data for new billing rates and customer usage portals. This local storage mitigates the risk of data loss if communications to the meter is lost for an extended time.

The chosen vendor may have shown the ability to backfill missed reads in the lab, but in reality back-filling missed data may only work for a limited number of meters before saturating the network – manual processes may still be required to instigate data recovery.

In the real world, with flickering power conditions and multiple unreliable communications paths, meter data storage may also be insufficient to cope with events and alarms in addition to the consumption history. Be prepared for gaps in customer data, even with automated validation/editing/estimating (VEE) tools, through the development of data management strategies and appropriate business processes. Longer gaps may require manual intervention.

Prior to the advent of customer web portals and time-of-use rates, missed meter reads were not such a problem since the only data being collected was the register read. Bills can still be estimated as long as there is a good read every few months and the bill will reconcile.

If there is a desire to engage the customer by displaying their detailed consumption interval history, or apply time-of-use rates that rely on that history to calculate peak rate billing buckets, loss of data is now evident to the customer.

Mitigating timestamp corruption

Bad timestamps have been received from AMI systems, which can wreak havoc on MDM systems trying to validate consumption gaps, flag changes to previously-billed data, or handle unexpected consumption from the future. Ask each AMI vendor if there are certain conditions that can cause meters to become confused about what time it is, or otherwise lead to corrupt timestamp data. Scenarios to consider should include brown-outs and momentary outages.

Another example of timestamp corruption is incorrect handling of daylight savings spring/fall changes. Past bugs in AMI systems have led to local time jumping by two hours instead of one. These bugs may not be apparent until after the AMI system has been deployed and billing is determined from the collected data.

Be prepared to correct production data as well as resolve AMI system issues after go-live, given that testing may not cover 100 percent of the possible scenarios that will occur in production.

Adding up all the consumption intervals in a month and comparing the total consumption to the register read difference from one month to the next should match fairly closely. Within the MDM system one could expect minor discrepancies due to estimations of gaps, but discrepancies can also occur in the AMI meter itself due to timestamp issues.

Integrating systems successfully

An AMI head-end can transfer duplicate meter data in a single data file if a meter hops between communications nodes. Some MDM systems may mark this data as bad and reject the whole file. Ask the MDM vendor about receiving duplicates from the AMI system in the same batch file, or processes that the AMI vendor uses to prevent its occurrence.

If the customer information system sends transactions to the MDM asynchronously over an enterprise message bus, the order of transactions may matter – such as when an existing customer account needs to be finalized and a new customer moves in same day. Be aware that the CIS may send the related account changes in the correct order but the MDM can process them out of order. Ask the MDM vendor for ways to support asynchronous messaging with processing order preserved.

Provisioning of new meters through web services may work well in small quantities. However, when exchanging non-AMI meters with AMI meters in large numbers (thousands per day during mass deployment), the meter provisioning may start failing due to time-outs.

A common theme with web services is that large batch transactions are not successfully handled due to request/response time limits. The workaround is to break provisioning into smaller chunks — but make sure the provisioning system can do that.

Getting the configuration right

When a system is misconfigured it may not be obvious. Mysterious unrepeatable memory failures in the middle of batch processing can corrupt data and chew up weeks of fruitless investigative time trying to get to the bottom of the problem — potentially due to a single-letter typo in an install script.

Ask each vendor if their system is configured primarily through a forms interface (to minimize the risk of typos in scripts). Configuration mistakes that cause database deadlocks or random program crashes can delay projects significantly.

Reducing the risk

Expect the unexpected. Select a system integrator and/or test team with strong project experience. They have lived these conditions on other projects. Actively participate in the user groups of the selected vendors; other utilities have lived similar challenges.

Actively seek out and engage those that have gone before you – specifically those with the vendor set that you have selected. Allow time in the project plan to implement workarounds – the problems are fixable, but the business case often depends on keeping the AMI/MDM project moving in the meantime.

A surprising number of issues can turn up during a smart grid project. Avoid the common pitfalls associated with AMI/MDM deployment by asking the right questions of vendors, leveraging experienced and knowledgeable partners, and developing a realistic timeline.

Author: With 28 years of experience in the energy/utility arena, Mr. Bisset is a technical leader with extensive experience in project planning and execution, vendor coordination, technical architecture, software engineering, system integration, application design and development. His expertise focuses on smart grid, geographic information systems (GIS), and related technology solutions for utility IT initiatives. Reach him at BissetJW@BV.com