Precision Forecasting: Using AMI, AI and Machine Learning to Manage Load Uncertainty

The challenge

Utilities big and small, regulated and unregulated, face unprecedented challenges to their operations, especially load forecasting. The proliferation of variable renewable capacity, the closure of predictable baseload capacity, more extreme weather and climate-related events combined with a diminishing lens into customer behavior and patterns is really the perfect storm. Simply put, the relationship between supply and demand is becoming more dynamic (erratic even), opaque, and imbalanced.  

We see this in all of the organized markets as well as areas without organized markets. From 2007 to 2016, 531 coal units representing 55.6 GW of capacity were retired across the U.S. Most of these retired coal units were in the PJM , and the Midcontinent ISO territories. Since the start of 2018, Texas alone has lost three coal plants. These closures will not be the end as more are predicted across the country– such as in New York, where all remaining coal plants coal plants are slated to close.

A new federal report, “Impacts of High Variable Renewable Energy Futures on Wholesale Electricity Prices, and on Electric-Sector Decision Making” confirms the addition of variable renewable resources onto the grid can have profound impact on pricing. In deregulated markets like ERCOT, prices drop to average lows of $10/MWh and then spike in the evening to an average of $80/MWh. In a market with a large number of intermittent renewable resources, price volatility and changing patterns, the report notes that ancillary service prices rise up to 8-fold. This already high price variability will likely be exacerbated this summer as warmer than usual temperatures are predicted. It is these shifts in generation mix that have given us the now infamous CAISO duck curve, and its cousins.

At the same time, the growth of behind-the-meter, customer-sited solar, which introduces supply unpredictability into power markets, continues unabated. Installations continue to grow as states develop new policies to drive deployment, such as the recent California mandate to require solar installations on almost all new home builds beginning in 2020. In antiquated models, this proliferation of solar shows up as a reduction in demand for utilities. In the case of California, this shift in generation is predicted to create demand challenges for energy providers, especially as storage lags behind rooftop installations. Additionally, this increasingly decentralized generation means energy providers are losing insight into load profiles and their ability to predict energy usage.

As the generation mix continues to evolve and delivery business models reinvent themselves, the legacy analytics and top-down forecasting approaches that worked in the 20th century simply do not work in today’s business environment. These approaches were built for predicting aggregate load obligations, and not the more granular customer level distinctions necessary in managing today’s grid. With large amounts of AMI data increasingly available, assumptive models do not allow for two-way data and disaggregation of consumer data. There are limitations that impact a utilities’ entire line of work–from load forecasting to management to matching customers with the right energy efficiency programs. There is a growing need for real-time data to be processed and analyzed in order for grid operators to make informed and profitable decisions.

As we get deeper into summer–the most volatile time of the year for utilities–grid operators require modern technology solutions to this fast-growing challenge.

The Opportunity

The ability to continuously anticipate demand and supply dynamics at every level of the value chain and at nearly every point in time is vital in so many ways, from energy procurement and risk mitigation, capacity planning, distribution grid resilience and reliability to personalized customer engagement, lower cost acquisition and retention. Legacy analytics and forecasting provides a top-down approach that simply cannot capture the increasingly intertwined relationship between supply and demand. 

The lack of accurate and granular customer level forecasts poses risks to the stability of our power system. The fact that we are not in a static production and delivery environment means that we are distributing power through a grid into a level of demand where we have less and less visibility. Companies can fill a big void by telling us what’s happening on the demand side of the equation, so we can design, plan and operate our grids for contingencies across the value chain.

Distributed resources, specifically solar provide challenges not only in terms of shifting demand, but also in terms of visibility and understanding customer usage. When the sun is shining, the generated solar being used appears in traditional models as a reduction in demand despite power consumption being the same. In the case of weather sensitivity, it is important for utilities to have insight into customer usage patterns in order to maintain grid stability and to avoid hedging costs. Weather sensitivity has profound impacts on the grid and can drive up costs, especially in competitive markets.

Fortunately for the modern utility, there is a solution. With the proliferation of smart meters, large volumes of granular AMI data are increasingly available, giving utilities greater insight into the areas of their operations that have become opaque as the energy landscape has changed, especially customer usage data. This treasure trove of granular customer usage data can give utilities far greater visibility at the grid edge than ever before and enable them to make actionable decisions based on that information. However, as many utilities have discovered, access to AMI data is only half the battle.

To use behind-the meter-data effectively to make planning and energy procurement decisions, systems have to be able to consume and use that data in near real time. This has created a paradox: the very data we need, and that is now largely available, simply cannot be brought back, processed and integrated into network modeling in time to enable the kind of real time decisions needed to manage today’s more dynamic operating environments. Even the most sophisticated network planning operating models provided by leading vendors are still based largely on assumptive models and aging research driven off of high-level assumptions of customer usage. This limitation has implications across a utility’s entire line of operations, but especially in load forecasting.

Companies which combine bottoms-up analysis of AMI data with machine learning and artificial intelligence develop precision forecasts that are 40-60 percent more accurate than incumbent approaches. This radically improved methodology has the power to transform a utility’s approach to forecasting and can lead to substantial savings in operating costs. In a deregulated market, for instance, a forecasting improvement of just a few percentage points at just the right hours can save REP millions of dollars and significantly decrease margin volatility.

This technology-enabled approach to forecasting is vital in 2018 and will only grow in importance as summers become hotter, weather becomes more volatile, dispatchable baseload power is retired, and distributed resources become more widely adopted. By feeding predictive intelligence that was previously not possible to customers, the grid and other vital utility functions, utilities can create value across the business.

As the energy value chain becomes more complex and dynamic, the utilities adopting more robust and accurate forecasting technologies will build a strong foundation for sustainable business growth, even as the utility business model continues to undergo significant, costly changes.

About the author: Bob Champagne is senior vice president at Innowatts, an energy technology company focused on transforming the energy value chain through its use of predictive intel, machine learning and AI enabled technologies. To date, Innowatts’ eUtility platform, via its clients–regulated and deregulated utilities, smart energy communities–has enabled over 15 million users with lower across-the-board costs and a personalized energy experience.


Previous articleABB wins order to grid-connect UK Hornsea offshore wind farm
Next articleEPRI wins $1.8 million for solar forecasting technology research
The Clarion Energy Content Team is made up of editors from various publications, including POWERGRID International, Power Engineering, Renewable Energy World, Hydro Review, Smart Energy International, and Power Engineering International. Contact the content lead for this publication at

No posts to display