How to Transform Data Into Value-added Information Through Analytics

by Bijoy Chatt and Sam Sankaran, Navigant

Large-scale deployment of advanced metering infrastructure (AMI) and intelligent energy devices (IEDs) combined with grid edge power conditioning equipment and availability of sensor data has created a tremendous need for data storage and analysis. The application of analytics, however, lags in several areas.

Big Data’s Growth

International Data Corp. (IDC), a provider of market intelligence in the information technology (IT), telecommunications and consumer technology markets, predicted big data will grow 30 percent in 2014 by developing “data-optimized cloud platforms” that will leverage high volumes of real-time and nonreal-time data streams. On the flipside, Gartner reported in 2013 that big data didn’t drive big growth in the worldwide business intelligence and analytics market.

“Even though big data hype reached a fever pitch in 2013, this did little to move the dial for analytics,” according to Gartner.

The business intelligence and analytics market grew some 8 percent to $14.4 billion in 2013; the uptick could have been even greater. Only 8 percent of organizations surveyed by Gartner have deployed a big data project, with some 57 percent still in the research and planning stages. Regardless, the analytics of the large volumes of data appears to be one of the most critical, yet lacking elements of big data. The analytics value proposition also remains unclear to many in the industry.

What is Big Data?

Big data refers to data sets with size beyond the ability of typical database software tools to capture, store, manage and analyze. Structured data is obtained from data that conforms to a construct or has a preset pattern that can be analyzed with traditional methods and business intelligence techniques. Unstructured data consists of large sets of unstructured information obtained from the social Web (i.e., social media, digital photographs, online videos), sensors and other sources that pose a more complex challenge in analysis but might provide greater value. The major attributes of big data are:

“- Volume. The vast amounts of data generated every second. Potentially greater than terabytes (1012) and petabytes (1015) and could reach the zettabytes (1021) and brontobytes (1027) range.

“- Velocity. The speed at which new data is generated and at which data moves. Big data technology allows for analysis of the data while it is being generated without ever putting it into databases.

“- Variety. The different types of data. Although the structured relational databases that neatly fit into tables are small, some 80 percent of the world’s data is unstructured and cannot be put into tables easily.

“- Veracity. The messiness or trustworthiness of the data. With many forms of big data, quality and accuracy are less controllable (e.g., Twitter posts with hashtags, abbreviations, typos and colloquial speech, as well as the reliability and accuracy of content).

Utility Data Growth

AMI has increased the volume of utility data by the order of thousands (see Figure 1). Before smart meters, each customer meter was read monthly: 12 reads annually. Smart meters produce data at 15-minute intervals: 35,040 reads annually, or some 400 MB of raw data per year per meter. This data grows in magnitude after it is analyzed. And, with some utilities’ considering five-minute intervals, the data volume continues growing. When Austin Energy deployed 500,000 smart meters, its data storage requirements grew to 200 terabytes based on 15-minute reads. Those needs are expected to grow to 800 terabytes if interval reads are increased from 15 minutes to five minutes. With some 140 million smart meters deployed across the U.S., it is expected that close to 100 petabytes of information will be generated over 10 years.

A Utility Perspective of Big Data

Utilities envisioned applications from the massive amounts of data being collected through AMI, supervisory control and data acquisition and other sensors, including:

“- Protecting revenue from theft;

“- Targeting demand response (prioritizing customers for energy conservation and demand response programs using geospatial techniques, energy density mapping);

“- Distribution operations planning (targeting customers with high peak loads to help reduce peaks by staggering power for ventilation, heating, cooling and lighting);

“- Transformer load management (identifying transformers that are overloaded or underused);

“- Quality assurance and quality control data (improving the quality of connectivity information including phase);

“- Voltage correlation (using analytics to link meters to transformers including phase);

“- Energy modeling (analyzing usage patterns including unmetered usage from streetlights and other devices);

“- Voltage deviation (identifying transformers with voltage deviating from the rated voltage by 2 to 3 percent or more);

“- Geospatial outage frequency analysis (analyzing outage patterns geographically);

“- Predictive analytics for electric vehicle (EV) adoption (identifying plug-in EV owners and predicting demand patterns to ensure adequate transformer capacity); and

“- Asset management.

Big Data Applications in Asset Management

There is an increasing trend of developing predictive maintenance programs for the efficient use and management of utility assets. The management of those assets requires data from different sources and structured and unstructured data. Thus, big data could be deployed to address predictive maintenance in the asset management space. One specific application for transformers is illustrated in Figure 2, where data comes from different data-gathering systems in either a structured or unstructured fashion. These data are processed, modeled and formatted for further analysis. Algorithms in the analytics engine correlate with different information to make a decision and provide a trending analysis of the equipment’s health.

Figure 3 provides an example of a transformer health assessment based on winding temperature, differential temperature of online tap changer and main tank temperature, dissolved gas analysis and hourly loading of the transformer, among other things. Although Figure 3 shows a simple qualitative assessment of the equipment’s health by developing a watch list of transformers that need early attention, a more sophisticated quantities assessment can be made easily from the collected data.


Transformers are just one example of how big data and analytics can aid utilities in asset maintenance. A similar analytical approach can be used for circuit breakers or other utility assets to assess equipment health conditions and for developing subsequent mitigation plans.


Bijoy Chattopadhyay, Ph.D., is a director with Navigant’s energy practice. He has more than 30 years of experience and is based in San Francisco. Reach him at bijoy.chattopad

Sam Sankaran is an associate director with Navigant’s energy practice. He has 14 years of electric utility experience and is based in Burlington, Massachusetts. Reach him at

More Electric Light & Power Current Issue Articles
More Electric Light & Power Archives Issue Articles

Previous articleIntelligent Energy Storage- Key to Unlocking the Smart Grid
Next articleIntelligent Undergrounding-Burying Highly Reliable Cable When, Where it Makes Sense

No posts to display