By Penni McLean-Conner, NSTAR
Increasingly, energy efficiency (EE) programs are being considered energy resources–just like generation–and consequently are built into integrated resource plans to meet forecasted load. The challenge is that most are unmetered resources. The impact the programs have in aggregate on meeting energy demand must be ascertained through evaluation, measurement and verification. There is great emphasis and reliance on evaluation to validate program performance.
Evaluation is the systematic field testing of assumptions used in planning. Evaluation achieves improvement and accountability. From an improvement perspective, evaluations assess program performance and implementation effectiveness. Lessons learned from the evaluations are incorporated into new programs and used to determine program objectives. From an accountability perspective, program administrators rely on evaluation to validate and document program results and ensure that programs are achieving their objectives and meeting resource-planning requirements.
Rigorous evaluation ensures that programs are cost-effective and that savings are sustained over time. Evaluation assesses whether a program achieved its goals, such as being a reliable energy resource. And program administrators rely on evaluation to demonstrate fiscal responsibly in the expenditure of public funds on EE programs.
The National Action Plan for Energy Efficiency has identified four types of evaluations: impact, process, market effects and cost-effectiveness. As utilities consider the development or expansion of EE programs, understanding and maximizing the utilization of evaluation, verification and measurement will be critical.
Impact evaluation quantifies the direct and indirect benefits of the program. The impact evaluation determines the program benefits and could include any or all of the following: gross energy savings, net energy savings, nonenergy benefits and demand savings. The gross savings are the savings at the meter, regardless of the program’s influence on the actions. Net savings are the percentage of gross energy savings attributable to the program only. Nonenergy benefits can include a variety of attributes and typically include water savings, economic impacts and reduction in utility arrears. Demand savings measure the impact of the program on the rate of consumption.
Process evaluations are used to assess the program performance. Process evaluations identify where costs can be remmoved from programs and where quality of service can be improved.
Process evaluations involve gathering information from all stakeholders in the delivery of a program, including program designers, implementers, participants, policymakers, trade allies and others. Improvements in efficiency and effectiveness will be identified where improvements in a program delivery can be made. For example, a process evaluation may identify how to reduce delivery time to participants, whether the program incentive is appropriate or if it’s too high or low.
Market Effects Evaluation
Market effects evaluation assesses how the overall supply chain and markets have been impacted by the program. For example, it is important to evaluate whether a program or end-use technology has achieved market transformation, at which point the technology is common and used by consumers without needing any additional incentives.
Program administrators must ensure the cost effectiveness of their programs. The cost-effectiveness evaluation quantifies the costs of the program and compares the costs with the benefits of the program. Several cost-effectiveness tests are used, often at the direction of regulatory bodies. Which test or tests may be used is driven by policy, which varies by state. More than one-third of states require multiple tests, according to “How Do We Measure Market Effects? Counting the Ways, and Why It Matters,” a 2004 oral presentation for the American Council for an Energy-Efficient Economy.
For program administrators, the cost-effectiveness tests are key components in assessing individual programs and building a program portfolio that will be approved by regulators. Program administrators will want to demonstrate fiscal responsibility in delivering programs that maximize the benefits and minimize the costs.
The type of cost-effectiveness evaluations or tests used often depends on public policy. The most commonly used cost-effectiveness test is the total resource cost test, which measures the net costs taking the perspective of the utility. This test also includes participant and nonparticipant costs. Other tests take different views, such as the participate test, which measures the quantifiable costs and benefits from the custome’s perspective, or the program administrator cost test, which measures net costs based on the administrative costs only. With programs rapidly increasing, the need to ensure that the programs are delivering as expected is even more important. As such, program administrators, regulators and other stakeholders are interested and involved in ensuring the evaluation process is robust and withstands external scrutiny.
Penni McLean-Conner is the vice president of customer care at NSTAR, the largest investorowned electric and gas utility in Massachusetts. McLean-Conner, a registered professional engineer, serves on several industry boards of directors, including the Massachusetts Technology Collaborative and CS Week. Her latest book, “Energy EfÃ´â‚¬â‚¬â‚¬ ciency: Principles and Practices,” is available at http://pennwell books.com.