Most state and utility energy efficiency programs undergo a “process evaluation” to assess how well the programs achieve their objectives, and how the programs could do better. Process evaluations can provide essential feedback from the perspective of program administrators, end-users (e.g., households and building owners) and suppliers (e.g., mechanical and lighting contractors). Feedback is great for improvement, right? Well, yes, if you can act on it, and if acting on it will have impact.
Findings you can act on
As a researcher, I have learned to begin with the end in mind. That is to say, before digging into the details of methodology and data, a researcher should speculate on the possible findings of the research and assess whether the results would be relevant to the research stakeholders. An essential question for process evaluation findings, methodology, and data collection is “What conclusions are actionable by program designers?” The answer to this question begins with identifying the levers or points of influence of program designers, and then assessing how and how much the levers could be adjusted to maximize program impact.
Findings with impact
As an engineer with experience in building design, commissioning, energy modeling, and energy measurement and verification (M&V), I have worked with and witnessed the many factors that can affect energy efficiency. These factors include efficiency program support, project budgets, owner requirements, equipment substitutions during construction (e.g., “value engineering”), installation quality, automated control setpoints, and end-use load (both peak demand and duration). Some of these factors can be influenced by efficiency programs and the rest pose uncertainties that could increase or decrease program impacts. Process evaluations should be designed to quantify the potential impact (energy and demand savings) of program changes/recommendations, and the uncertainty of factors that are outside of the program scope. Doing so can help to prioritize recommendations that are directly linked to the mission of energy and demand savings, and can help to find opportunities to minimize the uncertainty of program impact by expanding program scope, such as including a commissioning component to verify and ensure that control systems are optimized for expected energy savings.
The ability to quantify the impact of process evaluation recommendations will depend upon the available budget and data for the evaluation. At a minimum, a process evaluation should help program designers identify the barriers in the marketplace for energy efficiency, assess how well the existing program design and delivery overcomes them, and identify additional opportunities for the program to help mitigate or eliminate the barriers.
Energy efficiency process evaluations
In summary, the focus of energy efficiency process evaluations should be on findings/recommendations that are actionable and that have positive impact. This simple statement is important to keep in mind in process evaluations that span a broad range of program activities and stakeholders. Feedback is good, if you can act on it, and if acting on it will have impact!
