Ruth Bush and Niall Kerr, ClimateXChange Research Fellows looking at retrofit policy in Scotland, set out lessons on how evaluation can play a role in helping Scotland to meet its climate targets.

The Energy Efficient Scotland Route Map was launched in May 2018, setting out the Scottish Government’s plans for large-scale energy efficient retrofit of Scotland’s building stock alongside heat decarbonisation. Chapter 7 of the route map details plans to establish a monitoring and evaluation framework that will demonstrate the programme’s progress and “will allow [Scottish Government] to adapt and flex the programme where necessary”.

As ClimateXChange Research Fellows we use different types of evaluation to inform energy efficiency policy in Scotland. Experiences in our projects offer lessons about how to ensure evaluation can play a meaningful role in helping Scotland to meet its climate targets in the future.

At the European Environmental Evaluators Forum (EEEN) conference at the end of 2017 we came together with environmental agencies, consultancies and academic colleagues from across Europe to hear how other countries are using evaluation in policy development and to discuss Scotland’s approach.

From left to right: University of Edinburgh researchers Ruth Bush and Niall Kerr, Jack Causley from Scottish Government and Mark Winskel a Chancellor’s Fellow at UoE and the session chair at the EEEN. 

Evaluation approaches

Niall Kerr’s research:

• Evidence use in energy policy-making: decision making under social, technological and political uncertainty
• Systematic evidence reviews: systematically and transparently gathering potentially contradictory evidence
• Multiple criteria for policy effectiveness: efficient, equity and institutional fit
• Dedicated method of energy policy effectiveness in Scotland
• Pilot evidence review findings: encouraging households to invest in home energy efficient retrofit

Ruth Bush’s research:

• Learning lessons from local pilot projects, led by local authorities delivering energy efficiency retrofit in domestic and non-domestic buildings
• Using social surveys: To understand investment drivers and barriers in different contexts
• In-depth interviews: to understand programme design and management approaches
• Building capacity amongst local authorities – Sharing lessons from local pilots with other local authorities and Scottish Government

The importance of embedding a culture of evaluation into policy and delivery.

As retrofit programmes start to focus on some of the more challenging areas of energy retrofit like private non-domestic buildings, or moving away from gas central heating, we’re not going to get things right first time.

Embedding this type of learning culture takes time. Starting this process of embedding culture of evaluation at national and local levels will help us to innovate and learn from when things go wrong.

In the local SEEP pilots, many of the pilot leads had little or no experience of evaluation. They were in charge of coordinating technical monitoring and survey data collection within their projects, and we did training and workshops to develop a way of working that would deliver good quality and consistent data that would allow us to compare across the projects as well as draw local lessons.

The pilot leads now have a much more developed understanding of evaluation; not only about how to make the practicalities of data collection work, but also about what they can learn to help their own work.

Whilst the process of national policy decision-making may be unpredictable, many of the findings from a systematic evidence review are likely to have relevance over medium to long-term time periods. Reviewing multiple sources of evidence on a particular topic in energy policy entails summarising the most relevant details often from a mix of qualitative and quantitative sources. Inevitably some detail from the reviewed sources will be excluded. Whilst seeking to offer insight by identifying trends in the assembled evidence our reviews will also act as something of a reference document for those interested in the details of the policy question.

For readers that have a particular interest in a sub-section of the review the gathered evidence will allow an initial introduction to a potentially much more comprehensive evidence base. Building a consolidated knowledge base on the particularly complex issues of energy system transition is critical so that lessons are learned from historical and international policies.

The importance of the timing of evaluation?

Policy processes don’t wait for pilot projects to finish and outcomes of an evaluation to be published. It was important for our evaluation team to create opportunities to feedback what we were learning to Scottish Government and local authorities on the local pilots on an on-going basis. We did that formally through interim reports, but also held workshops with everyone involved and facilitated discussions on key emerging themes from our data. National policy makers could get a more in-depth insight into how delivery was working at the local level, and they also provided an opportunity for local authorities and their partners to exchange experiences and learn from one another.

An on-going relationship is important to making sure the evaluation is contributing to delivery and policy processes – no matter the speed of policy development.

 Systematic evidence reviews do involve a trade-off between the rigour of evidence examination and the length of time taken; more time means more evidence gathering and more consideration of potentially complex and contradictory findings. Policy decisions are, however, often made in an unpredictable and dynamic environment. This mismatch has led to the use of Rapid Evidence Assessments (REA) for energy policy. REAs are designed to apply a comprehensive and un-biased approach to evidence gathering, and are intended to take place over a relatively brief, roughly 6-month period. Certain policy questions may be suitable to longer reviews of 12 months or more, while shorter scoping reviews of around 1 month can be applied to more urgent issues. Our project intends to trial different lengths of evidence review throughout the energy policy effectiveness project.

Presentations and more information about the EEEN Forum 2017 can be found on the European Environment Agency Website: https://www.eea.europa.eu/themes/policy/events/eeen2017/6th-european-environmental-evaluators-network-forum