A look at the use of evaluation in Scotland’s Energy Efficiency Programme pilots

By Ruth Bush, Research Fellow CXC, University of Edinburgh and Professor Jan Webb, Heat and the City, University of Edinburgh

Our progress on emissions reductions over the next 10 – 20 years is critical for climate change mitigation. This is reflected in the ambition of the latest Scottish Government Draft Climate Change Bill which sets out draft targets for the domestic and service sectors to reduce emissions by 75% and 98% respectively by 2032.

To meet these targets, delivery programmes will need to radically increase in scale to bring about installation of building fabric energy efficiency measures, behaviour change campaigns, and low carbon heat solutions that will enable us to switch away from fossil fuel based natural gas and oil heating systems.

In this blog, researchers from the University of Edinburgh Heat and the City Research Group share their thoughts on their ClimateXChange research project, evaluating the 25 local authority-led pilot projects funded by Scottish Government to inform the design of Scotland’s Energy Efficiency Programme (SEEP).

The research project aims to bring academic best practice into evaluation of new local pilot delivery programmes, and draw lessons to inform both the local authorities coordinating delivery, and Scottish Government as they develop the national programme.

This blog was originally published as part of the Energy & Society Research Group blogseries on the Scottish Government Energy Strategy consultations in May 2017.

Policy and regulations are going to play a critical role in meeting the ambitious goals set out in the Scottish Government’s draft Climate Change Plan for the next 15 years. But the scale of change needed means we are unlikely to establish an interconnected suite of new regulations, standards, funding, financing, incentives, alongside supportive local and national governance structures, and get it right first time. We need to establish a constructive dialogue between local governments, the public sector, businesses and community organisations, that enables new approaches to be trialled – and sometimes to fail – without derailing the whole programme.

Lessons from the demise of the Green Deal

The politics of being in government mean that experimental approaches to developing new policy measures are susceptible to harsh critiques by opponents, resulting in loss of ‘political capital’ for incumbents and a correspondingly risk averse approach by officials. After the demise of the Green Deal at the UK level, post-mortems have highlighted the challenging political context that meant that “failure was not politically conceivable” (Rosenow & Eyre, 2016). Despite early evidence pointing to issues with the programme design, the political context did not allow space for adjusting problematic areas and subjecting proposals to a ‘reality check’.

By 2015, when the programme was eventually abandoned, the average delivery rate for loft insulation had dropped by 90%, cavity wall insulation by 62%, and solid wall insulation by 57%, compared to 2012 when the programme was launched (Rosenow & Sagar, 2015). This experience highlighted the importance of creating space for constructive dialogue and reflection as policies are developed. Perceived failure of one programme should not mean the end of a whole policy agenda, but present UK Government action for energy efficient buildings is stalling.

Embedding pilots and evaluation within policy development

It is great to see that the Scottish Government are not repeating these mistakes during the development of their ‘cornerstone’ policy – Scotland’s Energy Efficiency Programme(SEEP). They have already embedded a process for learning into policy development through supporting local pilot projects and embedding evaluation as an integral part of the projects and on-going delivery.

As the academic research team supporting the evaluation of the local SEEP pilots, we have a responsibility to contribute to establishing a constructive dialogue around the national development of SEEP. Research evaluation of such programmes is potentially influential in determining what is understood to be ‘successful’ more generally. However, choosing the criteria for robust evaluation is challenging given the diverse geographies and socio-economic circumstances within which programmes take place, alongside the differing contexts of skills and supply chains.

Collaborative evaluation can support lessons for delivery as well as policy development

There are currently 25 local authority-led pilots taking place across Scotland. As an evaluation team, we work collaboratively with both the local authority pilot leads and Scottish Government officials to understand their objectives and concerns during pilot design and delivery, and to determine the important dimensions of the local project from the local and national perspectives. This approach has highlighted common challenges and best practice, as well as areas where there is a need to gain more experience, such as approaches to targeting the commercial sector, or mixed-ownership tenement buildings.

As the number of pilot projects grows, local authorities are taking on an increasingly proactive role in shaping the evaluation process. This ensures that the evaluation speaks to the local context and plays a development role to support future programme delivery, as well as informing national policy design.

The hardest delivery challenges require political space for experimentation, and even failure

This use of evaluation within policy development is one way to provide a framework for constructive dialogue around the hardest delivery challenges we face in SEEP. It creates a formalised and evidence based way for local experiences to inform national policy design – rather than a blame culture where failed projects are seen only as a result of poor management and a waste of squeezed public resources.

But this won’t be enough on its own to create the political space for tackling the hardest delivery challenges for SEEP. Programmes that take on these hard issues will need political back-up from local and national politicians and senior officials, they will need time and space to develop and refine approaches, and they will need flexible resource allocation that can be shifted around different tasks and over time. A strong working relationship based upon trust and respect between national and local governments is critical here. We hope that this culture can continue to build over time to empower the officials and politicians supporting it to tackle the hard challenges posed for SEEP.

The SEEP pilot evaluation interim report (published May 2017) is now available on the Scottish Government’s website

Read more about the SEEP pilot evaluation project