Measuring the impact of climate change engagement in Scotland

Research completed: April 2026

Executive summary

Aims

This study was commissioned to help the Scottish Government strengthen how it assesses, evidences and communicates the impact of its Public Engagement Strategy on Climate Change (PES) (Scottish Government, 2021). The PES is an overarching framework that brings together a wide range of programmes (such as the Climate Action Hubs, Climate Engagement Fund, the Climate Policy Engagement Network, and others) to support public understanding, participation and action on climate change. In this report, we set out to address three related aims:

• to identify best-practice approaches for monitoring and evaluating complex public engagement strategies
  
• to understand how PES monitoring and evaluation currently operates in practice
  
• to provide practical, proportionate recommendations that can strengthen strategy-level M&E while remaining feasible and proportionate.

Findings

The research identified the following key findings:

• Scotland’s PES is at the forefront: Many countries run successful deliberative interventions and national awareness campaigns on climate change. However, few have attempted to deliver multi-stranded, locally embedded approaches that are tied into national Just Transition policy, like Scotland’s PES. Scotland remains at the forefront of practice in actively commissioning and operationalising a strategic-level evaluation capable of generating cumulative insights across multiple years and programmes of public engagement. See 6.3 International examples of public engagement strategies on climate change.
  
• Evaluating the PES is challenging: The PES encompasses a diverse set of interventions which operate in a complex system, designed to engage the full breadth of the Scottish public on climate change, across all sectors and geographies. The complexity poses challenges for evaluation, as the variables make it difficult to isolate the impact of the intervention. This limits the conclusions that can be drawn and the claims made about direct causality. However, there are a range of evaluation approaches, methods, and tools which are suitable and recommended for evaluations in these complex contexts. The Mid-Point Review of the PES used methods broadly in line with these approaches; however, this report provides guidance on how to implement these in a more holistic way, appropriate for full-term evaluation. See 7 First principles of evaluation in complex systems.
  
• The PES could be evaluated using a hybrid of several methods and approaches: Evaluating the PES at strategy level could benefit from embracing a hybrid design, combining a few key types of evaluation approaches and methods to suit the different scales and methods of delivery within the PES. Appropriate approaches include participatory, adaptive, and theory-based approaches such as systems mapping and contribution analysis. See 8 Key factors in evaluation design.
  
• Structured data collection can support evaluation: PES programmes are already collecting the right types of data. However, there is an opportunity to structure data collection and presentation to support a strategy-level evaluation framework. Data can be aggregated to provide evidence of impact, as well as to facilitate learning, listening, relationship-building and storytelling. See 9 Recommended approach: Applying contribution analysis to the PES at strategy level.
  
• Expectations on attributing impact must be managed: The recommended evaluation approaches have limitations in their ability to strictly attribute impacts to PES activities. As a result, there may be a gap between what stakeholders would like to know about how the PES is delivering, and what it is possible to conclude from evaluation. Future evaluators must manage expectations and communicate clearly about the thinking behind chosen approaches. Nonetheless, these approaches will enable Scottish Government to build an evidenced narrative of how the PES has contributed to its objectives. They can also capture important aspects of complex systems, which will allow Scottish Government to identify gaps and adapt the strategy accordingly. See 9.2.3 Telling the story of the PES to external stakeholders.

Recommendations for future PES monitoring and evaluation

The following recommendations will be helpful in integrating findings of this report into an updated and strengthened monitoring and evaluation (M&E) approach for the PES:

• Expand the PES Theory of Change (ToC): The PES has a high-level ToC in place. However, a more comprehensive, fully elaborated version could serve as a useful foundation for robust evaluation. A full discussion of how to approach this, including a template, has been provided in the report. See Appendix C – Using the Theory of Change as an evaluation tool.
  
• Work with stakeholders to create a common understanding of how interventions are delivering impact on the ground through expanding the current ToC. Adopting participatory methods, the Scottish Government could work with PES delivery partners to more clearly articulate how different climate engagement activities work together to deliver intended outcomes. Involving delivery partners in refreshing PES M&E could also build shared understanding, trust, and legitimacy. Scottish Government teams could help delivery partners develop ToCs for their own programmes, where these are not already in place. These would be proportionate to the scale of each programme and could be “nested” within the strategy-level ToC, articulating how programmes support strategy level and vice versa.
  
• Use contribution analysis to evaluate the PES: There are a variety of evaluation approaches and methods that could be applied to the PES, working within a Theory of Change framework. Contribution analysis provides a method to test and refine the causal pathways within the ToC for the PES or its component programmes. It does this by testing and refining the ToC, examining the strength of evidence for each causal link, integrating quantitative and qualitative data, and assessing whether rival explanations offer a better account of the change. Although it cannot offer definitive proof of attribution, it can provide reasoned and credible evidence of contribution. See 8.3 Feasible designs: Selecting appropriate evaluation approaches for the PES at strategy level.
  
• Set SMART targets: Set a small number of specific, measurable, achievable, relevant, and time-bound (SMART) targets within the PES, including the reasoning behind the level of change targeted. These will flow from the refreshed ToC and clarify and inform evaluation questions. See 9.1 Proportionality: How can contribution analysis address the challenges of PES M&E in a feasible way?
  
• Continue to encourage and enable innovation at programme level: Continue to encourage local delivery partners to innovate in their own engagement and evaluation methods. Programmes tailored to local contexts (such as Climate Action Hubs, Climate Action Schools and Climate Engagement Fund projects) are key drivers of innovation within the PES. Capturing qualitative data on what works in these contexts represents key learning as well as powerful storytelling material, an important aspect of evaluation. See 9.2 Communicating and embedding evaluation results with PES trusted messengers and other stakeholders.

The recommendations and approaches listed above will help evaluate the different layers, scales and styles of engagement delivered across the PES. The unified framework should help clarify the PES’s programme theory, articulating how PES programmes work together in unison to support the overarching strategic objectives, creating the basis for evaluation. These recommendations are relevant for:

• the Scottish Government Climate Change Engagement team as owners of the PES
  
• delivery partners and programme leads responsible for monitoring and reporting
  
• the wider network of trusted messengers using the PES to inform their activities
  
• policy teams using public engagement insights to shape decisions
  
• analysts and evaluators preparing for the first full PES evaluation.

Together, these steps should create a clearer, more consistent and more credible account of how public engagement supports Scotland’s transition to a net zero and climate-resilient future and ensure the PES continues to learn, adapt and demonstrate value.

Glossary

Intervention	Any policy, programme or other government activity meant to elicit a change
The Green Book	Guidance from the UK Government (HM Treasury, 2022) on the appraisal of policies, projects and programmes
The Magenta Book	Guidance from the UK Government (HM Treasury, 2020) on what to consider when designing an evaluation
Trusted messengers	Formal and informal partners, including individuals and groups from a range of organisations, who work with Scottish Government to communicate with the public and those in their sphere of influence on climate change

Abbreviations table

CEF	Climate Engagement Fund
COM-B	COM-B Model: Capability, Opportunity, Motivation, & Behaviour
GCS	UK Government Communication Service
ISM	ISM (Individual Social Material) model
KPI	Key performance indicator
PES	Scottish Government (2021) Net Zero Nation: Public Engagement Strategy
MPR	Scottish Government (2025) Climate Change – Public Engagement Strategy: Mid-Point Review
M&E	Monitoring & evaluation
OECD	Organisation for Economic Co-Operation and Development
SNAP3	The third Scottish National Adaptation Plan (SNAP3)
ToC	Theory of Change
UN	United Nations

Introduction

Aims of this work

The Scottish Government has identified a need to strengthen the evidence base supporting its ability to assess and communicate the impact of its climate-related public engagement activities. A range of programmes, campaigns, and partnerships contribute to delivering the Net Zero Nation: Public Engagement Strategy for Climate Change (PES) and there remains a gap in articulating how these collective efforts lead to strategic outcomes.

This project’s research questions are:

• What recommendations can be made to strengthen the approach to monitoring and evaluating the outcomes and impacts of the PES for its final review and future iteration?
  
• What can we learn about how the composite parts of the PES are currently monitored and evaluated in practice, and the opportunities to improve how progress towards outcomes and impacts is assessed?
  
• What practical tools for monitoring and evaluation could be used by both Scottish Government officials and external partners to better measure impact?

This project therefore:

• Reviewed existing literature and guidance on monitoring and evaluating public engagement and communications strategies
  
• Engaged with practitioners to assess how PES is monitored in practice
  
• Assessed how existing monitoring and reporting for PES activities align with best practice
  
• Identified gaps and opportunities for strengthening the PES Theory of Change (ToC)
  
• Provides practical recommendations, tools and templates to construct a proportionate, coherent monitoring and evaluation (M&E) system
  
• Strengthens the Scottish Government’s ability to communicate the cumulative impact of its engagement activity to internal and external audiences.

Discussions with Scottish Government colleagues have highlighted that, while the PES provides a strong overall framework, the links between understanding, participation, and acting on climate change are not yet fully articulated. In particular, there is a need to clarify the assumptions and dependencies that underpin how engagement contributes to wider system change. This work therefore provides both an analytical foundation and practical recommendations to help the Scottish Government refine its M&E approach, ensuring that the Strategy’s impact can be demonstrated more effectively across its multiple levels of delivery.

Overview of PES history, programmes & partners

The PES builds on a long-standing legislative and policy framework for climate action in Scotland. Since 2009, the Climate Change (Scotland) Act has required the Scottish Government to take action to promote public engagement and participation in climate change policy (Legislation.gov.uk, 2009). Early approaches emphasised individual responsibility for reducing emissions, encouraging personal lifestyle choices such as recycling or energy efficiency. Over time, the emphasis of engagement has evolved. The current strategy reflects a shift from individual to systemic change, focusing on the social and collective dimensions of climate action.

The development of the PES delivery encompassed several important shifts in how climate engagement activities have been coordinated and delivered in Scotland:

• The Let’s Do Net Zero (LDNZ) marketing campaign represented one of the first large-scale communications efforts under the new Strategy. Its initial focus was on building public understanding of climate change and Scotland’s national ambitions.
  
• In parallel, direct delivery grants supported organisations with existing relationships within their audiences, enabling the Scottish Government to complement other programmes with additional grants where strategically relevant, for example by continuing support to the Climate Action Schools programme.
  
• The Climate Engagement Fund was introduced to further diversify engagement, expanding support to new partners and audiences, and encouraging innovation in community-led communication and participation.
  
• The Climate Change Participation Programme was established to facilitate structured dialogue around Just Transition policy development, linking public engagement more directly to policymaking and implementation.

Rather than encouraging isolated behavioural changes, the PES aims to create the conditions for whole-of-society participation in Scotland’s transition to net zero. The framework is structured around three strategic objectives (Understand, Participate, and Act) which together provide a coherent foundation for influencing awareness, inclusion, and behaviour across diverse sectors and communities. Under the current delivery model, the PES has added additional programmes under the three strategic objectives (see Figure 1). For a full overview of the programmes and how they contribute to the overall vision of the PES, please refer to:

• Net Zero Nation: Public Engagement Strategy (Scottish Government, 2021)
  
• Climate Change – Public Engagement Strategy: Mid-Point Review (Scottish Government, 2025)

Together, these programmes represent a gradual evolution from top-down communications to a more distributed and partnership-based approach, built around trusted messengers who can translate the national strategy into local action.

The PES sits within a comprehensive policy landscape aimed at achieving net zero greenhouse gas emissions in Scotland by 2045, as mandated by the Climate Change (Scotland) Act 2009 (Legislation.gov.uk, 2009). It occupies a pivotal position within this landscape, functioning as the bridge between strategic ambition and public engagement, ensuring that the transition to net zero is grounded in public understanding, legitimacy, and shared ownership. While the Update to the Climate Change Plan (2018-2032) (Scottish Government, 2020) and the Just Transition Strategy (Scottish Government, 2021) set the technical and economic pathways to net zero, and the third Scottish National Adaptation Plan (SNAP3) (Scottish Government, 2024) focuses on achieving climate resilience and adaptation, the PES provides the social infrastructure through which these policies gain public understanding, legitimacy, and participation. Its emphasis on informing, enabling, and empowering citizens ensures that climate action is not only delivered to people but with them.

By linking with frameworks such as National Planning Framework 4 (Scottish Government, 2023), the Environment Strategy (Scottish Government, 2025), the Biodiversity Strategy (Scottish Government, 2024), and the Heat in Buildings Strategy (Scottish Government, 2021), the PES embeds people-centred approaches across diverse policy domains. In doing so, it provides the framework for strengthening coherence between climate, social, and economic policy, supporting the wellbeing economy, circular economy, and National Strategy for Economic Transformation (Scottish Government, 2022). Its effectiveness depends on how well engagement is integrated into decision-making and accountability processes across government, highlighting both the strategic importance of the PES and the ongoing challenge of ensuring that participation translates into influence and impact.

Working in partnership is at the core of the approach of the PES. The PES programmes listed in Figure 1 are primarily delivered by local organisations, embedded in regions across Scotland. Crucially, the primary audience for the PES itself are these trusted messengers:

“We know that public support is crucial in order to successfully achieve the transformational societal change required to reach net zero. We also know that the government cannot be the only communicator on climate change issues. This strategy [the PES] is therefore aimed at climate change communicators – or ‘trusted messengers’ – who can more effectively engage people in their sphere of influence.” (Scottish Government, 2021, p. 4)

The PES clarifies the Scottish Government’s direction of travel, sets the tone and language around how they talk about climate change, clarifies roles and responsibilities (i.e., what the Scottish Government is doing; what trusted messengers are doing; how members of the public can contribute). This enables trusted messengers and delivery partners to be aligned and consistent in their language and actions.

This report uses “delivery partners” to refer to those groups directly funded by the Scottish Government to deliver PES programmes. These include a range of public, private, and third sector organisations, for example Keep Scotland Beautiful (Climate Action Schools), third sector organisations delivering the Community Climate Action Hubs), the recipients of the Climate Engagement Fund (wide variety), and others.

However, other potential “trusted messengers” and the full audience for the PES extends far beyond those who are funded to deliver PES programmes. For example, Zero Waste Scotland, NHS Scotland, NatureScot, and Culture for Climate Scotland all engage the public on climate change and related issues. These organisations each have their own unique strategic objectives, priorities, and audiences; however, the PES and its outputs provide a central reference regarding government messaging and strategy for how to engage the public on climate change. The PES is designed to be useful to any and every organisation who wants to facilitate these conversations with their membership or sphere of influence. Overall, the PES provides a unifying framework for Scotland’s climate engagement activity, utilising a mix of methods to reach and connect with the public, enabling a whole of society approach to the transition to a net zero and resilient Scotland.

Previous and future evaluations

The evaluation of the PES is designed as a continuous, multi-stranded process. Since its inception, the PES has been reviewed once, via the Mid-Point Review (MPR), which assessed progress against the strategy’s three objectives (Understand, Participate, Act) over its first three years (2021–2024). The MPR drew evidence from existing programme evaluations (e.g., Climate Week, Climate Engagement Fund, Hubs) and national indicators derived from the Scottish Household Survey (Scottish Government, 2025). The MPR was supported by an independent evidence review, “A review of effective public engagement on climate and implications for Scotland” (Millar, et al., 2024). This synthesised international best practice on effective engagement methods to compare what had been delivered via the PES. The Millar et al. report found that the PES’s methods, content and delivery were indeed following best practices for effective public engagement on climate change.^[1] However, a key collective finding from previous evaluations was the need to strengthen how PES activities are monitored and evaluated. The practical challenges faced in monitoring and evaluating the PES, as referenced in the MPR, are:

• “More consistency in data gathering and reporting, aligned to the PES Theory of Change, would help strengthen the conclusions that can be drawn. There is opportunity to take action on this for the remainder of the PES delivery period, and particularly for designing the end of PES review.” (Scottish Government, 2025, p. 3)
  
• “Activities operate at a range of scales, with different measures of success which are difficult to aggregate or compare across.” (Scottish Government, 2025, p. 10)
  
• “A clear challenge to this approach is evidencing impact on policy outcomes. Millar et al. (2025) underline the importance of policy makers meaningfully engaging with the outputs and recommendations from participatory activities and explaining to participants how these will be acted upon. It is notable that less than half (45%) of the 67 respondents to the stakeholder survey felt they had had meaningful opportunity to participate in policy development, and many commented negatively on their experience of participation. This included feelings of over-consultation and a lack of clarity on how their participation in the design-stage led to meaningful action by Scottish Government.” (Scottish Government, 2025, p. 27)

“The lack of ability to track the impact of the outcomes [of Scotland’s Climate Assembly] on Scottish Government climate change policy is a significant weakness in terms of progress towards the Participate objective of the PES.” (Scottish Government, 2025, p. 22)

The next iteration of the PES is not expected to represent a fundamental overhaul, but rather an evolution of the existing framework. The focus will be on strengthening the layers that sit beneath the high-level objectives, with clearer articulation of how risks, assumptions, and dependencies are identified and tested. Future evaluation efforts will place greater emphasis on coherence across programmes, ensuring that findings can be meaningfully aggregated to assess the cumulative impact of engagement activity. There will also be a stronger focus on indirect impacts and learning from partners and networks. This shift reflects a wider ambition to move beyond activity-level monitoring toward a more strategic understanding of how public engagement contributes to long-term behavioural and societal change.

Research process

This research adopted a blended approach that moved between reviewing best practice and technical guidance, exploring international examples, engaging with delivery stakeholders, and testing recommendations in a workshop. This allowed emerging insights to be tested against both established evaluation principles and the practical realities of delivering a complex, multi-stranded public engagement strategy.

While many countries now undertake climate assemblies, public dialogues or funded engagement programmes, the review found few initiatives comparable in scale, structure or ambition to Scotland’s PES. Most international examples focus on discrete deliberative processes or short-term programmes, often without a unifying strategy, shared Theory of Change or coherent evaluation framework. As a result, there is no robust or settled body of “best practice” for evaluating long-term, portfolio-based public engagement strategies on climate change. See 6.3 International examples of public engagement strategies on climate change.

This finding strongly shaped the remainder of the report: with no available standard against which to benchmark Scotland’s PES, the analysis returns to first principles of evaluating complex systems, drawing on theory-based approaches and established government guidance to develop a tailored, credible and proportionate evaluation framework for the PES.

Literature review & international comparators: Key takeaways for the findings of this report

We reviewed the relevant UK, European and international literature and guidance on evaluating government programmes and strategies. There are limited examples of international governments who have implemented strategies or initiatives similar in scale and structure to Scotland’s PES, which is world-leading in this area. As a result, this research returned to first principles of designing evaluations for policy interventions operating in complex contexts.

Methodology

Literature review

This research began with a targeted literature review of UK and international evaluation guidance, public engagement strategies and relevant research on climate engagement with the aim of determining what is feasible and proportionate for the PES. The targeted and pragmatic review identified best-practice approaches for monitoring and evaluating complex, multi-stranded public engagement strategies by focusing on 18 authoritative UK and international sources (see 6.2 Results of literature review). Findings from each source were extracted into a structured matrix and synthesised across nine sub-questions under three themes: the purpose of strategy-level evaluation (“why”), defining outcomes and impacts (“what”), and identifying appropriate M&E methods (“how”).

Alongside this work, the research examined international examples of government-led public engagement on climate change to assess whether established models or evaluation approaches could be directly transferred to the Scottish context.

Document analysis

The document analysis systematically reviewed the existing PES M&E landscape to assess current data collection practices. Primary materials included programme-level M&E templates, reporting forms, project reports, and the Net Zero Nation PES Mid-Point Review (MPR) (Scottish Government, 2025). Indicators were mapped to established Theory of Change stages from the Magenta Book and Government Communication Service (GCS) Evaluation Cycle (Inputs, Outputs, Outtakes, Outcomes, and Impacts) to capture the implicit ‘programme theory’ of the PES (HM Treasury, 2020) (Government Communication Service, 2024). This comparison mapped current data flows, gaps, and the feasibility of collecting specific metrics. These findings directly inform the refreshed evaluation framework and guidance in Appendix E – Data and indicators for PES M&E.

Interviews with key stakeholders

Semi-structured interviews added practical depth and insight from those responsible for delivering climate engagement. Fifteen participants from various PES projects and programmes were selected to share programme, project and strategy level delivery experience. Interview questionnaires (see Appendix B – Interview guide example) were co-designed with the steering group, and all interviews followed agreed ethics procedures, including informed consent. Sessions were recorded, transcribed and analysed to extract key findings and lessons learned, which were integrated into the report to support interpretation.

Workshop with Scottish Government teams

A final stakeholder workshop with Scottish Government teams validated and refined the research findings. The session focused on six M&E improvement ideas derived from earlier interviews. Two activities structured the discussion: (i) defining the metrics required to capture delivery partners’ impacts, including behavioural and outcome-level change; and (ii) an effort-versus-impact prioritisation exercise to assess feasibility and likely effectiveness. Insights from the workshop shaped both practical recommendations and the strategic context presented in the final report.

For more detail on the project methods, see Appendix A – Full project methodology.

Results of literature review

Collectively, the sources provide a comprehensive foundation for assessing monitoring and evaluation at the strategy level.

Sources were analysed using a data-extraction matrix comprising core fields applicable to all documents (including source type, jurisdiction, relevance to project, transferability to the Scottish PES context, and relevance to the research questions), alongside thematic fields tailored to different source types. Thematic fields captured information on theory of change, evaluation methods, indicators, strengths and gaps, public involvement, transferable concepts, and implications for the development of robust, context-sensitive monitoring and evaluation frameworks.

From this review of UK and international guidance and case studies in M&E and public engagement, several conclusions were drawn which affected the direction of this research and subsequent steps:

• International guidance consistently highlights that high-quality monitoring and evaluation of complex, multi-stranded programmes depends on a clear Theory of Change that articulates causal pathways from activities to outcomes and supports the development of SMART, strategically aligned indicators. For such programmes, this implies an outcome-focused, theory-based approach to evaluation that works backwards from intended outcomes to assess progress, understand how and why change occurs, and examine the contribution of multiple interventions and partnerships within dynamic systems (see next section, 6.3 International examples of public engagement strategies on climate change).
  
• UK central government guidance provides clear steers on: determining which government interventions are evaluable; when to evaluate; designing and setting the scope of evaluations. Complex, multi-stranded interventions operating in systems affected by many variables (such as Scotland’s PES) require agile, flexible approaches to evaluation. This guidance is interpreted for the context of the PES in 7 First principles of evaluation in complex systems.

As detailed above, there are few examples of international governments who have implemented strategies or initiatives similar in scale and structure to Scotland’s PES. It follows that there are few robust evaluations or precedent for evaluating interventions like PES. Therefore, the next step in the process of this research was to consider:

• What is the best practice for evaluating complex, multi-stranded government interventions more generally (i.e. not specific to public engagement and/or climate change)?
  
• What are useful examples of where this has been done?

These principles are interwoven and provide the foundation for the following chapters of this report.

International examples of public engagement strategies on climate change

In addition to searching for relevant academic, government and grey literature, the research team also reviewed international examples of government-led public engagement on climate change. The United Nations (UN) and the Organisation for Economic Co-Operation and Development (OECD) have called for national governments to engage their citizens on climate change, highlighting this as essential to achieving the pace and scale of action required and for a coordinated global response. Many nations in Europe and some further afield have engaged citizens in deliberative processes such as citizens assemblies on climate change. However, it is a finding of this research that there are few international examples of government programmes or strategies similar to Scotland’s PES, which is a long-term, cross-government, partnership-based, funded and coordinated approach to public engagement on climate change. This section discusses the international context for Scotland’s PES in more detail.

Governments around the world increasingly recognise that achieving large-scale behaviour change on climate issues requires more than one-off participation exercises: it depends on long-term public engagement that builds understanding, shifts social norms and supports people to act within their own contexts. Decades of public-health experience (another policy area with a focus on society-wide behaviour changes, such as smoking cessation, diet, road safety and physical activity) show that meaningful change emerges gradually through a combination of communication, community support and structural conditions. Climate engagement is even more complex, because people’s capacity to understand climate change, participate in solutions and take action is shaped by local values, identities, opportunities and constraints; it cannot be driven solely by top-down messaging (Climate Outreach, 2020, p. 11). Both the UN and the OECD, among others, have explicitly called for governments to engage citizens on climate change, including through participation in policy development, public awareness and inclusive processes:

Action for Climate Empowerment (ACE) is a term used to refer to work under Article 6 of the UN Framework Convention on Climate Change (“reduce the impact of climate change by enabling society to be a part of the solution”) and Article 12 of the Paris Agreement (“Parties shall cooperate in taking measures, as appropriate, to enhance climate change education, training, public awareness, public participation, and public access to information”) (UNFCCC, 2025). ACE workstreams mirror the strategic objectives of Scotland’s PES, Understand, Participate and Act. The UNFCCC states that “Public participation is about long-term collaboration among different groups, as much as it is about the quality of how and the extent to which citizens partner with local and national governments to craft policies” (UNFCCC, 2025). Submissions under the UNFCCC negotiations call on Parties to “develop and implement climate policy based on inclusive stakeholder engagement and public participation processes,” explicitly linking climate policy quality with ongoing inclusive engagement (Grantham Research Institute on Climate Change and the Environment, 2024).

The OECD’s 2025 Engaging and Empowering Citizens for the Net-Zero Transition paper asserts that

“An effective government-citizen interface for climate action 1) provides access to transparent information and open data; 2) builds trust through clear and responsive public communication; 3) empowers and mobilises citizens at scale through people-centred policy-making based on meaningful citizen participation; and 4) empowers and sustains change through informed citizens driving environmentally sustainable behavioural shifts”, making citizen participation central to effective net-zero transitions (OECD, 2025, p. 6).

For this reason, several countries now fund programmes that build climate literacy, participation and agency through diverse, locally grounded projects rather than prescribing a uniform behavioural pathway. Approaches include deliberative processes, public consultations, awareness campaigns, participatory platforms, and other means to involve people in understanding, shaping or supporting climate policy. Place-based examples include:

Deliberative and participatory policy processes

• The Republic of Ireland convened a Citizens’ Assembly on climate change (2017) to gather recommendations on enhancing climate action, involving randomly selected citizens deliberating on climate policy measures (Knowledge Network On Climate Assemblies, 2025).
  
• Denmark established a Citizens’ Assembly on Climate Issues to involve a representative cohort of citizens in discussing climate policy and making recommendations to government (Danish Board of Technology, 2025).
  
• France organised a Citizens’ Convention on Climate where ordinary citizens were tasked with proposing measures to achieve 2030 climate targets. Austria has prepared a Citizens’ Climate Assembly to propose climate neutrality measures to government (IEA, 2021).
  
• Germany, Spain, Sweden, Luxembourg, and Finland have experimented with national or regional climate deliberative processes (cities/juries/assemblies) (Lorenzoni, et al., 2025).
  
• In the UK:
  • There was a UK Climate Assembly which ran in 2020. Over several months, Assembly members heard from experts, questioned stakeholders and discussed trade-offs before agreeing a set of recommendations covering areas such as energy, transport, food, land use and fairness. The Assembly demonstrated strong public support for ambitious climate action, provided it is fair, well-communicated and involves public participation in decision-making (Climate Assembly UK, 2020).
    
  • Scotland’s PES is mandated by the Climate Change (Scotland) Act 2009 and was first published in 2021 (Legislation.gov.uk, 2009). Scotland’s Climate Assembly ran between 2020-21 (Andrews, et al., 2022).
    
  • Wales has a published Public Engagement Strategy of its own. It is the most similar to Scotland’s and is discussed in this section as well as Appendix D – Case studies. The Northern Ireland Executive is in the process of developing a similar document.
    
  • DESNZ recently published a policy paper “Energising Britain: Your voice in our Clean Energy Superpower Mission” (Department for Energy Security and Net Zero, 2025). This is the latest statement on the UK Government’s approach to a transition to net zero and includes the intention to explore participatory policymaking and engage widely. The paper makes clear that DESNZ plan to align their approach with that of Scotland, Wales, and Northern Ireland, highlighting the leadership of these devolved nations in this space:
    • “We have sought to align our approach so that it is complementary to existing activities that are taking place across the UK. We will continue to work in partnership with the Scottish Government, Welsh Government and the Northern Ireland Executive to support work across the four nations and share both learning and resources.

Other instances of Open Government and multi-stakeholder participation

• Canada ran Generation Energy (2017-18), a large national dialogue which engaged over 380,000 Canadians in conversations about the nation’s energy future, which fed into planning. They also ran the Climate Action and Awareness Fund (2020-22), directly funding community-based climate action, discussed below (Government of Canada, 2025).
  
• Brazil’s Federal Government operates digital platforms such as Brasil Participativo to engage citizens on policy issues, including climate planning and climate action (Government of Brazil, 2025).
  
• Costa Rica created a Citizens Consultative Council on Climate Change under its Open Government Partnership plan to support climate data disclosure and public participation in policy design, including in deciding their Nationally Determined Contribution in line with the Paris Agreement (Energia, n.d.)
  
• Kenya and Sierra Leone have included multi-stakeholder processes for operationalising climate legislation and climate commitments, emphasising transparent climate information and participatory approaches (Robinson, et al., 2018).
  
• The first Global Citizens’ Assembly was held at COP26 in Glasgow. A panel of 100 members was drawn via a multistage civic lottery intended to mirror the global population. They were charged with addressing the question of “how can humanity address the climate and ecological crisis in a fair and effective way?” They deliberated for 68 hours over 11 weeks and developed the People’s Declaration for the Sustainable Future of Planet Earth. Plans are now being made to create a permanent Global Citizens’ Assembly, with 10 million people participating annually by 2030, to address climate change as well as AI, global public health, and other shared issues (Curato, et al., 2023) (OECD).

Within this international landscape, Scotland’s Public Engagement Strategy is distinctive in its scale and structure: it brings together national communications, climate-action programmes, multiple grant-funding streams and community-based delivery through a wide network of partners, forming one of the most integrated national approaches to climate engagement. The PES shares the same core aims that the UNFCCC’s Action for Climate Empowerment and OECD guidance stress: raise awareness, broaden participation, and enable action. Like many international examples, Scotland uses a mix of national campaigning, deliberative processes and funded partner networks to reach different audiences. In that respect, the PES is consistent with best practice, combining top-level messaging with locally rooted delivery, and explicitly framing participation as enabling co-design and community agency.

Where Scotland differs is in scale, institutional framing and how those strands are packaged. The PES has an explicit Theory of Change and nested programmes that together form a single, government-led ecosystem for engagement. Many countries run successful one-off deliberations or national campaigns, but fewer have attempted to align sustained local delivery, grant-backed experimentation and a strategy-level ToC under a single public engagement umbrella that is intentionally nested into wider climate and just transition policy. The combination of a statutory net zero commitment for 2045, a national strategy that maps a clear logic, ring-fenced small-scale innovation funding, and a distributed hub model to deliver place-based work is unique internationally. In short, while the individual elements of the PES exist in other nations, there is nothing exactly like Scotland’s PES in terms of the specific mix of programmes and ToC. However, Scotland’s PES sits in an international context of practice that values deliberation, public dialogue and participatory policy. Its structure gives it an opportunity to lead by example, particularly if it capitalises on the current window to strengthen its programme theory and show how local experimentation feeds strategy, thereby turning international endorsement of citizen engagement into examples of cumulative impact.

Detail on M&E for international public engagement on climate change

Scotland’s PES recognises climate change as a collective societal challenge and intentionally seeks to build public understanding, participation and action. It also seeks to monitor and evaluate engagement in a way that balances standardised metrics with sensitivity to local variation, supporting iterative learning while ensuring that people feel their contribution meaningfully influences progress. The review of international examples identified two examples of public engagement approaches which provide useful case studies for Scotland’s PES. These examples are given in Appendix D – Case studies.

Among these examples, Wales is at the earliest stage of delivery but has taken the clearest step toward constructing a structured evaluation architecture. The Welsh Government’s Public Engagement Strategy is underpinned by a detailed Theory of Change developed through participatory workshops, which clarifies assumptions, dependencies and intermediate outcomes across four “green choice” themes (Welsh Government 2023 p.19). Although a full evaluation framework has not yet been finalised, Wales has already identified a range of measurable outputs including media, press and visual assets, engagement analytics, demographic insight work and other activity metrics. It has also outlined how outcome measurement will draw on existing national datasets such as the Climate Change Perceptions and Actions Survey, the Welsh Government National Survey and the Climate Action Wales Behavioural Tracking Survey.

Canada’s Climate Action and Awareness Fund represents a very different model. Rather than a unified strategy, it provides funding to diverse external organisations, resembling Scotland’s Climate Engagement Fund. Each project or programme has its own objectives, reporting requirements, and spending targets, although these are not centrally collated and stated on the Government webpage. There is no overarching evaluation framework, no shared Theory of Change and no centralised data infrastructure for public engagement outcomes. Reporting is limited to project-level activity counts and summary outputs, with little evidence of systematic feedback loops. Without national-level KPIs or harmonised indicators, Canada’s approach generates significant activity but no mechanism to understand cumulative public engagement impact or system-level change.

Saint Lucia offers the strongest example of population-level monitoring among the three cases, although its programme-level evaluation remains nascent. Its Communications Strategy draws on a long-running series of Knowledge, Attitudes and Practices (KAP) surveys, which provide baseline data on public awareness, perceptions and climate-relevant behaviours. These datasets effectively function as Saint Lucia’s headline KPIs, enabling the government to track shifts in understanding and attitudes over time. While current communications activities rely mainly on one-way information tools such as media monitoring and virtual awareness materials, the repeated KAP surveys provide a rare longitudinal evidence base against which emerging engagement efforts can be assessed. Future evaluation work linked to the National Adaptation Plan is anticipated, but systems for programme-level assessment are still developing.

Across the three examples of public engagement at national scale, Wales illustrates the importance of establishing clear assumptions, risks and delivery dependencies at the outset through structured ToC development, while also acknowledging that meaningful evaluation depends on sustained dialogue between delivery partners and the communities involved (McNamara & Foss, 2024). In contrast, Canada’s Climate Action and Awareness Fund highlights the risks of operating without a coherent strategic framework: the lack of formal feedback mechanisms, the early curtailment of the programme and concerns raised by the Office of the Auditor General about weak planning all point to a disconnect between activity and evaluative rigour (Office of the Auditor General of Canada 2021 p.29). Saint Lucia demonstrates the value of anchoring public engagement in community expectations and lived experience, with respondents emphasising the need for greater education, sensitisation and community-based research to guide adaptation practices. Taken together, these cases underline that while many countries invest in climate engagement, Scotland remains at the forefront of practice in actively commissioning and operationalising a strategic-level evaluation capable of generating cumulative insights across multiple years and programmes of public engagement.

Scotland’s PES therefore sits at the forefront of this emerging area, attempting to do what few others have yet achieved: evaluate public engagement not as isolated interventions, but as a coordinated portfolio whose combined impact can be assessed over time. This research therefore echoes the findings of the Millar et al. report: “Scotland is already leading the way – not only in having a Public Engagement Strategy on Climate Change in place, but also in seeking to review and evaluate its progress” (Scottish Government, 2025, p. 3).

The absence of a clear international blueprint makes it both necessary and appropriate to return to first principles when considering how the PES should be evaluated. Where comparable strategies are limited, evaluation cannot rely on replication of existing models or indicators. Instead, it must be grounded in established guidance on how to evaluate complex, adaptive interventions operating within dynamic systems. The following section explores the complexity of the PES and its operating environment in more detail, exploring key variables and considerations relevant to the choice of evaluation approaches and providing guidance on how methods and metrics should be selected to generate credible, useful learning.

First principles of evaluation in complex systems

As detailed in 6.3 International examples of public engagement strategies on climate change, there are few examples of international governments who have implemented strategies or initiatives similar in scale and structure to Scotland’s PES. It follows that there are few robust evaluations or precedent for evaluating interventions like the PES. Therefore, the next step in the process of this research was to consider: What is best practice for evaluating complex, multi-stranded government interventions more generally (i.e., not specific to public engagement and/or climate change)? What are useful examples of where this has been done?

First principles of evaluation in complex systems: Key takeaways for the findings of this report

The PES encompasses a diverse set of interventions which operate in a complex system, designed to engage the full breadth of the Scottish public on climate change, across all sectors and geographies.

The complexity and diversity of variables and forces involved poses challenges for evaluation. This is because these confounding factors make it difficult to isolate the impact of the intervention, therefore limiting the conclusions that can be drawn and claims made about strict causality, tying changes observed at society level directly back to PES activities.

Evaluations of interventions operating within complex systems require a clear programme theory or Theory of Change (ToC), setting out the reasoning behind how the intervention is meant to work, including causal pathways, assumptions, and evidence. The PES has a high-level ToC in place; however, the current version will need to be expanded if it is to serve as a useful foundation for robust evaluation. A full discussion of how to approach this need, including a template, has been provided in Appendix C – Using the Theory of Change as an evaluation tool.

Specific methods which could be useful include participatory systems mapping, Most Significant Change, or outcome harvesting (Bicket, et al., 2020). These are methods that the Scottish Government policy teams could use with PES delivery partners and trusted messengers to capture their understanding of how the PES is delivering impact on the ground and articulate this through a full Theory of Change. This will address the identified need for the Scottish Government to be able to more clearly articulate how and why different climate engagement activities, such as those set out in the PES, can work together to deliver intended outcomes.

Best practice for evaluating complex policy interventions builds on the same principles that underpin evaluation across government: evaluation must be proportionate, fit-for-purpose, and embedded from the earliest stages of strategy and programme design. The Green Book (UK Central Government Guidance on Appraisal and Evaluation) provides the overarching structure for aligning objectives, monitoring and evaluation across strategies, programmes and projects (HM Treasury, 2022), while the Magenta Book (UK Central Government Guidance on Evaluation) sets out the core methodological requirements for credible, theory-based evaluation, including the need for a clear Theory of Change, well-defined evaluation questions and appropriate methods for addressing complexity (HM Treasury, 2020).

The Magenta Book defines any policy, programme or activity intended to bring about change as an “intervention” and stresses that evaluation planning begins before implementation. A credible evaluation plan must identify the objectives of the intervention, articulate a clear Theory of Change (ToC), outline any necessary literature reviews, specify the evaluation questions, identify when and by whom evidence will be needed, plan for detecting unintended outcomes, account for wider contextual factors and set out methods, data requirements and the resources required. Evaluation questions define the scope of the work and should be informed by the purpose of the evaluation, gaps or uncertainties identified in the ToC and the needs of stakeholders such as funders, implementers and affected groups. They should also reflect key decision points for the intervention and the expected use of findings, including short-term needs such as benefits realisation and longer-term learning about what works, why and for whom.

The Magenta Book sets out a staged process for evaluation:

• Scoping, which involves developing a detailed understanding of the intervention and choosing evaluation questions and approaches;
  
• Method and data design, which involves selecting methods, collecting data and analysing evidence;
  
• Use and dissemination of findings in order to support ongoing learning and accountability (HM Treasury, 2020, p. 20).

The Green Book complements this by recommending that interventions use a small number of SMART objectives supported by a clear rationale and evidence-based appraisal. These objectives should align across strategies, programmes and projects, with programme objectives framed as outcomes and project objectives framed as outputs that enable those outcomes (HM Treasury, 2022, p. 26).

What do these steps look like in the context of the PES? The PES operates within a complex, adaptive system, where interventions interact with an evolving policy context, diverse stakeholder groups and long-term feedback loops. In such environments, traditional, linear, “did‑it‑work” evaluations struggle to establish clear attribution due to the number of variables and influencers involved. Robust causality is difficult to infer, particularly in the absence of a counterfactual (i.e., “What would have happened in the absence of the PES?”). The Magenta Book discusses the aspects of complex systems which pose challenges for evaluation:

“Because of the multiple interactions and influences in complex systems, the relationship between a policy intervention and its outcomes may be far from straightforward. For example:

• Complex Systems may be in continual change, or might resist change, as different parts respond at different times, or adapt to maintain the ‘status quo’ (linked to the characteristics of adaptation, feedback loops, emergence and self-organisation and domains of stability);
  
• Context (and history) matters: the same intervention will often have different outcomes in different contexts, or if delivered in a slightly different way (linked to the characteristics of being an open system and path dependency);
  
• The nature of the change is unpredictable, going faster or slower, or taking a different path to the one expected (linked to the characteristics of non-linearity, unexpected indirect effects, feedback loops, levers and hubs and tipping points);
  
• Multiple perspectives: different actors within the system will often have different views of what is happening, and this can influence the way they respond to an intervention.” (HM Treasury, 2020, p. 18)

These confounding aspects of complex systems have consequences for the steps to developing and carrying out an evaluation. For a detailed definitions and discussions of the different properties of complex systems, see Magenta Book Supplementary Guide: Handling Complexity in Policy Evaluation (Bicket, et al., 2020). All of these properties apply to the PES and its wider context, to varying degrees. Therefore, how does this affect the steps for the Scottish Government to explore, scope evaluation, and select appropriate methods?

Appraisal: Building understanding of the PES and its context

At the outset of commissioning and managing an evaluation, it is essential to develop an initial understanding of the policy or intervention being evaluated and the context in which it operates. This early understanding underpins both the initial evaluation design and the deeper insights generated as the evaluation progresses. The Magenta Book recommends that evaluators develop a clear map of the intervention and its intended outcomes using tools such as logic models, policy mapping or a Theory of Change (ToC), particularly where this has not already been undertaken during policy planning or appraisal. This mapping stage becomes especially important where policies are complex, as it helps clarify how different elements are expected to interact and where risks and uncertainties may arise (Bicket, et al., 2020).

• A Theory of Change (ToC) is a structured, explicit explanation of how and why an intervention is expected to lead to desired outcomes. It sets out the causal pathways linking activities and outputs to short-, medium- and long-term outcomes, and makes explicit the assumptions, risks and external factors that must hold true for change to occur. A ToC is typically used to guide planning, monitoring and evaluation by clarifying what success looks like, what evidence is needed, and where learning should focus.
  
• A system map is a visual or analytical representation of the wider system in which an intervention operates, showing the multiple drivers, relationships, feedback loops and contextual factors that influence outcomes. Rather than focusing on a single intervention’s intended pathway, system mapping seeks to understand how change actually emerges within a complex system, including non-linear dynamics, interdependencies, leverage points and unintended effects. System maps are descriptive rather than prescriptive and are particularly useful for diagnosing complexity and revealing where interventions may interact or conflict.

Where appropriate, system mapping and modelling approaches can complement logic models or Theories of Change, either before or alongside them, to encourage a broader understanding of the multiple interacting factors that shape delivery and outcomes. Where system mapping has already been undertaken during appraisal (i.e., the “building understanding” step), it can provide a strong foundation for evaluation design. In highly complex interventions, the boundary between appraisal and evaluation may be blurred, with ongoing learning requiring regular review and adaptation of the intervention itself. In such cases, evaluation should actively feed back into decision-making, supporting continuous reflection, adjustment and, where necessary, reappraisal over the life of the policy.

The Defra complexity evaluation framework, as shown in Figure 2, illustrates the iterative nature of evaluation for complex policy frameworks. The framework emphasises that, for policy evaluations in complex contexts, developing understanding of the system and designing the evaluation are simultaneous, related, iterative processes, rather than taking place sequentially, as might be possible for simpler contexts and interventions. These processes continue throughout the evaluation’s life cycle, adapting it as necessary as understanding improves. Understanding and design allow evaluators to clarify the evaluation purpose, which sits at the centre of the framework. Understanding and designing are conducted with the support of stakeholders, therefore learning is fed back and embedded into relevant processes both inside and outside the evaluation. All of these evaluation components are managed by the evaluation team and/or commissioners in response to findings and evolving needs (Bicket, et al., 2020, p. 27).

The PES seeks to help enable change on a whole-of-society basis, and a broad array of factors and variables will influence its outcomes. Articulating a robust Theory of Change (and perhaps a broader system map) is important to clarify how the PES is meant to work, its “programme theory,” which evaluation then assesses. As discussed in the previous section, change rarely follows a simple or linear pathway from inputs to outcomes in complex systems. For this reason, early mapping should be treated as provisional rather than definitive. Its value can be significantly enhanced by involving stakeholders who have detailed knowledge of the delivery context or experience of similar interventions elsewhere, as they can help anticipate how complexity may influence outcomes.

The PES has an existing Theory of Change (Figure 7 in Appendix C). However, this version was designed to be a communication tool to summarise PES thinking, rather than to reflect the full scope of the PES or for use as an evaluation tool. Scottish Government teams, PES delivery partners, and the wider informal network of trusted messengers all hold significant knowledge about the PES, the practical realities of how it works, and insight into local contexts to which it is applied. Co-creating a comprehensive Theory of Change and potentially a wider system map would provide an opportunity to capture and visualise this knowledge, creating a resource which clearly communicates the logic and applications of the PES for internal and external audiences. This addresses the identified need to more clearly articulate how and why different climate engagement activities can work together to deliver intended outcomes.

The first step of a strengthened M&E system for the PES is therefore to expand this ToC. Practical approaches and methods for doing this are:

• Participatory Systems Mapping: Qualitative, participatory mapping approaches to draw out multiple perspectives and generate deeper shared understanding of the system, causal mechanisms and the process of change. Maps can be analysed using network analysis tools to help identify aspects of the system such as: key influences, trade-offs, feedbacks, and vulnerabilities (see practical guide from the Centre for Evaluation of Complexity Across the Nexus here).
  
• Most Significant Change: Significant change stories emanating from the field level, are periodically collected and the most significant selected by panels of designated stakeholders or staff. Once changes have been captured, groups sit down together, read the stories aloud and have regular and often in-depth discussions about why the stories are considered significant and in particular, to learn about the similarities and differences in what different groups and individuals’ value. It provides some information about impact and unintended impact but is primarily about clarifying the values held by different stakeholders (see comprehensive overview from BetterEvaluation here).
  
• Outcome Harvesting: A participatory methodology that collects (‘harvests’) evidence of what has changed (‘outcomes’) and then, working backwards, determines whether and how an intervention has contributed to these changes. Various data collection methods such as interviews and surveys (face-to-face, by telephone, by e-mail), workshops and document review can be employed. It is repeated as often as necessary to understand what the intervention is achieving (see comprehensive overview from BetterEvaluation here) (UK Government 2021 pp.59-60)

These are useful methods for building understanding of the intervention and its context, outputting an updated ToC and potentially a wider system map, which then informs evaluation design. The PES ToC should be expanded based on the knowledge and lived experience of trusted messengers and delivery partners. This directly addresses the need for this research, to more clearly articulate how and why different climate engagement activities can work together to deliver intended outcomes. Additional guidance on how to do this, including applying the above methods, is given in Appendix C – Using the Theory of Change as an evaluation tool. A robust, articulated Theory of Change is absolutely a prerequisite for evaluating a complex intervention operating within a complex system.

Key factors in evaluation design

As discussed in the previous section and visualised in Figure 2, building understanding and evaluation design are parallel, iterative processes. The key factors impinging on evaluation design are the system attributes (explored and clarified in the through the “building understanding” step, constructing a robust ToC and/or system map), the evaluation purpose (depends on the intervention’s stage/maturity and needs of key stakeholders), and what are feasible designs (depends on availability of data, expertise and resources) (Figure 3).

This section explores these key factors in detail and relates them specifically to the PES.

Key factors in evaluation design: Key takeaways for the findings of this report

Evaluation of public engagement is complex. Appropriate approaches are determined based on the context and specific goals of the intervention and evaluation. Complex, multi-stranded interventions operating in systems affected by many variables (such as Scotland’s PES) require agile, flexible approaches to evaluation. Together, these conclusions point to an evaluation style that:

• centres on a refined ToC that articulates the three PES objectives (Understand, Participate, Act) and the causal chains which clarify how PES activities and programmes support them
  
• adopts a mixed‑methods design that embraces different data types, combining high‑level quantitative tracking (national indicators like Scottish Household Survey, campaign reach) with deep qualitative inquiry (case studies, stakeholder interviews, participatory system mapping) for different PES programmes and activities, as appropriate
  
• uses theory‑based evaluation approaches (such as contribution analysis) to uncover mechanisms, which can handle these diverse data types and contextual conditions. These will give insight into the degree of contribution, while acknowledging that a pure experimental counterfactual is unlikely to be feasible.

Theory-based evaluations are based around a robust Theory of Change and are bespoke to the stated objectives of each intervention. Based on the evaluation purpose, system attributes, and feasibility, this report recommends a hybrid design for PES M&E, combining elements of participatory and theory-based to strengthen and address the challenges to M&E. The next section explores the opportunities and practical considerations to applying a contribution analysis approach to the PES.

Evaluation purpose: PES M&E challenges and needs at strategy level

The Magenta Book discusses several key purposes for evaluation: accountability (i.e., evidence of impact; Has the intervention achieved its aims? Has it represented good value for public money?); learning (i.e., How is the intervention working or not working? How can it be improved?); and listening and building (i.e., ensuring diverse voices play a part; builds trust and legitimacy across stakeholders) (Bicket, et al., 2020, p. 31).

All of these purposes are relevant and important for the PES:

Accountability: “We will draw together all the data we have collected from our programme evaluations to date, our national indicators and any bespoke commissioned research to provide an assessment of our overall progress towards our strategic objectives.”

Learning/process: “We will also use this as an opportunity to gather feedback from key stakeholders and delivery partners on what aspects of the strategy they think are working well and what could be improved. The findings of this review will be made available and we will use them to shape and inform the remainder of the strategy’s implementation.”

Listening and building: “Responses to our consultation highlighted the importance of involving wider stakeholders in our evaluation activity, including recognising the role of community-led methods and data collection. We are committed to working with our delivery partners in designing programme evaluations that not only track progress towards objectives but also provide opportunities for learning and development” (Scottish Government, 2021, p. 43).

The stakeholder workshop that was conducted as part of this research was also an important opportunity to understand the purposes of PES evaluation from the perspective of the Scottish Government teams who are responsible for shaping and steering the strategy, commissioning evaluations, and integrating resulting findings:

“That there’s so many different levels and lenses that you can look at the PES in terms of what it’s trying to deliver. And so, finding a way to organise that in the monitoring and evaluation that allows us to identify gaps or big successes, or maybe where more effort is needed, I think that’s the key thing. […] Saying, “Do we have the right combination of these kinds of tools and metrics and things to paint the whole picture and be able to then identify those highs and lows?” (Quote from workshop with Scottish Government)

It is clear from the above that PES evaluation will ideally cover all purposes of evaluation, based on the evaluation questions that key PES stakeholders want to see addressed. This therefore informs evaluation design and the selection of methods which can deliver on each of these for the PES at strategy level.

System attributes: Addressing different scales and styles of engagement within the PES

The PES is a complex and multi-stranded set of interventions because it is designed to address a complex and thorny problem. Scottish Government’s approach to climate action is described using the language of “widespread,” “transformational,” and the framing of the PES document emphasises a “holistic, systemic approach” to public engagement (Scottish Government, 2021). The PES includes a diversity of layers, scales, and styles of engagement in order to provide robust and comprehensive approach to the wide-ranging issue of how to meaningfully engage the public on climate change.

As discussed in 5.2 Overview of PES history, programmes & partners, the component programmes of the PES are designed to contribute to its strategic objectives, Understand, Participate and Act. PES programmes provide national coverage, aiming to reach as many members of the Scottish public as possible and providing opportunities to participate and get further involved. PES programmes are also tailored and delivered in local context. For example, Climate Action Hubs and Schools are delivered at regional level, adapting to the needs of their areas. This is intentional, delivering key climate messaging through trusted messengers:

“Community groups already have an existing audience that they work with, and they might communicate with, their social media, their notice board in their village hall etc. And they also know people and they have an understanding of their issues and their priorities” (Quote from interview)

Locally embedded PES delivery partners and wider trusted messengers translate strategic objectives into locally relevant activity. They also tie into the social fabric of each local area in different ways and can act as catalysts for widespread change^[2], as was shared by a PES stakeholder involved with Climate Action Schools:

“Quite often it’s actually local schools that act as, or have acted as a catalyst for climate action in a local area. In some parts of the country, schools are very active and outward-looking in terms of their work, and they do engage with local businesses and local community groups […]Local authority officers who have engaged with us and seen the value of what we offer continue to return, request further support, and try to encourage schools to participate. While their influence over schools is limited, they persist because they recognise that our offer supports their own local authority targets, climate action plans, and wider sustainability objectives.” (Quote from interview)

Climate Engagement Fund projects are delivered at a range of scales (including specific cities and regions) and by a range of organisations from different sectors (e.g., museums, universities, charities, etc.) (Figure 4).

Activity delivered under these programmes varies widely. The differing scales and demographics of target audiences of these programmes, or the aspects of the PES strategy that they are designed to deliver, determines how they shape their delivery and set their own KPIs.

• Programmes aiming to increase understanding across the full breadth of society, such as the Let’s Do Net Zero communications, Net Zero Nation website, and communications around Climate Week or other events represent very broad, but comparatively shallow engagement. They collect KPIs such as number of Scottish adults reached, how many times they saw activity, and data collected from website traffic such as followers, email engagement, and downloads of materials.
  
• Conversely, programmes delivering deeper engagement in narrower, local contexts, face-to-face with participants, such as the Climate Action Hubs, Climate Action Schools, and Climate Engagement Fund projects can collect more detailed and qualitative data such as demographics reached, changes in understanding, confidence, or intended behaviour changes, and a wide range of other metrics and case studies collected from people participating directly in PES programmes and activities (Scottish Government, 2025).

PES programmes each have their own monitoring and evaluation strategies in place, proportionate to their scale and maturity. Scottish Government teams collect data from projects and programmes managed by locally embedded delivery partners, such as quarterly monitoring forms from Climate Engagement Fund projects. The Climate Action Hubs have a fully refreshed M&E system, which includes a spreadsheet linking key data fields / KPIs to the Hubs ToC. Climate Action Hubs also set additional, individual KPIs based on the priorities of the local community:

“Spending a lot more time getting to understand the local landscape and what other funders are offering in the area and things like that has meant that, this year, our KPIs are probably a bit more bespoke, whereas in the first year they were probably more similar to some of the other Hubs.” (Quote from interview)

Climate Engagement Fund projects are encouraged to do the same, with a part of their monitoring forms left for free text entry about what they have achieved beyond the standard KPIs. This makes sense: PES programmes are all different, each designed to address a different part of the PES’s overall objectives, and require different monitoring approaches and metrics to capture and reflect the outcomes of their delivery. It would not make sense to compare them based on the same metrics. The diversity of trusted delivery routes strengthens the Strategy by increasing the likelihood that climate messaging reaches a wider and more diverse public. At the same time, it further underscores the importance of programme-specific monitoring frameworks that can capture how different forms of trust, audience reach and engagement contribute to the Strategy’s shared objectives rather than attempting to assess performance through standardised indicators alone. The following interview quote expands on this idea in the context of the PES’s three strategic objectives:

“For the Participate outcome, it makes a lot of sense to me to do the deep dive deliberative stuff. […] If you engage 100 people, you can have a really useful participatory process that really does change and inform policy in a useful way. You don’t need to talk to all 5 million people, you wouldn’t get that much more from talking to a lot more people, right? Whereas for the other two impact outcomes [Understand and Act], it’s more complicated, because you have to raise awareness across the majority of the population, or you have to get most people to act. […] We’ll never reach everybody, but we have to get enough people that they influence each other, and businesses, and other organisations.” (Quote from interview)

However, the diversity of scales, approaches, and corresponding data types poses challenges for summing up the impact the PES is having as a whole, coherent strategy, as highlighted in the MPR:

“At the time of this review the portfolio of activity in support of the PES represented a variety of approaches at different stages of delivery. Activities operate at a range of scales, with different measures of success which are difficult to aggregate or compare across. These factors add to the complexity and challenge of evaluation.” (Scottish Government, 2025, p. 10)

The central challenge for PES M&E at strategy level is to create a framework which can use this diversity of scales, approaches, and data types to assess and communicate the cumulative impact of the PES. This has implications for which evaluation approaches and methods are feasible, discussed in the next section.

Feasible designs: Selecting appropriate evaluation approaches for the PES at strategy level

The central challenge for PES M&E is to build an evaluation framework at strategy level that can:

• account for different scales and styles of engagement,
  
• remain adaptable and agile, and
  
• capture emergent and unintended outcomes.

There are a variety of potentially suitable evaluation approaches and methods that could be applied to the PES, working within a Theory of Change framework. The below list of approaches comes from the Magenta Book supplementary guide on handling complexity in policy evaluation, laying out a typology of evaluation approaches which can be used in complex contexts:

• Participatory, emancipatory and adaptive approaches such as developmental evaluation, action research and peer challenge, highly responsive and exploratory approaches in which stakeholders take an active part in the delivery of the evaluation providing real time feedback on the policy.
  
• Theory-based approaches which articulate a theory of how the policy is working to deliver change, then seek to test this to investigate whether, why or how the policy causes or contributes to observed results, and whether alternative explanations can be ruled out. Approaches include:
  • Systems mapping and modelling, used to generate, progress and test the theory of change through an iterative process of developing and testing a formal model of the system, and
    
  • Generative causation approaches, such as realist evaluation and contribution analysis, that seek to articulate underlying mechanisms or processes of change, and test the theory empirically to investigate whether, why or how the policy causes or contributes to observed results, and how context influences these.
    
• Configurational case-based approaches such as qualitative comparative analysis, which support systematic analysis of a number of cases of the intervention being evaluated to identify the configuration of factors, or combinations of factors, that appear necessary or sufficient, to success.
  
• Counterfactual approaches, including:
  • Experimental approaches such as randomised control trials, and quasi-experimental such as difference in difference, which provide a usually quantitative measure of the extent to which any observed changes in an outcome of interest were caused by the intervention (or treatment) by means of a comparison of results obtained for a treatment group with those in a non-treatment control, and
    
  • Predictive approaches, which attempt to predict what would have happened in the absence of the treatment using statistical or simulation modelling; in its simplest form, predictive approaches use informant opinion as to whether impacts would have happened in the absence of an intervention.
• Statistical association approaches, that look for correlations between cause and effect or between variables, to explore the influence of (usually) isolatable multiple causes on a single effect, while controlling for ‘confounders’.
  
• Synthesis designs, such as realist synthesis, which seek to draw conclusions by combining results from evaluations drawn from several contexts. (Bicket, et al., 2020, p. 42)

As discussed in previous sections, choosing feasible and appropriate evaluation approaches from this list depends on evaluation purpose and system attributes. Figure 5 plots these approaches based on two key attributes of complex systems, their degree of certainty and agreement. On the horizontal axis, policy areas that are far from certainty benefit from approaches which embrace systems thinking (i.e., developing a system map and ToC; seeking data about scope, context and interactions), whereas better understood systems may not require this depth. On the vertical axis, policy areas that are further from agreement benefit from more participatory methods (i.e., wide engagement, participatory mapping, agile and adaptive management), whereas areas where there is less of a diversity of positions may not require this breadth. This illustrates why participatory, adaptive, generative approaches are more suitable for multi-stranded interventions operating in complex systems, such as the PES. Adapted from (Bicket, et al., 2020, p. 44).

Counterfactual and/or statistical evaluation approaches provide the clearest and most robust evidence for strict attribution of impacts, i.e. “The intervention clearly has had these effects, because they are present in treatment cases and absent in control groups.” However, counterfactual approaches are often challenging in complex settings because they necessitate control groups and the tight controlling of variables (HM Treasury, 2020, p. 46). A central challenge of evaluation in the PES is that it is a very “open” system, with many possible variables and local settings affecting the delivery of its activities. This quote from the Magenta Book demonstrates the difficulties that this poses for experimental approaches:

“An experimental design used to evaluate a programme designed to help disabled people into work foundered when it was discovered that organisations operating in different locations delivered the intervention in different ways, tailoring this to the types and levels of disability they were dealing with, and the kind of work opportunities available locally. […] As well as making it difficult to clearly articulate, or standardise, the intervention, these factors also made it difficult to identify and collect uniform data from an intervention group and a control sample” (Bicket, et al., 2020, p. 23).

Counterfactual approaches are

“difficult to apply where causal pathways are complex, little understood and hard to unravel, making them unsuited to analysis through the experimental manipulation of single causal factors. Can, by their generalising nature, hide the initial emergence of new phenomena in a complex system” (Bicket, et al., 2020, p. 55).

While an updated ToC for the PES would clarify its assumed causal pathways leading from inputs to impacts, the breadth of the PES’s activities and target audiences make counterfactual and other reductive approaches unsuitable at this stage of its development.

Due to the complexity of the PES’s system attributes, as well as the diversity of its programmes, activities, and outcomes, PES M&E will need to embrace an evaluation design which is flexible, agile, and which embraces systems thinking and participation. These factors point towards participatory, emancipatory, adaptive, and theory-based approaches. These types of approaches have important advantages which make them suitable for strategy-level evaluation of the PES.

For policies that are multi-stranded, evolve over time, and interact with wider social, political and institutional systems (like the PES), evaluation approaches must be theory-based, flexible and capable of accommodating non-linear change, multiple causal pathways and uncertainty. An evaluation framework for the PES must be built from a clear understanding of what the Strategy is trying to achieve, how change is expected to occur, and which assumptions are most critical to test. By focusing on Theory of Change development, contribution rather than attribution, and the careful selection of proportionate methods, the approach aligns with Magenta Book guidance for complex interventions where experimental designs are neither feasible nor appropriate. This also creates space to explicitly consider system effects such as feedback loops, reinforcing mechanisms and contextual variation, features that are central to public engagement but often invisible in simpler evaluation models.

The main limitation of theory-based approaches is their inability to strictly attribute the change (the “effect size”) to the intervention. However, they are good at articulating why an intervention is working and exploring how it is working or not working in certain contexts, and whether this might be transferrable to other contexts. Theory based approaches can also embrace and reconcile different evaluation methods and data types (e.g., output and performance monitoring, surveys, interviews, focus groups, case studies, ethnographies) (HM Treasury, 2020, pp. 42-43). This makes an overarching theory-based approach a good choice for the PES at strategy level, as it could incorporate the outputs of programme-level evaluations in a hybrid design:

“None of these approaches or methods is mutually exclusive. In evaluation it is common to combine different methods so that conclusions are based on several different sources of information gathered in different ways. For complex evaluations, often what is required is a hybrid design in which two or more approaches are combined and tailored to meet the needs of the evaluation” (Bicket, et al., 2020, p. 43).

The Magenta Book explicitly recommends an adaptive management style for complex issues, embedding evaluation in implementation, using regular feedback loops and allowing the design to evolve as new insights emerge. Both the Magenta Book and the Guide stress that learning (risk‑management, improvement) and accountability (demonstrating impact) can be pursued simultaneously when the evaluation is proportionate, iterative and focused on contribution rather than strict causality.

Recommended approach: Applying contribution analysis to the PES at strategy level

Applying a contribution analysis approach to the PES at strategy level: Key takeaways for the findings of this report

This section demonstrates that contribution analysis provides a robust, proportionate framework for evaluating the PES at strategy level in the absence of experimental counterfactuals.

Rather than standardising or aggregating diverse programme data, it shows how a refreshed Theory of Change and theory-based evaluation can credibly integrate multiple data types to build a coherent contribution story across Understand, Participate and Act.

The analysis confirms that the PES already generates much of the right evidence; the key gap is not data collection but strategic synthesis, target-setting and sense-making.

Delivery partners and trusted messengers are central to this approach as contributors to appraisal, learning and innovation. Storytelling and case studies play a vital role in communicating impact where quantitative attribution is neither feasible nor appropriate.

To evaluate the impacts of the PES, we recommend the Scottish Government applies contribution analysis. This is a particularly useful method when experimental counterfactuals are impossible. It provides a structured, step-by-step approach to examining whether an intervention plausibly contributed to observed outcomes, whether the causal pathways in the ToC are supported by evidence and whether alternative explanations can be ruled out. It does this by testing and refining the ToC, examining the strength of evidence for each causal link, integrating quantitative and qualitative data, and assessing whether rival explanations offer a better account of the change. Although it cannot offer definitive proof of attribution, it can provide reasoned and credible evidence of contribution. It is most effective when supported by a clear and well-specified ToC and is particularly suitable for complex portfolios such as large public engagement programmes, where outcomes arise from multiple interacting influences and where interventions vary in type, scale and context (HM Treasury, 2020).

Other theory-based tools include realist evaluation, which examines what works, for whom, in what contexts. However, realist evaluation is more about how the intervention is working (process evaluation), while contribution analysis is skewed towards attribution, tying activities to impacts (impact evaluation) (HM Treasury, 2020, pp. 14-15). Process tracing and qualitative comparative analysis tests causal pathways in cases. These could be useful to investigate specific PES programmes which are less well understood or have not yet been evaluated. Innovative complements might include systems mapping (wider context / ecosystem), outcome harvesting (works back from observed changes), potentially useful for constructing the ToC, discussed in Appendix C – Using the Theory of Change as an evaluation tool (HM Treasury, 2020, p. 49).

Applying contribution analysis to the PES could provide a robust framework for Scottish Government teams to understand how their PES activities influence society. Since experimental designs are infeasible for the complexity of evaluating large-scale public engagement, contribution analysis allows teams to infer causality by verifying the ToC behind the PES, confirming that the expected chain of results occurred, and accounting for external factors. By following the six iterative steps below, Scottish Government teams can build a “contribution story” that provides a credible assessment of whether the PES has made a difference in how people in Scotland understand, participate in, and act on climate change.

Step 1: Set out the attribution problem to be addressed

The first step for Scottish Government teams is to acknowledge the attribution problem: determining the extent to which observed changes in public attitudes and behaviours are due to PES activities rather than external factors (e.g., global news, energy price fluctuations, or UK-wide policies). This involves moving from the question of, for example, “To what extent did the PES cause behavioural change across society?” to the manageable question: “Is it reasonable to conclude that the PES activities have made a difference to the public’s willingness to engage with net zero?”. This is a subtle but important change – While contribution analysis cannot strictly quantify the “effect size” of an intervention, it provides a framework to verify the links of the causal chains which lead to desired impacts.

Step 2: Develop the Theory of Change and the risks to it

As discussed in 7.1 Appraisal: Building understanding of the PES and its context, robust evaluation hinges on an updated PES ToC, which maps out how PES activities lead to the strategy’s three pillars: Understand, Participate, and Act. This involves creating chains that link outputs (what the government does) to immediate outcomes (reactions or outtakes), intermediate outcomes (e.g., behavioural or other observable changes), and final impacts. Within these chains, the method requires the identification of assumptions (the conditions that must exist for one step to lead to the next) and the risks that might prevent this progression.

The below list provides some example, high-level causal pathways for PES objectives. These illustrate some pathways through the PES Theory of Change, illustrating how Inputs / Activities lead to Outputs, which lead to Outtakes^[3], which lead to Outcomes, together building to contribute to Impacts. These are presented, as well as the types of evidence which can support each. Assumptions are listed, including some taken from the ToC for the Wales PES, where these are relevant to Scotland’s PES. For a full discussion of how to refresh and update the PES ToC and how this builds into the evaluation framework, see Appendix C – Using the Theory of Change as an evaluation tool.

• Understand: “Let’s Do Net Zero” Campaign (Inputs/Activities; Outputs, evidenced through monitoring data) → Public recall of campaign messaging (Immediate Outcome or Outtake, evidenced through surveys, e.g.) → Improved knowledge of how climate change relates to individual lives (Outcome, evidenced through surveys or poll, e.g.) → Increased awareness of Scotland’s collective net zero goals (Outcome, evidenced through surveys or poll, e.g.) → (Supports) Implement the updated Climate Change Plan (Impact, assumed)
  • Assumption: Increased understanding leads to further engagement and/or behaviour change, i.e., “if you know better, you do better.” Evidence for assumptions can be drawn from a wide variety of sources and constitutes the evidence base for the chosen approach overall. It should be robust and clear, showing how the intervention was developed based on the best available information (applies to all) (HM Treasury, 2020, pp. 36-37).
    
  • Assumption: Trusted messengers make climate change content more resonant and relevant to local contexts.
• Participate: Climate Policy Engagement Network / Participation Programme (Input/Activity; Outputs, evidenced via monitoring data) → Participants feel their voices are heard and valued (Immediate Outcome or Outtake, evidenced through surveys or poll, e.g.) → Public input is integrated into draft Just Transition Plans (Outcome, evidenced through internal monitoring or policy audit, e.g.) → Policies are more fair, inclusive, and supported by the community (Impact, assumed).
  • Assumption: “Involving the public, including disadvantaged groups, in decision-making on the climate will lead to better outcomes for the wider public and disadvantaged groups.” (McNamara & Foss, 2024, p. 30)
    
  • Assumption: There is a robust feedback loop in place, where the government explicitly communicates how participant recommendations were acted upon, maintaining trust in the process.
• Act: Funding for Climate Action Hubs (Input, evidenced via monitoring data) → Local organisations have increased capacity and networks (Output, evidenced through monitoring data) → Communities initiate and sustain local climate projects (Outcome, evidenced through monitoring data) → Taking action is normalised in households and communities (Impact, assumed).
  • Assumption: “Improvements are required to the capability, opportunity and motivation outcomes for there to be improvements in the behaviour outcomes. This is the core logic of the strongly evidenced COM-B model of behaviour change.” (McNamara & Foss, 2024, p. 55)
    
  • Assumption: “Partners have the institutional and technical capacity required to deliver and amplify the Strategy’s activities” (McNamara & Foss, 2024, p. 42). Long-term, intensive support is provided rather than one-off grants, because community-led action requires sustained capacity building to be effective.

Step 3: Gather existing evidence on the Theory of Change

In this step, teams utilise existing data (such as the data collected for and presented in the Mid-Point Review (MPR) to validate the links in the ToC. Evidence must be gathered in three areas: the implementation of activities, the occurrence of results, and the validity of assumptions.

Our review of the M&E in place across PES programmes finds that programmes are collecting many of the right types of data and indicators for quantifying the impact of government public engagement. For example, under the “Understand” objective, the MPR shows the “Let’s Do Net Zero” campaign reached 75% of Scottish adults, providing evidence for the first link in the results chain. For “Act,” independent evaluations of pathfinder Hubs confirm they have successfully empowered local organisations to initiate projects (Scottish Government, 2025). Furthermore, the MPR’s inclusion of the Millar et al. (2024) evidence review helps validate assumptions, such as the finding that the public wants honesty about the climate crisis balanced with practical, positive steps they can take.

As discussed in 8.2 System attributes: Addressing different scales and styles of engagement within the PES, programmes have their own M&E in place – The key job at strategy level is to collate the various scales and data types being collected across the PES and discuss how this evidences the strategy’s ToC, for which contribution analysis is a suitable method. For a full assessment and discussion on what data types are available to evidence different parts of the PES ToC, see Appendix E – Data and indicators for PES M&E.

Step 4: Assemble and assess the contribution story

Evaluators now critically evaluate the contribution story developed so far. This involves identifying which links in the causal chain are strong and which are weak. For instance, while there is strong evidence for the Understand objective’s reach (campaign recall), the MPR acknowledges that it is difficult to attribute national shifts in public attitudes solely to the PES (Scottish Government, 2025, p. 10). A challenge to the story might be: “Even if awareness is high, does the PES actually cause the behavioural change, or is it the rising cost of energy?”. Identifying and addressing these weaknesses head-on allows teams to see where the logic of the PES needs more support or where the influence of external factors need more recognition or exploration.

Step 5: Seek out additional evidence

Based on the gaps identified in Step 4, evaluators then gather new data to strengthen the story. The MPR and our review highlighted several areas where current evidence could be strengthened:

• Demographic data: There is a lack of demographic data on participants in the Participation Programme, making it difficult to prove that the Participate programmes are reaching under-represented groups (Scottish Government, 2025, p. 24). New evidence collection could include standardised demographic monitoring of all engagement activities to identify any gaps in representation.
  
• Long-term behavioural tracking: While the Scottish Household Survey tracks general trends, the PES delivery partners interviewed as part of this research expressed interest in the long-term impact of specific interventions, such as whether people engaging with PES activities maintain their increased motivation to act over several years.
  
• However, collecting additional data is time consuming and costly. See 9.1 Proportionality: How can contribution analysis address the challenges of PES M&E in a feasible way?

Step 6: Revise and strengthen the contribution story

Contribution analysis is iterative; as new evidence is gathered; the story is refined. If the data shows that a certain assumption is false, the ToC and delivery may need to be adjusted to include more direct support or different messaging. By following this approach, Scottish Government teams can create a revised story that clearly demonstrates how the PES’s multi-layered approach has collectively contributed to Scotland’s transition to a net zero economy. This final story will not strictly quantify the effect size the PES, but it will be more credible than solely reporting activity outputs.

For an external guide to contribution analysis, see (Mayne, 2008).

Some of the key challenges which have been identified for PES M&E are:

• “More consistency in data gathering and reporting, aligned to the PES Theory of Change, would help strengthen the conclusions that can be drawn. There is opportunity to take action on this for the remainder of the PES delivery period, and particularly for designing the end of PES review.” (Scottish Government, 2025, p. 3)
  
• “Activities operate at a range of scales, with different measures of success which are difficult to aggregate or compare across.” (Scottish Government, 2025, p. 10)

A Theory of Change and linked evaluation framework for the PES at strategy level will address this gap in comparing across the diversity of PES programmes and activities. They will do this by articulating how the different PES programmes are designed to support its overall objectives in different ways and specify what evidence is needed to evidence that this is working in practice. The primary way to evidence the cumulative impact of the PES is not to attempt to standardise metrics collected across programmes to try to compare apples to apples, when these programmes take fundamentally different approaches; Rather, it is to articulate and provide evidence for how this mix of approaches is necessary and sufficient to deliver the PES’s overall objectives (Figure 6).

Programmes need not be directly comparable or simply aggregated. The PES takes a range of approaches and scales to address the needs for engagement at different levels. These programmes each have proportional M&E strategies in place, selecting appropriate KPIs. Contribution analysis makes it possible to have a common evaluation framework across this diversity because it is method-agnostic. This means that evaluators can mix and match specific evaluation methods and data types (e.g., output and performance monitoring, surveys, interviews, focus groups, case studies, ethnographies) to capture the outputs of the different PES programmes contributing. A refreshed Theory of Change framework will link these activities together, articulating the thinking behind this mix of approaches and specifying what types of measurable cumulative impacts are reasonably expected to be observed as a result. A theory-based approach, specifically contribution analysis, emerges as a good approach for this need.

For examples of where contribution analysis has been applied or proposed in similar contexts (The EU Adaptation Strategy on Climate Change and the Climate Action Wales Public Engagement Strategy), see Appendix D – Case studies.

Proportionality: How can contribution analysis address the challenges of PES M&E in a feasible way?

Proportionality is a foundational principle of government evaluation, dictating that the level of scrutiny and resource investment must align with the scale, risk, and innovation of the policy. Evaluation must be fit-for-purpose and tailored to the needs of decision-makers. For a low-risk, well-evidenced intervention, light-touch monitoring may suffice. The scale and complexity of the PES and its context necessitate a robust, proportional evaluation (HM Treasury, 2020, p. 16).

Regarding the specific allocation of resources, official guidance clarifies that there is no “standard proportion” or fixed percentage of a policy’s total value that should be dedicated to evaluation (HM Treasury, 2020, p. 69). Instead, the scale of investment is generally proportionate to the policy’s ambition and the degree of uncertainty involved. Key factors affecting resourcing include:

• Pilots and innovations: Interventions that are breaking new ground or involve high levels of uncertainty typically require a higher proportionate spend on evaluation than routine, established policies.
  
• Data collection costs: Collecting new primary data often constitutes the largest portion of an evaluation budget. Integrating data requirements into routine monitoring and administrative systems early on can substantially reduce these costs. The MPR showcases the depth of data already collected and available for PES programmes.
  
• Specialist expertise: Complex evaluations often require specialised skills in theory-based methods, systems mapping, or software for data visualisation, which should be factored into the management case of the business plan.

For the Scotland PES, the high interdependence of its Understand, Participate, and Act objectives suggests that evaluation resourcing should be significant enough to support a wide data “net,” capable of capturing the multiplicity of factors influencing societal change. Because complex systems change over time, the evaluation must also be flexible, embracing adaptive management approaches. This means building contingency into the budget and allowing for the revision of evaluation questions as new properties emerge within the system, such as adaptation, levers and hubs, feedback loops, and/or other unintended effects (Bicket, et al., 2020, pp. 34-36).

This report finds that PES programmes are collecting many of the right types of data and indicators for robust and proportionate M&E already and have effective M&E strategies in place (see Appendix E – Data and indicators for PES M&E). This is consistent with the MPR, which concludes: “these measures and trends should be taken as an indication of progress being made towards the strategic objectives of the PES, as well as a source of evidence to inform how PES aims and activities are designed and delivered” (Scottish Government, 2025, p. 37). The key next step is to construct a coherent narrative and draw evidence together for how these programmes act in a coordinated way to support the strategic objectives.

However, given finite time and resources, how can Scottish Government prioritise among the many causal pathways which will compose the updated Theory of Change? How can Scottish Government teams carry out a contribution analysis for the PES at strategy level in a targeted, feasible way? What is the headline KPIs for the PES under this new evaluation framework?

“No individual method can provide answers to all evaluation questions. If time and resources are limited, questions and methods will have to be prioritised and trade-offs between methods will be needed. Iteration will be necessary. It is important to consider both the feasibility and appropriateness of a proposed method” (HM Treasury, 2020, p. 41).

An efficient way to track progress against the PES’s objectives will be to establish reasoned and challenging targets. The Green Book makes clear that government appraisal must be built on objectives that are Specific, Measurable, Achievable, Realistic and Time‑limited (SMART) (HM Treasury, 2022, p. 25), and the Magenta Book reinforces this by stating that evaluators should help develop “well‑defined objectives (e.g. SMART)” (HM Treasury, 2020, p. 12). The Green Book states that government interventions should typically limit themselves to “up to 5 or 6 SMART objectives” to retain focus and avoid dilution of effort (HM Treasury, 2022, p. 26). The Magenta Book does not prescribe an exact number of objectives, but it advises that the scope of an evaluation (and by extension the set of high‑level objectives that drive it) be kept manageable, recommending around 6 or 7 overarching evaluation questions (UK Government 2020 pp.28-29).

The PES and MPR do not use the language of “SMART,” but they do embed measurable targets and key performance indicators that fulfil the spirit of SMART criteria. The three strategic objectives are linked to concrete metrics such as percentages of people recognising climate urgency (baseline 68% in 2019), believing their behaviour matters (baseline 58% in 2018) and understanding required actions (baseline 74% in 2018) (Scottish Government, 2021, p. 42). Specific campaign targets are also recorded (e.g., 73% recognisers reporting increased awareness and a 50% action‑taken target, exceeded at 78% in 2021; a 60% action target in 2024, exceeded at 80%) (Scottish Government, 2025, p. 13), alongside quantifiable engagement goals for the Climate Week hashtag and website visits (p.16). However, explanation or reasoning for these targets is not provided.

The national-level indicators from the Scottish Household Survey are presented in the MPR as “the most robust Scotland-level data on public attitudes towards climate change over time” (Scottish Government, 2025, p. 7). However, the PES also recognises that there are limitations to the explanatory power of using these figures as evidence for PES efficacy, acknowledging that “there are a multitude of factors which affect these indicators” (Scottish Government, 2021, p. 42).

The refreshed PES ToC and wider strategy need to set clear targets, including linked indicators which are directly attributable to PES delivery.^[4] Depending on what is feasible to measure at strategy scale, these targets may need to be set within Outputs, Outtakes, or Outcomes, rather than at Impact level. The corresponding sections above identify which data types are available in the current reporting scheme, where the gaps are, and make recommendations for how to standardise data collection on those aspects of the PES. This is how the cumulative impacts of the PES can be communicated:

1. Set a small number of SMART targets within PES stages, including the reasoning behind the level of change targeted^[5]
   
2. Capture data across the stages of PES programmes, aggregated at strategy level where feasible, to report against these targets
   
3. Clarify the causal mechanisms, assumptions, and evidence behind the ToC, including how PES stages build on each other to contribute to Outcomes and Impacts that are not measurable or attributable solely to PES delivery (including national awareness and attitudes reported in the Scottish Household Survey, as well as national policy goals).

Some ideas for strategy-level KPIs which came out the evidence review and interviews from this project are:

• Inclusion reach (Output)

Metric: Percent of participants from unengaged or hard‑to‑reach demographics (monitoring data).

The PES aims for national coverage, but delivery often screens for sympathetic audiences rather than those most affected by the transition. This could be tracked to ensure partners (Hubs, Schools, Climate Engagement Fund) are using appropriate recruitment strategies, choosing different venues or using trusted messengers to access, for example, Gaelic speakers, rural residents, or low‑income groups.

• Collective efficacy (Outtake)

Metric: Percent reporting increased confidence that “people like me” can influence local climate decisions (survey).

High climate anxiety is best countered by a sense of collective efficacy, the belief that a community can work together to solve problems. This normalizes action at a local level (Scottish Government, 2025, p. 36). This could be boosted by Hubs and Climate Engagement Fund projects focusing on showcasing local success stories and peer‑to‑peer learning.

• Feedback loop completion (Outcome)

Metric: Percent of Participate activities that have delivered a “You Said / We Did” report back to the participants (monitoring data).

This emerged from interviews as the primary driver of long‑term public support and trust. The lack of evidence on how the Climate Assembly or Participation Programme influenced final policy is currently a “significant weakness” (Scottish Government, 2025, p. 22). Delivery partners can add a mandatory “Report Back” milestone to their project timelines, ensuring participants see the tangible results of their time. For a full discussion of feedback loops as a way to evidence the impact of participation, see in Appendix E – Data and indicators for PES M&E.

These are illustrative suggestions only – the most important KPIs will need to be discerned from the results of the refreshed ToC development process, ideally in collaboration with PES stakeholders and trusted messengers.

This target-setting approach has several key advantages for PES M&E: One of the most powerful uses of SMART targets is the ability to set the scope of strategy-level evaluations. It is important to share the background evidence with PES stakeholders to define what success looks like, focusing stakeholder attention on the most critical indicators. A strategy as broad as the PES cannot evaluate every outcome in every sector. By setting high-level headline targets, you effectively tell stakeholders: “These are the areas of greatest uncertainty and/or highest priority” (HM Treasury, 2020, p. 27). This focuses data collection efforts on evidence that is impact-orientated, increasing the efficacy and efficiency of the evaluation. For instance, if the literature indicates that “climate self-efficacy” is a key predictor of behavioural change, setting a SMART target around self-efficacy justifies why the evaluation prioritizes qualitative deep dives into this metric over other, more peripheral data.

Target setting provides a proactive tool for the communication or evaluation results. It allows evaluators to say: “We set out to achieve X because it is a proven indicator of success; here is the evidence that we achieved it”. This is particularly useful in complex systems where direct attribution is difficult, as is the case for the PES. By hitting specific, theory-backed targets, evaluators build a contribution claim that is credible and evidenced, even if other external factors were also at play. Headline targets serve as a storytelling device for audiences and key stakeholders. They translate complex technical data into accessible narratives about progress, following through on the commitment from the PES for accessible and inclusive evaluation (Scottish Government, 2021, p. 43). Demonstrating success against a predefined target also builds trust and legitimacy among the diverse stakeholders involved in the climate transition.

Targets themselves should be supported by evidence, as well as by stakeholders, and reviewed at each evaluation cycle. Poorly set policy targets can lead to poor or perverse outcomes and involving relevant stakeholders can help avoid this. When targets are developed through participatory methods, stakeholders are more likely to buy into the findings, even if those findings highlight areas for improvement (Bicket, et al., 2020, p. 45).

Anchoring the PES M&E strategy in this combination of contribution analysis and strategic target setting creates a cohesive, evidence-based narrative of societal change. This approach acknowledges complexity without being paralyzed by it, focusing on proving that the strategy’s mix of approaches is the necessary and sufficient driver of Scotland’s society-wide, net-zero transitions.

Communicating and embedding evaluation results with PES trusted messengers and other stakeholders

Trusted messengers outside of core PES programmes

The evaluation of the Public Engagement Strategy (PES) operates across two distinct spheres of influence, each presenting different challenges for measuring impact and establishing causality. The direct scope of evaluation includes the “umbrella” of Scottish Government-funded programmes, which have been the focus of this research, such as the Climate Engagement Fund, Climate Action Hubs, and Climate Action Schools (see Figure 1). For these initiatives, the Scottish Government has high data control through mandatory quarterly reporting, grant conditions, and bespoke monitoring templates. The indirect scope, however, encompasses a much broader landscape of trusted messengers, including public bodies like NatureScot and the NHS, as well as local community groups and industry representatives (Scottish Government, 2021, p. 9). While the PES is designed to act as a guiding framework for these messengers to ensure coordinated and consistent climate communication across all parts of government and society, these partners often operate under their own mandates and do not provide routine reporting to the PES team.

This creates a significant attribution challenge for the Scottish Government in seeking to include this activity in evaluation. While these wider messengers are essential for reaching unengaged groups and providing local relevance, it is difficult to determine to what extent their activities are influenced by the PES itself. The Strategy’s role is primarily that of an enabler and facilitator: it provides the direction of travel, messaging, evidence, and potentially networks that messengers use to engage their own spheres of influence. Once again, contribution analysis could provide a suitable method to tackle this challenge, moving away from seeking direct cause-and-effect proof in favour of building a credible narrative of how the PES has influenced the broader ecosystem. This would involve new, systematic data collection on this trusted messenger activity to enable:

• Testing the “enabler” logic: Gathering evidence to see if trusted messengers are aware of the PES, if they have used its Understand, Participate, Act objectives to design their own engagement, and if the PES principles (e.g., being positive, inclusive, and evidence-based) have influenced their messaging.
  
• Accounting for external drivers: Recognising that an organisation like Zero Waste Scotland (ZWS) might deliver engagement that aligns with the PES due to its own statutory duties, rather than direct PES influence. The analysis would look for the added value the PES provided, such as increased message consistency or improved access to under-represented groups through PES-funded networks.
  
• Triangulating evidence: Combining quantitative indicators (e.g., toolkit downloads) with qualitative insights (e.g., stakeholder interviews) to assess whether the PES influenced how these messengers communicate. To include these wider messengers in a future round of evaluation, these additional data types and collection methods could be helpful:
  • Targeted partner surveys: A standardised survey for climate leads in major public bodies and other identified trusted messenger organisations to measure their awareness of the PES, their use of PES messaging, and the perceived value of the support provided by the Scottish Government.
    
  • Administrative uptake metrics: Monitoring the reach and utility of central PES resources, such as the number of Let’s Do Net Zero toolkit downloads by external organisations and the frequency of Climate Week hashtag use by non-government partners (already monitored and presented in MPR).
    
  • Alignment content analysis: A periodic review of communication materials from a sample of trusted messengers to assess the degree of alignment with PES principles and core messaging themes.
    
  • Social Network Analysis (SNA) could be interesting to explore, if resources allow.^[6]

These data points and methods could absolutely be included in an evaluation of the PES in the future. However, Scottish Government will need to decide the relative priority of monitoring and evaluating the impact of external versus internal, funded partners and trusted messengers. Data collection would need to be built in from the outset (HM Treasury, 2020).

Delivery partners facilitating PES programmes in local communities

Delivery partners play a central and distinctive role within the PES. They are not only implementers of funded activity, but also trusted intermediaries, innovators, and sense-makers operating at the interface between national ambition and local context. Any strengthened, unified approach to M&E must therefore be designed with the involvement of delivery partners. This section sets out how delivery partners can be meaningfully included in the appraisal, design and ongoing operation of PES M&E, while remaining proportionate, non-burdensome, and supportive of innovation.

The Magenta Book emphasises that effective evaluation begins with developing a shared understanding of how an intervention is intended to work, the context in which it operates, and the assumptions underpinning causal pathways. Delivery partners hold critical experiential knowledge of how engagement unfolds on the ground, how audiences interpret and respond to climate narratives, and how contextual factors such as place, trust, identity and institutional relationships shape outcomes. Involving delivery partners in the appraisal and evaluation design stage therefore serves two complementary purposes. First, it improves the quality and credibility of the PES Theory of Change by grounding it in lived delivery experience. Second, it builds shared ownership and legitimacy for the resulting evaluation framework, increasing the likelihood that it will be used for learning rather than perceived as a compliance exercise.

Participatory methods are particularly well suited to this stage. Approaches such as participatory systems mapping can be used to surface delivery partners’ perspectives on how different PES programmes interact with each other and with wider social, economic and policy systems. Qualitative, story-based approaches such as Most Significant Change (MSC) and outcome harvesting are well suited to engaging delivery partners in articulating outcomes and early impacts. Used at the appraisal and design stage, these methods can inform the refinement of the PES Theory of Change, test assumptions, and help shape evaluation questions that are meaningful to both Scottish Government and delivery partners.

One of the defining strengths of the PES is the degree of innovation enabled through its delivery model. Funds such as the Climate Engagement Fund, alongside place-based programmes like Climate Action Hubs, intentionally allow partners to experiment with formats, narratives, audiences and modes of participation. This flexibility is essential in a complex and evolving system, where no single engagement model will work everywhere or for everyone (see Learning and Innovation in Appendix E – Data and indicators for PES M&E). A unified M&E approach must therefore avoid constraining innovation through over-standardisation. Delivery partners should continue to have the freedom to adapt and innovate in response to local needs, emerging opportunities and changing contexts. Rather than measuring success solely against predefined outputs or outcomes, strategy-level M&E should seek to document what works, for whom, in what circumstances, and why. Case studies are particularly valuable in this regard, illustrating how innovative approaches emerge, how challenges are navigated, and how unintended outcomes arise.

Taken together, these elements point toward a model of evaluation that is relational rather than transactional. Delivery partners are not passive data providers, but active contributors to understanding how the PES functions as a system. Their involvement is concentrated where it adds most value—at the stages of appraisal, sense-making and learning—while routine monitoring remains proportionate and streamlined. Embedding this approach will require clarity of roles. Scottish Government retains responsibility for strategy-level evaluation design, synthesis and accountability. Delivery partners contribute contextual insight, experiential knowledge and reflective learning, primarily through existing channels and periodic participatory processes. Independent evaluators play a facilitative role, translating between levels, ensuring methodological robustness, and managing the balance between standardisation and flexibility. This partnership-based approach is well aligned with the mechanisms of the PES itself. Just as the Strategy recognises that government cannot engage the public on climate change alone, so too must evaluation recognise that understanding impact in complex systems requires multiple perspectives.

Telling the story of the PES to external stakeholders

When presenting the results of a contribution analysis approach to external audiences and decision makers, the narrative should be built around three pillars: credibility, relevance, and accountability. “In a complex policy situation, there may be a gap between what key stakeholders would like to know, and what it is possible to conclude from the evaluation” (Bicket, et al., 2020, p. 32). This is due to the properties of these complex systems, which limit the certainty of outcomes and make precise causal attribution difficult. Nonetheless, the contribution analysis approach provides a robust contribution narrative.

Presenting the results of evaluation will mirror the concise discussion in 9 Recommended approach: Applying contribution analysis to the PES at strategy level: Start with the attribution problem, explaining that the PES operates in a complex policy environment where experimental counterfactuals are infeasible. Contribution analysis is therefore the most appropriate tool because it offers a transparent, step‑by-step process for judging whether the strategy plausibly contributed to observed outcomes, rather than claiming definitive causal proof. Emphasise that the method moves the question from “Did the PES cause the change?” to “Is it reasonable to conclude that the PES made a difference?”

Briefly walk the audience through the six-step contribution analysis process, detailing how it builds on itself and is iterative, feeding back and repeating. Translate the analytic outcomes into decision‑relevant claims – Because contribution analysis does not deliver a single effect‑size estimate, the claims that can be made are credible, bounded statements about contribution. Be explicit about the limits of contribution analysis, explaining what the evaluation can and cannot claim:

• Can claim: The PES contributed to higher awareness, greater public input into policy, and the emergence of community projects. Examples:
  • “The PES contributed to a measurable increase in public awareness of Scotland’s net‑zero goals, as evidenced by the 75% campaign reach and corresponding survey‑based recall.”
    
  • “Community Climate Action Hubs have contributed to the emergence of locally led climate projects, supported by independent evaluations.”
    
  • “The Participation Programme has likely improved the inclusiveness of climate policy design, although additional demographic data are required to confirm reach to disadvantaged groups.”
• Cannot claim: Exact percentages of change solely attributable to the PES (“effect size”), because other factors (e.g., rising energy prices, global news) also influence its impacts.

Framing the claims as contributions rather than guarantees builds trust while remaining honest about uncertainty. All claims are anchored in the six‑step contribution analysis framework, backed by the Mid‑Point Review data, programme level evaluations, and the refreshed Theory of Change. By communicating both the strengths of the evidence and the uncertainties, the Scottish Government can demonstrate methodological rigour, build stakeholder confidence, and demonstrate the need for ongoing public engagement investment.

Layered into this can be more qualitative, contextual anecdotes from across PES delivery, to demonstrate the qualities of work happening across different programmes, including different scales and approaches to engagement. This storytelling aspect of the PES was highlighted as important at our workshop with PES stakeholders:

“I think one of the challenges we have is being able to tell the stories of the impact that we’re having effectively. […] I think it’s particularly the storytelling aspect that I think is something that we could explore more, and how we can encourage that to happen at a project level, as well as thinking about it at the programme level, so it’s a bit more consistent.” (Quote from workshop with Scottish Government)

Storytelling can be used as an exploratory appraisal and evaluation method:

“Storytelling encourages people to make sense of their own experiences, reflect on the array of contextual factors that influence outcomes, and articulate possible futures. Personal stories provide a human face to evaluation data and so can strengthen communication of key messages about a policy, including the impact on people and communities, their emotions and perspectives in ways that descriptions cannot” (Bicket, et al., 2020, p. 61).

This is a key function and reason to collect case study data and examples of innovative methods, even where these do not fit squarely into evaluation questions or cannot be meaningfully aggregated, because they provide the context, colour, and depth that allows external stakeholders to meaningfully engage with the results of evaluation:

“Maybe it’s useful to think about the different purposes of M&E. […] If you think about it as more of a kind of comms and engagement tool, that then comes out of the M&E, then having some case studies at project level is useful at the programme level, if that makes sense. It doesn’t have to be aggregated; it can stay project level.” (Quote from workshop with Scottish Government)

“A lot of these smaller pieces get lost […] when we come to try and do the big measurement picture. […] I think innovative metrics is necessary to be able to capture the different ways in which we achieve impact, because you can reach a high number of people without really reaching them, whereas you can reach a very small number of people and have a really lifetime impact. And trying to work out how we reflect that and measure that is quite a challenge.” (Quote from workshop with Scottish Government)

“The people who get involved in these processes, for them, they could have started off this type of engagement with zero interest, zero knowledge of anything that we’re talking about and that they leave it feeling that they’ve actually contributed something. I think that’s incredibly important and the reason why we do our job and it’s not necessarily that easy to evaluate.” (Quote from interview)

This is being embraced in England’s approach to public engagement on climate change, in which DESNZ flagged some specific organisations who are supporting them with storytelling:

“We are also learning from organisations like the Local Storytelling Exchange and Round Our Way that are making sure that stories about the impacts of climate change on our day to day lives, as well as the action that people are taking in their communities to address it, are part of the national discussion and news reporting. These stories can also highlight the benefits that can be realised, including for health and wellbeing and protecting the places that we care about.” (Department for Energy Security and Net Zero, 2025)

For more detail on storytelling as an evaluation method, see the methods, tools and techniques from Community Research (Community Research, 2025).

Conclusions

This report was commissioned to support the Scottish Government in strengthening how it assesses, evidences and communicates the impact of its Public Engagement Strategy on Climate Change (PES). The core challenge facing PES monitoring and evaluation (M&E) is demonstrating how a diverse, multi-stranded portfolio of engagement interventions collectively contributes to its strategic objectives of understanding, participation, and action. The PES operates across different scales, geographies, audiences, and modes of engagement, within a complex and dynamic policy environment. These features make robust evaluation both essential and inherently challenging. In this report we set out to address three related aims:

• to identify best-practice approaches for monitoring and evaluating complex public engagement strategies
  
• to understand how PES monitoring and evaluation currently operates in practice
  
• to provide practical, proportionate recommendations that can strengthen strategy-level M&E while remaining feasible and proportionate.

Each of these aims has been addressed through a combination of literature review, international comparison, document analysis, stakeholder interviews and a validation workshop with Scottish Government teams.

A central finding of the research is that Scotland’s PES is world-leading its ambition and unique in its structure. Few other governments have attempted to coordinate national communications, funded local delivery, deliberative participation, and trusted messenger networks under a single strategic framework on climate change. However, this same strength creates challenges for evaluation, because there is a lack of similar interventions to provide precedent or best practice.

In response, we returned to first principles of evaluation in complex systems, drawing on the established literature, including the Green and Magenta Books. We find that counterfactual and experimental approaches to evaluation (which focus on attribution, uniform indicators or aggregation of standardised metrics) are ill-suited to capturing the PES’s nuanced approach. Attempts to force comparability across programmes risk obscuring, rather than illuminating, how engagement works in different contexts. Theory-based approaches provide the most credible and feasible route to strategy-level evaluation for the PES. Specifically, contribution analysis (supported by a strengthened Theory of Change) is identified as an appropriate overarching framework. This approach allows diverse forms of qualitative and quantitative evidence to be brought together in a structured way, testing the plausibility of causal pathways while remaining sensitive to context, emergence and indirect effects.

A key implication of this shift is that diversity within the PES is not a weakness to be corrected through harmonisation, but a defining feature to be understood and analysed. Contribution analysis can accommodate different data types, scales and evaluation practices by focusing on how evidence supports or challenges the programme theory, rather than on whether indicators are directly comparable. This resolves a tension identified earlier in the strategy’s evaluation journey: the need for coherence without imposing uniformity.

The report also highlights the critical role of PES delivery partners and trusted messengers in making this approach work. Delivery partners are not simply sources of monitoring data, but key holders of knowledge about how engagement unfolds in practice, how innovation emerges, and how outcomes are shaped by local context. By involving them meaningfully in appraisal, Theory of Change development, and evaluation (using participatory methods such as systems mapping, Most Significant Change and outcome harvesting) the Scottish Government can both improve the quality of its evaluation and strengthen trust, legitimacy and shared ownership.

Across the report, we framed practical recommendations around proportionality and feasibility. The proposed unified M&E approach builds on what already exists: a strategy-level Theory of Change, programme-level monitoring systems, national datasets and a growing body of qualitative insight. Rather than introducing a new layer of reporting, it focuses on aligning these elements within a coherent evaluation framework that can support accountability, learning, and adaptation over time. In doing so, it responds directly to the central challenges of consistency, coherence and communicating cumulative impact.

The PES is particularly well-placed to make these improvements now. A helpful (though high-level) Theory of Change is already in place (Figure 7 in Appendix C); all component programmes have operated for more than two years; and the first full-term evaluation has yet to be conducted. This creates an optimal window for articulating clearer causal logic and designing an evaluation framework that will support both the remainder of this strategy cycle and the refreshed PES due in 2026. By strengthening the Theory of Change, adopting contribution analysis as a unifying framework, and working in partnership with delivery organisations, the Scottish Government can build a compelling evidence base for how public engagement contributes to Scotland’s transition to net zero. Crucially, this approach embraces evaluation not only as accountability, but evaluation as learning, ensuring that the PES continues to evolve, innovate and respond to the complex societal challenge it was designed to address.

References

Andrews, N., Elstub, S., McVean, S. & Sandie, G., 2022. Scotland’s Climate Assembly Research Report: process, impact and Assembly member experience. [Online]
Available at: https://www.gov.scot/publications/scotlands-climate-assembly-research-report-process-impact-assembly-member-experience/
[Accessed January 2026].

BetterEvaluation, 2025. BetterEvaluation. [Online]
Available at: https://www.betterevaluation.org/
[Accessed December 2025].

Bicket, M. et al., 2020. Magenta Book Supplementary Guide: Handling Complexity in Policy Evaluation. [Online]
Available at: https://assets.publishing.service.gov.uk/media/5e96c98ed3bf7f412d7f7bb0/Magenta_Book_supplementary_guide._Handling_Complexity_in_policy_evaluation.pdf
[Accessed December 2025].

Climate Assembly UK, 2020. The Path to Net Zero. [Online]
Available at: https://www.climateassembly.uk/recommendations/index.html
[Accessed January 2026].

Climate Outreach, 2020. Theory of change: creating a social mandate for climate action. [Online]
Available at: https://climateoutreach.org/reports/theory-of-change/
[Accessed January 2026].

Community Research, 2025. Storytelling: Methods, tools & techniques. [Online]
Available at: https://whatworks.org.nz/storytelling/
[Accessed January 2026 ].

Curato, N. et al., 2023. Global Assembly on the Climate and Ecological Crisis: Evaluation Report., s.l.: s.n.

Danish Board of Technology, 2025. Citizen Assembly on the climate area. [Online]
Available at: https://tekno.dk/project/danish-citizen-assembly-on-climate/?lang=en
[Accessed January 2026].

Department for Energy Security and Net Zero, 2025. Energising Britain: Your voice in our Clean Energy Superpower Mission. [Online]
Available at: https://www.gov.uk/government/publications/energising-britain-your-voice-in-our-clean-energy-superpower-mission/energising-britain-your-voice-in-our-clean-energy-superpower-mission-accessible-webpage#collaborate-to-inform-and-inspire-climate-and-nature-ac
[Accessed January 2026].

Energia, M. d. A. y., n.d. Consejo Consultivo Ciudadano de Cambio Climático. [Online]
Available at: https://cambioclimatico.go.cr/5c-consejo-consultivo-cambio-climatico/
[Accessed 08 01 2026].

European Commission, 2016. Evaluation Roadmap for the Evaluation of the EU Adaptation Strategy. [Online]
Available at: https://ec.europa.eu/smart-regulation/roadmaps/docs/2016_clima_011_evaluation_adaptation_strategy_en.pdf
[Accessed January 2026].

European Union, 2021. Forging a climate-resilient Europe – the new EU Strategy on Adaptation to Climate Change. [Online]
Available at: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2021:82:FIN
[Accessed January 2026].

Government Communications Service, 2024. GCS Evaluation Cycle. [Online]
Available at: https://www.communications.gov.uk/publications/gcs-evaluation-cycle/
[Accessed January 2026].

Government of Brazil, 2025. Brasil Participativo. [Online]
Available at: https://brasilparticipativo.presidencia.gov.br/
[Accessed January 2026].

Government of Canada, 2025. Government of Canada. [Online]
Available at: https://www.canada.ca/en.html
[Accessed January 2026].

Grantham Research Institute on Climate Change and the Environment, 2024. Submission to the United Arab Emirates Just Transition Work Programme topic for the second dialogue. [Online]
Available at: https://www4.unfccc.int/sites/SubmissionsStaging/Documents/202409041453—JTWP%20COP29%202nd%20Dialogue%20-%20GRI%20Submission%20%2804.09.2024%29.pdf
[Accessed January 2026].

HM Treasury, 2020. The Magenta Book: Central Government guidance on evaluation. [Online]
Available at: https://www.gov.uk/government/publications/the-magenta-book
[Accessed December 2025].

HM Treasury, 2022. The Green Book: Central Government Guidance on Appraisal and Evaluation. [Online]
Available at: https://assets.publishing.service.gov.uk/media/6645c709bd01f5ed32793cbc/Green_Book_2022__updated_links_.pdf
[Accessed December 2025].

IEA, 2021. Recommendations of the Global Commission on People-Centred Clean Energy Transitions. [Online]
Available at: https://www.iea.org/reports/recommendations-of-the-global-commission-on-people-centred-clean-energy-transitions/recommendation-10?utm_source=chatgpt.com
[Accessed January 2026].

Knowledge Network On Climate Assemblies, 2025. [Online]
Available at: https://www.knoca.eu/
[Accessed January 2026].

Legislation.gov.uk, 2009. Climate Change (Scotland) Act 2009. [Online]
Available at: https://www.legislation.gov.uk/asp/2009/12/section/91
[Accessed January 2026].

Legislation.gov.uk, 2019. Climate Change (Emissions Reduction Targets) (Scotland) Act 2019. [Online]
Available at: https://www.legislation.gov.uk/asp/2019/15/contents
[Accessed January 2026].

Lorenzoni, I., Jordan, A. J., Sullivan-Thomsett, C. & Geese, L., 2025. A review of National Citizens’ Climate Assemblies: learning from deliberative events. Climate Policy.

Mayne, J., 2008. Contribution analysis: An approach to exploring cause and effect. [Online]
Available at: https://www.betterevaluation.org/sites/default/files/ILAC_Brief16_Contribution_Analysis.pdf
[Accessed January 2026].

McNamara, S. & Foss, A., 2024. Theory of Change for Climate Action Wales: Public Engagement Strategy 2023-2026. [Online]
Available at: https://www.gov.wales/sites/default/files/statistics-and-research/2025-02/theory-of-change-for-climate-action-wales-public-engagement-strategy-2023-to-2026.pdf
[Accessed December 2025].

Millar, C. et al., 2024. A review of effective public engagement on climate and implications for Scotland. [Online]
Available at: https://www.climatexchange.org.uk/wp-content/uploads/2025/03/CXC-Effective-public-engagement-on-climate-and-implications-for-Scotland-November-2024.pdf
[Accessed January 2026].

OECD, 2010. Quality Standards for Development Evaluation. [Online]
Available at: https://www.oecd.org/en/publications/2010/02/quality-standards-for-development-evaluation_g1ghc6e7.html
[Accessed January 2026].

OECD, 2019. Better Criteria for Better Evaluation. [Online]
Available at: https://www.oecd.org/en/publications/2019/12/better-criteria-for-better-evaluation_f7a307eb.html
[Accessed January 2026].

OECD, 2025. Engaging and empowering citizens for the net-zero transition. [Online]
Available at: https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/04/engaging-and-empowering-citizens-for-the-net-zero-transition_6bd3d8e7/8d869640-en.pdf
[Accessed January 2026].

OECD, 2025. OECD Recommendation on Public Policy Evaluation Implementation Toolkit. [Online]
Available at: https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/02/implementation-toolkit-for-the-oecd-recommendation-on-public-policy-evaluation_f24516be/77faa4fe-en.pdf
[Accessed January 2026].

Robinson, M., Worker, J. & Grinspan, D., 2018. Public Participation in Open Government Reforms Advances Climate Action. [Online]
Available at: https://www.opengovpartnership.org/stories/public-participation-in-open-government-reforms-advances-climate-action/
[Accessed January 2026].

Scottish Government, 2020. Securing a green recovery on a path to net zero: climate change plan 2018–2032 – update. [Online]
Available at: https://www.gov.scot/publications/securing-green-recovery-path-net-zero-update-climate-change-plan-20182032/
[Accessed January 2026].

Scottish Government, 2021. Climate change – Net Zero Nation: public engagement strategy. [Online]
Available at: https://www.gov.scot/publications/net-zero-nation-public-engagement-strategy-climate-change/
[Accessed December 2025].

Scottish Government, 2021. https://www.gov.scot/publications/heat-buildings-strategy-achieving-net-zero-emissions-scotlands-buildings/. [Online]
Available at: https://www.gov.scot/publications/heat-buildings-strategy-achieving-net-zero-emissions-scotlands-buildings/
[Accessed January 2026].

Scottish Government, 2021. Just Transition – A Fairer, Greener Scotland: Scottish Government response. [Online]
Available at: https://www.gov.scot/publications/transition-fairer-greener-scotland/pages/5/
[Accessed January 2026].

Scottish Government, 2022. Scotland’s National Strategy for Economic Transformation. [Online]
Available at: https://www.gov.scot/publications/scotlands-national-strategy-economic-transformation/
[Accessed January 2026].

Scottish Government, 2023. National Planning Framework 4. [Online]
Available at: https://www.gov.scot/publications/national-planning-framework-4/
[Accessed January 2026].

Scottish Government, 2024. Climate change: Scottish National Adaptation Plan 2024-2029. [Online]
Available at: https://www.gov.scot/publications/scottish-national-adaptation-plan-2024-2029-2/
[Accessed January 2026].

Scottish Government, 2024. Independent Evaluation of Scottish Climate Action Hubs. [Online]
Available at: https://www.gov.scot/binaries/content/documents/govscot/publications/independent-report/2025/01/independent-evaluation-scottish-climate-action-hubs/documents/independent-evaluation-scottish-climate-action-hubs/independent-evaluation-scottish-climate-actio
[Accessed January 2026].

Scottish Government, 2024. Scottish Biodiversity Strategy to 2045. [Online]
Available at: https://www.gov.scot/publications/scottish-biodiversity-strategy-2045/
[Accessed January 2026].

Scottish Government, 2025. Climate Change – Public Engagement Strategy: Mid-Point Review. [Online]
Available at: https://www.gov.scot/publications/public-engagement-strategy-climate-change-mid-point-review/
[Accessed December 2025].

Scottish Government, 2025. Environment Strategy Overview. [Online]
Available at: https://data.gov.scot/environment/environmentstrategy.html
[Accessed January 2026].

Scottish Government, 2025. Scottish Climate Survey 2024: Main findings. [Online]
Available at: https://www.gov.scot/binaries/content/documents/govscot/publications/statistics/2025/04/scottish-climate-survey-main-findings/documents/scottish-climate-survey-2024-main-findings/scottish-climate-survey-2024-main-findings/govscot%3Adocument/Scottish%2BCli
[Accessed January 2026].

Scottish Government, n.d. Climate Action Hubs – Updated Theory of Change, s.l.: Unpublished.

Sette, C., 2013. 52 weeks of BetterEvaluation: Week 8: Using social network analysis for M&E. [Online]
Available at: https://www.betterevaluation.org/blog/52-weeks-betterevaluation-week-8-using-social-network-analysis-for-me
[Accessed January 2026].

Smithers, R. et al., 2018. Study to support the evaluation of the EU Adaptation Strategy. [Online]
Available at: https://climate.ec.europa.eu/document/download/2697d445-b408-4bc0-b4f2-21fc3f96c254_en?filename=adapt_strat_eval_report_en.pdf
[Accessed December 2025].

UK Government, 2019. Regulatory Policy Committee – Case Histories. [Online]
Available at: https://assets.publishing.service.gov.uk/media/6836fcc19411f0341f3236cc/ME_guidance.pdf
[Accessed January 2026].

UNDP, 2021. UNDP Evaluation Guidelines. [Online]
Available at: https://erc.undp.org/pdf/UNDP_Evaluation_Guidelines.pdf
[Accessed January 2026].

UNFCCC, 2025. Action for Climate Empowerment. [Online]
Available at: https://unfccc.int/topics/education-and-youth/big-picture/ACE#International-cooperation
[Accessed January 2026].

Welsh Government, 2023. Climate Action Wales – Public Engagement Strategy 2023-26. [Online]
Available at: https://www.gov.wales/sites/default/files/publications/2023-07/climate-action-wales-public-engagement-strategy-2023-to-2026.pdf
[Accessed January 2026].

Appendices

Appendix A – Full project methodology

Literature review

The literature review was designed to identify best-practice approaches for monitoring and evaluating complex, multi-stranded public engagement strategies. Its purpose was to connect these insights back to the PES ecosystem and to determine what is both feasible and proportionate within the Scottish context. To achieve this, the review adopted a targeted and pragmatic approach, identifying 17 authoritative sources of M&E guidance from the UK and internationally which were selected from a broader review as being the most relevant and applicable to Scotland’s PES. These included core government evaluation frameworks such as the UK Magenta Book, the Green Book and OECD evaluation standards; the evaluation of the EU Climate Adaptation Strategy; the Theory of Change underpinning the Wales Public Engagement Strategy; and research on building a social mandate for climate action produced by Climate Outreach.

Findings from each source were systematically extracted into a structured data matrix and organised according to relevant thematic fields. The evidence was then synthesised across nine sub-research questions, grouped under three overarching themes: understanding the purpose of strategy-level evaluation (“the why”), defining outcomes and impacts (“the what”), and designing appropriate monitoring and evaluation methods (“the how”).

The findings from this stage of work were central to identifying best practice for monitoring and evaluating public engagement strategies, and they directly inform our recommendations on how to operationalise the Theory of Change as a practical evaluation tool.

Document analysis

This document analysis methodology was designed to systematically review the existing monitoring and evaluation (M&E) landscape of the Public Engagement Strategy (PES) to identify current data collection practices, assess their alignment with established best practices, and directly inform evidence-based recommendations for strengthening the PES evaluation framework.

The analysis began by defining the scope of the PES M&E data landscape. The primary sources reviewed included a comprehensive set of documents detailing programme-level M&E practices:

• M&E templates, return forms, and project reports provided by PES programmes and projects. These sources detail how project-level data is collected and communicated up to the strategy level.
  
• The Net Zero Nation Public Engagement Strategy for Climate Change Mid-Point Review (MPR), which serves as the most recent formal evaluation and provides high-level aggregation of KPIs and findings across the PES portfolio.

This review established a record of the current M&E practices and capability. The core of the methodology involved a structured classification of the data points and reporting mechanisms found in the source materials against two established evaluation frameworks. Notes were systematically recorded for each project to document the types of data and indicators collected, aligning them with the stages of the Government Communication Service (GCS) Evaluation Cycle, which establishes a clear taxonomy for assessing communications efforts. Indicators and data being collected were mapped to the different stages of the GCS Cycle: Inputs, Outputs, Outtakes, Outcomes, and Impacts. This process aimed to map current data collection against the inherent ‘programme theory’ of the PES, which captures inputs, outputs, outcomes, and impact.

By comparing observed M&E practices against established evaluation criteria, the analysis identified current data flows, highlighted existing data gaps (such as missing impact measurements or difficulties in aggregation across varying scales), and determined the feasibility and sensibility of collecting specific data types at different levels. This structured review directly informs the subsequent phase of the project: refreshing the evaluation framework and providing concrete guidance on applying the right metrics to future PES programmes. The results of this step of the research are detailed in Appendix E – Data and indicators for PES M&E.

Interviews with key stakeholders

To provide depth and lived experience of delivering climate change engagement activities to the evaluation of the PES, a qualitative data collection approach was utilised through semi-structured interviews. This methodology was designed to gather practical insights into M&E application, system strengthening opportunities, and lessons learned regarding tensions between strategic design and practical delivery.

A total of fifteen participants were interviewed, strategically selected to represent different functional levels of the PES. The interview design and specific questions, which were drawn from three tailored guides (for project, programme, and strategy levels), were discussed and agreed with the project steering group (see Appendix B – Interview guide example). Interviews were conducted virtually via MS Teams and lasted between 30-60 minutes. Ethical integrity was maintained by ensuring all participants signed information and consent forms prior to the interview, confirming their understanding of how the data would be managed. All sessions were recorded and subsequently transcribed for detailed analysis.

The analysis of the transcripts focused on pulling out key findings and capturing anecdotes and lessons learned to support the overall research aims. The resulting qualitative evidence is layered into the final report to enhance interpretation and provide narrative depth.

Workshop with Scottish Government teams

The final activity of this research was a stakeholder workshop with Scottish Government teams, which delivered crucial validation and operationalisation for the research findings. This methodology aligns with evaluation best practice by involving key audiences to ensure the utility and adoptability of the final recommendations. The session maintained the same rigorous ethics protocol (including informed consent and confidentiality) established for the preceding qualitative interviews.

The central objective was to refine six ideas for M&E improvement that emerged directly from initial delivery partner interviews. The session focused intently on the practical challenges of implementation, addressing the need for more consistency in data gathering and reporting identified by the Mid-Point Review. The core methodology involved two key activities:

• Metric definition: Discussion focused on the specific metrics needed to effectively capture the impacts achieved by delivery partners, extending beyond simple outputs to address measurable behavioural change and complex outcomes.
  
• Practicality mapping: Participants prioritised the proposed M&E improvements (and their corresponding M&E metrics) by mapping them onto an effort vs. impact matrix. This exercise gave the research team insights into the feasibility and perceived effectiveness of recommendations, ensuring they were pragmatic for Scottish Government implementation.

The rich context and concrete ideas generated during the workshop directly inform the final report sections, contributing to both practical recommendations and strategic context.

Appendix B – Interview guide example

This was the interview guide for project-level delivery partners (i.e. individual Climate Engagement Fund projects, Climate Action Hubs, etc.)

Interviewer to introduce themselves and the aims and outcomes of the evaluation.

Interviewer to confirm that participant has read, understood, and signed project information and consent form.

Ask for permission and start recording.

1. Role & Link to PES

• Can you describe your project and your role in delivering it?
• The [insert relevant programme, depending on Understand/Participate/Act pillars] is linked to the Public Engagement Strategy (PES). How aware are you of the PES, and do you feel your project connects to its pillars (Understand, Participate, Act)?
• The PES has seven engagement principles — were these clear when you applied, and how do you check your project follows them?
• Did you refer to PES documents (like the Theory of Change) when shaping your application?

2. Monitoring & Evaluation Practices

• What kind of monitoring and evaluation do you currently carry out for your project?
• What data do you collect (e.g. attendance, training numbers, surveys, stories, behaviour change, audience reach)?
• How do you know you’re reaching people who were previously unengaged?
• Did you establish any kind of baseline for your staff, students, or community participants? If not, what made this difficult?
• How did you set your milestones, outputs, and outcomes when applying for [insert name of programme / fund]? Did SG provide guidance?

3. Use of Findings

• How do you use your monitoring and evaluation findings within your project team? Do they help adapt your activities?
• Do you report these findings to SG, or to your community or partners? If so, how?
• Do you feel the reporting templates capture the real value of your work, or are some impacts invisible in the data?

4. Challenges & Gaps

• What barriers or challenges have you faced in monitoring and evaluation (e.g. time, resources, relevance of indicators)?
• Are there aspects of your work (like shifts in confidence, trust, or community attitudes) that are especially hard to measure?
• Climate engagement can lead to different kinds of change (understanding, motivation, reduced distress). Which of these outcomes do you see, and how do you try to measure them?
• Did the short funding period or reporting requirements make it harder to evaluate properly?

5. Suggestions for Improvement

• What would make monitoring and evaluation easier or more useful for your team? Is there any need for additional guidance, shared templates, or peer learning help?
• If you could change one thing about the reporting process, what would it be?
• Do you think your project has built capacity that will last beyond the funding period (e.g. staff, students, community capability)?

Appendix C – Using the Theory of Change as an evaluation tool

The research team understands that the existing PES Theory of Change (ToC) was developed primarily as a communication tool, giving an accessible overview of the strategy and its overall vision (see Figure 7). Rather than spelling out the precise causal mechanisms underlying the programme theory, the current ToC gives a streamlined, one-page picture of the PES’s key objectives. Some delivery partners have found this to be useful:

“We’ve been paying attention to the Public Engagement Strategy and […] trying to reorganise the way we report to government. We always keep in mind the overarching objectives the government wants to achieve through the PES and the 2030 target, and we ask ourselves how the work we are doing contributes to those aims.” (Quote from interview with delivery partner)

It is sensible to retain an updated version of the current one-page ToC overview for this purpose, making clear that it is an overview and signposting to a full ToC.

However, based on best practice, a key first step to developing a more robust M&E system for the PES is to develop that more complete ToC. A key objective of this work is to explore ways to better evidence how PES programmes and projects act cumulatively to contribute towards the Strategy’s objectives and vision. By articulating the underlying assumptions, mechanisms, and contextual factors that govern change, a ToC allows policy teams and evaluators to identify KPIs and what the data needs are to evidence progress. This need has been echoed by Scottish Government:

“This kind of goes back to the whole Theory of Change and trying to make the Theory of Change a bit more granular, so that you can say, “Why do we want partnership working?” And then it will break it down into, “Ok, we want some partnership working to facilitate this cross-programme learning,” right? […] So it’s that’s kind of where that more detailed Theory of Change I think will help draw out some of these issues. It’s just too broad of a thing, at the moment.” (Quote from workshop with Scottish Government)

The Climate Action Hubs have recently had a new M&E approach developed for them, which began with an overhaul of the Hubs ToC:

“Our first phase of redesigning the M&E approach for the Hubs was to look at the Theory of Change. So, with that, we were looking at the longer-term outcomes and agreeing those and working through them in workshops with the Hubs. What we’re doing now is, we’ve commissioned an evaluation of the Hub programme, so that’s underway, it will report in March next year, and that is directly looking at impacts and is speaking to people who have been involved in Hub activities.” (Quote from workshop with Scottish Government)

This supports the idea that evaluation starts with a robust ToC, developed in partnership with stakeholders, such that everyone agrees on the baseline against which they are being measured. Evaluation then follows, to put evidence behind the key points of the programme theory. The synthesis below addresses the key elements of a useful ToC, details what this looks like for a communications strategy, provides a critique of the current PES ToC, and lays out a guide for policy colleagues to refresh and update the framework.

A useful ToC differentiates itself by its ability to align large-scale, multi-part programmes for cumulative success. This usefulness is defined by:

• Utility for all stakeholders: The ToC can offer a practical tool for both the commissioning body (e.g., Scottish Government) and the delivery partners, enabling everyone to see clearly how their specific work supports the overall collective goals.
  
• Supporting cumulative assessment: A useful ToC maps the thinking behind how multiple, diverse activities collectively contribute to outcomes, providing a foundation for cumulative assessment necessary for complex portfolios where different component programmes may use varying measures to define success.
  
• Focusing activities and identifying gaps: By clarifying assumptions and cataloguing evidence, a fleshed-out ToC helps policy colleagues focus activities (“What does this programme do, specifically?”) and identify where the current strategy leaves gaps (“Which parts of our Theory of Change are not covered?”).
  
• Testing realism: Exploring why an intervention might fail (i.e. the “negative programme theory”) strengthens the ToC by testing assumptions and improving realism and risk identification (HM Treasury, 2020, p. 26).

For an engagement strategy like the PES, the Theory of Change can show how inputs lead directly to changes in audience knowledge, attitudes, and (where relevant) behaviour:

• Inputs/Activities: The resources and specific communication and engagement actions (e.g., campaigns, training, partnerships).
  
• Outputs: The direct results of the activities (e.g., number of people reached, materials produced, toolkit downloads).
  
• Outtakes: What the audience thinks, feels, or intends to do after exposure (e.g., awareness, attitude shift, sentiment, motivation).^[7]
  
• Outcomes: The desired changes or behavioural response (e.g., changes to social norms, understanding, or attitudes; starting, stopping, or maintaining a specific climate action; actively participating in policy design).
  
• Impact: The achievement of the long-term policy goal (e.g., delivering on Just Transition commitments).
  
• Learning and Innovation: Capturing how lessons learned through both delivery and evaluation are fed back into the Inputs stage and integrated into the intervention.
  

The PES has an underlying ToC structured around three high-level objectives—Understand, Participate, and Act—which aligns all initiatives toward an overarching vision. However, the Mid-Point Review and best practice guidelines suggest areas where the ToC requires strengthening to be a more rigorous alignment and evaluation tool:

• Causal specificity and assumptions: A robust ToC must detail precise causal mechanisms (the precise chain of events and steps that link the inputs and activities of the programme to the desired outcomes) and assumptions (the explicit conditions, beliefs, or circumstances that must hold true for each causal link to succeed). While mechanisms and assumptions are implicit in the current version and likely guide implementation, these are not sufficiently clarified or explicitly catalogued with supporting evidence.
  
• Alignment and cumulative assessment: A useful ToC enables cumulative assessment across all programmes. The complexity of the PES means activities have “different measures of success which are difficult to aggregate or compare across” (Scottish Government, 2025, p. 10). This difficulty suggests the high-level ToC needs refinement to standardise the flow of data up to the overarching indicators.
  
• Activity gaps: A ToC should help identify gaps in coverage. These could be made clear, for example, Millar et al. (2024) suggested that certain effective engagement approaches like creative activities and the use of humour are noted as underrepresented in the current PES and require greater focus going forward (p.19). A refreshed ToC would provide an opportunity to highlight this gap indicate plans to fill it.
  
• Contextual factors and alternative pathways: A robust ToC should also include an explanation of how contextual, behavioural, and organisational factors influence whether the outcomes occur, as well as any alternative mechanisms that could potentially lead to the same outcomes (HM Treasury, 2020, p. 24).
  

UK Government Communication Service (GCS) Evaluation Cycle

The GCS Evaluation Cycle is the Government Communication Service’s (GCS) practical adaptation of evaluation principles to the specific context of public communication. Whereas the Magenta Book provides the theoretical and methodological foundation, the GCS framework is designed for the operational realities of communication campaigns, where diverse audiences, fragmented delivery models and behaviour-change goals require more iterative, adaptive and real-time approaches to evidence. Although the PES is broader than a single communications campaign, its delivery strands aim to inform, motivate and empower the public to understand, participate in and act on climate change, objectives closely aligned with the GCS model’s focus on audience insight, behavioural barriers and the sequencing from awareness to action. The GCS Cycle embeds evaluation from the outset of a campaign, enabling communicators to integrate behavioural science tools into planning and make ongoing, evidence-informed adjustments throughout delivery.

Together, the Green Book, Magenta Book and GCS Cycle provide a coherent framework that supports learning, strengthens accountability and enables continuous improvement in the design and delivery of public engagement. These frameworks are complementary and mutually reinforcing. The GCS explicitly recommends that colleagues utilise the Magenta Book alongside the Evaluation Cycle to gain understanding of evaluation methods for different scenarios, data collection and analysis approaches (both quantitative and qualitative), and guidance on sharing evaluation results to inform better communications. Collectively, they provide the authoritative UK Government guidance on evaluation design and practice.

Given that the PES is a government-led communications initiative aimed at driving public understanding, participation, and normalising action on climate change, these frameworks are important sources for assessing best practice in evaluating the PES. The PES itself is grounded in evidence-based principles and explicitly incorporates a Theory of Change aligned with these frameworks. Using the Magenta Book and GCS Evaluation Cycle as reference standards allows for a systematic assessment of whether the PES evaluation approach adheres to current government best practice and identifies where alignment could be strengthened.

In the main body report, we have supplemented guidance from the Green & Magenta books with more specific material from the GCS. Specifically, this report adds “Outtakes” from the GCS, which is an intermediate category which is relevant to interventions which include public engagement. It also draws upon the metrics used to evidence different stages of the GCS cycle as examples. The GCS adds communications- and engagement-specific guidance to the more general best practice for evaluations found in the Green and Magenta Books.

A guide for reviewing and refreshing the PES Theory of Change

Developing a comprehensive, strategy-level ToC is an opportunity to clarify how PES inputs and outputs (including those deployed at strategy level as well as at programme level) cohere to deliver the strategy’s aims and objectives. It should show the thinking behind how the PES’s programmes are targeted towards addressing different parts of its objectives and how these are necessary to deliver in different contexts (i.e. across different scales, geographies, demographics, etc.). It should bring this together in one place without duplicating existing programme-level strategies. Clarifying the causal mechanisms and assumptions behind the individual programmes will directly identify the appropriate data and metrics required to evidence these (see next section, How does the Theory of Change become the evaluation framework?).

To align more closely with the Magenta Book guidance, the PES going forward could adopt a structured, iterative process to review and refresh the PES Theory of Change for the next strategy refresh (due 2026). Key steps to ensure the requirements of a robust ToC are met are:

1. Recategorise existing ToC actions / outcomes / impacts to align with Magenta Book categories: The “Indicative Actions” as well as the Outcomes and Impacts of the current ToC could be recategorised to align with the categories used in Theory of Change guidance. Clarifying how the ongoing PES activities align with the definition those key steps is an important first stage to building a more robust programme theory. Clarifying the difference between inputs / outputs etc. is crucial to selecting relevant metrics to measure each. For example, “Promote and maintain Net Zero Nation website” becomes an Input, aligning most closely with the GCS Input, “Content creation.” The research team has created a first draft of this recategorisation in Figure 8. This figure serves as a practical tool and template on which Scottish Government policy teams can build.
   
2. Add the full scope of Inputs, Outputs, Outtakes, Outcomes, and Impacts: Building on the template provided in Figure 8, add entries to the ToC to make it a complete picture of what the strategy aims to deliver and how. Prioritise strategy-level inputs / outputs etc. (see lists below).
   
3. Break down / subdivide some existing entries (e.g. “Deliver Just Transition Commitments,”) to make clear which of these relate to the PES and how it addresses them. Consider rephrasing these to make them fit more squarely into GCS stages.
   
4. Map causal mechanisms: For the PES at strategy level, explicitly map the predicted causal pathway from Input to Impact through the GCS Evaluation Cycle stages (Outputs, Outtakes, Outcomes). This structure ensures all programme elements cohere under the high-level PES objectives. Some suggestions or examples of this are also provided in the main body report in Recommended approach: Applying contribution analysis to the PES at strategy level.
   
5. Clarify and catalogue assumptions: Systematically list assumptions underlying each causal link. It is best practice to be clear regarding how the programme assumes that each (input / activity / output) will lead / contribute to each outcome / impact.^[8] Going further, it is best to link these assumptions to supporting theory or evidence where possible, i.e. how assumptions are supported by research or that approaches have been successful in the past or in similar contexts.^[9]
   

Specific methods which Scottish Government policy teams could use with PES delivery partners and trusted messengers to capture their understanding of how the PES is delivering impact on the ground and then articulate through a Theory of Change include participatory systems mapping, Most Significant Change, or outcome harvesting (Bicket, et al., 2020). This will address the identified need for the Scottish Government to be able to more clearly articulate how and why different climate engagement activities, such as those set out in the PES, can work together to deliver intended outcomes.

In addition, Scottish Government policy teams should consider how to integrate or “nest” the work of the PES’s component programmes within the strategy-level ToC. Many of the PES’s inputs and outputs are delivered by its component programmes – For example, the Climate Action Hubs have a very robust ToC of their own (Scottish Government, n.d.) .^[10] Scottish Government policy teams should discuss how best to include PES programmes within a strategy-level ToC in a way that is useful and appropriate to strategy-level evaluation. Ideally, programme-level logic models or ToCs should be “nested” within the PES ToC, showing what inputs are received from strategy level and how each programme delivers outputs and outcomes relevant to the strategy overall. The Climate Action Hubs ToC gives an example of how to do this, clearly delineating “Individual Hub Outcomes” and “Hub Network Outcomes.” Including nested programmes in this way would ensure that the strategy-level ToC is a complete picture of the PES, clarifying how programmes support the strategy’s outcomes and impacts, and identifying any gaps in delivery.

An improved ToC would also provide a place to capture those inputs, outputs, etc. which are delivered at strategy level, which provide the infrastructure, framework, and coordinating mechanisms within which individual programmes operate. For example:

Strategy-level Inputs

• Framework and governance development: Development and publication of the PES itself, including the Theory of Change; Creation of governance structures and stakeholder networks, such as the Climate Change Communications Stakeholder Group and Climate Policy Engagement Network (CPEN).
  
• Evidence-based strategy planning: Commissioning and analysing the independent evidence review on effective public engagement (Millar, et al., 2024), which informs how the strategy should evolve; Consultation exercises on the strategy itself (e.g., the 2020 consultation that received 178 responses and shaped the current PES) (Scottish Government, 2021, p. 10)
  
• Monitoring and evaluation architecture: Design of the multi-stranded monitoring and evaluation approach (national indicators, programme evaluations, MPR, full-term evaluation); Commissioning of independent evaluations and research (e.g., MPR, Millar et al. report, and this project)

Strategy-level Outputs

• Strategic documents and reports: Publication of the PES strategy document itself, MPR, full-term report, and publicising updates to the three national indicators from the Scottish Household Survey
  
• Data integration and synthesis: Analysis and synthesis of monitoring data across all programmes to assess cumulative progress toward the strategic objectives

Strategy-level Outtakes

• Organisational learning: Documented lessons on what works well and what needs improvement (e.g., the effectiveness of community-led action), e.g., stakeholder survey (67 responses from 280 organisations) to understand delivery partners’ experiences and perceptions of the PES (Scottish Government, 2025, p. 9); Identification of gaps in the current approach (e.g., underrepresentation of creative activities, need for stronger feedback loops on policy influence); Decisions to strengthen particular elements (e.g., increasing emphasis on trusted messengers, building in stronger communication of how public participation influences policy outcomes)

Strategy-level Outcomes

• Organisational adaptation: Incorporation of learning from the MPR and other reports into the refreshed PES (due 2026); Improved monitoring and evaluation framework for the next strategy period
  
• Instances of demonstrable policy action resulting from participation: Integration of participation insights into specific policy decisions and plans (e.g., Transport Just Transition Plan informed by Participation Programme outputs); Clear communication back to participants about how their input was used

Capturing strategy-level Inputs, Outputs, Outtakes, and Outcomes in the ToC is essential because it makes visible the coordination and learning infrastructure that transforms disparate programme activities into cumulative impact. Without these, Scottish Government policy teams cannot demonstrate how the full portfolio of PES activities coherently delivers against the overarching net zero vision—they can only report on individual programme success. Including and clarifying these strategy-level activities is useful for all PES stakeholders: It shows Scottish Government policy teams, delivery partners, and members of the public how their individual contributions connect.

The above recommendations represent a clear guide to refreshing the PES ToC and bringing it in line with best practice for the purposes of M&E for government interventions. Thinking longer-term, Scottish Government policy teams could consider going further to integrate the following:

• Ensure partner utility and stakeholder involvement: Ensure the ToC serves as a functional tool across the organisation and its partners. The ToC should be developed and reviewed in collaboration with a wide range of stakeholders, including policy designers, implementers (e.g., Climate Action Hubs, Climate Engagement Fund recipients), and beneficiaries. This ensures delivery partners have ownership over their contribution to the overall goals.
  
• Identify and incorporate gaps: Use the revised model to identify activities not currently covered (e.g., greater focus on creative activities and humour). Catalogue evidence learned from previous communications activities—including what did not work—to build upon prior performance.
  
• Explore the negative programme theory: Conduct a formal exercise to explore why and how the intervention might fail, thereby strengthening the ToC by testing realism.

The refined ToC should guide the development and implementation of the refreshed PES due in 2026, ensuring continuous improvement and institutionalising best practice.

Practical tools and templates: Theory of Change

The following resources provide a comprehensive toolkit for refreshing the PES ToC:

The Green Book and Magenta Book provide general guidance and technical background. The relevance of these websites to the PES is highlighted in the sections above. The Magenta Book also includes a comprehensive list of further resources, reading, and additional templates (HM Treasury, 2020, p. 26).

The template presented in Figure 8 synthesises the guidance of the Green Book, Magenta Book, and GCS to align the existing PES ToC with best practices from these sources. This adaptation provides a framework / “bones” to which Scottish Government policy teams can add detail, including the full suite of strategy-level inputs, outputs, etc., casual mechanisms, underlying assumptions, linked evidence, and how PES programmes “nest” and fit in.

The Climate Action Hubs example provides a robust, worked template for an effective ToC. The PES ToC could be modelled on this and should ensure that strategy- and programme-level programme theories are aligned and resolved, underpinned by the same frameworks and consistent language (Scottish Government, n.d.)

How does the Theory of Change become the evaluation framework?

The Theory of Change (ToC) is the intervention’s causal model; it articulates how the communication is theoretically expected to work. It maps out the assumed causal chain: inputs (resources and activities) are expected to generate outputs (audience reach and exposure), which are predicted to shift audience beliefs and feelings (outtakes), ultimately resulting in behaviour change (outcomes) and policy/organisational impact. Evaluation activities should then align with the GCS six-stage cycle (Figure 9).

1. Inputs/Activities: Evidence of planning and using learnings (from past delivery, based on audience insight, or findings from other previous interventions) to shape content and delivery, and setting SMART objectives linked to policy goals.
   
2. Outputs: Measuring successful delivery and audience exposure, such as reach, impressions, and distribution.
   
3. Outtakes: Capturing immediate audience reception, perceptions, attitudes, and intentions.
   
4. Outcomes: Measuring the actual changes or behavioural responses (e.g., changes to social norms, understanding, or attitudes; starting, stopping, or maintaining a specific behaviour; actively participating in policy design).
   
5. Impacts: Linking outcomes to the achievement of high-level policy and organisational goals, often including value-for-money assessments.
   
6. Learning and Innovation: Capturing strategic lessons that feed back into future planning cycles (back into Inputs stage).
   

The GCS six stages are the evaluation framework—they define how and what to measure at each point in the causal chain to test whether the ToC held true in practice. The two interact as follows:

• Inputs stage: The ToC is developed during the planning phase (the Inputs stage). The GCS explicitly states that “it may be beneficial to set out your assumptions and predictions for how your different audience segments will experience your communication activity (‘Outputs’) and how this is predicted to change their respective beliefs or feelings (‘Outtakes’), and ultimately lead to the desired audience behaviour (‘Outcomes’) in a Theory of Change.” (Government Communication Service, 2024, p. 33)
  
• Testing the theory: Collecting data (using a template / evaluation framework) on stages 2–5 (Outputs, Outtakes, Outcomes, and Impacts) directly measures the causal chain predicted in the ToC. If the ToC predicted that outputs would lead to outtakes, the GCS recommends collecting data on both and assess whether that relationship materialised in practice.
  
• Learning loop: The Learning & Innovation stage (stage 6) feeds insights back to refine the ToC for future iterations.
  

The ToC forms the conceptual foundation for evaluation by articulating the intervention’s logic (“the what”), while the GCS six-stage cycle serves as the methodology or framework (“the how”) to systematically test whether that logic holds true in practice.

The next step is to formalise the steps of the GCS into a fit-for-purpose evaluation framework that standardises data collection across PES programmes (e.g., Climate Action Hubs, Climate Engagement Fund, Climate Action Schools, and campaigns like ‘Let’s Do Net Zero’) to allow for cumulative assessment of the entire programme’s impact. The following steps outline how Scottish Government policy teams can build a fit-for-purpose evaluation framework for the PES, flowing directly from the updated PES ToC:

• Construct the evaluation framework: This should flow directly from the ToC, providing a space to capture relevant data to summarise progress against delivering the inputs and outputs and evidence of outtakes, outcomes, and impacts achieved. The PES requires a core data collection tool that uses mandatory indicators collected from programmes and mapped across the GCS stages. Indicator data could include quantitative and qualitative metrics for each GCS stage. Discussion and recommendations on relevant indicators and data are given in Appendix E – Data and indicators for PES M&E.
  
• Coherence, nesting, and streamlining: Replicating the “nesting” of programmes within the strategy-level ToC, the PES evaluation framework should show clearly where and how data collected from programmes feeds into reporting at strategy level. The evaluation framework should operationalise and evidence the theory behind how programmes relate to and deliver the overall strategy.
  
• Standardisation: To enable this nesting and streamlining, monitoring forms should be standardised across PES programmes, to the degree possible, to allow for easier cumulative assessment. The eventual framework should be designed to be simple, cross-compatible, and allow for easy linking of data to facilitate aggregation and potential autofill functions (e.g., in Excel) between different programme reports.

Refreshing and building out the PES ToC is the more complex task – A fit-for-purpose evaluation framework should be a shorter leap from there, simply collating the correct types of metrics to evidence each stage of the ToC.

Practical tools and templates: Evaluation framework

The following sources provide a comprehensive toolkit to implement an evaluation framework aligned with best practice:

The Government Communication Service (GCS) Evaluation Cycle provides clear guidance and definitions of each of its six stages, along with relevant metrics commonly used to measure each. It shows why measuring each stage matters and what can be claimed based on collecting that information. It also clarifies the theory behind a recurring, iterative evaluation cycle, showing how the stages feed into each other and back into Inputs, enabling learning and adaptation.

The Climate Action Hubs Revised Theory of Change and Revised Monitoring Template are robust tools which should be used as key sources and templates for building both the PES ToC and corresponding evaluation framework. The Hubs’ version shows the thinking behind how their activities are meant to address strategy level goals, delineates the difference between programme- and Hub-level outcomes, and their evaluation framework shows how each data collection step links back to the ToC. The Hubs template offers a ready-made structure for tracking Outputs, Outtakes (e.g., improved knowledge, concern, skills), and Outcomes (e.g., funding secured by community groups). The PES ToC and evaluation framework should be aligned with that of the Hubs such that data is cross-compatible they and can nest/integrate into each other effectively to allow for cumulative assessment by the Scottish Government.

Appendix D – Case studies

Case study 1: Lessons from EU Adaptation Strategy on Climate Change

The EU Adaptation Strategy for Climate Change provides a valuable reference point for the PES as an example of how monitoring and evaluation can be structured for a complex, multi-stranded climate change strategy operating across multiple governance levels. The formal evaluation of the EU Strategy, conducted between December 2016 and March 2018, assessed its implementation and performance since its launch in 2013. Covering all Member States, the evaluation applied five core criteria (Relevance, Effectiveness, Efficiency, Coherence, and EU Added Value) and drew on the original Impact Assessment to establish a 2013 baseline, planned inputs and activities, outputs, expected results, and operational objectives (European Commission, 2016). Key learnings for this research included:

• The absence of a clear counterfactual scenario was identified as a major methodological challenge. This led evaluators to adopt a theory-based evaluation approach rather than impact attribution in a strict causal sense.
  
• The evaluation was structured around a clear intervention logic, linking the need for policy action to objectives, actions, and anticipated impacts, and using this logic to guide assessment across the EU Strategy.
  
• The assessment was organised around a set of evaluation questions aligned with the five Better Regulation criteria. These questions were further translated into operational sub-questions tailored to different action areas and stakeholder groups.
  
• Evidence was drawn from multiple complementary sources, including literature review, targeted stakeholder surveys, interviews, workshops, open public consultation, and case studies. Evidence was triangulated to strengthen the robustness of findings.

Relevant and transferrable lessons from this example have been carried through to the recommendations for the PES in this research. As the Impact Assessment did not project a counterfactual scenario (which would have outlined how conditions would have evolved in the absence of the Strategy) and given the complexity of evaluating a multi-level intervention, the EU evaluation necessarily adopted a theory-based approach. Evaluation for the PES has similar starting conditions. The EU approach examined the Theory of Change embedded in the Strategy’s intervention logic, assessing progress for each objective and associated action. The current state of play was compared against the expected impacts and operational objectives defined in the Impact Assessment, which served as the principal points of comparison. Where suitable data were available, impacts were assessed quantitatively; in other cases, qualitative judgement was applied. To support robust conclusions, evidence from multiple sources was triangulated, with any inconsistencies identified and explicitly addressed (European Union, 2021). The recommendations of this report mirror this approach, identifying contribution analysis as a method which can combine multiple data sources to evidence a coherent contribution story.

Case study 2: Lessons from Wales PES

Wales is at an early stage of delivery but has taken a clear step towards establishing a structured evaluation architecture for its Public Engagement Strategy. The Strategy is underpinned by a detailed Theory of Change (ToC), developed through participatory workshops, which articulates how activities are expected to contribute to intermediate outcomes and longer-term impacts, and clarifies key assumptions and dependencies across four “green choice” themes (Welsh Government, 2023, p. 14). The ToC is structured around the COM-B behaviour change model, which assumes that behaviour change occurs only where Capability, Opportunity, and Motivation are present, and explicitly links public engagement activities to these components to set out the Strategy’s causal pathways. While a full evaluation framework has not yet been finalised, Wales has identified a range of measurable outputs and outlined how outcome measurement will draw on existing national datasets, including climate perceptions, behavioural tracking, and national survey data (McNamara & Foss, 2024). Key learnings from the Wales PES include:

• The ToC should be treated as a living framework and updated as the Strategy evolves, particularly as engagement activities become more clearly defined and as the scope of behaviours, including adaptation behaviours, expands over time.
• There is a risk that, without partners being fully aware of the Strategy and their role within it, activities may continue as disconnected strands of work rather than as a coherent public engagement strategy. This creates challenges both for attribution of outcomes and for the Strategy achieving its intended impact.
• Measuring outputs and outcomes is challenging due to the breadth and diversity of partners involved. Future evaluation should therefore consider how to assess the effectiveness of partner contributions by identifying what has changed as a result of the Strategy.
• The Wales ToC explicitly identifies assumptions, risks, and dependencies at each stage of the model. These provide a valuable foundation for shaping evaluation questions and identifying areas of focus when assessing the extent to which the Strategy has achieved its aims.

The Welsh ToC report reinforces the value of contribution analysis as a theory-based method for Scotland’s PES, that asks whether it is reasonable to conclude the strategy contributed to the desired outcomes, a fit for a policy where many parallel drivers exist:

“Recommendation 1: Use contribution analysis when evaluating. The Strategy will not directly lead to the COM-B outcomes or impacts described in the Theory of Change but aims to increase them. Future evaluation should therefore take a theory-based approach, using contribution methods (HM Treasury 2020). Rather than asking the questions such as, ‘did the Strategy lead to a Just Transition to net zero?’, contribution evaluation will assess questions such as, ‘is it reasonable to conclude the Strategy contributed to a Just Transition to net zero.’” (McNamara & Foss, 2024, p. 59)

Appendix E – Data and indicators for PES M&E

Evaluating a multi-faceted national program like the PES requires a data collection approach that is both robust and pragmatic. The selection of data types should be guided by the principle of proportionality, which ensures that evaluation efforts are fit-for-purpose, credible, and tailored to the questions being asked, the resources available, and the context of the intervention (HM Treasury, 2020, pp. 16-17). Within contribution analysis, the function of data collection is to gather sufficient evidence to test the causal pathways outlined in the Theory of Change (ToC) and build a credible narrative of the strategy’s influence.

The core data needed to achieve this must provide a triangulated view of performance, combining quantitative and qualitative sources to track progress from inputs to outcomes. These essential data types are generally cost-effective and can often be integrated into existing program management and reporting cycles.

• Core quantitative data: This includes administrative and monitoring data that tracks Inputs & Activities (e.g., total spend, number of events organised), and Outputs (e.g., estimated total reach, website traffic, volume of press coverage). It also includes survey data, which is crucial for measuring outtakes and outcomes not captured elsewhere, such as changes in public awareness, attitudes, and stated intentions to act. Existing sources like the Scottish Household Survey provide a baseline for national indicators.
  
• Core qualitative data: This includes stakeholder interviews, progress report narratives, and feedback forms. This data is essential for the process evaluation component, providing the contextual “why” behind the quantitative figures, exploring what worked well or less well, for whom, and how delivery could be improved.

The following sections of the report present more specific metrics and data types which are relevant for evidencing the different stages of the updated PES ToC. It also signposts to where these are being collected across PES programmes and highlights any gaps. Putting data behind the reorganised ToC, addressing these gaps, and streamlining data collection and sharing will strengthen PES M&E.

To gain deeper insights into complex or long-term impacts, this core dataset can be supplemented with more resource-intensive methods, layered on strategically. These data types, while more costly and time-consuming to collect, are vital for understanding contribution and capturing the nuanced, meaningful changes that quantitative metrics may miss. This includes in-depth case studies to explore participant journeys, deliberative methods like focus groups to understand community attitudes, or quasi-experimental designs where feasible to strengthen causal claims (HM Treasury, 2020, pp. 42-43). The decision to employ these methods should be proportionate to the evaluation’s needs; they may be reserved for pilot projects, specific high-priority research questions, or the final full-term evaluation where a deeper understanding of “what works” is required to inform future strategy. Options to integrate these and other methods are discussed under Evidence for Impacts, below.

The following sections are ordered based on a generic Theory of Change: Inputs/Activities, Outputs, Outtakes, Outcomes, Impacts, and Learning and Innovation. For each section/stage, its definition from the GCS is given. This is followed by the recategorised parts of the PES ToC to align with this definition (this replicates what is presented in Figure 8, so it may be helpful have that updated ToC to hand to follow along/read across). Then, a list of the relevant metrics which are recommended for measuring each stage is presented. This is followed by an overview of where these metrics and data types can be found across the range of PES programmes, any noteworthy gaps, and other discussion points as relevant.

Evidence for Inputs and Activities

“Inputs are what you put in at the start – the planning and research that informs your communications or campaign. This includes everything that must be done to prepare for the communication activity, which may include conducting research, reviewing previous learnings, planning evaluation design, determining budget and costs, etc. Wherever possible, planning should be based on research evidence, insights and learnings to maximise your chance of delivering a successful campaign or communication activity. Successful communication activities and campaigns rely upon sufficient preparation” (Government Communication Service, 2024, p. 12).

Collecting data on inputs and activities establishes the foundation of every engagement effort, showing what resources, research, and planning work were invested before any outcomes appear. Without that baseline, it is impossible to link observed changes in awareness, attitudes or actions to specific programmes, to calculate cost‑effectiveness, or to demonstrate that the strategy is using evidence‑based approaches. Transparent input data also lets policymakers identify gaps, replicate successful practices and hold the programme accountable for delivering value for public money across the whole PES.

Recategorised Inputs and Activities from draft updated ToC:

• The Scottish Government and Scotland’s key communicators deliver a consistent and impactful message about our climate change national endeavour
  
• Promote and maintain Net Zero website
  
• Ensure accessibility
  
• Promote positive messaging
  
• Apply lessons from Scotland’s Climate Assembly

Metrics for Inputs and Activities:

• Total spend to date (online and/or offline)
  
• Theory of Change (including evidence base) – Implementation of behavioural science in planning effective communication (Is it in place? Is it evidence-based?)
  
• Content creation (volume by type)
  
• Volume of press and/or social media releases (# of releases sent out)
  
• Evidence of implementing insights from research or learnings from previous evaluations (Is it in place? What did it inform?)

The MPR aggregates several metrics for Inputs/Activities, but the level of detail varies across the PES programmes. Budget information for the large‑scale programmes is provided, for example, the £9.5m allocated to Community Climate Action Hubs and the £800k cumulative spend on the Climate Engagement Fund (p.2), with further annual figures for Climate Week in Annex A (p.38). Narrative sections describe the creation of communication assets for the “Let’s Do Net Zero” campaign, website updates and tool‑kits, which together constitute the content‑creation evidence, although no consolidated count is provided (p.12). Social media reach is reported for specific initiatives, giving a partial view of the volume of press/social media releases. The MPR is clearly evidence-based and lists the research that has informed the strategy (e.g., external research, People’s Panel, stakeholder surveys, p.8). It also notes that the overall Theory of Change was tested, but it does not present a programme-by-programme Theory of Change, or a systematic log of how earlier evaluations were fed into design. References to behavioural‑science‑informed messaging appear in the discussion of motivational framing, yet there is no dedicated metric confirming its systematic application.

Gaps – Metrics absent or only partially recorded:

• Total spend to date (online and/or offline): Detailed spend is only available for a few large programmes (Hubs, Climate Engagement Fund, Climate Week). Most other initiatives – Climate Action Schools, Climate Action Towns, the Participation Programme and the Climate Assembly – have no aggregated financial input data in the MPR.
  
• Theory of Change (including evidence base): The MPR mentions the overarching ToC but does not provide a separate, documented Theory of Change for each programme, nor does it link activities to a specific evidence base.
  
• Content creation (volume): Asset creation is described qualitatively (campaign materials, tool‑kits) but no quantitative tally of pieces produced (e.g., number of videos, PDFs, lesson plans) is recorded for any programme.
  
• Volume of press and/or social‑media releases: only ad‑hoc social‑media metrics (hashtag use – see Evidence for Outtakes, below) are cited; no systematic count of press releases or social‑media posts per programme.
  
• Evidence of implementing insights from research or previous evaluations: the Review acknowledges use of external research but does not capture, per programme, which specific insights were incorporated or how they altered design.

Evidence for Outputs

“Outputs are objective measurements of what is delivered and how your audience encounters and interacts with your communications through reach, distribution and exposure. This captures how successfully your communication activity has reached your target audience, which may include press coverage, public relations and impressions, as well as low/no-cost activities such as stakeholder engagement” (Government Communication Service, 2024, p. 17).

Recategorised Outputs from draft updated ToC:

• There are many opportunities for people to have their views heard on Scotland’s response to climate change and to hear the perspectives of other people in Scotland
  
• Deliver public awareness campaigns
  
• Communicate climate impacts
  
• Facilitate climate conversations
  
• Build strong stakeholder partnerships

Metrics for Outputs:

• Estimated total reach (absolute number and proportion of target audience; offline / online)
  
• Direct contacts (# of direct online / offline contacts)^[11]
  
• Events organised (# of events, # of attendees, feedback from attendees)
  
• Volume of coverage (# of exposures)
  
• Partnerships secured (# partners providing amplifying support, promoting material or any form of in-kind or financial contribution)

The MPR contains data that maps directly onto the GCS output fields, but the information is spread across a series of narrative sections and annex tables rather than being presented as a single consolidated dashboard. For example, the Let’s Do Net Zero evaluation cited in the MPR provides the absolute reach for the COP‑26‑related communications: 75% of adults in Scotland, 3.5 million people saw the campaign, with an average exposure of 9.7 times. The same section also notes that the Net Zero Nation website receives 80‑100k visits per year, providing a baseline for ongoing online reach (pp.13-14). The Climate Engagement Fund section records that the Fund engaged 8,000 people in 2023‑24 and 7,000 people by Q3 2024‑25 (p.17). Annex A on Climate Week (p. 38) supplies the GCS metrics of “estimated total reach” (website visits, attendees), “direct contacts” (partner activity), “events organised” (12 events across 8 Ministerial portfolios), “volume of coverage” (3,588 hashtag mentions, although these could double as an Outtakes metric) and “partnerships” (number of partners, partner asset downloads). 

The Climate Engagement Fund Quarterly Grant Progress Report contains a dedicated “Audience Reach” grid that records total, in‑person and online reach for each grant and maps them against the three strategic objectives (Understand, Participate, Act). The Climate Action Schools Impact Report supplies its own reach and event figures: 27,500 pupils taught directly, 5,500 young people attending in‑person events and 1,160 schools participating – which are cross‑referenced in the MPR narrative as illustrative evidence of school‑level outputs (p.19). The Climate Assembly research report contributes the media‑coverage count (151 articles across four phases) and the number of direct engagement moments (106 assembly members, Ministerial Q&A) (Andrews, et al., 2022, p. 124). Finally, the Climate Action Hubs Revised Monitoring Template links the Output indicators (events organised, participants, partnership counts, coverage) to the hub‑level data collection worksheets. By pulling the figures from these sections and the annexed programme monitoring tables, Scottish Government can compile a strategy‑level dataset for the five GCS output metrics.

Gaps – Metrics absent or only partially recorded:

• Estimated total reach (proportion of target audience): Most programmes report absolute numbers (e.g., Schools, Hubs, Let’s Do Net Zero) but do not calculate the proportion of the defined Scottish audience that was reached (see 9 Recommended approach: Applying contribution analysis to the PES at strategy level.
  
• Direct contacts: a distinct “direct contact” count is missing; Contact is embedded in reach totals but not isolated as a separate metric (e.g., Climate Engagement Fund only captures audience reach, not a tally of unique contacts).
  
• Volume of coverage (media exposures): Only the Climate Assembly report provides a media‑exposure count; other programmes (Schools, Hubs, Climate Engagement Fund, Climate Week) lack a dedicated field for press cuts, broadcasts or digital impressions.
  
• Partnerships secured (amplifying support): Partnership activity is described qualitatively (e.g., “collaboration with community groups” in the Hubs template) but no programme reports a quantified count of partners that provide amplification or in‑kind support.
  
• Consistent offline vs. online split: While some reports separate online and in‑person reach (Climate Engagement Fund, Climate Week), others (Schools, Assembly) give only aggregated totals, preventing a clean offline/online comparison across the portfolio.

Measuring Outputs such as total reach, direct contacts, events, media coverage and partnership counts give the Scottish Government a factual baseline of what the PES actually delivered, turning narrative claims into verifiable numbers. Because the Theory of Change will link these delivery metrics to downstream Outtakes, Outcomes and Impact, a robust Outputs dataset lets PES teams demonstrate the scale of exposure and engagement that underpins any observed changes in attitudes, actions, or policy influence.

Evidence for Outtakes

“The audience perception – what they think, feel or intend to do as a result of your communication activities. Outtakes capture the reception, perception, intentions and reaction of your target audience to your communication activity. Outtakes are distinct from outcomes: while outtakes focus on audience beliefs, attitudes and feelings, outcomes focus on actual changes in behaviours.

Outtake metrics measure how your communication activity impacts your target audience’s awareness, understanding, attitudes, emotions, or intentions. Comparing Outtakes with the targets set in objectives enables you to understand which messaging has been effective for engaging your audience or different audience segments” (Government Communication Service, 2024, p. 19).

Recategorised Outtakes from draft updated ToC:

• People understand the causes and consequences of climate change and the need for ambitious mitigation and adaptation action
  
• People understand the actions that they are encouraged to take and how to take them
  
• People are aware of the action that all of Scotland is taking to tackle climate change and understand how it relates to their lives

Metrics for Outtakes:

• Sentiment (attitudes & emotions) (percent and/or case studies; assessment of press coverage)
  
• Online engagement: Click-through/view-through rates, Dwell time, Bounce rate, Passive engagements (shares/likes/retweets), Active engagements (comments/responses/hashtag use)
  
• Prompted/Unprompted campaign recognisers/awareness (# and % target audience)
  
• The extent to which different groups agree/disagree with messages related to the Theory of Change (5-point scale)
  
• Understanding and Agreement: Assessment of audience’s level of understanding of, and agreement with, campaign content (5-point scale)
  
• Stated or intended behaviour change: Proportion of target audience that claims they will act in accordance with the campaign aim (absolute number and proportion of target audience)
  
• Response rate (percent of contacts who respond)
  
• Attitudinal change: The degree to which people’s attitudes have changed in favour of the campaign (5-point scale)

The MPR pulls the evidence for Outtake metrics from several sources that are already collected across the Public Engagement Strategy (PES): National attitude and awareness trends are reported from the Scottish Household Survey (SHS) and the Scottish Climate Survey – these provide the “understanding & agreement” and “awareness” baselines used to track whether people know the causes of climate change and the actions they should take (p.2). The commercial “Let’s Do Net Zero” (LDNZ) campaign research provides comprehensive data on online engagement and supplies the core Outtake figures for campaign recognisers, including understanding of actions and having taken action (p.13).

Outcome tables for the Climate Engagement Fund projects list the number of people informed about Scotland’s climate programme (6,000 in 2023‑24) and give qualitative statements of increased understanding (p.17). The Climate Assembly research report provides the member‑level survey data that capture sentiment, attitudinal change, understanding and stated behavioural intention for the Assembly‑wide audience (Andrews, et al., 2022, p. 92). Program‑specific qualitative evidence (e.g., Climate Action Schools testimonials, Climate Action Hubs monitoring template, Climate Action Hubs evaluation) is referenced in the MPR as illustrative examples of audience perception and confidence (e.g., p.19).

Gaps – Metrics absent or only partially recorded:

• No standard 5‑point “understanding & agreement” or attitudinal‑change scale in many programme reports (Climate Action Schools, Climate Engagement Fund template, Climate Action Hubs monitoring).
  
• Stated or intended behaviour‑change figures are not routinely collected outside a few isolated surveys (LDNZ, Assembly).
  
• Sentiment/emotional‑response measurement is only anecdotal (e.g., educator quotes) and lacks quantitative scoring across programmes.

Evidence for Outcomes

“Outcomes are the response from your target audience in terms of changes in behaviour or active engagement, i.e., registrations to a website/service, adoption of positive habits, cessation of unwanted practices to comply with new laws/regulations, etc. It captures whether your audience’s feelings and motivations (“Outtakes”) really translate to actual behavioural change (“Outcomes”).

Outcomes determine how your communication activity influenced behaviour change and contributed to the policy objectives, i.e., whether your campaign encouraged your target audience to start doing something, stop doing something or maintain behaviour. Outcomes enable links to be made between any Outtakes (changes in beliefs and attitudes) and the resulting desired change in behaviour. Remember, even if your communication is primarily aimed at changing beliefs/ feelings (“Outtakes”), rather than behaviours (“Outcomes”), you should still consider whether (and if so, how) audience behaviour might change in response to your communications” (Government Communication Service, 2024, p. 22).

Recategorised Outcomes from draft updated ToC:

• Social norms are changed through increased local and community engagement and action on climate change
  
• People actively participate in shaping just, fair and inclusive policies that promote mitigation of and adaptation to climate change
  
• Taking action on climate change is normalised and encouraged in households, communities and places across Scotland
  
• Consultation process enhanced^[12]

Metrics for Outcomes:

• Behaviour change: Maintain, stop, or start a specific action; Quantifiable actions taken
  
• Completion/registration date (e.g., sign-up to a service)
  
• Decreased barriers to change behaviour as targeted by the campaign
  
• Decreased barriers to behaviour change that were not targeted by your campaign or cannot be changed through communications
  
• Capacity building, evidence of increased skills and confidence
  
• (All measured by recording absolute number and proportion of target audience, via survey/case study/other feedback)

The PES’s approach to behaviour change and implications for M&E

The PES it clear that it primarily targets understanding and attitudes, rather than acting as a direct driver of individual behaviour change. The strategy’s narrative stresses a “holistic, systemic approach […] to build a strong social mandate for the society‑wide, long‑term changes needed to transition Scotland to a net‑zero nation” and acknowledges that “behaviour change at the individual and household level is a key element […] but we cannot rely on nudging people towards one or two low‑carbon behaviours” (Scottish Government, 2021, p. 13). Within the programme theory for the PES, behaviour change is understood to be a subset of broader public engagement on climate change, rather than a directly measurable objective for the PES.

A critical principle drawn from GCS guidance is that not all outcomes are relevant or measurable for every campaign:

Detail from GCS on “Evaluating different types of communications”

“Awareness and attitudes: In the minority of cases, communications might primarily seek to raise awareness of an issue or to change people’s attitudes, with no behaviour changes anticipated, meaning measurement of Outcomes may not always be possible. However, even where the communication activity does not have a primary behavioural element, considerations should be made regarding how the communication objectives of increasing awareness or changing attitudes feed into overarching behavioural and policy objectives.

Behaviour Change: Most government communication seeks to change behaviours to implement government policy or improve society. This means that, in addition to tracking awareness metrics, evaluations should capture whether your target audience adopt the desired behaviour change, according to the purpose of “start”, “stop” or “maintain”. This is so that we can learn which methods, messages and channels are effective for encouraging successful behaviour change” (Government Communication Service, 2024, p. 10).

For the purposes of using the GCS metrics and definitions, the research team understands the PES as an awareness‑and attitude‑focused strategy. Drawing on best practice, PES evaluation should therefore concentrate on the Outtakes (knowledge, beliefs, motivation) that underpin the longer‑term cultural shift the strategy seeks, while still recognising that any downstream Outcomes will be mediated through those Outtakes. This is consistent with a contribution analysis approach, focusing on evidencing the causal links and making improvements within the programme’s sphere of influence, supporting the contribution story regarding downstream Impacts.

Practical tools and resources: Linking the PES to behaviour change (where relevant/desired)

Scottish Government policy teams should continue to make clear the programme theory behind the PES, including whether and how it is designed to enable and/or influence behaviour change. This could be included in the updated PES ToC, although it could be made clear that the PES does not directly target behaviour change, instead signposting to other programmes addressing that need. If there is a desire to connect the PES to behaviour change, the following resources provide a fit-for-purpose toolkit:

Awareness and attitude shifts (Outtakes) do not exist in isolation; they feed into the behavioural chain described by the COM‑B model (Capability, Opportunity, Motivation) (UK Government 2024 p.35). In the GCS framework, the Motivation component is directly linked to the outtakes that the PES is designed to influence. When motivation is raised, the COM‑B model predicts that the likelihood of Capability (skills, knowledge) and Opportunity (access to low‑carbon services, supportive policy environments) will determine whether a behaviour change occurs. The PES strategy document complements this with the ISM (Individual Social Material) model, a similar typology which considers the factors which influence behaviour and behaviour change (Scottish Government 2013). By mapping the PES ToC onto COM‑B + ISM, evaluators can make explicit where additional data (e.g., measures of opportunity such as infrastructure availability) are needed to explain gaps between attitudes and outcomes.

Integrating these insights into future PES evaluations can be done through a tiered design. Scottish Government policy teams could consider investing in further process evaluation components such as targeted case studies that trace the COM‑B pathway from motivation to capability and opportunity. The added value of this is a richer explanatory narrative that can inform policy refinements and demonstrate the systemic cultural shift the strategy promises. The main challenges are the resource demands of primary data collection, the need for inter‑programme data alignment, and the risk of over‑extending attribution claims when behavioural outcomes are measured without a robust counterfactual. By applying the proportionality principle—matching data intensity to evaluation questions and programme risk—the PES can maintain methodological rigour while remaining cost‑effective and focused on its core mandate of building awareness, shifting attitudes and, ultimately, fostering the societal conditions for lasting climate action.

Current reporting and potential lessons for Outcomes within the PES

In terms of reporting on Outcomes, the MPR describes the results from the Climate Action Hubs programme (improved community‑level mitigation and adaptation, confidence‑building and capacity‑building outcomes, p.32), and from the Climate Action Towns case‑studies that show climate actions embedded in local place plans (e.g., the Blackburn Future Plan, p.30). The Let’s Do Net Zero communications evaluation provides the recogniser‑to‑action link (e.g., 80% in 2024 reported having taken an action after seeing the campaign, p.13). The Climate Engagement Fund quarterly reports note that 8,000 people across eight projects reported taking climate‑action steps and give examples of practical workshops (e.g., historic‑building sustainability, p.17).

The MPR cites the Scottish Household Survey and the Scottish Climate Survey as the source of the “understanding & agreement” and “awareness” baselines that underpin the GCS metrics for sentiment, attitudinal change and stated/intended behaviour. The Climate Assembly research report lists the most popular climate actions for members, e.g., reductions in meat/dairy consumption, energy use (Andrews, et al., 2022, p. 16).

Gaps – Metrics absent or only partially recorded:

• Behaviour‑change counts (maintain/stop/start) are not captured in a standardised way for several major programmes (Climate Action Schools, Climate Assembly, Participation Programme) – the MPR notes only anecdotal evidence.
  
• Decreased‑barrier metrics are missing – the Climate Action Hubs monitoring template has no dedicated indicator for “reduction of targeted or non‑targeted barriers”, and the Climate Engagement Fund quarterly template does not request barrier‑reduction data.
  
• Quantifiable practical‑action tallies are fragmented – case‑studies (e.g., 89 bags of foodbank donations, 22,000 litter picks, Keep Scotland Beautiful 2025) exist, but there is no aggregated count across all projects; the MPR only cites project‑level totals (8,000 people taking, or encouraged to take actions via the Climate Engagement Fund, p.17) without a common metric.
  
• Capacity‑building evidence is largely qualitative – several programmes record knowledge gains or testimonials but lack a uniform 5‑point confidence or skill‑score that can be summed across the portfolio.

Evidence for Impacts

“Impact draws links between inputs, outputs, outtakes, and outcomes to determine how your communication activity has contributed to or impacted policy and organisational objectives. Impact is the comparison of actual outcome data with KPIs and objectives set to measure whether these were met. Organisational objectives are distinct from the policy objectives and include longer-term or wider considerations such as return on investment, revenue, cost reduction, compliance, retention, recruitment, positive contributions to physical/ mental health, environmental impact, etc.

Impact allows you to demonstrate whether your objectives were delivered. For paid-for campaigns you should also demonstrate whether your KPIs were met and if they were sufficient to deliver the targets set in your objectives. You should also consider how your communications activity contributed to broader organisational impacts” (Government Communication Service, 2024, p. 24).

Recategorised Impacts from draft updated ToC:

• Implement the updated Climate Change Plan
  
• Deliver Just Transition Commitments (including empowerment, community building, public support, equity and inclusion)
  
• Deliver COP26 engagement and legacy
  
• The Scottish Government listens to, acknowledges, and responds to people participating in policy design
  

Metrics for Impacts:

• Contribution to policy and/or organisational objectives (absolute number and proportion of targets met)
  
• Long-term compliance or retention (absolute number and proportion of targets met over time)
  
• Cost-per-person metrics (e.g., cost-per-person achieving awareness, behaviour change, etc.).
  
• Current return on investment (unit benefit multiplied by number of behaviour changes achieved)
  
• Evidence of outcomes being embedded in formal plans and/or policies

Impacts of participation: Feedback loops

The PES ToC includes participation in climate policy as one of its goals: “There are many opportunities for people to have their views heard on Scotland’s response to climate change and to hear the perspectives of other people in Scotland” (Outputs); “People actively participate in shaping just, fair and inclusive policies that promote mitigation of and adaptation to climate change” and “Consultation process enhanced” (Outcomes); “The Scottish Government listens to, acknowledges, and responds to people participating in policy design” (Impacts).

Demonstrating that public inputs are heard, acted upon and reported back is essential to prevent participation from being perceived as tokenistic and to preserve trust in government: the Scottish Climate Survey shows that 39% of respondents listed Scottish Government as a trusted source, (Scottish Government, 2025, p. 14), but that this trust could erode if people cannot see the impact of their involvement. The MPR identifies a key weakness in this regard: the current lack of ability to track how public engagement outcomes influence Scottish Government climate change policy, signalling a break in the causal chain that undermines the Participate pillar (Scottish Government, 2025, p. 22).^[13] Evidence from the Climate Assembly research report reinforces this point, showing that participant confidence declined when they could not see how their recommendations were taken forward, underlining the trust-building role of effective feedback (Andrews, et al., 2022, p. 15). The workshop with Scottish Government teams highlighted the challenges of establishing feedback loops:

“The challenge we have with feedback loops, specifically around things like participation and influence on policy, is being able to track that internally, and then being able to provide those feedback loops. I don’t know that measurement is our issue there, because I think that’s about policy measurement. It’s much harder to do that, from a PES perspective. […] I think we’ve gone almost as far as we can in terms of feedback loops now, and we do have a couple of systems. I mean, it’s more reflection loops than feedback loops in that we try and, on the back of a participative exercise, through the partner we’ve been working with, [provide] reflections based on the report and say, ‘Thank you for this input. This is how that’s going to feed into the process. These are some of the highlights that we really liked, really took from the report.’ And we’ve found partner feedback from that has been that that is incredibly useful. And it doesn’t go as far as […] really reflecting the impact on decisions. But it’s probably as far as we can go in terms of feedback loops on policy impact for participation, because it is a really funny, complex, sushi process that no one can really track.” (Quote from workshop with Scottish Government)

Both the Magenta Book and the Green Book warn that isolating how engagement activity cause policy changes is rarely possible; evaluations must therefore focus on contribution, using a ToC to trace plausible pathways, as discussed in the previous section. Despite these challenges, there are good examples in the Mid-Point Review (MPR) of closing the loop: the Participation Programme’s delivery tables show how evidence feeds into Just Transition Planning and the Transport Just Transition Plan (p.15), while Climate Policy Engagement Network documents contributions to SNAP3 and other Just Transition Plans (p.25). Evaluation frameworks such as the Independent Evaluation of the Climate Action Hubs further institutionalise this feedback function through explicit measures—such as “Output 8: Supporting policy input and decision making”—designed to capture evidence of influence on policy (Scottish Government, 2024, p. 42).

In summary, a robust feedback loop, systematically documenting how public input is taken up and communicating the response back to participants, where possible, is essential to maintain trust and demonstrate the value of public engagement. Future PES evaluations should continue to capture this data where possible. The recommendations in this report related to strengthening the ToC will help in joining up these efforts across PES stages to demonstrate a coherent approach within the PES to support impact on climate change policies. The OECD Citizen Participation Barometer could be a useful tool in the future, when available.

Value-for-money and return on investment

Value-for-money is one of the three main purposes of evaluation, alongside process and impact evaluations. Value-for- is of key interest for PES stakeholders, as evidenced by this interview quote:

“For me, the most important is, ‘Are we doing the right thing? Are we making any difference? Is this a good use of public money? Are we investing in the right programmes? Is it making any kind of a difference? Should we actually use this this money for retrofitting houses?’” (Quote from interview)

The main difficulty in calculating a clear value‑for‑money or return on investment for the PES is that the strategy is deliberately designed to generate understanding and attitudinal change rather than immediate, quantifiable behavioural outcomes. Government Communication Service guidance stipulates that return-on-investment objectives should be focused on measurable behavioural results (e.g., numbers of investments generated, or teachers recruited) and that impact analysis must move through a five‑step process of objectives, baseline, trend, isolation and externalities (Government Communication Service, 2024, p. 37). Because the PES treats behaviour change as a subset of broader public engagement and not a direct, measurable target, the Outcomes required for a traditional return on investment calculation are often absent. Instead, the programme is expected to deliver “Outtakes” such as knowledge, beliefs and motivation. Similar to the limitations on available national scale data discussed in 9.1 Proportionality: How can contribution analysis address the challenges of PES M&E in a feasible way?, it is difficult to establish a baseline and a trend for the desired Outcomes and to separate the PES’s contribution from other policy instruments, so attribution of any observed impact to the communication activity alone is uncertain.

In addition, the PES’s long‑term, systemic ambition to build a strong social mandate for net‑zero creates wide‑ranging externalities (positive cultural shifts, equity gains, indirect health benefits) that are difficult to monetise or link directly to the initial investment. Assumptions required to translate attitudinal shifts into monetary returns are therefore more speculative, and the need to track unintended consequences further adds uncertainty to the return-on-investment calculation. Consequently, standard GCS return on investment methodology—relying on concrete, behaviour‑based KPIs—does not align cleanly with the PES’s intended impact pathway, limiting the ability to demonstrate conventional value-for- money- or return on investment.

Learning and Innovation

“Formal learning forms the final stage of an evaluation cycle. At this stage, you would evaluate the effectiveness of inputs at each stage of the evaluation cycle to understand the impact on policy/organisation objectives and see what did or did not work. Formal learning can be used to drive innovation in the future. It would be useful to consider what approaches could be employed in future activities to overcome difficulties and leverage strengths in your approach. Where applicable, it is also useful to capture what innovations were applied in your communication activities and why they did or did not work.

For paid-for campaigns, this stage is where learnings from your 10% innovation investment should be reported in your evaluation. It is an opportunity to showcase innovations that could be “scaled up” across other communication activities but also to report other learnings (i.e., what didn’t work). The aim of this stage is not to justify new approaches but rather to inform future considerations.

If objectives or KPIs were not met, what reasons can be identified to explain the variation? If the objectives were surpassed, what has driven that? This identifies strategic insights and learnings that can be taken forward and shared.

What are they used for? This is so that, now or in the future, communications activities can capitalise on successful communication techniques and avoid embedding unsuccessful methods. Learnings that can be applied or scaled more generally should also be shared GCS-wide, across teams, and across government organisations” (Government Communication Service, 2024, p. 26).

Recategorised “Supporting activities to help bring about the changes (assumptions) required” from draft updated ToC:

• Support climate change education
  
• Involve children and young people
  
• Encourage community climate action
  
• Engage through culture and heritage
  
• Promote a place-based approach
  
• Support trusted messengers
  
• Encourage connections with nature

For the PES, learning and innovation should be understood not as a separate, occasional activity but as a continuous, project-driven feedback loop. This aligns closely with the GCS Evaluation Cycle, which positions “Learning & Innovation” as the final stage that closes the loop by analysing what worked, what did not, and feeding those insights directly into the next cycle of planning and Inputs (Government Communication Service, 2024, p. 8). While the PES sets the overarching strategic aims for understanding, participation and collective action, it is the programmes and projects that translate these ambitions into practical engagement with people on the ground. These are the spaces where experimentation happens: where different formats, audiences, and supporting activities (e.g., trusted messengers, place-based approaches, culture and heritage, community climate action, etc.) are tested in real contexts. Evaluating and learning from this project-level innovation is therefore the primary route through which the PES can evolve, refine its Theory of Change, and adapt to Scotland’s diverse social and geographic settings.

Public engagement initiatives are highly context-dependent: trust dynamics, local networks and culturally relevant messengers can determine whether an engagement mechanism succeeds or fails (Climate Outreach, 2020). Engagement is also non-linear and complex. Its effects tend to accumulate gradually through shifts in understanding, trust and perceived legitimacy, rather than through a single, traceable cause–effect pathway. Much of its impact only emerges when the right combination of conditions is present (Bicket, et al., 2020).

The MPR already demonstrates that the system is capable of project‑level innovation: the Climate Engagement Fund encourages grant‑making organisations to experiment with new formats and audiences (p.16); Climate Action Hubs are deliberately tailored to local contexts, learn from one another, trial new approaches, and feed insights back to Government (p.33). By mandating structured reflection (e.g., the Hubs Section 7 template) and a central “catalogue of evidence”, projects can capture best practice, challenges and suggestions for improvement, then push those insights up the hierarchy so the ToC and the input assumptions are regularly refreshed (Scottish Government, n.d.). Workshop participants reflected on these aspects of PES delivery:

“It kind of links to the co-design. For me, these kinds of things are about people having agency over how they report or how they measure success. And it gives a sense of control over the project […] Potentially, we’re already doing a lot of this, like naturally it just happens, and so it’s more a case of just being a bit more conscious of it and thinking about how it’s used, and how we can maximise the value of it to have the most impact.” (Quote from workshop with Scottish Government)

“In the Hubs evaluation, we did include in the contract quite a big emphasis on learning and knowledge sharing. And not necessarily like in an innovative way, but I think just in a way that works for the people involved in the Hub activities. So going beyond just publishing the evaluation report and leaving it out there. […] We’re setting up some learning sessions with the Hubs, which will be centred around different themes. The researchers will present findings to the Hubs, but then have more of an open discussion back and forth with them about how they might integrate the findings in their own work.” (Quote from workshop with Scottish Government)

These projects are already generating insights on what works, for whom and in what contexts—precisely the type of evidence highlighted in the Magenta Book’s guidance for complex social interventions. The challenge is not generating innovation, but capturing it systematically, synthesising it, and feeding it back into the Inputs stage of the PES so that assumptions, mechanisms and ‘supporting activities’ are regularly refreshed.

Robustly testing the efficacy of delivery methods and supporting activities, especially if adapting to the context of Scotland specifically, will require investment in more in-depth research. Proportionality is essential. Richer qualitative data (e.g., case study narratives, interviews, participant journeys, etc.) should be used selectively, where it adds explanatory power or illuminates causal mechanisms. Over time, repeating this cycle of project-level experimentation, structured reflection, and strategic refinement will allow the PES to build a credible evidence base for continuous improvement, strengthen its ToC, and ensure that Scotland’s public engagement system remains adaptive, place-sensitive and capable of evolving alongside societal needs.

All of these activities around learning and innovation should be captured as Inputs, specifically as “Insights and formal learnings” implemented into delivery at strategy level (see Strategy-level Inputs under A guide for reviewing and refreshing the PES Theory of Change and Evidence for Inputs and Activities).

How to cite this publication:

Stevens, B and Vester, S (2026) ‘Measuring the impact of climate change engagement in Scotland,’ ClimateXChange.

© The University of Edinburgh, 2026
Prepared by SAC Consulting on behalf of ClimateXChange, The University of Edinburgh. All rights reserved.

While every effort is made to ensure the information in this report is accurate, no legal responsibility is accepted for any errors, omissions or misleading statements. The views expressed represent those of the author(s), and do not necessarily represent those of the host institutions or funders.

This work was supported by the Rural and Environment Science and Analytical Services Division of the Scottish Government (CoE – CXC).

ClimateXChange

Edinburgh Climate Change Institute

High School Yards

Edinburgh EH1 1LZ

+44 (0) 131 651 4783

info@climatexchange.org.uk

www.climatexchange.org.uk

If you require the report in an alternative format such as a Word document, please contact info@climatexchange.org.uk or 0131 651 4783.

1. “Scotland is already leading the way, not just in have the PES in place but also having a built-in process of monitoring and evaluation. The PES should continue to provide a clear and positive vision for the future and include multiple approaches, including co-ordinating large-scale engagement and supporting smaller local engagement. It could explore more creative innovative activities than those currently used.” (Millar, et al., 2024) ↑

2. This point is echoed in the Magenta Book guide on evaluation in complex systems, which identifies “levers and hubs” as a key aspect of complex systems: “Some components of a system may have a disproportionate influence over the whole because of the structure of their connections. Their activity may help to mobilise or slow down change [i.e. depending on whether they are for or against the issue], and their presence or absence make a system vulnerable to disruption” (Bicket, et al., 2020). Using trusted messengers takes full advantage of this aspect to the benefit of PES objectives. ↑
   

3. “Outtakes” is an intermediate Theory of Change category included in the UK Government Communication Service Evaluation Cycle. They are a useful concept in any intervention which has an awareness-raising or engagement element. They are defined as “the audience perception – what they think, feel or intend to do as a result of your communication activities. Outtakes capture the reception, perception, intentions and reaction of your target audience to your communication activity. Outtakes are distinct from outcomes: while outtakes focus on audience beliefs, attitudes and feelings, outcomes focus on actual changes in behaviours.” (Government Communication Service, 2024) ↑
   

4. “SMART objectives should be objectively observable and measurable” (HM Treasury, 2022). ↑
   

5. To assist with this, the GCS Smart Targets Tool is expected in 2026. ↑
   

6. See for example the following blog from Better Evaluation: Using Social Network Analysis for M&E (Sette, 2013). ↑
   

   
   

7. See next section, GCS Evaluation Cycle. ↑
   

8. For example, explicitly state the assumption that providing funding through the Climate Engagement Fund (Input) will successfully mobilise local action (Outcome) because the funded organisations have trusted community relationships (Assumption). ↑
   

   
   

9. The Millar et al. (2024) provides a robust evidence base on best practice for engagement, commissioned for this purpose. ↑
   

   
   

10. The Climate Action Hubs ToC (not yet published but shared with the research team for the purposes of this report) is quite robust, with assumptions and key causal links identified. It includes an evaluation framework which builds directly from this, with fields for entering key evidence and how this links to specific parts of the ToC. Data and indicators collected by the Hubs, and how these support strategy-level evaluation, are included in Appendix E – Data and indicators for PES M&E.Programme- and strategy-level ToCs should be aligned and resolved, underpinned by the same frameworks and consistent language. ↑
    

    
    

11. The GCS metrics differentiate between “reach” (the size of the audience that could have been exposed to a communication piece; a one‑way, exposure‑only metric that estimates how many individuals the output could have touched) and “contacts” (the people with whom the strategy has had a direct, two‑way interaction, e.g., attending an event, signing up to a mailing list, responding to a survey, or any other activity that creates a record of the person’s participation and allows follow‑up). The GCS therefore recommends tracking “# contacts” as a separate indicator from reach, reflecting the depth of engagement rather than just potential audience size (UK Government 2024). ↑
    

    
    

12. Rephrased slightly from existing PES ToC, where this Indicative Action was “Enhance consultation process.” Rephrasing this as “Consultation process enhanced” enables it to fit as an Outcome, a better fit for this goal of the PES. No other PES Indicative Actions, Outcomes or Impacts were rephrased. ↑
    

    
    

13. This quote was specifically referring to the outcomes of Scotland’s Climate Assembly. However, interview and workshop findings confirm that linking the results of engagement to policy changes is a challenge faced by Scottish Government policy teams. ↑