EVALUATION METHODS: REFLECTIONS AND PROPOSITIONS FOR THE EVALUATION COMMUNITY
Moderator: Michelle McGuire, Databuild Research and Solutions Ltd.
This session will present some propositions for the evaluators of energy efficiency policies and programmes. Taking us from a programme level view, through a policy level view to a reflection on the whole practice of energy efficiency programme and policy evaluation – the authors in this session combine personal reflection, experiences and extensive analyses to propose new ways of thinking and approaching our role as evaluators to the IEPPEC community.
Beset by power outages and a long-term electricity supply shortfall, Eskom, the South African national electricity utility, introduced a cash rebate for installing residential Solar Water Heaters (SWH) in 2008. The rebate programme set an ambitious target of 925 000 SWH installations by 2013, but was suspended in 2015 with only 102 498 rebate payments made, 11% of the initial target. In their paper, Theo Covary and Karin Kritzinger provide an overview of the Eskom rebate programme, after which they measure the programme against attributes of internationally successful SWH incentive programmes namely quality installation standards; certainty and long term commitment; system performance targeted to avoid over-sized / over- priced systems; strong marketing campaign; holistic contractor training and customer education and; mandatory regulations for new buildings.
The paper outlines a study of the programme that found that only one of the attributes of internationally successful SWH incentive programmes was present, to understand why the programme was ineffective. Wider lessons are drawn about the need for planning and commitment in all government support programmes.
In their paper Christine Wörlen, Sarah Rieseberg and Ramina Lorenz propose The Theory of No Change, developed on the basis of two meta-reviews of evaluations of sectoral transformations – energy efficient appliances in Thailand and district heating systems in Poland. Seeing that theory-based evaluation are ‘the order of the day’, they reflect that many theory-based evaluations focus on programme logics rather than actual theories of change and argue that this is inadequate as it does not account for the potential relevance of these external factors.
As a result, they propose the Theory of No Change, a theory of change for sustainable energy programmes. The Theory proposes there are four main groups of stakeholders which can influence the effectiveness of a market transformation project or programme in the energy policy field: the users, the providers of goods and services (“supply chain”), the financiers and the policy makers. Each of these can face one or more of six generic types of barriers: lack of motivation, lack of awareness, lack of access to the technology, lack of technical expertise necessary to use the technology, lack of affordability and lack of cost effectiveness. The paper describes the Theory alongside the practical application of the theory in designing climate mitigation programmes.
Reflecting on her personal journey through evaluation and her observations on the field of energy efficiency programmes, in her paper, Keryn Hassall draws on her exploration of the literature and discussions with evaluators to suggest that mainstream evaluation provides constructs in how we think about evaluation that energy programme evaluators can use in their response to the increasing expectations of energy policies and programmes.
The paper suggests that it is time for the field of energy programme evaluation to move away from its explicitly technical focus, and to join with mainstream evaluation – the body of knowledge and practice that is maintained and advanced by the professional evaluation societies internationally. It describes the diversity of mainstream evaluation and the potential for collaborating to develop a comprehensive approach to energy programme evaluation.
The Theory of No Change [paper]
Christine Wörlen, Arepo Consult
Sarah Rieseberg, Arepo Consult
Ramona Lorenz, Arepo Consult
Learning from evaluation in other sectors: adjusting to the changing demands on evaluation and evaluators by joining with mainstream evaluation [paper]
Keryn Hassall, ARTD Consultants