Guidelines for Evaluation

IMPRINT

Investment Management

PUBLISHED BY:

Austrian Development Agency

Zelinkagasse 2, 1010 Wien Tel.: +43 (0)1 90399-0 Fax: +43 (0)1 90399-1290 [email protected], www.entwicklung.at

Management and editorial team:

Sigrid Breddy, Astrid Ganterer, Tonka Eibs Austrian Development Agency

Consultations and drafting:

Lisa Ringhofer, Silvia Weninger www.triple-minds.com

Graphic design and Layout:

Christina Brandstötter www.brandsisters.com

Vienna, July 2020

Order from:

Austrian Development Agency (ADA), Zelinkagasse 2, 1010 Vienna, Austria [email protected], www.entwicklung.at

Photo: © istockphoto/Patacharin Saenlakon

CONTENTS

PREAMBLE …………………………………………………………………………………………………………………………………. 5 LIST OF ABBREVIATIONS …………………………………………………………………………………………………………. 6 I. INTRODUCTION……………………………………………………………………………………………………………………….. 7 II. GUIDING PRINCIPLES…………………………………………………………………………………………………………… 8 III. TYPES OF EVALUATION MANAGEMENT AND ROLES AND RESPONSIBILITIES………11

IV. THE EVALUATION PROCESS IN 15 STEPS ……………………………………………………………………..13 PLANNING, PREPARING AND COMMISSIONING THE EVALUATION…………………………….14

STEP 1: STEP 2: STEP 3: STEP 4: STEP 5: STEP 6: STEP 7

Frame the evaluation interest and use ………………………………………………………………….14 Detail purpose and objectives ……………………………………………………………………………….. 15 De ne key evaluation questions ……………………………………………………………………………. 16 Outline evaluation design and approach ………………………………………………………………. 17 Estimate the budget …………………………………………………………………………………………………. 21 Develop Terms of Reference (ToR)………………………………………………………………………… 24 Select and commission evaluator(s) …………………………………………………………………….. 25

INCEPTION …………………………………………………………………………………………………………………………….26

STEP 8: STEP 9: STEP 10:

Kick-off and clari cation meeting ……………………………………………………………………………26 The evaluation matrix ……………………………………………………………………………………………….26 The inception report ………………………………………………………………………………………………….28

INQUIRY ………………………………………………………………………………………………………………………………….29 STEP 11: Data collection and analysis ……………………………………………………………………………………29

SYNTHESIS …………………………………………………………………………………………………………………………….31 STEP 12: Findings, conclusions and recommendations ……………………………………………………..31 STEP 13: The evaluation report ……………………………………………………………………………………………….32

WORKING WITH THE FINDINGS …………………………………………………………………………………………34 STEP 14: Disseminate evaluation ndings …………………………………………………………………………….34 STEP 15: Coordinate management response and follow-up ………………………………………………35

SUMMARY OF KEY OUTPUTS ALONG THE EVALUATION PROCESS ………………………………..37

3

PREAMBLE

It is now more than ten years ago that the Austrian Development Agency (ADA) rst developed guidelines on programme and project evaluations. After a decade of practice, we have now taken the opportunity to review the existing guidance.

The guidelines at hand have been inspired by our commitment to learn from the everyday lessons of our own work and pay tribute to recent developments within the global and national evaluation communities. We take a conscious stance to change ADA’s programme and project evaluation practice – away from requesting to evaluate each programme and project at least once in its cycle towards evaluating more selectively, purposefully and with a speci c focus on utilisation. This will allow us to both concentrate our resources where they are most useful and to enhance the quality of our evaluations and therefore, our development results. We are thereby also implementing speci c recommendations of the 2020 OECD/DAC Peer Review and the rst meta-evaluation of ADA programme and project evaluations completed in 2019.

The guidelines are based on international and national standards and good practice in the eld of development evaluation, and should be read together with the Evaluation Policy of the Austrian development cooperation and the Evaluation Criteria of the OECD/DAC. They set high standards for the quality and process of programme and project evaluations – we are convinced this is the way forward – and provide practical guidance and tools for their implementation.

Evaluation is a shared responsibility of everyone involved in the planning, implementation and monitoring of programmes and projects. This is why the new guidelines emphasise the need for consultations within ADA and with our partners: Starting from giving thought to what programmes and projects will most usefully be evaluated to earmarking the required resources, asking the right questions and choosing the most adequate evaluation approach and design. This is also important for creating a joint commitment to use evaluations ndings.

We will continue to remain committed to our institutional learning to ensure our work has the impact we aim for.

I invite you all to join us on this journey. Vienna, July 2020

Martin Ledolter Managing Director, ADA

5

I. INTRODUCTION

Welcome
to Module 1 of the Evaluation Policy of the Austrian development cooperation1: The Guidelines for

Programme and Project (PP) Evaluations at the Austrian Development Agency (ADA).2

These guidelines come at an important moment in time given the recent revision of the evaluation criteria along with relevant de nitions and principles for use by the OECD Development Assis- tance Committee (DAC)3 in December 2019 as well as the release of an interministerial Evaluation Policy by ve actors of the Austrian development cooperation4 in August of the same year.

The overall goal of these guidelines is to contribute to better PP evaluation and evaluation use and, therefore in the mid- to long-term, to better development results. To that end, they rst set out the key principles guiding programme and project evaluations at ADA (Chapter II). They then distinguish between di erent types of PP evaluation management and de ne respective roles and responsibilities (Chapter III). Finally, they o er practical guidance for each step within the evalua- tion process (Chapter IV). In addition, the guidelines provide tools and templates to help foster a joint understanding and approach to contribute to high quality programme and project evaluations at ADA (Annexes 2-10).

ADA sta and implementing partners (IP) at Headquarters (HQ) and in the eld5 are the main intended users of these guidelines. As such, the guidelines are aimed at assisting in the design, management, quality assurance and utilisation of evaluations of programmes and projects funded or implemented by ADA. They are also intended for use by external evaluators who need to un- derstand how PP evaluations are conducted in the context of the Austrian Development Cooper- ation (ADC) and its operational arm, the Austrian Development Agency, and by other interested audiences as well.

These guidelines are the result of an extensive review and consultation process and re ect learn- ings from the Meta-Evaluation of ADA Project and Programme Evaluations 2016-20186 as well as from ADA’s and IP’s evaluation practice over the past decade. They are based on international and national standards and principles as well as a review of available guidance from bi- and multi- lateral development actors. Consultations for the guidelines included key informant interviews,

a survey among key ADA sta and implementing partners at headquarters and in the eld, two workshops with a core group of ADA colleagues across organisational units at HQ and a nal feedback round with key ADA sta at HQ and in the eld. This participatory and learning-oriented approach to developing the guidelines was chosen to make them responsive to demand, user-friendly and practicable.

These guidelines are most useful if read together with the two following documents:

1. The Evaluation Policy of the Austrian development cooperation, which de nes the overall quality standards, principles and de nitions of relevance to Austrian development evaluation, and

2. The OECD/DAC evaluation criteria, standards and principles for use, which provide the recognised international framework for evaluation in development cooperation (and beyond).

Throughout the guidelines, links between the di erent chapters and annexes as well as to key external documents are provided for ease of reference and use.

Enjoy!

1. 1  The term ‘Austrian development cooperation’ denotes the entirety of Austrian ODA actors and contents and therefore extends beyond ADC, i.e. MFA and ADA. The term ‘Austrian Development Cooperation’ (ADC)’ on the other hand, is used as an institutional term, comprising exclusively the two development actors MFA and ADA. See MFA 2019a:3.

2. 2  These guidelines replace the Guidelines for Project and Programme Evaluations published by ADC in 2008. See ADC (2008).

3. 3  OECD (2019)

4. 4  MFA (2019a)

5. 5  For ADA, this includes ADA sta in Coordination O ces and Project Management Teams (PMT).

6. 6  ADA (2019a)

I. INTRODUCTION

7

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

II. GUIDING PRINCIPLES

Programme and project evaluations are de ned as the ‘[e]valuation of a single development mea- sure designed to attain speci c objectives with a pre-speci ed budget and a set plan of action (project evaluation) or an evaluation of a combination of measures put together to attain speci c development objectives at global, regional, national or sectoral levels (programme evaluation)”.7

The following principles underpin and inform relevant decision-making, thinking and practice with regards to programme and project evaluations at ADA.

1. Evaluate programmes and projects purposefully

Programmes and projects must be evaluated with a clear purpose in mind, rather than just as a matter of principle. Instead of aiming to evaluate every single intervention funded by ADA – the goal is to have the right things evaluated for the right reasons at the right time. ADA is committed to evaluate a minimum of 30 to 50 percent of its programmes and projects. To reach this ambi- tion, adequate re ection and decision-making on whether and when to evaluate need to occur within and across ADA’s organisational units, taking into account strategic considerations that go beyond individual programmes or projects.8 It is also important that ADA and its implementing partners jointly re ect on the purpose and use of an evaluation and plan accordingly from the outset of a programme or project. By default, all programmes and projects that are subject to approval by ADA’s Supervisory Board must be evaluated once in their programme/project cycle.9

The below criteria will help decision-making with regards to whether and when an evaluation may be advisable – or not. The list is not exhaustive and does not replace a consultative, contextual- ised re ection process. Budget and human resource issues need to be considered together with other factors when deciding whether and when to evaluate a particular programme or project.

1011

For guidance on scoping and framing the evaluation interest and use see Chapter IV, Step 1 (Frame the evaluation interest and use) of the guidelines.

Reasons supporting a decision to evaluate:

Reasons for refraining from an evaluation:

D Programmes and projects funded from ADA’s Small Project Fund (unless one or several criteria on the left are present)

D Contributions to funds or appeals, where ADA is only a small contributor (unless one or several criteria on the left are present)

D In case of ADA’s contributions to international organisations (IO), no separate evaluation is nec- essary if the IO conducts an evaluation. Instead, reference and use are made of the development partner’s evaluation ndings and recommendations (subsidiarity principle11)

D D

D D

D

D

8

There is a speci c knowledge interest for conduct- ing an evaluation10

Pilot programmes and projects or innovative approaches with a potential for replication or scaling-up

Programmes and projects are being considered for a subsequent phase

Programmes and projects address evidence gaps and demonstrate a high potential for learning (e.g. new approaches or themes, particularly successful or unsuccessful interventions)

Programmes and projects are of strategic impor- tance (e.g. strategic nature of partnership with im- plementing partner, strategic interest in area, theme, modality or other aspect of engagement)

High expected utilisation and usefulness of ndings

7. 7  MFA 2019a:6

8. 8  This could be done in the context of the annual work planning processes (Arbeitsfeldprogramm) at unit and/or departmental level or during team

   retreats. ADA’s thematic advisors and the EGSIM appraisal team should always be involved in this exercise.

9. 9  These are programmes and framework contracts with a budget exceeding 3 million euro as well as projects with a budget exceeding 2 million euro.

   See ADA 2017:2

10. 10  At times a speci c evaluation interest only develops during implementation, contingent on speci c events. In those cases, there should be exibility,

    also of budgetary nature, to reserve or re-allocate resources for evaluation within a programme or project budget.

11. 11  MFA 2019a:8

2. Use the OECD/DAC evaluation criteria selectively and thoughtfully

Evaluation criteria must be used and selected thoughtfully for programme and project evaluations at ADA. This is in line with the Evaluation Policy12 and the adapted de nitions and principles for use of the OECD/DAC evaluation criteria13 and is intended to help ensure that speci c informa- tion needs are addressed in a particular context and time. It further builds on a nding of the Meta-Evaluation of ADA Project and Programme Evaluations 2016-2018, which highlights the importance of clearly connecting the purpose of an evaluation with selecting and prioritising the relevant evaluation criteria to ensure adequacy and feasibility. This approach is expected to help lay the ground for enhanced programme and project evaluation quality and use.14

For guidance on how to select evaluation criteria depending on the evaluation purpose and objectives and/or on how to use them as guiding frame for developing evaluation questions
see Chapter IV, Steps 2 (Detail purpose and objectives) and 3 (De ne key evaluation questions) of the guidelines.

3. Apply evaluative thinking throughout a programme and project cycle

Evaluative thinking – that is a conscious way of thinking with a lens on how programmes and projects that are being planned or implemented can be evaluated and contribute to learning – is key to making programmes and projects more evaluable and to enabling a meaningful and useful evaluation. Important measures to be taken during the planning phase of a programme and project may include the inclusion of a programme or project inception phase where baselines for key indicators are being established (which an evaluation can draw upon and assess progress against), the development of a Theory of Change/Logic Model to map out how the intervention
is expected to deliver the desired results and the development of rigorous processes and tools to collect monitoring data (which is the foundation for robust and evidence-based evaluation ndings). Moreover, the systematic analysis of monitoring data during programme and project implementation is critical for revealing trends and dynamics that may indicate the need for an evaluation or alternative review, learning or audit processes.

For guidance on how evaluative thinking and relevant considerations throughout the programme and project cycle feed into the design of an evaluation see Chapter IV, Step 1 (Frame the evaluation interest and use) and Step 4 (Outline evaluation design and approach) of the guidelines.

4.Carefully balance scope, budget and time

Striking the right balance between what we really want to know (scope) with which resources (budget) and when along the project’s or programme’s life cycle (time) is key to ensuring high-quality evaluations. All those aspects are closely interlinked and interdepen- dent: The scope must be realistic in terms of available resources, i.e. time and budget. The evaluation ques- tions (what we want to know) determine what budget
is needed. The timing and timeliness of an evaluation are crucial to ensure uptake and use of ndings. De- cision-making with regards to timing and time further need to take into consideration the availability of stake- holders of an evaluation (e.g. farmers may not be able to support the evaluation during the harvest period).

12. 12  See MFA 2019a:9

13. 13  See OECD (2019)

14. 14  See ADA (2019a)

II. GUIDING PRINCIPLES

SCOPE

BUDGET

TIME

9

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

The Evaluation Policy recommends earmarking, where relevant, at least 3 percent of the respec- tive programme or project budget for an evaluation.15 For smaller programmes and projects16 it
is further recommended that at least 25,000 euro are earmarked for an evaluation. Establishing
a budgetary oor is important given ADA’s experience and the ndings of the Meta-evaluation, which show that conducting an evaluation below a certain budgetary threshold is both unrealistic and compromising of quality. Programmes and projects that cannot meet this oor may want to consider alternative review and learning processes.17 For larger programmes and projects exceed- ing 3 million euro, a budgetary ceiling of 90,000 euro is recommended for evaluations.

For guidance on how to balance scope, budget and time of an evaluation see Chapter IV, Step 1 (Frame the evaluation interest and use), Step 4 (Outline evaluation design and approach) and Step 5 (Estimate the budget) of the guidelines. Annex 11 (Bibliography) points to further reading on alterna- tive review and learning processes.

5. Use and learn from evaluations18

Evaluations must have a clearly de ned intended use (see principles 1 and 2). Yet this alone is not su cient. In order to contribute to better development results, evaluations must be used. The preconditions to promote learning and subsequent use of evaluation ndings are being set during the design phase of an evaluation, starting with scoping and developing the evaluation questions. Equally, evaluation use and learning from evaluation are directly linked to the quality and credi- bility of an evaluation and its ndings, conclusions and recommendations as well as the evalua- tion process itself. All of this underscores the need for solid evaluation design and application of standards and principles for good evaluation, including independence.19 The Evaluation Policy de nes preconditions and tools to help foster evaluation use and learning throughout the evalu- ation process and beyond, including the establishment of a reference group, the publication and dissemination of the evaluation report, the participatory development and timely implementation of a management response and the regular monitoring thereof.20

The use and learning from evaluation also require a conducive environment and a learning cul- ture within organisations – a learning culture and related processes that go beyond the use and learning from individual evaluations. It is therefore important to share ndings and recommenda- tions beyond the primary users of an evaluation and to proactively engage di erent stakeholders, organisational units and senior management in institutional learning processes. This is necessary for the evaluation to bring about changes at a broader level and to a larger scale.

For guidance on evaluation use see Chapter IV, Step 1 (Frame the evaluation interest and use), Step 14 (Disseminate evaluation ndings) and Step 15 (Coordinate management response and follow-up).

15. 15  MFA 2019a

16. 16  De ned as projects or programmes with a budget of approximately 833,000 euro or below, where application of the above percentage (3 percent)

    would result in a limited evaluation budget.

17. 17  See ADA (2019a)

18. 18  These can include lessons learned exercises at mid or end of programme or project, in person or virtually, learning workshops or a synthesis of already

    completed evaluations to learn from their ndings and recommendations. See Chambers, R. (2002) and Ramalingam, B. (2006) in the bibliography (see

    Annex 11), which o ers further reading on such learning formats.

19. 19  MFA 2019a:8 und MFA 2019:11

20. 20  MFA 2019a:8 and MFA 2019:13

10

III. TYPES OF EVALUATION MANAGEMENT AND ROLES AND RESPONSIBILITIES

In alignment with the Evaluation Policy, evaluations of programmes and projects that are funded or implemented by ADA are generally conducted by (an) external, independent evaluator(s).21 The role of ADA sta and implementing partners (IP) is therefore primarily concerned with evaluation management and quality assurance as detailed below.

1. Types of evaluation management: Partner-led and ADA-led

ADA distinguishes between two types of management of programme and project (PP) evaluation:

1. Partner-led evaluations: Evaluation management lies with ADA’s implementing partner. This is the case when a programme or project is funded by ADA and implemented by an IP through a grant agreement22 (Förderungen), with the planned evaluation budget included in the pro- gramme or project budget.23

2. ADA-led evaluations: Evaluation management lies with ADA. This is the case when:

   1. A programme or project is funded by ADA and implemented by an IP through a service contract (Aufträge) or where ADA commissions a programme or project evaluation outside the framework of the programme or project budget. In both cases, ADA has an overarching interest to manage and steer the evaluation.24 In these cases, evaluation management lies with the programme manager or advisor who is responsible for the programme or project at ADA HQ.25

   2. A programme or project is (co-)funded by a third party26 and implemented by ADA. In this case, evaluation management generally lies with the relevant Project Management Team (PMT) in the eld.27 Evaluation management can also lie with the Department Programmes and Projects International (PPI) at ADA HQ and/or with the ADC Coordination O ce in the eld when there is a particular overarching interest to manage and steer the evaluation.28

2. Roles and responsibilities

Regardless of where evaluation management lies, decision-making with regards to whether and when a programme or project will be evaluated as well as the scoping, purpose and objective(s) of an evaluation follows a consultative process between ADA and its implementing partners and/ or other national and bi- and multilateral donors as applicable. Please refer to Annex 1 for a work ow illustrating the relevant processes, roles and responsibilities as well as the support op- tions available to managers of partner- and ADA-led programme and project evaluations. In what follows, a brief summary is provided of the main lead and support roles when it comes to the evaluation of programmes and projects funded or implemented by ADA: The Evaluation Manager, the ADA Programme and Project Manager and ADA’s Evaluation Unit.

21. 21  MFA 2019a:11

22. 22  This includes, for example, individual grants, strategic partnerships and grants awarded through calls.

23. 23  Some programmes or projects implemented by an IP may be co-funded by a third party (for example by the Swiss Agency for Development and Coop-

    eration/SDC). In this case, ADA is co-funding the programme or project and not implementing it as in the case of 2 (B).

24. 24  Such an interest may arise, for example, when the evaluation purpose is to assess the relevance and e ectiveness of ADA’s sectoral engagement

    across di erent programmes and projects implemented by IPs.

25. 25  Concretely, these are programme managers or advisors in the organisational units L&R, T&Q, W&E, ZGI or EPOL, who are assigned the responsibility

    for a given programme or project in ADA’s funding management system (FMS) (Sachbearbeiter/in des Projekts).

26. 26  A third party may be a multilateral organisation (e.g the European Union) or fund (e.g. the Green Climate fund), a bilateral donor (e.g. Finland) or an

    Austrian actor (e.g. BMK or Bundesland Vorarlberg).

27. 27  In the context of international third-party funded projects, the procurement of services (including evaluation services) is regulated by the Project Opera-

    tional Manual (POM) (ADA 2019b:37-38). Procurement and evaluation management are therefore not necessarily led by the same person. This concerns

    mainly Step 7 in the evaluation process (see Chapter IV).

28. 28  Such an interest may arise, for example, when the evaluation purpose it to promote learning and assess broader aspects of relevance, e ectiveness and coherence and across di erent third-party funded programmes or projects implemented by ADA.

III. TYPES OF EVALUATION MANAGEMENT AND ROLES AND RESPONSIBILITIES

11

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

Evaluation Manager

The evaluation manager plays a crucial role in steering and coordinating the evaluation process and in upholding the principles and standards for good development evaluation set out in the Evaluation Policy and the Guidelines for Programme and Project Evaluations throughout. The evaluation manager also safeguards the quality and timeliness of programme and project evalua- tions, their ndings and recommendations. In the case of partner-led evaluations it is the respon- sibility of the evaluation manager to contact the ADA Programme and Project Manager for approv- al of the draft Terms of Reference (ToR), inception report (IR) and evaluation report (ER). Chapter IV details the speci c tasks of the evaluation manager for each step in the evaluation process.

ADA Programme and Project Manager

The term ADA Programme and Project Manager (PPM) refers to ADA sta responsible for a particular programme or project in ADA’s Funding Management System (FMS).29 The ADA PPM accounts for the quality and timely implementation of ADA-funded or implemented programmes and projects – including their evaluations. The approval of evaluation ToR, IR and the nal ER always lies with the ADA PPM, whether for partner-led or ADA-led evaluations. Prior to approval, the PPM needs to ensure that other organisational units are consulted as relevant.30 In particular, the PPM should always inform and involve ADA’s thematic advisors at HQ and in Coordination O ces as well as ADA’s EGSIM appraisal team in the review and feedback process. In the case of partner-led evaluations, the PPM acts as interface between the IP and relevant organisational units involved in the evaluation process at ADA HQ and in the eld (see Annex 1 and Chapter IV, Steps 1 to 15).

ADA’s Evaluation Unit

ADA’s Evaluation Unit (EVAL)31 based in Vienna provides technical advice and quality assurance support to programme and project evaluations when contacted by the ADA PPM. In line with ADA’s Public Disclosure Policy32, the executive summaries of programme and project evaluations are published for all programmes with a budget exceeding 3 million euro, and for all projects with a bugdet exceeding 2 million euro.33 These evaluations should always be reviewed and quality assured by ADA’s Evaluation Unit. It is the responsibility of the ADA PPM to make timely contact with ADA EVAL to request support.

12

29. 29  For the purpose of these guidelines, the term ADA PPM generally means the ADA sta responsible for the programme or project in ADA’s funding man- agement system FMS (Sachbearbeiter/in des Projekts). In the case of third-party funding where ADA sta responsible for third-party funded projects
    in the FMS are themselves funded with those funds, the Head of Unit International Third Party Funding (DMI) assumes the PPM function to warrant necessary independence.

30. 30  In case where the Head of DMI functions as PPM, this coordination task is delegated to the ADA sta responsible for the programme or project in ADA’s funding management system FMS (Sachbearbeiter/in des Projekts).

31. 31  The mandate of ADA’s evaluation unit includes the steering of strategic evaluations in line with ADC’s bi-annual evaluation plan, the coordination and monitoring of management responses to strategic evaluations, support to programme and project evaluations, contributions to the continuous develop- ment of ADC’s evaluation system, and represent ADC together with the MFA as competence centre on evaluation in international networks.

32. 32  ADA (2018a)

33. 33  This applies to all programmes and projects approved after 3rd July 2018.

BETTER DEVELOPMENT RESULTS

SUPPORT LEAD

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

STEP 1

Frame the evaluation interest and use

The process of framing the evaluation interest – or scoping an evaluation – is exploratory. It serves to de ne the overall direction before thinking of evaluation criteria or evaluation questions. It begins with asking the questions whether an evaluation would be useful and feasible (see Chapter II, principle 1). This re ec- tion lays the ground and sets the overall parameters for the evaluation process as well as the framing of the evaluation interest and use. It further helps re ect on what is really meaningful within the speci c context of a programme or proj- ect: What do we really want to learn from the evaluation? And what not?

Evaluation managers and ADA programme and project managers are encour- aged to deliberately exclude aspects that are of limited or no interest. This will help focus the evaluation and increase clarity, feasibility and ultimately, use. As a rst step, it is therefore recommended to de ne what should be within and outside the scope of an evaluation – in a so-called scoping exercise jointly with key stakeholders and following a consultative process (see Annex 2). This thinking needs to be made explicit and further detailed later when developing the relevant sections of the Terms of Reference (see Step 6).

Aspects to consider when scoping an evaluation include:

• Geographical aspects: Which regions, countries, areas, districts, target com-

  munities should be part of the evaluation? Which ones not?

• Time-related aspects: What period do you want to consider for this evalua- tion? The current programme or project cycle, multiple cycles, or only a speci c time period within a cycle?

• Thematic/structural aspects: Do you want to look at the entire programme or project or only at selected components?

• Evaluability aspects: Is enough data available and are key informants accessi- ble to enable solid data collection and evidence generation?

  The timing and timeliness of an evaluation are important elements to consid-
  er when framing the evaluation interest and use. In terms of timing, a broad distinction can be made between mid-term and end-term evaluations.36 Broadly speaking, mid-term evaluations intend to inform decision-making with regards to project or programme implementation to maximise the potential for achieving intended results. End-term evaluations are generally conducted to assess how and why results were achieved (or not), in order to inform decision-making with regards to programme and project continuation. Closely related to timing is the timeliness of an evaluation. When do evaluation ndings and recommendations need to be available so that they can be put to e ective use? This is largely determined by the needs of evaluation users and relevant decision-making processes. It is also important to consider: How quickly can ndings be made available and communicated in order to be available at the time when they are needed for decision-making?

36 Other types of evaluations that are typically implemented outside the framework of a speci c programme or project or alongside its implementation are ex-ante/ex-post evaluations and real-time/developmental evaluations (see MFA 2019a).

14

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

STEP 2

Detail purpose and objectives

Every evaluation must have a clearly de ned purpose to be of practical use. That is, it must meet the information needs of its intended users. If the purpose is unclear, there is a risk that the evaluation will focus on the wrong issues, draw invalid conclusions and provide recommendations that are neither useful nor used.

Evaluation in Austrian development cooperation performs three intercon- nected functions which serve as a guiding frame for de ning the evaluation purpose:

1. A learning function to understand why particular development interventions have worked or not;

2. A steering function to supply credible and reliable ndings for evidence-based decision-making at strategic and operational levels; and

3. An accountability and communication function to give account of the use of public funds and corresponding results achieved to partners, donors and the Austrian public at large.37

Even though in practice evaluations may serve multiple purposes, deciding on the main purpose is important in order not to lose focus of the evaluation.38 To de ne the purpose of the evaluation, the evaluation manager is encouraged
to start with asking the following questions: Why and for whom? Why is the evaluation being undertaken now? Who is asking for it? What are the intended bene ts of the evaluation and for whom? If the evaluation mainly aims to meet an accountability purpose, it is important to de ne accountability to whom and for what. If the main purpose of the evaluation is geared towards learning, it is important to be clear about learning by whom and how this learning is sup- posed to happen.

The objectives logically follow from the purpose and provide more details on what the evaluation seeks to accomplish and how the results will be used to ben- e t the programme or project, other interventions or the organisation at large (see Box 2 for an example). Sometimes evaluation objectives are formulated using
the OECD/DAC evaluation criteria (see Step 3). It is recommended to formulate between one and three objectives, yet there is exibility in this number. Evalua- tion should be objective driven rather than driven by methods or methodological considerations. Only when the objectives are clearly formulated does it become possible to determine the most suitable approach and methodology.

As the purpose and objectives are identi ed, it is important to specify the intend- ed users of the evaluation in order to ensure that their information needs and expectations are met. Intended users may include ADA sta and/or implementing partners, coordinators or other sta at HQ and/or in the eld, or stakeholders from similar programmes or projects. Their main interest from an evaluation may be to gain knowledge and insights into results achievement to help choose more e ective implementation strategies. Likewise, decision-makers who oversee projects or programmes such as senior management, policy makers or donors very likely require evaluation ndings to decide whether to continue, modify, or discontinue a programme or project. It is therefore critical to consult the intended users when de ning the purpose and objectives of an evaluation.

37. 37  See ADC 2019:5

38. 38  That said, all evaluations, whether primarily for accountability, learning or steering, are learning opportunities.

IV. THE EVALUATION PROCESS IN 15 STEPS

TIP

Make sure that the main purpose and objectives really re ect the needs and interests of the main users of the evaluation results.

15

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

STEP 3

De ne key evaluation questions

As a next step, the evaluation manger translates the main purpose, objectives and scope of an evaluation into speci c evaluation questions (EQ). The evalua- tion questions together drive the entire evaluation: They determine the evalua- tion design, including methodological approach and methods of data collection and analysis used, as well as the evaluation budget needed.

All evaluations of programmes and projects funded or implemented by ADA need to refer to and use the OECD/DAC evaluation criteria as the guiding frame for developing the evaluation questions (see Box 1). It is important to apply cri- teria thoughtfully and selectively, depending on the particular purpose and ob- jectives of an evaluation (see Chapter II, principle 1). Data availability, budgets, timing and methodological considerations may also inform how (and whether) a particular evaluation criterion shall be covered.39

Box 1: Overview of the OECD/DAC evaluation criteria

Evaluation questions should be clear and well-grounded in the purpose, objec- tives and scope of an evaluation. There is no standard rule as to the number
of evaluation questions, which is highly context-speci c. Yet there are trade- o s between the breadth and depth of an evaluation: The more evaluation questions, the less depth in analysis – and vice versa. When developing the evaluation questions, it is important to consider the feasibility for evaluation questions to be answered accurately within the scope, timeframe and budget of an evaluation as well as the data that are available and accessible.42 Can the

39. 39  OECD 2020:5

40. 40  OECD 2020

41. 41  OECD 2020:3

42. 42  There is however general consensus among the development evaluation community not to formulate more than two to

    three questions per criterion, as having too many may result in the evaluation losing focus. See for example, ALNAP (2016) or UNODC (2017).

16

IV. THE EVALUATION PROCESS IN 15 STEPS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

TIP

Rather than formulating spe- ci c evaluation questions for cross-cutting issues and basic principles, (implicitly) include them in every question.

STEP 4

questions be answered by drawing on more than one source of information in order to allow for triangulation? Careful consideration should also be given to involving relevant stakeholders in the formulation process. In addition to the intended users of an evaluation this may include the partner government or civil society. This will help to ensure that the information gained from each EQ is of high importance to stakeholders and generate interesting ndings that are likely to be used.

Here is some practical guidance for developing good evaluation questions:

• Formulate open-ended questions that commonly start with to what extent, to what degree, how well or how (e.g. To what extent did the intervention have an e ect on institutional change?)

• Use analytical (why?) rather than descriptive (what?) questions

• Use action-oriented questions, as they strongly focus on how ndings will ac- tually be used (e.g. How could we better support the inclusion of marginalised youth in Village Saving Groups?)

• Formulate questions with a single focus
  (not comprising several questions in one questions)

• Avoid questions that already implicate part of the analysis and response Outline evaluation design and approach

  Once the evaluation questions are developed, the evaluation manager de nes the evaluation design and approach through which these questions will be43 answered. For the purpose of these guidelines, design is understood as the overall strategy chosen for assessing and analysing change; while approach
  is understood as the methodological approach including the selection of data collection and analysis methods. There is no single right or best evaluation de- sign or approach, which needs to be tailored to the speci c evaluation purpose, objectives and questions. It is also important to consider the political and social context of a programme or project being evaluated as well as the available budget and timeframe. Identifying the best possible evaluation design and approach also requires balancing what is best and what is feasible.44 Finally, robust methods and data collection tools as well as triangulation are precon- ditions for obtaining solid and reliable data, which, in turn, are the basis for deriving credible and useful evaluation ndings.45

  At this point in the evaluation process it is neither necessary nor possible to develop a fully detailed evaluation methodology and set of data collection methods as this will be done by the evaluator(s) during the inception phase (see Step 9). What is necessary at this point is to indicate aspects of design that are perceived as suitable and/or necessary for answering the evaluation questions. Which design will best help structure the data collection and analysis process for answering the evaluation questions and ful l the purpose of the evaluation?

43. 43  As opposed to closed-ended questions – those questions that can be answered by a simple ‘yes’ or ‘no’ – as they are usually very limited in their scope and the analysis and answer that they require.

44. 44  UNODC 2017:130

45. 45  Triangulation means using multiple approaches, methods and sources for data collection and analysis to verify and sub-

    stantiate information. This helps overcome the bias that comes from single informants, methods, observations or perspec- tives. Validity refers to the accuracy and relevance of data, i.e. how accurately a method measures what it is intended to measure. Reliability to consistency in results using the same method, i.e. how consistently a method measures something (UNODC, 2017:132). The concepts of triangulation, validity and reliability are not speci c to evaluations. They are concepts and quality standards pertaining to research. See for example Pierce (2008:79-99).

TIP

Ensure the suggested approach and methods are human rights based, gender-sensitive and inclusive.

17

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

In terms of evaluation design, a common distinction is made between exper- imental, quasi-experimental and non-experimental design.46 They all aim to assess the causal links between the programme or project intervention and observed changes (causal attribution), but using di erent approaches. An experimental and quasi-experimental design uses a counterfactual approach47 to explain causality, while a non-experimental design identi es patterns that would be consistent with a causal relationship, which is usually grounded in a well-developed Theory of Change, and then seeks con rming and discon rming evidence.

• Experimental design: Involves the random selection of a group to an intervention (intervention group) and non-intervention (control group), pre- and post-measurement of each group and a comparison of the two.

• Quasi-experimental design: Uses a comparison where the group to an inter- vention is not randomly selected (comparison group) and attempts to take into account the challenges of doing a true experiment in real life.

• Non-experimental design: Considers the extent to which change has oc- curred only for those a ected by the programme or project without using a comparison between assisted and non-assisted groups.

  In terms of methodological approach, a common distinction is made between a qualitative, quantitative or mixed-methods approach depending on the meth- ods – that is the tools, techniques and processes – used to collect and analyse data.48

• Quantitative approaches and methods: They measure and assess what can be studied with numbers. They answer the ‘what’ questions. Quantitative meth- ods use structured approaches that provide precise data that can be statistically analysed.

• Qualitative approaches and methods: They analyse and explain what can be studied with words. Qualitative methods use semi-structured techniques to provide data than can provide an in-depth understanding of attitudes, percep- tions and behaviours.

  For the evaluation of programmes and projects funded or implemented by ADA, the use of a mix of methods is recommended to increase the variety of infor- mation and insights, and to allow for method and data triangulation in order to enhance the reliability and credibility of ndings.

  Again, focus at this point is on identifying data collection methods that are considered the most realistic and useful in a particular evaluation context, rather than listing the entire range of possible methods. Detailed information as to what methods will be used for answering the evaluation questions will be developed by the external evaluator(s) during inception and illustrated in the evaluation matrix (see Step 9).

  Table 1 provides an overview of some commonly used qualitative and quanti- tative data collection methods and techniques used in programme and project evaluations.49 Di erent options and methods for data analysis will be described in the context of data collection and analysis during inquiry (see Step 11).

  46. 46  UNODC 2017:130. As a general rule, experimental and quasi-experimental designs tend to be costlier and more time-in- tensive for preparation and implementation than non-experimental designs.

  47. 47  A counterfactual is a comparison between what has actually happened because of a project or programme, and what would have happened in its absence (see Rogers, 2014).

  48. 48  See for instance Cresswell (2014) and UNODC (2017:133).

  49. 49  Adapted from UNODC (2017:136-137). These methods refer to both, primary and secondary types of data and are listed in alphabetical order and not in order of relevance.

TIP

Encourage the selection of a varied mix of methods to get a richer set of data.

18

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

Table 1: An overview of data collection methods and techniques

IV. THE EVALUATION PROCESS IN 15 STEPS

50. 50  MSC is not merely a data collection method, but rather a process that involves the collection of signi cant change stories emanating from the eld level and the systematic selection and analysis of these (see Davies & Dart, 2005).

51. 51  PRA is not merely a data collection method but rather a process that involves data collection, analysis and conclusion by the community (see Kumar, 2002).

19

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

Deciding on the adequate mix of data collection methods and developing the tools and instruments needed for actual data collection is complex and time consuming. For example, a good survey re- quires careful selection of survey participants and a balanced mix of closed-ended and open-ended questions. Similarly, interviews and focus group discussions require careful preparation of interview/ discussion guides taking into account the sequencing and wording of questions. This needs to be taken into account and budgeted for when designing an evaluation (see Step 5).

It is also important to allow su cient time for data collection in the eld. Oftentimes too many interviews and/or focus group discussions are scheduled during (often very condensed) eld trips, ultimately compromising quality. Practical guidance52 and experience suggests the following:

Key informant interviews:

• Two weeks eldwork may include 25 to 50 key informant interviews

• Typically, no more than four or ve interviews per day

• Theoretical saturation53 occurs at six to twelve interviews of a particular type

  Group interviews:

• Two weeks eldwork may include ve to twenty group interviews

• Typically, no more than two or three per day

  Focus group discussions:

• Two weeks eldwork normally include approximately ten focus group discussions

• Typically, no more than one or two per day

20

52. 52  See ALNAP (2016)

53. 53  Saturation takes place when further interviews yield no more new information.

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

IV. THE EVALUATION PROCESS IN 15 STEPS

STEP 5

Estimate the budget

Determining an adequate evaluation budget depends on the purpose, objec- tives, scope, design and approach suggested for an evaluation. It also depends on the thematic and/or methodological evaluation expertise needed as well
as the expectations with regards to eld and other travel and related logistical arrangements throughout the evaluation process.

These guidelines recommend earmarking at least 3 percent of the programme or project budget for an evaluation. At the same time, a budgetary oor of 25,000 euro and a budgetary ceiling of 90,000 euro are recommended for the evaluation of a programme or project funded or implemented by ADA. (See Chapter II, principle 4). This is in recognition of the fact that conducting an evaluation below a certain budgetary threshold is both unrealistic and compro- mising of quality.

While a provisional budget estimate for an evaluation is usually earmarked within the respective programme or project budget, the evaluation manager needs to develop a detailed evaluation budget once a decision is made to evaluate. The single largest cost of any evaluation is the fees of the external evaluator(s). To ensure quality, it is crucial to have a realistic and adequate cost estimate. Box 2 provides guidance on how to calculate a detailed cost estimate for external evaluator(s) based on a concrete project example. This example is illustrative and relevant cost estimates provided need to be adjusted for each evaluation as necessary, depending on both the size and characteristics of

a particular programme or project being evaluated (in terms of geographical location, language, safety etc.), and the purpose, objectives and scope of the evaluation as well as the choice of evaluation design and methods.

Besides budgeting for external evaluator(s), it is also important to account for the time investment needed by the evaluation manager in commissioning and managing the evaluation. Su cient time (and budget, where relevant) should therefore be allocated for debrie ngs, engaging stakeholders and the evalu- ation reference group (as applicable) and for reading and commenting on the inception, draft and nal evaluation reports.

Finally, the utilisation phase of an evaluation needs to be budgeted for – both in terms of nancial and human resources. This may include the translation of the evaluation report or the development of evaluation briefs and other communi- cation tools to promote the dissemination and use of evaluation ndings. These costs need to be kept in mind and factored in by the evaluation manager when determining whether to conduct an evaluation and what resources to plan for. In case the evaluator(s) play a role in dissemination, this must be re ected in ToR (Step 6). Developing a (costed) communication plan (see Step 14) is a good practice.

21

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

Box 2: Example to illustrate steps 1 to 5

54 This project example is based on a real ADA-funded project.

22

IV. THE EVALUATION PROCESS IN 15 STEPS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

55 This calculation is based on net amounts. Value added tax (VAT), as applicable, has to be added when considering the total costs.
56 The daily fee displayed here is an example. Applicable fees vary and will be lower or higher based on a number of factors, including the local context, the scope and complexity of

the evaluation as well as the evaluator’s expertise. Daily fees should always respect the principles and standards that govern ADA’s work. See ADA 2018b. 57 One of the most common approaches is qualitative content analysis (e.g. Mayring, 2014).

23

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

STEP 6

Develop Terms of Reference (ToR)

The Terms of Reference (ToR) of an evaluation bring together the conceptual thinking undertaken during all the previous steps in the evaluation process (Step 1 to Step 5). As such, they outline 1. Why the evaluation is being under- taken (purpose, objectives and users), 2. What is being examined and why now (scope and time), 3. The key criteria and evaluation questions being addressed and how these could be answered (methodology and methods), 4. The available budget and 5. The expected timeframe and deliverables. As with the earlier steps in the evaluation process, evaluation managers should follow a consulta- tive process when developing the ToR.58 This is important to establish a shared understanding of the evaluation purpose and to clarify and manage expecta- tions among relevant stakeholders. It also helps capitalise on existing know- ledge and facilitates ownership of the evaluation process.

The ToR are a key reference document that will become part of the contractual agreement between the commissioning organisation and the external evalua- tor(s). They set out the overall framework and determine the general direction of an evaluation. As such, they serve as the key frame of reference for evaluator(s) when developing a proposal for conducting an evaluation. There are no stan- dards with regards to the length of ToR. Yet the ToR need to be comprehensive and concise in spelling out the key parameters of an evaluation including the expectations and requirements with regards to the evaluator(s)’ quali cations, the estimated timeframe and a budget range based on the calculations in

Step 5. The ToR should also include a contact person for evaluator(s) to refer to for clari cations and questions and set a realistic timeframe for the submission of proposals. Annex 4 provides a checklist for evaluation managers and ADA programme and project managers to consider when developing and approving the ToR.

Similarly, there are no standards with regards to the size and composition of evaluation teams. While one evaluator may su ce for the evaluation of projects that are of smaller size and complexity, the evaluation of larger and more com- plex (e.g. multi-country) programmes may require two (or more) evaluator(s), one lead and one (or more) support evaluators. Especially for more complex evaluations it is recommended that evaluation teams combine international and local evaluators. Having an appropriate gender-mix within the evaluation team is also important, especially in cultures with strictly assigned male and female gender roles. It is important for evaluation managers to specify the require- ments or preferences with regards to the size and composition of the evaluation team in the ToR, and to budget accordingly, so that evaluator(s) can take this into account when submitting a proposal.

58 This may be done through the establishment of an Evaluation Reference Group (ERG), which is generally composed of a small number of key stakeholders and intended users, which supports and provides inputs at key stages of the evaluation process, for example, the evaluation design (scoping and ToR), the preparation of a stakeholder matrix, the draft inception report, the presentation of preliminary ndings and the draft evaluation report (including the re nement of recommenda- tions).

24

PLANNING, PREPARING AND COMMISSIONING THE EVALUATION

IV. THE EVALUATION PROCESS IN 15 STEPS

STEP 7

Select and commission evaluator(s)

The process of nding good and available evaluator(s) can be di cult and time-consuming. For evaluation managers to reach the best quali ed evalu- ator(s) for conducting a particular programme or project evaluation, the ToR should be disseminated widely, targeting professional organisations and com- petent individuals.

There are several places to begin the search for (an) appropriate evaluator(s). One option would be to target evaluators and consulting rms that are already known to the commissioning organisation from previous assignments, or that were found via a targeted internet search. Other options for obtaining o ers may include the advertisement of the ToR59 (e.g. in newspapers or magazines) and/or their publication on the Internet or via known e-mail distribution lists for networks of freelance evaluators and consulting rms. Moreover, many national, regional and international evaluation networks/societies60 are a good source for disseminating ToR, as many of these have a free newsletter function accessible to non-members as well.

Programme and project evaluations need to be commissioned following an open, transparent procurement process in keeping with the Federal Procure- ment Act61 or other appllicable procurement law and relevant rules stipulated by ADA.62 The selection of the evaluator(s) is done on the basis of a technical and a nancial (price) o er, which are assessed against the requirements set out in the ToR. For the assessment itself, it is recommended that the evaluation man- ager develops an assessment matrix, and that an assessment commission63 is established to ensure impartiality and objectivity in the selection process.

When planning an evaluation, it is important to allocate su cient time for the procurement process, including for the advertisement of Terms of Reference. Experienced and well-quali ed evaluators typically have limited availability and must often be contracted months in advance to ensure availability. Evaluation managers are therefore encouraged to reach out to potential evaluator(s) and inquire about their interest and availability for submitting an o er soon after the decision for conducting an evaluation is made.

59. 59  For the procurement of services above a certain budgetary threshold, the advertisement of ToR via speci c channels and the minimum number of o ers to be obtained are prescribed in the Federal Procurement Act. (see BVergG 2018 as last amended)

60. 60  Such as the European Evaluation Society (EES), the DeGEval Evaluation Society (DeGEval), the Swiss Evaluation Society (SEVAL), the Vienna Evaluation Network (VEN), and Voluntary Organizations for Professional Evaluation (VOPEs). They are a particularly useful platform for distributing ToR for country-speci c evaluations.

61. 61  BVergG 2018 as last amended.

62. 62  See ADA’s General Terms and Conditions of Contracts.

63. 63  For the procurement of services above a certain budgetary threshold, evaluation committees are prescribed by the Federal

Procurement Act (see BVergG 2018 as last amended).

25

KEY POINTS IN BRIEF

> Use the meeting to provide con- text, clarify expectations and un- clear issues, review the workplan and agree on next steps.

> Ensure the meeting takes place after (and not before) signing the contract with the evaluator(s).

KEY POINTS IN BRIEF

> The evaluation matrix sets out a plan for answering each evaluation question.

> Make sure that the evaluation matrix includes 1. evaluation criteria, 2. evaluation questions, 3. indicators, 4. sources and

5. methods for data collection.

TIP

Evaluators may reformulate, re- group, reprioritise and sometimes even remove evaluation ques- tions as long as this is justi ed and agreed upon with the evalua- tion manager and the ADA PPM.

KEY POINTS IN BRIEF

> Make sure that the inception report includes
1. a preliminary desk review

summary;
2. an evaluation matrix;
3. a stakeholder mapping; and 4. a workplan.

> Allocate enough time for internal review and approval processes.

TIP:

Make sure the focus of the inception report is on the meth- odological part, not on context description.

KEY POINTS IN BRIEF

> It is important to draw on multiple sources and to triangulate data in order to have valid evidence.

> Make sure data analysis is ade- quately budgeted for and docu- mented in the inception report and evaluation report.

TIP:

Make sure that gender-sensi- tive language is used and that data collected is disaggregated by gender and other relevant dimensions.

DATA PROCESSING

(PURELY DESCRIPTIVE)

Example:

“30% of all participants have been able to expand their business”

DATA INTERPRETATION

(COMPARATIVE ASSESSMENT)

Example:

“This number is twice as high as last year.”

KEY POINTS IN BRIEF

> Make sure there is a logical ow from ndings to conclusions and recommendations.

> Findings must be based on trian- gulated data and information.

> Recommendations must be clear, actionable and targeted to specif- ic stakeholders.

RECOMMENDATIONS

PROPOSED ACTIONS FOR MANAGEMENT

CONCLUSIONS

INTERPRETATIONS AND JUDGEMENTS BASED ON – THE FINDINGS

FINDINGS

EMPIRICAL FACTS COLLECTED DURING THE EVALUATION

FINDINGS

EMPIRICAL FACTS COLLECTED DURING THE EVALUATION

FINDINGS

EMPIRICAL FACTS COLLECTED DURING THE EVALUATION

KEY POINTS IN BRIEF

> Make sure that the report presents evidence-based ndings, conclu- sions and recommendations.

> Use a feedback matrix when com- menting the draft report and leave it up to the evaluator(s) to decide whether the feedback is accepted or rejected (with justi cation).

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

WORKING WITH THE FINDINGS

STEP 14

Disseminate evaluation ndings

The timely availability of evaluation reports along with the e ective presentation70 and communication of ndings are key requirements to promote their utilisation and use.71 In addition, the transparent publication of evaluation reports contrib- utes to enhanced credibility. ADA’s Public Disclosure Policy requires that exec- utive summaries and RAFs of all evaluation reports of programmes and projects exceeding a certain threshold are made available via ADA website.72

To ensure maximum outreach and use of evaluation ndings, it is important
to consider the intended users and uses of an evaluation early on. Evaluation managers are therefore encouraged to develop a communication plan identi- fying potential channels and products to best meet the information needs of di erent audiences at the beginning of the evaluation process (see Step 1). Di erent recommendations of an evaluation may speak to di erent stakeholders within (and outside of) an organisation, pointing up the need for targeted and tailored communication approaches.73 It is also important to factor in the costs associated with communicating evaluation ndings and to make necessary arrangements accordingly (see Step 5), for example by costing the communica- tion plan.

Guiding questions to facilitate e ective dissemination of evaluation nd- ings may include:

• Why does this information need to be communicated?

• What do the di erent audiences need to know? What would they like to know?

• Are there any special considerations or limitations to be kept in mind (e.g. patchy internet connection, language, high sta turnover)?

• When is the best timing for dissemination (e.g. upcoming strategy revision, new planning cycle)?

  Evaluation ndings need to be presented and communicated in a user-friendly and accessible manner to facilitate use. For example, traditional and text-heavy evaluation reports may not speak to its intended users. Some approaches for e ectively communicating evaluation results may include story-telling elements and the use of visual aids such as diagrams, pictures, charts and graphs. A creative use of innovative information and communication technologies and social media channels (e.g. LinkedIn, twitter, Facebook, blogs, Flickr.com74) is equally important. The website FreshSpectrum75 is a great resource for explor- ing user-friendly innovative ideas for tailoring evaluation messages. Language is another factor that needs to be considered when disseminating evaluation ndings.

70. 70  See ADA (2018a)

71. 71  MFA 2019:12

72. 72  ADA (2018b)

73. 73  See ALNAP (2016:341)

74. 74  Flickr.com may be used for an evaluation photo story.

75. 75  https://freshspectrum.com

34

WORKING WITH THE FINDINGS

IV. THE EVALUATION PROCESS IN 15 STEPS

STEP 15

Coordinate management response and follow-up

A management response (MR) is an e ective tool to facilitate the utilisation of evaluation ndings.76 It allows relevant stakeholders to position themselves vis- á-vis the evaluation and its recommendations and to articulate how they will go about taking them forward. The process of developing a management response can also foster organisational learning and accountability.

The role of the evaluation manager is to coordinate the development of the MR bringing together relevant stakeholders targeted by the recommendations of an evaluation. Recommendations may be accepted, partially accept or rejected.
In case of the latter two, a rationale needs to be provided to substantiate the decision. In the MR, stakeholders de ne speci c measures, responsibilities and timeframes for implementing the recommendations (if accepted or partially accepted). A template for developing a management response to programme and project evaluations can be found in Annex 10.

It is important to allocate su cient time for developing the management re- sponse and to start the process soon after completion of the evaluation report. The MR should be nalised within a period of three months. To help ensure timely implementation and use of recommendations, it is advised to regularly monitor and update the status of implementation of the management respon- se,77 and to de ne clear roles and responsibilities in that regard. The extent to which the recommendations of a programme or project evaluation are imple- mented may also be assessed as part of a future evaluation.

76. 76  MFA 2019:12

77. 77  If not all recommendations can be implemented immediately it might be useful to specify a prioritised time frame.

35

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

DESIGN IMPLEMENTATION UTILISATION

36

BETTER DEVELOPMENT RESULTS

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

ANNEXES

ANNEX 1

Work ows for partner-led and ADA-led evaluations

D The work ows below describe the work ow for partner-led and ADA-led evaluations. They are designed to visualise and thus easily outline roles and responsibilities along the evalua- tion process.

PARTNER-LED EVALUATIONS

WORK PLAN / PROGRAMME AND PROJECT (PP) PLANNING PHASE

ADA discusses within and across relevant organisational units whether an evaluation should be planned in relation to a particular proposed pro- gramme or project and states related expectations, including a reference to the ADA guidelines for PP evaluations, at tendering/procurement stage.

ADA PPM must consult with his/ her organisation- al unit and can consult with ADA EVAL before taking a decision

If it is decided that no evalu- ation will take place, ADA PPM must document this, including the reasoning in ADA’s Funding

If an evaluation is planned, ADA PPM ensures before contract nalisation that a preliminary scoping is done and a rst cost estimate and timing are in line with the standards set by the ADA guidelines for PP evaluations

Applicant IP includes this in the PP proposal.
ADA PPM documents decision after PP is approved

Management System (FMS)

ADA PPM and IP discuss and document how learning will take place nevertheless during process cycle. IP documents this in project document and budgets for it, ADA PPM doc- uments this in ADA’s Funding Management System (FMS)

Ensure budgetary exibility in case the need for an evaluation arises at a later stage

!

ADA PPM can ask ADA EVAL to join this discussion
or meet them in preparation

ADA PPM can and in some cases, where ADA’s Disclosure Policy applies, must in- clude ADA EVAL in quality check loop

DESIGN PHASE

ADA PPM and IP hold a conference call/meeting to conduct scoping, de- ne the evaluation’s purpose and objectives and identify key evaluation questions, relevant criteria and adequate approach and methodology

IP drafts ToRs and submits them for comments and quality check to ADA PPM within agreed deadline

ADA PPM comments the draft ToR and performs quality check and returns them to IP for review and nalisation

IP nalises the ToR and sends them to ADA PPM for approval ADA PPM reviews and approves ADA PPM reviews and

ToR rejects ToR

IP disseminates the ToRs and selects and commissions the evaluator(s)

Additional feedback loop

38

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

ADA-LED EVALUATIONS

WORK PLAN / PROGRAMME AND PROJECT (PP) PLANNING PHASE

ADA PPM discusses within and across relevant organisational units whether a particular proposed programme or project should be evaluated and takes a related decision in line with the ADA guidelines for PP evaluations

ADA PPM must consult with his/ her organisational unit and can consult with ADA EVAL before taking a decision

If it is decided that no evaluation will take place, ADA PPM must document this, including the reasoning

ADA PPM discusses and document how learning will take place nevertheless during process cycle. PPM documents this in proj- ect document and budgets for it, ADA PPM documents this In FMS

Ensure budgetary exibility in case the need for an evaluation arises at a later stage!

If an evaluation is taking place, ADA PPM conducts a preliminary scoping and delinates a rst cost estimate and timing in line with the standards set by the ADA guidelines for PP evaluations

ADA PPM re ects this decision in the relevant PP documentation

DESIGN PHASE

Evaluation manager and ADA PPM conduct scoping, de ne the evalu- ation’s purpose and objectives and identify key evaluation questions, relevant criteria and adequate approach and methodology

Evaluation manager drafts ToRs, submits them for comments with relevant ADA colleagues at HQ or in the eld

Evaluation manager consolidates ToR based on the comments received and performs a quality check

ADA PPM nalises ToR and approves them. Evaluation manager disseminates ToR and manages tendering process

!

ADA PPM can ask ADA EVAL to join this discussion or meet in preparation

ADA PPM can and in some cases, where ADA’s Disclosure Policy applies, must in- clude ADA EVAL in quality check loop

40

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

ANNEX 2 Scoping Exercise

D The following questions78 are designed to support ADA programme and project managers and implementing partners in their rst brainstorming with regards to the evaluation. The initial scoping exercise can take di erent forms and shapes – meetings, phone calls, work- shops – and may be documented to help lay the ground for the elaboration of the Terms of Reference (ToR).

16. WHAT? (TOPIC)
Consider what is going to be evaluated:

• >  What is the essence of the programme and project that you want to evaluate?

• >  What is the context of the programme or project that you are looking at?

• >  Do you want to look at the whole programme or project or only at (a) selected component(s)?

• >  What do you want to exclude from the evaluation? Think of geographical, time-related, thematic, structural and/or other aspects that are of less interest or aspects that do not seem to be evaluable.

  17.WHY? (PURPOSE)
  Consider how the idea about the evaluation came up:

• >  Why do you want to do this evaluation? Why now? Who is asking for it?

• >  What kind of information would you like to obtain and which issues do you seek to address?

• >  How are the results of the evaluation going to be used and by whom?
  How can the evaluation bene t the programme or project, other interventions, the organisation?

• >  Are you interested in a speci c evaluation product besides the evaluation report and the executive summary (e.g. two-page brief, infographic page, poster, video, podcast)?

  18.WHO, WHAT, WHEN? (SCOPE)

  Consider what should be included and excluded from the scope of the evaluation:

• >  What period do you want to cover by the evaluation?
  Do you want to look at the entire programme or project phase, multiple phases or a speci c period therein (i.e. the last two years)?

• >  Who are the main stakeholders at di erent levels (HQ and eld) and what should their involvement in the process be?

• >  What is the best timing for the evaluation to take place?
  Are there any speci c deadlines or processes to consider for the nal evaluation report to be available in time?

  19.QUESTIONS? (GENERAL AND SPECIFIC)

  Consider the issues that you want to learn about with regards to the programme or project:

• >  Feel free to think of as many questions as you need at this stage – no matter if very general or very speci c. They will be netuned at a later stage.

• >  Please share any working hypothesis or working assumption underlying your questions.

42

78 This scoping exercise is adapted from MSF (2019).

ANNEXES

ANNEX 3 Checklist of Documents for Programme and Project Evaluations

D The following documents may be useful for external evaluator(s) when conducting an evaluation of programmes and projects funded or implemented by ADA. They need to be provided by the evaluation manager to the external evaluator(s) at the time of the kick-o . Some documents may not be readily available and need to be obtained from relevant stakeholders in advance. The checklist is not exhaustive and needs to be adapted and expanded for each evaluation on a case-by-case basis, as appropriate.

43

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

ANNEX 4 Quality Checklist for Terms of Reference (ToR)

D This checklist79 is designed to support evaluation managers and ADA sta and implement-
ing partners when preparing the Terms of Reference for an evaluation. It provides the basic structure of the ToR and serves as guidance for evaluation managers when reviewing and ADA programme and project managers when approving the ToR.

D Terms of Reference should be structured as follows:

1. Context and Background

2. Purpose and Objectives

3. Scope

1.

2.

3.

4.

79

4. Evaluation Questions

5. Design and Approach

6. Workplan

7. Evaluation Management Arrangements

8. Requirements for the Evaluator(s)

9. Speci cations for the Submission of O ers

10. Annexes

Context and Background

Purpose and Objectives

Scope

Evaluation Questions

This quality checklist adapted from the United Nations Evaluation Group (UNEG) Quality Checklist for Evaluation Terms of Reference and Inception Reports. UNEG (2010a).

44

5. Design and Approach

ANNEXES

6. Workplan

7. Evaluation Management Arrangements

8. Requirements for the Evaluator(s)

9. Speci cations for the Submission of O ers

10. Annexes

80 This may include participation of duty bearers and rights holders, especially women and vulnerable groups, the documentation of how data collection will be human rights based, foster environmental sustainability, gender sensitive, and include the disaggregation of data by sex, ethnicity, age, disability etc

45

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

ANNEX 5 Quality Checklist for Inception Report (IR)

D This checklist81 is designed to provide guidance to evaluation managers and ADA pro- gramme and project managers when assessing and approving the inception report. It also serves as guidance for evaluator(s) when structuring the IR, to ensure that it meets ADA requirements.

D The inception report82 should be structured as follows:

1. Background, Purpose and Objectives

2. Evaluation Design and Approach

   2.1. MethodologyandMethods 2.2. EvaluationMatrix
   2.3. Data Collection Instruments 2.4. Data Analysis

   2.5. Limitations, Risks and Mitigation Measures

3. Quality Assurance and Ethical Considerations 4. Workplan
5. Annexes

1. Background, Purpose and Objectives

2. Evaluation Design and Approach 2.1. Methodology and Methods

2.2. Evaluation Matrix

81. 81  This quality checklist is adapted from the United Nations Evaluation Group (UNEG) Quality Checklist for Evaluation Terms of Reference and Inception Reports. UNEG (2010a).

82. 82  In addition to the speci c chapters outlined below, the report also should include a title page, a table of contents, a list lists tables/graphs and gures and a list of acronyms.

46

2.3. Data Collection Instruments

2.4. Data Analysis
2.5. Limitations, Risks and Mitigation Measures

3. Quality Assurance and Ethical Considerations

4. Workplan

5. Annexes

ANNEXES

83 MFA 2019.

47

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

ANNEX 6

Quality Checklist for Evaluation Report (ER)

D This checklist84 is designed to provide guidance to evaluation managers and ADA programme and project managers when reviewing and approving the evaluation report. It also serves as guidance for evaluator(s) when structuring the evaluation report85,
to ensure that it meets ADA requirements.

D The evaluation report should be structured as follows:

1. Executive Summary

2. Introduction

3. Background and Context Analysis

4. Evaluation Design and Approach
   4.1. Methodological Approach
   4.2. Data Collection and Analysis Tools
   4.3. Limitations, Risks and Mitigations Measures

5. Findings

6. Conclusions

7. Recommendations

8. Annexes

1. Executive Summary 2. Introduction

3. Background and Context Analysis

48

84. 84  This quality checklist adapted from the United Nations Evaluation Group (UNEG) Quality Checklist for Evaluation Terms of Reference and Inception Reports. UNEG (2010a).

85. 85  In addition to the speci c chapters outlined below, the report also should include a title page, a table of contents, a list lists tables/graphs and gures and a list of acronyms.

4. Evaluation Design and Approach 4.1. Methodological Approach

ANNEXES

4.2. Data Collection and Analysis Tools

4.3. Risks, Limitations and Mitigations Measures

5. Findings

49

GUIDELINES FOR PROGRAMME AND PROJECT EVALUATIONS

6. Conclusions

7. Recommendations

8. Annexes

50

ANNEX 7

Template for Evaluation Matrix

ANNEXES

D This template is designed to help structure and document how an evaluation will go about answering the questions. The evaluation matrix is instrumental for setting the scene for an adequate and realistic evaluation and forms an integral element of the inception report. The below template can be adapted and expanded by the external evaluator(s) as relevant, but must at least contain relevant information captured in the four columns below.

D The Evaluation Matrix template (in Excel format) can be downloaded from the ADA website.

51

COMMENT PROVIDED BY TO REPORT SECTION PAGE ABOUT COMMENT TEXT REACTION EVALUATOR(S)

ANNEX 9

Results Assessment Form (RAF)

ANNEXES

D The Results Assessment Form is a mandatory annex to the evaluation report of ADA PP evaluations. It serves to help ADA to assess how integrated results based management is implemented at the level of programs/projects and make an assessment of the extent to which programmes and projects contribute to the achievement of results at di erent lev- els.86 Part 1 of the RAF needs to be lled in by the evaluation manager and the ADA PPM, while part 2 needs to be lled in by the evaluator(s).

D The RAF template (Excel format) can be downloaded from the ADA website.

* A drop-down list with the scoring scale is provided in the RAF template in Excel format available on the ADA website.

86 See ADA 2015:45.

53

Recommendation fully accepted (B)

Recommendation partially accepted (B)

Recommendation not accepted (B)

Timeline for implementation (E)

Recommendation of the evaluation (insert number and brief title) (A)

Recommendation partially accepted or not accepted, please, explain reasons (C)

Recommendation fully or par- tially accepted: please, de ne concrete measures to implement the recommendation (D)

Name of the organisation and, department respon- sible for the implementa- tion of the recommenda- tion (F)

Current status of the implementation and date (G)

1.

Yes/ Yes/ Yes/ No No No

2. 3. etc.

ANNEX 11

Bibliography