The Lancet editorial Evaluation: the top priority for global health states: "Evaluation must now become the top priority in global health. Currently, it is only an afterthought. A massive scale-up in global health investments during the past decade has not been matched by an equal commitment to evaluation. This complacency is damaging the entire global health movement. Without proper monitoring and accountability, countries and donors--and taxpayers--have no idea whether or how their investments are working."
Evaluation can be defined both as a means of assessing performance and to identify alternative ways to deliver: as example the new Canadian Federal Evaluation Policy developed by the Treasury Board of Canada defines evaluation as "the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.”
As you explore the resources below, consider this case scenario:
Imagine that you're a member of a project team who have been commissioned to run a nutrition project in Peoplesville, a fictional town where there are high levels of malnutrition and stunting in children. Your donors are very keen to see the impact of their investment, and so you've been tasked with planning the evaluation of the project, to demonstrate your impact and learn lessons for the next project cycle. Remember that even though the evaluation takes place at the end of the project, you have to think about it from the beginning so that you can collect baseline data, monitoring data throughout the programme, and final data at the end so that you can show an impact - if you only think about evaluation at the end, then you won't necessarily have the baseline data so you won't know if you've had an impact.
The BetterEvaluation Rainbow Framework organises different evaluation options (methods and strategies) in terms of different tasks in an evaluation, organised into seven clusters. Please browse through - it makes evaluation come alive and supports learners through the steps thinking about the theory of change in their programme. You may find it useful if you do want to evaluate a programme.
Donabedian and Kruk and Freedman tell us that we should develop an evaluation framework with a set of specific indicators. When thinking about indicators, consider what makes a good indicator, e.g. "health status" is not a good indicator for outcomes. Why is this? Well, what do you mean by 'health status'? How will you measure it? A neat acronym for developing good indicators is "SMART" - Specific, Measurable, Achievable, Realistic, Timely. An example of this would be "under 5 mortality per 1,000 population per year" - it's specific, you can measure it, you can achieve a reduction in that indicator with your programme so it's achievable and realistic, and you'll be able to collect data on in in the time-frame you need so it's timely.
A simple structure, process, outcome model should include:
1. Inputs/Structure (staff, buildings, systems, policies)
2. Process
- Effectiveness (access and quality for the whole population)
- Equity (access and quality for marginalised groups; participation and accountability)
- Efficiency (adequacy of funding; costs and productivity; administrative efficiency)
3. Outcome
- Effectiveness (health outcomes; patient satisfaction)
- Equity (health outcomes for disadvantaged groups)
- Efficiency (cost-effectiveness of the whole programme)
Here is an example:
Evaluation in the project cycle
There are many different ways to think about the 'project cycle'. Here is one from the United Nations Environment Programme (UNEP) handbook
- Phase 1: Project identification - analyse the situation, identify the project, and prepare the project proposal
- Phase 2: Project formulation and implementation - feasibility assessment, establish baseline and target data, and plan implementation
- Phase 3: Review and approval - multi-stakeholder review and approval of project plan
- Phase 4: Project implementation - project implementation according to plan, monitoring and reporting of data, and risk assessment and management
- Phase 5: Evaluation - mid-course evaluation for amendments and improvements, and end of project evaluation for generation of lessons learned