Evaluations normally happen at the end of a project – or at a pre-defined point within a long period of project interventions (for example, a midterm evaluation) – and entail a reflection and assessment of what has been achieved and learned.
The focus of evaluations is on outcomes and how effective project interventions are in achieving development changes. Outcomes are development changes between the completion of outputs and the achievement of impact, and are achieved in partnership with others.
Traditionally, evaluation tended to be managed with an outsider perspective, often giving little recognition to local expectations and the potential for stakeholder contributions. In effect, stakeholders were the objects of evaluations rather than key participants. Beneficiaries, local organizations and governments in recipient countries were left without substantive roles.
Over the past years, the value of engaging key stakeholders has become critically linked to the achievement of downstream performance results. Experience has shown that if stakeholders have participated in the development of evaluation results, they are more likely to contribute to their implementation.
A participatory evaluation is an opportunity for the stakeholders of a project to stop and reflect on the past in order to make decisions about the future. Through the evaluation process key stakeholders can become integrally involved in:
Overall, participatory evaluation serves to increase the analytical capacities of community members, and empower them to assess their environment, reflect on development achievements and take action to respond to issues and challenges. Participatory evaluation can, therefore, provide more comprehensive information on efficiency, relevance, sustainability, impact and effectiveness of work in progress.
Possible steps in a participatory evaluation process include:
Step 1: Initial discussion with the project team and key stakeholders to determine the composition of the evaluation team as well as groups and people to be involved in the evaluation process
Step 2: Organize a pre-evaluation workshop for the evaluation team to discuss the purpose and scope of the evaluation and to determine key evaluation questions.
The planning stage of participatory evaluations is extremely important. Initial questions to be addressed include:
Step 3: Conduct evaluation training for the evaluation team in participatory evaluation tools as well as other methods which could be used of the evaluation. Discuss guiding principles such as how to engage communities, observation and listening skills.
Step 4: The evaluation fieldwork includes the collection of quantitative and qualitative data. As part of the evaluation field work there should be an on-going analysis of findings/learning within evaluation team.
Step 5: Following the data collection process, a data analysis workshop is conducted attended by all data collection team members. During this workshop, information collected by all the evaluation team members is analyzed and summarized.
Step 6: During the evaluation feedback session preliminary findings and recommendations from the evaluation are presented to different stakeholders in a final meeting for validation purposes.
Step 6: Report writing
Step 7: Once the evaluation report has been finalized and agreed on, a post evaluation action plan is developed for ensuring that the evaluation findings will be followed up and will lead to improvement of on-going work.
Just as a carpenter has many tools in his toolbox, evaluators also must have numerous tools at their disposal. The types of tools used during the course of a participatory evaluation will vary. Ideally an evaluation should use a mix of tools in order to draw out multiple viewpoints. Selecting the right method involves many factors. Some methods are better for gathering quantitative data, others for qualitative data. Some are better for particular audiences than others.
When designing evaluation tools and selecting evaluation methods, it is useful to consider the cultural contexts of the communities in which programmes operate. Here are some guiding questions to consider ensuring that evaluation methods and tools are culturally appropriate:
What is important however, is not the tools that are used, but the space that is created for open and honest sharing of the impact of the intervention on people’s lives.
Below I would like to present some useful tools for engaging people in participatory evaluation to capture changes and to analyse impacts.
Mapping is a method which can be used in the planning phase of a project, as well as in evaluation. In an mid-term evaluation of an Area Development Programme conducted in Lao PDR, village maps prepared during the assessment phase were used as a visual aid to help villagers discuss progress and to involve them in evaluating the project’s work.
As a first step villagers were asked to add new households, new infrastructure built (e.g. roads, water systems, schools, electricity), new fields and reflect on land use changes since 2005 (e.g. new types of crops, size of fields, etc.).
Secondly villagers re-assessed the poverty status of each household based on poverty criteria developed in 2005.
The comparison of the wealth ranking exercise conducted in 2005 with the actual situation in 2010 was used to differentiate the benefits that different community members have gained from development interventions. A comparison of the old village map with the updated village map showed that:
River of Life exercises provide an opportunity to reflect collectively on: 1) the positive forces that helped the project to achieve its outcomes and goal, 2) the blocks and barriers that hindered effective project implementation, and 3) major successes and breakthroughs. In the process of doing this exercise, project stakeholders can explore the project’s past, its present and its future.
How to do it?
1. Draw a river/ highway/ path as a symbol of the journey of development work completed in a selected period of time (e.g. the past 5 years);
2. Identify symbols representing the factors that have helped and hindered the project in accomplishing its outcomes and goals. For example, with the river image supportive factors can be represented in the drawing as brooks leading into the big river, a big fishes is in the river or a message in a bottle. Blocks or barriers can be depicted as dams, troubled times can be depicted as rapids;
3. Once the picture has been completed, the river/ highway/ path image can be used to help the project team to better understand the origins of their accomplishments and struggles and to look ahead to examine opportunities available for improvements and discuss potential barriers the project must overcome to achieve success.
The use of secondary data in an evaluation represents tremendous cost and time savings, and every effort should be made to establish what secondary data exist and to assess whether or not they may be used for the evaluation. Possible useful secondary data sources for an evaluation include: local health clinic records, education statistics, etc.
However, it is important that secondary data sources are used with care and judgment triangulated with primary data whenever possible. Therefore, before using secondary data, project team members should evaluate the data to identify any limitations on their use.
The use of interviews in evaluation is inevitable. In an evaluation, interview usually takes the form of a face-to-face discussion between an evaluation team member and project stakeholders (e.g. beneficiaries, partners, project team, [...].
Key Informant Interviews are particularly useful for gaining insight into the experience of individuals and households. They help to provide information from knowledgeable people on:
It is highly recommended to develop a few questions to guide the interview in advance to ensure that issues discussed with different project stakeholders help answer evaluation questions.
Children drawings can help the evaluation team to find out which changes had the greatest impact on children’s lives.
Proposed process to use children drawings about change during the evaluation process:
1. Ask a group of children/ youth to draw a picture of their community 5 years ago on a A0 paper. Then ask the same group of children/ youth to draw a map of their village as it is today;
2. 1-2 facilitators observe the children while they are drawing their community. They listen to their discussions, record key points and notable quotes on a piece of paper.
3. Ask the children to explain both drawings. What did they draw? What are people doing in the pictures?
4. Ask the children to explain differences between both drawings. What changes have they observed in their local communities? IMPORTANT: The pictures have to be analysed by the children themselves, not by the evaluation team!
5. 1-2 note takers record key points and notable quotes;
6. Take a photo of all the drawings.
Field visits conducted to selected target villages can be used to discuss programme achievements, significant changes and areas to improve with beneficiaries and other key stakeholders (e.g. Village Development Committees).
The achievement rating/ smiley scale is a good tool to be used during mid-term and end-of-project evaluations to involve communities and partners to make an assessment of progress against in delivering outputs, and achieving outcomes and impact.
A detailed step-by-step description on how to use the achievement rating/ smiley scale can be found in the 'monitoring section' (5.4.3.)