6. Evaluation

Evaluations normally happen at the end of a project – or at a pre-defined point within a long period of project interventions (for example, a midterm evaluation) – and entail a reflection and assessment of what has been achieved and learned.

The focus of evaluations is on outcomes and how effective project interventions are in achieving development changes. Outcomes are development changes between the completion of outputs and the achievement of impact, and are achieved in partnership with others.

Traditionally, evaluation tended to be managed with an outsider perspective, often giving little recognition to local expectations and the potential for stakeholder contributions. In effect, stakeholders were the objects of evaluations rather than key participants. Beneficiaries, local organizations and governments in recipient countries were left without substantive roles.

Over the past years, the value of engaging key stakeholders has become critically linked to the achievement of downstream performance results. Experience has shown that if stakeholders have participated in the development of evaluation results, they are more likely to contribute to their implementation.

A participatory evaluation is an opportunity for the stakeholders of a project to stop and reflect on the past in order to make decisions about the future. Through the evaluation process key stakeholders can become integrally involved in:

  • deciding what is to be evaluated and who will be involved;
  • selecting the methods and data sources;
  • carrying out the evaluation;
  • analyzing information and presenting evaluation results;
  • proposing solutions/directions to respond to issues/challenges.

Overall, participatory evaluation serves to increase the analytical capacities of community members, and empower them to assess their environment, reflect on development achievements and take action to respond to issues and challenges. Participatory evaluation can, therefore, provide more comprehensive information on efficiency, relevance, sustainability, impact and effectiveness of work in progress.

Possible steps in a participatory evaluation process include:

Step 1: Initial discussion with the project team and key stakeholders to determine the composition of the evaluation team as well as groups and people to be involved in the evaluation process

Step 2: Organize a pre-evaluation workshop for the evaluation team to discuss the purpose and scope of the evaluation and to determine key evaluation questions.

The planning stage of participatory evaluations is extremely important. Initial questions to be addressed include:

  • What resources (time, money) are available for the evaluation process?
  • What key questions would different stakeholders like to see answered in the process?
  • What data collection methodologies and tools will be used?
  • Who will be involved in the evaluation?
  • How will the data collection process be organized?

Step 3: Conduct evaluation training for the evaluation team in participatory evaluation tools as well as other methods which could be used of the evaluation. Discuss guiding principles such as how to engage communities, observation and listening skills.

Picture 26: Conducting evaluation training for the evaluation team
Picture 26: Conducting evaluation training for the evaluation team

Step 4: The evaluation fieldwork includes the collection of quantitative and qualitative data. As part of the evaluation field work there should be an on-going analysis of findings/learning within evaluation team.

Step 5: Following the data collection process, a data analysis workshop is conducted attended by all data collection team members. During this workshop, information collected by all the evaluation team members is analyzed and summarized.

Step 6: During the evaluation feedback session preliminary findings and recommendations from the evaluation are presented to different stakeholders in a final meeting for validation purposes.

Picture 27: Evaluation feedback session
Picture 27: Evaluation feedback session

Step 6: Report writing

Step 7: Once the evaluation report has been finalized and agreed on, a post evaluation action plan is developed for ensuring that the evaluation findings will be followed up and will lead to improvement of on-going work.

6.1. Evaluation methods/ tools

Just as a carpenter has many tools in his toolbox, evaluators also must have numerous tools at their disposal. The types of tools used during the course of a participatory evaluation will vary. Ideally an evaluation should use a mix of tools in order to draw out multiple viewpoints. Selecting the right method involves many factors. Some methods are better for gathering quantitative data, others for qualitative data. Some are better for particular audiences than others.

When designing evaluation tools and selecting evaluation methods, it is useful to consider the cultural contexts of the communities in which programmes operate. Here are some guiding questions to consider ensuring that evaluation methods and tools are culturally appropriate:

  • Are data collection methods relevant and culturally sensitive to the population being evaluated?
  • Do the instruments consider potential language barriers that may inhibit some people from understanding the evaluation questions?
  • Are multiple methods being used, so that information can be analyzed in a variety of ways?

What is important however, is not the tools that are used, but the space that is created for open and honest sharing of the impact of the intervention on people’s lives.

Below I would like to present some useful tools for engaging people in participatory evaluation to capture changes and to analyse impacts.

6.1.2. Village Maps

Mapping is a method which can be used in the planning phase of a project, as well as in evaluation. In an mid-term evaluation of an Area Development Programme conducted in Lao PDR, village maps prepared during the assessment phase were used as a visual aid to help villagers discuss progress and to involve them in evaluating the project’s work.

Picture 28: A map produced to provide a visual image of village conditions at the start of a project. Somboun Village, Phoukhoun District, Lao PDR (2005)
Picture 28: A map produced to provide a visual image of village conditions at the start of a project. Somboun Village, Phoukhoun District, Lao PDR (2005)

As a first step villagers were asked to add new households, new infrastructure built (e.g. roads, water systems, schools, electricity), new fields and reflect on land use changes since 2005 (e.g. new types of crops, size of fields, etc.).

Secondly villagers re-assessed the poverty status of each household based on poverty criteria developed in 2005.

Picture 29: People from Somboun village in the process of updating the village map during the mid-term evaluation in 2010
Picture 29: People from Somboun village in the process of updating the village map during the mid-term evaluation in 2010
Picture 30: Map of Somboun village updated during the mid-term evaluation in 2010
Picture 30: Map of Somboun village updated during the mid-term evaluation in 2010

The comparison of the wealth ranking exercise conducted in 2005 with the actual situation in 2010 was used to differentiate the benefits that different community members have gained from development interventions. A comparison of the old village map with the updated village map showed that:

  • Between 2005-2010, the total number of households increased from 55 to 66;
  • 28 households were able to improve their status (10 HHs ‘very poor/ hungry’ → ‘poor’, 11 HHs ‘very poor/ hungry’ → ‘food secure’, 7 HHs ‘poor’ → ‘food secure’);
  • 20 households kept the same status as in 2005 (7 HHs ‘very poor/ hungry’, 6 HHs ‘poor’, 7 HHs ‘food secure’);
  • 2 households got worse (‘food secure’ → ‘poor’).
Table 4: Comparison of the wealth ranking exercise conducted in 2005 with the new wealth ranking exercise conducted in 2010
Table 4: Comparison of the wealth ranking exercise conducted in 2005 with the new wealth ranking exercise conducted in 2010

6.1.3. River of Life

River of Life exercises provide an opportunity to reflect collectively on: 1) the positive forces that helped the project to achieve its outcomes and goal, 2) the blocks and barriers that hindered effective project implementation, and 3) major successes and breakthroughs. In the process of doing this exercise, project stakeholders can explore the project’s past, its present and its future.

Picture 31: The River of Life – the journey of a development project presented as a river
Picture 31: The River of Life – the journey of a development project presented as a river

How to do it?

1. Draw a river/ highway/ path as a symbol of the journey of development work completed in a selected period of time (e.g. the past 5 years);

2. Identify symbols representing the factors that have helped and hindered the project in accomplishing its outcomes and goals. For example, with the river image supportive factors can be represented in the drawing as brooks leading into the big river, a big fishes is in the river or a message in a bottle. Blocks or barriers can be depicted as dams, troubled times can be depicted as rapids;

3. Once the picture has been completed, the river/ highway/ path image can be used to help the project team to better understand the origins of their accomplishments and struggles and to look ahead to examine opportunities available for improvements and discuss potential barriers the project must overcome to achieve success.

6.1.4. Secondary Data Collection

The use of secondary data in an evaluation represents tremendous cost and time savings, and every effort should be made to establish what secondary data exist and to assess whether or not they may be used for the evaluation. Possible useful secondary data sources for an evaluation include: local health clinic records, education statistics, etc.

However, it is important that secondary data sources are used with care and judgment triangulated with primary data whenever possible. Therefore, before using secondary data, project team members should evaluate the data to identify any limitations on their use.

Picture 32: An evaluation team member collecting secondary data from a sub-district health centre
Picture 32: An evaluation team member collecting secondary data from a sub-district health centre

6.1.5. Key Informant Interviews

Picture 33: Left: An evaluation team conducting Key Informant Interviews with kindergarten teachers, right: a programme staff conducting a Key Informant Interview with a village leader
Picture 33: Left: An evaluation team conducting Key Informant Interviews with kindergarten teachers, right: a programme staff conducting a Key Informant Interview with a village leader

The use of interviews in evaluation is inevitable. In an evaluation, interview usually takes the form of a face-to-face discussion between an evaluation team member and project stakeholders (e.g. beneficiaries, partners, project team, [...].

Key Informant Interviews are particularly useful for gaining insight into the experience of individuals and households. They help to provide information from knowledgeable people on:

  • Facts and information for the verification of facts;
  • Opinions and perspectives;
  • Analyses;
  • Recommendations that can improve a project’s performance;
  • Reactions to the evaluator's hypotheses and conclusions.

It is highly recommended to develop a few questions to guide the interview in advance to ensure that issues discussed with different project stakeholders help answer evaluation questions.

6.1.6. Drawings about change

Children drawings can help the evaluation team to find out which changes had the greatest impact on children’s lives.

Proposed process to use children drawings about change during the evaluation process:

1. Ask a group of children/ youth to draw a picture of their community 5 years ago on a A0 paper. Then ask the same group of children/ youth to draw a map of their village as it is today;

2. 1-2 facilitators observe the children while they are drawing their community. They listen to their discussions, record key points and notable quotes on a piece of paper.

Picture 34: “Our community 5 year ago and now…” Tien Phuoc District, Vietnam (2006)
Picture 34: “Our community 5 year ago and now…” Tien Phuoc District, Vietnam (2006)

3. Ask the children to explain both drawings. What did they draw? What are people doing in the pictures?

4. Ask the children to explain differences between both drawings. What changes have they observed in their local communities? IMPORTANT: The pictures have to be analysed by the children themselves, not by the evaluation team!

5. 1-2 note takers record key points and notable quotes;

6. Take a photo of all the drawings.

3.1.7. Guided village transect walks/ site visits

Field visits conducted to selected target villages can be used to discuss programme achievements, significant changes and areas to improve with beneficiaries and other key stakeholders (e.g. Village Development Committees).

3.1.8. Achievement rating/ smiley scale

The achievement rating/ smiley scale is a good tool to be used during mid-term and end-of-project evaluations to involve communities and partners to make an assessment of progress against in delivering outputs, and achieving outcomes and impact.

Picture 35: Interventions are evaluated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory
Picture 35: Interventions are evaluated on a four-point rating scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory
Picture 36: Children performing a smiley scale rating
Picture 36: Children performing a smiley scale rating

A detailed step-by-step description on how to use the achievement rating/ smiley scale can be found in the 'monitoring section' (5.4.3.)