The author Kurt Vonnegut once wrote, “How complicated and unpredictable the machinery of life really is.” This is certainly true in international development, where the people and societies we work with are constantly shifting and evolving. Given this inherent complexity, how do we know that our programs and interventions are truly doing what they’re supposed to do? And whatever the outcomes—good or bad—what is actually attributable to our work?
Enter Outcome Harvesting, which, in the world of monitoring, evaluation, and learning (MEL), is one of five complexity-aware monitoring approaches (CAMs); the other four being Most Significant Change, Sentinel Indicator, Stakeholder Feedback, and Process Monitoring of Impact. Each of these can be useful when measuring a project that’s active across multiple countries, in complex operating environments, and/or when several implementers are working in the same space. But Outcome Harvesting is especially valuable when exploring issues of cause and effect.
Outcome Harvesting is forensic in nature. As opposed to examining progress toward fixed objectives, as with many evaluation approaches, Outcome Harvesting works in reverse. It collects—“harvests” if you will—evidence of changes and examines if and how a specific intervention contributed to them.
The process must be highly participatory and involves successive feedback loops with informants and other actors. In all likelihood, you’re already using the same modalities needed for Outcome Harvesting—interviews, focus group discussions, etc. The most difficult thing is not leading respondents. Evaluators must be intentional about it and ask the right questions at the right time.
The difference between Outcome Harvesting and traditional methods lies in its approach to question-crafting. Through a series of open-ended inquiries, Outcome Harvesting identifies any and all outcomes, both intended and unintended. For example, the Ford Foundation notes these illustrative questions in their write up of Outcome Harvesting:
- What happened?
- Who did it (or contributed to it)?
- How do we know this?
- Is there corroborating evidence?
- Why is this important?
- What do we do with what we found out?
But what does this look like in practice? Dexis has successfully applied Outcome Harvesting across a range of sectors and environments, from education to agriculture to youth employment. In just one example, we successfully used this approach for the USAID Securing Water for Food Grand Challenge Fund (SWFF) evaluation.
SWFF doesn’t follow a typical development project model. It’s designed to involve as many people across as many sectors as possible to help grow more food with less water. But that makes measurement and attribution messy. Our evaluation team used Outcome Harvesting to ask questions about how changes had occurred and how the implementers, partners, and end-users contributed to those changes.
With informants from across South Africa, Kenya, Uganda, Ghana, Egypt, and Bangladesh, the SWFF evaluators asked such questions as:
- Did SWFF-supported projects increase water efficiency/make water more accessible?
- Did SWFF-supported projects lead to more agricultural productivity and resilience to climate change?
- Have vulnerable groups (the poor, women, ethnic minorities) been positively and/or negatively impacted (through income, employment, water/environmental) from SWFF-supported innovations?
- To what extent have private funds been generated that contribute to the developmental objectives of the program both during and following SWFF awards?
The final evaluation noted, “Results indicate that SWFF strongly contributed to outcomes: analysis of surveys find 90% of beneficiaries have improved access to water and 95% water efficiency directly due to SWFF’s innovations.” USAID commended the evaluation team for the use of Outcome Harvesting as extremely thoughtful, which offered a depth in reporting not seen in other qualitative evaluations.
Cause-and-effect relationships and attribution are difficult to ascertain with so many players and outcomes that are either unintended or unknown. Complexity-aware monitoring approaches like Outcome Harvesting allow evaluators to still gauge whether positive change has occurred because of the project or activity even when a myriad of other factors is in effect.
As such, it can also be a good approach for grand challenges, where the idea is to identify multiple pathways to a common issue or objective (often complex or intractable by nature), and support the need to understand which pathway ultimately has had an impact.
But Outcome Harvesting is useful even when you are tracking specific outcomes with more traditional monitoring and evaluation. In fact, Outcome Harvesting can be used to complement these traditional approaches. Once you master the fundamentals, Outcome Harvesting is straightforward to administer. In essence, it’s how you’re framing your questions and how well you’re listening.
Stephanie Monschein is a Technical Advisor on the Dexis Monitoring, Evaluation, and Learning team.