You are here

Impact Evaluation Decision

Impact evaluations are important opportunities for learning, in a precise way, about the extent and intensity of project effects as well as about the degree to which those effects can be said to have been caused by a specific intervention. In a project Concept Paper's section on Monitoring, Evaluation and Learning, USAID staff are expected to identify the project interventions for which an impact evaluation will be undertaken, including project approaches that are being pilot tested or which involve development hypotheses that have not previously been tested.

Missions may conduct a project impact evaluation whenever they judge the opportunity for learning about the effects of a well defined intervention to merit this type of evaluation investment. USAID also requires that impact evaluations be carried out for pilot project and those which involved untested hypotheses or include approaches that are being considered for replication or scaling up. USAID's Evaluation Policy uses the term intervention, rather than program or project, when describing the focus of an impact evaluation.

The term "impact evaluation" involves a specialized meaning of the word impact. In common usage, impact means everything from high level results, in a very general sense, to long term outcomes from a project, to people-level outcomes. When joined with the term evaluation, however, "impact evaluation" is defined by most international development donors as a type of evaluation that involves a structured test of one or more hypotheses underlying a program or project intervention that involves a comparison between those affected by the intervention and a comparable group or area that was not affected by the intervention. The need for impact evaluations with these characteristics was highlighted for the development community in the Center for Global Development's publication When Will We Ever Learn. As USAID's review of Trends in International Development Evaluation Theory, Policy and Practices demonstrates, most donors, including USAID, have responded to the challenges of this report by formally introducing requirements for impact evaluations where their learning benefits are expected to be substantial.

Increasingly, impact evaluations that are being conducted in developing country environments are demonstrating that hypotheses testing that have advanced the development of effective medicines, agricultural practices can be cost-effectively applied to a much wider range of development assistance and partner government program interventions. The effect that this type of evaluation attributes to an intervention is the difference between these groups after exposure to the intervention. In some cases, differences between groups will be fully explained by an intervention; in others, both groups may improve as a function of factors to which everyone was exposed, but the group that received the project intervention will have improved more than the group that did not receive that intervention. Impact evaluation techniques are also being used to detect differences in the effects produced by two or more alternative approaches to solving a given development problem.

When considering whether to undertake an impact evaluation it is important to understand that in some circumstances this type of evaluation may not be appropriate.  For example, it may not be possible to conduct an impact evaluation, as USAID defines this term, when:

  • The intervention is a national policy or program for which no comparison group (counterfactual) can be identified.
  • The number of units (individuals, locations) available for assignment to groups that will and will not receive the intervention is small and may not be sufficient to accurately detect important differences between treatment and comparison groups.

To be sure that an adequate number of people or locations are available to be assigned to treatment and comparison group, it is wise to carry out a power analysis before investing in an impact evaluation that uses an experimental design or a very similar quasi-experimental design.