Including an analysis in a Project MEL Plan of the need for evaluations during the project (tied to some threshold or key decision) and at the end of the project (either for decisions or to capture learning) lays the foundation for allocating sufficient evaluation resources and planning in a way that allows the use of the best methods for quality evaluation.
Performance evaluations, as defined in ADS 201, encompass a broad range of evaluation methods. They often incorporate before–after comparisons but generally lack a rigorously defined counterfactual. Performance evaluations may address descriptive, normative, and/or cause-and-effect questions. Performance evaluation questions may include, but are not limited to, the following topics:
No single evaluation design or approach will be privileged over others; rather, the selection of method or methods for a particular evaluation should principally consider the appropriateness of the evaluation design for answering the evaluation questions as well as balance cost, feasibility, and the level of rigor needed to inform specific decisions.
In recent years, studies carried out for PPL/LER have shown that:
Illustrative Types of Performance Evaluation
There is no commonly accepted taxonomy of performance evaluations on which all evaluators would agree. The range of evaluation questions addressed by performance evaluations is wide, and includes questions about project or activity results or outcomes; implementation processes and their effectiveness; what has been sustained since a project or activity ended; how cost effective was the program compared to existing practice or another approach; was the project or activity viewed as being relevant, or given positive ratings by intended beneficiaries; were men/women, or elderly, or poor, differentially affected by the project or activity. The range of data collection methods used may be as expansive as the list of questions, and thus many performance evaluations are self-described as Mixed Method Evaluations. Using a mix of methods, evaluators undertake all of the sub-types of performance evaluations described below – and more.
Process or Formative Evaluations
Process or formative evaluations are most often undertaken mid-way through USAID activities. Process evaluations focus on how the activity is working and whether expectations about implementation and beneficiary response to the activity are in line with expectations. Both process and formative evaluations may include questions about the initial results of an activity, such as what goods/services have been delivered, or to what degree outputs have been achieved, such as teachers trained, or new seed varieties planted. While USAID does not have a Technical or How To Note on Process Evaluation, guides produced by other organizations are helpful for understanding this type of performance evaluation and where it may be useful, including the Evaluation Brief on Conducting a Process Evaluation that can be downloaded here.
Outcome or Summative Evaluation
Among USAID "final" evaluations, most fall into the outcome or summative sub-types of performance evaluations, and address questions that focus on whether planned results and targets were achieved, as well as whether activities had unintended consequences. While many evaluations in this cluster are single point in time studies, some are more formal "pre-post" evaluations that fund both a baseline and endline round of data collection, but only for the activity’s intended beneficiaries. This performance evaluation subtype does not include a comparison group, which is one of the characteristics that differentiates it from an USAID impact evaluation. USAID outcome and summative evaluations tend to consider results at all levels of an activity or project Logical Framework, i.e., whether outputs were produced, to what degree purpose was achieved, and whether any change in the status of the goal could be detected. UNDP, which has a useful guide on Outcome-level Evaluation, uses a narrower definition of the term "outcome", focusing on what USAID would call the purpose and goal levels of an activity or project, but not the outputs.
What Has Been Sustained? (Ex-Post)
The presence of this question in a performance evaluation tends to signal that it is an ex-post evaluation, often with an outcome focus, though in some cases USAID may be just as interested in whether services or processes were sustained beyond the funding period for the activity as they are in whether benefits to particular beneficiaries were sustained. Performance evaluations in this cluster can be asked to empirically determine what was sustained, which is generally not possible in mid-course or final USAID evaluations that as whether an activity is likely to be sustained? The latter involves a hypothetical, and often the best that evaluators can do is determine whether protocols that would help sustain an activity are in place. Since sustaining services and benefits from an activity often involves the people who live in a project or activity location and will continue to live there, USAID’s Local Systems Framework, a systems thinking in evaluation, more generally, can be helpful for structuring this type of performance evaluation. Given the retrospective nature of this type of evaluation, it may also be important to be able to reconstruct baseline data, which the World Bank paper highlighted under this heading addresses.
In the trade arena, the European Commission uses ex-post evaluations to examine the results of regulatory changes, such as evaluations of Free Trade Agreements.
In its 2016 update of USAID’s evaluation guidance in ADS 201, USAID introduced a requirement for evaluations that look beyond the results of a single activity implemented by a single partner, calling upon each Mission to conduct at least one "whole-of-project" evaluation during the lifetime of a CDCS. As USAID’s ADS 201 Additional Help paper indicates, evaluations in this cluster are characterized more by their scope and questions than by the mix of techniques they use to gather information.
Whole-of-Project Evaluation Questions
The following are illustrative evaluation questions for Missions to consider and revise per their learning needs.
To examine the contribution from all constituent parts of a project to the Project Purpose:
To examine strengths and weaknesses of the project theory of change:
To examine the interaction among activities as they contribute to the Project Purpose:
Causality Questions in Performance Evaluations
The presence of questions about cause and effect in a performance evaluation does not define a cluster of evaluations, in the same way as do the other headings above. Questions about causality are found in USAID final evaluations as well as in ex-post evaluations. What draws attention to them in a performance evaluation is the need this kind of question creates for an approach for dealing with causality that does not involve a counterfactual comparison of the results for activity beneficiary to the results for some rigorously constituted comparison group. Over the years, a variety of evaluation approaches for addressing questions about cause-and-effect have emerged for use in situations where an impact evaluation is either not feasible or will not be undertaken for other reasons. This range of techniques is often grouped under the term “non-experimental designs.” Illustrative of the nature of these evaluation techniques are the following.
A number of these techniques are treated in greater detail in a volume from 3ie entitled Addressing Attribution of Cause and Effect in Small N Impact Evaluations: Towards an Integrated Framework, which is highlighted on this page.
Performance Evaluation Staffing
As ADS 184.108.40.206 states, all required evaluations must be external evaluations, meaning that that Team Leader will be an independent expert from outside the Agency, who has no fiduciary relationship with the implementing partner for the project or activity to be evaluated. The requirement for evaluations to be external applies all three of USAID’s required types of evaluations.
Beyond this, USAID's evaluation policy encourages USAID staff as well as evaluators from partner countries to serve as members of evaluation teams. More generally, USAID guidance and experience indicates that, on occasion, USAID may elect to undertake an evaluation on a joint basis, together with its country partner or other donors. Evaluations of this type require close coordination at a number of points, and may require that both USAID staff and the evaluation team dedicate more time to this type of effort than might be expected for other evaluations. Similarly, when USAID elects to undertake a Participatory Evaluation, in which beneficiaries play a more active role, additional evaluators and USAID staff time may be required to facilitate this process. Decisions about team composition for mid-term and final evaluations have M&E budget implications that are worth considering when the evaluation component of a Project M&E Plan is developed.
Other options USAID may wish to consider include joint evaluations conducted collaboratively with other donors or country partners. Useful Guidance for Managing Joint Evaluations has been developed by the OECD and can be accessed through the line above.
Performance Evaluation Methods
Decisions about the specific data collection and analysis methods to be applied to address performance evaluations questions are an important step in the planning of every performance and impact evaluation, but the need not be made prematurely.
USAID requires that the main types of evaluations to be undertaken during a CDCS period be specified in a PMP, noting that PMPs can and should be updated, as needed, to reflect additional evaluations a Mission schedules in response to facts that “trigger” supplementary evaluations during the CDCS implementation period. Indicating in a PMP which evaluations will be performance evaluations and which will be impact evaluations, including the identification of one whole-of-project evaluation that will be undertaken, provides USAID with an early basis for understanding whether all of the Agency’s evaluation requirements will be met.
At the Project MEL stage and in Activity MELs, USAID may decide to identify specify subtyles of performance and impact evaluations they are considering. Data collection methods associated with performance indicators are needed at these stages in the program cycle, but ADS 201 does not focus on the data collection and analysis methods for evaluations until it describes the required elements of an evaluation Statement of Work (SOW) and even there it is flexible saying that USAID should:
Specify proposed data collection and analysis method(s) or request that prospective evaluators propose data collection and analysis method(s) that will generate the highest quality and most credible evidence on each evaluation question—taking time, budget, and other practical considerations into account.