USAID Evaluation Policy Emphasizes Important Links to Project Design
For each project, consideration will be given during the design phase to the performance evaluation(s) and, in some cases, impact evaluation(s) that will be undertaken. Identifying key evaluation questions at the outset will both improve the quality of the project design, and will guide data collection during implementation.
USAID's call for evaluation questions in a project Concept Paper iterates a parallel requirement for a CDCS. Both of these requirements reflect USAID's evaluation policy commitment to improving the degree to which evaluation is an integral element of the program cycle.
When developing project evaluation questions for inclusion in a Concept Paper, project Design teams may find it useful to use the Project Evaluation Questions Worksheet included in this kit section to identify questions by levels in a project's Logical Framework on or other issues, as illustrated below.
|Project Purpose and Sub-Purpose
||What types of improvements and/or types and level of benefits do assisted firms attribute to their involvement with this project ? disaggregated by male/female firm owners and workers?
|Project hypotheses/ theory of change
||What evidence do the records of assisted firms provide concerning a causal linkage between a reductions in the time and cost of moving goods across borders and an expansion of their exports/imports?
||What has been the project experience to date with the retention of trained customs staff assigned to land border crossings and what does retention experience suggest in terms of the sustainability of project services and benefits?
When developing questions that might be asked in project evaluations, it is important to remember that USAID undertakes two types of evaluations: performance evaluations and impact evaluations, as defined below.
- Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. Any activity within a project involving untested hypotheses or demonstrating new approaches that are anticipated to be expanded in scale or scope through US Government foreign assistance or other funding sources will, if feasible, undergo an impact evaluation.
- Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. Operating units are required to conduct at least one performance evaluation of each large project it implements; where large is defined as larger than the average for the unit's portfolio.
For performance evaluations, which are carried out at a single point in time, rather than over the course of an intervention, the timing of a proposed evaluation has implications for the types of questions to be asked. For example:
- In mid-project evaluations process questions may help identify opportunities for improving performance, as may questions about the realism of the project design, adherence to the project's budget and schedule, and early indications about the achievement of expected results and the validity of project assumptions.
- Questions about project results, including any differential impact on men and women, or their effects on youth or other target populations are likely to be answered more fully in an end-of-project evaluation, as are questions about a project's actual cost-effectiveness compared to project design stage expectations.
- Questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project's services and benefits were sustained.