USAID evaluation training courses for Agency staff anticipate that whenever a decision is made to undertake a Performance Evaluation or an Impact Evaluation, someone on USAID's staff will be designated as the Evaluation Manager for that effort. This may be the Operating Unit's Evaluation Point of Contact, or it may be another staff member. This individual helps ensure that evaluations are carried out in a manner that is consistent with USAID's Evaluation Policy.
The more an Evaluation Manager understands about USAID’s evaluation process and where quality control opportunities exist within that process, the more likely it is that they will be able to ensure that the evaluations they oversee meet USAID’s standards. To this end, it is useful to conceptualize the evaluation process as consisting of a series of distinct stages, each of which includes specific steps and evaluation products. While any division of the evaluation process into stages is arbitrary, the five stages shown in the table below help to isolate blocks of time over a process that may, in total, take six months to a year or more to complete, depending on the specific evaluation. A more detailed version of the Evaluation Management Process table can be accessed from this page. To complement this five stage evaluation process table, the kit provides an Illustrative Evaluation Manager’s Checklist, which is featured on this page.
|The Evaluation Management Process – Summary View|
|Stage 1||Stage 2||Stage 3||Stage 4||Stage 5|
|Decision to Evaluate to Issuance of SOW||Proposal Review to Approval for Data Collection to Begin||Support During Data Collection and Analysis||Initial Evaluation Results Briefing to Final Report||Dissemination of Final Report; Post-Evaluation Review/Decisions; Action on Evaluation Findings|
One of the new elements of the USAID evaluation process is an Evaluation Dissemination Plan. This product, introduced in USAID’s Evaluation Policy is produced by USAID. An initial version is to be created before an Evaluation Statement of Work (SOW). Producing a preliminary Evaluation Dissemination Plan early in the evaluation process helps ensure that an evaluation SOW reflects USAID’s needs for evaluation products in hard and soft copies as well as oral briefings for various audiences to support evaluation dissemination and utilization.
The expanded Evaluation Management Process table also includes several steps that are described as Quality Control Checkpoints (QQC). Quality Control Checkpoints (QCC) are opportunities during the evaluation process for reviewing work completed to date with an eye towards identifying evaluation quality issues early enough to correct them. As the graphic below suggests, at least one such opportunity exist in Stage 1, 2 and 4 of the evaluation process. QQCs in this graphic, it should be noted, reflect “best practices” in evaluation, but are not checkpoints identified or referred to in USAID’s Evaluation Policy. Two of the checkpoints included in the graphic below (QCC#1 and QCC # 4) are, however, already considered to be routine steps in USAID’s evaluation process: namely a review of an evaluation SOW and a review of a draft evaluation. To strengthen its quality control at these two steps USAID has developed an Evaluation SOW Review Checklist keyed to ADS 201 and an Evaluation Report Review Checklist based on its Evaluation Policy and ADS 201, both of which can be filled out online in subsequent sections of this kit.
Two additional Quality Control Checkpoints included in the diagram above are less consistently part of the USAID evaluation process. They can easily be added by an Evaluation Manager. One of the best ways to do this is to incorporate these QQCs into an Evaluation SOW’s list of deliverables.
QQC#2 Desk Review/Inception Report and Detailed Evaluation Plan: USAID guidance stresses that evaluation should build on what is being learned from performance monitoring and that before an evaluation team begins its field work there is a well developed evaluation design/methodology in place including instruments and a data analysis plan. This QQC is intended to make sure that these steps are completed before the start of field work is authorized.
A requirement for USAID approval of an evaluation team’s desk/inception report and detailed evaluation plan prior to the start of field work can be written into an Evaluation SOW as a milestone. While delay at the start of an evaluation is not ideal, it may sometimes be the best way to ensure that a final evaluation product meets USAID standards.