USAID uses various methods to help ensure that USAID evaluation reports, and the evaluation SOWs that generate these reports, are of high quality, including peer reviews undertaken as these product are developed and PPL/LER organized technical audits and random checks on policy compliance. Missions and other operating units can also monitor and work to improve evaluation quality using simple meta-evaluation tools.
An important goal of USAID’s evaluation policy is to raise the quality of evaluations on a Mission-wide basis, or for all trade or economic growth evaluations across a region or bureau. Monitoring the quality of clusters of evaluations, or the SOWs used to produce clusters of evaluations, is fairly simple and the results, even over the course of year, can provide USAID operating units with important feedback on their strengths and weaknesses.
Most systems for monitoring evaluation quality – or meta-evaluation as this is often called – involve the use of checklists that capture some, if not all, important characteristics of SOWs or evaluation reports being produced by an operating unit. To conduct a meta-evaluation, either by looking backwards over a stack of evaluations that has accumulated, or to monitor evaluation quality on a running basis, scoring each evaluation as it appears, adding the scores to a simple Excel data base, and watching trends emerge over time, Missions already have the basic tools.
When a checklist approach to meta-evaluation is used, the results can be visually displayed in a way that quickly informs users about patterns across a set of evaluations – what checklist items are consistently handled well in Mission products, and which checklist items are not always in line with USAID expectations, as the highly simplified illustration of a checklist based meta-evaluation below demonstrates. Meta evaluation checklists of this sort have been used internally by PPL/LER (and its predecessors) off and on over the past several decades to understand evaluation quality. Missions and other operating units can also monitor the quality of their evaluation reports, and evaluation SOWS, in this way.
|Evaluation Checklist Meta-Evaluation Items||Mission Evaluation Report (ER) Checklist Scores for 2013||Sum of Correct Scores|
|ER #1||ER #2||ER #3||ER #4||ER #5||ER #6||ER #7||ER #8||ER #9|
|2-3 page executive summary mirrors contents of report||7|
|Evaluation methods and instruments are included||6|
|Study limitations are explicitly stated||7|
|All evaluation questions are addressed||7|
|The findings sections makes clear which methods data came from||4|
|Findings-Conclusions-Recommendations progression is easy to follow||6|
|Recommendation are specific as to action and who is to take the action||5|