Impact Evaluations of Large-Scale Public Health Interventions: Experiences from the Field

Author(s): Skiles MP, Hattori A, Curtis SL

Year: 2014

Abstract:

Agencies are articulating the need to causally attribute health outcomes to investments in an era of shrinking resources and increasingly complex development environments. The opportunity to uncover essential information for program planning and resource allocation is a strong motivation for impact evaluations. Findings from large-scale impact evaluations (IEs) can be instrumental for decision making, yet they are not without challenges and costs. 

In this paper, we share field experiences from a number of evaluation studies undertaken during MEASURE Evaluation Phases II and III. A series of case studies highlight design and implementation challenges that required creative solutions to move forward; plus analysis across studies revealed common reoccurring themes and valuable lessons. Examples of these cross-cutting themes presented include: challenges with identification and selection of program beneficiaries, random assignment in complex environments, identification of a robust comparison or control group for estimating the counterfactual, heterogeneity of program impacts, timing of baseline data collection, and absence of baseline data and a counterfactual. 

Field experiences from MEASURE Evaluation project demonstrate the need for transparency and collaboration among the key partners, the inevitable balancing of technical requirements with programmatic priorities, and the flexibility required to adapt designs in order to answer the most valuable evaluation questions. Interest in accountability of funding of public health interventions continues to grow, promising continued interest in IEs. Evaluators, implementers, and funders can share in these learnings as we move forward with expanding our understanding of the costs and benefits for rigorous evaluations.