Matching the gold standard: comparing experimental and nonexperimental evaluation techniques for a geographically targeted program

Link to Document:
ja-10-114

Author(s): Handa S, Maluccio JA

Year: 2010

Abstract:
We compare non-experimental impact estimates using propensity score matching with those from a social experiment to determine whether this non-experimental approach can match the gold standard. The social experiment we use was carried out to evaluate a conditional cash transfer program implemented in Nicaragua in 2000. The outcomes we assess include total and food expenditure and a variety of children's health outcomes including vaccinations, morbidity, and breast feeding. We find that PSM does better at replicating the benchmark for individual outcomes but does poorly for expenditure outcomes. Judicious choice of sample improves the performance of PSM for all outcomes. A more detailed analysis of the components of expenditures shows the degree of bias is related to the importance of the item in the household budget and persists even when differences in prices and consumption habits are controlled for by comparing households from the same geographic region. The PSM technique seems most promising for evaluating individual, and easily measured outcomes, such as those related to child schooling and health, but less so for more complex outcomes such as expenditures.