Seeking Consensus on How to Measure the Impact of Family Planning Total Market Approaches
- Flickr: UN photo - Bamako, Mali, by Marco Domino. Women and children from the community wait in line to receive their free consultations.
However, because TMA combines multiple players in various ways across disparate country settings, it can be difficult to develop evidence that demonstrates what combinations of players, strategies, and activities work best.
Achieving the aim to describe the optimum mix requires a larger evidence base and the development of agreed-upon methods to assess the medium- and long-term impact of TMA on FP outcomes. To answer this need, the United States Agency for International Development (USAID)-funded MEASURE Evaluation project produced a guide to help implementers to plan for evaluations of their TMA activities.
The Guide for Assessing the Impact of a Total Market Approach to Family Planning Programs, published in October 2019, seeks to compare definitions of TMA that implementers are using; to identify objectives and components of current or past TMA implementations; to outline methods for conducting an impact evaluation of TMA; and to use an example from Cambodia to demonstrate how a TMA might be evaluated for impact.
Developing and implementing TMA activities are still in their infancy and, therefore, few studies exist to guide an impact evaluation. This new guide offers a step-by-step approach for evaluating TMA activities. The guide complements earlier published guides and includes a discussion of a rapid assessment technique that can avoid the high cost of long-term studies.
The authors conducted a systematic review of current and past implementations of TMA and supplemented those findings with key informant interviews among implementers and supporters of TMA. They offer six recommendations for those who are planning TMA activities:
- Clearly identify specific objectives that will determine which indicators to prioritize. It is advantageous if these objectives are documented in public documents such as a national reproductive health policy.
- Be aware that an interrupted time series design is likely to be the most feasible rigorous design. It can be supplemented with an artificially constructed comparison group using pre-intervention trends to project what likely would have happened in the absence of TMA (i.e., a counterfactual projection).
- Ensure there are enough pre-intervention data points to enable reliable estimates of pre-intervention trends. If these are not available, the authors recommend a focus on evaluating if the TMA activity achieved its stated objectives rather than attempting to evaluate its net impact.
- Try to compress the time between the last pre-intervention data point and the start of TMA activities to avoid a wide time gap that may confound measuring differences pre- and post-intervention. If this isn’t possible, the authors recommend a rapid assessment survey of key outcome measures immediately before the start of TMA activities.
- To get timely information about the impact of TMA, the authors recommend conducting annual or biannual rapid assessment surveys post-intervention to enable a timelier start of the impact evaluation and create more data points, which will also make the impact estimates more robust.
- TMA implementers should systematically document when any events took place in their program (e.g., price increases on commodities). This will help later in interpreting an interrupted time series design. Likewise, it is important to document external confounding factors that took place that may have affected outcome measures.
