Routine Data, Uncommon Evaluation: An Innovative, Mixed-Methods Assessment of Malaria Interventions in Nigeria
Since 2011, the U.S. President’s Malaria Initiative (PMI) has supported malaria interventions in Nigeria, including artemisinin-based combination therapies, intermittent preventive treatments for pregnant women, and long-lasting insecticidal nets. PMI also supports efforts to train health workers in malaria diagnosis and treatment.
But how effective have these interventions been in reducing malaria disease burden? To find out, the U.S. Agency for International Development (USAID) commissioned the Malaria Intervention Assessment (MIA) with funding from PMI and other implementing partners.
Objectives
To lead the study, USAID chose MEASURE Evaluation, a project it funds to improve and strengthen health information systems in developing countries. MIA has documented and assessed malaria control interventions from 2008 to 2016 in four Nigerian states: Cross River, Ebonyi, Nasarawa, and Sokoto.
MEASURE Evaluation partnered with Nigeria’s National Malaria Elimination Program to:
- Describe state malaria interventions;
- Record trends in key malaria prevention and case management indicators, and malaria morbidity and mortality in hospitals;
- Assess quality of care and monthly malaria data in PMI-supported and non-PMI-supported primary healthcare facilities; and
- Document changes in contextual factors, such as local culture, knowledge, and beliefs, likely to affect malaria interventions and outcomes.
Methods
In June 2015, the project team met with partners to design the protocol, which was later approved by a U.S. institutional review board and an ethics committee in Nigeria. The assessment used a quasi-experimental design and mixed-methods approach to capture data from:
- Health information systems at 560 primary healthcare facilities and their referral hospitals;
- Approximately 2,800 client exit interviews, 38 key stakeholder interviews, and observations of malaria interventions at the healthcare facilities; and
- Household surveys and document review.
With an approved design protocol in place, the team began training evaluators who would work in the field. “We first trained supervisors in Lagos,” says Ana Claudia Franca-Koh, a senior research, monitoring, and evaluation specialist leading the project for MEASURE Evaluation. “And then those supervisors facilitated statewide trainings of data collators and exit interviewers with support from our subcontractor, Nielsen Nigeria.”
Challenges
As is common with all research, the project team encountered challenges along the way. The first was to adjust to a major fuel shortage in Nigeria. Despite being one of the world’s biggest oil producers, Nigeria imports most of its fuel. In the spring of 2016, the country faced a severe fuel shortage due to outstanding debts to fuel distributors, a currency crisis caused by the slump in global oil prices, and the end of government fuel subsidies that had for years made gasoline cheaper at the pump.
“The fuel crisis made our fieldwork more difficult logistically,” says Franca-Koh, “but we managed to work with our subcontractor and get it done.”
Another challenge occurred when they discovered older health records were missing from healthcare facilities. “We had to send one data collator from each team to the state offices to pull data from the original registers there,” she says.
The team also had to shift their plans when exit interviewers could not find enough people with specific conditions—those who had come in with fever or pregnant women in their second or third trimester—to complete the five interviews needed from each facility. “We had to do revisits,” says Franca-Koh. “But because we had trained more exit interviewers than were originally called for, we could revisit the facilities to get the information we needed.”
Analysis
MIA is uncommon in its extensive use of routine health data, which is often considered unreliable due to inconsistent or incomplete records at health facilities. But in recent years, many countries, and Nigeria in particular, have focused on training health workers to complete their registers properly and strengthening the routine health system. Plus, Franca-Koh and other evaluators triangulated the data with exit interviews, in-depth stakeholder interviews, document review, and state-level analysis.
As they collated and used the routine data to report on malaria trends, Franca-Koh and her team also assessed data quality—how complete, timely, and accurate the information has been. “This assessment not only helps us understand our evaluation results,” Franca-Koh notes, “but also provides information for others who are trying to strengthen routine health data systems. We can all reflect on how well the system is doing.”
Sharing preliminary results
After completing fieldwork in early June 2016 and concluding data analysis this fall, Franca-Koh and her team shared preliminary results with malaria stakeholders in Abuja.
“We’ve engaged with stakeholders from the beginning,” Franca-Koh says. “Our partners participated in the training and visited us during fieldwork. It’s been a very collaborative, positive experience.”
On November 17, 2016, Franca-Koh presented MIA results at the 65th Annual American Society of Tropical Medicine & Hygiene Conference in Atlanta, Georgia. The presentation included key findings, strengths, and challenges of MIA, as well as lessons learned for improving malaria control interventions and the routine health information system.
“We are excited about the study results,” says Franca-Koh. “They should help guide future malaria programming and prepare for pre-elimination efforts in Nigeria.”
For more information
To learn more about MIA and its results, email measure@unc.edu. For more information about MEASURE Evaluation’s work to assess health system performance and impact in developing countries, visit https://www.measureevaluation.org/our-work/evaluation.
