How a Functional Health Information System Can Improve Program Evaluation

Heidi Reynolds, right, at the annual American Evaluation Association meeting.

By Heidi Reynolds, PhD, MPH, Director of Evaluation, MEASURE Evaluation

WASHINGTON, DC—A functional health information system (HIS) can improve program evaluation in two ways: one, by producing data that can be used for evaluations, and two, by making those data easily accessible.

Let’s delve in to that first point: a functional HIS produces data that can be used for evaluations. Evaluations often collect primary data—for example, interviews at the household or program beneficiary level. Evaluations also can draw on data that are collected for other purposes, such as client records, program monitoring data, or surveys. All those data sources—including primary data from evaluations—are among the data available in a HIS.

To illustrate how existing data sources can be used in evaluation, let’s take the example of routine data. Typically, these are data produced at the facility and community levels that gather individual client care information and are aggregated to show characteristics about service use and quality. Routine data can be used for evaluation on their own or in combination with other data sources. For example, census data can help with the calculation of population denominators. Civil registration and vital events data help provide information about impact when questions relate to program effects on births and deaths. Other outcomes—HIV prevalence, or contraceptive prevalence rates, for example—might be obtained via a household survey or modeling. But without information about program exposure, surveys alone are insufficient to understand program effects or impact.

For many HIS in low- and middle-income countries, this level of functioning—producing all these data sources with high quality—is a goal, but not yet a reality. However, as the HIS strengthens, these analyses will be increasingly feasible.

And about that second point: a functional HIS makes data easily accessible for use in evaluation. In a lower-functioning HIS, which is more consistent with current reality for many countries, an evaluator wanting to use existing data often must go from source to source to obtain the data. For example, the evaluator may need to visit facilities to abstract client records, request that the ministry of health provide access to routine health data, or email the researcher to see if s/he will share their data. Once the data are obtained, there may still be steps required to clean the data, check data quality, create new variables, and organize the databases to facilitate analysis. Those steps consume resources in terms of costs for data collection, management and analysis, and time.

With the proliferation of electronic tools—such as mobile phones, electronic health records, and open-source, cloud-based applications such as DHIS 2—data are increasingly available, sometimes in real time. Digital health platforms are facilitating access by allowing data from different data sources to be interoperable—such as data from health programs for HIV and tuberculosis—or in different systems, such as logistics and facility data. When data are interoperable, they’re easier to use. Making data interoperable takes efforts at all levels. For example, the national level of an HIS would need to define data standards, rationalize indicator definitions, and standardize data collection forms.

Other advances that are facilitating access include the global push for open data. It is increasingly possible to make an online request to an organization for permission to download cleaned data sets with supporting documentation (see, for example, the Demographic and Health Surveys or the Carolina Population Center Dataverse). Platforms are being designed and implemented to make more data available for evaluation purposes and facilitate access to those data. For instance, Johns Hopkins University’s National Evaluation Platform (NEP) works at the country level to identify relevant program or policy questions; to assist the evaluator to identify and compile relevant datasets; and to conduct analyses and produce communication materials. The Tanzania Open Data Portal makes data from a variety of sectors available in machine-readable formats and includes visualizations from those data. The future functional HIS will see more digital health platforms that improve data access by aggregating information from multiple sources and combining those data in a way that makes them ready for use in evaluations.

Investments in strengthening HIS will ensure that they produce good-quality and timely data across all data sources, plus provide that data in ways that are helpful for evaluation. These investments would not only assist in making evaluation more possible and less costly, they are good investments for the future efficiency of HIS and have the potential to decrease the need for expensive special studies.

For more information on MEASURE Evaluation’s work on evaluation, see https://www.measureevaluation.org/our-work/evaluation. Also visit the Health Information System Strengthening Resource Center at https://www.measureevaluation.org/his-strengthening-resource-center.

Read additional blogs from Evaluation 2017 on big data and equity in evaluation.