Secondary Logo

Journal Logo

Supplement Articles

Evaluating the systems used to monitor HIV populations accessing therapy and care in low-income and lower-middle-income countries

Hoskins, Susana; Jahn, Andreasb; Somi, Geoffreyc; Semenenko, Igord; Kirungi, Wilforde; Kaleebu, Pontianof; Phiri, Samg; Malyuta, Rusland; Fakoya, Adeh; Weller, Iani; Porter, Kholouda

Author Information
doi: 10.1097/QAD.0b013e32835bde24

Abstract

Introduction

The scale-up of HIV treatment programmes in low-income and middle-income countries (LMICs), where most of the epidemic is concentrated, is one of the greatest public health successes of the 21st century [1]. With the extension to life expectancy afforded by antiretroviral therapy (ART), it has become possible to consider HIV as a chronic disease which requires long-term monitoring [2]. Good chronic disease management is based on reliable information systems and a longitudinal record tracking a patient's data and health outcome [3–6]. Aggregated data from such systems can then be used to monitor the welfare of groups of patients in care and to generate reliable indicators to measure and benchmark the performance of a programme [7,8].

Monitoring the health status of patients on ART in LMICs was initially based on the manual aggregation of data from individual patient treatment cards [9]. However, as the number of patients receiving ART increased 10-fold between 2002 and 2007, the need to track the health of a growing number of patients forced monitoring systems to adapt accordingly [2,10,11]. ART clinic registers were introduced to capture health status at enrolment, ART initiation and details of follow-up appointments for patients under care in a facility. Programmatic monitoring became, and remains, based on aggregated register data reported in the form of totals, proportions, means and medians, at regular intervals [9]. These ‘indicators’ are usually calculated manually as electronic systems are rare.

The need for monitoring systems to adapt quickly to match the pace of expanding ART roll-out, and the number of stakeholders involved in various aspects of HIV care, each with differing rationales for programme monitoring, has resulted in complex monitoring systems. Recognizing this, international organizations and national governments have committed to developing a common monitoring and evaluation system throughout a country with appropriate mechanisms to assess progress [12,13]. In 2006, WHO published guidelines for monitoring patients on ART, and templates of standardized tools to collect and report such data were made available online [3]. More recently, the UNAIDS consolidated over 200 HIV care monitoring indicators from different organizations in an online indicator registry and published 25 core and 15 recommended indicators for HIV care [14,15]. Two of these indicators relate to assessing the health of patients on ART: the proportion known to be on treatment 12 months after its initiation and the percentage of estimated HIV-positive incident TB cases that received TB and HIV treatment. In order for national programmes to be able to compile and report against these two indicators, data must be captured at the patient and clinic level using the WHO-recommended monitoring tools. It is important, therefore, to review the extent to which these recommendations have been adopted into national monitoring systems.

In this paper, we present the results of the extent to which WHO-recommended monitoring tools and UNAIDS indicators have been adopted into the national monitoring systems in four LMICs and discuss some of the challenges of the monitoring systems used to assess the health of increasing numbers of patients accessing ART.

Methods

Through a consortium, Evidence for Action (EfA) [www.evidence4action.org], funded by UK Department for International Development, we gathered data items collected and reported on in HIV treatment and care programmes in Malawi, Tanzania, Uganda and Ukraine. The current study was part of a collaboration between the investigators of the three African countries and the UK to validate the indicators used to assess the outcome of HIV-positive individuals in treatment programmes. We widened the remit of the current study to include Ukraine for two main reasons. First, whereas the three African countries are classified as low-income economies, that is with gross national income (GNI) per capita of US$ 1005 or less, Ukraine is classified higher as a lower-middle-income economy (GNI of US$ 1006–3975). Second, in Ukraine, unlike Africa, injecting drug use is a major route of HIV transmission and different co-morbidities may, therefore, prevail. The system in Ukraine may, therefore, have different reporting needs and may also be in a better position to adapt quickly to international recommendations.

Our aim was to document and highlight similarities and differences between the data gathered and reported to ministries of health in each of the four countries and how these relate to WHO and UNAIDS recommendations. We, therefore, examined the contents of the following four records/reports in each country.

Patient record card: each patient will have one record card which includes their longitudinal health record of attendance at the clinic.

Clinic ART register: a record of all patients who have started on ART in a clinic is maintained in the register. Each patient is recorded in a row in the register and updated with patient ART attendance.

Cross-sectional report: aggregated data from the clinic is reported to the Ministry of Health in a cross-sectional manner, for example number started on ART in last quarter or to date.

Cohort report: aggregated data is reported for a cohort, for example number in care 12 months after starting ART.

We identified templates of the standardized monitoring tools from the Web site of the WHO for patient record card, clinic ART register and cohort monitoring report. No template was available for the cross-sectional report. We also retrieved the list of indicators recommended by UNAIDS through the United Nations General Assembly Special Session (UNGASS), available on the UNAIDS Web site. We did not examine reports used by external donors or other nongovernmental organizations. Our work focused on tools used to assess the health of the treated population and did not include tools used to monitor pre-ART care.

For each country, we compared each monitoring tool used against the recommended WHO ART programme monitoring tools and the inclusion of data required to calculate the UNGASS indicators. We also documented any additional data collected/reported in the four countries, which are not included in the WHO recommended tools, focusing on the current use at end of 2010. As no WHO tool was available for the cross-sectional reporting, we confined comparison of items reported by each of the four countries. Results are presented as numbers of data items/parameters used in country systems and as a proportion of those recommended by WHO, where a tool exists, and visually as a tick box for each item recommended or not recommended and its use in each country. Given the number of data items included in recommended and used tools, in order to aid readability, we broadly categorized items into whether they are collected at baseline (i.e. enrolment or ART initiation), as a primary outcome, a secondary outcome or at follow-up (i.e. at each appointment).

We compared the systems in the four countries at the end of 2008 and 2010, 2 years after the respective publications of the WHO monitoring tools in 2006 and UNGASS indicators in 2008, thereby allowing a minimum 2-year period for these tools and indicators to have been incorporated into country systems.

The clinic ART register data used in Malawi in 2008 were no longer available electronically and, in Ukraine, no clinic ART register or cohort reporting was used in 2008.

Finally, we report on the differences in age stratification used in reports.

Results

In 2008, Malawi, Uganda and Tanzania were using all the tools recommended in the WHO 2006 guidelines to monitor the health of patients on ART. Ukraine was using only a patient record card and a cross-sectional report. By 2010, all four countries were using all four types of ART programme data collection and reporting tools.

Specific data items included within reporting tools differed between countries, however. Table 1 summarizes the number of data items recommended for collection by WHO in their published tools. Although items included as proportions of those recommended are generally high, and have largely increased in 2010 compared with 2008, none of the four countries are collecting or reporting on all items recommended by WHO in any of the tools with the exception of Tanzania for cohort reporting. The degree of concordance among the four countries examined in specific items collected, however, is low given that in 2010 there was concordance on only 14 (48%), four (27%) and four (31%) data items collected by all four countries for patient record card, clinic ART register and cohort report, respectively (Table 1). Of interest, we noted that data collected in clinic ART registers in Uganda and Tanzania, as a proportion of those recommended in the WHO tool, was lower in 2010 than in 2008.

Table 1
Table 1:
Number of data items collected and reported by ministries of health within HIV treatment and care programmes in four countries in 2008 and 2010 as a proportion of those recommended by WHO tools.

Detail of the specific data items recommended by WHO and UNGASS and collected through patient record cards and clinic ART registers in the four countries in 2008 and 2010 is provided in Table 2. The only data items collected in patient record cards by the four countries in 2008 were patient demographics, weight (and height for children) and WHO stage or CD4 cell count at enrolment into care, and none in the clinic ART register. By 2010, the number of data items collected by all had increased. Much of the discordance of data collected by the four countries is within the items collected at enrolment where, of 11 items recommended for patient record cards, only four are recorded by all four countries, five by three countries and one by two countries in 2010. In 2010, of the four countries, only Uganda collected information on the HIV status of family members and Tanzania on functional status at enrolment. Of note, data needed to report on the two recommended UNGASS indicators were collected by all countries in 2010 except for loss to follow-up which was not being recorded in patient record cards or the ART clinic register in Ukraine.

Table 2
Table 2:
Specific WHO-recommended data items collected through individual patient record cards and clinic antiretroviral therapy registers in HIV treatment and care programmes in four countries in 2008 and 2010.

Within the ART register, in 2008, Uganda and Tanzania, the only countries with available data or using the tool, collected all the recommended data relating to ART initiation, outcome and treatments, with the exception of isoniazide prophylaxis which was not collected in Tanzania at the time. By 2010, all four countries were collecting the recommended ART initiation data on demographics, ART start date and regimen type. Remaining recommendations at initiation on WHO clinical stage or CD4 cell count, weight and height for children and functional status and recommended data at follow-up were recorded to varying degrees.

Table 3 provides details of the specific data items recommended by WHO and UNGASS and used for cohort and cross-sectional reporting to ministries of health in each country in 2008 and 2010. In the 2008 cohort reports, recommended primary outcome data, which are essential to calculate the ‘survival’ indicator as accurately as possible, were not commonly reported. Only in 2010 did the three African countries report the percentage of the original cohort alive and on ART, distinguishing between patients who had died, stopped treatment or become lost to follow-up. Ukraine did not report on losses to follow-up when calculating this key indicator. Adherence data were reported only from Tanzania and Malawi as number picking up drugs in 6 months and proportion missing any dose, respectively.

Table 3
Table 3:
WHO-recommended parameters reported on to ministries of health in four countries in 2008 and 2010.

The variability in data collected in the cross-sectional reports, both in 2008 and 2010, was high. Variation was in part due to reporting being done on the cumulative number of patients in care or on the current group of patients in care. Furthermore, although data on, for example the number of patients alive and on ART at the end of the reporting period, were recorded in all countries, they were not commonly disaggregated by sex and age. Although the number of patients still on a first-line regimen was recorded in all countries, only Malawi and Ukraine additionally reported the number on alternative first-line or nonstandard regimens.

A number of additional items, not included on the WHO recommendations, were collected and reported within the four countries (Table 4), only one of which was common to all four countries: CD4 cell count or percentage at ART initiation. These data items tended to reflect issues relevant within country, for example viral load data and information on injecting drug use in Ukraine, or to information relevant to costs, for example ART funding source, number of CD4 cell counts and additional laboratory measurements performed, number of adults with CD4 cell counts above a certain threshold and use of prophylaxis and treatment for opportunistic infections.

Table 4
Table 4:
Additional data items not specifically recommended in WHO template tools but used in Ministry of Health records and reporting in one or more countries (Malawi, Tanzania, Uganda and Ukraine) in 2008 and 2010.

We also noted that in the absence of patient outcome indicators specific to paediatric populations, reporting was disaggregated by age. However, different age bands were used across the tools by the four countries making comparisons problematic. For example, infants were categorized as aged less than 17 months in Malawi, less than 12 months in Uganda and Tanzania and included in the 0–3 years age band in Ukraine. Within the same cross-sectional reports, data on the number on first-line and second-line regimens were disaggregated into broad categories of ‘children’ in Malawi and aged 0–14 years in Uganda, Tanzania and Ukraine.

Discussion

Our work starkly highlighted, even by 2010, the lack of common use of recommended data in national tools used to monitor the clinical progress of the treated population. Additionally, we observed the collection of items not recommended in the templates or used to calculate UNGASS indicators, such as the number in a cohort picking up drugs for all 6 months, which closely resembled indicators recommended by international donors such as the Global Fund [16]. Some data items collected in 2008 were no longer collected in 2010, whereas others recommended in the templates were not taken up by any of the four countries. Sometimes data were country-specific relating to features of the epidemic such as those on injecting drug use in Ukraine. It is thus clear that national monitoring systems are designed not solely based upon one set of international recommendations but also include indicators required by other funders or organizations and also based on national experiences of which indicators are feasible and useful to collect.

Although the monitoring of HIV programmes can be used to identify a ‘good’ programme, enabling lessons to be learnt from one programme to improve the performance of others, the variety of indicators and age stratifications used make comparisons across programmes challenging. Moreover, this lack of consensus in number and type of priority data collected and reported across the range of monitoring tools highlights the uncertainty regarding which data collected at the patient-level, and which reported indicators are, in fact, most useful to assess the impact on the health of the treated population.

Indicators used to report on the health of the population in care have never been evaluated programmatically and it is not known which, if any, most accurately reflect the health of the treated population [17]. Consequently, it is not known which data items are important enough to be collected for indicator computation. Although measures such as CD4 cell count or weight gain may indicate improved immunological status at the patient level, the validity of using manually derived aggregates to assess the health of the treated population has never been examined. It is vital to understand which data reported by sites are actually useful in assessing the performance of a treatment programme. It would be of value to know for example, whether the data on the number on ART at end of a reporting period shown in Table 3, common to all four countries, are enough to monitor the progress of the treated population within ART programmes. However, as the internal, external, construct validity and predictive value of current indicators have never been evaluated, it is not known whether fundamental indicators, including survival and retention, capture the construct they intend to measure. This compromises the ability of programme managers to monitor their population in care accurately. Furthermore, as it is not known which indicators can predict the longer-term outcome of the patient population, it is also not known which key indicator(s) can enable managers to respond to potential predictors of failure early.

The lack of evidence for the choice of HIV monitoring indicators has resulted in complex and varied monitoring systems recommended by the multiple stakeholders involved in aspects of HIV care. HIV programme managers, donors, national ministries of health and international organizations, including the WHO, UNAIDS, PEPFAR and the Global Fund, each have differing rationales for programme monitoring. They have a responsibility to track patient status over time, monitor the performance of HIV programmes and detect immediate problems, evaluate a programme's impact and ensure accountability for funds. Over the years, this has resulted in a plethora of guidelines, recommendations, glossaries, tools and frameworks on how best to monitor HIV programmes in LMICs [15,18–25]. Associated with this has been the generation of a wealth of similar yet slightly different monitoring indicators, as demonstrated in the consolidation of over 200 HIV care monitoring indicators from various stakeholders in the UNAIDS online indicator register [14]. This highlights the uncertainty of how best to report on the health of the population in care which may explain the differences we found in our study.

The publication of the WHO tools in 2006 and of the UNGASS indicators in 2008 has been the most comprehensive step in ART monitoring standardization to date. However, in the absence of further ART outcome indicators recommended by UNGASS, programme funders, accountable for their spending, will continue to request feedback on multiple areas of ART care, resulting in a complex matrix of nonoverlapping or uncollectable data. Recently, assessments of pharmacy refill adherence have been shown to be as accurate as CD4 cell counts for detecting patients at high or low risk of virological failure [26]. This is an important step to evaluating the validity and predictive value of that specific indicator. It is now vital to assess the validity and predictive value of other indicators, to inform target setting and enable best practice for HIV treatment rollout.

Acknowledgements

The authors wish to acknowledge and thank Dr Saidi Kapiga (Mwanza Interventions Trials Unit, Tanzania) for his help in coordinating the data collection in Tanzania.

Author contributions: S.H., A.J. and K.P. conceived the study, A.J., G.S., I.S., W.K., P.K., S.P. and R.M. collated data items reported in the four countries and S.H. wrote the first draft and handled the revisions. All authors contributed to drafts of the manuscript and in presenting data comparisons.

Conflicts of interest

The manuscript is an output from a project funded by the Evidence for Action on HIV treatment and care systems (EfA) research consortium. EfA is funded by the UK Department for International Development (DFID), for the benefit of developing countries. The views expressed are not necessarily those of DFID.

The authors have no conflicts of interest to declare.

References

1. Geng EH, Bwana MB, Kabakyenga J, Muyindike W, Emenyonu NI, Musinguzi N, et al. Diminishing availability of publicly funded slots for antiretroviral initiation among HIV-infected ART-eligible patients in Uganda. PLoS One 2010; 5:e14098.
2. Horberg MA, Aberg JA, Cheever LW, Renner P, O’Brien Kaleeba E, Asch SM. Development of national and multiagency HIV care quality measures. Clin Infect Dis 2010; 51:732–738.
3. WHO. Patient monitoring guidelines for HIV care and ART; 2006. http://www.who.int/hiv/pub/ptmonguidelines.pdf. [Accessed 10 February 2010]
4. Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ 2010; 341:c4564.
5. Bussmann H, Wester CW, Ndwapi N, Vanderwarker C, Gaolathe T, Tirelo G, et al. Hybrid data capture for monitoring patients on highly active antiretroviral therapy (HAART) in urban Botswana. Bull World Health Organ 2006; 84:127–131.
6. Wagner EH. Chronic disease management: what will it take to improve care for chronic illness?. Eff Clin Pract 1998; 1:2–4.
7. Fraser HS, Jazaveri D, Nevil P, Karacaoglu Y, Farmer PE, Lyon E, et al. An information system and medical record to support HIV treatment in rural Haiti. BMJ 2004; 329:1142–1146.
8. Nash D, Elul B, Rabkin M, Tun M, Saito S, Becker M, et al. Strategies for more effective monitoring and evaluation systems in HIV programmatic scale-up in resource-limited settings: implications for health systems strengthening. J Acquir Immune Defic Syndr 2009; 52 (Suppl 1):S58–S62.
9. Harries AD, Gomani P, Teck R, de Teck OA, Bakali E, Zachariah R, et al. Monitoring the response to antiretroviral therapy in resource-poor settings: the Malawi model. Trans R Soc Trop Med Hyg 2004; 98:695–701.
10. UNAIDS. Report on global AIDS Epidemic; 2008. http://www.unaids.org/en/KnowledgeCentre/HIVData/GlobalReport/2008/2008_Global_report.asp. [Accessed 10 January 2010]
11. Hill A, Pozniak A. A normal life expectancy, despite HIV infection?. AIDS 2010; 24:1583–1584.
12. UNAIDS. ‘Three Ones’ key principles, ‘Coordination of National Responses to HIV/AIDS’: Guiding principles for national authorities and their partners. In: Proceedings of the UNAIDS Conference; 2004; Washington; Paper 1. http://data.unaids.org/una-docs/threeones_keyprinciples_en.pdf. [Accessed 9 September 2001]
13. United Nations General Assembly Political Declaration on HIV/AIDS. Resolution 60/262; 2006. http://data.unaids.org/pub/Report/2006/20060615_hlm_politicaldeclaration_ares60262_en.pdf. [Accessed 10 February 2010]
14. UNAIDS. Online Indicator Registry. http://www.indicatorregistry.org/public.aspx?view=home&public=1. [Accessed 10 February 2010]
15. UNAIDS. Monitoring the Declaration of Commitment on HIV/AIDS: Guidelines on construction of core indicators, 2010 reporting; 2009. http://data.unaids.org/pub/manual/2009/jc1676_core_indicators_2009_en.pdf. [Accessed 10 January 2011]
16. Global Fund. Monitoring and Evaluation Toolkit; 2004. http://www.theglobalfund.org/en/me/documents/toolkit/. [Accessed 2 November 2011]
17. Ross-Degnan D, Pierre-Jacques M, Zhang F, Tadeg H, Gitau L, Ntaganira J, et al. Measuring adherence to antiretroviral treatment in resource-poor settings: the clinical validity of key indicators. BMC Health Serv Res 2010; 10:42.
18. WHO. Monitoring and reporting on the health sector's response towards universal access to HIV/AIDS treatment, prevention, care and support 2009–2010. In: WHO Framework for global monitoring and reporting; 2009. http://www.who.int/hiv/universalaccess2010/hiv_me_framework_2009-10.pdf. [Accessed 6 August 2012]
19. WHO. HIV Drug Resistance Early Warning Indicators: WHO indicators to monitor HIV drug resistance prevention at antiretroviral treatment sites; 2008. http://www.who.int/hiv/topics/drugresistance/technical_information/en/index.html. [Accessed 13 January 2012]
20. WHO. A guide to indicators for monitoring and evaluating national antiretroviral therapy programmes; 2005. http://www.who.int/hiv/pub/me/en/naparv.pdf. [Accessed 6 August 2012]
21. UNAIDS. Glossary: Monitoring and Evaluation terms; 2010. http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/11_ME_Glossary_FinalWorkingDraft.pdf. [Accessed 6 August 2012]
22. UNAIDS. A national evaluation agenda for HIV; 2010. http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/9_3-National-Eval-Agenda-MEF.pdf. [Accessed 6 August 2012]
23. UNAIDS. 12 Component monitoring and evaluation system strengthening tool; 2010. http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/1_MERG_Assessment_12_Components_ME_System.pdf. [Accessed 6 August 2012]
24. PEPFAR. President's Emergency Plan for AIDS Relief, Planning and Reporting: Next generation indicators reference guide; 2009. http://www.pepfar.gov/documents/organization/81097.pdf. [Accessed 6 August 2012]
25. Global Fund to fight AIDS Tuberculosis and Malaria. Monitoring and Evaluation Toolkit: HIV, Tuberculosis and Malaria and Health Strengthening Systems; 2009. http://www.theglobalfund.org/en/me/documents/toolkit/. [Accessed 6 August 2012]
26. Bisson GP, Gross R, Bellamy S, Chittams J, Hislop M, Regensberg L, et al. Pharmacy refill adherence compared with CD4 count changes for monitoring HIV-infected adults on antiretroviral therapy. PLoS Med 2008; 5:e109.
Keywords:

indicators; low-income countries; monitoring and evaluation; WHO

© 2012 Lippincott Williams & Wilkins, Inc.