Warner-Smith, Matthew MPH; Rugg, Deborah PhD; Frescura, Luisa MSc; Moussavi, Saba MPH
In 2001, Member States of the United Nations, along with civil society groups, including people living with HIV, convened in the General Assembly for a Special Session on HIV/AIDS. The session led to the adoption of a Declaration of Commitment (DoC) to intensify efforts to prevent HIV infection and to increase the quality and coverage of services for people living with HIV and affected by AIDS. The Declaration stipulated that progress should be reviewed at a high-level meeting every 2 years.1
The Special Session and the DoC coincided with a period of intensified attention on HIV in the international political arena. In 2000, a Global Summit on Development agreed on the Millennium Development Goals for 2015, including the halting and reversal of the HIV epidemic (goal 6). In 2004, the World Health Assembly committed to ensure that 3 million people needing antiretroviral therapy would be receiving that treatment by 2005 “3 by 5”. Although the goal of 3 million by 2005 was not met, treatment services expanded very significantly and the target number of 3 million people was reached in 2007. At the high-level meeting on HIV of 2006, Member States adopted the principle of Universal Access to HIV prevention, treatment, care and support, expanding on the 2006 commitment to universal access to treatment adopted by the G8 nations.
In the DoC, targets were set in selected priority areas with the intention of catalyzing intensified programming. The targets included increasing young people's knowledge about how to prevent HIV transmission (2005 target: 90%; 2010 target: 95%); improving the effectiveness of efforts to prevent mother-to-child transmission of HIV (2005 target: reduce infant infections by 20%; 2010 target: reduce infant infections by 50%, ensure 80% of women have access to prevention mother-to-child transmission services); and reducing HIV prevalence (2005 target: reduce prevalence by 25% in the most affected countries; 2010 target: reduce prevalence by 25% globally).1
After the adoption of the DoC, the United Nations Secretary-General charged the Secretariat of the Joint United Nations Programme on HIV/AIDS (UNAIDS) with the responsibility of monitoring the implementation of the Declaration. This article describes the system that was established for monitoring progress made in national responses to HIV and discusses response rates and the quality of data received (Fig. 1). This article does not discuss the findings of United Nations General Assembly Special Session on HIV/AIDS (UNGASS) reporting, which are described elsewhere.2-5
The UNAIDS Secretariat, using the technical expertise of its Monitoring and Evaluation Reference Group, has established a system for the regular monitoring of national HIV epidemics and countries' policy and programmatic responses to those epidemics. The resultant UNGASS reporting system is based on the biennial submission of Country Progress Reports against a set of core indicators.
The UNGASS indicator set was designed to cover the full range of key HIV programs and to also address the important structural issues of fiscal resources and the policy environment. The set is described in Guidelines on the Construction of Core Indicators.6-9 The performance of this indicator set was reviewed after each round of reporting and revised in response to these reviews and to new programmatic developments (Appendix 1). In recognition of the fact that not all the UNGASS indicators are necessarily applicable to each national HIV epidemic, the 2008 reporting round required that countries state whether each indicator was relevant to their particular context.
The Guidelines on the Construction of Core Indicators6-9 describe a process for reporting which recommends that each country should conduct data needs assessments; establish reporting plans; process data (including cleaning, validation and entry into a single database); conduct data vetting and triangulation workshops to obtain consensus on the values to be reported; and achieve consensus among stakeholders on the final report to be submitted. Indicators are reviewed by UNAIDS' Monitoring and Evaluation Reference Group after each round of reporting and any revisions and/or additions to the core set of UNGASS indicators are included in the subsequent guidelines (Fig. 1).
Before 2003, there was no established system for the routine reporting of indicator data on national multisectoral responses to HIV. The first round of reporting used paper-based reporting forms which countries submitted by mail. An electronic data management system, the Country Response Information System (CRIS), was released in 2004. This included the revised indicators for 2005 reporting and permitted the production of a standard electronic indicator data file to accompany the submission of the narrative section of Country Progress Reports. Electronic submission using CRIS obviated the need for translation and for manual data entry.
In 2007, a relational database was developed specifically for the management of data submitted in Country Progress Reports. This Global Response Database now includes cleaned quality-assured data from 3 rounds of reporting on the full set of UNGASS indicators.
Technical and Financial Support
Before 2004, no systematic technical assistance was provided for reporting on national multisectoral responses to HIV. In 2004, UNAIDS recruited, trained and deployed technical field staff who were mandated to support the development of systems for the production, synthesis, interpretation and reporting of data on national responses to HIV. These staff would also support Member States in reporting on progress made with respect to the DoC. In addition to the deployment of technical monitoring and evaluation advisors, direct financial assistance was provided on request to low-income and middle-income Member States to support reporting. To facilitate the provision of technical assistance for national reporting, training in the preparation and submission of Country Progress Reports, including the use of the CRIS and UNAIDS' recommended methods for epidemiological estimations, was provided to monitoring and evaluation staff from monitoring and evaluation consultants.
A communications strategy was implemented targeting 3 different audiences associated with reporting: political leaders, technical staff and international supporting partners. They were informed via correspondence of the reporting process and were provided with the guidelines and reporting software. Within the United Nations system, joint correspondence was sent from UNAIDS, the World Health Organization (WHO) and the United Nations Children's Fund (UNICEF) to country offices informing them of the recommended process for reporting and emphasizing the importance of coordinated support from the United Nations system to national counterparts. Technical information, including the guidelines and the UNGASS reporting edition of CRIS, was placed on the website and mailed to national AIDS coordinating bodies and UNAIDS country offices. A dedicated UNGASS reporting email address was used as a portal for managing requests for technical advice.
Data Quality Assurance
Country Progress Reports received by UNAIDS were systematically checked for calculation errors, illogical values and missing data fields. When issues were detected, they were discussed with technical staff from the relevant Member State. Quality issues that were resolved were documented and data in the Global Response Database were updated accordingly.
Due to differences in the timing and coordination of data collection in different countries, data reported to cosponsor United Nations agencies and partners may differ from those received at UNAIDS. Data received in Country Progress Reports were therefore reconciled against data for the same indicators reported by national health authorities to the WHO and data reported to UNICEF. Where discrepancies were identified, national governments and United Nation country staff were contacted to ascertain the reason for the discrepancy and to identify a single commonly accepted value for that indicator. To ensure comparability of data across countries, it was necessary to publish estimates obtained through the UNAIDS-endorsed estimation methodology and values obtained through the Demographic and Health Surveys (DHS) alongside country reported estimates for indicators 4, 5, 7, 10, 13, 15, 16 and 17 (Appendix 1).
Reporting Response Rates
Of the 189 Member States of the United Nations who adopted the 2001 DoC, 102 submitted a Country Progress Report in the first round of reporting in 2004 (54%). By 2006, the number of Member States in the United Nations had grown to 191, and 122 of them submitted reports (64%). In the most recent round of reporting, the number of Member States had risen to 192, and the number of reports received in time for inclusion in the Global Report increased to 147 (77%) (Fig. 2). In addition, use of UNAIDS data entry software increased from 62% to 73% between 2006 and 2008 reporting rounds.
Reporting in the 3 UNGASS rounds to date has been consistently the highest among countries with generalized epidemics (Fig. 3), followed by countries with concentrated and low-level epidemics. The percentage increase in reporting between 2004 and 2008 reporting rounds is similar across the 3 epidemiological levels, with 26% for generalized epidemic countries, 22% for concentrated epidemic countries and 25% for low-level epidemic countries.
Reporting has also remained consistently highest in low-income countries, 89% of which reported in 2008 compared with the 82% of middle-income countries and 53% of high-income countries that reported.
In 2008, the most frequently reported indicators were the National Composite Policy Index (with 137 reporting countries), followed by indicators measuring clinical services such as blood safety, antiretroviral therapy, and prevention of mother-to-child transmission. The indicators related to men who have sex with men, sex workers and injecting drug users remain the least frequently reported, particularly for prevention programs among injecting drug users (with only 21 countries reporting data in 2008).
Country-reported relevance suggests that, although most indicators are considered pertinent to countries' respective national HIV epidemics (Fig. 4), data may not always be available for global reporting (although the data may be available at the national level). For example, more than 80% of reported indicators related to men who have sex with men, sex workers and injecting drug users are identified as relevant, yet data is available for only 40% of those indicators. Data availability was highest for service delivery indicators that measure the level of provision of care and treatment, followed by national survey-based indicators which are representative household surveys measuring demographic characteristics and behavioral patterns.
The UNGASS reporting process emphasizes country ownership of data with the aim of strengthening national monitoring and evaluation of HIV programs. Therefore, the onus of data collection, cleaning, validation and aggregation rests with each reporting country. Once country-reported data reach the global level, quality assurance is limited to the reconciliation of data values from different sources and the correction of illogical values. In 2008, 107 reporting countries (73%) used UNAIDS data entry software which reduced the time required for data processing at the global level and potential data entry errors. In addition, 122 countries were contacted regarding data reconciliation issues. Clarification was received from 88% of these countries regarding coverage for antiretrovirals and 50% for other indicators. Although the quality of source data (in terms of measurement and processing errors) cannot be assessed at the global level, there are several other dimensions of data quality that pertain to the UNGASS dataset, namely accessibility, transparency, relevance, interpretability, coherence, timeliness and completeness.
“Accessibility” refers to the ease with which data can be obtained by the user. Country Progress Reports are posted without alteration on the UNAIDS website. Cleaned and validated data are then disseminated in the Report on the Global AIDS Epidemic10-12 and a variety of other publications, including The Secretary-General's Report13 and articles in the peer-reviewed literature.
“Transparency” refers to the clarity of documentation regarding assumptions, methods and limitations of the data. Comprehensive guidelines on reporting, including full indicator definitions, are posted on the UNAIDS website and are disseminated widely in print and electronic versions.
“Relevance” is concerned with whether the information sheds light on issues of interest to potential users. The UNGASS indicator set was developed for monitoring the extent of progress of national responses to HIV and the scope of its users, and uses, are far reaching. International organizations, governments, civil society and the private sector use the data to shape international and national political discourse and to inform strategic and programmatic decisions.
“Interpretability” refers to the availability of information to aid users in understanding and analyzing the data. Countries are requested to report metadata, including measurement tools used, the dates of data collection, the sample sizes for survey-based indicators and descriptions of methodologies used for data collection. In 2008, 85% of reported indicators had sufficiently complete metadata (metadata was considered complete if the data collection tool, dates of data collection and, where applicable, sample size were all reported). That figure reflects both a positive response to the need for higher quality data and refinements made to the UNAIDS data collection tool which ensure that these fields are completed upon submission.
“Coherence” pertains to the ability to compare methodologically consistent data between countries and across years. Although there have been 3 successive rounds of UNGASS reporting, refinements in indicator definitions and methods mean that not all indicators are directly comparable across time. Currently, 6 of the 23 core UNGASS indicators have been collected without alteration for at least 2 reporting years. However, for UNGASS 2010 reporting, all indicator definitions have remained unchanged (Appendix 1) and data will be comparable for all indicators between 2008 and 2010. A review of the metadata reported in 2008 (indicators were considered consistent if the reported measurement tool, period of data collection and additional country comments indicated the methodology was consistent with the indicator definition outlined in the UNGASS guidelines) found that more than 60% of reported indicators in 2008 had methodologically comparable data.
“Timeliness” refers to the lag time between data collection and data dissemination. All Country Progress Reports received by UNAIDS are posted unaltered on the UNAIDS website, making the raw data immediately available to the public. Cleaned, validated and analyzed data is disseminated within 6 months of receipt by UNAIDS, through the publication of the Report on the Global AIDS Epidemic. In addition, the UNGASS guidelines provide recommendations on the frequency of data collection for each indicator. Differences in the timeliness of data reported to UNAIDS are evident according to type of data collected (Table 1). For example, 90% of population-based indicators reported in 2008 were timely, compared with 28% for impact indicators where annual data collection is recommended.
Completeness of reported data is an essential aspect of data quality because it enables analysis and interpretation and allows for more valid conclusions to be drawn. The reporting of disaggregation by either sex, age or school level is recommended for 19 of the 23 national program indicators, and approximately 80% of data for these indicators is reported with at least partial disaggregation. For 5 UNGASS indicators where sex disaggregation was requested in both 2006 and 2008, the reporting of such disaggregation increased by 20%. In 2008, 82% of reported data for these 5 indicators was disaggregated by sex, compared with 62% in 2006.
The UNGASS reporting system has strong national ownership of reporting and is based on a broad consultative process across both government sectors and civil society. The development of the reporting system required a combination of high-level political commitment and extensive technical collaboration between various elements of the United Nations system and bilateral and multilateral development agencies, national governments and civil society.
Although the system was developed to meet the reporting requirements of the United Nations General Assembly, it was also intended to provide a catalyst for the development of systems for monitoring national responses to HIV. At the time of the Special Session, very few countries had systems that could provide a clear overview of progress across the breadth of a comprehensive multisectoral response to HIV.3 Country Progress Reports had the potential to provide such a perspective and to stimulate the development of systems for collecting and synthesizing data from a range of sources and the dissemination of those data.
It is noted that reporting is only 1 component of comprehensive monitoring and evaluation systems,3 one that is arguably of less immediate utility to national responses than other components. Indicator-based monitoring data, and other forms of information such as evaluation-based data and program reviews, are of greatest value when used within national contexts for resource mobilization, strategic planning and program management.
Although it is impossible to determine what effect the UNGASS reporting process has had on the course of HIV epidemics, it has the potential to enhance national HIV responses by promoting data collection where routine systems to monitor HIV and AIDS may not yet exist; providing methodological guidance and tools to collect the highest quality information possible; and encouraging the use of this data for policy and programmatic planning.
The observed increase in response rates, combined with an increase in the quality of reported data, probably reflects both the increasing levels of support provided to the process and an increasing awareness of UNGASS reporting. These improvements can be seen as an endorsement of the value national partners place on reporting.
Despite its widespread endorsement, the reporting system does have many limitations. Most prominent among them is the lack of data to report; the poor quality of some of the reported data; and, possibly, political indifference to reporting on particular populations. Most of these shortcomings stem from the underlying limitations of the systems used to generate the data and from the technical difficulties inherent in such data collection. This is particularly true of those indicators related to most-at-risk populations, where data collection continues to be challenged by the methodological difficulties of accessing hidden and heavily stigmatized populations. However, not all the quality issues evident in the reported data can be attributed to the limitations of the underlying systems. An example of quality issues independent of the system are cases where discrepancies were found between data submitted to different agencies for the same indicator for the same period, and where data was known to be available and yet was not reported. Both of these issues suggest a lack of coordination between different institutions that are responsible for collecting and disseminating data within countries and the entities responsible for reporting that data through international processes. Strengthened coordination and consultation efforts at national level could yield substantial improvements in the quality and extent of reporting by addressing those 2 issues.
A second limitation of the reporting system is the difficulty inherent in ensuring the validity of the data reported. Data validity issues start with questions about the suitability of the indicator for collecting valid data. The UNGASS indicators balance considerations of data availability with considerations of the technical merits of the measure. Indicator selection and development was undertaken as a collaborative effort with international development agencies and national governments specifically to find an appropriate balance between these 2 considerations. Measurement of the coverage of prevention services for injecting drug users is a prime example of this. The indicator uses a superficial definition of service coverage to allow for data collection through a relatively simple cross-sectional survey. It has been argued that a more rigorous definition of prevention services (including all the key elements of harm reduction) should be used and that the measure should tally all those who receive these services as a proportion of the total population of injecting drug users. However, such an approach would rule out the vast majority of countries that do not provide a full complement of harm reduction services, that cannot reliably track which individuals receive which services, or that are unable to derive reliable estimates of the size of their injecting drug user populations.
The validity of reported data is of more interest than the validity of the indicators per se. National ownership and government accountability underpin UNGASS reporting, which fundamentally is a process for Member States to report to the General Assembly on their fidelity to a political commitment. As such, data are limited to what countries report, which raises questions in some quarters about the validity and impartiality of these data. Although international partners cannot address these concerns by assuming the responsibility of governments in primary data collection, they can play a role in strengthening data validity by promoting national monitoring and evaluation capacity building and in encouraging in-country consultations on the validity and reliability of the data that is to be reported.
Questions about data validity emerge when reported data seem to stray from global trends. These outliers, which may often be due to methodological differences in data collection, pose a challenge for data analysis and the resulting messaging which is presented in the Report on the Global AIDS Epidemic. Approaches to deal with these outliers include the removal of methodologically inconsistent data from analysis and the calculation of medians rather than means to account for wide data ranges.
A broad formal consultation to review the UNGASS reporting system is to be undertaken after the 2010 round of reporting, coinciding with the 10th anniversary of the DoC. This review could be used to identify indicators that may have higher validity. However, any increases in validity obtained through the replacement of indicators must be weighed against the loss of trend data that arises from changing indicators. Having consistent, global, time-series data on the HIV response remains a key strength of UNGASS, as it allows for the collection of a wide range of information on the HIV response. Even allowing for imperfect measures, this breadth of information can inform overall direction in planning and policy and can be used as vital markers of the HIV response.
Country Progress Reports are official Member State submissions to the United Nations and political considerations require that these data be accepted at face value at the international level. Nonetheless, the reporting system includes mechanisms for vetting the validity of reported data. Foremost among these is a recommended consultation process for data vetting at national level before submission of the data to UNAIDS. This process is effectively a peer review that assesses whether the values to be reported are based on the most robust data available. At the global level, the data cleaning process increases data validity through a reduction of illogical values. Finally, reconciliation against other data sources and the resulting dialogue with Member States to identify reasons for discrepant values has been found to improve data validity, such as identifying instances where data tables were misread or where an incorrect indicator was used.
Despite these processes, greater attention to data validity would strengthen the reporting system. It has been suggested that establishing data audit trails for internationally reported data may lead to improved data quality.14 This is an approach that warrants further investigation. The potential benefits of such data audit trails would need to be balanced against the increased resources this would require and possible losses of data that otherwise would have been reported had it not been for the lack of data audit trail.
Although the 2 approaches are not mutually exclusive, external data validity checking may be more effective and more cost effective in stimulating more rigorous reporting than establishing data audit trails. This approach would have the additional benefit of modelling good practise in monitoring and evaluation systems and would assist Member States in strengthening their monitoring systems. As with data audit trails, this approach has major resource implications. Hence the efficacy of this approach would need to be demonstrated through pilot testing before being implemented systematically.
There also remain a number of challenges at the international level. Foremost among these is the need to bridge the gap between a demand for predictable routine reporting requirements and the relatively less predictable political demand for data. This tension has led to what is widely acknowledged as an insufficient amount of time between the end of a reporting cycle and the due date for reports.
This demand for high quality data at the international level also creates a tension between data quality and national ownership. If data quality is the only consideration in the preparation of global reports, it would be most efficient to obtain data directly from the original source of origin or from the collation of data from a variety of secondary sources. However, national ownership and the strengthening of national systems are important values that underpin UNGASS reporting. This dictates that national governments should take responsibility for ensuring both the availability of data in the UNGASS reporting system and the quality of that data.
A final limitation of the UNGASS system is the low-response rate from high-income countries. This stems from a variety of possible factors, including a perception that UNGASS reporting relates exclusively to low-income and middle-income countries and a possible lack of awareness on the part of relevant technical agencies about the need to report. Although these issues are not necessarily exclusive to high-income countries, the lack of a presence of international development partners in these countries limits the opportunities for addressing such misperceptions.
The UNGASS reporting process is a mechanism for monitoring global trends in the HIV response through the collection and dissemination of consistent global data across time. The process features a strong political mandate, a broad base of support from the international development community and civil society, and a high degree of national ownership. Consistent increases in reporting rates and the quality and completeness of data received suggest that the process is gaining strength as a reporting system and has become an established benchmark for international monitoring of the HIV response. Despite these strengths, the process faces the continued challenges of data validity, data vetting and impact evaluation.
The continued strengthening of national capacity for data collection, monitoring and evaluation and the continued provision of normative guidance and tools to support collection of high quality data are viable methods to overcome challenges identified in the process. This is particularly true for indicators pertaining to difficult-to-reach populations such as sex workers, men who have sex with men and injecting drug users.
Promoting and supporting data vetting in-country before submission of Country Progress Reports, encouraging collection of disaggregated data and encouraging countries to report more details regarding the methodology used to collect their data could increase data quality and facilitate better assessment of data consistency across countries during analysis. Finally, data analysis and data use could be strengthened through triangulation of data reported through the UNGASS process with other sources of global data pertaining to HIV.
The authors wish to thank all the individuals who were involved in the preparation and submission of Country Progress Reports and those who reviewed and provided feedback on the early draft.
© 2009 Lippincott Williams & Wilkins, Inc.