Throughout several African countries, various agencies, including community-based organizations (CBOs), have been using the CSI. Although the extent of CSI utilization is unclear, a 2009 survey distributed to OVC organizations elicited responses from 18 organizations operating in 9 countries.6 Of these, most were nongovernmental organizations.
The primary motivation for developing the CSI was to establish a tool for widespread use. Thus, it is critical to evaluate the tool's ability to generate valid information regarding child vulnerability. This study is an independent assessment of the construct validity of the CSI.
Study Site and CSI Training
The study was conducted in Malawi, where HIV prevalence is 12% among adults aged 15-49 years, and the OVC population is approximately 1.8 million or 18% of all children.7 Mchinji is a rural district in eastern Malawi, where most households rely on subsistence farming.
Before study commencement, we conducted community outreach with village leaders to explain the study's purpose. In August 2009, 6 CBOs that served children living in family care had sufficiently large child populations and expressed interested in utilizing the CSI were identified to participate in the study. The CBOs agreed to be trained to use the CSI and collect CSI scores. Two staff members from each CBO, mainly directors or coordinators, were trained over 6 days by an independent, experienced CSI trainer. Each CBO obtained CSI scores for the children on their program rosters.
To assess construct validity of the CSI, we developed two instruments, the ‘Comprehensive Child Welfare’ (CCW) surveys, one for 5-year to 11-year olds and one for 12-year to 17-year olds. To determine whether the CSI would yield similar information as globally-recognized indicators, the CCW surveys included validated instruments or best practice indicators in the same domains and factors as the CSI (Table 2). The CCW surveys were age specific and differed in developmentally appropriate ways. Initial drafts of the surveys were refined with input from stakeholders, including United States Agency for International Development personnel, CSI tool developers, and our Malawi-based research team. The instruments were translated into Chichewa (Mchinji's local language), back translated into English, and reviewed for accuracy. Final revisions occurred after pilot testing.
Sampling Frame and Population
CBOs serving OVC in Malawi typically have a client population of 100-150 children. Using a 95% confidence level, a 5 percentage point margin of error, and power of 80%, we anticipated needing a minimum of 80 children in each age group to detect statistically significant differences in the correlation coefficients for the CSI and CCW scores at or above 0.28, very near the threshold for a weak correlation. We added about 20 children to each sample to ensure sufficient statistical power.
Our sampling frame included all children served by the 6 CBOs described above who had been scored using the CSI. Using the CBO rosters, we stratified children by age, gender, and CBO and selected a random sample of children in each age stratum. For each age group, we aimed to enroll equal proportions of male and female children. The final recruited sample included 102 children aged 5-10 years, with their caregivers, and 100 children aged 11-17 years.
The study was reviewed and approved by the Boston University Ethical Review Board and the Malawian Ministry of Health.
Our research team consisted of 4 local research assistants from the Center for Social Research, University of Malawi, who had extensive experience in OVC-related research. The team was trained to administer the CCW surveys. To reduce the likelihood that changes in the children's welfare would bias the findings, data collection using the CCW surveys took place in September to October 2009, immediately after CBO collection of CSI scores in early September. Surveys were conducted in the local language in children's homes. Children aged 11-17 years were surveyed and measured in private to encourage candid responses. For younger children, we interviewed the child's main caregiver and then measured the child.
Data from the CCW surveys were entered by the local research team into a CSPro version 4.0 database (US Census Bureau, Washington, DC)8 and analyzed in Boston using Statistical Analysis Software version 9.0 (SAS Institute, Cary, NC).9 We performed separate age-specific analyses. We retained the CSI 0-3 scoring system employed by the Mchinji CBOs, in which 0 = “good” and 3 = “very bad”.
To measure construct validity,10 or the degree to which a scale correlates with the concept it attempts to measure, we selected items from the CCW instruments that corresponded to each of the CSI factors and compared the 2 sets of data using Spearman Rank Correlation Coefficients. This approach yields ranked coefficients, without requiring that the variables have a linear relationship. If the 2 variables increase together, the coefficient is positive; if, conversely, 1 increases as the second decreases, the coefficient is negative. Perfect correlation yields a coefficient of 1.00; an inverse relationship yields a coefficient of −1.00; no relationship yields a coefficient of 0. We utilized data from single questions, composite scores comprised of multiple questions, and variables we constructed such as “underweight Z scores”, a frequently used measure of the deviation of a child from the mean weight-for-age (Table 3). We used this standard interpretation: a coefficient of 0.00-0.30, 0.30-0.70, or 0.70-1.00 indicates a weak, moderate, or high correlation, respectively.11
For both CSI scores and CCW data, we calculated frequencies for categorical data and means for continuous data. We also calculated cross tabulations for each factor to better understand the discordance between the CSI scores and CCW data.
Sample Characteristics and CSI Scores
Both samples of children contained approximately the same number of males as females. Among the younger children, nearly one-half were aged 9-10 years and just over one-third were aged 5-6 years. The number of children per age was more evenly distributed in the older group of children.
Table 4 provides the CSI scores for each factor, by age group. For nearly all factors, the majority of children in both age groups were rated as “fair” or “good”.
The Spearman Rank Correlation Coefficients for selected comparisons are provided in Table 5. Generally, correlation coefficients were weak and insignificant. Below, we briefly present our main findings by CSI domain and factor.
Domain 1: Food Security and Nutrition
For the first CSI factor, the CCW food insecurity composite score had a moderate (0.40) and significant correlation (P < 0.0001) with CSI “food security” scores. Were the 2 measures highly correlated, the CCW food insecurity composite scores would steadily increase as CSI scores decline from 0 to 3 (from “good” to “very bad”). The CCW “food insecurity” score ranged from 0-9. Children rated by the CSI as “good” had an average score of 6.7; children rated as “bad” had a score of 7.9, while children rated as “very bad” had a score of 7.0.
The correlations for the second factor, “nutrition and growth”, were weak primarily because CSI scores failed to identify children with poor nutrition, as reported by caregivers in the CCW. In fact, of the children with “good” or “fair” CSI scores, more than 75% of caregivers reported 6 or more of the following issues for the child or household in the previous 4 weeks: (1) worrying about insufficient food; (2) not eating preferred foods; (3) limiting variety of foods; (4) eating what they did not want to; (5) eating less than preferred; (6) eating fewer meals; (7) having no food in the household; (8) going to sleep without food; and (9) going a day or night without food. More importantly, the expected positive relationship between CSI scores and physical height was absent, as children with “fair” and “bad” CSI scores had virtually the same mean height-for-age Z score, −1.80 and −1.79, respectively. Similarly, no 11-year to 17-year olds were scored by the CSI as “very bad” on nutrition and growth, despite the fact that 25% were stunted (≥2 SDs below mean height for age).
Domain 2: Shelter and Care
Correlations for the first factor, “shelter”, were weak for both age groups. Cross-tabulation analysis showed that 30% of children scored as “good” or “fair” for shelter by the CSI had 2 or more indicators of bad housing, including the following: grass roofing without plastic (which does not always keep inhabitants dry during the rainy season), mud floors only, and no toilet facilities. However, some children with “very bad” CSI housing scores only had 1 indicator of bad housing.
For the “care” factor, the correlations between CSI scores and a “bad care” composite (and individual indicators) from the CCW surveys were virtually uncorrelated in both age groups. Among the 47 11-year to 17-year olds, who were scored as having “good” care by the CSI, 40% reported not getting enough to eat when food was available; 38% reported having to wear dirty, torn clothes, even though there were ways of getting better/new ones; 32% reported not being cared for when sick; 36% reported feeling uncared for; and 19% said they felt as though there was never anyone looking after or helping them when care was needed most. At the same time, 5% of children with a “bad” or “very bad” CSI care score reported none of these issues.
Domain 3: Protection and Abuse
For “abuse and exploitation”, there were weak correlations (0.07-0.20 for 11-year to 17-year olds) between CSI scores and CCW indicators. In the older age group, no child was scored as “bad” or “very bad” by the CSI. However, the CCW data showed substantial proportions of children that reported the following “sometimes” or “many times”: being (1) screamed at (36%); (2) called names or cursed (36%); (3) spanked or beaten with an object (24%); (4) pushed, grabbed, or kicked (22%); (5) locked out of the house (22%); (6) threatened with abandonment (21%); and (7) told they should never have been born or should die (14%).
Similarly, all comparisons yielded weak correlations on the second factor, “legal protection”. The vast majority of younger children were scored by the CSI as having “good” legal protection. However, among children receiving a “good” CSI score, 12% worked more than 20 hours per week; 1 child worked over 40 hours per week. Among caregivers of younger children, 77% reported that the child's birth was unregistered and most did not know that births should be registered.
Domain 4: Health
We found weak to moderate correlations between CSI scores and CCW health and chronic illness indicators, albeit with some major discrepancies (Table 5). Among children with “bad” or “very bad” CSI health scores, 8% were reported in “excellent” health by their caregivers on the CCW. Of 35% of 5-year to 10-year olds who were stunted (n = 36), only 1 had a “very bad” CSI score and 8 were scored as “bad”. Among older children, 11% reported being sick for more than 1 month, but nearly all of these children received “good” or “fair” CSI scores. Although 25% of older children had height-for-age Z scores over 2 SDs from the mean, only 1 was rated as having “bad” health.
The correlations between CSI scores and corresponding CCW indicators were low (0.05-0.08) and statistically insignificant for the second factor, “healthcare”. CBO staff scored 80% of younger children as having “good” or “fair” health care with the CSI. Among younger children scored as having “good” health care, 22% of caregivers reported 5 or more indicators of bad or no health care, including that the child had not received vaccinations or care for the last fever, and that they would not seek treatment if the child had blood in his/her stool; could not drink; was breathing fast; was injured; or had trouble breathing.
Domain 5: Psychosocial Wellbeing
For the factor, “emotional health”, we found no meaningful correlations between the CSI and CCW indicators (Table 5). Among younger children, 89% received “good” or “fair” CSI scores. However, the mean Child Depression Index (CDI) scores—which increase with greater emotional distress—were 4.8, 5.8, 4.1, and 4.0, respectively, for children scored on the CSI as “good”, “fair”, “bad”, and “very bad”. Thus, children with the most severe emotional distress, as reported by caregivers, were rated as having “fair” emotional health by CBO staff using the CSI. Furthermore, on the full scale, CDI scores ranged 0-16. Among sampled children, 25% scored beyond the midpoint, indicating that they experienced 8 or more indicators of emotional distress, including experiencing feelings of sadness, worry, or being withdrawn or unable to concentrate. The caregivers of children with “good” or “fair” CSI scores reported various specific emotional problems for these children, including: the child did not have fun (22%); the child did not want to be with people (16%); and the child cried all the time (15%).
This pattern was repeated in the older age group. Although 94% received CSI scores of “good” or “fair” emotional health, children reported multiple signs and symptoms of emotional distress. The mean CDI scores were similar for children scored “good,” “fair,” and “bad” by the CSI (39.0, 38.6, and 35.2) (the range was 10-42). Children with CSI scores of “very bad” had a mean CDI score of 50.0, which may be appropriate. However, among children with “good” or “fair” CSI scores, substantial proportions scored above the midpoint for various issues as follows: negative mood (21%); ineffectiveness (15%); anhedonia (loss of joy/pleasure in life) (13%); and 27% scored above the midpoint for the total composite CDI score, indicating emotional distress.
For the second factor, “social behavior”, correlations were also weak between CSI scores and all CCW indicators. More than 80% of children in both groups were scored as “good”, yet numerous problems were evident from the CCW data. Among older children, 26% scored above the midpoint for problem behavior and yet none were rated as “bad” or “very bad” by the CSI.
Domain 6: Education and Life Skills
The CSI scores for “performance” and CCW data on grade for age were uncorrelated. We found that 66% of children aged 11-17 years who were enrolled in school, yet behind by 2 or more grades, were scored as having “good” or “fair” performance. Among 5-year to 10-year olds, 6% of children behind in grade by 3 years or more were scored as having “good” or “fair” performance. Finally, the CSI score for “education and work” and the CCW data on school enrollment had low to moderate correlations for younger and older children, respectively.
In this study, we were unable to find strong correlations between CSI and CCW scores in either age group. Most correlation coefficients were low and weak. In some cases, the relationships were the inverse of what would be expected. A sharp disconnect emerged between the picture created by the CSI scores—one of generally healthy children who were only vulnerable on food security and shelter factors—and the picture that emerged from the CCW survey data. From the latter, we found children that displayed substantial vulnerability on numerous measures of health and well being, including abuse, health care, and emotional health. The CSI scores did not identify children who reported living in severe distress in 1 or more domains.
There are several possible explanations for this disconnect. We believe the most important issue is that the CSI tool combines multiple, rather than unitary, concepts in each factor. Thus, it is unclear which concept individual users are capturing in the score assigned to each child. A valid tool must capture unitary concepts to yield valid and reliable results. The CSI tool attempts to gather too much information in each domain. For example, under the factor, “abuse and exploitation” (Table 1), a score of “fair” is defined as “there is some suspicion that [the] child may be neglected, over-worked, not treated well, or otherwise maltreated” and a score of “bad” means that the child “is neglected, given inappropriate work for his or her age, or is clearly not treated well in household or institution”. Each score combines several different problems and includes the concepts of suspicion versus certainty, and severity. If a child is rated as “fair”, it is not clear whether there is suspicion that the child is neglected or overworked or if the child experiences mild or infrequent abuse.
Cultural and situational factors may also limit the ability of CBO staff to collect accurate data. In the Malawi context, food insecurity is common, households generally have inadequate shelter, and children are often disciplined sharply. Still, different cultural expectations do not fully explain the weak correlations. In some cases, the correlation coefficients were negative, indicating the opposite of what would be expected if both sets of data ranked the same children from better to worse.
Additionally, it is possible that the CSI training for CBO staff was inadequate. However, the CBO staff expressed confidence in their ability to use the CSI-tool.6 Moreover, if the CSI tool was used widely, the quality of the training would vary due to the limited number of CSI trainers. Tool designers envisioned a situation where CBOs would be able to download the tool and use it after reading the manual. This study suggests that it may not be feasible for CBO staff to use the CSI to gather accurate information without training or even with the current amount of training offered.
Another possible explanation for the discordant findings is that the caregivers and older children in this study provided inaccurate information. However, our data collection team had considerable experience conducting research in this population and had been trained to detect discrepancies between reported and observable indicators. They found that caregiver reports seemed consistent with their surroundings and that children were forthcoming with issues such as abuse and depression when asked in private and in a sensitive manner.
We acknowledge that a limitation of the study was the absence of a rigorous validation of the indicators within the CCW surveys in Malawi—an undertaking that was precluded by time and funding constraints. However, the CCW surveys were developed by incorporating a number of previously validated child-oriented tools and best practice indicators used by United Nations Children's Fund, United States Agency for International Development, The World Health Organization, and The World Bank. Our approach was to accurately assess the welfare of children and then compare the resulting data with the CSI scores from the same children. This was the strongest approach possible. We are confident in the capacity of the CCW surveys to capture accurate data. Moreover, we used an exhaustive analytic approach that included comparing CSI scores with single and composite indicators. We only lacked numerous comparison options with the education factors, where we were limited by few well-validated indicators. We acknowledge that stunting as a marker of health and food security may be limited, particularly in older children, where for example, stunting at any early age may not necessarily indicate poor nutrition or health in adolescence.
Although this was a single evaluation in one country, our findings can nonetheless be useful in furthering the discussion regarding the effectiveness of the CSI. Based on our findings, we recommend the following measures. First, we recommend caution by prime users—particularly CBOs, nongovernmental organizations that support CBOs, and policy makers at all levels—in interpreting CSI scores with regards to detection of children in extreme situations, priority setting, and the evaluation of services. Second, the CSI tool should be revised. One approach would involve focusing each factor on a single issue to permit the clear identification of children in extreme distress. Third, we advise strengthening the training of CBO staff through increased explanation of scores and practice. Finally, after revisions, further validation research of the CSI is warranted to ensure that the tool can effectively be used to understand child vulnerability.
The authors gratefully acknowledge the assistance and support of the numerous individuals who made this work possible. We thank members of the OVC-CARE team at the CGHD, in particular Malcolm Bryant, Jonathon Simon, and Deirdre Pierotti, for their help in implementing each stage of this project. We acknowledge the important support received from Andrea Halverson, Christian Fung, Beverly Nyberg, Krista Stewart, and Gretchen Bachman at USAID, and from Karen O'Donnell and Florence Nyangara at Measure Evaluation. Our sincere gratitude is extended to our research assistants in Malawi: McDonald Chitekwe, Allan Dyless, Emmanuel Kambalame, and Zione Themba. Finally, we are very grateful to the village heads, CBO staff, and caregivers and children who generously spent time learning about this project and providing information about their lives. Without their support, this project would not have been possible.
1. United States President's Plan for AIDS Relief. Orphans and other vulnerable children programming guidance (July 2006). Available at: http://www.pepfar.gov/guidance/78161.htm
. Accessed October 6, 2010.
3. O'Donnell K, Nyangara F, Murphy R, et al. Child Status Index: A Tool for Assessing the Well-Being of Orphans and Vulnerable Children—Manual
. Chapel Hill, NC; Measure Evaluation; January 2009.
4. O'Donnell K, Nyangara F, Murphy R, et al. Child Status Index (CSI): Public Domain
. Durham, NC; US President's Emergency Fund for AIDS Relief through USAID to Measure Evaluation & Duke University; 2008.
5. Nyangara F, O'Donnell K, Murphy R, et al. Child Status Index: A tool for Assessing the Well-Being of Orphans and Vulnerable Children—Field Users' Guide
. Chapel Hill, NC; Measure Evaluation; January 2009.
6. Miller C, Brooks B, Tsoka M, et al. Evaluation of the Child Status Index: A Validation Study in Malawi
. Boston, MA: Center for Global Health and Development, Boston University; 2010.
7. Office of the President and Cabinet of the Republic of Malawi. Malawi HIV and AIDS Monitoring and Evaluation Report: 2008-2009
. Chapel Hill, NC; January, 2010.
8. Census Survey Processing System
[computer program]. Version 4.0. Washington, DC: United States Census Bureau; 2009.
9. Statistical Analysis Software (SAS)
[computer program]. Version 9.0. Cary, NC: The SAS Institute; 2009.
10. Cook TD, Campbell D. Quasi-Experimentation: Design and Analysis Issues for Field Settings
. Boston, MA: Houghton-Mifflin; 1979.
11. Nunnally J, Bernstein I.Psychometric Theory
. 3rd ed. New York, NY: McGraw-Hill; 1994.
Keywords:© 2011 Lippincott Williams & Wilkins, Inc.
children; evaluation; Malawi; OVC; vulnerability