Share this article on:

A Computational Future for Preventing HIV in Minority Communities: How Advanced Technology Can Improve Implementation of Effective Programs

Brown, C. Hendricks PhD*; Mohr, David C. PhD; Gallo, Carlos G. PhD*; Mader, Christopher BS; Palinkas, Lawrence PhD§; Wingood, Gina PhD; Prado, Guillermo PhD*; Kellam, Sheppard G. MD; Pantin, Hilda PhD*; Poduska, Jeanne PhD#; Gibbons, Robert PhD**; McManus, John PhD††; Ogihara, Mitsunori PhD‡‡; Valente, Thomas PhD; Wulczyn, Fred PhD‖‖; Czaja, Sara PhD‡‡; Sutcliffe, Geoff PhD¶¶; Villamar, Juan MSEd*; Jacobs, Christopher BS¶¶

JAIDS Journal of Acquired Immune Deficiency Syndromes: June 1st, 2013 - Volume 63 - Issue - p S72–S84
doi: 10.1097/QAI.0b013e31829372bd
Supplement Article

Abstract: African Americans and Hispanics in the United States have much higher rates of HIV than non-minorities. There is now strong evidence that a range of behavioral interventions are efficacious in reducing sexual risk behavior in these populations. Although a handful of these programs are just beginning to be disseminated widely, we still have not implemented effective programs to a level that would reduce the population incidence of HIV for minorities. We proposed that innovative approaches involving computational technologies be explored for their use in both developing new interventions and in supporting wide-scale implementation of effective behavioral interventions. Mobile technologies have a place in both of these activities. First, mobile technologies can be used in sensing contexts and interacting to the unique preferences and needs of individuals at times where intervention to reduce risk would be most impactful. Second, mobile technologies can be used to improve the delivery of interventions by facilitators and their agencies. Systems science methods including social network analysis, agent-based models, computational linguistics, intelligent data analysis, and systems and software engineering all have strategic roles that can bring about advances in HIV prevention in minority communities. Using an existing mobile technology for depression and 3 effective HIV prevention programs, we illustrated how 8 areas in the intervention/implementation process can use innovative computational approaches to advance intervention adoption, fidelity, and sustainability.

*Department of Public Health Sciences, Miller School of Medicine, University of Miami, Coral Gables, FL;

Department of Preventive Medicine, Northwestern University, Chicago, IL;

Center for Computational Science, Miller School of Medicine, University of Miami, Coral Gables, FL;

§School of Social Work, University of Southern California, Los Angeles, CA;

Department of Behavioral Sciences and Health Education, Rollins School of Public Health, Emory University, Atlanta, GA;

Department of Mental Health, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD;

#American Institutes for Research, Washington, DC;

**Department of Health Statistics, University of Chicago, Chicago, IL;

††Department of Marine Biology and Fisheries, Rosenstiel School of Marine and Atmospheric Science, University of Miami, Coral Gables, FL;

‡‡Department of Psychiatry and Behavioral Sciences, Miller School of Medicine, University of Miami, Coral Gables, FL;

Department of Preventive Medicine, University of Southern California, Los Angeles, CA;

‖‖Chapin Hall, University of Chicago, Chicago, IL; and

¶¶Department of Computer Science, University of Miami, Coral Gables, FL.

Correspondence to: C. Hendricks Brown, PhD, Center for Family Studies, Miller School of Medicine, University of Miami, 1425 NW 10th Avenue, Miami, FL 33136. (e-mail:

The authors gratefully acknowledge the support of the NIDA and OBSSR for funding on the Center for Prevention Implementation Methodology (Ce-PIM: P30DA027828; CHB, GP, HP, SGK, MO, CG, JV, CM, CJ, GS, TV, LAP, SC, JM, RDG) and NIDA as well for R01DA025192 (GP, HP, CHB) and R01DA030452 (JP, CHB); NIMH for funding of the Center for Behavioral Intervention Technologies (CBITS: P20MH090318; DCM) and the Advanced Center to Improve Pediatric Mental Health Care (ACIPMC: P30MH074678; LAP, FW, CHB) as well as grants P20 MH090318 (DCM) and R01MH066302 (RDG); NIAID, NCI, NICHD, NHLBI, NIDA, NIMH, NIA, FIC, and OAR for supporting the Miami Center for AIDS Research (CFAR; P30AI073961; GP, CHB); NIAID for supporting the Emory Center for AIDS Research (CFAR: P30AI050409; GW), the National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention (NCHHSTP) for U01-PS000671 (GP), NCHHSTP for supporting R18PS001157 (GW), NCATS and NIMHD for supporting the Miami Clinical and Translational Science Institute (CTSI: UL1TR000460; CHB, SC, CM), IES for R305A090446 (JP, CHB) and the William T. Grant Foundation (Grant ID# 10648; LAP, CHB). We also thank the Center for Computational Science at the University of Miami (CG, CM, CHB). Additional support for the supplement was provided by R13 MH-081733-01A1.

The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

The authors have no conflicts of interest to disclose.

Back to Top | Article Outline


Nearly 50,000 new HIV infections occur in the United States every year, with African Americans having 7.6 times the incidence rate and Hispanics having 2.9 times the incidence rate of white non-Hispanics in 2009.1 To counter these disparities, behavioral and public health researchers, in partnership with minority communities, have developed and tested a range of preventive interventions targeting HIV risk factors. Recent meta-analyses show that overall such programs, when directed toward African American communities,2–6 are moderately efficacious, and certain programs specifically designed for African Americans have been shown to be highly efficacious or effective.7–11 There are fewer programs designed specifically for Hispanics,6,12 but among these, several have been identified as particularly efficacious.13–16 Most of the approaches in the literature so far have focused on small group interventions to affect behavioral changes that reduce HIV exposure. Even with the successes of these programs, there is an undeniable gap in their delivery as they need extensive efforts to engage the intended population,15,17 serve their specific needs,18 and address the need for extensive training of facilitators and implementation challenges of organizations who would deliver such programs.19 Recently, several strategies that rely on DVD, Internet-based,20–22 or mobile phone–based technologies have been used in an attempt to overcome such limitations, with varying levels of effectiveness. These include interactive videos in STD clinics,23 anonymous partner notification (eg,,24 and self-management (eg, +Click).25 Although there are a number of mobile phone interventions to support antiretroviral therapies, there are still only a few that support prevention of HIV itself.26,27

In this article, we discuss 2 important but largely untapped opportunities to address HIV prevention in minority communities. These 2 opportunities involve the use of sophisticated computational approaches. One general computational approach is to apply behavioral intervention theories and strategies to deliver new interventions that use technology, especially the use of the web, mobile phone, and sensor technology. Such approaches are known collectively as behavioral intervention technologies (BITs),28 and although these have grown in sophistication for depression treatment, they have not yet been used anywhere near to their full potential in HIV prevention. We discuss below how a BIT-called Mobilyze could serve as a model for innovative interventions for HIV prevention in minority communities. A second broad area of technological innovations that has great promise but little development involves the use of computational tools to improve the implementation of already existing effective behavioral interventions. These behavioral interventions could target either the risk factors leading directly to HIV transmission (eg, risky sex) or indirectly by targeting antecedent risk factors (eg, childhood aggressive disruptive behavior). These computational approaches would not change the intervention per se but would be used to improve adoption, fidelity, and sustainability of evidence-based behavioral interventions.29–32 Three effective behavioral interventions for minority populations are used as examples, Sisters Informing Sisters about Topics on AIDS (SISTA), Familias Unidas (FU), and the Good Behavior Game (GBG). We also provide a general framework that describes different computational approaches and pinpoints 8 areas across the intervention–implementation system where sophisticated computational approaches can be used.

Back to Top | Article Outline


To illustrate potential opportunities for an innovative BIT intervention to reduce sexual risk behavior, we consider Mobilyze,28 which uses mobile phone technology to provide near continuous access to a wide range of behavioral strategies for depressed patients. In its current format, Mobilyze delivers a BIT intervention for depressed patients, which includes lessons, tools, and feedback. Some of the didactic lessons are taken from web-based approaches (eg, moodManager)33; this material is primarily text based, which is intended to explain treatment rationale and describe behavior change strategies. Tools are interactive components that include core features of behavioral activation,34 such as scheduling and monitoring pleasant events, and learning to identify and overcome avoidance. Feedback includes interactive graphic tools that allow users to track and explore relationships between mood, activities, social context, and locations. The intervention is supported by a coach who provides initial training over the phone to ensure that the patient can use the intervention and initial engagement. Coaches follow-up with brief weekly phone calls to support adherence.35

The core innovation of this intervention is the integration of a context sensing system, which is designed to use data from the sensors embedded in the phone to identify patient states without requiring the patient to input data. This continuous access allows for ongoing assessment of patient states and activities, and just-in-time outreach at moments conducive to intervention. Asking patients to repeatedly input data on their states and activities can be burdensome and an interference rather than a support in one's daily life. So Mobilyze incorporates a context sensing system in which data from embedded sensors in the smart phone are downloaded, and individualized algorithms are developed to identify patient states including location, activity, social context, and mood.

We briefly describe the mechanics of how Mobilyze is currently implemented to address depression. Powered by a machine learner, Mobilyze! uses a context-inference system that utilizes the large amount of data collected by embedded sensors in the mobile phone to aid the user in mood management. Data from the sensors (eg, GPS, WiFi, Bluetooth, accelerometer, use of onboard applications, etc.) on a mobile phone are shipped to the study servers. These observations are then collected by a context aggregator and made into features that are passed to a machine learner that maps this continuous data stream into likely states (eg, alone, at home, depressed).36 These predicted states are then passed to the action components, which provide mechanisms for relaying this information to other external outreach applications including displaying the information in graphs, changing their status in response to the information, or triggering outreach events based on the predictions (eg, send a short text message or notify a clinician).

There are a number of ways that a mobile BIT could have potential benefit for preventing HIV/AIDS for minority youth and young adults, those who are major users of mobile phones.37 First, depressed males and females are more likely to engage in risky sexual behavior in early to mid-adolescence compared with nondepressed youth,38 unlike depressed older adolescents and adults where sexual drive and number of sex partners is diminished.39 Thus, a focus on identifying and intervening in early or mid-adolescence when depression arises could potentially reduce HIV infections. Second, applying the same principles around specific HIV risk behaviors could lead to a novel intervention involving personalized reminders about condom use and substance abuse when vulnerable youth are located in places and times where risk is likely to be high. A BIT that identifies user contexts associated with risky social contexts that are identified through GPS coordinates and other data could provide just-in-time notifications to support safe behaviors. Such approaches would require testing for effectiveness using a carefully designed randomized trial.

The second general approach we discuss blends well-tested behavioral interventions with sophisticated technologies that support their implementation, rather than creating new interventions. The first of these we discuss is (SISTA), a well-known behavioral intervention for reducing sexual risk in African American women. SISTA is a 5-session HIV prevention program for African American women, 18–29 years of age, who report unsafe sex.40 SISTA focuses on enhancing ethnic and gender pride, self-worth, and self-actualization in being an African American woman. SISTA also provides knowledge on HIV risk reduction practices. Role playing focuses on enhancing sexual communication skills, assertiveness without demeaning one's partner, and skills to refuse unsafe sex. SISTA also focuses on enhancing correct condom use skills and promotes norms supportive of consistent condom use. Compared with a control, women in SISTA showed increased consistent condom use [odds ratio (OR) = 2.1, 95% confidence interval (CI): 1.03 to 4.15, P = 0.04], greater sexual communication (OR = 4.1, 95% CI: 1.67 to 10.01, P = 0.002), greater sexual assertiveness (OR = 1.8, 95% CI: 1.01 to 3.27, P = 0.05), and increased partner norms supporting consistent condom use (OR = 2.1, 95% CI: = 1.08 to 3.87, P = 0.03).40

The Centers for Disease Control and Prevention (CDC) has defined SISTA as an “evidence-based HIV intervention” (EBI) for African American women.41 For the past several years, CDC has disseminated SISTA to agencies nationally, conducting 1-week trainings with more than 1200 community agencies in almost every state in the United States. SISTA is one of the most widely disseminated EBI in the CDC's Dissemination of Evidence-Based Intervention (DEBI) program (see Figs. 1A, B). SISTA is currently being adapted for faith institutions. A description of the creation of faith-based SISTA has been recently published.42



The 2 other behavioral interventions we discuss involve a somewhat different prevention paradigm. Rather than intervene at a time of high risk, as SISTA does, these intervene earlier on key risk and protective processes before the time of high risk. These developmental epidemiology43–45 interventions target modifiable antecedent factors for HIV transmission in an earlier stage of life before that during which most of the risk of sexual transmission occurs. We mention 2 such interventions: FU, a family intervention for middle school Hispanic adolescents, and the GBG, a classroom based for elementary school focused on aggressive/disruptive behavior that has been tested primarily with African American children in an urban setting.

FU is a culturally specific, family-based, drug abuse and HIV preventive intervention for Hispanic middle school youth and their families.13 FU has been evaluated and found to be efficacious in multiple randomized trials in dramatically reducing the rate of drug use from 34% to 25% over 30 months postintervention, OR = 0.65, P = 0.02,14 risky sexual behavior, risk ratio = 0.61 (95% CI, 0.39–0.87), P < 0.01, for inconsistent condom use during vaginal sex 6 months postintervention,13,46 and their co-occurrence with more than a 50% reduction in the days having unprotected sex while under the influence of illicit drugs in the intervention condition versus control, risk ratio = 0.36 (95% CI, 0.22–0.58), P < 0.01.13 The effects of FU on multiple outcomes have been mediated by improvements in family functioning.46 FU is ranked among the highest among the Substance Abuse and Mental Health Services Administration (SAMHSA) National Registry of Evidence-Based Programs47 and is currently being tested in a large effectiveness trial across 24 middle schools in Miami, FL. It is also being retooled so that it can be implemented in primary care and other settings.

The GBG48 is a classroom-based behavior management strategy that targets early aggressive/disruptive behavior, a robust risk factor for sexual risk taking, drug use, and delinquent and criminal behavior.49 GBG uses group contingencies to help children master the role of student and be successful at the key demands of the classroom including sitting still, paying attention, and completing school work. Through GBG, children work together to create a positive learning environment by monitoring their own behavior and that of their classmates. As a universal intervention, GBG is directed at the entire class. In a randomized effectiveness trial with long-term follow-up into young adulthood, GBG and a control condition were delivered to first graders with the intervention continuing through second grade.50 Especially for high-risk males, GBG led to lower levels of aggressive behavior,44,51,52 drug and alcohol dependence/abuse disorders,53 antisocial behavior,54–57 suicide ideation and behavior,58 and lower sexual risk behavior through young adulthood, fully 13 years later, in a mostly African American population.59 Specifically, males in the GBG classrooms who were in the persistently high aggressive/disruptive class were more than twice as likely to use condoms compared with males in control classes (89% lifetime use during vaginal sex through young adulthood compared with 41% for controls, P = 0.01) and delayed vaginal sex by 2 years (14.4 versus 12.3 years, P = 0.035). Furthermore, only 4% of GBG males reported co-occurrence of substance use and unprotected sex compared with 15% of control males (P = 0.04), with much stronger effects among those males in the high persistent class (1% for GBG versus 44% for controls, P = 0.01).59 There is an ongoing randomized trial of GBG to test questions of implementation, including testing 2 different models of training and support.60,61 GBG is on SAMHSA's list of effective programs47 and is also being implemented in 21 US school districts through funding by SAMHSA.

Back to Top | Article Outline


Although these last 3 behavioral interventions have been demonstrated through rigorous randomized trials to reduce HIV sexual risk behavior in minority populations, proving that a preventive intervention is effective is only the first step that we need to take in delivering prevention programs that will reduce the incidence of HIV infection in populations. A commonly held view is that only about 15% of interventions reach the stage of quality implementation and that occurs only after a long time after publishing, typically around 17 years.62 The RE-AIM model63 provides an important framework for having such programs improve the health of populations through increased Reach into the community, delivery of Effective programs, Adoption by communities and organizations, Implementation with fidelity, and Maintenance, as all of these are necessary to improve outcomes. An important HIV prevention policy, as well as scientific challenge, involves learning how to package these and other successful programs for large-scale delivery and to move these prevention programs into practice.64 Fundamental questions include how to increase program adoption, how to deliver these with fidelity, and how to sustain these or expand their delivery to large populations.29,30 All 3 of these tasks comprise the domain of implementation science,65 an emerging field that will require new methods,30 new collaborations, and partnerships.61,66,67

In this period of transition in emphasis from effectiveness to implementation research, there is a growing realization that the delivery of successful prevention approaches at the community level will require much more personalized interventions, so that they fit into the fabric of people's lives rather than requiring people to make major changes in their customs and habits. In particular, behavioral interventions will need to be delivered to people at the appropriate time and place and via a modality that is familiar to the individual, rather than the current traditional approach where the intervention is delivered by an expert under highly restricted times and places. This is especially important for delivery of prevention programs to minority populations who are challenged by limited transportation, child care, and available time. Second, we need what can be called contextualized implementation for addressing the complex social systems that would ultimately house these prevention efforts. Key challenges include the following: delivery of personalized interventions that are cost-effective in terms of human resources, building support structures to enhance fidelity of intervention delivery, and sustaining and expanding these prevention programs in social systems. Ultimately, we seek ways to deploy our interventions that match or exceed the effectiveness of current programs and are accepted and used by large numbers of people, yet cost far less—say by one or more orders of magnitude—than those that have now been tested in effectiveness research.

The knowledge that we have derived to date from effectiveness trials to accomplish these implementation goals is in its infancy68–72 and new implementation approaches are needed.30 We anticipate that “social and behaviorally informed” technologies have a great potential to be used in meeting these challenges. These technologies often come under the term Systems Science36,69,70,73,74 or Systems Engineering,75–77 which collectively comprise a set of methods that deal with complex interactions and include social network analysis, agent-based modeling and other simulations, and system dynamics. For the purpose of this article, we considered that many of these methods, as well as others, are closely connected to modern computational science, and we focus on human–computer interactions that involve improvement via synchronous or asynchronous communications. The focus on a wide assortment of computational approaches leads to a variety of techniques and tools to address such system-level challenges. This article describes how innovative computational perspectives can lead to radically different approaches to advance the prevention of HIV, especially in minority populations.

Back to Top | Article Outline


In Figure 2 below, we provide a simplified social ecological model of implementation for an evidence-based intervention program embedded in multiple systems. In describing this model, we use Mobilize and the implementation of SISTA, FU, and GBG as examples. Large vertical arrows in this diagram identify focal points where computational approaches potentially can be used to improve either the intervention itself or the implementation of an evidence-based preventive intervention program. To specify particular components in this model, we use the following names: the Target is the population in which we want to affect behavior change, for example, the student for GBG, the Hispanic adolescent for FU, the African American woman for SISTA, or the patient of a BIT. The Intervention is the program or practice that is delivered to the target to effect change; we have given an overview of the Mobylize BIT above and in the sections below, we provide more of the mechanics involving Mobylize, SISTA, FU, and GBG.



We define Intervention Agent as the person or device that is delivering the evidence-based program. For GBG, it is the child's classroom teacher and the trainer who trains the teachers. For FU, there are 2, the facilitator who delivers FU to a group of parents and at facility and to families in the home, and the parent, most often the mother, whose is responsible for communicating the messages of FU to the youth. For Mobilyze, it is the smart phone application and server software and in some cases includes a coach. The Implementation Agency is defined as the system(s) responsible for delivering the intervention, for example, the school system for GBG and likewise the current implementation agency for FU is also the school system. Implementation Agencies have not yet been fully defined for BITs in the United States. But web-based treatments can be prescribed through primary care clinics in the United Kingdom78 and are available through government-supported institutions in Australia.79

We also use the term Developer or Purveyor as the one who supplies the behavioral intervention and its technical assistance regarding training and supervision. For GBG, there are currently 3 technical assistance groups that provide training for a prevention implementation program in 21 school districts now being funded by the SAMHSA as mentioned earlier. For FU, the purveyor is the Center for Family Studies at the University of Miami. Purveyors for BITs include the group that makes a mobile application available (eg, Northwestern's Center for Behavioral Intervention Technology for Mobilyze) and may include coaching services as well.

The term Broker, sometimes called Intermediary,80 refers to individuals or organizations that develop partnerships between Agencies, Purveyors, and funders and evaluators, to support implementation and quality improvement. CDC's DEBI program links community-based organizations with researchers and hosts technical assistance. Brokers for GBG have varied by locale and include the American Federation of Teachers in Houston (the teachers' union); Oxford Brookes University in Oxfordshire, England; and the Nebraska Department of Education. The Broker for BITs can be a government agency or a private company. For example, Beating the Blues, the most widely used web-based treatment of depression, is now disseminated through Ultrasis, PLC.

Finally, we include the Funder, which may be a local, regional, or federal governmental agency; philanthropy organizations; or potentially public–private partnerships.81 For SISTA, the CDC has played a pivotal role, which for GBG the primary funder for implementation currently is SAMHSA. These funders are deeply interested in being effective in all 3 implementation stages of adoption, implementation with fidelity, and sustainability.

Back to Top | Article Outline


Several potential gains can come from computational approaches to delivering or implementing behavioral interventions in minority populations. Certainly, the World Wide Web has greatly extended our ability to broadcast behavioral messages or search for information like nothing before it. But technology also allows us to reach a specific target audience as well. We can now “narrowcast” messages to those who may be most vulnerable. This could be of major value to minority populations who are often the last to receive gains from standard interventions that are often designed and tested on majority populations. Of even more potential is the ability to “timecast” messages to the appropriate target audience at the appropriate time and place where they may be ready for behavioral change. Mobile technologies, for example, can be used to assess context through location (GPS), time of day, activity state (motion sensor), and emotional state (mobile screening, processing of voice, and pattern of ingoing and outgoing messages). The type of message to be delivered (or not) can be algorithmically decided based on this perceived context. Although not able to measure context and status perfectly, such systems have potential greater impact, as research has demonstrated that receptivity of messages is highly sensitive to context.33–35 Furthermore, mobile-based interventions are all relevant to minorities, especially youth and young adults. In general, minorities in the United States are less likely to have access to the web, but they are more likely to use mobile technologies than white non-Hispanics,37 and therefore, m-Health interventions may play an important part, especially when sensors can be used to effectively identify times and places for intervention.

Back to Top | Article Outline


Mobile sensing is inexact, and asking subjects repeatedly to input data on their states and activities can be burdensome. To avoid this burden, Mobilyze uses a supervised machine learning paradigm in which Mobilyze is trained by asking the user to label his or her states while developing prediction algorithms. Thus, subject interaction is high only during this supervised learning experience. At that point, Mobilyze will be able to infer when the target subject is likely to be engaging in the positive events, in which case positive feedback is provided, and when the subject is nonadherent, in which case encouragement and suggestions are provided. Initial field testing found significant reductions in depressive symptoms among users.28

When direct response from a subject is required through mobile phones, computerized adaptive testing (CAT)82 can be used to minimize subject burden as well. In place of traditional multi-item assessments that give a fixed set of items to everyone, CAT draws items from a large databank and adaptively selects items for each subject that are maximally informative based on his or her responses to items already answered. As only the most discriminating items for each person are administered, this allows precise measurement with a minimum number of questions for each subject. The paradigm shift is from the standard assessment approach of fixing the number of items but allowing measurement uncertainty to vary, to allowing the number of items to vary but fixing uncertainty at a predefined level suitable for the specific measurement problem at hand. A CAT implementation relies on querying findings from a preexisting psychometric study involving a large group of subjects asked to respond to many items. CAT is delivered through a computer program that selects the next item in response to that person's on last answers. The CAT-DI, a tool to assess depressive symptoms, for example, requires an average of 12 items per subject—half the size typically used—and is correlated 0.95 with a total score from 389 depression items. A related computerized adaptive diagnostic assessment of Major Depressive Diagnosis can be made with 4 items, having a sensitivity of 0.95 and specificity of 0.87 against a clinician-based Structured Clinical Interview for DSM-IV (SCID/DSM).83

Back to Top | Article Outline


Computational methods in general have the advantage of simplifying repetitive tasks; increasing accuracy; reducing costs; increasing speed and power for information retrieval, processing, and decision making; and extending the system to other settings and human languages with relatively little effort.59,84 These are all relevant to a general strategy of replacing repetitive human tasks in improving health with computers due to advantages in cost and accuracy. In this section, we describe how computational approaches can be used to address current limitations in implementation of our 3 evidence-based prevention programs mentioned earlier. Personalized and contextualized m-Health reminder systems, such as the one we mentioned above, provide one example of how the technology of BITs can potentially be applied to distinct components of the implementation system and to an intervention delivered to the target population. This is discussed with regard to the GBG.

Back to Top | Article Outline


GBG is a classroom management tool that is used by teachers (intervention agents) in elementary schools (implementation agent) to provide clear messages to students (target) about expected behavior and to provide group-based contingencies for behavior, especially aggressive/disruptive behavior. As mentioned, GBG has shown positive impact for a wide range of externalizing outcomes through adolescence and adulthood. Classroom behavior management is a primary reason teachers leave the profession, yet elementary school teachers are rarely taught ways to manage classroom behavior unless they are trained in special education.85 Without training, the contingent responses of teachers to student behavior are usually all individually based and inconsistent, resulting in coercive interaction cycles rather than a reduction in aggressive disruptive behavior.86–89 GBG employs a structured team-based game approach in which teams of students receive rewards if their team has fewer instances of breaking the class rules than a specified criterion while the game is being played. Of note is that all teams, hence all students, can win. The game requires consistency and precision in a number of areas. Teachers arrange the teams so that each is heterogeneous in membership with regard to behavior, sex, and academic achievement. Teachers and students work together to make explicit the connections between the class rules and the behaviors that support or are infractions of the rules. Through GBG, teachers learn to address infractions of the rules neutrally and with consistency. Over the school year, the number of games played per week increases as does the duration of each game; games are played in varying subjects and contexts throughout the day; and the rewards move from tangible to intangible and are delivered on a delayed time structure. Although there exist several models for technical assistance for GBG (purveyors), training and support for teachers that is in keeping with the randomized field trials providing evidence for GBG effectiveness consists of up to 20 hours of group-based training for teachers followed by regular individualized support by coaches. The coach's role is to help teachers bring together the knowledge and skills learning in training with their daily instructional practices.90–92

There are several ways computational approaches can increase the spread and fidelity of GBG. Many of the tasks to set up GBG and record its implementation could be aided by mobile phone applications. For example, teachers could set the schedule for playing the game and rearrange the teams for balance as part of their instructional planning and keep records such as team points more easily and potentially more accurately using a mobile phone in contrast to entering the data on a computer or onto pen and paper forms. Information to support ongoing fidelity of implementation can also benefit from technology and can be assessed in automated fashion. For example, a core element in the game is that teachers respond to infractions of the rules in a consistent neutral fashion. A schedule for teachers to audio record the classroom at times when they are playing the game and not playing the game could be set up. Voice recognition software trained to the teacher's voice can be used to provide automatic transcription in text form93–95; then, text mining software96,97 can be used to evaluate the quality of the teacher's response during the game and how well the teacher is generalizing the skills throughout the day. A more extensive use of these same computational linguistics tools is described below for FU.

Technology and computational approaches can also inform the next stage of training and support for teachers. Strategies that employ technology, for example, synchronous and asynchronous web-based training, are becoming widely used in the provision of training and supporting to teachers and are beginning to be developed for GBG. Such approaches have the potential to increase the reach of GBG and similar interventions, especially in locales in which reliance on face-to-face models of training and support is inefficient. Computational approaches can also strengthen our ability to develop individualized adaptive training and support for teachers informed by data and decision rules in conjunction with professional judgment. Data on teacher demographics and characteristics, school and community context, GBG implementation and outcomes, and the training and support provided by the coach collected via the web can be brought together in a manner to support data mining and the determination of decision rules.

Back to Top | Article Outline


There is strong evidence that proven programs delivered with low fidelity provide little or no change on outcomes,53,98–100 so a fidelity monitoring and feedback system is critical to implementation success of a behavioral intervention. Although there are many ways to measure fidelity, process, adherence, competence, and the like,98 existing systems are all labor intensive and expensive. Efficacy and effectiveness trials can often afford to pay for fidelity monitoring through the research funding, but this labor and expense is often beyond what institutions such as schools- and community-based organizations can afford in full-scale implementation. Obtaining automated fidelity ratings is a promising approach to reducing these costs while maintaining validity and reliability. Below we explore its use with FU.

FU is delivered to families both in parent groups and family visits. In the parent groups, discussions aim to increase parents' understanding of their role in protecting their adolescent from harm and to facilitate parental investment (eg, parental involvement, positive parenting) in the home, at school, and with peers. For the family visits, facilitators supervise parent–adolescent discussions to encourage bonding within the family and help parents implement the skills learned in the parent groups (eg, discussing behavior management, peer supervision issues, and homework). Each group visit is video recorded, and 25% of the family visits are recorded. Facilitators are currently being rated based on 6 content and process dimensions of adherence to the intervention manual on each 30-minute segment of the family visit and each group visit. These ratings are used in supervision of the facilitators.

We propose that advanced technologies can be used as an important part of the systems to monitor fidelity from existing video/audio tapes. Computational linguistics101,102 in combination with speech processing can both simplify and provide additional feedback regarding the fidelity of intervention delivery. For instance, existing technologies of speaker recognition,103 automatic transcription,104 identification of dialog turns (how long a speaker holds the floor), and105 identification of gestures106 and topic segmentation,107 along with computation linguistics techniques can be harnessed to generate fidelity scores. Although these technologies aim to replicate certain aspects of human tasks, they by no means replace them all together. Inexpensive automated coding can be used as a first stage fidelity check for all videos, followed by the more expensive step human coding on selected video segments based on statistical sampling of facilitators, tapes, and times that computational algorithms identify as uncertain or important to focus on during supervision. Researchers at the Center for Prevention Implementation Methodology (Ce-PIM) are currently developing such an automated fidelity system for implementing FU intervention, and this system can be adopted for use with other preventive intervention programs.

Back to Top | Article Outline


There is a current adaptation of SISTA now funded that will allow SISTA to be implemented in numerous faith institutions serving African American women. In this section, we discuss how simulation methods can be used to investigate alternative approaches to wide-scale implementation in mega churches having several thousand members of the congregation. A fundamental challenge is to design a system for delivering SISTA in a small group format led by key leaders in this congregation that maintains both high fidelity and reach into this community. We conceptualize the delivery of SISTA as a system involving (1) the facilitators (intervention agents from the congregation) who deliver SISTA; (2) 2 types of training, 1 for training of facilitators and 1 for training of trainers; (3) the monitoring, feedback, and supervision components for facilitators; and (4) the network relations between facilitators and the target women. Factors that we hypothesize affect effectiveness of SISTA include the quality of training; the selection of and network position of facilitators; the influence over a woman's sexual attitudes by the church, friends, and partners42; the norms reflected about behavioral risk in mega church hierarchy and congregation; and the baseline attitudes of each woman. The first 3 of these factors are modifiable by the implementation strategy: the higher the skill level of those who are selected to be trained as facilitators, the higher the fidelity, but those who are in key influence positions in a network may be more effective in changing behavior.108,109 By using different ways to select trainers, trainers of trainers, facilitators, and women who attend SISTA based on the skills and network positions, we can expect widely different population impact and cost. Because these factors interact in complex ways, it is difficult to predict what choices are optimal. No closed form solutions exist to designing the most effective training, monitoring, and feedback system, and simulations are necessary to predict which implementation strategies are appropriate. Simulations would take into account the existing influence network data108,110–112 and complex interactions that can be modeled with agent-based models113–119 to model the behavior of alternative strategies, compare simulations, and feed findings back to the churches. An agent-based model involves interactions for different types of “agents,” that is, trainers, trainers of trainers, facilitators, and women in the congregation. Their interactions follow rules and methods that describe how each agent can obtain information about itself, other agents, and the environment, and how it can interact to affect other agents and the environment. Ultimately, this modeling approach could be used to “learn” effective strategies for advancing the adoption and implementation of SISTA and other interventions.

Back to Top | Article Outline


This article illustrated some novel approaches for using computational systems to improve HIV prevention in minority populations. We argue that there are 2 fundamental approaches that could be beneficial. The first is the use of sophisticated computationally driven interventions to reduce risky sex behaviors. The second involves computational supports that increase effective implementation of behavioral interventions that have been shown to be effective in reducing HIV risk behaviors in minority populations.

With these examples as illustrations, we return to Figure 2, which identifies 8 places where computational approaches could be used. Starting with interventions focused on the target (location 1), there are 2 main challenges. Foremost is increasing the reach of an intervention into minority populations. Owing to the wider access and acceptance of mobile phones among minorities, this technology has an untapped potential for HIV prevention. After improvement in reach, these new interventions have to be tested for effectiveness with rigorous randomized trials. Next, we can use computational methods to explore different delivery mechanisms (location 2) of the same intervention. For FU and similar parent interventions for African Americans, it is still unclear whether web-based interventions would be as successful as group-based interventions that have the same content because they have limited social interaction mechanisms used to shape behaviors in small groups. Computational approaches may play an important supportive role in simplifying and supporting intervention agents (location 3) as we illustrated with mobile technologies for the GBG. Eventually, human-mimicking computational systems, such as virtual coaches120,121 may become sufficiently sophisticated as to become the sole intervention agent themselves, directly interacting with end users. In the more immediate future, virtual systems may be more useful in delivering training or supplementing on-site or distance training (location 4). In addition, computational linguistics systems may be able to monitor and provide feedback about the fidelity of a limited set of responses, such as a teacher is required to do in the GBG when a child breaks a rule. We also suggest that computational methods may be used first to model and then improve the system that an implementation agency (location 5) uses as we illustrated with SISTA in mega churches. Again moving upstream, the developer can use computational systems to improve the quality of an intervention (location 6). For example, the acceptability and impact of one intervention component versus another can be tested against one another rapidly in web-based environments.122,123 The role of brokers (location 7) in developing partnerships and facilitating complex implementation strategies124 could be facilitated by computational systems that automatically monitor the implementation process and detect inefficiencies. Finally, the fundamental role of the funder in deciding and implementing effective policies, including incentives, can be supported through simulation modeling.

We predict that a number of computational tools, now being used in behavioral research, will be useful in improving the implementation of HIV prevention programs. In particular, social network analysis, agent-based modeling, and system dynamics have been identified as 3 general system science approaches that are critical for prevention.74 These and related tools are the methods one uses for a new generation of in silico behavioral science studies that provide an important new method to the types of in vitro, or small-scale laboratory studies and in vivo or full-scale community studies that are well known among social and behavioral scientists in HIV prevention. We note that the power of these highly flexible computational tools to reduce the solution space for effective implementation strategies depends heavily on the quality and quantity of data that are available. For example, simulation that use variation in impact estimates derived from a randomized trial can be used to predict population-level impact,125 explore unintended consequences brought about by unexpected causal feedback loops, and identify which policy interventions offer the greatest leverage (ie, impact per dollar spent).126 Social network interventions can be analyzed to identify and select more efficient adoption strategies.127 When simulation and network analytic approaches are combined with data from electronic sources, including system-level process and administrative records, the potential for real-time integration of macro-to-micro policy decisions, with accurate prediction of target, intervention agent and agency responses, is likely to result in improved implementation and ultimately a reduction in HIV infections.

It is worth considering what types of support structures would be needed in a computational future of prevention if such technologies are found to be effective and selected for wide-scale implementation. People are willing to use computational systems as long as they work as intended, but everyone has little tolerance for computational systems that are faultily designed or implemented. To fully realize the potential of large-scale delivery, cost savings, increased accuracy, and other benefits offered by the application of computational technologies within preventive interventions will require the production of high-quality software tools and systems that can be sustained and updated as required. Computational technologies (ie, software) can be easily copied and transferred between independent organizations, but this is only easily done when these systems have been well engineered, tested, and documented. In addition to high-quality tools, successful adoption of a computationally supported intervention will require the tool developer to provide technical support during adoption and after to the Implementation Agency.

Sustaining preventive interventions that rely on computational systems requires a systematic approach to maintaining and updating the software. Software engineering as a discipline is concerned with the application of systematic approaches to the design, construction, enhancement, and ongoing maintenance of software systems and is therefore critical in long-range sustainability. The application of software engineering methodologies and best practices enables the production of more robust and reusable code, ultimately resulting in lower long run costs, than generally results from a more organic approach to software development. Production of high-quality software for application in preventative interventions will require the establishment and organizational system, which has the capabilities to develop, enhance, and sustain support for the systems they develop over a period of many years.

Alternative organizational models are possible for building and sustaining software tools for preventative interventions; 2 common models are dedicated development organizations and community development organizations. In a dedicated development organization, all software development is conducted by a single organization. This is the usual organizational model of commercial software companies (eg, Microsoft), which competes in the marketplace against other software developers. In nonprofit scenarios, such an organization could be funded by one funding provider who funds the organization to provide a standard technology for a specific community of interest. This is the model used by the National Cancer Institute, which funds the Biomedical Informatics Grid.128 The community development model is an alternative model under which open source software is developed. In this case, development is done by a network of developers collaborating together for quality improvement. Development is coordinated by a governance organization that establishes and oversees the software engineering standards and conventions used by the community. The Apache Software Foundation129 is one well known example of an open source development community that has been successful in producing and maintaining high-quality widely used software. The establishment of such entities to support the development of computational tools for preventative intervention, direction, and sufficient resources should be provided so that these organizations can emerge in parallel with the development of the associated technologies.

The roles of funders and oversight will be critical in implementing computational systems such as the ones that we described above. Funders could include NIH, CDC, HHS, or other federal agencies that support effectiveness testing and implementation research; federal agencies that deliver services, including SAMHSA and the CDC, who can provide direct support and training or technology assistance for community delivery; state or territorial service organizations who may provide support to local community institutions, organizations, or coalitions, and local groups, themselves, who procure funding as well as provide access to communities and their constituents. Oversight refers to both formal and informal community, institutional, or governmental groups that give permission for research and/or services to be conducted or provide evaluations. The Food and Drug Administration, for example, is charged with regulating the use of medical devices, and their actions have begun to affect some m-Health interventions.130 Especially for minority communities, local community coalitions are the key stakeholders who provide access, and ultimately, all activities are accountable to these community-based groups. Their oversight in developing computationally aided systems for their own communities would be essential.

This article would be remiss if it did not acknowledge that advanced technologies and the systems that deliver these technologies, including the biomedical and research communities, are often viewed with suspicion in many minority populations. Technologies that involve human–computer interactions, especially if they engender mistrust (eg, accessing sensitive information), or work counter to existing patterns, values, and meaning could easily lead to nonacceptance or even iatrogenic effects. Technology has the capacity to reduce interpersonal communication, which is culturally dystonic for many minority families.131 Technological systems can also leave users behind if they require more training and experience than readily available. The Internet, social media, and mobile phone technologies, for example, create high communication barriers for parental communication, monitoring, and supervision of their children, who are more technologically literate. However, technological improvements and fixes may remain on the drawing board, trailing users who want a more intelligent interaction with an automated system. Both individuals and organizations can be uncomfortable depending on technology that they themselves do not maintain, must retrieve updates, or must continue to pay for. Professional organizations that deliver prevention programs, including those representing teachers, nurses, and other health professionals, have attitudes about technology already formed by past experiences in their jobs and private lives, and these may create an important barrier to wide-scale acceptance. From an organizational standpoint, technologies that report on system implementation can threaten management and employees' visions, policies, and practices, and perhaps their own jobs. More efficient reporting on the implementation status of a system cannot only be threatening; it may provide a crude and inaccurate reflection of a system or an inaccurate modeling of the consequences of alternative decisions. Our perspective is that the introduction of any new technology cannot happen without working closely with the ultimate users of these technologies so they have maximal cultural fit, integrate seamlessly (and nearly invisibly) in their lives, and continue to have utility.132

Although these technologies could ultimately simplify the delivery of effective programs, they cannot be used until trust in both the technologies and the stakeholders engaged in their implementation has been established. Thus, the process of embedding technologies requires that it be preceded with a trusting and sustainable community–research partnership.133 Such a partnership is founded on the principle of knowledge exchange (of the technology, its use and the rigorous process for its evaluation versus knowledge of the community and the technology's end users) and resources (ie, personnel, funding, and equipment versus access to the community).134 In successful partnerships, such an exchange inevitably leads to a transformation of values that motivate the behavior of both the community and the researchers involved in introducing the new technologies.135 However, such a transformation requires an understanding of the value hierarchies of the respective partners, interpersonal processes that require the development and nurturance of mutual reciprocal trust, a willingness and ability to engage in negotiation and compromise, the exercise of a certain degree of “cultural humility,”136 and the presence of a culture broker.135 Thus, rather than serving as a barrier to interpersonal communication, technologies such as those described in this article and the partnerships required to implement and sustain their use may ultimately lead to the emergence of a new cultural system that serves as a model of and a model for prevention research and practice.

The methods presented in this paper are part of a more expanded approach that we have termed "scientific equity." In parallel to the term health equity,137 scientific equity refers to equality in the amount of scientific knowledge that is available or being produced to address health disparities. If there are insufficient research studies focused only on minorities, if there are too few minorities included in research studies to permit a synthesis across studies,138 or if advanced technologies are not applied to improve the delivery of effective interventions to minorities, there is little chance that we will learn what is necessary to counter existing disparities in minority populations. Thus, scientific equity means that sufficient scientific resources and infrastructure are applied to address all steps necessary in removing a disparity. It is difficult to predict at this point how well these advanced computational approaches will work in community settings and how long it would take to implement them with success. But the potential that these new approaches could fill the huge gap between what we know works and their use in high HIV-exposed communities is a strong argument for a broad investment in these approaches. Just as the emergence of the HIV epidemic forced a major transformation in public health, service delivery, and public policy, the failure to these same institutions to stem the tide of HIV infections in minority communities decades later, must and will force a major rethinking of the strategies of prevention implementation policy and practice. In the future, advanced computational technologies can play an important role.

Back to Top | Article Outline


1. Prejean J, Song R, Hernandez A, et al.. Estimated HIV incidence in the United States, 2006–2009. PLoS One. 2011;6:e17502.
2. Darbes L, Crepaz N, Lyles C, et al.. The efficacy of behavioral interventions in reducing HIV risk behaviors and incident sexually transmitted diseases in heterosexual African Americans. AIDS. 2008;22:1177–1194.
3. Henny KD, Crepaz N, Lyles CM, et al.. Efficacy of HIV/STI behavioral interventions for heterosexual African American men in the united states: a meta-analysis. AIDS Behav. 2012;16:1092–1114.
4. Marshall K, Crepaz N, O'Leary A. A systematic review of evidence-based behavioral interventions for African American youth at risk for HIV/STI infection, 1988–2007. In: McCree D, Jones KT, O'Leary A, eds. African Americans and HIV/AIDS: Understanding and Addressing the Epidemic. New York, NY: Springer; 2010:181–209.
5. Crepaz N, Marshall KJ, Aupont LW, et al.. The efficacy of HIV/STI behavioral interventions for African American females in the United States: a meta-analysis. Am J Public Health. 2009;99:2069–2078.
6. Crepaz N, Horn AK, Rama SM, et al.. The efficacy of behavioral interventions in reducing HIV risk sex behaviors and incident sexually transmitted disease in black and Hispanic sexually transmitted disease clinic patients in the United States: a meta-analytic review. Sex Transm Dis. 2007;34:319–332.
7. Jemmott JB III, Jemmott LS, Fong GT, et al.. Effectiveness of an HIV/STD risk-reduction intervention for adolescents when implemented by community-based organizations: a cluster-randomized controlled trial. Am J Public Health. 2010;100:720–726.
8. DiClemente RJ, Wingood GM, Rose ES, et al.. Efficacy of sexually transmitted disease/human immunodeficiency virus sexual risk-reduction intervention for African American adolescent females seeking sexual health services: a randomized controlled trial. Arch Pediatr Adolesc Med. 2009;163:1112–1121.
9. Wingood GM, DiClemente RJ. Enhancing adoption of evidence-based HIV interventions: promotion of a suite of HIV prevention interventions for African American women. AIDS Educ Prev. 2006;18(4 suppl A):161–170.
10. Wingood G, DiClemente RJ, Robinson-Simpson L, et al.. Efficacy of an HIV intervention in reducing high-risk human papillomavirus, nonviral sexually transmitted infections, and concurrency among African-American women: a randomized controlled trial. J Acquir Immune Defic Syndr. 2013;63:S36–S43.
11. Murry VM, Berkel C, Brody GH, et al.. The strong African American families program: longitudinal pathways to sexual risk reduction. J Adolesc Health. 2007;41:333–342.
12. Herbst JH, Kay LS, Passin WF, et al.. A systematic review and meta-analysis of behavioral interventions to reduce HIV risk behaviors of Hispanics in the United states and Puerto Rico. AIDS Behav. 2007;11:25–47.
13. Prado G, Pantin H, Huang S, et al.. Effects of a family intervention in reducing HIV risk behaviors among high-risk Hispanic adolescents: a randomized controlled trial. Arch Pediatr Adolesc Med. 2012;166:127–133.
14. Pantin H, Prado G, Lopez B, et al.. A randomized controlled trial of Familias Unidas for Hispanic adolescents with behavior problems. Psychosom Med. 2009;71:987–995.
15. Villarruel AM, Jemmott JB III, Jemmott LS. A randomized controlled trial testing an HIV prevention intervention for Latino youth. Arch Pediatr Adolesc Med. 2006;160:772–777.
16. Wingood GM, DiClemente RJ, Villamizar K, et al.. Efficacy of a health Educator–Delivered HIV prevention intervention for Latina women: a randomized controlled trial. Am J Public Health. 2011;101:2245–2252.
17. Villarruel AM, Jemmott LS, Jemmott JB, et al.. Recruitment and retention of Latino adolescents to a research study: lessons learned from a randomized clinical trial. J Spec Pediatr Nurs. 2006;11:244–250.
18. Latham TP, Sales JM, Boyce LS, et al.. Application of ADAPT-ITT: adapting an evidence-based HIV prevention intervention for incarcerated African American adolescent females. Health Promot Pract. 2010;11(3 suppl):53S–60S.
19. Griffith D, Allen J, DeLoney E, et al.. Community-based organizational capacity building as a strategy to reduce racial health disparities. J Prim Prev. 2010;31:31–39.
20. Bull S, Pratte K, Whitesell N, et al.. Effects of an internet-based intervention for HIV prevention: the Youthnet trials. AIDS Behav. 2009;13:474–487.
21. DiNoia J, Schinke S, Pena J, et al.. Evaluation of a brief computer-mediated intervention to reduce HIV risk among early adolescent females. J Adolesc Health. 2004;35:62–64.
22. Kiene S, Barta W. A brief individualized computer-delivered sexual risk reduction intervention increases HIV/AIDS preventive behavior. J Adolesc Health. 2006;39:404–410.
23. Carey MP, Senn TE, Vanable PA, et al.. Brief and intensive behavioral interventions to promote sexual risk reduction among STD clinic patients: results from a randomized controlled trial. AIDS Behav. 2010;14:504–517.
24. Levine D, Woodruff A, Mocello AR, et al.. inSPOT: the first online STD partner notification system using electronic postcards. PLoS Med. 2008;5:1428–1431.
25. Markham CM, Shegog R, Leonard AD, et al.. +CLICK: Harnessing web-based training to reduce secondary transmission among HIV-positive youth. AIDS Care. 2009;21:622–631. doi:
26. Lester R, Ritvo P, Mills E, et al.. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet. 2010;376:1838–1845.
27. Swendeman D, Rotheram-Borus MJ. Innovation in sexually transmitted disease and HIV prevention: internet and mobile phone delivery vehicles for global diffusion. Curr Opin Psychiatry. 2010;23:139–144.
28. Burns MN, Begale M, Duffecy J, et al.. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res. 2011;13:e55.
29. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4–23.
30. Landsverk J, Brown CH, Chamberlain P, et al.. Design and Analysis in Dissemination and Implementation Research. London: Oxford University Press; 2012.
31. Proctor EK, Landsverk J, Aarons G, et al.. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36:24–34.
32. Horwitz S, Chamberlain P, Landsverk J, et al.. Improving the mental health of children in child welfare through the implementation of evidence-based parenting interventions. Adm Policy Ment Health. 2010;37:27–39.
33. Mohr CD, Duffecy J, Jin L, et al.. Multimodal E-mental health treatment for depression: a feasibility trial. J Med Internet Res. 2010;12:e48.
34. Martell CR, Dimidjian S, Herman-Dunn R. Behavioral Activation for Depression: A Clinician's Guide. New York, NY: Guilford; 2010.
35. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res. 2011;13:e30.
36. Schensul JJ. Community, culture and sustainability in multilevel dynamic systems intervention science. Am J Community Psychol. 2009;43:241–256.
37. Smith A. Mobile Access 2010: Pew internet and the American life project, July 7, 2010, Accessed April 19, 2013.
38. Miller-Johnson S, Winn DM, Coie J, et al.. Motherhood during the teen years: a developmental perspective on risk factors for childbearing. Dev Psychopathol. 1999;11:85–100.
39. Lansford JE, Yu T, Erath SA, et al.. Developmental precursors of number of sexual partners from ages 16 to 22. J Res Adolesc. 2010;20:651–677.
40. Wingood GM, DiClemente R. A randomized controlled trial of an HIV sexual risk—reduction intervention for young African-American women. JAMA. 1995;274:1271–1276.
41. Center for Disease Control (CDC) Division of HIV/AIDS Prevention. SISTA. Available at: Accessed July 01, 2012.
42. Wingood GM, Simpson-Robinson L, Braxton ND, et al.. Design of a faith-based HIV intervention: successful collaboration between a university and a church. Health Promot Pract. 2011;12:823–831.
43. Kellam SG, Koretz D, Moscicki EK. Core elements of developmental epidemiologically based prevention research. Am J Community Psychol. 1999;27:463–482.
44. Kellam SG, Rebok GW, Ialongo N, et al.. The course and malleability of aggressive behavior from early first grade into middle school: results of a developmental epidemiology-based preventive trial. J Child Psychol Psychiatry. 1994;35:259–281.
45. Kellam SG, Rebok GW. Building developmental and etiological theory through epidemiologically based preventive intervention trials. In: McCord J, Tremblay RE, eds. Preventing Antisocial Behavior: Interventions from Birth through Adolescence. New York, NY: Guilford Press; 1992:162–195.
46. Prado G, Pantin H, Briones E, et al.. A randomized controlled trial of a parent-centered intervention in preventing substance use and HIV risk behaviors in Hispanic adolescents. J Consult Clin Psychol. 2007;75:914–926.
47. SAMHSA. NREPP: SAMHSA's national registry of evidence-based practices and programs. Available at: Accessed July 09, 2012.
48. Barrish HH, Saunders M, Wolf MM. Good behavior game: effects of individual contingencies for group consequences on disruptive behavior in a classroom. J Appl Behav Anal. 1969;2:119–124. doi:
49. Capaldi DM, Stoolmiller M, Clark S, et al.. Heterosexual risk behaviors in at-risk young men from early adolescence to young adulthood: prevalence, prediction, and association with STD contraction. Dev Psychol. 2002;38:394–406.
50. Brown CH, Kellam SG, Ialongo N, et al.. Prevention of aggressive behavior through middle school using a first grade classroom-based intervention. In: Tsuang MT, Lyons MJ, Stone WS, eds. Towards Prevention and Early Intervention of Major Mental and Substance Abuse Disorders. Arlington, VA: American Psychiatric Publishing, Inc; 2007:347–370.
51. Dolan LJ, Kellam SG, Brown CH, et al.. The short-term impact of two classroom-based preventive interventions on aggressive and shy behaviors and poor achievement. J Appl Dev Psychol. 1993;14:317–345.
52. Muthén BO, Brown CH, Masyn K, et al.. General growth mixture modeling for randomized preventive interventions. Biostatistics. 2002;3:459–475.
53. Kellam SG, Brown CH, Poduska JM, et al.. Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug Alcohol Depend. 2008;95(suppl 1):S5–S28. Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.drugalcdep.2008.01.004. doi:
54. Brown CH, Wang W, Kellam SG, et al.. Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug Alcohol Depend. 2008;95(suppl 1):S74–S104. Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.drugalcdep.2008.01.005. doi:
55. Schaeffer CM, Petras H, Ialongo N, et al.. A comparison of girls' and boys' aggressive-disruptive behavior trajectories across elementary school: prediction to young adult antisocial outcomes. J Consult Clin Psychol. 2006;74:500–510. doi:
56. Schaeffer CM, Petras H, Ialongo N, et al.. Modeling growth in boys’ aggressive behavior across elementary school: links to later criminal involvement, conduct disorder, and antisocial personality disorder. Dev Psychol. 2003;39:1020–1035.
57. Petras H, Kellam SG, Brown CH, et al.. Developmental epidemiological courses leading to antisocial personality disorder and violent and criminal behavior: effects by young adulthood of a universal preventive intervention in first- and second-grade classrooms. Drug Alcohol Depend. 2008;95(suppl 1):S45–S59. doi:
58. Wilcox HC, Kellam SG, Brown CH, et al.. The impact of two universal randomized first- and second-grade classroom interventions on young adult suicide ideation and attempts. Drug Alcohol Depend. 2008;95(suppl 1):S60–S73. Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.drugalcdep.2008.01.005. doi:
59. Kellam SG, Wang W, Mackenzie ACL, et al.. The impact of the good behavior game, a universal classroom based preventive intervention in first and second grades, on high risk sexual behaviors and drug abuse diagnoses into young adulthood. Prev Sci. doi:10.1007/s11121-012-0296-z.
60. Poduska J, Kellam SG, Brown CH, et al.. Study protocol for a group randomized controlled trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: integrating effectiveness and implementation research. Implement Sci. 2009;4:56.
61. Poduska J, Gomez M, Capo Z, et al.. Developing a collaboration with the Houston independent school district: testing the generalizability of a partnership model. Adm Policy Ment Health. 2012;39:258–267.
62. Balas EA. From appropriate care to evidence-based medicine. Pediatr Ann. 1998;27:581–584.
63. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–1327.
64. Rotheram-Borus MJ, Swendeman D, Chovnick G. The past, present, and future of HIV prevention: integrating behavioral, biomedical, and structural intervention strategies for the next generation of HIV prevention. Annu Rev Clin Psychol. 2009;5:143–167.
65. Davies P, Walker A, Grimshaw J. A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implement Sci. 2010;5:14.
66. Kellam S. Developing and maintaining partnerships as the foundation of implementation and implementation science: reflections over a half-century. Adm Policy Ment Health. 2012;39:317–320.
67. Brown CH, Kellam SG, Kaupert S, et al.. Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: experiences of the prevention science and methodology group. Adm Policy Ment Health. 2012;39:301–316.
68. Mabry PL, Olster DH, Morgan GD, et al.. Interdisciplinarity and systems science to improve population health: a view from the NIH office of behavioral and social sciences research [erratum]. Am J Prev Med. 2008;35:611.
69. Norman CD. Health promotion as a systems science and practice. J Eval Clin Pract. 2009;15:868–872.
70. Maglio PP, Mabry PL. Agent-based models and systems science approaches to public health. Am J Prev Med. 2011;40:392–394.
71. Glass TA, McAtee MJ. Behavioral science at the crossroads in public health: extending horizons, envisioning the future. Soc Sci Med. 2006;62:1650–1671.
72. Landsverk J, Brown CH, Rolls Reutz J, et al.. Design elements in implementation research: a structured review of child welfare and child mental health studies. Adm Policy Ment Health. 2011;30:54–63.
73. Chambers DA. Advancing the science of implementation: a workshop summary. Adm Policy Ment Health. 2008;35:3–10.
74. Lich KH, Ginexi EM, Osgood ND, et al.. A call to address complexity in prevention science research. Prev Sci. 2012;14:279–289.
75. Rivera DE, Pew MD, Collins LM. Using engineering control principles to inform the design of adaptive interventions: a conceptual introduction. Drug Alcohol Depend. 2007;88 (suppl 2):S31–S40.
76. Czaja SJ, Nair SN. Human factors engineering and systems design. In: Salvendy G, ed. Handbook of Human Factors and Ergonomics. 3rd ed. Hoboken, NJ: John Wiley & Sons Inc; 2006:32–49.
77. Kossiakoff A, Sweet WN. Systems Engineering: Principles and Practices. New York, NY: John Wiley & Sons Inc; 2003.
78. National Institute for Health and Excellence (NICE), ed. Computerised Cognitive Behaviour Therapy for Depression and Anxiety. (Review of TA 51): Technology Appraisal 97. Accessed April 19, 2013.
79. Christensen H, Hickie IB. E-mental health: a new era in delivery of mental health services. Med J Aust. 2010;192:S2–S3.
80. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).
81. Martin M, ed. Understanding the True Potential of Hybrid Financing Strategies for Social Entrepreneurs. 2011. Impact Economy Working Papers; Vol 2. Impact Economy: Switzerland. Accessed April 19, 2013.
82. Gibbons RD, Weiss DJ, Kupfer DJ, et al.. Using computerized adaptive testing to reduce the burden of mental health assessment [see comment]. Psychiatr Serv. 2008;59:361–368.
83. Gibbons RD, Weiss DJ, Pilkonis PA, et al.. The CAT-DI: a computerized adaptive test for depression. Arch Gen Psychiatry. 2012;69:1104–1112.
84. Bennett GG, Glasgow RE. The delivery of public health interventions via the internet: actualizing their potential. Annu Rev Public Health. 2009;30:273–292.
85. Kellam SG, Mackenzie ACL, Brown CH, et al.. The good behavior game and the future of prevention and treatment. Addict Sci Clin Pract. 2011;6:73–84.
86. Reid JB, Eddy JM. The prevention of antisocial behavior: some considerations in the search for effective interventions. In: Stoff DM, Breiling J, Maser JD, eds. Handbook of Antisocial Behavior. New York, NY: Wiley; 1997:343–356.
87. Patterson GR, Reid JB, Dishion TJ. Antisocial Boys. Eugene, OR: Castalia Pub. Co.; 1992.
88. Reid JB. Prevention of conduct disorder before and after school entry: relating interventions to developmental findings. Dev Psychopathol. 1993;5:243–262.
89. Snyder JJ, Stoolmiller M. Reinforcement and coercion mechanisms in the development of antisocial behavior: the family. In: Reid JB, Patterson GR, Snyder JJ, eds. Antisocial Behavior in Children and Adolescents: A Developmental Analysis and Model for Intervention. Washington, DC: American Psychological Association; 2002:65–100.
90. Spouse J. Bridging theory and practice in the supervisory relationship: a sociocultural perspective. J Adv Nurs. 2001;33:512–522.
91. American Institutes for Research, ed. Conceptual Overview of the Coaching Component of the Reading Professional Development Impact Study. Washington, DC: American Institutes for Research; 2005.
92. Sweeney D. Learning Along the Way: Professional Development by and for Teachers. Portland, ME: Stenhouse; 2003.
93. Hoste L, Dumas B, Signer B. SpeeG: a multimodal speech-and gesture-based text input solution. Proceedings of the International Working Conference on Advanced Visual Interfaces, Naples, Italy. May 22–25, 2012. Tortora G, Levialdi S, Maurizio T., eds. 2012:156–163.
94. Povey K, Ghoshal A, Boulianne G, et al.. The Kaldi Speech Recognition Toolkit. Presentation at ASRU 2011, December 11-15, Honolulu, Hawaii. Procedures ASRU; 2011.
95. Walker W, Lamere P, Kwok P, et al.. Sphinx-4: A Flexible Open Source Framework for Speech Recognition. Mountain View, CA: Sun Microsystems, Inc; 2004. doi:
96. Miner G, Elder J, Hill T, et al.. Practical Text Mining and Statistical Analysis for Non-Structured Text Data Applications. Waltham, MA: Academic Press; 2012.
97. Hall M, Frank E, Holmes G, et al.. The WEKA data mining software: an update. ACM SIGKDD Explorations Newsletter. 2009;11:10–18.
98. Schoenwald SK, Garland AF, Chapman JE, et al.. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health. 2010;38:32–43.
99. Schoenwald SK, Sheidow AJ, Letourneau EJ. Toward effective quality assurance in evidence-based practice: links between expert consultation, therapist fidelity, and child outcomes. J Clin Child Adolesc Psychol. 2004;33:94–104.
100. Kellam SG, Brown CH, Poduska J, et al.. Summary of cohort 2 analyses—supplement to “Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes”. Drug Alcohol Depend. 2008:Available at: and 10.1016/j.drugalcdep.2008.01.004.
101. Williamson DM, Xi X, Breyer FJ. A framework for evaluation and use of automated scoring. Educ Meas Issues Pract. 2012;31:2–13.
102. Xie S, Evanini K, Zechner K. Exploring content features for automated speech scoring. Proceedings of the NAACL-HLT, Montreal, Canada, June 3–8, 2012.
103. Beigi H. Fundamentals of Speaker Recognition. New York, NY: Springer-Verlag; 2011.
104. Jurafsky D, Martin D. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2nd Ed. Upper Saddle River, NJ: Pearson Prentice Hall; 2009.
105. Tamarit V, Martinez-Hinarejos CD, Benedi JM. Estimating the number of segments for improving dialogue act labelling. Nat Lang Eng. 2012;1:1–19.
106. Jeong YM, Lim KT, Lee SE. mGlove: enhancing user experience through hand gesture recognition. Adv Electron Eng Commun Manag. 2012;1:383–386.
107. Riedl M, Biemann C. How text segmentation algorithms gain from topic models. Proceedings of NAACL-HLT Montreal, Canada, June 3–5, 2012. 2012:12.
108. Valente TW. Social Networks and Health: Models, Methods, and Applications. New York, NY: Oxford University Press; 2010.
109. Valente TW, Davis RL. Accelerating the diffusion of innovations using opinion leaders. Am Acad Polit Soc Sci. 1999;566:55–67.
110. Valente TW, Ritt-Olson A, Stacy A, et al.. Peer acceleration: effects of a social network tailored substance abuse prevention program among high-risk adolescents. Addiction. 2007;102:1804–1815.
111. Valente TW, Pumpuang P. Identifying opinion leaders to promote behavior change. Health Educ Behav. 2007;34:881–896.
112. Valente T. Opinion leader interventions in social networks: can change HIV risk behaviour in high risk communities. Br Med J. 2006;333:1082–1083.
113. Morell JA, Hilscher R, Magura S, et al.. Integrating evaluation and agent-based modeling: rationale and an example for adopting evidence-based practices. J Multidisciplinary Eval. 2010;6:32–57.
114. Heath B, Hill R, Ciarallo F. A survey of agent-based modeling practices (January 1998 to July 2008). J Artificial Soc Social Simulation. 2009;12:A143–A177.
115. Ormerod P, Rosewell B. Validation and verification of agent-based models in the social sciences. In: Squazzoni F, ed. Epistemological Aspects of Computer Simulation in the Social Sciences: Second International Workshop, EPOS 2006, Brescia, Italy, October 5–6, 2006: Revised Selected and Invited Papers. Vol. 5466 LNAI. Berlin/New York, NY: Springer; 2009:130–140.
116. Epstein JM. Generative Social Science: Studies in Agent-Based Computational Modeling. Princeton, NJ: Princeton University Press; 2007.
117. Miller JH, Page SE. Complex Adaptive Systems: An Introduction to Computational Models of Social Life. Princeton, NJ: Princeton University Press; 2007.
118. Grimm V, Revilla E, Berger U, et al.. Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science. 2005;310:987–991.
119. Axelrod RM. The Complexity of Cooperation: Agent-Based Models of Competition and Collaboration. Princeton, NJ: Princeton University Press; 1997.
120. Gratch J, Wang N, Gerten J, et al.. Creating rapport with virtual agents. 7th International Conference on Intelligent Virtual Agents. 2007.
121. Watson A, Bickmore T, Cange A, et al.. An internet-based virtual coach to promote physical activity adherence in overweight adults: randomized controlled trial. J Med Internet Res. 2012;14:e1.
122. Muñoz RF, Lenert LL, Delucchi K, et al.. Toward evidence-based internet interventions: a Spanish/English web site for international smoking cessation trials. Nicotine Tob Res. 2006;8:77–87.
123. Munoz RF, Beardslee WR, Leykin Y. Major depression can be prevented. Am Psychol. 2012. Am Psychol. 2012;67:285–295.
124. Chamberlain P, Saldana L, Brown CH, et al.. Implementation of multidimensional treatment foster care in California: a randomized trial of an evidence-based practice. In: Roberts-DeGennaro M, Fogel S, eds. Empirically Supported Interventions for Community and Organizational Change. Chicago, IL: Lyceum Books, Inc. 2010; 218–234.
125. Goldhaber-Fiebert J, Bailey S, Hurlburt M, et al.. Evaluating child welfare policies with decision-analytic simulation models. Adm Policy Ment Health. 2012;39:466–477.
126. Wulczyn F, Kogan J, Dilts J. The effect of population dynamics on performance measurement. Soc Serv Rev. 2001;75:292–317.
127. Valente T. Network interventions. Science. 2012;337:49–53.
128. National Cancer Institute. Cancer Biomedical Informatics Grid. U.S. National Institutes of Health. Available at: Accessed June 20, 2012.
129. The Apache Software Foundation. Apache. Available at: Accessed June 20, 2012.
130. Food and Drug Administration (FDA). FDA outlines oversight of mobile medical applications. FDA Press Release, July 19, 2011. Accessed June 20, 2012.
131. Pantin H, Schwartz SJ, Coatsworth JD, et al.. Familias unidas: A systemic, parent-centered approach to preventing problem behavior in Hispanic adolescents. In: Szapocznik J, Tolan P, Sambrano S (Eds). Preventing Youth Substance Abuse: Science-Based Programs for Children and Adolescents. Washington, DC: American Psychological Association; 2007:211–238.
132. Wallerstein NB, Duran B. Using community-based participatory research to address health disparities. Health Promot Pract. 2006;7:312–323.
133. Madison SM, McKay MM, Paikoff R, et al.. Basic research and community collaboration: necessary ingredients for the development of a family-based HIV prevention program. AIDS Educ Prev. 2000;12:281–298.
134. Begun AL, Berger LK, Otto-Salaj L, et al.. Developing effective social work university—community research collaborations. Soc Work. 2010;55:54–62.
135. Palinkas LA, Soydan H. Translation and Implementation of Evidence Based Practice. New York, NY: Oxford University Press; 2012.
136. Tervalon M, Murray-Garcia J. Cultural humility versus cultural competence: a critical distinction in defining physician training outcomes in multicultural education. J Health Care Poor Underserved. 1998;9:117–125.
137. Braveman P, Gruskin S. Defining equity in health. J Epidemiol Commun Health. 2003;57:254–258.
138. Perrino T, Howe G, Sperling A, et al.. Advancing science through collaborative data sharing and synthesis. Persp Psychol Sci. In press.

implementation science; systems science; behavioral intervention technology; machine learning; computational linguistics; timecast; scientific equity

© 2013 Lippincott Williams & Wilkins, Inc.