The most common mechanisms for integrating processes and problem-solving across the human subject research protection program were ad hoc meetings (23/45; 51%), standing meetings (18/45; 40%), and the use of a shared reporting mechanism (17/45; 38%). The most common mechanisms for integrating processes and problem-solving across institutional CTSA core function groups were standing meetings (27/45; 60%), ad hoc meetings (23/45; 51%), and/or the use of a shared reporting structure (20/45; 44%). Some institutions reported no mechanism for integrating these functions into the human subject research protection program (7/45; 16%) or across institutional CTSA core function groups (5/45; 11%).
Survey Part II:
Respondents attributed a wide variety of services across the protocol life cycle to RSAs programs, often in collaboration with other departments. RSAs participated in, but were usually not the dominant organizers of, required training in human subject protection or good clinical practices (GCP). RSAs led activities related to informed consent and supported the design and review of data and safety monitoring plans and boards. They were the main providers of education in DSMP requirements (24/44; 55%), elective education in human subject protection (24/44; 55%) and GCP (19/44; 43%), training in adverse event reporting (16/44; 36%), regulatory compliance updates (13/44; 30%), and training in response to audit findings (12/44; 27%). Many respondents also indicated a role for the RSA in addressing rights and safety concerns for CTSA-supported projects. These advocacy activities included real-time compliance oversight (25/42; 60%) and the investigation of complaints initiated by staff (21/42; 50%) or participants (26/42; 62%) regarding research conduct (see Table 4).
RSAs both shared in the delivery of collaborative services and provided unique services to protocols affiliated with the CTSA center. For non-CTSA research protocols, the IRB and compliance group provided otherwise-shared services without RSA participation. Respondents also reported that services provided uniquely by the RSA for CTSA research, such as data and safety monitoring plans and boards assistance, direct advocacy, and consent oversight, were often not conducted for non-CTSA research. The most common of these not conducted activities for both non-CTSA (11/39; 28%) and CTSA research (9/41; 22%) was “verification of program readiness to implement a protocol” (see Supplemental Digital Table 1, http://links.lww.com/ACADMED/A102).
Survey Parts I and II: Evaluation activities
In Part I, respondents were asked to describe “a program or activity that provides exceptional value, importance, or innovation in the fulfillment of the RSA functions” and to describe how the quality and value of the program or activity is assessed. Respondents most often described programs for the education of researchers or coordinators (12/26; 46%) or programs to enhance participant safety (8/26; 31%). Other self-reported programs included informed consent oversight, quality assurance, and support of research ethics (each at 7/26; 27%). Among the 26 RSA programs, respondents described three types of assessment methods: (1) qualitative assessments, including both general feedback such as satisfaction surveys, questionnaires, and verbal praise (15/26; 58%) and feedback provided in response to specific activities such as monitoring and/or the review of audit reports (4/26; 15%), (2) quantitative assessments, including tallies of provided services, protocols reviewed, investigators/trainees assisted, and audits performed (4/26; 15%), and (3) outcome-based measures, such as an evaluation of the impact of the RSA services on protocol review turnaround time, adverse events, audit findings, and the elimination of specific research conduct errors after corrective education (3/26; 12%). Six of the 45 institutions (13%) specifically reported that they did not assess the value of their RSA activities. No respondents reported measuring participant-based outcomes to evaluate their research participant advocacy or human subject protection activities. In Part II, narrative descriptions of these evaluation methods provided no additional information for us to assess.
Most programs tracked collaborative and uniquely RSA-provided activities electronically—DSMP design and development (24/38; 63%), informed consent process and document review (30/38 [79%] to 32/38 [84%]), data safety monitoring (22/37; 59%), and the investigation of complaints lodged by staff (19/37; 51%) or participants (21/37; 57%) about research conduct.
The first step in a multistep process to develop a robust system of evaluation for the RSA Best Practice Functions is to assess the current state of practice. The RSA taskforce survey collected information on the organizational structure, activities, and evaluation methods of current CTSA center RSA programs. Currently, these programs provide a wide variety of RSA activities, many of which are complementary to or integrated with other institutional programs to support the safe and ethical conduct of research, and some of which are provided solely by RSA programs. The survey also revealed that RSA programs generally have senior-level supervision and CTSA funding. The inventory of specific RSA activities that we compiled allows us now both to examine how RSA programs fulfill the RSA Best Practice Functions and to identify important issues to consider when designing formal evaluation recommendations.
Function 1: Inclusion of reporting pathways that lead to the appropriate authority and are conflict-of-interest free
RSA programs are led by senior institutional officials within a variety of underlying organizational structures. In general, these structures provide the appropriate reporting pathways for access to individuals who have the authority to implement and act on institutional policy. Thus, we can assess an institution’s fulfillment of Function 1 by examining the authority and reporting pathway afforded to those who implement the RSA functions and the institution’s support for alternate reporting pathways in the event of a conflict of interest or of commitment.
Special challenges emerge for the large CTSA centers for which the official overseeing the fulfillment of the RSA functions has no formal authority at the affiliated institutions otherwise within the scope of the RSA functions, and for which there is no binding reporting pathway. Multiinstitutional CTSA centers may require new organizational models to ensure that Function 1 is fulfilled, perhaps modeled after those recently developed for aligning IRB functions across many independent but cooperating institutions.13
Function 2: Facilitation of integrative, complementary, and unique activities
RSA programs include many complementary and integrated activities that support the safe and ethical conduct of research. In addition to providing broadly applicable research education, RSA programs also fulfill needs that are context-specific and enhance human subject protections through education, oversight, or advocacy. Whereas federally mandated research education may focus on the regulatory aspects of human protections, RSA-provided education targets operational training, and training and assistance with protocol-specific research ethics or safety challenges. These contextual, responsive services are common mechanisms for fulfilling RSA Best Practice Function 2. Of note for future evaluation, we found (1) that some duplicative functions exist, which should prompt institutions to assess the safety net value of this redundancy against the need to streamline and ensure the cost-effectiveness of their RSA activities, and (2) that institutions reported some activities uniquely provided by RSA programs were not conducted for non-CTSA research. This contrast in provision of services affords a unique opportunity for evaluation of the impact of those activities in these two groups.
Function 3: Promotion of an authority with the ability to temporarily suspend activities for safety or ethical reasons
We found that for RSA programs the authority to influence the course of a clinical research activity could be conferred formally to the designated RSA by the senior RSA official, or informally through the RSAs relationships, status, and credibility within the CTSA. We assessed who held this authority indirectly, relying on surrogate questions on our survey about committee membership and voting authority in the hope of mitigating concerns that the institutional integration of RSA activities might contribute to misleading responses. We also found that RSAs were represented on IRBs, scientific review committees, and CTSA council or governance boards at approximately half of the institutions and often RSAs held voting rights. Holding these positions provides RSAs credibility within the clinical research enterprise that then can afford them the necessary influence to effect change in a research project, averting the need to halt a study. In addition, these relationships provide RSAs with access to and influence on the appropriate authorities who can halt a study, if they themselves do not hold that power. In retrospect, we should have included both surrogate and direct questions on our survey to learn more about the authority that RSAs hold within their institutions.
Function 4: Act as a resource to the research community and to research participants
We found that most institutions engage the expertise of the RSA in areas such as regulatory compliance and participants’ rights for the benefit of both their research community and their research participants. As a resource for investigators and staff, RSAs provide expertise by delivering operational and specialized research training and consultation and on-demand targeted services in conduct, oversight, and protection of rights and safety primarily for, but not limited to, CTSA-associated protocols. As a resource for human subject research participants, RSAs often fulfill unique roles by providing services to assure participants’ rights and advocacy, informed consent oversight, and participant safety protections. Recently, several RSA programs have collaborated to serve as a resource to the public at large through community engagement initiatives to raise awareness of participant protections and rights.14
Our study had a few notable limitations. First, some centers found Part I of the survey difficult to complete because of the complex nature of their program’s organization or Part II difficult to complete because the relatively limited survey response choices could not accurately capture complex program activities. Second, our survey did not directly assess Function 3, relying instead on surrogate questions. Finally, the survey data that we collected provided limited definitive information about the fulfillment of the RSA Best Practice Functions or the impact of RSA programs. Although these limitations may be perceived to reflect our survey design, we believe that they reflect the current state of the research subject advocacy field and the limitations of existing evaluation methods. In this regard, our survey represents a critical, early step in the process of developing robust evaluation mechanisms.
Recommendations for the future of RSA Best Practice Functions evaluation
The CTSA consortium model of research subject advocacy is based on the fulfillment of the RSA Best Practice Functions rather than on the provision that institutions conduct specific activities. The RSA activities reported here generally fulfill the broadly worded RSA functions of appropriate reporting, service and education integration, ability to halt a study for ethical reasons, and acting as a resource to the research enterprise. Challenges to the development of more meaningful evaluation methods for assessing these activities include (1) heterogeneous organizational structures, (2) the lack of a definition for what constitutes fulfillment of programmatic and organizational objectives, and (3) few existing measures to assess the magnitude of RSA activity value and impact.
Respondents primarily reported qualitative approaches to evaluating RSA programs, including measuring investigator satisfaction and activity tallies using locally defined metrics. Rarely did they report outcome-based measures. Although most institutions reported tracking both compliance data and RSA activities, few described initiatives that specifically correlate program and compliance outcomes using available data. This gap may represent an important opportunity for RSA programs to incorporate the compliance outcome data that they already collect into an evaluation plan with which to assess, organize, and implement their programs. Qualitative evaluation data remain important as they provide feedback on how best to deliver RSA services. To advance RSA evaluation, we must develop CTSA consortium consensus both to define the expected outcomes of RSA programs and to develop measures for those outcomes while continuing to respect the heterogeneity of locally appropriate program structures.
We recommend dividing potential outcome measures for RSA functions into three categories: operational, research team-based, and participant-based. Operational outcomes should assess the impact of policy, teaching, and services on the research team-based and participant-based outcomes. Research team-based outcomes should measure a program’s adherence to protocol and policy and research documentation. Notably, RSA programs should assess these outcomes by analyzing training, IRB, and compliance data and investigating any correlations with RSA services. Participant-based outcomes should measure the efficacy of investigator training (e.g., informed consent) or the impact of direct-to-participant research advocacy activities on participant outcomes. Although we can use compliance data to glean quantitative assessments of operational and research team-based outcomes, we have no validated measures for assessing participant-based outcomes. To address this gap, two of the authors (R.G.K., K.G.S.) have led and continue to participate in an effort by 15 academic research centers to develop validated participant-based measures of the research experience.15 These outcome measures assess aspects of the participant experience, such as the adequacy of informed consent, a reflection of policy, training, and conduct, and may prove to be valuable tools for evaluating participant-based outcomes across the CTSA consortium.
We find it particularly challenging to evaluate how well RSA programs mitigate risk and prevent ethical or safety lapses from occurring by providing support for appropriate protocol design, DSMPs, or real-time oversight. We may be able to detect reductions in protocol deviations or adverse events by comparing the frequency of these lapses before and after an RSA intervention; however, to do so will require novel approaches to data collection. For decades, the Joint Commission has required hospitals to conduct analyses of the potential impact of their “near misses” in patient care using the Failure Mode and Effects Analysis (FMEA) tool.16 Only one report in the literature describes applying this tool to assess and reduce risk in research.17 Applying the FMEA tool, or a modification of it, may represent an opportunity for us to assess the impact of RSA-prevented harms.
On the basis of the RSA taskforce survey results that we reported here, we recommend the following steps to advance the development of methods to evaluate the value and impact of RSA programs:
- Foster a culture among RSA programs that values the capture and utilization of existing data sources to evaluate the impact of ongoing RSA activities on regulatory compliance, scientific integrity, and participants’ rights and safety.
- Implement pilot demonstration projects to develop common definitions and procedures for a limited test set of outcome measures and disseminate the results to the CTSA consortium. Such projects could include (1) comparing the type and frequency of protocol violations and deviations before and after the delivery of RSA services or any relevant changes in policy, and (2) incorporating RSA-provided activities into classic performance improvement initiatives conducted in response to participant-based outcomes.
- Develop formal RSA program outcome-based measures in alignment with RSA Best Practice Functions, for use consortium-wide.
The implementation of such evaluation methods is complex. The heterogeneity of RSA programs not only grew out of the flexibility that the NCRR encouraged during the early stages of the RSA program but also from the diverse structures and needs of institutions that persist across the CTSA consortium. The development of outcome-based evaluation measures must take this heterogeneity into account while advancing the value and effectiveness of RSA programs.
Acknowledgments: The authors wish to thank Dr. Jody Sachs for her encouragement, insightful comments, and administrative support during the development, fielding, and analysis of this survey, and Ms. Tyler-Lauren Rainer for her technical assistance. The authors also wish to thank the members of the Research Subject Advocacy Survey taskforce who were involved in the design or analysis of this survey: Jane Alexander, Dr. Enrico Cagliero, Dr. Dawn Lantero, Dr. Hal Jenson, Robert Kolb, Liz Martinez, Halia Melnyk, Andrea Nassen, Nancy Needler, Dr. Kathy Powell, Eric Rubinstein, Laurel Yasko, and Jan Zolkower.
Funding/Support: This project was funded in part by the National Center for Research Resources, National Center for Advancing Translational Sciences, and National Institutes of Health, and through the CTSA Program (UL1 TR000043, UL1RR025755, and UL1 RR024140) as part of the Roadmap Initiative, Re-Engineering the Clinical Research Enterprise. The CTSA consortium publications committee approved this report.
Other disclosures: None.
Ethical approval: The IRB chairman at the Rockefeller University confirmed that this study did not constitute human research and therefore was exempt from IRB review
2. Neill KM. Research subject advocate: A new protector of research participants. Account Res. 2003;10:159–174
3. O’Lonergan T. Creative solutions: Research subject advocates: Increase in reports of human subject protection deficiencies bring scrutiny as well as more efforts at education and support. Prot Hum Subj. 2003. Spring:10–11
5. Society of Research Subject Advocates Web site. http://www.srsa.us
. Accessed May 14, 2012.
6. Bramstedt KA. Research subject advocates: To whom are they loyal? Clin Invest Med. 2003;26:64–69
7. Carroll PRProstate Cancer Advocates. . The impact of patient advocacy: The University of California–San Francisco experience. J Urol. 2004;172(5 pt 2):S58–S61
8. Easa D, Kim K, Kato K, et al The research subject advocate at the University of Hawai’i Clinical Research Center: An added resource for protection of human subjects. Hawaii Med J. 2006;65:50–52
9. Martinez RA. Role of research subject advocates in the development of data safety and monitoring plans. J Investig Med. 2004;52:464–469
10. Stroup S, Appelbaum P. The subject advocate: Protecting the interests of participants with fluctuating decisionmaking capacity. IRB. 2003;25:9–11
11. Silber TJ. Protection of children in research: Beyond pediatric risk levels: The emergence of the research subject advocate. J Clin Ethics. 2010;21:221–223
12. Rhonda G. Kost, Chair, Research Subject Advocacy taskforce, Regulatory Knowledge Committee. Personal communication with Meryl Sufian, Evaluation Committee. February 13, 2012
13. Bierer B, Winkler S. Alternative IRB approvals for multisite trials, Harvard reliance agreement. Paper presented at: 4th Annual CTSA Clinical Research Management Workshop; 2011 Bethesda, Md
14. Winkler S. Making sausages from silos. Paper presented at: CTSA Regulatory Knowledge Key Function Committee Face to Face Meeting; 2011 Bethesda, Md
15. Kost RG, Lee LM, Yessis J, Coller BS, Henderson DKResearch Participant Perception Survey Focus Group Subcommittee. . Assessing research participants’ perceptions of their clinical research experiences. Clin Transl Sci. 2011;4:403–413
17. Cody RJ. Anticipating risk for human subjects participating in clinical research: Application of failure mode and effects analysis. Cancer Invest. 2006;24:209–214
Supplemental Digital Content
© 2012 by the Association of American Medical Colleges