Kost, Rhonda G. MD; Reider, Carson PhD; Stephens, Julie; Schuff, Kathryn G. MD; on behalf of the Clinical and Translational Science Award Research Subject Advocacy Survey Taskforce
In 2000, the National Institutes of Health National Center for Research Resources (NCRR) introduced the requirement for institutions to create a research subject advocate (RSA) position at NCRR-funded general clinical research centers (GCRCs) to enhance human research subject protections. The primary function of RSAs would be to ensure that human research studies were “designed and conducted safely and ethically with protection of human subjects accorded the highest priority.”1 The RSA program was initially deployed in 78 NCRR-funded GCRCs; each center was responsible for defining the specific activities of its RSA,2,3 maximizing the flexibility of the position and preserving local discretion in fulfilling institutional and center-specific needs according to the general guidelines.1 Although some members of the Clinical and Translational Science Award (CTSA) RSA taskforce4 and of the national Society of Research Subject Advocates5 have broad experience with the roles fulfilled by RSAs in the GCRCs, no one has compiled a systematic inventory of how GCRCs incorporated the position. Historically, only a few RSA programs have published evidence about their RSA activities.6–11
The implementation of the CTSA program presented an opportunity for the NCRR to extend the platform for research subject advocacy while preserving local control of the funding, design, and scope of RSA roles and responsibilities. Without specific guidance, some CTSA institutions dismantled their RSA programs, either distributing research subject advocacy-specific functions across other institutional entities or abolishing them entirely. Other CTSA centers expanded their RSA programs, adding responsibilities, complementary oversight, and resources. In 2007, to clarify the role of RSAs at CTSA centers, the Consortium Executive Committee (CEC) asked the Regulatory Knowledge and Support (RKS) Key Function Committee (KFC) to convene a taskforce to recommend best practice functions for RSA programs. Through a collaborative, iterative process, the taskforce proposed and the executive committee endorsed four RSA Best Practice Functions in January 2008 (see List 1).4
Subsequently, the CEC asked the RSA taskforce to explore models of RSA Best Practice Function implementation and to make recommendations for the evaluation of these models. Currently, there are no CTSA-endorsed evaluation metrics for assessing the implementation of the RSA Best Practice Functions.12 Anecdotes of RSA program heterogeneity, and the absence of standards or a recent inventory of RSA program implementation strategies, led the taskforce to develop a multistep initiative to (1) describe the models of RSA program organization, associated activities, and current modes of evaluation, (2) define meaningful outcome measures for those activities, and (3) develop the methods to assess these outcomes. In this report, we describe the method and results of a survey designed to address the first step of this initiative. On the basis of the results, we then present recommendations for the next step of the process—defining outcome measures to enable CTSA centers to evaluate whether they have fulfilled the RSA Best Practice Functions.
The taskforce created a two-part, Web-based survey that they deployed from May to October 2010. Part I contained 24 questions regarding leadership, organizational structure, governance, scope, degree of collaboration and integration, and funding of RSA activities implemented to fulfill the Best Practice Functions. Part I also asked respondents to identify any particularly valuable RSA practices and to describe the methods for their evaluation. Part II presented an extensive list of activities that could potentially fulfill the four RSA Best Practice Functions. Respondents were asked to attribute the conduct of each activity to the office at their institution that performed that activity (e.g., “Education,” “RSA,” “Quality assurance/compliance,” “Institutional review board [IRB]”) and to describe “any methods used to measure the value of these activities” (see Supplemental Digital Appendix 1, http://links.lww.com/ACADMED/A101).
The questions in Parts I and II of the survey align with the RSA Best Practice Functions (see List 1). For example, the questions on governance and organization asked about the reporting structure of each RSA program (Function 1). Other questions assessed the integrative and complementary nature of RSA activities (Function 2). Questions on RSAs’ representation on committees and their voting authority indirectly assessed their ability to temporarily suspend a study for safety or ethical reasons (Function 3). The detailed questions about specific RSA activities in Part II considered programs’ provision of resources to the research community and to research participants (Function 4).
In 2010, the members of the RSA taskforce who direct or implement RSA programs conducted alpha and beta testing of the survey. They refined the overall approach through an initial face-to-face meeting. Taskforce leadership then revised the survey and iteratively tested its face and content validity with taskforce members to optimize readability, content, and clarity. They piloted the survey with 10 members, who field-tested the survey with colleagues familiar with RSA activities, and provided feedback to the leadership. They chose to use surrogate questions for Function 3 based on testers’ feedback on the difficulty of localizing RSA authority among multiple delegated or highly integrated programs. The CTSA RKS KFC and its principal investigator liaison endorsed the final survey and deployed it through the RKS voting representatives.
Each CTSA’s voting representative to the RKS KFC received an e-mail explanation of the project and a link to the online survey. One of us (C.R.) provided telephone and e-mail support for questions. We distinguished whether a survey response reflected an RSA program spanning multiple entities within a CTSA center or only a single institution, and whether RSA services were provided to CTSA and non-CTSA researchers. Nonresponding institutions received e-mail and telephone reminders to encourage survey completion. We did not collect any personal information other than contact information for the survey respondents. In consultation with the Rockefeller University IRB chair, our survey was deemed exempt from IRB review. We analyzed our data using descriptive statistics (SAS 9.2; SAS Institute, Cary, North Carolina).
Between May and October 2010, we received responses from 45 RSA programs to Part I, and from 42 to Part II. Overall, 43 of the 46 (93%) CTSA centers funded at the time participated. Included in Table 1 are characteristics of the respondents and the RSA programs.
Survey Part I
Organization and scope.
A majority of respondents reported that the highest oversight authority for their RSA program was either a senior university official or the CTSA center leadership. Most respondents reported a designated RSA at their institution as well as a university-recognized RSA program. However, some institutions indicated that RSA activities were distributed across multiple programs without a primary locus of responsibility (see Table 1).
All but two RSA programs were funded with CTSA resources. Of the institutions with one or more persons designated as an RSA (36/45; 80%), all reported that their program was supported by their CTSA and for 16 of those 36 (44%) by their RKS core. Approximately two-thirds (24/36; 67%) of programs with a designated RSA provided services across their entire CTSA center; the remainder (12/36; 33%) served only a single institutional entity such as a research center. Of the institutions with an office designated for RSA functions, half provided services to both CTSA and non-CTSA-supported projects (16/32; 50%). Respondents also reported that RSA services were not provided for some elements of CTSA research (11/45; 24%) and non-CTSA research (19/45; 42%) at their centers (see Table 1).
Service, collaboration, and integration.
The categories of activities provided by RSA programs were broad, including educational, oversight, and policy development activities, and provision of services to investigators and participants. Respondents viewed most of the activities performed by the RSA programs as collaborative, complementary, and/or integrated with other institutional services. The most commonly reported services uniquely provided by the RSA program included data safety and monitoring plan (DSMP) development (16/45; 36%), research subject rights/advocacy (11/45; 24%), and informed consent oversight (10/45; 22%) (see Table 2).
The most common mechanisms for integrating processes and problem-solving across the human subject research protection program were ad hoc meetings (23/45; 51%), standing meetings (18/45; 40%), and the use of a shared reporting mechanism (17/45; 38%). The most common mechanisms for integrating processes and problem-solving across institutional CTSA core function groups were standing meetings (27/45; 60%), ad hoc meetings (23/45; 51%), and/or the use of a shared reporting structure (20/45; 44%). Some institutions reported no mechanism for integrating these functions into the human subject research protection program (7/45; 16%) or across institutional CTSA core function groups (5/45; 11%).
Survey Part II:
Respondents attributed a wide variety of services across the protocol life cycle to RSAs programs, often in collaboration with other departments. RSAs participated in, but were usually not the dominant organizers of, required training in human subject protection or good clinical practices (GCP). RSAs led activities related to informed consent and supported the design and review of data and safety monitoring plans and boards. They were the main providers of education in DSMP requirements (24/44; 55%), elective education in human subject protection (24/44; 55%) and GCP (19/44; 43%), training in adverse event reporting (16/44; 36%), regulatory compliance updates (13/44; 30%), and training in response to audit findings (12/44; 27%). Many respondents also indicated a role for the RSA in addressing rights and safety concerns for CTSA-supported projects. These advocacy activities included real-time compliance oversight (25/42; 60%) and the investigation of complaints initiated by staff (21/42; 50%) or participants (26/42; 62%) regarding research conduct (see Table 4).
RSAs both shared in the delivery of collaborative services and provided unique services to protocols affiliated with the CTSA center. For non-CTSA research protocols, the IRB and compliance group provided otherwise-shared services without RSA participation. Respondents also reported that services provided uniquely by the RSA for CTSA research, such as data and safety monitoring plans and boards assistance, direct advocacy, and consent oversight, were often not conducted for non-CTSA research. The most common of these not conducted activities for both non-CTSA (11/39; 28%) and CTSA research (9/41; 22%) was “verification of program readiness to implement a protocol” (see Supplemental Digital Table 1, http://links.lww.com/ACADMED/A102).
Survey Parts I and II: Evaluation activities
In Part I, respondents were asked to describe “a program or activity that provides exceptional value, importance, or innovation in the fulfillment of the RSA functions” and to describe how the quality and value of the program or activity is assessed. Respondents most often described programs for the education of researchers or coordinators (12/26; 46%) or programs to enhance participant safety (8/26; 31%). Other self-reported programs included informed consent oversight, quality assurance, and support of research ethics (each at 7/26; 27%). Among the 26 RSA programs, respondents described three types of assessment methods: (1) qualitative assessments, including both general feedback such as satisfaction surveys, questionnaires, and verbal praise (15/26; 58%) and feedback provided in response to specific activities such as monitoring and/or the review of audit reports (4/26; 15%), (2) quantitative assessments, including tallies of provided services, protocols reviewed, investigators/trainees assisted, and audits performed (4/26; 15%), and (3) outcome-based measures, such as an evaluation of the impact of the RSA services on protocol review turnaround time, adverse events, audit findings, and the elimination of specific research conduct errors after corrective education (3/26; 12%). Six of the 45 institutions (13%) specifically reported that they did not assess the value of their RSA activities. No respondents reported measuring participant-based outcomes to evaluate their research participant advocacy or human subject protection activities. In Part II, narrative descriptions of these evaluation methods provided no additional information for us to assess.
Most programs tracked collaborative and uniquely RSA-provided activities electronically—DSMP design and development (24/38; 63%), informed consent process and document review (30/38 [79%] to 32/38 [84%]), data safety monitoring (22/37; 59%), and the investigation of complaints lodged by staff (19/37; 51%) or participants (21/37; 57%) about research conduct.
The first step in a multistep process to develop a robust system of evaluation for the RSA Best Practice Functions is to assess the current state of practice. The RSA taskforce survey collected information on the organizational structure, activities, and evaluation methods of current CTSA center RSA programs. Currently, these programs provide a wide variety of RSA activities, many of which are complementary to or integrated with other institutional programs to support the safe and ethical conduct of research, and some of which are provided solely by RSA programs. The survey also revealed that RSA programs generally have senior-level supervision and CTSA funding. The inventory of specific RSA activities that we compiled allows us now both to examine how RSA programs fulfill the RSA Best Practice Functions and to identify important issues to consider when designing formal evaluation recommendations.
Function 1: Inclusion of reporting pathways that lead to the appropriate authority and are conflict-of-interest free
RSA programs are led by senior institutional officials within a variety of underlying organizational structures. In general, these structures provide the appropriate reporting pathways for access to individuals who have the authority to implement and act on institutional policy. Thus, we can assess an institution’s fulfillment of Function 1 by examining the authority and reporting pathway afforded to those who implement the RSA functions and the institution’s support for alternate reporting pathways in the event of a conflict of interest or of commitment.
Special challenges emerge for the large CTSA centers for which the official overseeing the fulfillment of the RSA functions has no formal authority at the affiliated institutions otherwise within the scope of the RSA functions, and for which there is no binding reporting pathway. Multiinstitutional CTSA centers may require new organizational models to ensure that Function 1 is fulfilled, perhaps modeled after those recently developed for aligning IRB functions across many independent but cooperating institutions.13
Function 2: Facilitation of integrative, complementary, and unique activities
RSA programs include many complementary and integrated activities that support the safe and ethical conduct of research. In addition to providing broadly applicable research education, RSA programs also fulfill needs that are context-specific and enhance human subject protections through education, oversight, or advocacy. Whereas federally mandated research education may focus on the regulatory aspects of human protections, RSA-provided education targets operational training, and training and assistance with protocol-specific research ethics or safety challenges. These contextual, responsive services are common mechanisms for fulfilling RSA Best Practice Function 2. Of note for future evaluation, we found (1) that some duplicative functions exist, which should prompt institutions to assess the safety net value of this redundancy against the need to streamline and ensure the cost-effectiveness of their RSA activities, and (2) that institutions reported some activities uniquely provided by RSA programs were not conducted for non-CTSA research. This contrast in provision of services affords a unique opportunity for evaluation of the impact of those activities in these two groups.
Function 3: Promotion of an authority with the ability to temporarily suspend activities for safety or ethical reasons
We found that for RSA programs the authority to influence the course of a clinical research activity could be conferred formally to the designated RSA by the senior RSA official, or informally through the RSAs relationships, status, and credibility within the CTSA. We assessed who held this authority indirectly, relying on surrogate questions on our survey about committee membership and voting authority in the hope of mitigating concerns that the institutional integration of RSA activities might contribute to misleading responses. We also found that RSAs were represented on IRBs, scientific review committees, and CTSA council or governance boards at approximately half of the institutions and often RSAs held voting rights. Holding these positions provides RSAs credibility within the clinical research enterprise that then can afford them the necessary influence to effect change in a research project, averting the need to halt a study. In addition, these relationships provide RSAs with access to and influence on the appropriate authorities who can halt a study, if they themselves do not hold that power. In retrospect, we should have included both surrogate and direct questions on our survey to learn more about the authority that RSAs hold within their institutions.
Function 4: Act as a resource to the research community and to research participants
We found that most institutions engage the expertise of the RSA in areas such as regulatory compliance and participants’ rights for the benefit of both their research community and their research participants. As a resource for investigators and staff, RSAs provide expertise by delivering operational and specialized research training and consultation and on-demand targeted services in conduct, oversight, and protection of rights and safety primarily for, but not limited to, CTSA-associated protocols. As a resource for human subject research participants, RSAs often fulfill unique roles by providing services to assure participants’ rights and advocacy, informed consent oversight, and participant safety protections. Recently, several RSA programs have collaborated to serve as a resource to the public at large through community engagement initiatives to raise awareness of participant protections and rights.14
Our study had a few notable limitations. First, some centers found Part I of the survey difficult to complete because of the complex nature of their program’s organization or Part II difficult to complete because the relatively limited survey response choices could not accurately capture complex program activities. Second, our survey did not directly assess Function 3, relying instead on surrogate questions. Finally, the survey data that we collected provided limited definitive information about the fulfillment of the RSA Best Practice Functions or the impact of RSA programs. Although these limitations may be perceived to reflect our survey design, we believe that they reflect the current state of the research subject advocacy field and the limitations of existing evaluation methods. In this regard, our survey represents a critical, early step in the process of developing robust evaluation mechanisms.
Recommendations for the future of RSA Best Practice Functions evaluation
The CTSA consortium model of research subject advocacy is based on the fulfillment of the RSA Best Practice Functions rather than on the provision that institutions conduct specific activities. The RSA activities reported here generally fulfill the broadly worded RSA functions of appropriate reporting, service and education integration, ability to halt a study for ethical reasons, and acting as a resource to the research enterprise. Challenges to the development of more meaningful evaluation methods for assessing these activities include (1) heterogeneous organizational structures, (2) the lack of a definition for what constitutes fulfillment of programmatic and organizational objectives, and (3) few existing measures to assess the magnitude of RSA activity value and impact.
Respondents primarily reported qualitative approaches to evaluating RSA programs, including measuring investigator satisfaction and activity tallies using locally defined metrics. Rarely did they report outcome-based measures. Although most institutions reported tracking both compliance data and RSA activities, few described initiatives that specifically correlate program and compliance outcomes using available data. This gap may represent an important opportunity for RSA programs to incorporate the compliance outcome data that they already collect into an evaluation plan with which to assess, organize, and implement their programs. Qualitative evaluation data remain important as they provide feedback on how best to deliver RSA services. To advance RSA evaluation, we must develop CTSA consortium consensus both to define the expected outcomes of RSA programs and to develop measures for those outcomes while continuing to respect the heterogeneity of locally appropriate program structures.
We recommend dividing potential outcome measures for RSA functions into three categories: operational, research team-based, and participant-based. Operational outcomes should assess the impact of policy, teaching, and services on the research team-based and participant-based outcomes. Research team-based outcomes should measure a program’s adherence to protocol and policy and research documentation. Notably, RSA programs should assess these outcomes by analyzing training, IRB, and compliance data and investigating any correlations with RSA services. Participant-based outcomes should measure the efficacy of investigator training (e.g., informed consent) or the impact of direct-to-participant research advocacy activities on participant outcomes. Although we can use compliance data to glean quantitative assessments of operational and research team-based outcomes, we have no validated measures for assessing participant-based outcomes. To address this gap, two of the authors (R.G.K., K.G.S.) have led and continue to participate in an effort by 15 academic research centers to develop validated participant-based measures of the research experience.15 These outcome measures assess aspects of the participant experience, such as the adequacy of informed consent, a reflection of policy, training, and conduct, and may prove to be valuable tools for evaluating participant-based outcomes across the CTSA consortium.
We find it particularly challenging to evaluate how well RSA programs mitigate risk and prevent ethical or safety lapses from occurring by providing support for appropriate protocol design, DSMPs, or real-time oversight. We may be able to detect reductions in protocol deviations or adverse events by comparing the frequency of these lapses before and after an RSA intervention; however, to do so will require novel approaches to data collection. For decades, the Joint Commission has required hospitals to conduct analyses of the potential impact of their “near misses” in patient care using the Failure Mode and Effects Analysis (FMEA) tool.16 Only one report in the literature describes applying this tool to assess and reduce risk in research.17 Applying the FMEA tool, or a modification of it, may represent an opportunity for us to assess the impact of RSA-prevented harms.
On the basis of the RSA taskforce survey results that we reported here, we recommend the following steps to advance the development of methods to evaluate the value and impact of RSA programs:
1. Foster a culture among RSA programs that values the capture and utilization of existing data sources to evaluate the impact of ongoing RSA activities on regulatory compliance, scientific integrity, and participants’ rights and safety.
2. Implement pilot demonstration projects to develop common definitions and procedures for a limited test set of outcome measures and disseminate the results to the CTSA consortium. Such projects could include (1) comparing the type and frequency of protocol violations and deviations before and after the delivery of RSA services or any relevant changes in policy, and (2) incorporating RSA-provided activities into classic performance improvement initiatives conducted in response to participant-based outcomes.
3. Develop formal RSA program outcome-based measures in alignment with RSA Best Practice Functions, for use consortium-wide.
The implementation of such evaluation methods is complex. The heterogeneity of RSA programs not only grew out of the flexibility that the NCRR encouraged during the early stages of the RSA program but also from the diverse structures and needs of institutions that persist across the CTSA consortium. The development of outcome-based evaluation measures must take this heterogeneity into account while advancing the value and effectiveness of RSA programs.
Acknowledgments: The authors wish to thank Dr. Jody Sachs for her encouragement, insightful comments, and administrative support during the development, fielding, and analysis of this survey, and Ms. Tyler-Lauren Rainer for her technical assistance. The authors also wish to thank the members of the Research Subject Advocacy Survey taskforce who were involved in the design or analysis of this survey: Jane Alexander, Dr. Enrico Cagliero, Dr. Dawn Lantero, Dr. Hal Jenson, Robert Kolb, Liz Martinez, Halia Melnyk, Andrea Nassen, Nancy Needler, Dr. Kathy Powell, Eric Rubinstein, Laurel Yasko, and Jan Zolkower.
Funding/Support: This project was funded in part by the National Center for Research Resources, National Center for Advancing Translational Sciences, and National Institutes of Health, and through the CTSA Program (UL1 TR000043, UL1RR025755, and UL1 RR024140) as part of the Roadmap Initiative, Re-Engineering the Clinical Research Enterprise. The CTSA consortium publications committee approved this report.
Other disclosures: None.
Ethical approval: The IRB chairman at the Rockefeller University confirmed that this study did not constitute human research and therefore was exempt from IRB review