Dyrbye, Liselotte N. MD; Thomas, Matthew R. MD; Papp, Klara K. PhD; Durning, Steven J. MD
The ethical conduct of research is essential to maintaining the social contract necessary for researchers to advance science. In recent years, substantial attention has been given to the responsible conduct of research (RCR), mostly in response to researchers’ misconduct. Academic health centers and academic and scientific societies are educating researchers in areas of RCR (e.g., data acquisition and conflict of interest), although training goals vary.1,2 RCR instructional programs have typically been designed for clinician investigators engaged in biomedical research, because these investigators commonly conduct research requiring informed consent.
Within academic health centers, however, a growing group of investigators are conducting medical education research. As with biomedical research, medical education research must be conducted in a rigorous and ethical fashion to generate trustworthy and generalizable results needed to facilitate change and gain the trust of fellow researchers, whose work should build on existing studies. Conduct of medical education research requires familiarity with RCR principles as they apply to medical education. Because the subjects of medical education research are often medical students—who are considered a special class of subjects by the Office for Human Research Protection in the U.S. Department of Health and Human Services—researchers engaged in medical education research need to take into consideration how to best uphold confidentiality and recruit learners as research subjects without implicit or explicit coercion, to optimize students’ ability to give true consent (e.g., students may volunteer for research for the sole purpose of gaining a favorable recommendation or grade).3,4 Although there is some debate about what constitutes research in medical education,5 most academic health centers require that all research activities involving human subjects (e.g., students) that lead to generalizable knowledge (such as program evaluation where the results are disseminated beyond the institution) must be reviewed by the local institutional review board (IRB) to ensure that the aforementioned concerns are adequately addressed.4–6
Clinician educators, as opposed to clinician investigators, conduct the majority of the medical education research, because they have access to learners, work in the field of medical education, and seek to publish to achieve academic promotion. As such, they should be aware of federal and institutional human research guidelines needed to protect the confidentiality of student participants in research and to build safeguards (avoid mandatory prospective research participation, provide reassurances that there are no ill consequences for not participating in research, etc.) to protect students from coercion and maximize their ability to give true consent.5 RCR areas previously identified as important in medical education research include recruitment of students, informed consent and confidentiality practices, and use of existing medical education data for research (analogous to chart review of existing data initially obtained for nonresearch purposes in the clinical research realm).6 Yet, it is not known how familiar clinician educators are with their local IRB policies pertaining to these RCR areas. Recent publications suggest that some clinician educators have experienced unnecessary frustration during the IRB review process.7–9 These publications highlight difficulties with the process of IRB review (tediousness of required paperwork, multiple paper copies, prolonged delays, etc.) for medical education research.8,9 Such difficulties, and the reported substantial variability in revisions requested by multiple IRBs to a single medical education research protocol,8 present potential barriers to much-needed multiinstitutional medical education research.
To facilitate medical education research, clinician educators and their local IRBs need a productive working relationship to advance science and protect students. Clinician educators need awareness of RCR as it applies to medical education, and IRBs need to communicate their policies in a clear manner and then apply them consistently. An understanding of how familiar clinician educators engaged in medical education research are with the areas of RCR most pertinent to them, and their views of the clarity of their local IRB policies pertaining to medical education research, could be helpful to IRBs and to academic health centers and academic and scientific societies currently providing instruction in RCR. Insight into clinician educators’ levels of satisfaction with their IRBs’ reviews of medical education research protocols and those educators’ ideas for facilitating medical education research could provide further impetus for change.
With these considerations in mind, we asked an organization of North American clerkship directors, the Clerkship Directors in Internal Medicine (CDIM), to include certain questions in their annual survey of institutional members (typically clinician educators who are clerkship directors in internal medicine). These questions were about (1) familiarity with, and clarity of, three IRB policies of particular importance to medical education research, (2) satisfaction with IRB review of medical education research protocols, and (3) how medical education research may be facilitated. Given that RCR programs have not been designed for clinician educators, and IRBs have less experience with reviewing medical education research protocols when compared with clinical investigation protocols, we hypothesized that clinician educators would report struggles with pertinent IRB policies, dissatisfaction with IRB review of medical education research protocols, and a need for consistent education and application of RCR principles as they apply to medical education research.
In April 2006, CDIM surveyed its U.S. and Canadian institutional members. A total of 110 of 143 departments of medicine in North America have institutional members in CDIM. We sent all CDIM institutional members an e-mail message with a cover letter linked to the Web-based survey. Nonresponders were contacted up to three additional times by e-mail, regular mail, or telephone. Participants were blinded to any specific hypothesis of the study. Participation was elective, and all those surveyed were told that their and their institutions’ identities would not be revealed. The IRB at the Uniformed Services University of the Health Sciences F. Edward Hébert School of Medicine approved this study.
The survey included questions pertaining to demographic information about the respondent (e.g., age, sex, academic rank, number of matriculating medical students, number of years in their primary job description). One section assessed clinician educators’ perceptions and experiences with IRBs. For that section, two of us (L.N.D. and S.J.D.) had used the available literature and our own expertise to develop the questions. The questions for all sections were then shown to a group of experienced educators and further refined, then reviewed and modified by members of CDIM’s research committee. After the research committee modifications, the questions were presented to the CDIM council and, after additional revisions, accepted by the CDIM council for inclusion on CDIM’s 2006 survey.
The final questions in the section labeled “Institutional Review Board and Medical Education Research” included questions about each member’s past experience of submitting a medical education research protocol to an IRB; the number of medical education research protocols the member submitted to an IRB within the past three years; the number of those IRB submissions where the member was the principal investigator or coprincipal investigator; the number of submitted medical education research protocols considered exempt, of minimal risk (thus receiving expedited review), or of more than minimal risk (thus receiving full review) by an IRB; and the number of protocols rejected/not approved by an IRB. The remainder of that section of the survey covered three topics, discussed below: (1) familiarity with and clarity of three IRB policies of particular importance to medical education research, (2) satisfaction with IRB review of medical education research protocols, and (3) ways in which medical education research may be facilitated.
Familiarity and clarity of three IRB policies.
Within the topic of familiarity with and clarity of three IRB policies, respondents were asked to indicate whether they were familiar with the IRB policy on each of the following three education-research-related policies: recruitment of students for medical education research, medical education research consent and confidentiality practices, and use of existing medical education data for research. For each policy with which the respondent indicated familiarity, the respondent was asked to specify the clarity of the IRB policy on a four-point Likert-type scale (1 = very clear, 2 = somewhat clear, 3 = somewhat unclear, and 4 = very unclear).
Satisfaction with IRB review of medical education research protocols.
Respondents with experience submitting a medical education research protocol to an IRB were asked about their satisfaction with the IRB’s overall efficiency, the expertise of the IRB committee members in medical education research, the revisions to the protocol asked for by the IRB, and the required IRB paperwork. Response categories to the satisfaction questions used a four-point Likert scale (one to four, from satisfied to dissatisfied), with an option to indicate unknown or not applicable. This section of the survey also included questions about respondents’ experience with multiinstitutional medical education research projects in the past three years, whether there had been notable differences in expectations of the various involved IRBs, and whether multiple reviews of the same IRB protocol resulted in helpful suggestions or improvements to the protocol.
Facilitation of medical education research.
The section regarding facilitation of medical education research asked respondents whether the IRB’s policies affected their ability to do research (response categories: 1 = very encouraging, 2 = somewhat encouraging, 3 = neutral, 4 = somewhat inhibits, and 5 = greatly inhibits) and whether they thought that a national consensus statement regarding the IRB’s role in medical education research would be of benefit to themselves and/or their local IRBs.
Descriptive summary statistics were used to summarize responses to all questions. Responses to the questions about the clarity of IRB policies regarding medical education research and satisfaction with the IRB are presented as three categories for simplicity and because of the small sample size. Responses to the questions regarding clarity of IRB policies were collapsed into clear (very clear and somewhat clear), unclear (somewhat unclear, very unclear), and unfamiliar with the policy. For responses to the questions about satisfaction with the respondent’s local IRB, responses were collapsed into satisfied (satisfied, somewhat satisfied), dissatisfied (somewhat dissatisfied, dissatisfied), and unknown or not applicable. We hypothesized that respondents who had not previously submitted a medical education research protocol to an IRB would not be familiar with IRB policies related to medical education research. Thus, familiarity with IRB policies was dichotomized as familiar versus not familiar. Respondents with experience submitting a medical education research protocol to an IRB were separated into two groups depending on their responses to questions pertaining to clarity of IRB education-research-related policies and satisfaction with the specified aspects of their local IRBs (i.e., clear versus not clear and satisfied versus not satisfied). Faculty covariates considered included age, sex, academic rank, years in primary role, and number of previous medical education research protocols submitted to an IRB within the last three years, when applicable. Categorical data were analyzed using the chi-square test. Because of skewed distributions for some measures, nonparametric statistical procedures (i.e., Kruskal-Wallis and pairwise Wilcoxon rank-sum test) were used for continuous variables.
Eighty-three of 110 members (a response rate of 76%) completed the survey. Among the respondents, the majority were middle aged (mean 44.86 ± 6.98 years) and male (51/82, 62%; see Table 1). Most held the academic rank of associate professor (39/83, 47%) and had been in their primary role for an average of seven years (mean 7.21 ± 5.43 years, range 6 months to 24 years). Respondents reported a mean of 140.25 ± 46.54 matriculating students at their medical school.
Only 50 of the 83 responders (60%) had ever submitted a medical education research protocol to an IRB. Among those, the mean number of such submissions in the past three years was 3.1 ± 2.7. Forty (40/46, 87%) respondents had served as the principal investigator or coprincipal investigator on one or more of the medical education research projects they had submitted. The responding clinician educators had submitted a total of 154 medical education research protocols to an IRB. Nearly all of those proposals were deemed exempt (74/154, 48%) or considered minimal risk (71/154, 46%); few (9/154, 6%) went to full review, and no protocols were rejected or not approved by the IRB.
Familiarity and clarity of three IRB policies.
Nearly a quarter of respondents indicated they were unfamiliar with their IRB’s policy on recruitment of students for medical education research (19/73, 26%), consent and confidentiality practices for medical education research (20/74, 27%), or use of existing medical education data for research (20/74, 27%; see Table 2). Surprisingly, 25% (18/73) of respondents were unfamiliar with all three of the just-mentioned IRB policies. Respondents unfamiliar with all those policies were similar with respect to age, sex, academic rank, and years in primary role, but they were less likely to have ever submitted a medical education research protocol than were those respondents familiar with one or more of those IRB policies (6/18; 33% versus 42/55; 76%, P = .0003, chi-square test, Fisher exact test, two tailed). Approximately one in seven of those who had previously submitted a medical education research protocol to an IRB was unfamiliar with one or more of the three IRB policies (7/48, 15%).
Among those respondents who had some familiarity (i.e., did not indicate that they were “not familiar with the policy”) with the three IRB policies, approximately half considered their local IRB’s policies regarding recruitment of students for medical education research, research consent and confidentiality practices for medical education research, and use of existing medical education data for research to be very clear or somewhat clear (47%–52%; see Table 2). Indicating that the policy regarding recruitment of students for medical education research was clear or unclear did not vary by age, sex, academic rank, year in primary role, or previous submission of a medical education IRB protocol (all P > .05; Kruskal-Wallis or chi-square tests). Whether respondents considered the consent and confidentiality practices policy for medical education research and/or use of existing medical education data for research policy as clear or unclear also did not vary by age, sex, year in primary role, or previous submission of a medical education IRB protocol, but a difference was found by academic rank. Seventy-five percent (6/8) of professors, 24% (7/29) of associate professors, and 29% (5/17) of assistant professors indicated that the policy on consent and confidentiality practices of their IRB was unclear (P = .02 between groups [chi-square]), with only pairwise difference between professors and associate professors significant (P = .01; Fisher exact test, two tailed). Similarly, 75% (6/8) of professors, 24% (7/29) of associate professors and 35% (6/17) of assistant professors reported that their IRB’s policy on use of existing medical education data for research was unclear (P = .03 for difference between groups, with only pairwise comparison between professor and associate professor significant; P = .01; Fisher exact test, two tailed).
Satisfaction with IRB review of medical education research protocols.
Half to two thirds of respondents with experience submitting a medical education research protocol to an IRB were satisfied or somewhat satisfied with the overall efficiency of the IRB process, such as the turnaround time for protocols (25/49 [51%] and 10/49 [20%]; see Table 3); the revisions, such as changes to the cover letter, requested by the IRB (17/49 [35%] and 10/49 [20%]); and the expertise of IRB committee members in medical education research (10/49 [20%] and 18/49 [37%]). Respondents were split with respect to satisfaction with the required paperwork. Being satisfied (i.e., satisfied or somewhat satisfied) or dissatisfied (i.e., somewhat dissatisfied or dissatisfied) with these process-related aspects of the IRB inquired about did not vary by age, sex, academic rank, year in primary role, or number of medical education IRB submissions within the last three years (all P > .05; Kruskal-Wallis or chi-square tests).
More than a third of respondents (31/83; 37%) had been part of a multiinstitutional research project in the past three years; 60% of those respondents (18/30) agreed that there were notable differences in the expectations of various institutional IRBs, with only two (<10%) reporting that the multiple IRB reviews resulted in helpful suggestions or improvements to the protocol.
Facilitation of medical education research.
Nearly half of respondents (33/69; 48%) viewed IRB policies at their local institution as having no effect on their ability to do medical education research. The remaining 20 respondents (29%) considered their IRB’s policies to somewhat inhibit (19; 28%) or greatly inhibit (1; 1%) their ability to do research. Fewer felt somewhat encouraged (9; 13%) or very encouraged (6; 9%) to do research as a result of the IRB’s policies at their local institution.
Nearly all respondents agreed (66/73; 90%) that they would benefit from a national consensus statement regarding the IRB’s role in medical education research. Most (63/72; 88%) also thought their local IRB would likewise benefit from such a statement. Respondents with experience submitting a medical education research protocol to an IRB were just as likely to agree with both statements regarding a need for a national consensus as were respondents with no such experience (both P > .05; chi-square test, Fisher exact test). Similarly, belief of personal and/or institutional benefit from a national consensus did not vary by response to questions about familiarity with IRB policies, satisfaction with IRB processes, or clarity of IRB policies. Half (37/73; 51%) indicated they would be more likely to conduct medical education research if they had a better understanding of the IRB’s role and requirements in medical education research.
Clinician educators work on the front line of medical education. Because scholarly activity is typically required of clinician educators, it comes as no surprise that the majority of our respondents (60%) had experience with submitting a medical education research protocol to an IRB. Most of those protocols were deemed exempt or minimal risk (94%), and none were rejected. This suggests that CDIM institutional members are actively engaged in medical education research that is almost always being deemed exempt or is approved by their IRBs.
Despite these apparent successes, our findings suggest that clinician educators may be unfamiliar with key RCR principles—confidentiality, coercion, and informed consent4—as they apply to medical education research. Nearly one out of every four respondents were unfamiliar with their IRB policy on recruitment of students for medical education research (26%), medical education research consent and confidentiality practices (27%), or use of existing medical education data for research (27%). One in four (28%) were unfamiliar with all three of the aforementioned IRB policies. Because individuals often learn best when the need arises, it was not surprising to find that those with experience submitting a medical education research protocol were more likely to be familiar with these RCR areas than were respondents without such experience (P = .0003). Despite this experience, however, one in seven remained unfamiliar with one or more of the three IRB policies (14.6%).
Although we cannot ascertain from our data the reasons for the lack of familiarity with key RCR principles for medical education research, we speculate from our own experience with RCR training that the chasm in knowledge among our respondents is a reflection of typical RCR education.
On review of products from the U.S. Office of Research Integrity–Association of American Medical Colleges collaborative for educating scientists and trainees in RCR,10 only one addresses issues pertaining to educational research.11 This product, described in Family Medicine, briefly mentions the need for IRB review of educational research, the need for students to give consent for participation, and the risk of coercion. To empower researchers to perform RCR, they first need to know about policies relevant to their work.12 Although RCR principles pertinent to biomedical research and medical education research overlap, the work of medical education researchers has some unique considerations. As such, the goals of RCR education for clinician educators should be tailored to meet their needs and the needs of their subjects. As half of the respondents indicated they would be more likely to conduct medical education research if they had a better understanding of the IRB’s role and requirements in medical education research (51%), such training might facilitate important advancements in the field of medical education research as well as in the careers of clinician educators.
Although the majority of respondents familiar with the key educational IRB policies found them to be clear, those with experience in multiinstitutional research reported notable differences in IRBs’ expectations between institutions, and also a lack of perceived improvement to the research protocol from multiinstitutional review; these finding are consistent with perspectives previously reported.8 The differences may be attributable to variability in how policies are interpreted and applied, and they threaten the validity of the IRB process for medical education research. Furthermore, the lack of perceived benefit from multiple IRB reviews, particularly considering the cumbersome paperwork and delays involved,8 may impede much-needed multiinstitutional research. Because multiinstitutional, generalizable studies are greatly needed in medical education research,13–16 finding solutions to better enable multiinstitutional research is vital if such research is to advance.
In the clinician investigator area, consensus statements have assisted with promoting the validity of reporting research findings.17 Most respondents thought that they and their local IRBs would similarly benefit from a national consensus statement regarding the IRB’s role in medical education research (90% and 88%, respectively), regardless of their personal experience in submitting a medical education research protocol, familiarity with IRB policies, and/or satisfaction with IRB processes.
Our study has several strengths. First, to our knowledge, this is the largest report of clinician educators’ experiences with submitting medical education research protocols to their IRBs. Our response rate of 76% makes response bias less likely. Respondents to this survey had demographic characteristics similar to those reported for respondents to prior CDIM institutional member surveys.18 Second, the clinician educators involved in this study were from 83 different private and public U.S. and Canadian medical schools, lending generalizability to the findings. Third, our finding that most of the clinician educators with experience in multiinstitutional IRB review agreed that there were notable differences in the expectations of various institutional IRBs builds on the findings by Dyrbye et al,8 who reported on six clinician educators’ experiences with IRB committees’ responses to a single, uniform educational research protocol involving medical students.
Our study also has several limitations. First, we surveyed only CDIM institutional members; other clinician educators may have different experiences with their IRBs. The CDIM institutional members, however, represent a diverse group of clinician educators, most of whom are on a clinician educator promotion track requiring publications. Their experiences in submitting medical education research protocols to their IRBs are likely similar to those of other clinician educators trying to navigate the system designed primarily for biomedical research. (Even though almost all—94%—of the respondents’ IRBs deemed their proposals exempt or of minimal risk, this study’s findings, our previous research,8 and anecdotal evidence suggest that the navigation process, no matter what the outcome, can sometimes be more difficult—sometimes much more difficult—because of the biomedical-research bias of the system.)
Second, we surveyed a relatively senior group of clinician educators (mean 7.2 ± 5.4 years in the job). We suspect that the needs identified may actually be greater among clinician educators with less job experience.
Third, we did not ask whether IRB training had been received. As a proxy, we asked whether a medical education research protocol had been submitted to an IRB. Although we presume that most IRBs require RCR training before submission of a protocol, we do not know whether RCR training was required by all IRBs before protocol submission.
Last, the survey did not include any questions to allow for respondents to describe their IRBs. Future research may explore the nature of the relationship between IRB-related characteristics and the IRB’s conduciveness to medical education research.
Our study suggests that clinician educators perceive a need for a national consensus statement regarding the IRB’s role in medical education research, and that such a consensus statement could benefit both clinician educators and IRBs. An outgrowth of such a consensus statement could include courses in RCR tailored to medical education researchers. Such education might empower more clinician educators to engage in medical education research. Another outgrowth of a consensus statement could be a modification of the IRB submission and review process with special considerations for multiinstitutional research, as has been suggested by others.8 As clinician educators face multiple hurdles to conducting medical education research (lack of funding,19 IRB review,8 workload, etc.), we hope this report fuels the discussion of how best to facilitate medical education research while safeguarding students.