Antes, Alison L. MS; Wang, Xiaoqian MA; Mumford, Michael D. PhD; Brown, Ryan P. PhD; Connelly, Shane PhD; Devenport, Lynn D. PhD
Responsible conduct of research (RCR) education, the label commonly applied to research ethics instruction for scientists, is receiving increasing attention by the scientific community as a potential remedy for research misconduct. This attention is evident in the growing number of resources available from organizations such as the Responsible Conduct of Research Education Committee (RCREC)1 and the Office of Research Integrity (ORI),2 as well as in mandates for RCR instruction from funding agencies, including the National Institutes of Health and National Science Foundation.3,4 Another sign of the increasing focus on RCR instruction is the convening of university administrators and RCR instructors and researchers at the first Biennial ORI Conference on RCR Education, Instruction, and Training held in April 2008 and sponsored by the ORI and Washington University.5 One of the most common concerns expressed at this conference was the reality that although RCR instruction is being adopted to manage the integrity of scientific work, little evidence is available to demonstrate the effectiveness of such instruction. Needless to say, evaluation evidence is critical for determining whether these training efforts are indeed fruitful and for guiding future directions in instructional development. Therefore, the purpose of this study was to provide some initial evidence bearing on the effectiveness of existing RCR courses.
Not only is evaluation evidence limited, but when evaluation studies are conducted, outcome measures vary by study, making it difficult to compare conclusions about effectiveness across studies.6 Recent meta-analyses of scientific and business ethics courses concluded that ethics instruction as currently conducted is minimally effective.6,7 These meta-analyses, however, suffer from the limitation that instructional effects were aggregated across a wide variety of outcome measures. The present study examines the effects of existing RCR instruction using a single index across courses.
Evaluation efforts must first specify the intended outcomes of instruction. Without clear delineation of the desired outcomes, measurement and interpretations of the outcomes regarding course effectiveness are not possible.8 A number of outcomes might be suggested: (1) knowledge, particularly of RCR principles and guidelines, (2) skills, such as solving problems and making ethical decisions, and (3) attitudes, such as believing that research ethics are important and that researchers have a responsibility to behave ethically.9 Ultimately, however, the goal of ethics instruction is to change the behavior of those trained.
Because of the distal nature of behavioral outcomes and the difficulty of measuring real-world behaviors, behavioral outcomes are rarely assessed as training outcomes.10 In fact, one commonly applied assessment approach is to ask participants to rate their liking of, or satisfaction with, the course. Although this assessment approach is relatively easy and provides useful information about trainee reactions, it does not directly access actual learning. Thus, we proposed measuring ethical decision making, because this is a proximal outcome that will likely influence real-world behavior. This assessment approach captures whether and how trainees apply the learned knowledge and skills needed to make decisions in response to ethical problems.11 Hence, the current study examined several components of ethical decision making, including the ethicality of decisions made in four domains of research and two types of underlying social–cognitive processes driving ethical decision making, specifically meta-cognitive reasoning strategies and social–behavioral responses.
Prior research has identified four key behavioral domains of research ethics: data management, study conduct, professional practices, and business practices.12 Data management pertains to handling, storing, sharing, and reporting data. Study conduct relates to the treatment of human and animal subjects, adherence to institutional review board guidelines, and maintenance of confidentiality and anonymity. Professional practices concern adherence to professional commitments, mentoring, and treatment of collaborators. Finally, business practices pertain to contract and grant bidding, the use of physical resources, conflicts of interest, and laboratory management. Because the effectiveness of RCR instruction might vary across these domains, it is important to consider the multidimensionality of ethical decision making in research and to assess the effectiveness of instruction across these domains.
In addition, examining social–cognitive process variables underlying ethical decision making can provide additional insights into the effectiveness of RCR instruction and, ultimately, provide more specific feedback regarding the impact of instruction. In the present study, we assessed seven meta-cognitive reasoning strategies known to be critical to ethical decision making (see Table 1).11,13–15 Meta-cognitive reasoning strategies are the mental processes engaged in by an individual to actively think about the situation and work through the ethical problem.13,14 Use of these strategies involves focused analysis of the problem, reflection on and consideration of individuals involved, and anticipation of possible immediate and long-term consequences. Specifically, one might question one's own perceptions and motivations in a situation or anticipate the likely impact of one's decisions on oneself and others involved. It is particularly important to consider the impact of instruction on these meta-cognitive reasoning strategies because they are transferable training outcomes that can be applied across situations and ethical problems.
In addition to influencing how people think through ethical problems, RCR instruction might affect how individuals approach the social dimensions of ethical problems, such as whether an individual accepts personal responsibility for his or her actions or whether an individual is honest with others. Thus, we examined seven social–behavioral responses that are common to ethical problems (see Table 1). The importance of considering social dimensions becomes particularly clear when one considers the social nature of scientific research, which requires interaction with many other people, from collaborators to research participants.14 In fact, many ethical guidelines emphasize the social dimensions of scientific work. Overall, the intent of this study was to use a reliable and valid measure across existing RCR courses to assess the effectiveness of RCR instruction in terms of critical learning outcomes, including the ability to reason through ethical problems, navigate the social dimensions of ethical problems, and ultimately make ethical decisions.
Courses and participants
The University of Oklahoma's institutional review board reviewed and approved this study. Participation in this study was open to any RCR course being conducted at any institution in the United States. Participant recruitment began in the fall of 2005 and continued concurrently with data collection throughout the spring of 2007. Recruitment announcements stated that a nationwide, online study was being conducted to examine the impact of RCR instruction on the ethical decision making of scientists. Instructors of RCR courses at all levels (e.g., graduate student and faculty) in the biological, health, or social sciences were encouraged to contact the research team via a specified e-mail address to acquire additional information and enroll in the study.
The study recruitment effort yielded over 200 contacts with faculty at 21 institutions across the nation. About 40 (20%) instructors agreed to enroll in the study, and ultimately 21 of these courses yielded data from course participants, for a total of 173 RCR students. With the exception of four courses at private universities, the RCR courses took place at public universities. According to the Carnegie Foundation classifications,16 with the exception of one small medical school and one medium four-year research university, the universities were large four-year research universities. Additionally, these research universities were designated as institutions with high or very high research activity. In terms of location, the 21 responding institutions were distributed across regions of the United States, including four on the West Coast, six on the East Coast, and several from the Midwest (n = 3), South (n = 4), and Southwest (n = 4).
Twelve instructors from the 21 courses returned the background data questionnaire. On average, the instructors were 57 (SD = 7.1) years of age, and the majority of instructors reported having PhD degrees, with the remaining having MD, DVM, or EdD degrees. Most instructors reported that their current field of study was biology or medicine, and all instructors, with the exception of two, reported teaching between 2 and 15 research ethics courses in the past. All instructors but one reported being awarded at least one contract or grant during their career and attending at least one conference within the last two years. With the exception of one instructor, all reported authoring at least one publication, with the majority authoring between 14 and 21 publications. Thus, the instructors were active mid- to senior-level faculty.
Instructors from all courses completed a 68-item course content questionnaire, surveying the key characteristics of their courses (e.g., the target audience and instructional approach). This survey revealed that the courses were typically semester-long, required courses aimed primarily at graduate students in biomedical fields. Instructors reported that their courses were moderately time-intensive, and all reported covering the nine RCR topics (e.g., human subjects and publication practices) recommended by the ORI.2 Instructors reported using moderately complex cases and both individual and group activities. They also reported discussing, to a moderate extent, approaches for solving ethical problems or making ethical decisions. Any differences in scores on these scales were not statistically significant (P > .05). Thus, the courses were quite similar in their content and delivery.
The course participants consisted of 173 individuals, with 131 participants identifying themselves as graduate students pursuing PhD or MD degrees. The remaining participants identifying themselves as junior-level (n = 10), midlevel (n = 1), or senior-level professionals (n = 6), and the remaining 25 participants, did not report. This composition of course participants is typical for RCR courses because they generally focus on PhD-level graduate students. The course participant background information also suggested that the average course participant was reasonably representative of a typical RCR course participant at U.S. research universities. Specifically, the average course participant was 27 years old (SD = 4.7), and the majority were female (58%), with 35% being male (7% were unreported). Sixty percent of participants identified themselves as Caucasian, 14% as Asian, 5% as Hispanic, 4% as African American, and 8% as Middle Eastern, Native American, or other (9% did not report). The majority of students reported studying biological science (84%), and the remaining reported studying health (11%) and social science (5%). Thus, our sample represented biomedical course participants fairly well, but it was limited in terms of social scientists. This participant composition reflects the emphasis on RCR instruction in the biomedical sciences and the more limited emphasis in the social sciences. Seventy-one percent of participants reported being required to complete the course as a degree requirement, and 15% reported that it was an elective (the remaining were unreported).
We distributed the recruitment message via four primary outlets. These outlets included (1) posting the announcement on the ORI's Web site, (2) contacting groups involved in ethics or RCR education (e.g., the Poynter Center and the RCREC) and asking them to post the message online and/or to provide the message to their list of e-mail contacts, (3) contacting coordinators of major conferences, workshops, or other events focusing on RCR instruction and asking them to provide the announcement as a flyer to conference participants, and (4) contacting the compliance officers and/or the vice presidents for research at institutions classified by the Carnegie Foundation16 as doctoral research universities-extensive and medical schools (n = 205) and asking them to forward the announcement to individuals involved in RCR instruction at the institution, or to provide instructor contact information directly to the research team.
Upon contacting the research team, instructors were asked to complete and return electronically an informed consent form, a background form, and a survey about the content of their course. Next, we provided instructors with a unique instructor identification code that would allow their students access to the online measures. We also provided instructors with a series of messages that they could either e-mail their students or pass out in class to introduce the purpose of the study and note that they volunteered to make the study available to their students. This information emphasized that student participation was voluntary, that responses would be anonymous, and that the instructor would not be aware of who had or had not participated. Students who chose to participate were then able to read the procedures for accessing the online Web site.
Upon accessing the online Web site, students created a unique username and password, which allowed linkage of pre- and postcourse measures but did not allow identification of specific students by the researcher or the instructor of the course. Upon accessing the online Web site, participants indicated their field (social, health, or biological), so that the measure applicable to their scientific field would be provided to them. The participant instructions indicated that participants should log on to the Web site within the first two weeks of the course to complete the precourse materials. These materials were (1) an informed consent form, (2) a background form, and (3) the precourse ethical decision-making measure. After the conclusion of their course, the participant instructions indicated that they should log on to the Web site within two weeks to complete the postcourse materials, which included the postcourse ethical decision-making measure and a debriefing document. Several short e-mails were sent to course instructors at the start and conclusion of their course asking them to provide a reminder message to course participants.
The outcome measure (Chart 1) used to evaluate the effectiveness of RCR courses was the ethical decision-making measure developed and validated by Mumford and colleagues.11,15,17 The measure assesses ethical decision making in relation to four major dimensions of ethical behavior—specifically, data management, study conduct, professional practices, and business practices.12
This ethical decision-making measure consists of 12 research scenarios for each of the social, health, and biological sciences, so that participants receive scenarios relevant to their respective fields. Each of these scenarios provides the context for three ethical problems that follow from the original scenario and map to the four domains of ethical conduct. Thus, the measure consists of 36 ethical problems, with 18 constituting the premeasure and 18 constituting the postmeasure. For each ethical problem, participants are provided with eight potential responses that reflect low (1), moderate (2), or high (3) levels of ethicality, as determined by field experts using field-relevant norms and guidelines. Participants were instructed to choose two out of the eight options that they thought were most appropriate for addressing the ethical problem. Decision ethicality scores were obtained by averaging point values associated with participants' two chosen response options and then aggregating the responses selected for each of the questions subsumed under the four dimensions of ethical behavior.
In addition to assessing the ethicality of decisions made in response to ethical problems, the measure provides scores for the seven meta-cognitive reasoning strategies and seven social–behavioral responses underlying the choices endorsed by each participant. This scoring of the measure was developed by three expert judges who evaluated the response options according to the extent to which each response option reflected the application of the meta-cognitive reasoning strategies and the endorsement of the social–behavioral responses using a seven-point scale (0 = to no extent, 6 = to a great extent). Participants' scores on the meta-cognitive reasoning strategies and social–behavioral dimensions were obtained by averaging the scores from the two responses endorsed per problem and then aggregating across all problems.
The course content data were analyzed by conducting analyses of variance in which course was treated as the independent variable and course characteristic scores obtained in the course content survey were treated as dependent variables. The course content analysis revealed no significant differences in course characteristics across courses. Thus, for the subsequent analyses, the data from all course participants were treated as one group.
Next, to examine the effects of RCR instruction, we tested for pre–post changes in decision ethicality scores for the four types of decisions, the seven meta-cognitive reasoning strategies, and the seven social–behavioral responses using dependent-sample t tests comparing pretest means with posttest means on all dependent variables. For this analysis, data were used from those participants who completed both the pre and the post measure (n = 53). Before doing so, however, participants who only completed the pretest (n = 86) or the posttest (n = 34) were compared with those who completed both, to determine whether any systematic differences might exist between these groups. This analysis revealed no systematic differences between the groups in terms of the participants, instructors, course characteristics, or main dependent variables of interest. Therefore, the pre–post group is a reasonable representation of the larger group of participants.
Finally, to examine the magnitude of the effects of instruction, the Cohen d statistic for repeated measures was computed. This effect size estimate takes into account the standardized difference between the pre–post means. Cohen d values up to 0.20 are considered small, values up to 0.50 are considered moderate, and values up 0.80 are considered large.18
First, the analysis revealed no significant changes in the ethicality of data management, t(51) = −1.26, P = .21, study conduct, t(51) = 0.31, P = .76, or professional practices decisions, t(52) = 0.84, P = .41. The ethicality of business practices decisions, however, decreased significantly after RCR instruction, t(52) = −2.04, P < .05. Table 2 provides the means, standard deviations, and effect sizes for the ethicality of decisions.
Next, we examined changes in meta-cognitive reasoning strategies (see Table 3). The analysis revealed significant increases in four strategies, including recognizing circumstances, t(52) = 4.61, P < .01, questioning one's judgment, t(52) = 5.90, P < .01, managing emotions, t(52) = 2.14, P < .05, and analyzing personal motivation, t(52) = 3.54, P < .01. In contrast, the findings with respect to seeking help, t(52) = −9.33, P < .01, revealed statistically significant decreases after instruction, and anticipating consequences, t(52) = −1.53, P = .13, and considering others, t(52) = −1.66, P = .10, decreased as well but did not reach statistical significance at the P < .05 level. Thus, mixed findings emerged with respect to meta-cognitive strategies.
Finally, Table 4 presents the results for changes in the social–behavioral responses. After training, participants endorsed responses to ethical problems that involved retaliation, t(52) = 7.79, P < .01, deception, t(52) = 8.20, P < .01, avoiding responsibility, t(52) = 4.45, P < .01, and closing off the possibility for future decisions or actions, t(52) = 3.73, P < .01. In terms of responses characterized by active engagement, participants' scores decreased after instruction, t(52) = −1.88, P = .07, but this change did not reach statistical significance at the P < .05 level. Changes for involving others in the decision-making process, t(52) = 1.09, P = .28, and selfishness, t(52) = 0.61, P = .55, were minimal and not statistically significant.
This study revealed mixed effects of RCR instruction on ethical decision making. The ethicality of decisions made with respect to data management, study conduct, and professional practices did not improve or decline after instruction. However, the ethicality of decisions pertaining to business practices decreased. Changes in meta-cognitive reasoning strategies indicated that instruction encouraged the use of certain helpful strategies, such as recognizing the elements of the situation and analyzing personal motivations, as individuals think through ethical problems. However, the use of other critical strategies, such as seeking help, diminished after RCR instruction. Moreover, the negative effects observed for the social–behavioral responses, which indicated that course participants were more deceptive, retaliatory, closed, and neglectful of personal responsibility in their responses after instruction, countered the positive effects on the meta-cognitive reasoning strategies. Overall, the pattern of effects indicated that, although course participants considered and analyzed situational elements more skillfully, with regard to the social elements of ethical problems, they were more internally focused and closed off when making decisions. Given these countervailing effects for underlying ethical decision-making processes, the lack of improvement, and even decrement, in the ethicality of decisions is not surprising.
Overall, these findings point toward an unexpected conclusion—that a real need exists to reassess a common assumption about RCR instruction. Specifically, we typically take for granted the benefit of RCR instruction. That is, we assume that training in ethics leads to increases in ethical decision making or other integrity-related outcomes. Even those who question its efficacy would probably assume at the very least that such training produced no effects (that is to say, it “does no harm”). However, the present data lead to an important question: Is it possible that ethics training could have detrimental effects? The present findings suggest that not only might RCR instruction not be beneficial, but it might actually be a harmful endeavor. Thus, we must consider why RCR instruction may not work and how it could produce negative effects.
First and foremost, RCR instruction might encourage self-protective behavior.19 After RCR instruction, participants' responses reflected inappropriate closure, less consideration of others, and defensive response patterns, including failure to take personal responsibility, deception, and retaliation. Ethics instruction may induce this internally focused, socially closed-off response to ethical problems if course participants internalize one or both of two messages. First, ethical situations cause serious trouble and ruin people's careers, and second, other researchers are unethical and, thus, untrustworthy.14 Instruction emphasizing the nature of severe forms of misconduct (i.e., fabrication, falsification, and plagiarism) and the serious consequences associated with such misconduct is especially likely to exacerbate this focus on protecting oneself from ethical situations and others who might engage in misconduct.
In addition to inducing self-protection, RCR instruction might be ineffective, or even detrimental, if it plays into people's biases about themselves. People believe that they are better than the average person both cognitively and socially, and such a self-enhancement bias about oneself, especially with respect to ethical behavior, could make it difficult for course participants to view ethics instruction as relevant to them.20 Indeed, instructional effectiveness hinges on student receptiveness and engagement, which is tied to how self-relevant students perceive the training.21 Of note here are research findings showing high-performing scientists to be highly self-confident, dominant, and even arrogant, suggesting that they may be especially susceptible to self-enhancement biases and thus likely to view ethics training as irrelevant to them.22
Not only might natural self-enhancement tendencies render instruction ineffective, but ethics instruction might actually encourage self-enhancement. First, overconfidence could result from completing an ethics course if an individual subsequently believes that he or she knows how to handle all ethical issues and is thus invulnerable to unethical behavior. Such a mindset could promote disengagement in ethical problems and careless decision making. This explanation seems particularly plausible because self-enhancement biases operate especially when people make predictions about their likely future behavior.23 Thus, when thinking about how one might behave in dealing with a future ethical problem, people assume they will behave ethically. In fact, participation in an RCR course may encourage people to affirm their ethical virtues, which may actually allow for misbehavior later because this affirmation promotes unconscious rationalization of one's questionable behavior.24 This phenomenon is particularly likely if instruction overemphasizes extreme case examples where, by comparison, an RCR participant could only view him- or herself as highly ethical, thus enhancing the perception that he or she would never engage in such extremely unethical behavior.
Finally, students' attitudes toward their RCR courses present another plausible explanation for detrimental effects of RCR instruction.25 For instance, an attitude regarding ethics instruction as just another “pesky requirement” needed to obtain a degree or to engage in research might block the effectiveness of instruction. In fact, students might view just sitting through the course week-after-week as a hindrance to their careers, as they could otherwise spend that time on research endeavors. Clearly, future research must seek to examine the explanations for ineffective or harmful instruction proposed here, in addition to other potential risks of RCR instruction and how these obstacles might be overcome.
In future research examining where ethics instruction might go wrong, it is critical to consider that ethics instruction is fundamentally different from instruction in traditional academic courses. First, ethics instruction is not as fact based as standard courses. Instead, in addition to basic knowledge of rules and principles, it is important to learn strategy and process.15,26 Thus, ethics courses must teach participants how to think through ethical problems. Research examining how instruction might accomplish this objective is in high demand. The social nature of ethics instruction proves another unique characteristic in that the ethical behavior of scientists has direct implications for their field and even for society. Moreover, RCR instruction itself is a highly social endeavor, and this social dimension to instruction can produce powerful dynamics (e.g., social comparison processes, conformity) that can either enhance or undermine the goals of ethics instruction. Thus, research examining the unique social elements of ethics instruction is an area ripe for future investigation.
Future research should also focus on developing models for evaluating the effectiveness of RCR instruction. The unique nature of RCR instruction raises important questions about how to best assess its effectiveness. Commonly, assessment of RCR course effectiveness centers on knowledge of ethical principles and rules and/or student reactions to the course, typically liking of the course or subjective ratings of its effectiveness. Although knowledge and perceptions of the course are important, they do not address whether people can apply knowledge in a context. Thus, ethical decision-making measures which require people to make decisions about complex, realistic ethical problems offer one way to require course participants to apply knowledge in a more realistic, performance-oriented manner.11 Other assessment tools requiring realistic performance, such as problem solving or handling social conflict, and capturing the complex, ambiguous nature of real-world ethical problems, might be developed, and research along these lines is greatly needed to further the field of RCR instruction. These assessment tools should take into account the multidimensional nature of research ethics and consider the importance of social–cognitive processes in ethical decision making and behavior.
Before concluding, we should note the limitations of the current study. One primary limitation was the lack of control over the test-taking environment, given the nature of online studies. In addition, missing data led to a limited sample size for the pre–post comparison. Moreover, the voluntary nature of the study (at both the instructor and student levels) allows for the possibility that those who chose to participate differ from those who did not choose to participate. Fortunately, courses included in the study came from universities across the United States, and the available course participant information suggested that they were reasonably representative of RCR course participants in the United States. Nevertheless, the voluntary nature of the study leaves open the possibility that this sample was not representative of the broader population of RCR courses or course participants. As a result, the generalizability of these findings to more senior-level trainees and to trainees in fields beyond biomedical is limited by the nature of the present sample.
Furthermore, we did not assess individual difference and situational variables likely to impact responding on the ethical decision-making measure. A common issue in studies of educational interventions is the lack of control over external factors that might impact performance on the outcome measures. For instance, poor mentoring or observation of misconduct could influence outcomes associated with training. Finally, this study examined instructional effectiveness using only one measure. Although the measure is valid, and consistent dependent variables are desirable for comparing different RCR courses in the same effort, other outcomes could be examined, and additional ethics measures should be developed and included in future examinations of RCR course effectiveness.
In conclusion, our exchanges with RCR instructors revealed a great desire for information about what constitutes effective ethics instruction and considerable concern that RCR instruction may not be working. This study suggests that not only might RCR instruction be ineffective as currently conducted, but it could be harmful. Thus, the findings presented here ultimately raise more questions than they address, but overall it should be noted that the social–cognitive mechanisms underlying ethical decision making and ethics instruction are critical areas needing further investigation. Although we typically seek to understand what promotes effective instruction, it may be just as important to ask what might hinder effective instruction. Ultimately, the current study offers us a line of research that may facilitate our understanding of research ethics and ethical decision making and how we might construct effective RCR instruction.
The authors would like to thank Drs. Stephen Murphy and Ethan Waples for their contributions during the early stages of this project. Caleb Murphy is also thanked for his technical assistance in designing the online data collection Web site.
This research was supported by grant #5R01-NS049535-02 from the National Institutes of Health, Michael D. Mumford, Principal Investigator.
This research was approved by the University of Oklahoma Institutional Review Board.
An early version of this paper was presented at the 2008 First Biennial ORI Conference on Responsible Conduct of Research (RCR) Education, Instruction, and Training, St. Louis, Missouri.