Anderson, Melissa S. PhD; Horn, Aaron S. MA; Risbey, Kelly R. MEd; Ronning, Emily A. MA; De Vries, Raymond PhD; Martinson, Brian C. PhD
Training and mentoring in the responsible conduct of research (RCR) are widely promoted as the best available means of ensuring the integrity of science. Federal mandates requiring instruction in research integrity for all federally funded research trainees, and the educational initiatives of the U.S. Office of Research Integrity all support the assumption that RCR instruction—and, to a lesser extent, mentoring—are critical to the promotion of ethical behavior and the prevention of misconduct. Instances of behaviors that fall under the current federal definition of misconduct are rare,1 which might be taken as assurance that RCR training and mentoring work.
The responsible conduct of research, however, involves far more than avoidance of fabrication, falsification, and plagiarism. It requires practices that ensure the trustworthiness of research findings, the propriety of the methods used, and the legality and ethicality of the systems that support the research. Our concern here is with behaviors that may compromise the integrity of science, in addition to those that fall under the specific rubric of misconduct. We refer to these behaviors as problematic or questionable.
Our study is, to date, the only national, cross-disciplinary analysis of training in research integrity and mentoring in relation to researchers’ actual problematic behaviors. It is a first attempt to respond to the need for evidence that training and mentoring actually do what they are intended to do; as Heitman et al2 have noted, “formal RCR education can improve trainees’ knowledge of standards, but establishing its effects on their behavior remains the Holy Grail.” Our findings are based on data from a major national survey of early- and midcareer biomedical and social science researchers in the United States. Unlike other assessments of training in research integrity, this study examines actual—rather than prospective or hypothetical—behavior as reported by the respondents. Also, the two samples of researchers provide a unique opportunity to compare associations between research-integrity training and problematic behavior among scientists who received their doctoral degrees before and after the advent of federally mandated RCR training. The midcareer group completed their doctoral degrees, on average, one year before the NIH issued its special RCR training grant requirement in 1989, and well before the mandated training became widely available. In contrast, the early-career group, who were all NIH trainees, received their doctoral degrees an average of eight years after the requirement was issued.
We conducted a national survey of early- and midcareer biomedical and social science researchers in the United States in 2002. The study was approved by the institutional review boards of the University of Minnesota and HealthPartners Research Foundation. Respondents were assured anonymity. We drew samples from two databases maintained by the National Institutes of Health (NIH): one of all postdoctoral fellows supported by NIH and the other of all the researchers who received NIH R01 awards (basic research funding). We first identified those who were supported on an NIH postdoctoral fellowship (T32 institutional or F32 individual) during 2000 or 2001 (early career), and those who received their initial research grant (R01) in the period of 1999 to 2001 (midcareer). We then randomly drew the early- and midcareer samples, of sizes 4,160 and 3,600, respectively, from these two populations for a total of 7,760 subjects. Mailed surveys were administered in the autumn of 2002, with follow-up mailings to initial nonresponders.
The survey measured training in the ethical aspects of research through items that asked about researchers’ instruction on ethical issues, the responsible conduct of research, and related matters during their graduate education or subsequent postgraduate career. We created four classifications of respondents: those who received research-integrity training only through separate coursework on ethical issues, those whose training in ethics was completely integrated with their other coursework, those who received both separate and integrated training, and those who had neither. It is important to note that we removed from the analysis the respondents who did not answer the survey questions on training; those whom we classified as not having received training actually told us so on the survey.
We measured mentoring with survey questions about the kinds of help that the respondents received from mentors in their graduate education, postdoctoral work, or related professional experience. The three response categories were none (0), some (1), and a lot (2). We constructed each of four mentoring variables as the sum of responses to two of the mentoring items. Research mentoring related to good research practice and writing. Financial mentoring had to do with grant and contract proposals and obtaining other financial support. Survival mentoring involved guidance on how to build relationships and survive in the profession, and personal mentoring was reflected in ongoing interest and emotional support. The fifth mentoring variable, ethics mentoring, was constructed in the same way from two items in the training battery that measured informal mentoring on ethical issues apart from coursework.
To measure behaviors that may compromise the integrity of science, we derived a list of 33 problematic behaviors from six focus-group discussions that we conducted with 51 biomedical and social science researchers.3 Our use of the terms problematic and questionable to describe these behaviors stems from our focus-group participants’ concern about the damage that such behaviors could do to the integrity of science. Of the 33 behaviors, 27 were included in this analysis; the others fell outside the scope of training or mentoring in research ethics. We asked survey respondents to indicate whether they had or had not engaged in these specific behaviors within the previous three years. The items were grouped into eight conceptually related categories: data, methods, policy, use of funds, outside influence, peer review, credit, and cutting corners. Our analytical results are based on logistic regressions, with the odds of engaging in problematic behavior serving as the dependent variable.
Our analyses included several control variables: gender, type of highest degree (PhD versus other), and location of institution that awarded the highest degree (inside versus outside the United States). The degree variables controlled for possible differences in training among those who have a professional degree (usually the MD) but no PhD and those whose doctoral training took place outside the United States. We also included academic discipline in broad categories (medicine, biology, chemistry, physics/mathematics/engineering, social science, and miscellaneous, with medicine serving as the reference group).
We performed all analyses with the Statistical Package for the Social Sciences, version 14.0. We present all analyses separately for early- and midcareer respondents.
We received 1,479 useable surveys from the early-career sample and 1,768 from the midcareer sample. After adjustment for the number of surveys returned as undeliverable (876), the survey response rates were 43% for the early-career sample and 52% for the midcareer sample. The early-career respondent group included 787 (53%) women, compared with the midcareer group with 580 (34%). Those whose highest degree was other than the PhD accounted for 19% of each respondent set (275 in the early-career group and 340 in the midcareer group). Those who received their highest degree outside the United States represented a larger proportion of the midcareer group (358, or 21%) than of the early-career group (115, or 8%).
Table 1 shows the numbers and percentages of early- and midcareer respondents who reported having received research-ethics instruction and mentoring in various forms. Three quarters of the early-career respondents and half of the midcareer respondents reported having coursework that focused specifically on ethical issues. Fewer than two thirds of the early-career researchers and fewer than half of the midcareer group reported RCR training integrated with their other coursework. The early-career respondents were thus more likely than midcareer respondents to report having received some training in ethical issues, but we emphasize that a quarter reported having had no separate coursework on ethical issues and that more than a third had no integrated units or material in this area. More than half of the midcareer respondents reported the same. Eliminating overlap among the training categories yielded the following distribution for early-career scientists: a total of 174 (12%) had separate training only, 142 (10%) had integrated training only, 907 (63%) had both, and 216 (15%) had neither. Of the midcareer respondents, 130 (8%) had separate training only, 248 (14%) had integrated training only, 706 (41%) had both, and 633 (37%) had neither.
Respondents’ training in research ethics differed significantly by the characteristics of their degree programs. Ninety-one percent of early-career scientists in medicine and the social sciences reported having received separate or integrated training in research ethics or both, compared with 78% or fewer of those in biology, chemistry, and the combined areas of physics, math, and engineering. In the midcareer group, those in medicine and the social sciences also reported the highest levels of training (71% and 77%, respectively), but only 42% to 51% of the respondents in the other three disciplinary areas reported having received training of any kind in research ethics. Both samples showed significant differences in training between those with and without a PhD degree: those with a PhD were less likely to report having had either separate or integrated training in research ethics (83% in the early-career sample and 60% in the midcareer sample), compared with those who had not earned a PhD (95% and 78%, respectively). Among both early- and midcareer respondents, there were no statistically significant differences in training between those with and without their highest degree from a U.S. institution. (In the paragraph above, the numbers of respondents corresponding to the percentages were not included because of the possibility of misinterpretations. They are, however, available from the corresponding author.)
In general, respondents in both samples were more likely to say that they had received mentoring than instruction in research integrity, particularly in the areas of research practice, writing, developing professional relationships, and having discussions about ethics. Most had a mentor who expressed continuing interest in their progress. In some areas, however, there were gaps in mentoring. More than a quarter of respondents in both samples had not received help in learning the art of survival or emotional support when needed. Likewise, more than a quarter had not received ethics mentoring through workshops, conferences, or roundtable discussions. In addition, nearly 30% of the midcareer respondents said that their mentors had never given them any instruction in writing grant and contract proposals.
Mentoring showed some differences by type of degree program. Early-career researchers in the social sciences, medicine, and the physics/mathematics/engineering group were more likely than their counterparts in chemistry and biology to have received mentoring in ethics. Those in biology and physics/mathematics/engineering were the most likely to have received financial mentoring, and those in the social sciences and physics/mathematics/ engineering had the highest levels of survival mentoring. Among midcareer researchers, those in the social sciences were most likely to report having received ethics mentoring, followed by those in biology and medicine. Those in the social sciences and biology were most likely to have had financial mentoring. There was only one difference in mentoring by degree: midcareer respondents with no PhD were more likely to have received financial mentoring than those with a PhD. Having attained one’s highest degree in the United States was associated with significantly higher levels of mentoring in five cases: ethics mentoring among midcareer respondents, research mentoring among early-career respondents, financial mentoring among respondents in both samples, and personal mentoring among the midcareer researchers. (In the paragraph above, the numbers of each category of respondents were not included because of the possibility of misinterpretations. They are, however, available from the corresponding author.)
Table 2 presents the numbers and percentages of respondents in each sample who indicated that they had engaged in one or more of the potentially problematic behaviors in a given category within the previous three years. The purpose of Table 2 is not to show the prevalence of individual behaviors, but rather to list the components and indicate the ranges of the variables used in the logistic regression analyses below. Most of the item-specific response distributions have appeared elsewhere.1 The percentages reported here are dependent, of course, on the number and nature of items included in each category, and so we caution against any particular emphasis on these numbers.
Tables 3 and 4, corresponding to the early- and midcareer samples, respectively, present the logistic regression parameter estimates of the effects of training and mentoring on the odds of reported problematic behavior in each of eight categories. A parameter estimate greater than one indicates increased odds of having engaged in the given behavior, whereas parameter estimates less than one indicate lowered odds. The omnibus χ2 goodness-of-fit tests of the logistic models were all significant, except the early-career model of behavior in the area of outside influence, which was marginally significant at the P < .05 level.
Overall, training in research ethics showed little relationship with problematic behavior. Mentoring showed both positive and negative associations with problematic behavior in the early-career sample, but virtually none in the midcareer sample.
Among early-career researchers, training in research ethics through separate or integrated coursework (or both) showed no significant effects on problematic behavior, with one exception: those with separate training or with both separate and integrated training were more likely to report having engaged in problematic behaviors in the data category. Ethics mentoring, research mentoring and personal mentoring each lowered the odds of a researcher having engaged in problematic behaviors in certain categories. Ethics mentoring lowered the odds of questionable behavior in the areas of methods and cutting corners. Research mentoring was inversely related to problematic behavior in the areas of data, methods, use of funds, and cutting corners, whereas personal mentoring lowered the odds of questionable behavior in the categories of methods, outside influence, and peer review. By contrast, financial and survival mentoring were associated with increased odds of reported problematic use of funds, and survival mentoring had a similar association with behaviors in the categories of methods and peer review.
For the midcareer respondents, neither separate nor integrated coursework alone was associated with questionable behaviors, but combined they lowered the odds of reported problematic behavior in matters of policy, use of funds, and cutting corners. Mentoring was not associated with questionable behavior among the midcareer group, except that ethics mentoring lowered the odds of problematic behavior in the policy area.
Women were in general less likely to report engaging in the behaviors noted. Respondents without PhDs in the early-career group showed less problematic behavior in four categories (data, methods, policy, and use of funds), compared with those with PhDs. In the midcareer group, those without PhDs were more likely to report having cut corners. Among the early-career respondents, having received the highest degree from a non-U.S. institution was not associated with questionable behavior, but among the midcareer group it was associated with reduced odds of reported behavior in the categories of use of funds and cutting corners. Compared with the reference group of medicine, those in the social sciences had lower odds of reported problematic behavior in the areas of policy and (in the midcareer group) use of funds, but among the early-career respondents, the social scientists had higher odds of questionable behaviors related to peer review and authorship credit. Respondents in other disciplines differed little in behavior from the reference group of medicine.
Our respondents provided one final, subjective indication of the need for greater attention to training in research ethics. When asked, “How prepared do you feel you are to deal with ethical issues that are likely to arise in your work?,” only 73% (1,063) of the early-career researchers and 75% (1,288) of the midcareer researchers responded well prepared or very well prepared. In effect, then, about a quarter of the respondents (386 in the early-career group, and 443 in the midcareer group) felt less than well prepared to deal with ethical issues in their work.
It would have been reassuring had we found many significant, inverse relationships between respondents’ training in research integrity and behaviors that may compromise the integrity of science. We did not. In fact, we found no such inverse relationships among the early-career group, and among the midcareer group, inverse relationships appeared only among those who had both separate and integrated training in research ethics. Among the early-career respondents, mentoring proved to be related to problematic behaviors in both positive and negative ways, depending on the type of mentoring and type of behavior. These associations between mentoring and questionable behavior were not found among the midcareer researchers.
We found disturbingly high proportions of researchers who reported that they had not received separate or integrated training in research ethics. We reiterate that all respondents whom we categorized as not having received ethics training reported so on the survey. It is possible that some of these respondents did not remember their research-ethics training, but this possibility is not encouraging.
In general, respondents who received training in research integrity differed little in their subsequent reported behavior from those with no training in research integrity, at least in the categories of problematic behavior that we examined. It may be that RCR instruction does not typically address these behaviors, though it would seem that an effective research-integrity program would provide some guidance in these areas. It is also possible that RCR instruction teaches students less about how to avoid doing these things than about how to deal with ambiguities in their work, by showing them, in a nuanced way, under what special circumstances some of these problematic behaviors might be acceptable. Perhaps many of these behaviors are not, in fact, seen as inappropriate by most researchers. In any case, we emphasize that our list of problematic behaviors was based on our focus-group participants’ uneasiness and concern about such behaviors’ potential to compromise the integrity of science.
In our analyses, mentoring had more significant associations with questionable behavior than did training in research integrity among the early-career researchers, although not among their more senior counterparts. Ethics mentoring, research mentoring, and personal mentoring all had inverse relationships with problematic behavior in certain categories. It is important to note that four of our five measures of mentoring (research, financial, survival, and personal) represented general mentoring, not specific to ethical issues. Compared with classroom instruction, mentoring is typically highly personal, context specific, experiential, and relevant to the work at hand. Mentors frequently act as role models, in that their observed behavior is instructive. From such strong researcher–mentor connections come strong messages about how to conduct research.
Messages from mentors are equally strong, of course, when they have to do with making it in the competitive world of science. When early-career researchers need guidance for their professional survival, their mentors’ advice and modeled behavior may suggest strategic decisions that do not match textbook descriptions of proper behavior in research. In the early-career group, survival mentoring was associated with increased odds of questionable behavior in methods, use of funds, and peer review, and financial mentoring had the same association with use of funds. Competition has always been present in science, but increasingly competitive pressures may stem from entrepreneurial initiatives or corporate involvement, which are associated with greater competition and secrecy among scientists.4 As federal funding has also become more competitive, the dynamics of funding may lead scientists to be strategic in ways that contradict traditional normative cultures in science.5 Indeed, some of our focus-group respondents proved themselves to be quite Machiavellian in their drive to compete successfully in science.3
We note that our analyses revealed only associations, not causal links, between instruction or mentoring and problematic behavior. It is possible, for example, that some of our respondents’ questionable behaviors or misconduct led to sanctions that included mandatory instruction in the responsible conduct of research. The different time frames associated with our survey questions about instruction and mentoring (during graduate education or subsequent career) and problematic behavior (within the previous three years) were designed to lessen the influence of this possibility, but it cannot be entirely discounted.
It will come as no surprise to our focus-group participants that our survey respondents reported having engaged in problematic behaviors. Indeed, it was the presence of these behaviors that troubled our discussants. The discussants expressed little concern about outright misconduct (fabrication, falsification, and plagiarism) because they saw these as rare, as our survey results confirmed.1 They saw a greater threat to the integrity of science in what we have elsewhere termed normal misbehavior.3 Such behaviors fall outside current definitions of misconduct, but their potential to compromise the integrity of science demands attention.
As in any profession, a few researchers engage in misconduct, sometimes with dramatic public exposure. Many more admit to a kind of everyday behavior that is less than exemplary, again as in other professions. In science, however, integrity is a matter of critical concern because the public must be able to trust scientific findings that affect their health, environment, economy, industry, and society in general. Researchers also need to be able to trust each others’ work, in order to build further on it. Insofar as much of investigators’ research is federally supported and the purpose of their teaching is to prepare the next generation of scientists, the public’s trust is critical to ongoing support for scientific research.
It is important to note again that the measures of questionable behavior reported above are based on federally funded researchers’ reports of their own behavior within the previous three years. Our findings are therefore subject to the limitations of self-report data; however, in this case, it seems likely that self-report measures underestimate the extent of misbehavior, suggesting that our results are based on conservative estimates.
Our study suggests a need for further investigation of the associations between aspects of training in the responsible conduct of research and subsequent behavior. In particular, attention should focus on differences in content, pedagogy, and mode of delivery of RCR instruction in relation to various categories of problematic behavior.
There is reason to be optimistic about the power of mentoring to affect behavior of early-career researchers, but it would not be appropriate to assign responsibility for ethics training solely to mentors. First, our results have shown that mentors’ influence was sometimes associated with increases in problematic behavior. In effect, some of the influence that mentors exert is undesirable. Second, informal mentoring systems can leave some students without any mentor. Formal mentoring programs that ensure that each student is mentored, with clear expectations for what mentors are to accomplish, might be able to leverage the power of personal connections and context-specific learning to teach students about research ethics. An instructional component of such a program might ensure that students get perspectives from multiple mentors and that students study a wider range of issues than they typically would with only one mentor.
Our findings have called into question the effectiveness of research-ethics training in obviating problematic behavior. We found that mentoring has the potential to influence behaviors in ways that both increase and decrease the likelihood of problematic behavior. Finally, no one with a stake in scientific research should be comfortable knowing that any federally funded scientists—let alone a quarter of our sample—feel less than well prepared to deal with the ethical issues that their work may present.
This research was supported by the Research on Research Integrity Program, an ORI/NIH collaboration, grant #R01-NR08090. Raymond De Vries’ work on this project was also supported by grant #K01-AT000054-01 (NIH, National Center for Complementary and Alternative Medicine).
The authors acknowledge Joseph B. Shultz’s contributions to the preliminary conceptualization of this paper.