Schreier, Alan A. PhD; Wilson, Kenneth PhD; Resnik, David PhD, JD
Good record-keeping is central to the scientific process.1–4 Good research records encompass much more that just research data. They include but are not limited to planning and protocol descriptions, data manipulations and analysis procedures, personal and group interpretations of the results, and important communications and group decisions among collaborators. Data management is a subset of the broader concept of research record-keeping. Research records are important for managing and planning research, for replicating results, for documenting collaborations, for publishing and peer review, and for complying with governmental and institutional rules and regulations. In recent decades, legal and regulatory uses of research records have become prominent. Research records have long been used to resolve intellectual property disputes.2 However, research misconduct scandals in the 1980s and 1990s involving falsification and fabrication of research records provoked the federal government to require universities to implement research misconduct policies.2 Examination of research records is a central feature of inquiries and investigations under such university policies.
We recently conducted a survey of 96 university officials who are responsible for such inquiries and investigations as part of a research project on the impact of research records in research misconduct cases.5 We found that over half of the officials who reported experience with misconduct cases at their institutions also reported that they had been hampered in their inquires/investigations by inadequate research records. Also, another recent survey of investigators who had been funded by the National Institutes of Health (NIH) on the prevalence of questionable research practices noted that 27% of the 3,247 respondents admitted to “inadequate record-keeping related to research projects.”6
Given the importance of good research records, it is somewhat surprising that formal standards for such records are the exception rather than the rule in academic research laboratories. Although governments have mandated standards for good research records for certain segments of the research community—most notably in the area of human health and safety research through the stringent regulations of the U.S. Food and Drug Administration (FDA) and the Environmental Protection Agency (EPA)7,8—the majority of academic researchers are not constrained by any external set of record-keeping guidelines. In fact, most academic scientists find the mandated FDA record-keeping practices both onerous and unnecessary. Academic researchers prefer informal guidelines rather than formal standards for record-keeping.
During the last half of the 20th century, technological changes in how records are produced, collected, analyzed and stored, coupled with social changes in the nature of research groups, have created new challenges for research record-keeping.9 Traditional practices for such record-keeping are either no longer sufficient or, at worse, obsolete for the modern researcher. New record-keeping practices have arisen to meet these challenges; however, very little research has been done on defining in a systematic way the best practices* for research record-keeping in the 21st century academy. The literature is replete with descriptions of record-keeping practices, but they are usually personal assessments. In this article, we will discuss the modern challenges to traditional research record-keeping practices. We will then articulate best practices (principles and specific practices) for keeping research records that have emanated from our research and which, we believe, can help researchers respond to these challenges. Because these principles and practices were developed in a systematic way, we believe that they represent current norms within the scientific community (excluding FDA- or EPA-regulated research). We offer these best practices as ethical guidelines for researchers, research group leaders, and research institutions, not as absolute rules or legal requirements.
Modern Challenges to Traditional Record-Keeping Practices
During the 19th and early 20th centuries, a tradition for research record-keeping developed in the academic physical and biological sciences that focused on the research notebook as the primary tool for organizing research projects, planning experiments, recording data, analyzing the results, and storing these records for future reference. Research groups were small, allowing group leaders to personally train young researchers and to regularly examine primary research records; this process ensured the transmission of the record-keeping tradition to the next generation. During the last half of the 20th century, a number of trends in research have complicated the traditional record-keeping process and interfered with the training of new researchers in record-keeping practices.9 Chief among these trends were (1) the rise of large research groups; (2) the introduction of research instruments and methods that self-recorded data either on paper printouts or photographically (e.g., X-ray films, slides, and photographs); and (3) the rise of computers in the laboratory.
The current environment for research training, where a graduate student or a postdoctoral fellow is usually a part of the large research group, encourages the teaching of record-keeping practices to new researchers by peers rather than by the senior group leader. Oversight of research is often focused only on the results of a young researcher’s project; it appears to be rare for a researcher’s primary research records to be examined by a senior research group leader. If there are no standards imposed by the research group leader, the actual record-keeping practices within the laboratory group may deviate from accepted norms. The practices actually in use will depend on an individual researcher’s own attitude toward record-keeping and on the attitudes and practices of peers. An extreme example of this occurs when foreign students and postdocs keep their research records in their native languages if there is no requirement to keep records in a common language. There are many anecdotal cases of this practice because international students and postdocs are prevalent in academic research groups. Kanare10 describes a case where a research group leader discovered, after his Japanese postdoc had returned to Japan, that the postdoc had written his notebooks in Japanese.
The introduction of self-recording research instruments and new photography-based techniques were a major technical advance for science, but they have had the unfortunate byproduct of reducing the utility of the traditional bound research notebook, and more important, of beginning the fragmentation of the research record that is so common today. The first self-recording instruments in the 1940s and 1950s produced paper-based output (they now produce digital files). Photographic recording techniques also appeared at about this time, such as imaging radiolabels on paper chromatographs. These types of records were difficult to incorporate into traditional bound laboratory notebooks due to the cumbersome need to glue or tape such output into notebooks and the sheer volume of such records. Many research groups abandoned the traditional bound notebook in favor of loose-leaf binders or manila folders for both the collection of data output (instrument and photographic data) and for the planning of experiments. Soon, the processing and analysis of data was also placed into loose-leaf binders or manila folders, thus contributing to the physical fragmentation of research records.
The rise of computers in the laboratory was another major technical advance. They made possible the collection and processing of monumental amounts of data that were critical for the advancement of many fields (e.g., X-ray crystallography, genomics, and proteomics). However, the computer also promoted the fragmentation of the research records in academic laboratories and made oversight more difficult. Researchers routinely use many different computer applications in the course of their work and produce a variety of research records (e.g., word processing files, spreadsheets, instrument data output files, etc.). For a single research project, relevant computer files can be scattered among several computers within a research group, depending on the number of individuals working on the project. Within each computer, the pertinent files may be scattered among apparently unrelated folders on the hard drive. In addition, the rapid obsolescence of computer software and hardware (often only three to five years per generation) has made the long-term electronic retrieval of research records problematic, if not impossible.
Technological Solutions to Modern Record-Keeping Challenges: Different Responses of Academia and Industry
Computer technology has also been called upon to provide solutions to these modern record-keeping challenges. Laboratory information management systems (LIMS) and electronic laboratory notebooks (ELNs) are the two major types of tools that have been developed.11–13 Industry has embraced these systems due, in part, to intellectual property concerns and stringent FDA regulations. LIMS systems in particular have been extremely helpful in managing laboratories that generate a great deal of data and use standardized procedures, which is often the case in industrial and clinical labs and pharmaceutical quality control facilities. However, in academic (discovery-type) research, protocols and methods can change frequently, and the amount of data collected tends to be relatively limited. Because LIMS systems need to be customized for each process they handle, such systems are therefore viewed as being too inflexible for the typical academic laboratory.12 The exceptions here are areas like genomics and proteomics, whose procedures have become highly standardized with a need to process high amounts of data. ELNs supposedly mimic the bound paper notebook, but again due to flexibility issues, anecdotal evidence suggests that such tools are not commonly used in academic laboratories.12
These technological solutions themselves are limited by the basic assumptions used in their design about what constitutes good records and good record-keeping. The best practice principles that we developed and have described below can be used to guide the development of future generations of LIMS and ELNs. We believe that these principles transcend the mechanism for their implementation, whether they be paper- or computer-based.
Genesis of the Best Practices
We developed the best practices described below as an adjunct to our research-on-research-ethics project entitled “Scientific Record Keeping and Responsible Research Conduct” funded by a grant from the National Science Foundation (NSF).5 We carried out the research to determine the actual record-keeping practices of active NSF- and NIH-funded scientists and how these influence the adjudication of research misconduct investigations. To help prepare a national survey of active NSF and NIH scientists on their record-keeping practices and to aid in the analysis of the responses, it was necessary to have a set of generally recognized best practices, that is, normative standards for research record-keeping. We derived these best practices from four sources: (1) the research training and ethics literature; (2) institutional policies and guidelines; (3) interviews with university officials responsible for investigating allegations of scientific misconduct; and (4) focus groups composed of faculty members from both medical and nonmedical faculties of Duke University, East Carolina University, North Carolina State University, and the University of North Carolina at Chapel Hill.
The research training and ethics literature was our primary source of record-keeping practices. The regular research literature (e.g., journal articles, monographs, conference proceedings, etc.) was not excluded from our review; however, this literature usually does not discuss research record-keeping practices. The ethics and research training literature is large and often included detailed and discipline specific recommendations for record-keeping. Many of the references for record-keeping practices were in textbooks or primers for students learning laboratory techniques for the first time. They usually focused on the proper construction of a research notebook and more, recently, its electronic counterparts. The monograph Writing the Laboratory Notebook by Kanare10 is the acknowledged classic in this field. The practices described in this book and the other references are the practices that an individual researcher should follow when he or she is planning, collecting, recording, and analyzing research. We named these specific practices individual best practices. Only recently have references appeared that speak to higher-level best practices, such as those that research group leaders, departments and institutions could emulate to promote good record-keeping. Selected primary documents used in our compilation of our best practices are listed in the references at the end of this article.2–4,9,10,14–25
Institutional policies and guidelines that deal with research records from many research universities were reviewed. Although our review was far from inclusive, we found that such policies and guidelines usually spoke to issues of retention, ownership, access and transfer of research records. Only rarely did a policy or guideline (e.g., those of the University of California, San Francisco)18 set specific standards for record-keeping beyond these administrative issues. Sources that contained representative policies and guidelines that were useful are noted in the reference list.18–25
Our research ethics project included interviews with university officials from 96 research universities.5 These officials provided us with a number of positive practices for good record-keeping based on their experiences. In almost every case, however, these practices reiterate those that appeared in the literature. Thus, the university officials provided anecdotal evidence for the value of these practices.
As the number of specific best practices grew, they became too unwieldy to be a useful guide if compiled into a single list. An organizational structure was needed. We divided the best practices into three hierarchical groups: best practices for individual researchers, best practices for research group leaders, and best practices for departments and institutions. We also developed summaries—essentially statements of principles for the best practices for each group (see Lists 1, 2, and 3)—and compiled the detailed best practices separately. The resulting document, containing both principles and specific practices, was provided to the focus groups for their review and suggested revisions.
The focus groups were particularly helpful in refining the best practices. The focus-group members totaled 48 active senior scientists (12 on each campus), who have NSF or NIH funding in a variety of research fields (from basic sciences like physics and chemistry through clinical fields to social and behavioral sciences). Focus-group sessions of four hours in length were held in the spring and summer of 2004. Interestingly, the members of the groups directed most of their comments toward best practices for research group leaders and departments and institutions, the areas that were least discussed in the literature. They stressed the pivotal role of the research group leader in setting standards and procedures for his or her group. The focus-group members identified some new types of research records that had previously not been mentioned in the literature, such as records of e-mails and teleconference transcripts and minutes. These types of records have greatly increased in importance due to the highly collaborative nature of modern research, where the collaborating scientists and their research groups can be physically quite distant from each other. Records of communications about research results, plans for future research, and decisions regarding results or future plans need to be documented to prevent misunderstandings and miscommunications. Another area of extreme importance to all focus groups was the concern that any set of best practices resulting from our work would be adopted as mandated requirements by federal agencies, resulting in inflexible and onerous regulation of their research.
The Best Practices
Lists 1, 2, and 3 summarize the best practice principles for research record-keeping at all three levels described above. A complete document with detailed best practices is available at 〈http://www.research2.ecu.edu/BestPractices_Sep04.doc〉.
The best practices described above and in the three lists are a synthesis of traditional and modern practices from a variety of sources. They emphasize principles over detailed practices, which are subject to obsolescence and disciplinary restrictions. They also emphasize the critical role of the research group leader in setting standards and in the training of new researchers. In addition, they explicitly acknowledge the role of departments and institutions in setting the organizational context within which the research groups operate. Our focus-group members felt that these two higher-level sets of practice principles were applicable to any research discipline or institution. These principles address the effective management of scientific laboratories and work groups, and are aimed at promoting good communication between current research participants and future participants (through stored records).
It is our opinion that a breakdown in the training and supervision of new researchers has contributed to the decline of research record-keeping. The fragmented and still-evolving nature of modern research records has contributed to this breakdown by making many of the old record-keeping practices obsolete, with no generally accepted replacements in sight. Young scientists in training and new research group leaders may find it difficult to understand the linkages between the specific, detailed processes they follow in their research and the general norms of scientific research. By identifying a separate set of responsibilities assigned to the research group leader, they may be able to more clearly understand how to manage their groups and train their students more effectively.
We present this new synthesis of best practices to the research community as a potential aid in thinking about the challenges of record-keeping in the first decades of the 21st century. We hope that it will guide the development of better practices among research group leaders, as well as encourage departments and institutions to adopt practices and policies that will aid research group leaders in their responsibility to keep good records of their research.
Dr. Schreier is a university administrator charged with coordinating university research compliance activities including the review of allegations of research misconduct. He is primarily responsible for the development of the best practices document described in the present article. Dr. Wilson, the principal investigator of the NSF grant, is an experienced survey sociologist who is primarily responsible for the development and execution of the surveys and focus groups employed in the NSF project. Dr. Resnik, a bioethicist formerly on the faculty of East Carolina University, provided the initial ideas for the NSF project as well as critical review of all steps in the project.
The authors would like to thank Dr. Robert P. Lowman (Chapel Hill), Dr. Albert Collier (Chapel Hill), Dr. Joseph M. Corless (Duke University), and Mr. Mathew K. Ronning (NCSU) for their assistance with the focus groups and Ms. Angel Griffin (ECU) for her technical assistance. This project was supported by a grant # SES-0322752 from the National Science Foundation and the intramural research program of the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). The ideas and opinions in this project do not represent the views of the NSF, NIEHS, or NIH.
2 Shamoo A, Resnik D. Responsible Conduct of Research. New York: Oxford University Press, 2003.
3 Macrina F. Scientific Integrity: An Introduction with Case Studies, 2nd ed. Washington, DC: American Society of Microbiology Press, 2000.
5 Wilson K, Brantley A, Schreier A, Resnik D. Scientific record-keeping and responsible research conduct [manuscript in preparation].
6 Martinson BC, Anderson MS, deVries R. Scientists behaving badly. Nature. 2005;435:737–38.
9 Wright DE. Data management: recording, access, retention, ownership. In: Davis T (ed). Management of the Biomedical Research Laboratories: Proceedings of a National Conference. Washington DC: Office of Research Integrity, 1998.
10 Kanare H. Writing the Laboratory Notebook. Washington, DC: American Chemical Society Press, 1985.
12 Bonneta L. Toward a Paperless Lab? Scientist. 2003;17:40–42.
13 Gibbon G. A Brief History of LIMS. Lab Automat Inform Manage. 1996;32:1–5.
20 Data Management: Research Records. In: New Investigators: A Quick Guide To Starting Your Research at UCSF. 〈http://www.research.ucsf.edu/QG/orQgDm.asp
〉. Accessed 2 October 2005. San Francisco: University of California.
24 University of California, San Francisco. Office of Research Guidelines for Laboratory Notebooks. In: New Investigators: A Quick Guide to Starting Your Research at UCSF 〈http://www.research.ucsf.edu/QG/orQgNb.asp
〉. Accessed 2 October 2005.
*Best practices and good practices are terms of art common in the literature on ethical and regulatory aspects of clinical trials and laboratory science. For example, the Food and Drug Administration10 has good clinical practices (GCPs) and good laboratory practices (GLPs) for the conduct of clinical trials, and the Environmental Protection Agency11 has good laboratory practices (GLPs) for laboratory research. Our best practices are intended to supplement but not supplant GCPs, GLPs, and other best practices for research. Cited Here...