Cloft, Harry J. MD, PhD; Shengelaia, George G. MD; Marx, William F. MD; Kallmes, David F. MD
Dr. Cloft is assistant professor and Dr. Shengelaia is research associate, Department of Radiology, Emory University Hospital, Atlanta, Georgia; Dr. Marx is assistant professor and Dr. Kallmes is assistant professor, Department of Radiology, University of Virginia Health Sciences Center, Charlottesville, Virginia.
Correspondence and requests for reprints should be addressed to Harry J. Cloft, MD, PhD, Department of Radiology, Emory University Hospital, 1364 Clifton Road, NE, Atlanta, GA 30322; telephone: (404) 712-4991; fax: (404) 712-0293; e-mail: 〈firstname.lastname@example.org〉.
Purpose: To test the hypothesis that articles published as “preliminary” or “pilot” reports are followed by more definitive publications in only a minority of cases.
Method: A survey of Medline was performed for reports pubished in 1992 in journals listed in the Abridged Index Medicus that had the word “preliminary” or “pilot” in the title. For identified reports, a Medline search of publications in 1992 through 1999 was performed, using lead author's name, second author's name, and senior (last) author's name, and at least one keyword based on the publication title. Preliminary and pilot publications were subdivided by type of study (controlled clinical study, case series, laboratory or nonclinical) and by the report of either positive or negative results. Rates of publication based on study design and publication bias were compared using the chi-square test for statistical significance.
Results: The rate of publication of follow-up reports within seven years of the initial publication was 27%. Follow-up studies of controlled clinical studies (40%) were published more frequently than were those of laboratory or nonclinical studies (31%) or case series (22%), but these differences were not significant (p > .10). There was no statistically significant difference in follow-up publication rates based on publication bias.
Conclusion: Only 27% of studies published as preliminary or pilot reports were subsequently followed by a more definitive publication. While the words preliminary and pilot suggest that publication of further, refined work is pending, this is often not the case.
The publication of preliminary and pilot studies, such as scientific presentations at meetings, is an important means of rapidly conveying new information, summarizing current research, and focusing future research efforts. Using the word “preliminary” or “pilot” in an article's title implies that further, more definitive work is pending. However, since previous investigators have determined that less than half of all abstracts presented at scientific meetings are eventually followed by publications in peer-reviewed journals,1–8 one might suspect that there is a similarly low publication rate for follow-up reports of preliminary reports, which would be detrimental to the body of medical literature, because the results of preliminary and pilot studies often document important phenomena that require further investigation and support.
We sought to determine the rate of publication of follow-up reports in indexed medical journals for preliminary and pilot studies published in journals listed in the abridged Index Medicus. We defined a preliminary or pilot study as any publication with the word “preliminary” or “pilot” in the title, and searched Medline to find articles published in 1992 with either of these words in the title. This search yielded 146 reports. We defined follow-up report as a subsequent publication by one or more of the authors of the initial report in which the main focus of the article was the same as that of the initial report. For each preliminary report identified, we searched Medline for articles published from 1992 through 1999 to identify follow-up reports by using the lead author's name, the second author's name, the senior (last) author's name, and at least one keyword based on the publication title.
We subdivided the preliminary and pilot studies based on the type of study (controlled clinical study, case series, and laboratory or nonclinical). We calculated an overall rate of follow-up publication and the rates of publication for the individual study designs. We tested for univariate significance for differences in publication rates based on study design using the chi-square method.
In order to assess the role of publication bias in the rate of follow-up study publication, we examined each initial report to determine whether it reported a positive or a negative result. The results of a study were defined as positive if the study variable had produced positive (beneficial) results or statistically significant results,9 and all other studies were defined as negative. If there was more than one result, the study was identified as positive if any one result was positive. The rates of publication of follow-up reports for initial studies reporting positive and negative results were then determined and compared using the chi-square method.
Finally, we measured the times in years between publication of the initial and follow-up reports.
Results are shown in Table 1. The overall rate of publication of follow-up reports within seven years of the initial preliminary report was 27%. Follow-up studies of controlled clinical studies (40%) were published more frequently than were follow-up studies of case series (22%), but this diference was not significant (p > .10). Follow-up laboratory or nonclinical studies were published at a rate (31%) that was not significantly different from the rate of publication for case series (p > .10) or for controlled clinical studies (p > .10). There was no statistically significant difference (p > .10) between the rates of follow-up report publication for initial studies reporting positive results (34%) and those reporting negative results (33%).
The time course of follow-up report publications are shown in Figure 1. Approximately half of the follow-up reports were published within two years, and 98% were published within six years of the initial publication of the preliminary reports.
Our results indicate that less than one third of articles published as preliminary reports are later followed by reports published in an indexed journal. To our knowledge, no other study of publication rates of follow-up reports has been conducted. However, similar studies have been conducted on publication rates of reports following the presentation of abstracts at national meetings. These investigators determined that less than half of all abstracts are eventually followed by publications in peer-reviewed journals, with publication rates varying between 21% and 60% based on medical specialty.1–8 A meta-analysis of these studies appeared in JAMA in 1994, and the overall rate of publication was 49% (after eliminating one article limited to randomized clinical trials).6
In our study, follow-up reports to case series were published less often than were follow-up reports to controlled clinical studies and laboratory or nonclinial studies, but these differences were not significant. Controlled clinical studies and laboratory or nonclinical studies, by nature, are generally prospective rather than retrospective, and they require careful forethought and planning to carry out. Case series, in contrast, frequently are retrospective and take advantage of data that already exist in patient databases. One might expect that a plan to follow a preliminary report with the publication of a more definitive report would be given more forethought in a prospectively planned study than it would be given in a case-series study, and therefore one might expect higher rates of publication of follow-up reports in controlled clinical and laboratory or nonclinical studies. Unfortunately, our sample size was too small to establish statistically significant differences in publication rates based on study design. It is important to note, however, that the publication rate was never more than 40% for any type of study design.
One factor that could affect rates of publication of follow-up reports is publication bias, such that reports with positive results tend to be published preferentially over those with negative results.9 We did not find evidence of publication bias, but, again, our sample was too small to test this rigorously with statistical methods. Several studies have shown that abstracts presented at scientific meetings are less likely to be subsequently published if their results are considered negative, leading to potential problems with publication bias when meta-analyses of the existing literature are performed.9,10
We chose to study preliminary reports published in 1992 because this time point allowed a reasonable amount of time for the publication of a follow-up report. Previous authors have noted that most reports based on previously presented abstracts are published within four years of an abstract's presentation.2–7 Our results support the assertion that, if a follow-up paper is not published within seven years of the preliminary or pilot report, the data from the original report should be viewed in the context of whatever uncertainties were present in that preliminary study.
We studied preliminary reports published in journals included in the Abridged Index Medicus only. The Abridged Index Medicus was created by the National Library of Medicine and covers 119 English-language journals chosen for their accessibility, quality, and clinical relevance. The journals included are among the best in medical publications. Our results may, therefore, reflect subsequent publication rates for these better journals only. Less prestigious journals may have very different publication rates for follow-up reports. The Abridged Index Medicus includes publications from nearly all medical specialties. We did not stratify our results for various medical specialties because we did not have a large enough sample for such subgrouping of data. Future investigations might focus on publication rates of follow-up reports for journals that focus on particular medical specialties.
The methods we used to search Medline may have overlooked some published manuscripts. Our search for preliminary reports was reasonably but arbitrarily limited by searching for only those reports with the word “preliminary” or “pilot” in the title. However, it is unlikely that any other combination of terms would have yielded a more complete representation of preliminary reports. To identify follow-up reports, we cross-indexed at least the authors' names with relevant words from the abstract title. It is possible, but unlikely, that the final publication of some follow-up reports would not include either the presenting author or the senior author. Additionally, it is unlikely that the subject of the paper would change so radically that the most identifying word in the title would not appear in the text of the manuscript.
Failure to publish a follow-up report after a preliminary or pilot report is deleterious to the body of medical literature, because preliminary and pilot reports often document important phenomena that require further investigation and support. The reasons underlying failure of publication of follow-up reports include: (1) the investigators have lost interest; (2) the follow-up results were negative; (3) the subsequent work was much more difficult to perform and fund; (4) the initial investigation and its results were weak and the investigators tried to rationalize this to editors by declaring their work to be preliminary, having no intention of a subsequent, more definitive report. Thus, if a follow-up study is never published, the results of the initial report must be viewed with caution.
The use of the terms “preliminary” and “pilot” should be reserved for reports published when there is an earnest intent of the authors to publish a follow-up report.
1. Corry AM. A survey of the publication history of randomly selected IADR/AADR abstracts presented in 1983 and 1984. J Dent Res. 1990;69:1453–5.
2. De Bellefeuille C, Morrison CA, Tannock IF. The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol. 1992;3:187–91.
3. Goldman L, Loscalzo A. Fate of cardiology research originally published in abstract form. N Engl J Med. 1980;303:255–9.
4. Juzych MS, Shin DH, Coffey JB, Parrow KA, Tsai CS, Briggs KS. Pattern of publication of ophthalmic abstracts in peer-reviewed journals. Ophthalmology. 1991;98:553–6.
5. Meranze J, Ellison N, Greenhow DE. Publications resulting from anesthesia meeting abstracts. Anesth Analg. 1982;61:445–8.
6. Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts. A meta-analysis. JAMA. 1994;272:158–62.
7. Yentis SM, Campbell FA, Lerman J. Publication of abstracts presented at anaesthesia meetings. Can J Anesth. 1993;40:632–4.
8. Marx WF, Cloft HJ, Do HM, Kallmes DF. The fate of neuroradiology abstracts presented at national meetings in 1993: rate of subsequent publication in peer-reviewed, indexed journals. Am J Neuroradiol. 1999;20:1173–7.
9. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA. 1998;280:254–7.