Frumovitz, Michael MD, MPH; Kriseman, Maya L. MD; Sun, Charlotte C. DrPH; Blumenthal-Barby, Jennifer PhD; Sood, Anil K. MD; Bodurka, Diane C. MD; Soliman, Pamela T. MD, MPH
From the Department of Gynecologic Oncology, the University of Texas MD Anderson Cancer Center, and the Department of Obstetrics and Gynecology and the Center for Medical Ethics and Health Policy, Baylor College of Medicine, Houston, Texas.
See related editorial on page 493 and related article on page 498.
Supported in part by the National Institutes of Health through the University of Texas MD Anderson Cancer Center's Support Grant, CA016672.
Corresponding author: Michael Frumovitz, MD, MPH, Department of Gynecologic Oncology, CPB6.3244, Unit 1362, The University of Texas MD Anderson Cancer Center, 1155 Herman Pressler, Houston, TX 77030; e-mail: firstname.lastname@example.org.
Financial Disclosure The authors did not report any potential conflicts of interest.
OBJECTIVE: Selection of physicians for fellowships in obstetrics and gynecology subspecialties has become increasingly competitive. The number and quality of research publications is an important factor in the selection process. We sought to estimate the incidence of unverifiable publications among gynecologic oncology fellowship applicants.
METHODS: We reviewed the applications to a single gynecologic oncology fellowship program during 2004–2008. Articles and book chapters reported as published, in press, submitted, or in progress were searched for systematically by three reviewers using PubMed and Google. χ2 analysis was used to evaluate associations between demographic factors and unverifiable publications.
RESULTS: Two hundred forty-three applications met the inclusion criteria. Of the 35 applicants who listed membership in Alpha Omega Alpha, four (11%) were not listed on the organization's web site as inductees. Of the 464 articles reported as published or in press, only 387 (83%) could be verified. Of the 148 applicants who reported at least one published or in press article, 44 (30%) had at least one unverifiable publication. On multivariable analysis, only male gender increased the likelihood of unverifiable (“ghost”) publications (odds ratio 2.1, 95% confidence interval 1.1–4.1). Of the 282 manuscripts reported as submitted or in progress, only 126 (45%) were published. Of the 124 applicants who reported at least one submitted or in progress manuscript, 88 (71%) had at least one unverifiable manuscript.
CONCLUSION: The proportion of unverifiable publications listed on gynecologic oncology fellowship applications is concerning. Stringent review of applications before interview invitations and match list submission is warranted.
LEVEL OF EVIDENCE: III
In the past decade, the number of applicants to fellowship programs in obstetrics and gynecology tripled.1 Subspecialty training in obstetrics and gynecology has evolved from unofficial “apprenticeships” to board-certified fellowships. Whereas in the past, fellows were selected and recruited with an informal interview and a handshake, current graduating residents must undergo a rigorous application process including submission of a written curriculum vitae, letters of recommendation, and a personal statement. Scores from the U.S. Medical Licensing Examination and Council on Resident Education in Obstetrics and Gynecology as well as medical school grades also are submitted for review. Applicants then are selected for fellowships through the National Residency Match Program.
At the same time that the fellowship application process has become more rigorous, the fellowship programs themselves have formalized the structure of their curricula. In gynecologic oncology, a premium has been placed on research and every fellowship program requires 1–2 years of dedicated research time. There are numerous references on fellowship web sites devoted to training future academicians as a mission of gynecologic oncology training. Our fellowship web site states that graduates are expected “to become academic gynecologic oncologists.”2 Other programs speak of “supporting successful academicians”3 and preparing trainees for “an academic career in the subspecialty of gynecologic oncology.”4 Not surprisingly, it has become well recognized among applicants to gynecologic oncology fellowship programs that a key to success in the application process is to stress a commitment to becoming an academic physician after training. In an effort to back their credibility as potential researchers, applicants are openly encouraged by their mentors to perform research to bolster their curricula vitae. As interviewers, we often look to publications as a measure of a candidate's “academic potential.” Because there exists a strong assumption that listing published papers on an application is considered highly favorably, we wondered if applicants might list on their applications publications that could not be verified on careful review. The objective of this study was to estimate what percentage of publications listed on applications to our gynecologic oncology fellowship program were verifiable as published manuscripts. Other medical training programs have reported a wide range of misrepresentations (6–50%) by their applicants.5–11
MATERIALS AND METHODS
After obtaining approval from the institution's institutional review board, we reviewed all data submitted during 2004–2008 by applicants for the Gynecologic Oncology Fellowship at The University of Texas MD Anderson Cancer Center. To protect the identity of applicants who ultimately matched into the program, data on those applicants were excluded from the analysis as required by the institutional review board.
The types of data abstracted from the applications are listed in Box 1. Undergraduate major was categorized as science or engineering compared with liberal arts. Membership in the Alpha Omega Alpha Honor Medical Society was verified using the member search function on the organization's web site (www.alphaomegaalpha.org). For married female applicants who were not found on the Alpha Omega Alpha web site using their legal names and who had a maiden name listed on their application, attempts were made to find the applicant using the maiden name. Residency programs were classified as “academic” if they were affiliated with a university or medical school and “community” if they were associated solely with a teaching hospital or community health center. Citations were grouped as 1) published or in press (publications that one would expect to be verifiable); 2) submitted or in progress (articles that might not be verifiable given that neither submitted nor in progress implies that the work may or may not be published); and 3) book chapters.
Articles in peer-reviewed and nonpeer-reviewed journals and book chapters were reviewed. Listed articles were searched for in PubMed and on the Internet using Google by the data abstractor (M.L.K.). All citations that could not be verified by the data abstractor were then independently searched for by two full-time faculty members in the Department of Gynecologic Oncology at MD Anderson (M.F. and P.T.S.) for a total of three independent attempts to verify each listed article. Searches were done between June 1, 2010, and July 31, 2010, to allow adequate time for manuscripts listed as in press or submitted to appear on the appropriate search engines.12 Publications in foreign journals not included in PubMed and abstracts were not included because these proved difficult to accurately verify.
Descriptive methods were used to summarize the entire cohort of applicants. Further analysis was then undertaken of variables related to unverifiable manuscripts listed as published or in press. Using the presence or absence of at least one unverifiable published or in press journal article or at least one unverifiable book chapter, statistical analysis of this binary group using Mann-Whitney and χ2 tests was used to evaluate differences between groups. P values of <.05 were considered statistically significant. Missing data were coded as unknown and were excluded from the analysis. Variables found to be significant on univariable analysis were examined on multivariable analysis using a backwards stepwise regression. All data were analyzed using SPSS 17.0 for Windows.
From 2004 to 2008, 258 obstetrics and gynecology residents or graduates applied for the Gynecologic Oncology Fellowship at The University of Texas MD Anderson Cancer Center. After exclusion of the 15 applicants who ultimately matched into our program, 243 applicants were included in this analysis. Table 1 provides an overall description of the applicant cohort. Of the 35 applicants who listed membership in the Alpha Omega Alpha Honor Medical Society, four (11%) were not listed on the Alpha Omega Alpha web site as members.
One hundred forty-eight applicants (61%) stated on their application that they had at least one article published or in press (median 1, range 1–38). Of these 148 applicants, 44 (30%) had at least one publication that was not verifiable (37 applicants listed at least one unverifiable article as published, four listed at least one unverifiable article as in press, and three applicants listed both an unverifiable published and in press article). These 44 applicants represented 18% of the entire cohort of applicants (including those who listed no articles as published or in press). Of the four applicants whose Alpha Omega Alpha membership was unverifiable, two had no publications listed, one had all listed publications verified, and the last listed unverifiable publications. In total, 464 articles were listed as published or in press. Of these, only 387 (83%) were verified.
One hundred twenty-four applicants (51%) stated on their application that they had at least one manuscript submitted or in progress (median 1, range 1–9). Of these 124 applicants, 88 (71%) had at least one manuscript that would not be verifiable at the time data were abstracted. These 88 applicants represented 36% of the entire cohort of applicants (including those who listed no articles as submitted or in progress). In total, 282 manuscripts were listed as submitted or in progress. Of these, only 126 (45%) were verified as published.
Finally, 33 applicants (14%) listed at least one book chapter on their application (median 1, range 1–11). Of these 33 applicants, eight (24%) had at least one book chapter that was not verifiable. These eight applicants represented 3% of the entire cohort of applicants (including those who listed no book chapters). In total, applicants listed 54 book chapters. Of these, only 41 (76%) were verifiable.
Twenty-five of the 97 male applicants (26%, 95% confidence interval [CI] 18–35%) had at least one unverifiable publication compared with only 19 of the 146 female applicants (13%, 95% CI 8–20%; P=.02). Nineteen graduates of non-U.S. medical schools (28%) had at least one unverifiable publication compared with 25 U.S. medical school graduates (14%) (P=.03). Applicants from community programs were more likely to have unverifiable publications than applicants from university-based programs (25% compared with 14%, P=.03). Finally, applicants who had applied previously were more likely to have unverifiable publications than those applying for the first time (38% compared with 17%, P<.05). On univariable analysis, there were no differences in percentages of applicants with unverifiable publications by ethnicity, undergraduate major, type of medical degree (MD compared with DO), membership in Alpha Omega Alpha, or advanced degrees (any compared with none). On multivariable analysis, only male gender remained a significant predictor with an odds ratio (OR) of 2.1 (95% CI 1.1–4.1) for listing at least one unverifiable publication. Although applicants who had previously applied had an OR of 2.6 (95% CI 0.9–7.8) and those who attended an academic residency program had an OR of 0.54 (95% CI 0.3–1.1) of submitting an unverifiable publication, neither was statistically significant.
In summary, we were unable to verify articles listed as either published or in press in almost one third of gynecologic oncology fellowship applications. Although many readers may find the high rate of unverifiable publications startling, the rate is similar to the rate of publications found to be “misrepresented” in previous studies of applicants to residency programs (22–34%)5–7 and applicants to other fellowship programs (16–50%).8–11 Unlike previous authors, we took care to use the word “unverifiable” as opposed to “misrepresented” because of the severely negative connotation of the latter. In addition, we found no evidence of publication of 56% of manuscripts listed as submitted or in progress. For anyone involved in residency or fellowship interviews, this finding is less surprising. Finally, we were unable to find a listing for 11% of applicants who listed membership in the Alpha Omega Alpha Honor Medical Society on the organization's web site.
Gussman and Blechman13 speculated as to possible reasons behind the high rates of unverifiable content on postgraduate medical training applications. They proposed that applicants might be careless, which could lead to errors; might be unable to understand the directions on the application clearly; might be receiving poor mentorship from their advisors; or might purposely misrepresent themselves in “a desire to strengthen the application.”13 Although mistakes resulting from any of these reasons would make us question an applicant's attention to detail and his or her ability to perform at the highest levels required in medicine, we hope that most cases of unverifiable publications are a result of carelessness, misunderstanding, or lack of good advice from superiors. Unfortunately, results of a recent meta-analysis suggested that some form of purposeful misrepresentation of research experience is likely present in more than 10% of applications submitted to residency or fellowship programs.14 This is particularly concerning given the Accreditation Council for Graduate Medical Education's designation of professionalism and adherence to ethical principles as one of six core competencies.15
Should gynecologic oncology fellowships share in the blame for falsifications on applications? The fellowships have become highly competitive: 38% of applicants did not match in 2011.16 At the same time, our program descriptions and web sites stress our commitment to training academicians. Is it any wonder, then, that applicants might feel the need to bolster their research experience?
Although fellowship programs, ours included, stress the strong desire to train academicians, we have little objective data on the applications to predict which potential fellows will actually stay in academia. In addition, at current training rates, gynecologic oncologists might expect a 20% increase in caseloads over the next 40 years.17 Certainly much of the future need will be away from academic centers, in rural areas where women already are less likely to get optimal care for their gynecologic malignancies.18
The study presented here obviously has some limitations. Misspellings or other small errors in listed citations may have made verifying articles difficult. In addition, name changes attributable to marriage or other reasons may have added to the number of articles that we were unable to verify. We tried to anticipate some of these issues by verifying unfound citations using other authors listed, the article title, using multiple search engines, and having multiple investigators attempt to validate. We understood that many articles listed as submitted or in progress ultimately would never get published and therefore we did not consider those unauthenticated listings as attempts to mislead. We report those numbers simply out of interest to fellowship interviewers. We are, however, at a loss for how to explain our inability to verify membership in Alpha Omega Alpha for 11% of the applicants who listed themselves as members.
In conclusion, a large number of fellowship applicants listed publications on their applications that were unverifiable by three independent reviewers. Although we have taken care to describe these publications as unverifiable, findings from a recent meta-analysis suggest that a significant proportion of these unverifiable documents are overt attempts to misrepresent prior experience.14 Fortunately, the Electronic Residency Application Service now has a field for the PubMed identifier, and for next year's applications, we will ask applicants to provide this number. In addition, we will encourage applicants to e-mail or fax copies of any articles that are not listed in PubMed. Furthermore, because there may be many reasons a person may change his or her name, our cover letter to potential applicants will ask specifically for all names previously used to help us better verify information presented on their resume. Finally, we would emphasize not just the importance of detection of misrepresentation, but also the importance of prevention of misrepresentation, which could be accomplished by educating learners (medical students, residents, and fellows) as well as faculty and mentors about academic integrity and research ethics. The importance of such education is supported by The American College of Obstetricians and Gynecologists' Committee on Ethics in their recent statement on Professional Responsibilities in Obstetric–Gynecologic Medical Education and Training.19
1. Fang YM, Egan JF, Rombro T, Morris B, Zelop CM. A comparison of reasons for choosing obstetrician/gynecologist subspecialty training. Conn Med 2009; 73: 165–70.
5. Dale JA, Schmitt CM, Crosby LA. Misrepresentation of research criteria by orthopaedic residency applicants. J Bone Joint Surg Am 1999; 81: 1679–81.
6. Gurudevan SV, Mower WR. Misrepresentation of research publications among emergency medicine residency applicants. Ann Emerg Med 1996; 27: 327–30.
7. Konstantakos EK, Laughlin RT, Markert RJ, Crosby LA. Follow-up on misrepresentation of research activity by orthopaedic residency applicants: has anything changed? J Bone Joint Surg Am 2007; 89: 2084–8.
8. Bilge A, Shugerman RP, Robertson WO. Misrepresentation of authorship by applicants to pediatrics training programs. Acad Med 1998; 73: 532–3.
9. Glazer JL, Hatzenbuehler JR, Dexter WW, Kuhn CB. Misrepresentation of research citations by applicants to a primary care sports medicine fellowship program in the United States. Clin J Sport Med 2008; 18: 279–81.
10. Panicek DM, Schwartz LH, Dershaw DD, Ercolani MC, Castellino RA. Misrepresentation of publications by applicants for radiology fellowships: is it a problem? AJR Am J Roentgenol 1998; 170: 577–81.
11. Sekas G, Hutson WR. Misrepresentation of academic accomplishments by applicants for gastroenterology fellowships. Ann Intern Med 1995; 123: 38–41.
12. Kuo PC, Schroeder RA, Shah A, Shah J, Jacobs DO, Pietrobon R. 'Ghost' publications among applicants to a general surgery residency program. J Am Coll Surg 2008; 207: 485–9.
13. Gussman D, Blechman A. Verification of publications, presentations and posters by applicants to a residency in obstetrics and gynecology. J Reprod Med 2007; 52: 259–61.
14. Wiggins MN. A meta-analysis of studies of publication misrepresentation by applicants to residency and fellowship programs. Acad Med 2010; 85: 1470–4.
16. National Resident Matching Program, Results and Data: Specialties Matching Service 2011 Appointment Year. National Resident Matching Program, Washington, DC. 2011.
17. Wallace AH, Havrilesky LJ, Valea FA, Barnett JC, Berchuck A, Myers ER. Projecting the need for gynecologic oncologists for the next 40 years. Obstet Gynecol 2010; 116: 1366–72.
18. Carney ME, Lancaster JM, Ford C, Tsodikov A, Wiggins CL. A population-based study of patterns of care for ovarian cancer: who is seen by a gynecologic oncologist and who is not? Gynecol Oncol 2002; 84: 36–42.
19. . Committee opinion no. 500. American College of Obstetricians and Gynecologists. Committee on Ethics. Obstet Gynecol 2011; 118: 400–4.