Secondary Logo

Journal Logo

Invited Commentaries

A Crisis of Trust Between U.S. Medical Education and the National Board of Medical Examiners

Gesundheit, Neil MD, MPH

Author Information
doi: 10.1097/ACM.0000000000003131
  • Free

Abstract

The National Board of Medical Examiners (NBME) is a powerful, well-funded organization and gatekeeper to the medical profession; it sells products and services and provides state medical boards with a common system for licensing physicians in the United States. Here, I comment on the Perspective by Carmody and Rajasekaran in this issue of Academic Medicine that explores whether the NBME’s score-reporting policy, along with its current products and services, constitutes a conflict of interest.1 Specifically, I explore the history and more recent financial growth of the NBME, I argue that the NBME constitutes a self-regulated monopoly, I examine the influence of what Carmody and Rajasekaran characterize as “Step 1 mania,” and I outline 4 steps the NBME can take to regain the trust of the medical education community.

The Origins and Evolution of the NBME and the United States Medical Licensing Examination (USMLE)

Founded 5 years after the Flexner Report of 1910, the NBME was established with the noble goal of creating a rational, standardized system by which physicians could demonstrate their knowledge and suitability to practice medicine. The underlying goal of the NBME was to serve the public and ensure that practicing physicians in the United States met standards of competency in medical knowledge.

For its first 75 years, the NBME competed with an alternate pathway to licensure, the Federation Licensing Examination (FLEX) administered by the Federation of State Medical Boards (FSMB). In 1992, the NBME and FSMB engaged in a collaborative agreement to cosponsor and unify their licensure exams in the form of the United States Medical Licensing Examination (USMLE). The USMLE initially comprised 3 parts, and in 2004, the NBME and FSMB added a fourth clinical competency component, Step 2-CS (clinical skills). The NBME has primary oversight of Step 1 and Step 2-CK (clinical knowledge), the organizations share oversight of Step 2-CS, and the FSMB has primary oversight of Step 3.

The Financial Growth of the NBME: A Nonprofit With Growing Revenues, Margins, and Net Assets

Organizations, such as the NBME, that receive 501(c)(3) status under the U.S. Internal Revenue Code are considered nonprofits and, as such, do not have owners or shareholders and cannot distribute earnings; however, these registered nonprofits are often encouraged by their governing boards and market forces to grow their revenues and increase net assets.

The NBME reports its financial status to the Internal Revenue Service (IRS) annually on IRS Form 990 (the nonprofit equivalent of IRS Form 1120 for corporations and Form 1040 for individuals). Although financial details of the NBME’s operations are not provided in the organization’s 2018 annual report,2 IRS Tax Forms 990 are available to the public.3 In 2017, the NBME reported program service revenue of $153.9 million (M) and earnings (revenue less expenses) of $8.0M.3 Over the 5-year period from 2013 to 2017, the NBME’s overall program service revenue was $712.1M, with an annual growth rate of 5% per year.3 The aggregate earnings during this 5-year period were $39.7M.3 The NBME’s profit margin ratio, defined as net income (revenue less expenses) divided by net sales (program service revenue) over this 5-year period, was approximately 6%. For comparison purposes, the margins of the for-profit companies Walmart, Target, and Ford Motor Company over this same period averaged, respectively, 3%, 4%, and 4%.4 Because of these years of positive revenue and robust margins, the NBME’s net assets grew to $177.6M at the end of tax year 2017.3

Carmody and Rajasekaran have dug deeper into NBME’s finances and have shown that the organization’s revenues more than tripled between 2001 and 2017, far outpacing inflation.1 The fee for Step 1, a main revenue generator, increased from $435 in 2004 to $630 in 2019, again exceeding the rate of inflation.1 Further from 2008 to 2017, the NBME workforce grew to include more than 200 employees earning more than $100,000 per year.1 In 2017, the NBME’s retiring president received over $1M, and the incoming president received over $820,000, in total compensation.3 Fourteen employees earned more than $400,000.3 This compensation for top NBME executives in 2016–2017 was easily in the top 0.5% of charitable nonprofits, according to Charity Navigator (charitynavigator.org), an organization that tracks the salaries of nonprofit CEOs.5(pp9–10)

In many ways, the NBME has become a big business, with growing revenues of more than $700M over the 5-year period ending in 2017, new products, positive earnings of nearly $40M during this same period, strong margins, and highly paid staff and executives. While this growth and financial health is in many ways to the credit of NBME management, the figures also highlight the inherent risk of placing the organization at the nexus of key decision making regarding requirements for the use of its products.

The High NBME Fees Increase the Cost of Medical Education and Medical Student Debt

Notably, the consumers of NBME services and products are medical students, which, as Carmody and Rajasekaran outline, has consequences for the students themselves and medicine.1 Medical students ultimately pay the cost of nearly all of NBME services, in direct and indirect charges. The direct cost for U.S. applicants who take the USMLE is $630 for each step exam: Step 1 and Step 2-CK. Costs are approximately 50% higher for international medical graduates. Students incur further direct costs when they purchase NBME practice exams. Because the Step 1 and Step 2-CK tests have become high-stakes exams, most medical students take NBME-sponsored practice exams—the Comprehensive Basic Science Self-Assessment (CBSSA) for Step 1 and the Comprehensive Clinical Science Self-Assessment (CCSSA) for Step 2-CK—at a cost of $60 per exam. The number of CBSSA exams purchased by medical students grew 10-fold between 2005 and 2016.1 With the growth of Step 1 mania, practice exams have become one of the largest areas of product and profit margin growth at the NBME.1 Because of the popularity of practice exams, the NBME constantly refreshes its offerings (“forms”), ensuring strong revenues for this product. Although anxiety about Step 1 is well established, anxiety over Step 2-CK has also been increasing, and medical students are purchasing increasing numbers of CCSSA tests, also at a cost of $60 per test. Assuming medical students each purchase a total of 5 Step 1 and Step 2-CK practice exams, additional expenses incurred equal $300 per applicant. The additional costs of Step 2-CS ($1,290) and Step 3 ($875) bring the total expenses incurred directly by students for USMLE services to $3,725 (which does not include expenses for traveling to the Step 2-CS test site). These expenses do not include supplementary preparatory (“test prep”) materials, which are also in common use.

There are also indirect costs to medical students. Most medical schools require students to take NBME knowledge (shelf) exams during their clinical training. The cost of these exams to medical schools is approximately $46 per exam per student. Further, assuming that students take at least 5 shelf exams during medical school, the cost for this additional testing, when passed on as student fees or tuition, is at least $230 per student. Students may bear the burden of additional expenses incurred by medical schools that use the NBME’s Customized Assessment Services (CAS); this service allows schools to create exams from the USMLE question bank.

Not including expenses for CAS, the total fees for USMLE services are approximately $4,000 (including both indirect and direct costs) per U.S. medical student.

To summarize thus far, the cost of USMLE assessments is significant, particularly in light of the high and rapidly rising medical student debt, which now approaches a median of $200,000.6 The NBME, while registered as a 501(c)(3) nonprofit, enjoys large revenues and strong margins, employs a highly paid executive staff, and charges significant fees that contribute to medical student debt.

The NBME Is a Self-Regulated Monopoly

As mentioned, before 1992–1994, most states accepted 2 pathways for medical licensure, the NBME and the FLEX. The 1992 agreement to merge licensure testing created a monopoly—that is, a commodity or service in exclusive control by a company or group. Most public-service monopolies, whether for-profit or nonprofit, are regulated by an outside, impartial board that safeguards the public trust against the risk of abuse. As an example, electricity and natural gas services in southern California are provided nearly exclusively by the Southern California Edison (SCE) Company. Any change in pricing for SCE’s services must be approved by the California Public Utilities Commission (CPUC). CPUC commissioners are appointed by the governor and confirmed by the California State Senate. When SCE registers to raise its rates for electricity, it must document, for the CPUC, the greater cost of generating electricity or purchasing natural gas, any alternative sourcing that it has sought, and the effect of the price increase on communities. Public hearings often take place wherein the company faces scrutiny and must defend its pricing decisions. CPUC commissioners then vote to approve or reject SCE’s proposals. In short, neither SCE nor its board unilaterally set SCE’s rates.

By contrast, the NBME is regulated by an internal board, and changes in pricing strategies or new products do not require approval by a public group or an indisputably independent body, such as the Liaison Committee on Medical Education of the Association of American Medical Colleges (AAMC) or the AAMC’s Council of Deans. Although the NBME has a sizeable and diverse 80-member board, it is not clear whether changes in pricing structures are openly discussed and debated beyond perhaps closed meetings of the NBME Executive Board (of approximately 10 members). In addition, the expertise of the members of the NBME Executive Board is primarily education. In contrast to a public utility that is prevented from monopolistic abuses by external regulation, few checks and balances help control and regulate the NBME. The price charged for Step 1 and Step 2-CK examinations, as an example, can be raised without extensive justification or the scrutiny of the medical education community. The NBME can also raise its passing cutoff score for Step 1 and Step 2-CK without strong, evidence-based justification that a higher number predicts greater clinical success. More exam failures generate more revenue both from retesting and from increased sales of practice materials to students hoping to surpass the higher bar (see below). While it may not be the organization’s intent, the NBME benefits whenever pass thresholds increase.

Given that the NBME benefits from retesting and test preparation materials, its board members are likely not objective participants or arbiters of the current Step 1 score-reporting debate. The organization’s desire for growth and influence may be jeopardizing its original noble mission and its source of influence.

The NBME Is a Registered Nonprofit, But It Is Not Charitable

Perhaps equally distressing to medical students as the cost of USMLE testing is that the NBME and FSMB charge full fees to students who fail and must retake any of the 4 USMLE exams. This policy of charging a retesting fee is in line with the policy of the American Board of Medical Specialties, but there is a key difference. Medical specialists who take board exams are licensed physicians on the threshold of earning average salaries of approximately $300,000 annually.7 Medical students taking USMLE exams have a mean debt of $200,0006 and are on the threshold of 3 to 7 years of training before they are able to begin to pay off debt. Thus, in addition to the humiliation, psychological damage, and stigma of failing a licensing exam, a medical student bears the added burden of paying a second test fee. Given my years working with students, I can attest to the financial struggles students face. One medical student I mentored required 3 attempts before passing Step 2-CS and incurred $4,000 of debt in the process. Surely the NBME and FSMB, with significant net assets and an IRS classification in a category that includes charities, can be more charitable in their treatment of physicians-in-training.

The NBME and Step 1 Mania—A Perfect Toxic Storm

National debate about the disproportionate emphasis placed by graduate medical education (GME) program directors (PDs) on the USMLE Step 1 exam has been accelerating, especially given the negative consequences not only on medical students and their mental health but also on undergraduate medical education (UME).8 As the primary sponsor of Step 1, the NBME will undoubtedly bear financial consequences for any change to the format or grading of the Step 1 exam; thus, how the NBME interacts with the decision to change Step 1 score reporting is a relevant concern for medical educators.

Some of the concerns about the Step 1 exam and Step 1 mania have been discussed previously in this journal8,9 and in other forums. The dilemma has 3 basic aspects:

  1. The Step 1 exam is used as a gatekeeper or filter by many residency fields and PDs, even though it was designed as a pass/fail exam and has no rigorous data to validate its use to predict residency performance;10
  2. Because of the way many residency PDs use Step 1, the exam is perceived by students as the “ticket” to residency; and
  3. Because of this perception, many medical students make achieving a high Step 1 score their foremost objective for their preclerkship studies. They downgrade other topics and activities (e.g., understanding disease mechanisms; enhancing critical thinking and clinical reasoning; improving physical examination, interview, and communication skills; understanding the psychosocial context of illness; undertaking early clinical experiences and community engagement; volunteering; maintaining physical and emotional health and healthy relationships) as they, instead, elevate Step 1.

At Stanford, despite a faculty with 8 Nobel laureates and extensive clinical facilities that include 5 teaching hospitals, some medical students virtually ignore the preclerkship curriculum in favor of one or more parallel curricula they create on their own. Their teachers are First Aid (https://firstaidteam.com; published by McGrawHill Education), U World (https://www.uworld.com; Dallas, Texas), Pathoma (https://www.pathoma.com; Chicago, Illinois), Sketchy Medical (https://sketchymedical.com; Los Angeles, California), or one of 20 additional resources designed to help students maximize their Step 1 score.8 As the Step 1 exam approaches, classrooms and voluntary laboratory exercises are lightly attended. Students decrease interactions with one another. Community clinics, normally supported by medical students, are understaffed. Moreover, although this parallel curriculum no doubt has some merit, many of the techniques it inculcates are aimed to improve factual recall rather than understanding; that is, as Wartman has described, they focus on building “information” rather than the higher goal of building “knowledge.”11 Information is easily and reliably obtained in clinical practice by using a search tool on a smart phone, laptop, or portable tablet computer. Why graduate students would forgo an in-depth and multidimensional medical curriculum that helps them achieve knowledge and an understanding of medicine for, instead, learning tools that primarily encode memorization of facts might seem inexplicable. But students do, and the reason is Step 1 mania and the toxic storm it has created.

The NBME, whether by intent or happenstance, has been the beneficiary of Step 1 mania. With the increasing pressure to perform well on Step 1, students have used NBME practice exams and resources to boost their Step 1 scores. In response to rising national scores, which are in turn the result of better espionage by the burgeoning Step 1 prep industry, the NBME has asked question developers to create new material, often resulting in the testing of obscure and less relevant facts. This pursuit of medical trivia has affected medical education. For example, in the endocrine physiology course that I have taught for 20 years, when my teaching colleagues and I discuss hormonal signaling, we review the mechanisms and diversity of signal transduction and the ways that it can be modulated and disrupted. We discuss more than 50 hormones, but emphasize that memorizing exactly how each hormone signals is not important since specifics can always be looked up if they become relevant to patient care or research. Yet students report that our advice has been misguided; the Step 1 exam, in some iterations, has indeed tested for detailed facts about specific hormone signaling mechanisms. This drive for detail appears to define the “excellent” candidate for PDs who use Step 1 as a guide to identify trainees worthy of acceptance into their residency programs. Thus, the Step 1 exam has badly distorted the priorities of U.S. medical students. Assessment drives curriculum (even if inadvertently),12 and the current Step 1 climate has had a severe and adverse effect on curriculum development and student–faculty engagement.

As mentioned, this issue of Academic Medicine features a timely exposé by Carmody and Rajasekaran.1 The InCUS (Invitational Conference on USMLE scoring) meeting held in March 2019 was an opportunity to discuss the topic of Step 1 score reporting (pass/fail vs numerical score) and its effect on students and medical education, but the fact that the NBME and FSMB were organizers at the epicenter of the meeting undermined its legitimacy. I agree with Carmody and Rajasekaran: the NBME has a deep and inescapable financial conflict of interest in determining the future direction of the Step 1 exam. Below, I outline 4 steps, including recusal from discussion and decision making on the topic of Step 1 score reporting, that the NBME can take to regain the trust of the medical education community.

A Proposal With 4 Steps to Regain Trust in the NBME

The 4 steps I propose—recusal, transparent financial reporting, an independent oversight committee, and free retesting—will implement checks and balances with regard to the activities of the NBME and will begin the process of restoring confidence in the organization.

1. The NBME should recuse itself from current discussions and decision making about the reorganization of the Step 1 exam

The NBME may feel that it “owns” the Step 1 exam, but medical educators (including UME faculty and GME PDs), students (future physicians), and state medical boards should be the true owners of the exam. The NBME’s primary role is to implement a licensing exam to safeguard the public, not to influence medical education. Because of the NBME’s conflicts of interest, as described by Carmody and Rajasekaran1 and with which I agree, the NBME should recuse itself from participating in discussions and decision making about changes to the grade reporting, minimum passing score, and other issues surrounding Step 1.

2. The NBME must be transparent in disclosing its full finances

The NBME’s annual reports must show the organization’s full finances, including the sources and amounts of revenues and expenses. The NBME should disclose salaries paid to the most highly compensated employees, which can be shown in an anonymized format. Annual reports must also describe the decisions made by the NBME’s Compensation Committee. This information is currently missing from its 2018 56-page annual report.2 For-profit corporations regularly provide this information either in their annual reports or in quarterly 10-K forms. The NBME’s size and importance to medical education demand a higher level of transparency than the organization has heretofore provided. Disclosing the information on annual IRS Form 990 is required by law, but it is hidden from easy public view and, thus, is not sufficient.

3. New products, changes in prices for current products, and changes to pass thresholds must be approved by an independent oversight committee

The NBME provides required services for which there is no competition; therefore, changes to its products and pricing should be approved by an independent oversight committee. Currently the NBME maintains a board composed of approximately 80 diverse and talented members, mostly medical educators, who represent many constituencies and provide vision for the organization’s activities.2 An executive board of approximately 10 members is elected from the larger board.2 This system provides the appearance of independence, but NBME management may still exert influence in the selection of some board members. Despite the talented constituents of the NBME board and its smaller executive board, the AAMC or another suitable third party should organize a financial oversight committee that is unquestionably and fully independent of the NBME. Any new product, change in pricing, or change in pass threshold of an NBME licensing exam should be approved by such an independent oversight committee—just as an oversight committee reviews and approves changes to prices charged by a public utility company. Similarly, given the NBME’s status as sole provider of required services, the committee members must be appointed through an unquestionably independent process.

4. The NBME should not charge a fee for retaking a failed licensing exam

Failure on a licensing exam achieves its purpose—to identify a student who needs to improve knowledge and/or skills in the tested area—but recognizing that students suffer humiliation and psychological trauma when they fail an exam is important. Adding a steep retesting fee is unnecessary and compounds harm incurred by a student who is likely already in debt. Given the NBME’s considerable net assets of nearly $180M and positive earnings of $40M in 2013–2017, this retest fee should be waived.

In Sum

The NBME, although registered as a 501(c)(3) nonprofit, has grown to become a powerful organization that resembles a for-profit corporation. It enjoys significant revenue and earnings, has increased its offerings of revenue-enhancing products, and employs a highly compensated executive management team. Medical students are ultimately the source of nearly all the NBME’s revenue. The growth of the NBME has coincided with and contributed to the increase in medical student debt, which is at historically high levels. The NBME has operated essentially as a monopoly since its agreement in the early 1990s with the FSMB to cosponsor the 4-part USMLE, which is required for licensing physicians in the United States. Although the NBME has developed many valuable products and is ostensibly governed by a talented and capable board, the NBME has inherent financial conflicts of interest, appears to profit from its status as gatekeeper to the medical profession, and should, therefore, recuse itself (or be forced to be recused) from discussions or policy-making decisions related to changing the grade reporting of (and other aspects related to) the USMLE Step 1 exam. Further, the NBME should be transparent in all aspects of its finances, and new NBME products and pricing structures should be approved by an independent oversight entity. Finally, the NBME should not charge medical students and residents to retake USMLE licensing exams in cases of failure.

Although this Invited Commentary and the accompanying article by Carmody and Rajasekaran have addressed concerns about the NBME,1 similar concerns likely apply to the NBME’s partner organization, the FSMB. Next steps should include scrutinizing its practices and any real or apparent conflicts of interest.

References

1. Carmody JB, Rajasekaran SK. On Step 1 mania, USMLE score reporting, and financial conflict of interest at the National Board of Medical Examiners. Acad Med. 2020;95:1332–1337.
2. National Board of Medical Examiners. Engagement and Action: 2018 Annual Report. Philadelphia, PA: National Board of Medical Examiners. https://www.nbme.org/PDF/Publications/2018Annual-Report.pdf. Accessed November 27, 2019.
3. Cause IQ. Cause IQ organization profile: National Board of Medical Examiners. https://www.causeiq.com/organizations/national-board-of-medical-examiners,231352238. Accessed November 27, 2019.
4. Macrotrends. Walmart Profit Margin 2006-2019 | WMT. https://www.macrotrends.net/stocks/charts/WMT/walmart/profit-margins. Accessed December 10, 2019.
5. Charity Navigator. 2016 Charity CEO compensation study. https://d20umu42aunjpx.cloudfront.net/2016+CEO+Comp+Study/2016+CEO+Compensation+Study.pdf. Published October 2016. Accessed November 27, 2019.
6. Association of American Medical Colleges. Education debt manager for matriculating and graduating medical school students. https://store.aamc.org/downloadable/download/sample/sample_id/260. Published 2019. Accessed November 27, 2019.
7. O’Connell B. How much do doctors make in 2018? TheStreet. https://www.thestreet.com/personal-finance/how-much-do-doctors-make-14779617. Published November 13, 2018. Accessed November 27, 2019.
8. Chen DR, Priest KC, Batten JN, Fragoso LE, Reinfeld BI, Laitman BM. Student perspectives on the “Step 1 Climate” in preclinical medical education. Acad Med. 2019;94:302–304.
9. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 Scores in residency selection. Acad Med. 2016;91:12–15.
10. Hamdy H, Prasad K, Anderson MB, et al. BEME systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach. 2006;28:103–116.
11. Wartman SA. The Empirical Challenge of 21st-Century Medical Education. Acad Med. 2019;94:1412–1415.
12. Wormald BW, Schoeman S, Somasunderam A, Penn M. Assessment drives learning: An unavoidable truth? Anat Sci Educ. 2009;2:199–204.
Copyright © 2019 by the Association of American Medical Colleges