We completely agree with the author's argument that the American Board of Surgery's long-standing practice of using hotel rooms to deliver high stakes oral examinations is problematic. Furthermore, we can agree that the ideal location is consistent, neutral, secure, and safe. We wholeheartedly disagree with the assertion that the ABS hides behind arguments such as “every other board-certified surgeon has gone through this process before”, or “the examination is a rite of passage,” which may be a reflection of the author's own implicit bias about the surgeons who develop and deliver the examination. Improving the quality, reliability, relevance, and fairness of our examinations has been a major focus of the ABS for the last decade. The challenge of delivering examinations during the pandemic set the stage for a dramatic increase in rate of change.
We have focused on potential bias in the construction and delivery of examinations, from examination content to examiner pool. To mitigate this bias, we have standardized the examination scenarios as well as the rosters of scenarios given to candidates, thus eliminating the possibility that a candidate might receive a set of scenarios that were significantly more difficult or esoteric than others. An important way to mitigate against bias is to have a highly diverse examiner pool. In 2018, the ABS moved from the practice of examiners being invited to examine by individual directors, to a vetted, trained, and committed volunteer pool. The pool, designed to have examiners across specialty, geography, practice type (academic, community, and others), sex, race, and ethnicity, represents the most diverse group of volunteers the ABS has ever had. The author's assertion that the examination process has failed to reflect the demographics of the doctors taking the examination is inaccurate. The associate examiner pool was designed to reflect the demographics of the chief resident class.
In 2019, the board voted to collect additional demographics including race and ethnicity, which will allow us to study these variables and their relationship to examination performance. Before 2019, sex was the only demographic characteristic collected by the ABS. We looked at our general surgery certifying examination for evidence of sex bias and found none.1 Individual examiner scoring patterns also undergo analysis looking for sex bias and the (rare) outliers are counseled. The authors assert that the examination format is sex biased is not born out by research.
The ABS started the process of moving out of hotel bedrooms to all-suite hotels in 2010. Our last examination in a hotel bedroom was in 2017. Since then we have exclusively used the living room of suites, sharing the concern that the author expresses about being within eyeshot of a bed during a high-stakes professional assessment. We have explored renting space from other boards with examination centers, using convention center settings, or even leasing and developing our own examination center. All these solutions had major drawbacks, and ultimately were found to be either impossible or to add prohibitive expense and risk to a process that was already in evolution. The pandemic allowed or forced the leap to virtual examinations. We were the first of the 24 ABMS member boards to do so and have served as a resource to other boards that are developing their own virtual examination delivery.2
We agree that travelling to an examination, with the associated expenses and the possible stress of being away from home is an issue for candidates. Post-examination surveys after our virtual oral examinations revealed a very high degree of satisfaction with the virtual model, and candidates specifically identified not having to travel and the lower cost as advantages. Examiners, however, preferred the in-person examination on surveys. They cite the value of in-person team discussions about roster delivery, the subtleties of in-person communication, and the ability to work closely with seasoned examiners both during and after the examination. Very few examiners identified travel or being away from home as an issue. Three years ago, when the ABS first instituted the associate examiner pool, we put out a call for volunteers to become examiners. We made it clear that they would need to spend 10 days/ year away from home. Twenty-one hundred hopefuls filled out the application, for a pool of 130 new examiners. Although we acknowledge that even more surgeons might be willing to examine if we take travel out of the equation, access to a highly diverse group of examiners has not been an issue.
In summary, we agree on the important substance of the author's perspective. Aspects of the examination that might impact outcome but are unrelated to knowledge, judgment, and communication skills should be eliminated to the full extent possible. Our mission statement states “the American Board of Surgery serves the public and the specialty of surgery by providing leadership in surgical education and practice, by promoting excellence through rigorous evaluation and examination, and by promoting the highest standards for professionalism, lifelong learning, and the continuous certification of surgeons in practice.” We are committed to fulfilling the mission by an ongoing process of evaluating ourselves, to ensure fair and equitable delivery of our exams and to uphold the public trust.
1. Ong TQ, Kopp JP, Jones AT, et al. Is there gender bias on the American Board of Surgery General Surgery Certifying Examination? J Surg Res
2. Jones AT, Barry CL, Ibáñez B, et al. The development of a virtual pilot for the American Board of Surgery Certifying examination. Am J Surg
2021; S0002-9610(21)00067-2. doi: 10.1016/j.amjsurg.2021.01.040. Epub ahead of print. PMID: 33563463.