Secondary Logo

Journal Logo

Association Between Participation and Performance in MOCA Minute and Actions Against the Medical Licenses of Anesthesiologists

Zhou, Yan PhD*; Sun, Huaping PhD*; Macario, Alex MD, MBA; Keegan, Mark T. MB, BCh; Patterson, Andrew J. MD, PhD§; Minhaj, Mohammed M. MD, MBA; Wang, Ting PhD*; Harman, Ann E. PhD*; Warner, David O. MD

doi: 10.1213/ANE.0000000000004268
Medical Education: Original Clinical Research Report
Free

BACKGROUND: In January 2016, as part of the Maintenance of Certification in Anesthesiology (MOCA) program, the American Board of Anesthesiology launched MOCA Minute, a web-based longitudinal assessment, to supplant the former cognitive examination. We investigated the association between participation and performance in MOCA Minute and disciplinary actions against medical licenses of anesthesiologists.

METHODS: All anesthesiologists with time-limited certificates (ie, certified in 2000 or after) who were required to register for MOCA Minute in 2016 were followed up through December 31, 2016. The incidence of postcertification prejudicial license actions was compared between those who did and did not register and compared between registrants who did and did not meet the MOCA Minute performance standard.

RESULTS: The cumulative incidence of license actions was 1.2% (245/20,006) in anesthesiologists required to register for MOCA Minute. Nonregistration was associated with a higher incidence of license actions (hazard ratio, 2.93 [95% confidence interval {CI}, 2.15–4.00]). For the 18,534 (92.6%) who registered, later registration (after June 30, 2016) was associated with a higher incidence of license actions. In 2016, 16,308 (88.0%) anesthesiologists met the MOCA Minute performance standard. Of those not meeting the standard (n = 2226), most (n = 2093, 94.0%) failed because they did not complete the required 120 questions. Not meeting the standard was associated with a higher incidence of license actions (hazard ratio, 1.92 [95% CI, 1.36–2.72]).

CONCLUSIONS: Both timely participation and meeting performance standard in MOCA Minute are associated with a lower likelihood of being disciplined by a state medical board.

From the *The American Board of Anesthesiology, Raleigh, North Carolina

Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, California

Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, Minnesota

§Department of Anesthesiology, Emory University, Atlanta, Georgia

Department of Anesthesia & Critical Care, The University of Chicago, Chicago, Illinois.

Published ahead of print 5 June 2019.

Accepted for publication May 6, 2019.

Funding: Institutional and/or departmental.

Conflicts of Interest: See Disclosures at the end of the article.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website.

Reprints will not be available from the authors.

Listen to this Article of the Month podcast and more from OpenAnesthesia.org® by visiting http://journals.lww.com/anesthesia-analgesia/pages/default.aspx.

Address correspondence to Huaping Sun, PhD, The American Board of Anesthesiology, 4208 Six Forks Rd, Suite 1500, Raleigh, NC 27609. Address e-mail to huaping.sun@theABA.org.

See Editorial, p 1197

All certificates issued by the 24 Member Boards of the American Board of Medical Specialties (ABMS; Chicago, IL) are now time limited, requiring participation in a Maintenance of Certification (MOC) process. ABMS standards mandate a 4-part framework for MOC based on ABMS/Accreditation Council for Graduate Medical Education core competencies. One of the requirements (part III) is an assessment of knowledge, judgment, and skills using a periodic, secure, high-stakes written examination. The value of this requirement has been questioned. For example, only approximately half of anesthesiologists certified by the American Board of Anesthesiology (ABA; Raleigh, NC) felt that the Maintenance of Certification in Anesthesiology (MOCA) cognitive examination, a secure 200-item written examination for which a passing score was required for every 10-year MOC cycle, is relevant to their practice.1

In January of 2016, the anesthesiology MOC cognitive examination was supplanted by MOCA Minute, a web-based longitudinal assessment comprised of 120 questions annually. A study conducted during development showed that voluntary enrollment and participation in the MOCA Minute pilot program were associated with improved performance on subsequent anesthesiology MOC examinations, suggesting its efficacy in promoting learning.2 Sufficient time has not yet elapsed to determine whether any enhanced learning produced by MOCA Minute translates to better physician performance in clinical practice. Regardless of its potential effects on learning, it is possible that MOCA Minute participation may serve as a marker for characteristics important to physician performance; that is, physician characteristics that motivate participation in the MOCA Minute may be associated with other physician characteristics that determine performance. If such a relationship exists, it would need to be accounted for in future studies of how the MOCA Minute may affect physician learning and performance. We and others have used actions against the medical licenses of physicians by state medical and osteopathic boards as one method to ascertain significant performance deficiencies.3–7 This provides a means to assess the relationship between MOCA Minute experiences and physician performance in terms of a measure that physicians and the public may find meaningful.

The goal of this study was to determine the association between participation and performance in the MOCA Minute program and deficiencies in physician performance as measured by license actions. We tested 2 hypotheses: (1) timely participation in the MOCA Minute program is associated with a lower incidence of license actions and (2) not meeting the MOCA Minute performance standard is associated with a higher incidence of license actions.

Back to Top | Article Outline

METHODS

This study was deemed exempt from review by the Mayo Clinic Institutional Review Board (Rochester, MN).

Back to Top | Article Outline

MOCA Minute Program

Details of the MOCA Minute concept have been previously presented.2 As implemented in January 2016, 120 multiple-choice questions are delivered annually via a mobile app or website. Once a question is accessed, 1 minute is allowed for a response. A subsequent screen provides the correct answer, rationale, a brief critique reviewing the topic, and links to additional resource materials for further studying if desired. In 2016, 90 (75%) questions assessing core knowledge were distributed to all participants, with the remainder based on a participant-specified clinical practice profile. Forty-five (38%) questions were repeated, with the probability of a question being repeated depending in part on whether it was previously answered incorrectly. No >30 questions can be answered in a 24-hour period. In addition, no >30 questions can be answered in any 3-month period, unless participants had not answered all 30 questions in a previous period.

Performance on MOCA Minute was quantified using measurement decision theory (MDT).8 Based on Bayesian theory, this method estimates the probability that physicians possess the knowledge commensurate with that of a board-certified anesthesiologist based on their patterns of question responses. The assumption is that all board-certified physicians initially possess this level of knowledge, but over time, some may not maintain this level. Physicians can thus be classified into groups that do and do not maintain their knowledge. Each response to a question provides additional information regarding whether a physician continues to maintain his or her knowledge. Based on historical pass rates of the anesthesiology MOC examination, we assume a prior probability of .97 that a physician’s knowledge would be maintained. With each question response, this probability is updated based on whether the question is answered correctly, taking into account the difficulty of the question. Correct and incorrect responses increase and decrease the MDT probability, respectively. The MDT approach has several advantages, including providing contemporaneous feedback to each physician regarding their cumulative performance. It also allows for a customized mix of questions to be presented to each physician because the method adjusts for the difficulty of each item, with item difficulties calibrated among the physician pool.

For 2016, the ABA set a MDT probability of .10 as the performance threshold; that is, physicians with a MDT probability <.10 at the end of 2016 were considered to fall into the group not maintaining their knowledge. To meet the MOCA Minute standard for 2016, physicians were required to both complete 120 questions and achieve a MDT probability ≥.10 as of December 31, 2016.

Back to Top | Article Outline

Outcome

The outcome of this study is incidence of prejudicial license actions. A previous report details the method to ascertain incident license action cases.6 Information from all US state medical and osteopathic boards on license actions is collated by the Federation of State Medical Boards, which disseminates this information to the ABMS Member Boards via the Disciplinary Action Notification Service. License actions are classified as nonprejudicial (ie, lifting of probation conditions) or prejudicial (eg, loss or restriction of license). Nonprejudicial actions were not considered because they do not imply censure by the medical boards. A license action incident case was defined as a physician experiencing ≥1 prejudicial license actions. For a physician receiving multiple prejudicial actions, only the first one was considered an incident case. For this study, physicians were at risk for actions from the date of their initial certification to December 31, 2016.

Each license action has ≥1 bases describing why the physician was disciplined. These were classified into 1 of 11 basis categories: substance use, malpractice, fraud, inappropriate prescribing, criminal activity, impairment, license/board violation, records violation, failure to supervise, unprofessional conduct, and other inappropriate activity. The complete list of bases and their classification were described in a previous study.6

Back to Top | Article Outline

Study Design

Each hypothesis was evaluated using a separate set of analyses. Although MDT probability was calculated based on Bayesian theory, these analyses were conducted using a frequentist approach.

Back to Top | Article Outline

Registration for MOCA Minute and License Actions.

The first set of analyses examined the association between participation in the MOCA Minute program and incident license actions. These analyses initially included all anesthesiologists with valid time-limited certificates from the ABA (ie, certified in 2000 or after) who were eligible and required to participate in MOCA Minute (n = 20,112). To be eligible to participate in MOCA Minute, physicians must hold a valid unrestricted medical license. Physicians who were certified in 2000 or after but had their certificates revoked by the ABA (and thus were not eligible to participate in MOCA Minute, n = 12) or were known to have retired (n = 10) or died (n = 7) by December 31, 2016 were not included. The incidence of license actions was first compared between those who registered as required by December 31, 2016 and those who did not. Then, among those who registered for MOCA Minute, those who registered in quarters 1 and 2 of 2016 (“early adopters”) were compared with those who registered in quarters 3 and 4 (“late adopters”). Finally, among those who registered, those who did and did not complete all of the required 120 questions were compared.

Back to Top | Article Outline

Performance on MOCA Minute and License Actions.

The second set of analyses examined the association between performance in the MOCA Minute program and incident license actions. These analyses included all anesthesiologists with time-limited certificates who registered for MOCA Minute in 2016. The incidence of license actions was compared between those who did and did not meet the MOCA Minute standard in 2016, which included answering 120 questions and maintaining a MDT probability ≥.10 as of December 31, 2016.

Back to Top | Article Outline

Statistical Analyses

Anesthesiologists who experienced license actions before the date of their initial certification were excluded from the analyses. The incidence of license actions was compared using survival analysis, with the time to event defined as the number of years elapsed from the date of certification to the time of first prejudicial license action. Physicians without license actions were considered to be right-censored from the analysis on December 31, 2016 (ie, end of follow-up). Group differences in the incidence of license actions were tested using Cox proportional hazards models. In adjusted analyses, sex and medical school country (American medical school graduates [AMGs] versus international medical school graduates [IMGs]) were considered a priori as covariates for inclusion, based on the previous study finding that they could be associated with license actions.6 Cumulative incidence was visualized using Kaplan–Meier curves, with the x-axis representing number of years since initial certification and the y-axis representing cumulative proportion of individuals who have not had a license action.

Physicians with substance use disorder are more likely to experience license actions, which may or may not be associated with recognized performance deficits.9 To assess if the results were sensitive to the inclusion of physicians with this condition, each set of analyses was repeated examining only license action cases not related to substance use disorder, with incident cases related to the disorder treated as noncases.

To assess the potential impact of unmeasured confounding variables, a sensitivity analysis was conducted by simulating such a variable (eg, whether a physician was clinically inactive) and comparing the estimated effect (ie, odds ratio) in the logistic regression with and without the adjustment for this confounder.10 Simulations were conducted by assuming a confounding variable prevalence rate of 5%, 10%, or 20%. The strength of association between the confounder and either not registering for MOCA Minute or not meeting MOCA Minute standard, expressed as an odds ratio, was varied over a range from 1.5 to 4.0 (eg, clinical inactivity increases the likelihood of not registering for MOCA Minute or not meeting MOCA Minute standard). Similarly, the strength of association between the confounder and license actions was also varied over a range of odds ratios from 1.5 to 4.0. This odds ratio was assumed to be the same regardless of MOCA Minute participation or performance status. To account for the randomness of the simulation process, each simulation was repeated 20 times and the average estimate was calculated. Although our main analysis used survival analysis with hazard ratio (HR) as the estimated effect, a sensitivity analysis with odds ratio should give a comparable evaluation of impact from unmeasured confounders.

This study is based on population data, thus no a priori statistical power calculation was done to determine the sample size. A P value of <.05 was considered to indicate statistical significance. All statistical analyses were performed in R version 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria).

Back to Top | Article Outline

RESULTS

Registration for MOCA Minute and License Actions

Of the 20,112 anesthesiologists with valid time-limited certificates who were required to participate in the MOCA Minute, 106 had license actions that occurred before the date of their initial certification and were excluded. Of the remaining 20,006 anesthesiologists, 18,534 (92.6%) registered for the MOCA Minute by December 31, 2016 and 1472 (7.4%) did not. There were 245 incident license action cases among these anesthesiologists: 193 (1.0%) among anesthesiologists who registered and 52 (3.5%) among those who did not (Table 1). The most common basis code for license actions was malpractice, followed by unprofessional conduct and license/board violations (Table 2). In unadjusted analysis, the incidence of license actions was significantly higher in those who did not register compared with those who did (HR, 3.00 [95% confidence interval {CI}, 2.21–4.08]; Table 1; Figure). Results were similar in the analysis that adjusted for sex and medical school country (AMG versus IMG) (HR, 2.93 [95% CI, 2.15–4.00]; Table 1). In the sensitivity analysis (Supplemental Digital Content, Table 1, http://links.lww.com/AA/C850), this association was robust over the ranges of simulated characteristics for the unmeasured confounder (prevalence, association with registration for MOCA Minute, and association with license actions), with the 95% CIs for each estimate of the adjusted odds ratio excluding one.

Table 1.

Table 1.

Table 2.

Table 2.

Figure.

Figure.

Further analyses explored the role of participation patterns among those physicians who registered for the MOCA Minute. Compared with the “early adopters” (those who registered in the first 2 quarters of 2016), the “late adopters” (those who registered in the last 2 quarters of 2016) had a higher incidence of license actions in both unadjusted and adjusted analyses (HR, 1.73 [95% CI, 1.03–2.88] for adjusted analysis; Table 1). Compared with those physicians who completed all 120 questions as required in 2016, those who completed <120 questions had a higher incidence of license actions in both unadjusted and adjusted analysis (HR, 1.87 [95% CI, 1.30–2.68] for adjusted analysis; Table 1).

Table 3.

Table 3.

Results of all analyses were similar when physicians with substance use disorder were treated as noncases (Table 3), except that the difference between “early adopters” and “late adopters” was no longer statistically significant.

Back to Top | Article Outline

Performance on MOCA Minute and License Actions

Among the 18,534 physicians who registered for the MOCA Minute by December 31, 2016, 16,308 (88.0%) met the standard and 2226 (12.0%) did not. There were 152 (0.9%) incident license action cases among those who met the standard and 41 (1.8%) cases among those who did not. In unadjusted analysis, the incidence of license actions was significantly higher in those who did not meet the standard compared with those who did (HR, 1.93 [95% CI, 1.36–2.72]; P < .001; Table 1). Results were similar in analysis that adjusted for the effects of sex and medical school country (HR, 1.92 [95% CI, 1.36–2.72]; P <.001; Table 1; Figure). In the sensitivity analysis (Supplemental Digital Content, Table 2, http://links.lww.com/AA/C850), this association was robust over the ranges of simulated characteristics for the unmeasured confounder (prevalence, association with meeting MOCA Minute standard, and association with license actions), with the 95% CIs for each estimate of the adjusted odds ratio excluding one for most scenarios.

Physicians could fail to meet the standard either by not completing 120 questions as required or by achieving a MDT probability <.10. Most who failed to meet the standard (n = 2226) did so because they did not complete all 120 questions (n = 2093, 94.0%). Among those who completed 120 questions (n = 16,441), 133 (0.8%) physicians achieved a MDT probability <.10, and 4 (3.0%) of them had a license action; of the 16,308 physicians achieving MDT probabilities ≥.10, 152 (0.9%) had a license action (HR, 2.28 [95% CI, 0.84–6.17]; P = .10, unadjusted).

Results of all analyses were similar when physicians with substance use disorder were treated as noncases (Table 3).

Back to Top | Article Outline

DISCUSSION

The major findings of this study were that for board-certified anesthesiologists, both timely participation in the MOCA Minute program and meeting the MOCA Minute standard were associated with a modestly lower incidence of license actions by state medical and osteopathic boards.

Previous work showed that initial certification is associated with several measures of physician performance, including patient outcomes,11–16 quality measures,12,14 and license actions.3–5 Our previous work found that anesthesiologists who could not achieve initial certification by the ABA were approximately 3-fold more likely to experience a license action.6 In contrast, the evidence is more limited regarding how physician participation and performance in MOC is associated with physician performance.

Current ABMS standards mandate a 4-part framework for MOC: professionalism and professional standing (part I); lifelong learning and self-assessment (part II); assessment of knowledge, judgment, and skills (part III); and improvement in medical practice (part IV).17 Meeting MOC requirements requires participation in each element. In addition, part III requires a summative examination, typically using multiple-choice questions.17 Several studies have found a relationship between scores on part III examinations and measures of performance,18–20 including a study of anesthesiologists which found that a lower score on the part III anesthesiology MOC examination, a secure 200-item multiple-choice question assessment that diplomates were required to pass every 10 years, was associated with a history of license actions.21 However, only 2 previous studies examined the relationship between “passing” a part III examination and physician performance. A study of anesthesiologists participating in MOC before the introduction of the MOCA Minute found that not meeting MOC requirements in a timely fashion was associated with an increased risk of license actions.22 In this study, most physicians not meeting requirement had deficiencies in multiple elements of MOC; achieving a passing score on the anesthesiology MOC examination alone had little value in predicting license actions. A study of general internists found that for those certified by the American Board of Internal Medicine and taking their MOC examination within 10 years after initial certification, a passing score was associated with a lower risk of subsequent license actions.23 They did not report whether those who failed were also deficient in other MOC elements. Of note, those taking and passing their examinations after 10 years had approximately the same frequency of actions as those who failed the examination within 10 years, suggesting that timely participation was also an important determinant of risk.

In the present study, measures of participation in the MOCA Minute program were associated with license actions, including some evidence for a relationship between the intensity of participation and this outcome (eg, “early” versus “late” adopters and those who completed all 120 questions versus those who did not). In addition to willingness or interest, there are other reasons that might explain varied levels of participation. Anesthesiologists could voluntarily relinquish certification (eg, retirement, leaving clinical practice). Significant life events such as major illness, family distress, or military deployment could also affect the ability to participate. The circumstances that lead physicians to experience adverse license actions may affect their motivation or ability to participate. Although the ABA collects information on self-reported practice area at the time of MOCA registration, for those who did not register, it is not known whether or not they were actively practicing. Regardless of reason, the association between participation in the MOCA Minute program and physician performance does not reflect a causal effect, because the behaviors prompting license actions most likely occurred before participating in the program. Rather, participation serves as a marker for other physician characteristics associated with a lower risk for license actions. This may represent a confounding factor for future studies attempting to examine the effectiveness of MOC to improve physician performance.

Among those physicians who answered 120 questions, the HR for the incidence of license actions in physicians who did not and did meet the MDT probability threshold of ≥.10 was positive but not statistically significant, likely due to the small number of physicians who fell below the MDT probability threshold and the small number of license actions in this group. Rather, far more physicians experiencing license actions were identified according to participation status rather than performance. There were 93 incident license actions among those physicians who either did not register or did not meet the standard (answering 120 questions and maintaining a MDT probability of ≥.10). Of these, only 4 (4.3%) were attributed to physicians who failed to meet the standard because their response patterns of 120 questions resulted in a MDT probability <.10. Thus, although performance on MOCA Minute may provide some additional information, the criteria of non- or incomplete participation, not the pattern of responses among those who do participate, identified most physicians who experienced license actions among those not meeting the standard.

Conversely, even though MOCA Minute participation and performance were negatively associated with incidence of license actions, the majority of license action cases in this population (152 of 245 cases [62%]) occurred in physicians who met the standard. Thus, meeting this standard is not a guarantee that a physician has been or will be immune from a license action, but rather an indicator of a lower probability of having experienced a license action. In terms of positive and negative predictive values, if not meeting the standard is considered a positive “screening test,” and license actions are considered the condition being sought, overall, the positive predictive value of not meeting the standard is 2.5% (ie, this proportion of physicians who either did not register or meet the standard had experienced license actions) and the negative predictive value of meeting the standard is 99.1% (ie, this proportion of physicians meeting the standard had not experienced license actions).

This work has several limitations in addition to those already mentioned. As discussed in our previous work, there are both advantages and disadvantages of using license actions as an indicator of physician performance.6 Although the Disciplinary Action Notification Service system provides a comprehensive means to ascertain license actions among US physicians, there can be variations among state medical and osteopathic boards with regard to the standards for disciplining physicians and how the standards are implemented.24,25 In addition, this method assesses only performance issues of sufficient severity to warrant medical licensing board actions and does not capture other potentially relevant aspects of physician performance. Another important limitation of this analysis is that there were relatively few events, despite the large cohort size, affecting the ability to detect more modest associations. Also, unmeasured confounding variables could contribute to observed associations (eg, anesthesiologists who are not clinically active may be less likely to register for MOCA Minute). However, our sensitivity analysis with a simulated unmeasured confounder showed that the associations were robust. Finally, physicians with substance use disorder are more likely to experience a variety of adverse outcomes, including license actions.9 However, the pattern of results in the current analysis was not sensitive to the inclusion of these physicians, suggesting that the performance issues identified by the other basis codes are also associated with the risk of license actions.

In conclusion, both timely participation in the MOCA Minute program and meeting the MOCA Minute standard were associated with a lower incidence of prejudicial actions against the medical licenses of anesthesiologists, suggesting that these attributes serve as markers for physician characteristics associated with a lower frequency of such actions. Future research will be needed to determine if participation in MOC activities such as MOCA Minute can actually improve physician performance. The association between participation and performance may represent a significant confounding factor in such research.

Back to Top | Article Outline

DISCLOSURES

Name: Yan Zhou, PhD.

Contribution: This author helped design the study; manage, analyze, and interpret the data; and draft the manuscript.

Conflicts of Interest: Y. Zhou is a staff member of the American Board of Anesthesiology.

Name: Huaping Sun, PhD.

Contribution: This author helped design the study; manage, analyze, and interpret the data; and draft the manuscript.

Conflicts of Interest: H. Sun is a staff member of the American Board of Anesthesiology.

Name: Alex Macario, MD, MBA.

Contribution: This author helped design the study; manage, analyze, and interpret the data; and draft the manuscript.

Conflicts of Interest: A. Macario serves as a Director for the American Board of Anesthesiology.

Name: Mark T. Keegan, MB, BCh.

Contribution: This author helped design the study, interpret the data, and draft the manuscript.

Conflicts of Interest: M. T. Keegan serves as a Director for the American Board of Anesthesiology.

Name: Andrew J. Patterson, MD, PhD.

Contribution: This author helped design the study, interpret the data, and draft the manuscript.

Conflicts of Interest: A. J. Patterson serves as a Director for the American Board of Anesthesiology.

Name: Mohammed M. Minhaj, MD, MBA.

Contribution: This author helped design the study, interpret the data, and draft the manuscript.

Conflicts of Interest: M. M. Minhaj is a nondirector member of the American Board of Anesthesiology Research Committee.

Name: Ting Wang, PhD.

Contribution: This author helped design the study, interpret the data, and draft the manuscript.

Conflicts of Interest: T. Wang is a staff member of the American Board of Anesthesiology.

Name: Ann E. Harman, PhD.

Contribution: This author helped design the study, interpret the data, and draft the manuscript.

Conflicts of Interest: A. E. Harman is a staff member of the American Board of Anesthesiology.

Name: David O. Warner, MD.

Contribution: This author helped design the study; manage, analyze, and interpret the data; and draft the manuscript.

Conflicts of Interest: D. O. Warner serves as a Director for the American Board of Anesthesiology.

This manuscript was handled by: Edward C. Nemergut, MD.

Back to Top | Article Outline

REFERENCES

1. Culley DJ, Sun H, Harman AE, Warner DO. Perceived value of board certification and the Maintenance of Certification in Anesthesiology Program (MOCA®). J Clin Anesth. 2013;25:12–19.
2. Sun H, Zhou Y, Culley DJ, Lien CA, Harman AE, Warner DO. Association between participation in an intensive longitudinal assessment program and performance on a cognitive examination in the Maintenance of Certification in Anesthesiology Program®. Anesthesiology. 2016;125:1046–1055.
3. Kohatsu ND, Gould D, Ross LK, Fox PJ. Characteristics associated with physician discipline: a case-control study. Arch Intern Med. 2004;164:653–658.
4. Lipner RS, Young A, Chaudhry HJ, Duhigg LM, Papadakis MA. Specialty certification status, performance ratings, and disciplinary actions of internal medicine residents. Acad Med. 2016;91:376–381.
5. Morrison J, Wickersham P. Physicians disciplined by a state medical board. JAMA. 1998;279:1889–1893.
6. Zhou Y, Sun H, Culley DJ, Young A, Harman AE, Warner DO. Effectiveness of written and oral specialty certification examinations to predict actions against the medical licenses of anesthesiologists. Anesthesiology. 2017;126:1171–1179.
7. Jones AT, Kopp JP, Malangoni MA. Association between maintaining certification in general surgery and loss-of-license actions. JAMA. 2018;320:1195–1196.
8. Rudner LM. Scoring and classifying examinees using measurement decision theory. Practical Assessment, Research & Evaluation. 2009;14. Available at: https://pareonline.net/getvn.asp?v=14&n=8. Accessed November 20, 2018.
9. Warner DO, Berge K, Sun H, Harman A, Hanson A, Schroeder DR. Risk and outcomes of substance use disorder among anesthesiology residents: a matched cohort analysis. Anesthesiology. 2015;123:929–936.
10. Groenwold RH, Nelson DB, Nichol KL, Hoes AW, Hak E. Sensitivity analyses to estimate the potential impact of unmeasured confounding in causal research. Int J Epidemiol. 2010;39:107–117.
11. Chen J, Rathore SS, Wang Y, Radford MJ, Krumholz HM. Physician board certification and the care and outcomes of elderly patients with acute myocardial infarction. J Gen Intern Med. 2006;21:238–244.
12. Haas JS, Orav EJ, Goldman L. The relationship between physicians’ qualifications and experience and the adequacy of prenatal care and low birthweight. Am J Public Health. 1995;85:1087–1091.
13. Norcini JJ, Lipner RS, Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction. Med Educ. 2002;36:853–859.
14. Pearce WH, Parker MA, Feinglass J, Ujiki M, Manheim LM. The importance of surgeon volume and training in outcomes for vascular surgical procedures. J Vasc Surg. 1999;29:768–776.
15. Prystowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon’s training, certification, and experience. Surgery. 2002;132:663–670.
16. Silber JH, Kennedy SK, Even-Shoshan O, et al. Anesthesiologist board certification and patient outcomes. Anesthesiology. 2002;96:1044–1052.
17. American Board of Medical Specialties. Standards for the ABMS Program for Maintenance of Certification (MOC). Available at: http://www.abms.org/media/1109/standards-for-the-abms-program-for-moc-final.pdf. Accessed November 20, 2018.
18. Hess BJ, Weng W, Lynn LA, Holmboe ES, Lipner RS. Setting a fair performance standard for physicians’ quality of patient care. J Gen Intern Med. 2011;26:467–473.
19. Hess BJ, Weng W, Holmboe ES, Lipner RS. The association between physicians’ cognitive skills and quality of diabetes care. Acad Med. 2012;87:157–163.
20. Holmboe ES, Wang Y, Meehan TP, et al. Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries. Arch Intern Med. 2008;168:1396–1403.
21. Sun H, Culley DJ, Lien CA, Kitchener DL, Harman AE, Warner DO. Predictors of performance on the Maintenance of Certification in Anesthesiology Program® (MOCA®) examination. J Clin Anesth. 2015;27:1–6.
22. Zhou Y, Sun H, Macario A, et al. Association between performance in a maintenance of certification program and disciplinary actions against the medical licenses of anesthesiologists. Anesthesiology. 2018;129:812–820.
23. McDonald FS, Duhigg LM, Arnold GK, Hafer RM, Lipner RS. The American Board of Internal Medicine Maintenance of Certification examination and state medical board disciplinary actions: a population cohort study. J Gen Intern Med. 2018;33:1292–1298.
24. Federation of State Medical BoardsSummary of 2007 Board Actions. Available at: http://www.patientsafetyasap.org/pdf/2007_SummaryMedBoardActions.pdf. Accessed January 23, 2019.
25. Law MT, Hansen ZK. Medical licensing board characteristics and physician discipline: an empirical analysis. J Health Polit Policy Law. 2010;35:63–93.

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2019 International Anesthesia Research Society