Share this article on:

Using Machine Learning to Assess Physician Competence: A Systematic Review

Dias, Roger D., MD, MBA, PhD; Gupta, Avni, BDS, MPH; Yule, Steven J., PhD

doi: 10.1097/ACM.0000000000002414
Review: PDF Only

Purpose: To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different competence domains in several medical specialties.

Method: In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students’, residents’, fellows’, or attending physicians’ competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted.

Results: Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24, 34.8%) and radiology (15, 21.7%) were the most studied specialties; natural language processing (24, 34.8%), support vector machine (15, 21.7%), and hidden Markov models (14, 20.3%) were the ML techniques most often applied; and patient care (63, 91.3%) and medical knowledge (45, 65.2%) were the most assessed competence domains.

Conclusions: A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.

R.D. Dias is instructor in emergency medicine, Department of Emergency Medicine and STRATUS Center for Medical Simulation, Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts; ORCID:

A. Gupta is project manager, Center for Surgery and Public Health, Brigham and Women’s Hospital, Boston, Massachusetts.

S.J. Yule is associate professor of surgery, Harvard Medical School, and faculty, Department of Surgery and STRATUS Center for Medical Simulation, Brigham and Women’s Hospital, Boston, Massachusetts.

Supplemental digital content for this article is available at and

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Correspondence should be addressed to Steven J. Yule, STRATUS Center for Medical Simulation, Brigham and Women’s Hospital, 75 Francis St., Boston, MA 02115; telephone: (617) 525-9588; e-mail:

© 2018 by the Association of American Medical Colleges