An Investigation of the Relationship Between COMLEX-USA Licensure Examination Performance and State Licensing Board Disciplinary Actions

Abstract

Purpose 

Passing the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) serves as a licensing requirement, yet there is limited understanding between this high-stakes exam and performance outcomes. This study examined the relationship between COMLEX-USA scores and disciplinary actions received by osteopathic physicians.

Method 

Data for osteopathic physicians (N = 26,383) who graduated from medical school between 2004 and 2013 were analyzed using multinomial logistic regression to assess the relationship between COMLEX-USA scores and placement into one of 3 disciplinary action categories relative to no action received, controlling for years in practice and gender.

Results 

Less than 1% of physicians in this study (n = 187) had a disciplinary action(s). Controlling for all COMLEX-USA levels, years in practice, and gender, higher Level 3 scores were associated with significant decreased odds for all action categories: revoked licensed (odds ratio [OR] = 0.51, 95% confidence interval [CI] 0.36, 0.72; P < .001), imposed limitations to practice (OR = 0.59, 95% CI 0.41, 0.84; P < .01), and other action imposed (OR = 0.48, 95% CI 0.33, 0.69; P < .001), relative to not receiving an action. In these same models, higher Level 2 Performance Evaluation Biomedical/Biomechanical Domain scores decreased the odds for an action that revoked a license (OR = 0.75, 95% CI 0.58, 0.98; P < .05) and imposed limitations to practice (OR = 0.64, 95% CI 0.49, 0.84; P < .001).

Conclusions 

These findings provide evidence that the COMLEX-USA delivers useful information regarding the likelihood of a practitioner receiving state board disciplinary actions.

Physicians who obtain a medical license in the United States move sequentially through medical education and a licensure process designed to assure that all who qualify demonstrate the minimum level of competency for general undifferentiated medical practice. For physicians to become licensed, they must meet minimum requirements set by the state licensing board within the state where they intend to practice. All jurisdictions in the United States require U.S. and internationally medical trained graduates to obtain a medical school degree, successfully complete a multilevel licensing examination process, and complete at least 1 year of postgraduate training. Once these requirements are met, the individual is eligible to receive a full and unrestricted medical license based on each state’s licensing regulations.¹

In addition to granting medical licensure, state licensing boards investigate complaints filed against physicians ranging from substandard quality of care or disruptive behavior to impairment or unprofessional conduct. The state licensing board serving the jurisdiction where the physician practices has the authority to impose disciplinary actions (e.g., require additional education; impose a fine; in the severest cases, suspend or revoke the physician’s license) when concerns arise around physician performance or behavior. Once a complaint against a physician is received, the process typically involves several steps before the board determines if and what type of disciplinary board action(s) is imposed. Board orders include details about disciplinary actions that are sent to and recorded by the Federation of State Medical Boards (FSMB).¹

For osteopathic physicians, all state licensing boards accept the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) for initial licensure.¹ All osteopathic candidates who graduated after December 1, 2007, must pass the Level 1 Exam, Level 2-Cognitive Exam (CE), and Level 2-Performance Exam (PE) Biomedical/Biomechanical Domain (BD) and the Humanistic Domain (HD) of the COMLEX-USA before graduation. Level 3 is typically taken after the first or second year of postgraduate training. Seven osteopathic medical competencies in knowledge and skills related to patient care and clinical presentations common to osteopathic physicians are assessed by the COMLEX-USA (see www.nbome.org for additional details). Candidates sequentially take the COMLEX-USA Level 1, Level 2-CE, Level 2-PE, and Level 3 exams, where each component of the series must be passed before moving to the next level. Level 2-CE and Level 2-PE can be taken in either order. The National Board of Osteopathic Medical Examiners (NBOME) proposes the validity argument that those who pass the COMLEX-USA have met minimal standards of competence required to receive an undifferentiated license to practice medicine. This validity argument is predicated on the assumption that performance at each stage across the COMLEX-USA series indicates an osteopathic physician’s readiness for practice.

It is important for test agencies to collect empirical evidence to support inferences on what a licensing test is purported to measure.^2–4 Review of the current literature shows important relationships between performance on high-stakes licensing and credentialing tests and later behavioral outcomes related to disciplinary actions directly or indirectly linked to patient care and safety (e.g., drug or alcohol dependence, unethical practice, etc.). A recent study of internal medicine physicians showed state board disciplinary actions were highest and more severe for noncertified internal medicine physicians than for those who had passed a certification exam.⁵ Another study showed a statistically significant inverse relationship between the United States Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) scores and receiving state board disciplinary actions, where there was approximately a 25% lower odds of receiving a state disciplinary action with one standard deviation unit increase in Step 2 CK scores.⁶ In a prospective cohort study, family physicians who passed the Québec Family Medicine Certification Examination were followed up to 7 years; findings showed that certification scores taken at the end of medical school provided a sustained relationship with preventive care and acute patient care in primary practice over 4 to 7 years.⁷ An inverse relationship between the USMLE Step 2 CK scores and in-hospital mortality rates for international medical graduates has been reported.⁸ Evidence is provided that the COMLEX-USA Level 2-CE is an early indicator of future performance on the American Board of Family Medicine’s Maintenance of Certification for Family Physicians (MC-FP) examination, where of all persons who passed the Level 2-CE exam on their first attempt, 90% also passed the MC-FP examination on their first attempt.⁹ Review of the literature provides evidence supporting the validity of licensing and credentialing exams as an indicator of later behavior among practicing physicians.

The intent of this study was to investigate the relationship between COMLEX-USA Level 1, Level 2-CE, Level 2-PE, and Level 3 scores from undergraduate and graduate medical education with examinees subsequent disciplinary actions. Our central research question was: what is the relationship between COMLEX-USA Level 1, Level 2-CE, Level 2-PE, and Level 3 scores and state licensing board disciplinary actions of licensed osteopathic physicians, controlling for physicians’ time in practice and gender?

Method

Data

We obtained retrospective data for physicians who graduated from an osteopathic medical college between years 2004 and 2013 and had taken the full COMLEX-USA series from the NBOME and FSMB. This time frame was selected because it includes when the COMLEX Level 2-PE first began, and it also seemed like a reasonable period of time for this cohort of physicians to be issued a medical license and be in practice. Records within this time frame included 35,737 osteopathic physicians. The FSMB identified 26,383 (73.8%) of these osteopathic physicians who were issued a state board medical license and 187 of these physicians who received a state medical board disciplinary action(s) by March 2018. This study was reviewed by the Center for the Advancement of Healthcare Education and Delivery Institutional Review Board and qualified for exemption because of minimal to no risk to study participants.

Variables

It is possible for a physician to receive multiple disciplinary actions associated with single or multiple incidents from one or multiple state licensing boards. The 2016 FSMB census reported 21.7% of physicians to hold 2 or more active medical licenses.¹⁰ Within the time frame of this study, there were 187 osteopathic physicians issued one or more disciplinary actions. Using methodology similar to that used by Lipner et al,⁵ we retained the most severe action, which we classified into one of 3 groups: license revocation (i.e., license suspended, surrendered, or revoked), imposed limitations to practice (i.e., license denied, license restricted, probation, or conditions imposed), or other action imposed (i.e., fine, reprimand, continuing medical education required, or administrative/other action imposed). The board action categories for these 187 physicians are shown in Table 1. Physicians with board actions were compared relative to those who had not received a disciplinary action.

Table 1: Frequency and Percentage of First Most Severe Disciplinary Action Received by 187 of 26,383 Physicians Who Graduated From an Osteopathic Medical School 2004–2013, From a Study of the Relationship Between Licensure Examination Performance and State Licensing Board Disciplinary Action

We used COMLEX-USA Level 1, Level 2-CE, Level 2-PE, and Level 3 first-attempt scores as measures of performance. Our rationale for using physicians’ first-attempt scores is that this measure provides a common baseline for advancement to medical licensure. COMLEX-USA Level 1 and Level 2-CE are computer-based examinations taken during supervised practice in undergraduate medical education and are designed to assess the application of osteopathic medical knowledge related to foundational biomedical and clinical sciences, patient presentations, and physician tasks. Level 3 is a computer-based examination typically taken during the first or second year of residency, designed to assess skills related to the application of clinical sciences, patient safety, foundational competency domains, and clinical presentations. For these computer-based examinations, the number of items answered correctly are summed and transformed to a 3-digit standard score with a mean typically being in the 500–550 range.

Level 2-PE is a standardized patient-based performance examination taken in the third or fourth year of undergraduate medical education, designed to assess the fundamental clinical skills essential for osteopathic patient care.¹¹ The Level 2-PE is a conjunctive exam that falls into 2 domains: the Humanistic Domain (HD) assesses the doctor–patient communication skills, interpersonal skills, and professionalism of the examinee; the BD assesses ability to gather pertinent patient information, develop a differential diagnosis and plan, and synthesize as well as document that information. The Level 2-PE HD includes communication and interpersonal clinical skills assessed on six 9-point scales that are calculated as the average HD score. The Level 2-PE BD includes 3 clinical skill measures (checklist, patient note, and osteopathic manipulation) that are combined into a percent scale and calculated as the weighted BD score.

A physician’s experience in practice has been shown to be a likely indicator of disciplinary action.¹² To establish a common baseline, we retained the first jurisdiction issuing a medical license to a physician to calculate time in practice to a disciplinary action issued. For physicians who had not been issued a disciplinary action, time in practice was calculated from their medical license issue date to the time frame of this study. Gender, coded as 1 = female and 0 = male, was included because previous research has shown that the likelihood of receiving a board disciplinary action differs between male and female physicians.^5,6,12–15

Analyses

We analyzed data for this retrospective cohort study using multinomial logistic regression (MLR) statistical procedures. MLR is a direct extension of binary logistic regression when the outcome has more than 2 nominal categories.¹⁶ For a baseline approach, one of the outcome categories is chosen as the baseline or reference group to compare with each of the other outcome categories. Specifically, in this study, M represented the 3 disciplinary action categories, then M - 1 dichotomous dependent variables were coded as: license revocation = 1 versus no disciplinary action = 0; imposed limitations to practice = 1 versus no disciplinary action = 0; and other action imposed = 1 versus no disciplinary action = 0. Physicians who had not received a disciplinary action were selected as the baseline group to calculate the odds of an individual being in one of the groups that received a disciplinary action. Using MLR simultaneously estimates M - 1 separate binary logistic regressions models for each of the disciplinary action categories relative to the reference category. Each M - 1 regression model has its own intercept and predictor coefficients. When the outcome has more than one category, using MLR simplifies to not having to compute separate binary logistic regressions. Full discussion of MLR is provided by several resources.^17–19 We analyzed a generalized logit model using SAS statistical software, version 9.4 (SAS Institute Inc., Cary, North Carolina), using the logistic procedure.

Results

Characteristics of the 35,737 osteopathic physicians who were in our dataset and provided demographic information include the following: 46.5% (15,221) female, 77.5% (21,372) white, 16.0% (4,405) Asian, 3.4% (939) African American, 0.4% (99) Hispanic, 2.4% (765) other racial groups, 6.3% (1,194) who identified themselves with Hispanic origin, and 96.1% (18,327) who spoke English as their primary language. The FSMB identified 26,383 (73.8%) of these osteopathic physicians who were issued a state board medical license and 187 (< 1%) of these physicians who received a state medical board disciplinary action(s) by March 2018. COMLEX-USA score statistics by state licensing board disciplinary action groups are shown in Table 2. Collinearity diagnostics showed collinearity was no longer indicated after rescaling COMLEX-USA exam scores and physicians’ years in practice to each have a mean equal to 0 and standard deviation equal to 1. Means, standard deviations, and paired correlations of this transformation are shown in Table 3. The MLR procedure in SAS automatically checks and reports if separation is detected. For our model, no separation was detected.

Table 2: Predictors for State Licensing Board Disciplinary Action Category for Physicians Who Graduated From an Osteopathic Medical School 2004–2013, From a Study of the Relationship Between Licensure Examination Performance and State Licensing Board Disciplinary Action

Table 3: Correlations, Means, and SDs of Transformed Variables for Physicians Who Graduated From an Osteopathic Medical School 2004–2013, From a Study of the Relationship Between Licensure Examination Performance and State Licensing Board Disciplinary Action^a

MLR odds ratios (ORs) are shown in Table 4. Relative to osteopathic physicians not issued a disciplinary action, the odds of being issued a license-revoked board action for physicians having taken the COMLEX-USA series of examinations resulted in a nonsignificant relationship with Level 1 scores (OR = 1.33; 95% confidence interval [CI] 0.91 1.95; P = .142), Level 2-CE scores (OR = 0.94, 95% CI 0.63, 1.41; P = .76), and Level 2-PE HD scores (OR = 0.95, 95% CI 0.74, 1.23; P = .71). There was a significant relationship with Level 2-PE BD scores (OR = 0.75, 95% CI 0.58, 0.98; P = .03) and Level 3 scores (OR = 0.51, 95% CI 0.36, 0.72; P < .001), while controlling for years in practice and gender.

Table 4: Multinomial Logistic Regression Odds Ratios for Physicians Who Graduated From an Osteopathic Medical School 2004–2013, From a Study of the Relationship Between Licensure Examination Performance and State Licensing Board Disciplinary Action

Relative to osteopathic physicians not issued a disciplinary action, the odds of being issued an imposed limitations to practice board action for physicians having taken the COMLEX-USA series of examinations resulted in a nonsignificant relationship with Level 1 scores (OR = 1.27, 95% CI 0.85, 1.87; P = .24), Level 2-CE scores (OR = 1.17, 95% CI 0.77, 1.77; P = .47), and Level 2-PE HD scores (OR = 1.13, 95% CI 0.86, 1.47; P = .38). There was a significant relationship with Level 2-PE BD scores (OR = 0.64, 95% CI 0.49, 0.84; P < .001) and Level 3 scores (OR = 0.59, 95% CI 0.41, 0.84; P < .01), while controlling for years in practice and gender.

Relative to osteopathic physicians not issued a disciplinary action, the odds of being issued other actions imposed board action for physicians having taken the COMLEX-USA series of examinations resulted in a nonsignificant relationship with Level 1 scores (OR = 1.43, 95% CI 0.95, 2.15; P = .09), Level 2-CE scores (OR = 0.70, 95% CI 0.46, 1.07; P = .10), Level 2-PE HD scores (OR = 0.99, 95% CI 0.75, 1.29; P = .92), and Level 2-PE BD scores (OR = 0.95, 95% CI 0.71, 1.27; P = .74). There was a significant relationship with Level 3 scores (OR = 0.48, 95% CI 0.33, 0.69; P < .001), while controlling for years in practice and gender.

Discussion

State medical licensing policies are in place to protect the public, keep patients safe, and ensure that those who are granted the privilege of practicing as osteopathic physicians have demonstrated minimal competence to do so. Even with these policies in place, there is a small percentage of physicians who subsequently receive state licensing board disciplinary actions. Fortunately, within this study’s time period, less than 1% of osteopathic physicians received any disciplinary action. Using MLR, we investigated and provided support to the validity argument that COMLEX-USA performance provides useful early information about the likelihood of an osteopathic physician receiving a board action against them.

In our study, higher COMLEX-USA Level 2-PE BD scores showed significantly lower odds in receiving a board action revoking a physician’s license and imposing limitations to practice, controlling for scores at other levels, years in practice, and gender. Further, higher Level 3 scores showed significantly lower odds of receiving a disciplinary action in all 3 groups, controlling for the other variables. These inverse relationships provide support to the validity of the information provided by these areas of the COMLEX-USA examination. Even though COMLEX-USA Level 1, Level 2-CE, and Level 2-PE HD scores did not show a significant relationship with disciplinary actions, it would be an overstatement to claim no relationship exists between these measures and the criterion when the null hypothesis cannot be rejected. Passing Level 3 of the COMLEX-USA series is required by state licensing boards for eligibility to receive an undifferentiated license to practice, and candidates must pass Level 1, Level 2-PE, and Level 2-CE to graduate and become eligible to take Level 3. Of the COMLEX-USA scores included in this study, Level 3 scores provided the most consistent indicator of licensed osteopathic physicians’ risk of receiving a disciplinary board action.

COMLEX-USA Level 1, Level 2-CE, and Level 2-PE are administered during undergraduate medical education and are intended to provide an objective assessment of whether the individual meets minimum standards of competency to enter postgraduate training. Individuals who fail the COMLEX-USA are not eligible to graduate or be granted a license for independent practice. Because each exam is specifically designed to assess developing knowledge and skill competencies deemed important for practice, findings from this study appear to follow this progressive sequence. Finding no significance for Level 1, Level 2-CE, and Level 2-PE HD is not completely unexpected and is similar to the findings of the previous investigation regarding the initial step of a medical licensing exam.⁶ These components are administered during earlier years of undergraduate osteopathic medical education, and the eligibility for licensure is not solely determined by successful completion of these exams.

Our study also showed significantly lower odds of receiving a disciplinary action in all 3 groups associated with more years in practice and for female physicians compared with their male counterparts. These findings are relatively similar to what has been reported in previous studies¹² and indicate further research is needed to better understand these aspects of osteopathic physicians on the likelihood of being issued a disciplinary action.

Limitations of the study must be taken into account when interpreting our findings. While our study included important information indicating the likelihood of an osteopathic physician receiving a disciplinary action, it did not encompass all potential risk factors. For example, we did not have information regarding other performance measures (e.g., Medical College Admission Test scores, residency directors’ ratings, length of time spent in graduate medical education, etc.), physicians’ specialty in practice, multiple licensing jurisdictions, workload, working in solitary practice, all of which could potentially affect a physician’s risk of receiving a disciplinary action. We were able to include years in practice and gender in the logistic model to help account for individual variation. Another limitation is that we categorized disciplinary board actions into broad groupings. Future studies are needed to investigate the relationship between COMLEX-USA scores and disaggregated offenses to further identify specific disciplinary events, such as dependence abuse, arrests, or administering medication errors.

To the best of our knowledge, this is the first study to focus investigation on the COMLEX-USA computer-based and standardized patient-based exam scores. Findings from our study support the validity of the intended interpretation of the COMLEX-USA with regard to competency assessed during undergraduate medical education and residency. That is, the progressive content of each component of the series provides useful information to authorities within state jurisdictions who make decisions involving the likelihood of being at risk for a disciplinary action for an osteopathic physician. The COMLEX-USA assesses similar constructs to external criterion measures as indicated by convergent and discriminant validity evidence from previous work, where several significant positive correlations were shown between Level 2-PE clinical component scores and program director competency ratings of first-year osteopathic residents in pediatrics.^20,21 The high-stakes Level 2-PE exam exhibited convergent validity with overall program director competency ratings. Perhaps physicians who have been required to take Level 2-PE engaged in greater reflection and synthesis of these less traditionally assessed competencies during their educational training and continue to do so in practice. It is noted that Level 2-PE is the newest component of the COMLEX-USA series; therefore, the length of time for physicians in practice is relatively short compared with the other components of this exam series. Future research will be needed to show if Level 2-PE findings from this study are consistent over time.

It is critical for high-stakes test agencies to periodically examine the validity evidence regarding interpretation and uses of the exam. This study adds to the literature supporting that higher performance on national standardized assessments used for medical licensure of osteopathic physicians indicates a lower risk of being disciplined in their practice over time. The public depends on physicians to function at a safe, highly professional safe manner at all times. Part of ensuring this expectation is using validated, high-stakes assessments of knowledge and clinical skills throughout the physician’s education, training, and practice.