Secondary Logo

Journal Logo

Predicting Clinical Performance

Domains of Unprofessional Behavior During Medical School Associated with Future Disciplinary Action by a State Medical Board

Teherani, Arianne; Hodgson, Carol S.; Banach, Mary; Papadakis, Maxine A.

Author Information
  • Free

Abstract

As research on professionalism in medicine has acquired momentum, attempts to define and isolate behaviors indicative of unprofessional conduct have gained attention. The American Board of Internal Medicine defined professionalism as an entity that “requires the physician to serve the interests of the patient above his or her self-interest. Professionalism aspires altruism, accountability, excellence, duty, service, honor, integrity and respect for others.”1 Several studies have attempted to define domains of unprofessional behaviors. Arnold and her colleagues2 surveyed medical students and residents about their professional attitudes and behaviors to assess a scale for measuring professional behaviors. The subscales or domains of excellence, honor/integrity, and altruism/respect were identified. These behaviors provided an important perspective on what medical students and residents observe and value in practice as professionalism. The measurement of professionalism has also occurred through critical incident techniques.3 The behaviors most frequently found on critical incident reports were personality problems, fabrication, and abdication of responsibility. Another study examined written comments and formal evaluation session notes from 18 students who did not complete their core clerkship in medicine due to deficiencies in professionalism. The investigators applied behaviorally based descriptors as references, coding the comments into the domains of professionalism.4 The domains that were commonly cited for deficiencies were reliability and commitment, response to instruction, and working relationships. Despite the impressive work that has been accomplished to define unprofessional behavior, there is no consensus about which domains of unprofessional behavior are most important, nor has there been a study that has linked domains of unprofessional behavior to future disciplinary outcomes.

The search for “rigorous qualitative techniques” and “refined quantitative assessments” to define and measure professionalism continues.3 Recommendations that professionalism be evaluated effectively require that abstract definitions of what constitutes professionalism be replaced by observable behaviors and techniques for monitoring and measuring those behaviors to identify individuals in difficulty must be developed.5 The identification of observable behaviors is an important consideration and a daunting task. It is anticipated that this process can then lead to remediation and eventually a reduction in the need for future disciplinary action.

Our previous work demonstrated an association between unprofessional behavior in medical school and subsequent disciplinary action by a state medical board when the students became practicing physicians.6 This study expands on that work by identifying the domains of unprofessional behavior in the medical students that were linked to the disciplinary actions. Our research questions were: (1) Can a medical school physicianship form be used to generate domains that can then be used to reliably code comments found in students’ records?, and (2) Do cases and controls differ in the domains and number of unprofessional behaviors that are recorded in medical school records?

Method

The physicians in this study were University of California, San Francisco (UCSF), School of Medicine graduates. Their medical school files contained their application to medical school, course grades and evaluations, the dean’s letter, and any other documents. A retrospective case-control design was used to demonstrate that unprofessional behavior in medical school was associated with subsequent disciplinary action by the Medical Board of California.6 Demographic characteristics of these subjects and the sampling methodology have been previously described.6 The case physicians (n = 68) were UCSF graduates disciplined by the Medical Board of California between 1990 and 2000. Control subjects (n = 196) were UCSF graduates matched to year of graduation (within one year) and specialty. Kohatsu et al.7 have described the characteristics of physicians disciplined by the Medical Board of California.

All negative and questionable comments were extracted verbatim from any portion of the student files by a research assistant. Positive comments (e.g., “excellent student”) were not extracted. All files from which comments were extracted were reviewed by one investigator (MP) to ensure that all pertinent comments were retrieved. The same investigator also randomly checked approximately 10% of the files with positive or no comments to ensure that comments were not inadvertently missed. Human subjects approval was obtained from the UCSF Committee on Human Research.

Domains from the UCSF Physicianship Evaluation Form were used to code the extracted comments.8,9 The form details domains of behaviors deemed as unprofessional by the faculty. Two versions of the form, one for medical school years 1–2 and one for medical school years 3–4, were condensed to create a composite Physicianship Evaluation form. The research assistant who abstracted the negative comments and the investigators involved in coding were blinded as to the case or control status of the physicians. The domains of behavior or physicianship codes listed in the Physicianship Evaluation form were used to analyze the entire sample of graduates to determine the occurrence of a code (yes/no). Subsequently, differences in the occurrence and sum of the codes were used to compare subject status (i.e., cases versus controls).

The coding process was accomplished in two stages. During the first stage, one investigator with significant experience with medical students and professionalism issues (Dean of Student Affairs, MP) read the extracted comments to garner and categorize additional codes using the Physicianship Evaluation form. The goal of this derivation stage was to use a form of open coding to ensure that the Physicianship Evaluation form was complete and all codes were categorized appropriately.10 Subcodes were created to further define and expand each of the overall codes. For example, a subject who was repeatedly late or absent from clinic sessions was considered as lacking in reliability and responsibility. A subject written up to be “apathetic” or “unmotivated” was categorized as lacking of initiative and motivation.

The final, comprehensive version of the Physicianship Evaluation form was used by three investigators (MP, AT, CH) to separately code comments for the first 90 subjects. Whether the code occurred for each subject was considered in the coding process (coded 0–1). Differences in coding between the investigators at this stage were recorded for interrater agreement purposes. If two raters coded the comments similarly, the code was selected as the final code. Two investigators (MP, AT) coded the comments for remaining subjects (91–264). Where differences existed, investigators arrived at conclusions through discussion about the final codes assigned.

Finalized codes for each subject were entered into a database. Code occurrence was recorded for each subject. Statistical Program for the Social Sciences version 12.0 (SPSS Inc., Chicago, IL) was used to analyze the data. Cohen’s kappa was used to compute agreement among raters. The frequency of the physicians in each group who displayed any of the domains of the behavior (yes versus no) was analyzed using chi square tests. The severity of the behaviors was determined by examining the sum of the number of subcodes for each subject. To determine if the number of subcodes was able to further distinguish subject status, a multivariate analysis of variance was used. Post hoc univariate analyses were performed to test differences between groups on each code.

Results

The kappa values ranged between 0.89 and 1.00 for code occurrence for two of the raters. Table 1 lists the frequencies of the domains of behavior in the case and control physicians. The three domains of (1) poor reliability and responsibility; (2) lack of self-improvement and adaptability; and (3) poor initiative and motivation occurred significantly more often in the case physicians than in the control physicians.

Table 1
Table 1:
Frequency of Domains of Unprofessional Behavior in 68 Physicians Disciplined by the Medical Board and 196 Matched Control Physicians

The total occurrences per physician for the three significant domains identified in Table 1 by chi-square analyses were used as dependent variables in the multivariate model. There was a significant main effect for cases vs. controls (F = 2.57, df = 3, p = .05). Table 2 displays the results for the univariate analyses. There were significant univariate results for the domains of poor reliability and responsibility and poor initiative and motivation. The total number of occurrences of self-improvement and adaptability approached significance.

Table 2
Table 2:
Mean Ratings of Domains of Unprofessional Behavior Related to Disciplinary Outcomes for 68 Disciplined Physicians and 196 Matched Controls Physicians

Discussion

This study identified three domains of unprofessional behavior among medical students associated with subsequent disciplinary action by a state medical board when these students became practicing physicians. The three domains were poor reliability and responsibility, lack of self-improvement and adaptability, and poor initiative and motivation. Individuals experiencing difficulty in any of these three areas were more likely to be disciplined in future practice. Within the two domains of reliability and responsibility and initiative and motivation, disciplined physicians were more likely to exhibit a greater number of those types of behaviors.

In an earlier study, overall concerns about a student’s professionalism during medical school predicted later disciplinary outcomes by a state medical board.6 This study builds on that research by defining domains of unprofessional behavior observed during medical school that are associated with subsequent disciplinary action.

The lack of reliability and responsibility is a critical domain of unprofessional behavior that has been identified in earlier studies.3,4 This study identified two additional areas of unprofessional behavior—lack of self-improvement and adaptability and poor initiative and motivation—worthy of further examination.

A few reasons may account for why some domains and subdomains did not predict future disciplinary action. The researchers were limited to the evaluation of existing data. The frequency with which some domains and subdomains emerged may not be enough to contribute to the prediction. Certain terms, such as those concerning maturity and anxiety, may imply different gradients of severity by different evaluators. What one faculty member considered “immature” might have been more serious than another faculty member who considered “immature” a natural state of development. These individual perceptions of meaning need further exploration. Future research should also focus on methods to train faculty to consistently evaluate behaviors and assess outcomes accordingly.

A recent concern has been that although medical schools report they have incorporated professionalism into their curriculum, they still seek “valid and reliable means” for the evaluation of professionalism.11 A goal of this study was to address this concern by expanding and testing the validity of an instrument which measures professional behaviors by linking those behaviors to future disciplinary outcomes. Institutions aiming to evaluate unprofessional behavior during the course of medical school would benefit from knowing about and consequently tracking the domains identified in this study. In practice, the inclusion of these behaviors on professionalism forms and in the training of faculty to watch for such behaviors, could be beneficial. Some research has suggested that there is not consistent agreement among faculty about which behaviors are unprofessional.12 Findings from this study, if replicated, could help develop consensus about the most problematic domains of unprofessional behaviors as the identified domains were evidence-based through their linkage to the outcome of disciplinary licensure actions.

This study had several strengths. The student files at UCSF were comprehensive, which allowed for detailed analysis. The concordance among at least two of the raters when coding the comments was high, which increased the reliability of coding process. These findings indicate that a worthy relationship is present and future research with a larger sample size may yield valuable outcomes.

A limitation of this study was the retrospective research design. Second, the study was of graduates from a single institution. Thirdly, the sample size may not have been adequate for all the subgroup analyses. A larger sample size will be required to replicate and validate the results of these preliminary analyses. However, these preliminary analyses did generate some significant trends worthy of further exploration.

This study defined three critical domains of unprofessional behavior in medical students that are associated with disciplinary action. Validation of these findings could lead to focused remediation strategies and policy decisions about a student’s ability to graduate from medical school if successful remediation has not occurred.

References

1 American Board of Internal Medicine. Medical professionalism in the new millennium: a physician charter. Ann Intern Med. 2002;136: 243–46.
2 Arnold EL, Blank LL, Race KE, Cipparrone N. Can professionalism be measured? The development of a scale for use in the medical environment. Acad Med. 1998;73:1119–21.
3 Arnold L. Assessing professional behavior: yesterday, today, and tomorrow. Acad Med. 2002;77:502–15.
4 Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well three evaluation methods detect deficiencies in medical students’ professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75:167–73.
5 Ginsburg S, Regher G, Hatala R, et al. Context, conflict, and resolution: a new conceptual framework for evaluating professionalism. Acad Med. 2000;75:S6–S11.
6 Papadakis MA, Hodgson CS, Teherani A, Kohatsu ND. Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Acad Med. 2004;79:244–49.
7 Kohatsu ND, Gould D, Ross LK, Fox PJ. Characteristics associated with physician discipline: a case-control study. Arch Intern Med. 2004;164:653–58.
8 Papadakis MA, Osborn EH, Cooke M, Healy K. A strategy for the detection and evaluation of unprofessional behavior in medical students. University of California, San Francisco, School of Medicine Clinical Clerkships Operation Committee. Acad Med. 1999;74:980–90.
9 Papadakis MA, Loeser H, Healy K. Early detection and evaluation of professionalism deficiencies in medical students: one school’s approach. Acad Med. 2001;76:1100–6.
10 Strauss A, Corbin J. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Thousand Oaks, CA: Sage Publications, 1990.
11 Kao A, Lim M, Spevick J, Barzansky B. Teaching and evaluating students’ professionalism in US medical schools, 2002–2003. JAMA. 2003;290:1151–52.
12 Ginsburg S, Regher R, Lingard L. Basing the evaluation of professionalism on observable behavior: A cautionary tale. Acad Med. 2004;79:S1–S4.

Section Description

Moderator: Lee Manchul, MD

Discussant: Eric Holmboe, MD

© 2005 Association of American Medical Colleges