The Power of Subjectivity in the Assessment of Medical Trainees

Abstract

Objectivity in the assessment of students and trainees has been a hallmark of quality since the introduction of multiple-choice items in the 1960s. In medical education, this has extended to the structured examination of clinical skills and workplace-based assessment. Competency-based medical education, a pervasive movement that started roughly around the turn of the century, similarly calls for rigorous, objective assessment to ensure that all medical trainees meet standards to assure quality of health care. At the same time, measures of objectivity, such as reliability, have consistently shown disappointing results. This raises questions about the extent to which objectivity in such assessments can be ensured.

In fact, the legitimacy of “objective” assessment of individual trainees, particularly in the clinical workplace, may be questioned. Workplaces are highly dynamic and ratings by observers are inherently subjective, as they are based on expert judgment, and experts do not always agree—for good, idiosyncratic, reasons. Thus, efforts to “objectify” these assessments may be problematically distorting the assessment process itself. In addition, “competence” must meet standards, but it is also context dependent.

Educators are now arriving at the insight that subjective expert judgments by medical professionals are not only unavoidable but actually should be embraced as the core of assessment of medical trainees. This paper elaborates on the case for subjectivity in assessment.

For at least the last 50 years, “objectivity” has been an almost undisputed pursuit for those designing tests, including medical educators.¹ In 1961, De Groot^2(p172) defined objectivity as judgment “without interference or even potential interference of personal opinions, preferences, modes of observation, views, interests or sentiments.” The search for objectivity in testing was effectively enabled with the introduction of multiple-choice questions (MCQs) in written assessments, as the growing numbers of students in schools and universities stimulated the need for more automatic scoring systems. MCQs offered opportunities to exclude the personal opinions of examiners when assessing student knowledge and, therefore, offered a much-needed response to disputes about fairness and standards.

In medical education, efforts toward objective testing soon extended to the assessment of more sophisticated training goals beyond factual knowledge. Tests such as the patient management problem³ and the triple jump exercise⁴ were developed to “objectively” assess clinical reasoning and problem-solving skills, although these were plagued with issues of case and context specificity and, therefore, required many hours of testing to achieve appropriate levels of reliability.⁵ More successfully, performance-based tests of clinical skills, exemplified by the objective structured clinical examination,⁴ were integrated into many undergraduate and postgraduate medical education programs and national examinations.^6,7 With the introduction of mini-clinical evaluation exercises, clinical encounter cards, clinical work sampling, direct observation of procedural skills, and other tools, the search for objectivity also extended to workplace-based assessment.⁸ More recently, the search for objective assessment in the clinical workplace was given impetus through the introduction of competency-based medical education (a pervasive movement that started roughly around the turn of the century), with its injunction to move away from unclear and local standards of clinical competence in time-based apprenticeship models toward transparent, structured, outcomes-oriented clinical performance assessment.⁹

There is no question that these innovations in testing have had a valuable impact on the assessment of knowledge and skill in medical education. In this paper, however, we wish to raise some concerns and doubts about this drive toward objectivity as the exclusive, or even best, mechanism for achieving clarity, transparency, fairness, and validity in assessment processes, particularly for undergraduate and postgraduate training in the clinical workplace. We will first suggest that objectivity may not, in fact, represent what current efforts are achieving. Rather, these purported efforts toward “objectivity” might better be understood as negotiating a “shared subjectivity,” a convergence on a single, but still socially constructed, perspective. Although such convergence might achieve consensus, this should not be mistaken for bias-free objectivity. Second, we will suggest that in many situations relevant to effective health care delivery, negotiating a single shared perspective among assessors fails to represent authentic practice, which represents a range of perspectives on competence and interpretations of performance from a multitude of stakeholders. We will end with a discussion of the implications of embracing subjectivity for determining the quality of assessment data and decision making about those being assessed.

The Myth of Objectivity

Objectivity, from a positivist, classical test theory perspective, suggests that for each desired learner quality to be measured, a true score exists. With any existing assessment tool, the derived score will deviate from this true score (the “measurement error”). However, in domains such as medicine, in which students must learn to solve problems rather than produce undisputed answers, very often the objectivity of true scores or standards can be questioned. For example, if “objective” is defined as “precluding personal bias of the assessor,” then it could be argued that even large-scale MCQ tests are not objective. Indeed, every test question is created by an individual, often an expert, and represents a value judgment regarding what material is worth testing and sometimes even what the best answer is. Different experts are very likely to differ in their opinions in this regard, as is acknowledged in some test formats.^10,11 Conversations to arrive at a test blueprint, determining the topics to be included and/or the weighting of topics that are included, are seldom straightforward. Similarly, standard setting often requires a highly complex negotiation among experts, not only regarding how much a minimally competent candidate should know but also how to ensure that a test does not fail an inappropriately high proportion of candidates (e.g., if the expert-determined standard on a national licensing examination were found to fail half the candidates, there would undoubtedly be a strong tendency to adjust standards to bring the failure rate in line with expectations). Even the answers to questions (such as the most likely diagnosis) may be subject to negotiation, and consequently some recent test models have tried to incorporate a variety of expert opinions in the scoring rubric.¹⁰ Thus, even in the purest tests of knowledge, the best approximation of objectivity is often simply a (grudging) consensus among a numerical majority of experts, resulting in what might, therefore, be considered a (negotiated) shared subjectivity rather than objectivity.

This negotiation of shared subjectivity becomes even more obvious in rater-based assessments. The consistent demonstration of psychometric weaknesses,^12–15 even when preceptors rate the same performance,^16–19 has led to numerous efforts at rater training, including frame-of-reference training,²⁰ to negotiate a common perspective. Interestingly, the relative lack of success for such training efforts has led some to simply exclude “inherently inconsistent” raters to ensure a common perspective and a perceived reliability among the remaining raters.¹⁶ It is hard to argue that this approach excludes subjectivity—at best, it masks subjectivity behind a constructed consensus.

Not only do raters cause problems for the notion of objectivity, so does the context. Context specificity (i.e., the observation that an individual’s performance on a particular problem or in a particular situation is only weakly predictive of the same individual’s performance on a different problem or in a different situation²¹) is a commonly recognized thorn in the side of psychometricians trying to assess performance. Indeed, Norcini has been credited with suggesting that context specificity is “the one fact of medical education.”^22(p1220) Given the widespread prevalence of this “fact,” perhaps it is time to suggest that competence does not reside in the individual but, rather, in the individual’s interaction with a highly variable context.^23,24 Further, Gingerich²⁵ has suggested that the judgment of clinical competence is an inherently social activity and that social judgments are necessarily interpretations of the performance. If so, then a perceiver’s (rater’s) interpretation of an individual’s performance is a part of the context, and multiple perceivers means multiple contexts. For example, one perceiver may experience a performance as reassuring (confident and knowledgeable), and another may see it as off-putting (controlling and arrogant), but each experience of the performance is “true” for that perceiver. Thus, the most appropriate conclusion to draw from variations in assessment for a given performance is not that there is noise in the ratings and a problematic lack of objectivity but, rather, that the performance can be perceived in importantly different ways, so there is no single “objective” truth about the performance (much less, the performer). This is a constructivist view, rather than a positivist one, which, as mentioned above, would suggest that there is only one truth.

Assessment in the clinical setting complicates the notion of “objectivity” still further. In the clinical context, assessment of trainees implies an evaluation of their readiness to be entrusted with care,²⁶ and therefore, the assessment of learners and decisions around patient care are inextricable.^27–29 As medical trainees work under the supervision of a practitioner, the evaluation of their readiness to engage in patient care involves a continuous balancing of the benefits and risks for both the learner and patient.^30,31 That is, when a medical specialist who is charged with the care of a patient is simultaneously evaluating a trainee’s capacity to participate in the care of that patient, the interest of providing the best care will, and likely should, affect the judgment. In this regard, an intimate and personal acquaintance with the learner is needed to develop the confidence needed to make meaningful entrustment decisions.^32,33 These moment-by-moment, ad hoc entrustment decisions must, by definition, be subjective and situation specific. Applying a criterion of “objectivity” to clinical assessment also carries with it the assumption that judgments can always be expressed as documentation that can be shared and understood acontextually. In practice, preceptors’ gut feelings or intuitions about trainees likely have an important role in guiding decisions about their readiness to practice alone.²⁷ Similar to expert judgments about patients, these intuitions might be shared meaningfully among other experts with similar experiences but are likely to lose some of their essence when formalized in documented words or numbers.^34,35 Thus, to arrive at a summative decision about a given individual, either to guide further training or to determine readiness for certification,³⁶ it is necessary for a team, such as a clinical competency committee, to examine the breadth of these subjective assessments over some time period and to negotiate, in light of the complex patterns of data and informed by their own personal knowledge and experience, until the team feels comfortable in making a coherent collective (rather than “objective”) determination.^37,38

The Power of Embracing Subjectivity

To this point, we have argued that the pursuit of objectivity is problematic because of the “single truth” that it implies. In this section, we will suggest, consistent with other recent authors,^39,40 not merely that subjectivity cannot be avoided but that, in fact, it should be embraced. We previously pointed out that there might be multiple legitimate perspectives on a single performance and that each of these perspectives might be “true” in the experience of the individual perceiver. If so, the effort to average those perspectives to find the “signal in the noise” or to try to negotiate a single common perspective among perceivers is problematic not only in its representation of the individual but also in its preparation of the individual for effective future performance. The popularity and widespread acceptance of multisource feedback (MSF) as a legitimate approach to building a valid image of a trainee exemplifies our point.^41,42 It is because of the differences between assessors, not despite them, that MSF is so useful.

Adaptability to the context is a particularly important feature of a skillful practitioner, and as previously suggested, assessors are part of that context. What the community (patients, health professionals, hospitals, etc.) would like to see in a high-quality practitioner is the ability (and propensity) to monitor his or her impact on other individuals in an interaction and, when needed, to modify his or her behaviors in ways that accommodate the feedback received about his or her actions as experienced by each person. If others perceive one’s actions or approach positively, then continuing in this way is appropriate. However, if it becomes apparent that others are finding one’s actions or approach off-putting, then it becomes necessary to adapt appropriately. To effectively monitor and accommodate in this way, it is critical for the actor to know that some people find his or her style off-putting (or see his or her actions or approach as arrogant) so that he or she can be alert to this concern and adapt if and when this sort of reaction is being perceived. Thus, in contrast with an assessment process that suggests that there is just one best way to act in a particular situation, a more appropriate message to relay to the trainee might be the various ways in which his or her behavior was interpreted so that he or she can be alert to these sorts of interpretations and respond accordingly, in a situationally appropriate way. The fact that learners often react to “inconsistent” feedback with frustration, therefore, might be an important signal that they believe there is a single objectively correct way to act. This frustration suggests that these learners are currently not well prepared for the variability in interpretations of their behavior that they will face in clinical practice.

Interestingly, embracing subjectivity not only offers the possibility of richer feedback by defensibly representing differing perspectives on performance across assessments, but it also enables better thinking about the value and defensibility of the moment-by-moment judgments being made by preceptors to enable ad hoc entrustment. Using Crossley and colleagues’⁴³ notion of construct-aligned scales, assessment is shifting away from statements about the individual being assessed and focusing instead on the level of participation that the preceptor feels comfortable allowing for a certain learner at a certain moment.^44–46 Entrusting learners with clinical tasks implies an assessment of perceived risk, as the anticipated level to which the learner will be able to perform the task is weighed against the patient’s safety in that particular context.^29–31,47 Importantly, this shift in focus empowers the preceptor to probe and document his or her subjective experience rather than forcing him or her to document a context-free inference about the learner in the guise of objectivity. Ironically, therefore, the move to subjectivity as a framing of assessment in the workplace places the preceptor in a substantially more defensible position with regard to his or her documentation. A learner might legitimately question the “objective truth” in statements such as “below average” or “meets expectations” or challenge the fairness of differences in “objective scores” given to him or her as compared with different leaners. However, it is difficult for a learner to challenge a statement such as “I am just not comfortable with you performing this procedure,” “I’ll not have you lead that patient conversation on your own yet,” or “I’m now comfortable leaving the operating room while you complete this part of the procedure.”⁴⁶ In other words, at the level of an individual assessment of a single performance, documentation of the preceptor’s subjective experience is the only truly defensible proposition. “Objective truth” statements are always open to being questioned. Even descriptions of expected behavior at different developmental stages^48,49 can, at best, be a suggested reference for raters; they can never serve as “objective” milestones.⁵⁰

Implications and Future Directions

Acknowledging and celebrating the reemergence of subjectivity in assessment, Hodges⁴⁰ has described health professions education as moving into a “post-psychometric era.” We agree that there is a growing and appropriate challenge of the psychometric premise of objectivity and its attendant construction of variability in assessment as simply noise masking a single “true” signal about an individual or performance. However, we wish to strongly suggest that this should not lead to the return of a “pre-psychometric” mind-set about data and assessment. We cannot ignore, and, in fact, must build on, the lessons of the psychometric era. It is important to remember that the psychometric pursuit of objectivity included, in no small part, an effort to achieve fairness in assessment. Lessons from the past repeatedly demonstrate that unfettered subjectivity can easily lead to the (implicit or explicit) systematic disadvantaging and even outright exclusion of individuals from different social groups. However, as noted by Gould,⁵¹ lessons from the past also suggest that the development of “objective” measures has not infrequently produced similar results. Seeking fairness in assessment remains an important goal. But learners should realize that fairness results from the interaction of ability (observed behavior) with context (including the expert rater and the circumstances), making comparisons among learners challenging and inherently less transparent (“I saw you [learner A] doing very well with an easy case” versus “I saw you [learner B] struggling with a difficult case” could lead to a similar rating, but could make learner A feel that he or she was being treated unfairly). Nonetheless, it is important to remember that not all data are created equal, and as educators move toward embracing subjectivity in assessment, they must develop new ways to determine the legitimacy and meaningfulness of the judgments supervising clinicians make, whether for summative or formative purposes. Clinicians, as content experts, should receive support and guidance on how to explain entrustment decisions to learners.

There are already some intriguing explorations in this direction for the assessment of individual pieces of written work (as reviewed and elaborated on by Kuper⁵²). In considering how to translate these ideas into the clinical realm, one promising direction is to deeply explore what makes professionals trust their colleagues as practitioners.⁵³ Expert judgment, although fraught with subjectivity, is unavoidable, but its quality increases with experience. As Hodges^40(p37) has argued, clinical assessment of trainees might best be likened to clinical judgment: “With experience, expert clinicians become more rapid and more accurate in their recognition of patterns. There is no reason to believe that this process does not also operate in education.” Yet, this process need not be treated as a “black box.” We are hopeful that by unpacking the process by which ad hoc entrustment decisions are being made, researchers can develop opportunities to inform and shape these practices in positive ways.

Increasingly, it is being recognized that uniqueness among individual practitioners is not something to be avoided. In fact, recent models of patient safety have suggested that “everyday performance variability provides the adaptations that are needed to respond to varying conditions, and hence is the reason why things go right. Humans are consequently seen as a resource necessary for system flexibility and resilience.”^54(p4) This suggests that much of the context-dependent ability that comprises expert clinical performance may legitimately vary among practitioners and requires subjective judgment. Educators must explore how to compile subjective data to compare across people or against some standard for the purposes of high-stakes decision making. This will require enough data to be able to discern patterns and interpret individual data points in context. This does not necessarily mean discarding outliers or averaging such that varying opinions are lost in the summative representation of the individual (as happens with the use of “central tendency” statistics) but, rather, interpreting the variability of data according to each data point’s context and giving the data their weight based on their importance rather than on their consistency.