Why Focus Our Assessment Framework on Strugglers?
Recent studies have demonstrated that medical school faculty and administrators, as well as regulatory bodies, are becoming increasingly dissatisfied, if not discouraged, with issues surrounding strugglers.1–3,25 For example, concerns about individual medical student confidentiality and the ethical problems associated with the potential bias that may occur if information about poor performance is shared with other (future) course and clerkship directors often present a barrier to developing effective remediation plans, let alone to fulfilling societal obligations. Compounding these issues are pressures to increase medical school class size and the emergence of new medical schools, which could add to the overall number of strugglers.26
Given these problems, an emergent challenge for medical educators involves developing effective ways to not only accurately identify strugglers but also to better understand the primary causal factors underlying their poor performance. Identifying the reasons for one's poor performance in medical school, residency, or practice is a key first step in developing adequate remediation plans.27
Identification of strugglers: Traditional frameworks of evaluation
Current frameworks in medical education focus on the identification, evaluation, and remediation of struggling trainees and can be broadly grouped into analytic, developmental, synthetic, and competency-based approaches. Within each framework, the methods or measurements for identification of strugglers include a host of instruments that can be used in a formative and/or summative manner.28–30 Some methods involve direct observation of the trainee, whereas others rely on less authentic means, such as the use of multiple-choice examinations.
The analytic framework takes the trainee's performance “apart”—Greek: ana (whole) plus lytic (apart)—typically dividing learning outcomes into the domains of knowledge, skills, and attitudes. Developmental frameworks, such as the Dreyfus and Dreyfus28 approach, describe learner performance in progressive steps (e.g., novice and early beginner through expert), typically within a domain of performance. Synthetic models put the learner's performance together into discrete “steps,” or levels, with each step requiring a synthesis of greater knowledge, skill, and attitude. An example of the synthetic approach is RIME (Reporter, Interpreter, Manager, Educator).30 Finally, competency-based frameworks include the widely adopted competencies of the Accreditation Council on Graduate Medical Education, as well as those of CANMEDS and the Good Doctor.31,32 Synthetic and competency-based frameworks are also an example of “role-based” frameworks.33 Unfortunately, current methods of assessment and remediation in medical education tend to explain little variance (typically, studies explain less than 30% of the variance) in the outcomes of interest, such as professionalism, basic science knowledge, and/or clinical skill.1,5–8 If these frameworks were optimal, one would expect to explain more of what contributes to the outcomes of interest. The fact that little variance is explained suggests that other factors are contributing to these outcomes of interest, a notion that is particularly important for those who struggle.
In addition, medical education assessment frameworks, per se, typically do not focus on identifying the causal factors underlying poor performance of strugglers and thus provide minimal guidance to medical educators on how best to remediate these students. For example, if a student fails a written examination, there could be a number of potential causes, such as poor preparation, lack of higher-order thinking skills (e.g., relying on rote recall rather than transformational learning strategies), test anxiety, lack of effort, illness, or a personal crisis. Or, if a resident cannot successfully complete a post-encounter form for an objective structured clinical examination station, the reasons for suboptimal performance could include poor knowledge, poor time management, communication deficiencies, or poor self-awareness or self-assessment, to name just a few.
Without identifying and understanding the proximal causes of trainee underachievement, it is difficult to pinpoint the most effective ways to assist trainees in improving their performance. In fact, without clearly linking what has been learned in the assessment with subsequent remediation strategies (and the evaluation of “success” during that remediation) or the structured framing of the remediation (e.g., retaking a clinical rotation), we may be unintentionally setting students up to repeatedly fail. Furthermore, because our current assessment frameworks are generally not based on well-researched educational models or theories, it is challenging for educators to use and integrate various types of assessment data to generate practical recommendations for the assessment of struggling trainees and professionals. Collectively, these shortcomings are important reasons to consider alternative approaches, such as the SRL-MAT, an assessment framework designed to generate information about trainees' motivation and regulatory skills as they engage in authentic tasks in medical school and residency, such as taking comprehensive exams and performing clinical tasks. In sum, we believe the SRL-MAT can complement current assessment and training frameworks in medical education by delineating an innovative assessment process that identifies deficient self-regulatory processes in trainees and thus provides a direct link between assessment and remediation.
Self-regulation defined (and how it differs from self-assessment)
Social–cognitive researchers have defined SRL as a cyclical process whereby individuals use self-generated feedback about their learning to optimize their strategic pursuit of personal goals.34,35 From a social–cognitive perspective, self-regulation is typically conceptualized as a teachable skill that operates as a three-phase cyclical loop (see Figure 1). In short, processes preceding action (forethought) have an impact on learning efforts (performance control), which in turn influence how learners react to and judge their performance successes and failures (self-reflection). Forethought processes, such as goal-setting and strategic planning, help to mobilize students' efforts to learn. Once trainees engage in learning tasks, such as studying for a major exam or drawing blood from a patient, they typically enlist performance-phase processes such as using specific tactics to facilitate task performance (self-control tactics) and tracking the effectiveness of these methods (self-monitoring). The information that trainees gather from task performance is used to judge whether they succeeded or failed (self-evaluation), to judge why this performance occurred (attributions), and to decide what they need to do to optimize future performance (adaptive inferences).
It is important to note that self-regulation and, more specifically, SRL, differs from self-assessment (calibration of one's perceived performance with actual performance36); self-assessment constitutes just one of the several features of SRL. The medical education literature has found that poor performers are often the most unaware, that is, have the poorest self-assessment skill.37,38 This deficit between actual performance and awareness (self-assessment)38,39 also suggests that strugglers may possess poor knowledge of task demands or requirements, as well as of their own levels of knowledge or expertise, and may exhibit other problematic self-regulatory skills, such as ineffective use of learning strategies and maladaptive motivational beliefs.
Decades of work by educational psychologists suggest that students can be taught to become more independent and autonomous learners through self-regulation training.2,18 Of particular importance is that researchers have begun to show that struggling students can improve the accuracy of their self-assessments regarding personal capabilities and knowledge if they are provided with effective feedback from content experts or teachers and if they are taught specific strategies to generate internal feedback, such as error detection and self-monitoring.40–42 It is our belief that the SRL-MAT can provide valuable assessment data that can be used by educators to generate and provide task, strategic, and self-regulatory feedback.
The SRL-MAT Framework
The SRL-MAT framework is grounded in social–cognitive theory and research and thus makes a variety of important assumptions about student learning, as well as the essential techniques needed to evaluate student functioning (see Figure 1). According to Bandura's13,14 model of reciprocal determinism, student behaviors or skills, personal processes (beliefs, affect), and environmental factors (teachers, classroom climate) interact and influence each other in reciprocal ways. However, primary emphasis is placed on the role of human cognition and, in particular, self-efficacy perceptions, as a key personal process affecting student behavior. Thus, a medical student who possesses low self-efficacy, or low task-specific confidence, for drawing blood from a patient may exhibit low motivation and/or avoid these types of clinical experiences, which may further elicit negative feedback from his or her professors in medical school. Conversely, if faculty advisors or professionals provide this student with strategic feedback on how to improve his or her clinical skills, the student's perceptions of competence may increase while anxiety about this procedure lessens—personal factors that may prompt higher levels of behavioral engagement and motivation.
Another important assumption of the SRL-MAT framework is that behaviors and beliefs are dynamic and fluid in nature and thus will often vary across educational contexts, as well as for specific tasks within those contexts. For example, a medical student may show adequate diagnostic skills when evaluating a patient with new-onset polyuria who has diabetes, but that same student may show poor skill in recognizing a patient with congestive heart failure. From a practical and diagnostic perspective, this context-specific assumption is important because it suggests that effective self-regulation assessment tools should be tailored to specific contexts and tasks or activities within those contexts. The SRL-MAT uses an emergent assessment approach called SRL microanalysis, a procedure that involves asking temporally sequenced questions about specific regulatory processes as trainees engage in an authentic task or activity.23,24,43,44
In contrast to traditional self-report measures that require students to retrospectively provide accounts of their perceptions, beliefs, and behaviors without reference to situational demands or particular contexts, microanalytic measures are customized to particular academic tasks or activities and involve using context-specific questions targeting the various processes of the three-phase cyclical loop (see Figure 1). In this sense, SRL microanalysis seeks to evaluate how students approach, execute, and reflect during (as opposed to a long time after) specific tasks or events. The advantage of conceptualizing self-regulation in terms of a cyclical loop relative to specific tasks is that one can evaluate student regulation across virtually any activity or event (e.g., an examination, a history and physical, a written progress note, response to external feedback, or an interaction with a peer) as long as the event has a clear beginning and end (see reference 43 for a more detailed discussion). We have provided simplified example scenarios using the SRL-MAT framework in Scenarios 1 and 2.
Assessment of self-regulation
From our vantage point, trainees who struggle in medical school or residency might lack the regulatory skills to consistently monitor and adapt their learning efforts to maximize their performance. As a result, evaluating these skills and processes during medical training may not only help medical trainers better understand the processes that inhibit growth in their students but also may help them devise remediation or instructional plans to help these students improve. In the self-regulation literature, a variety of assessments are discussed, including paper-and-pencil self-report scales, direct observation, think-aloud protocols, and self-regulation microanalysis.16,45–48 Although self-report scales, such as the Motivated Strategies and Learning Questionnaire, are the most frequent type of self-regulation assessment tools and typically have evidence of strong psychometric properties,46,47 they are often problematic because of recall and other response biases, as well as the decontextualized nature of the scales. This latter problem is of particular concern because such scales often fail to capture the dynamic, context-specific nature of self-regulation. Furthermore, asking trainees to retrospectively report how they study or learn in medical school or the general strategies that they use (e.g., time management, organization) to succeed will yield minimal information about the thoughts and behaviors that they actually display while engaged in authentic clinical tasks, such as interviewing a patient, performing a procedure, or making a specific medical diagnosis. In contrast, assessment tools designed to directly measure self-regulation processes as they unfold in real time during essential clinical tasks and activities have the potential to provide medical educators with more meaningful information that can be used to not only understand potential causes of poor performance in medical school but also to provide insight into the type of feedback and remediation that strugglers may need.
Self-regulation researchers during the last 20 years have developed a variety of alternative assessment tools, such as think-aloud protocols, behavioral traces, and microanalytic assessment approaches, to directly measure self-regulation during specific events.16,47,49 The SRL-MAT endorses the use of think-aloud protocols and direct observations, but it places primary importance on using SRL microanalytic methodology to evaluate the regulatory skills and processes of trainees. In general, this assessment methodology involves comprehensively targeting the cyclical phase processes of self-regulation (e.g., goal-setting, metacognitive monitoring, strategy use, causal attributions) as an individual engages in a well-defined task or activity.24 It is a largely cognitive approach that seeks to identify faulty regulatory processes so that examiners can isolate gaps or deficiencies in how individuals approach (forethought), monitor (performance), and reflect (self-reflection) during their engagement in some specific task.43,50 Some of the essential features of SRL microanalysis include the use of (1) individualized assessment protocols, (2) open-ended and closed-ended questions targeting processes within all three phases of the self-regulation cyclical loop (see Table 1), and (3) customized self-regulation questions that are administered at strategic points during learning or performance on a specific task (see reference 43 for a more detailed overview of this procedure).
A variety of research studies have been conducted to illustrate the procedures and utility of SRL microanalysis. Cleary and Zimmerman49 conducted one of the first studies to microanalytically examine a comprehensive array of regulatory processes. Using a sample of high school students, the authors elected to use basketball free-throw shooting as the task around which to embed the forethought and self-reflection phase questions. Immediately preceding a free-throw practice session, the participants were asked specific questions regarding their goals, strategic plans, and self-efficacy in relation to shooting free throws. In that study, for example, the goal question was, “Do you have a goal when practicing your free throws? If so what is it?”; whereas the strategic planning question was, “What do you need to do to accomplish that goal?” The players were also asked a series of questions targeting their self-reflective thoughts following failure. For example, to examine their attributions and adaptive reactions following two missed shots in a row, the participants were asked, “What is the main reason why you missed those last two shots?” and “What do you need to do to improve your performance?” Examples of specific questions from a medical education context are shown in Table 1, with specific scenarios illustrating the application of these questions in Scenarios 1 and 2.
An important point here is that in Cleary and Zimmerman's basketball study, all questions were targeting specific regulatory processes, were customized to the specific task of interest (basketball free-throw shooting), and were administered as participants practiced their free-throw shooting. Such a procedure allowed the researchers to directly examine student cognition and regulatory processes as they unfolded in real time. Of empirical importance, the authors were able to show that the use of microanalytic questions clearly differentiated expert, nonexpert, and novice free-throw shooters. That is, expert free-throw shooters, who were defined as those shooting over 70% from the free-throw line, displayed higher levels of self-efficacy, set more specific goals, used more effective shooting and technique-related strategies, and were more strategic in their self-reflection following missed free throws than the two other groups.
Several other microanalytic studies have also illustrated the application, psychometric quality, and importance of SRL microanalysis procedures across diverse fields.23 In fact, Kitsantas and Zimmerman22 illustrated that SRL microanalytic-based questions not only differentiated among expert, nonexpert, and novice volleyball players but that, when collapsed into a single score, they accounted for 90% of the variance in volleyball serving skill.
Integrating SRL Microanalysis Into Medical Education
SRL microanalytic methodology can be applied to virtually any well-defined task, event, or activity. In medical education, some of the key tasks on which we expect medical students to reach proficiency include history taking, physical examination, and communication. SRL microanalytic questions and methodology can be developed around several of the expected tasks in order to provide educators with real-time, authentic information about the regulatory deficiencies that may underlie poor clinical performance. The basic idea is to identify a task that has both a clear beginning and end, and then wrap the microanalytic assessment approach—involving forethought (before), performance (during), and self-reflection (after)—around such a task. This would allow one to examine how students approach, learn or perform, and reflect on particular outcomes. In essence, we see the SRL-MAT as an innovative assessment and intervention approach that can be used with current training frameworks for comprehensively identifying, evaluating, and remediating strugglers across the medical education continuum. At this point, however, empirical support for the utility of the SRL-MAT in medical education is needed, and so we hope that investigators will pursue studies to further address such use.50
Medical educators' understanding of medical students' or residents' deficiencies in performance (and strugglers' assessment of their performance) is limited. This may be because the component parts of performance in medicine—particularly emphasizing motivation and context—have not been sufficiently defined, assessed, or allowed for real-time feedback to trainees. SRL microanalysis would provide medical educators (and trainees) with information about deficient motivational beliefs, metacognitive skill, and strategy use that adversely affects trainees' performance in medical school and in clinical practice. By generating this real-time feedback, educators would have more pertinent information to guide their remediation attempts with students who are disengaged and at risk for failure or for engaging in poor clinical practice. Some specific examples could include (1) studying strugglers as they take an examination that has given them difficulty, (2) assessing a skill, such as inserting a peripheral venous or central arterial catheter,50 (3) breaking down performance while performing a history or physical exam on a patient, or (4) creating an analysis or plan for a patient. Though such techniques require training of investigators (faculty) and fairly intensive use of resources for an individual trainee, we believe such an approach could (and should) be used for “high-risk” individuals, or strugglers. It could be argued that by not using a more inclusive, theory-driven approach for the identification and remediation of strugglers, we are, in essence, preparing strugglers to engage in self-defeating cycles of failure and frustration.
Finally, we believe that the rewards from faculty development training in SRL-MAT would go beyond the individual strugglers. Through improved understanding and enhanced skills for further identifying, evaluating, and remediating strugglers with the SRL-MAT, we believe that dissatisfaction, and even discouragement, with trying to identify and remediate strugglers could dissipate among faculty members.
In conclusion, although strugglers are a minority of medical trainees, effective remediation can often be a challenge, and the outcomes are usually uncertain. Our proposed framework, the SRL-MAT, capitalizes on work being done in other educational settings and is grounded in educational theory. What is more, the SRL-MAT offers the opportunity to identify strugglers, as well as to design and monitor remediation strategies that could potentially help these trainees become not just better learners but, ultimately, better doctors.
The views expressed in this article are those of the authors and do not represent the official views of the Department of Defense, the United States Air Force, the United States Navy, the Uniformed Services University, or other federal agencies.
6 Greenburg DL, Durning SJ, Cohen DL, Cruess D, Jackson JL. Identifying medical students likely to exhibit poor professionalism and knowledge during internship. J Gen Intern Med. 2007;22:1711–1717.
11 Hamdy H, Prasad K, Anderson MB, et al. BEME systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach. 2006;28:103–116.
13 Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall; 1986.
14 Bandura A. Self-Efficacy: The Exercise of Control. New York, NY: W.H. Freeman; 1997.
15 Boekaerts M, Pintrich PR, Zeidner M, eds. Handbook of Self-Regulation. San Diego, Calif: Academic Press; 2000.
16 Zimmerman BJ. Investigating self-regulation and motivation: Historical background, methodological developments and future prospects. Am Educ Res J. 2008;45:166–183.
17 Puustinen M, Pulkkinen L. Models of self-regulated learning: A review. Scand J Educ Res. 2001;45:269–286.
18 Butler D. The strategic content learning approach to promoting self-regulated learning: A report of three studies. J Educ Psychol. 1998;90:682–697.
19 Dignath C, Buttner G. Components of fostering SRL among students. A meta analysis of intervention studies at primary and secondary school level. Metacognition Learn. 2008;3:231–264.
20 Weinstein CE, Husman J, Dierking DR. Self-regulation interventions with a focus on learning strategies. In: Boekaerts M, Pintrich P, Zeidner M, eds. Handbook of Self-Regulation. Orlando, Fla: Academic Press; 2000:727–747.
21 Butler DL, Beckingham B, Lauscher HJN. Promoting strategic learning by eighth-grade students struggling in mathematics: A report of three case studies. Learn Disabil Res Pract. 2005;20:156–174.
22 Kitsantas A, Zimmerman BJ. Comparing self-regulatory processes among novice, non-expert and expert volleyball players: A microanalytic study. J Appl Sport Psychol. 2002;14:91–105.
23 Cleary T, Zimmerman BJ, Keating T. Training physical education students to self-regulate during basketball free-throw practice. Res Q Exerc Sport. 2006;77:251–262.
24 Cleary T, Zimmerman BJ. Self-regulation empowerment program: A school-based program to enhance self-regulated and self-motivated cycles of student learning. Psychol Sch. 2004;41:527–550.
28 Dreyfus SE, Dreyfus HL; California University Berkeley Operations Research Center. A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition. http://www.stormingmedia.us/15/1554/A155480.html
. Accessed December 26, 2010.
33 Van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309–317.
34 Zimmerman BJ. A social cognitive view of self-regulated academic learning. J Educ Psychol. 1989;81:329–339.
35 Zimmerman BJ. Attaining self-regulation: A social–cognitive perspective. In: Boekaerts M, Pintrich P, Zeidner M, eds. Handbook of Self-Regulation. Orlando, Fla: Academic Press; 2000:13–39.
36 Alicke MD, Klotz ML, Breitenbecher DL, Yurak TJ, Vredenburg DS. Personal contact, individualization, and the better-than-average effect. J Pers Soc Psychol. 1995;37:833–846.
37 Kruger J, Dunning D. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-awareness. J Pers Soc Psychol. 1999;77:1121–1134.
40 Butler DL, Winne PH. Feedback and self-regulated learning: A theoretical synthesis. Rev Educ Res. 1995;65:245–281.
41 Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77:81–112.
42 Ramdass D, Zimmerman BJ. Effects of self-correction strategy training on middle school students' self-efficacy, self-evaluation, and mathematics division learning. J Adv Acad. 2008;20:18–41.
43 Cleary T. Emergence of self-regulated learning microanalysis: Historical overview, essential features, and implications for research and practice. In: Zimmerman B, Schunk D, eds. Handbook of Self-Regulation of Learning and Performance. New York: Routledge; 2011.
44 Schmitz B, Wiese BS. New perspectives for the evaluation of training sessions in self-regulated learning: Time-series analyses of diary data. Contemp Educ Psychol. 2006;31:64–96.
45 Azevedo R. Understanding the complex nature of self-regulated learning process in learning with computer-based learning environments: An introduction. Metacognition Learn. 2007;2:57–65.
46 Pintrich PR, Smith DAF, Garcia T, McKeachie WJ. Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educ Psychol Meas. 1993;53:801–813.
47 Winne PH, Perry NE. Measuring self-regulated learning. In: Boekaerts M, Pintrich P, Zeidner M, eds. Handbook of Self-Regulation. Orlando, Fla: Academic Press; 2000:532–568.
48 Cleary TJ. The development and validation of the self-regulation strategy inventory-self report. J School Psychol. 2006;44:307–322.
49 Cleary T, Zimmerman BJ. Self-regulation differences during athletic performance by experts, non-experts and novices. J Appl Sport Psychol. 2001;13:61–82.
© 2011 Association of American Medical Colleges
50 Cleary TJ, Sandars J. Using microanalysis of self-regulatory processes to understand performance of clinical skills by medical students: A pilot study [unpublished].