Self-directed learning and self-regulation are pillars of the medical profession that depend fundamentally on accurate self-assessment. Yet, self-assessment conceptualized narrowly as a “personal, unguided reflection on performance” is essentially flawed, a task we rarely perform well.1–5 A broader conceptualization, couched within theories of social psychology,6 incorporates the conscious use of external data to inform self-assessment processes and offers additional means to understand how self-assessment might be used in education and practice.1,4,7–10 Accurate self-assessment also requires appropriate external standards for measuring one's performance and the ability to judge the extent to which that performance meets these standards.11,12
Epstein et al13 have integrated the above concepts into a comprehensive definition of self-assessment: “Self-assessment is a process of interpreting data about our own performance and comparing it to an explicit or implicit standard,” and “The power of self-assessment lies in two major domains—the integration of high-quality external and internal data to assess current performance and promote future learning, and the capacity for ongoing self-monitoring during everyday clinical practice.”
This more comprehensive definition enables framing self-assessment as a multifaceted construct comprising numerous discrete activities. These include, for example, selection of appropriate external data and standards, awareness of one's internal state, and critical reflection on one's own performance and external data.11,14,15 Viewing self-assessment in this way, as an externally informed reflective assessment process, enables us to consider questions regarding how learners and practitioners use self-assessment in their learning and practice. For example, on what external resources do learners draw to inform their self-assessments? To what extent do clinicians rely on their own judgments when generating self-assessments? To what extent are external sources such as professional standards, peers, or educators intentionally used? What factors influence the uptake of external information? Generating answers to these questions is essential to understanding how learners' and physicians' practices of self-assessment are informed as well as understanding self-assessment's contributions to learning, attainment and maintenance of competence, performance improvement, and ongoing self-monitoring.
Our goals were (1) to understand how learners and physicians familiar with structured self-assessment interventions perceived and used self-assessment in clinical learning and practice, and (2) to determine the components and processes that comprise self-assessment and the factors that influence them. Because we were particularly interested in the perceptions of individuals informed about the construct of self-assessment, we selected learners and physicians engaged in formal approaches to self-assessment.
We designed an international qualitative study that used focus groups and was guided by the principles of grounded theory.16,17 We selected focus groups to examine the multidimensional concept of self-assessment, because participant interaction facilitates new understanding and richer responses.17 Our research team included PhD and physician educators and researchers with collective experience across the continuum of medical education. Team members brought complementary theoretical perspectives and knowledge to the research including expertise in, for example, assessment; linking assessment, learning, and practice improvement; reflection; cognition; and small-group and workplace learning. Participating institutions' research ethics review boards approved the study. Focus-group participants provided written consent following institutional protocols.
Using purposive sampling, we selected eight programs available to us (Table 1),18–28 representing low, medium, and high degrees of structure/rigor in self-assessment activities, spanning the educational continuum of undergraduate/postgraduate learners and physicians, and situated in Canada, the United States, the United Kingdom, the Netherlands, and Belgium. To classify the degree of structure and rigor, we considered the breadth and depth of the self-assessment activity, particularly the extent to which external information and objective standards were used. Structured activities employed by the programs included portfolios (compiling learning and assessment experiences and reflections on them),18 formalized assessment of objectives and competencies (formally assessing one's learning and performance against external standards),20 personal learning plans (designing a plan for one's learning based on assessed performance gaps),21 multisource feedback (MSF, using formal feedback from multiple reviewers to inform perceptions of performance),25 chart audit (using clinical practice guidelines to audit patient records and determine gaps in performance),23 and practice-based small-group learning (participating in facilitated discussions of evidence and practice to identify gaps).27
In 2008, program directors of the eight programs invited participants in their programs by letter to volunteer for face-to-face focus groups. Focus groups were 1.5 hours in length and were recorded and transcribed. We conducted 17 groups, 2 from each program and an additional MSF group included to increase data saturation for general practitioners. For consistency, one researcher (J.S.) moderated all groups, assisted by at least one other team member or research designate familiar with the specific program and cultural contexts. Open interview questions explored participants' perceptions of self-assessment, use of formal program activities for self-assessing their clinical practice, identification of other activities enabling self-assessment, factors influencing accurate self-assessment, and linkages between self-assessment and reflection. Consistent with grounded theory principles, we revised interview questions on the basis of preliminary data analysis from each group to expand data collection on important concepts and emerging themes.
We conducted the analysis iteratively as a team. Two team members independently open-coded the first transcript and compared their interpretations of themes, which were then reviewed and verified by another team member and then through group discussion. After several focus groups, we developed an initial coding structure organized around the topics of the interview guide and emerging themes. One team member computer-coded the data, using NVivo8 software (QSR International Pty Ltd, Victoria, Australia). At least three other team members analyzed each transcript individually and then discussed theme development in a series of seven 1.5-hour teleconferences, continually informing revision of the coding structure. We held a face-to-face meeting to confirm emerging themes and formed subgroups to conduct detailed analyses of related categories of data. At a subsequent two-day meeting, we discussed these in depth, continued axial coding,16 and constructed a preliminary conceptual diagram of self-assessment integrating major themes. To facilitate concept mapping, we used MindManager Pro 7 software (MindJet Corporation, San Francisco, California). We interpreted themes through constant comparison and explored their interrelationships. Team subgroups refined the conceptual model components and verified their grounding in data. Throughout the analyses, we resolved differences in interpretation through discussion and reexamining transcripts and coded data. Researcher triangulation and high levels of team involvement and collaboration throughout all research stages enhanced the interpretive rigor.17
Overall, 85 learners (undergraduate, postgraduate) and 49 physicians (family physicians, internists) participated, for a total of 134 participants. See Table 1 for participation by program.
Emerging from our analysis of the transcripts is an understanding of the complexity of self-assessment and the processes and dimensions that inform it. We present these in a conceptual model, processes and dimensions of informed self-assessment (Figure 1), comprising five main components. The first three components describe a phased process by which data to inform self-assessment are accessed, interpreted, and responded to. These three components are each contextual, dynamic, and iterative. The first, “sources of information,” includes data from both internal (cognitive, affective) and external (formal, informal) sources, which can be accessed to inform self-assessment. The second component is “interpretation of information,” by which internal and external data are synthesized and assimilated to inform self-assessment. The third component, “responses to information,” corresponds to how the learner or physician uses the data. The three-component process seems highly influenced by the fourth component, which is multiple “external and internal conditions and influences.” A central dimension of the model, the final component, is a balancing or weighing of “tensions” arising from competing internal and external data and conditions.
Notably, these various components are each phenomena to be understood in their own right; that is, each could independently provide the focus for an extensive research inquiry. As such, the results outlined below are not to be treated as a complete examination of each aspect of self-assessment but, rather, as an attempt to better understand how the various elements fit together to inform a more comprehensive whole. Self-assessment appears as a multifaceted, contextualized, ongoing balancing act evident across the continuum of medical education and practice, although individual factors, influences, and responses may vary. In the following, we briefly describe the five components and suggest their interrelationships. To exemplify our findings, we provide quotes identified by a letter code assigned to the focus group and a numerical code assigned to the participant (e.g., C5).
Sources of information
Participants described accessing information from an array of external and internal sources to inform self-assessments.
Both “processes” and “people” informed self-assessment directly and indirectly. People included peers, consultants, other health professionals, supervisors, learners, and patients. Processes included structured activities of the respective programs (Table 1) and other formal and informal activities including OSCEs, clinical evaluations, CME courses, clinical practice guidelines, rounds, reading, audits, and recertification. Importantly, undergraduate and postgraduate learners valued formal and informal feedback from supervisors and senior peers. However, they also reported that feedback was sometimes conspicuous both by its absence and lack of usefulness. Lack of feedback left them feeling isolated, uncertain, and concerned that they might be unaware of inadequate performance. One participant commented,
I feel like I'm … just kind of working under my own steam, and thinking, well, “I hope I'm doing enough, I hope I'm doing the right thing” … you think, “Well okay, I've got a satisfactory, that's all right. I'll carry on doing what I was doing.” But it might be nice for someone to give a bit more feedback in between about how we're actually doing, 'cos sometimes you can feel very alone.
Peer interaction was an important informal source of information for both learners and physicians because it provided a benchmark, a measure against which to calibrate one's own performance. Learners described discussions with peers over lunch or after work to compare performances; likewise, physicians valued informal interactions. For instance:
And so you bat back and forth with your colleagues, “Oh, I didn't know you did that. Oh, I should start doing that.” I find lots of information we give to each other and then you reflect, “Maybe I should start thinking about doing that more….”
Whereas learners sought or accessed feedback from supervisors, physicians reported few sources of direct feedback. Some reported relying only on broad measures to inform their self-assessments, as this participant described:
I see enough patients over and over again. I have enough who've been with me for 25 years, I think I must be doing something right.
Whereas participants reported that structured learning and assessment activities could inform self-assessments, they also noted that informal interactions with patients, peers, students, coworkers, and supervisors raised awareness about gaps in knowledge or performance when these other people simply asked questions or shared their views.
Internal sources of information.
“The self,” especially one's emotions and internal states, seemed instrumental in informing self-assessment. Participants described positive states including confidence and certainty, and negative ones such as uncertainty, anxiety, and fear. Physicians often referred to a vague sense of something wrong or undefined feelings of discomfort as an indicator of something missed, as one participant explained:
I think as you grow older in practice, you get a lot of hunches…. I don't think they're real guesses, I think it's sort of a subconscious experience being stored … and sometimes you come to conclude there is something wrong with this person and you don't know what.
Emotions could be a particularly powerful source of information. In the following quote, a family physician describes how fear, arising at not knowing what to do when called in the middle of the night regarding a delivery, informed his self-assessment, with the response that he eliminated from practice an area in which he no longer felt proficient:
One self-assessment I encountered is basically fear. In the days that I did deliveries, one night I got phoned from the case room. They [described the situation] and I did not know what to do at that moment. I said, “It is time for me to get out of this.” That was fear.
Interpretation of information
Participants reported interpreting and reflecting on both external and internal data. They described critically reflecting “in action,” that is, during an activity or throughout the day, as an ongoing self-monitoring activity. An experienced physician reported,
I think we do a lot of it without thinking of it as reflection. We do it every day when we look at a patient's chart. You look back and see the last visit, “What did I do, or should I have done something different?” I mean that's reflection, but yet I wouldn't have thought of that as self-assessment or self-reflection, but we do it dozens of times a day.
They also described reflecting “on action,” that is, after a task was completed, at the end of the workday, or at night, like the physician who said,
When I go to sleep I think, “Oh god! I should have done this.”
Although some physicians described critical reflection as integral to their practice, others noted that not all physicians intentionally monitor their performance. One observed,
Some people end up in ruts, they sorta stop thinking.
Participants offered reasons for this, including high-volume practices not conducive to providing optimum care, lack of time to consciously reflect, and a tendency to rely on what is familiar in high-pressure situations rather than incorporating new information into practice.
Alternatively, some learners and physicians described the usefulness of structured learning activities in encouraging, even forcing, them to reflect on their performance, accept feedback, and seek ways to improve. One learner's insights were
At first I thought it [self-assessment] was something you did automatically … you'd think about what's happened. But in actual fact if something goes badly you don't wanna think about it. You know, taking blood with this patient went really badly, I didn't get the blood and there was a mess. I just don't want to think about it. Whereas if you're forced to do portfolio it makes you think about what went wrong and try and find solutions.
Interpretation of one's performance may also require collecting, weighing, and synthesizing information from multiple sources, as noted by a participant:
[In the inpatient setting] … you're interacting with multiple groups of people—with people that know less than you, know more than you, your peers. And synthesizing that information, doing different things and watching what happens to the patient. You are kind of gathering, well—“The attending feels this way, the fellow feels this way, the intern thinks this, and I think this.” And you see how the patient goes through that. And based on that you kind of see what the outcome of your actions was. Was the attending right? Was I right? Was there not a right answer and the patient was sick either way?
Building confidence seemed critical for learners and was intertwined with their interpretations of how they were doing. They spoke of feeling confident in a particular activity as indicating satisfactory performance and viewed lacking confidence as indicating a need for improvement. Alternatively, physicians spoke of the need to admit not knowing and their vulnerability to error, as one remark exemplifies:
If at the end of the day, if you haven't asked yourself some questions, then that's a problem. If you go through the whole day and think you know all about all the problems that faced you, then you don't know. You have to ask yourself why you didn't because you can't know everything.
Responses to information
Participants' responses to interpretation of data varied. When feedback disconfirmed their own self-assessments, being either more positive or more negative, some participants ignored or rejected this information. They gave varied reasons for this—for instance, lack of credibility of data from supervisors who rarely observed their learners, or in the case of physicians, attributing the disconfirming feedback to circumstances beyond their control. One participant, for instance, noted,
I flunked [the insurance company audit] because my population is unique and I'm by myself and have paper charts.
Others described initial negative emotional responses followed by reflection and processing, leading to an integration of this information with self-assessments and acceptance that improvement was needed. In particular:
Somebody gave me a “below-standard” score for something, and at first you're like, “Oh, my life, I can't believe it. I'm not fit to be a doctor!” And then you kind of, you think about it and it actually makes you think, “Yeah, okay, I am weaker in that area, I need to work on it.” And it does kind of give you a care really to be like, “Okay, in the future that's something that I'm gonna have to do.”
Others seemed quite matter-of-fact about receiving negative feedback, assimilating it into their own self-assessment, and using it to improve. A student commented,
If you get negative feedback … then you're going to look out for more opportunities to do that again, until you're satisfied in your own way of doing it, until you get positive feedback.
For learners and physicians, feedback confirming or validating one's present level of performance was valued and sought. Particularly for learners it seemed to aid learning and build confidence, as one student indicated:
So I called the boss [supervisor] again and said, “Okay this was the problem, I did this, this and this, was that right?” I was very pleased that she said “Good that you make this call” and “Yes, you did the right thing for this patient.” That really makes me feel like “Okay, this was a good way of … learning.”
However, interpreting external feedback and responding to it was not always straightforward. It was complicated, for example, by varying individual views of appropriate practice for one participant:
I think you need that feedback from them saying, “Okay, you can do it now.” And if you can go back to the same senior it's a bit more useful than seeing different seniors that kind of go, “Oh, you do it completely different.” That's the only bad thing about it really, you get different seniors going, “Oh, I've not done it that way before, you're doing it completely wrong.” When you're not doing it wrong, just everybody does it their own way, and you've just gotta build your own way of doing it and then you get a senior to go, “Yeah. Okay. You've done it, bit different to mine, but that's fine, you can do it.”
Other responses to accessing and interpreting data included moving beyond the present standard of performance to a higher level of achievement. As this physician described,
I was able to expand my scope of care in [a specific specialty area] by having some feedback. So I now know the first one or two steps before I need to refer.
External and internal conditions
Participants described four interrelated categories of conditions influencing the availability, interpretation, acceptance, and use of information to inform self-assessment: (1) learning/practice climate, (2) relationships with others, (3) credibility of information processes and information, and (4) personal attributes.
The learning/practice climate encompasses the medical culture and the specific clinical setting, and it exerted varying positive and negative influences. For example, both physicians and learners described the value of collaboration and teamwork. Physicians in practice-based learning groups described how collaboration facilitated their asking questions, sharing knowledge, seeking help, and problem-solving together, thus increasing available information, receptivity, and engagement. Learners assigned to specific clinical teams compared this experience with assignment to general ward-based rotations and observed that they were more active, were more engaged in patient care and their own learning, had better relationships with supervisors, and received more feedback.
Participants at all levels reported relationships with others as critical influences. Importantly, the quality of the relationship was a critical factor. Trust, respect, and nonjudgmental attitudes fostered self-assessment at all levels by creating a safe environment where questions could be asked, knowledge gaps revealed, and mistakes discussed with openness. A physician's comments are illustrative:
Because there is that level of trust within the group, I don't mind my peers knowing that I might not know the answer to something. Because I don't feel they would judge me by the fact I don't know the answer to something that comes up as part of this.
Physicians' practice settings (e.g., group versus solo, academic/hospital versus community) moderated opportunities for professional and social relationships and collegial interaction. Physicians and learners viewed isolation as a “red flag”; that is, working alone limited access to external information. Diversity in relationships provided access to a wider range of knowledge and experience for comparison purposes, providing richer information to process in self-assessment. A physician commented,
Anytime that you have to interact with a colleague, be it at a consultant level, personal, group practice level or as a teacher, that always provides feedback. It is a general amalgamation of all those things that gives you an assessment of how you are doing.
Alternatively, the desire to maintain comfortable relationships may result in reluctance to provide colleagues with critical feedback necessary for improvement.
You know you have to be comfortable enough with the person to say “I don't agree with you.” And you know, there's a lot of friendships…. “I can talk to you now, but we still have to work with each other another year, another three years,” there's a line that you can come to [between friendship and giving critical feedback].
For all, the perceived credibility of feedback processes influenced acceptance of the feedback. For some, the structure of specific assessment processes lacked credibility, and participants perceived these processes to be of little real value, as this comment illustrates:
Reflective learning is just … another hoop that you've gotta jump through.
For learners, feedback from supervisors was especially valued if based on direct observation over a period of time by an interested and skilled provider. They described credible and effective feedback as being constructive, specific, descriptive, and timely. Learners sometimes perceived peer feedback as more credible than that from supervisors, as one student noted:
Your clinical [learning] partner … they know your strengths and weaknesses and they're probably gonna give you quite useful feedback. Maybe a consultant watched you one time, they might not have the whole picture of what you're capable of and what you normally do, so sometimes I think it is useful to have feedback from a clinical partner.
The final category of conditions moderating participants' seeking, interpreting, and using external and internal data was personal attributes. These included emotions and characteristics such as motivation, confidence, curiosity, engagement, mindfulness, and self-directedness. As noted in the above discussion of internal sources of information, personal attributes and emotions could be a source of information. They also appeared as moderators and influences of one's ability to self-assess. The physician below describes how being mindful, that is, being attentive to what one is saying and feeling, positively influences one's ability to self-assess accurately and identify gaps in knowledge or practice:
You are sometimes aware when you are talking to patients that perhaps you are fudging the issue a little bit or you are filling in gaps that really shouldn't be gaps, that kind of thing. I think we all do that to come across as being coherent, but one should be able to recognize when one is doing that, and go and fill in the gaps. That's part of it too, is recognizing when there is a gap.
Participants' descriptions of their self-assessment experiences revealed multiple tensions. These emerged from complex interactions among internal and external data and conditions influencing acceptance and use of those data. Like other components of informed self-assessment, they appeared as contextual, dynamic, and variable and prevailed across participant groups. We identified three categories of tensions: between people, within people, and in the learning/practice environment.
Tensions existing between people arose generally from conflict between wishing to learn and improve and wishing to seem knowledgeable and confident to one's supervisors and colleagues. For example, relationship safety and openness were sometimes at odds and limited one's opportunity to acknowledge knowledge gaps. A physician commented,
There is this sense of intimidation that you don't want to speak up in case you might just be deficient in your medical knowledge.
The second category, tensions within people, often arose from discord between one's own assessment of competence and that conveyed by incoming information. For example, a student mused,
Thinking of how I function by myself [without feedback] is always different than how others think of me.
The third category encompassed tensions in the learning/practice environment. Learners described discordances between the formal curriculum and assessment processes, and what they perceived and experienced in clinical placements. For example, there was tension between giving and receiving feedback seriously and honestly versus “playing the evaluation game.” A postgraduate learner reported,
Some supervisors will say, “Oh, what do you want me to, what shall I put down for you?” That's what you get asked, and they just tick the boxes and it's meaningless.
The overarching tension seemed to exist between accessing and receiving external feedback for the purpose of improvement, and preserving and maintaining one's self-image as a learner or practitioner. It was evidenced by wanting feedback yet fearing negative information. Physicians spoke of an initial hesitancy to participate in structured assessment activities, explaining that feedback, although wanted, creates anxiety because of its potential threat to self-perceptions. One comment is illustrative:
Physicians like to get feedback. I think they are always a little worried about negative feedback. If it is going to be negative, how negative is it? But, you have to face it sometime.
Learners generally wanted feedback but noted that a degree of confidence in one's performance was requisite to being prepared to ask for and receive it. Tension existed between knowing one needed feedback to learn from one's mistakes and lacking the confidence to ask for that feedback. An illustrative comment is
When you are more confident, you can ask for more detailed feedback.
—Undergraduate learner H9
Viewing self-assessment as a set of processes informed by external and internal data enables consideration of its potential contributions to learning, assessment, improvement, and self-regulation.13 To this end, we explored how undergraduate/postgraduate learners and physicians perceived and used self-assessment in clinical settings; and through this, we identified specific activities informing self-assessment and the multiple influences on those activities. We then generated a model depicting components of informed self-assessment, a flexible, phased process of accessing, interpreting, and responding to external and internal data that requires balancing competing internal and external data and conditions. Central challenges to informing self-assessment are the dynamic interrelationships and underlying tensions within and among the components.
We attempted to depict the diversity and fluidity of the components. However, in a two-dimensional model (Figure 1), it is difficult to transmit the essence of the “tangled web” of self-assessment. Components are not static, and relationships are not linear and predictable. Realizing this can help us understand why self-assessment accuracy seems to be frequently unreliable.1–4 In a paper describing how young doctors make decisions about whether or not to call a senior in situations of clinical uncertainty, a similar complex process is described through the metaphor of a child's mobile gently and unpredictably moving in the breeze.29–30 Likewise, self-assessment is a multidimensional construct in which actions and reactions cannot, at least as yet, be predicted.
On a more specific level, this study reinforces important findings from prior research and suggests new directions. Foremost is the requirement for clear, timely, specific, constructive feedback, preferably offered by trusted, credible supervisors in a safe environment, to inform learners' self-assessment.11,31–35 We found this often lacking. Perhaps to replace missing feedback from supervisors, learners sought feedback from peers to inform their progress and self-assessments. Physicians reported rarely receiving formal feedback and frequently using peer interaction and feedback to inform their self-assessments. Hence, it seems that quality matters in terms of that peer network and the feedback that peers provide.27,36–40
Critical and deliberate reflection on assessment and other external and internal data appeared as a key activity of the interpretation phase of self-assessment. Learning much more about critical reflection—how and when it functions, its rigor, and its universality among learners and practitioners—seems to be important to understanding self-assessment and how it is informed.14,41–45 Similarly, understanding the contribution of “self,” that is, one's emotional responses, internal states, and personal characteristics to self-assessment, seems central.13,46
Tensions seemed integral to and inevitable within self-assessment. There were tensions between wanting to know how one is doing and fear of learning one is not doing as well as one should, between wanting to learn and improve and the anxiety of disclosing that one does not know, and between the potential for learning from structured assessment activities and the loss of that potential when these activities are perceived as not credible. Tensions often seemed linked to features of clinical and educational environments, perceived professional expectations, and the hidden curriculum of medical education.13,47–51 It seems that much needs to be done to address these factors before we can make great gains in understanding self-assessment and its relation to self-directed learning and professional self-regulation.
This study was intended to be comprehensive and provide an overarching view of self-assessment and how it is informed across the education continuum and multiple locations. It has helped us to identify, understand, and conceptualize interrelationships among the components of informed self-assessment. Each component now requires further research in its own right. The study's main limitation is the relatively small number of participants from each program and level of education. Paradoxically, this is also its strength as the model is grounded in evidence from diverse participants at various points on the medical education continuum. The need now is to identify and examine differences among groups and to explore each of the components of informed self-assessment as perceived by larger groups of learners and practitioners at various levels, including those not formally engaged in structured self-assessment activities, to confirm, further explain, and question the findings to date.
The authors thank Mieke Embo (University College Arteveldehogeschool, Ghent, Belgium) for facilitating participation of students at the Arteveldehogeschool; Lonneke Bokken, PhD (University of Maastricht, Maastricht, the Netherlands), for research support and cofacilitation of the Maastricht and Ghent focus groups, and for verifying those transcripts; and Tanya Hill, MSc (Dalhousie University, Halifax, Nova Scotia, Canada), for providing administrative support and reviewing the manuscript.
The Medical Council of Canada (MCC), the American Board of Internal Medicine (ABIM), and Dalhousie University, Faculty of Medicine, Office of Continuing Medical Education provided funding for the study. The following participating organizations provided in-kind support: Office of CME, Faculty of Medicine, University of Calgary, Canada; Faculty of Medicine, Manchester University, Manchester, United Kingdom; Department of Educational Development and Research, University of Maastricht, Maastricht, the Netherlands; and University College Arteveldehogeschool, Ghent, Belgium.
Ethical approval was granted by the Dalhousie University Health Sciences Research Ethics Board, Halifax, Nova Scotia, Canada; the University of Calgary Conjoint Health Research Ethics Board, Calgary, Alberta, Canada; the New England Institutional Review Board, Wellesley, Massachusetts; and the Stockport Research Ethics Committee, Stockport, Greater Manchester, United Kingdom. The Research Ethics Committees of the University of Maastricht, Maastricht, the Netherlands, and the University College Arteveldehogeschool, Ghent, Belgium, did not require ethical approval.
1Eva KW, Regehr G. “I'll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof. 2008;28:14–19.
2Eva KW, Regehr G. Self-assessment in the health professions: A reformulation and research agenda. Acad Med. 2005;80(10 suppl):S46–S54.
3Regehr G, Eva K. Self-assessment, self-direction, and the self-regulating professional. Clin Orthop Relat Res. 2006;449:34–38.
4Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence. JAMA. 2006;296:1094–1102.
5Kruger J, Dunning D. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77:1121–1134.
6Ross L, Nisbett RE. The Person and the Situation: Perspectives of Social Psychology. New York, NY: McGraw-Hill Inc.; 1991.
7Colthart I, Bagnall G, Evans A, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach. 2008;30:124–145.
8Galbraith RM, Hawkins RE, Holmboe ES. Making self-assessment more effective. J Contin Educ Health Prof. 2008;28:20–24.
9Sargeant J, Mann K, van der Vleuten C, Metsemakers J. “Directed” self-assessment: Practice and feedback within a social context. J Contin Educ Health Prof. 2008;28:47–54.
10Musolino GM. Fostering reflective practice: Self-assessment abilities of physical therapy students and entry-level graduates. J Allied Health. 2006;35:30–42.
11Boud D. Enhancing Learning Through Self-Assessment. London, UK: Kogan Page Limited; 1995.
12Boud D. Reframing assessment as if learning were important. In: Boud D, Falchikov N, eds. Rethinking Assessment in Higher Education. New York, NY: Routledge; 2007:14–25.
13Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: A challenge for medical educators. J Contin Educ Health Prof. 2008;28:5–13.
14Mamede S, Schmidt HG. The structure of reflective practice in medicine. Med Educ. 2004;38:1302–1308.
15Topping K. Self and peer assessment in school and university: Reliability, validity and utility. In: Segers M, Dochy F, Cascallar E, eds. Optimising New Modes of Assessment: In Search of Qualities and Standards. Dordrecht, Netherlands: Kluwer Academic Publishers; 2003:55–87.
16Corbin J, Strauss A. Basics of Qualitative Research. 3rd ed. Thousand Oaks, Calif: Sage; 2008.
17Liamputtong P, Ezzy D. Qualitative Research Methods. 2nd ed. Victoria, Australia: Oxford University Press; 2005.
18Driessen E, van Tartwijk J, Vermunt JD, van der Vleuten CP. Use of portfolios in early undergraduate medical training. Med Teach. 2003;25:18–23.
19Bachelor of Midwifery, College Arteveldehogeschool, Gent, Belgium. Arteveldehogeschool Web Site. Available at: http://www.arteveldehs.be/emc.asp?pageId=1857
. Accessed December 19, 2009.
20Dornan T, Brown M, Powley D, Hopkins M. A technology using feedback to manage experience based learning. Med Teach. 2004;26:736–738.
21Davies H, Archer J, Southgate L, Norcini J. Initial evaluation of the first year of the Foundation Assessment Programme. Med Educ. 2009;43:74–81.
22The Foundation Programme. The Rough Guide to the Foundation Programme. 2nd ed. Available at: http://www.foundationprogramme.nhs.uk/pages/home/key-documents#rough-guide-to-the-foundation-programme
. Accessed December 19, 2009
23Duffy FD, Lynn LA, Didura H, et al. Self-assessment of practice performance: Development of the ABIM Practice Improvement Module (PIM(SM)). J Contin Educ Health Prof. 2008;28:38–46.
24American Board of Internal Medicine. Improve your practice with PIMs. Available at: http://www.abim.org/pims/default.aspx
. Accessed December 19, 2009
25Violato C, Lockyer J, Fidler H. Changes in performance: A 5-year longitudinal study of participants in a multi-source feedback programme. Med Educ. 2008;42:1007–1013.
26Physician Achievement Review Program Web Site. Available at: http://www.par-program.org/PAR-Info.htm
. Accessed December 19, 2009.
27Armson H, Kinzie S, Hawes D, Roder S, Wakefield J, Elmslie T. Translating learning into practice: Lessons from the practice-based small group learning program. Can Fam Physician. 2007;53:1477–1485.
28The Foundation for Medical Practice Education. Small group information. Available at: http://www.fmpe.org/en/programs/pbsg.html
. Accessed December 19, 2009.
29Stewart J. To call or not to call: A judgement of risk by pre-registration house officers. Med Educ. 2008;42:938–944.
30Sargeant J. To call or not to call: Making informed self-assessment. Med Educ. 2008;42:854–855.
31Rees C, Shepherd M. Students' and assessors' attitudes towards students' self-assessment of their personal and professional behaviours. Med Educ. 2005;39:30–39.
32Sargeant J, Mann K, Sinclair D, van der Vleuten C, Metsemakers J. Challenges in multi-source feedback: Intended and unintended outcomes. Med Educ. 2007;41:583–591.
33Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(suppl 10):S70–S81.
34Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77:81–112.
35Shute VJ. Focus on formative feedback. Rev Educ Res. 2008;78:153–189.
36Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29:855–871.
37Sluijsmans DMA, Brand-Gruwel S, van Merriënboer JJG, Bastiaens TJ. The training of peer assessment skills to promote the development of reflection skills in teacher education. Stud Educ Eval. 2002;29:23–42.
38Shue CK, Arnold L, Stern DT. Maximizing participation in peer assessment of professionalism: The students speak. Acad Med. 2005;80(suppl 10):S1–S5.
39Arnold L, Shue CK, Kritt B, Ginsburg S, Stern DT. Medical students' views on peer assessment of professionalism. J Gen Intern Med. 2005;20:819–824.
40Sargeant J, Mann K, Sinclair D, et al. Learning in practice: Experiences and perceptions of high-scoring physicians. Acad Med. 2006;81:655–670.
41Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: A systematic review. Adv Health Sci Educ Theory Pract. 2009;14:595–621.
42Sargeant J, Mann K, van der Vleuten C, Metsemakers J. Reflection: A link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract. 2009;14:399–401.
43Schon DA. Educating the Reflective Practitioner. San Francisco, Calif: Jossey-Bass Inc.; 1987.
44Van Merrienboer JJG, Sluijsmans DMA. Toward a synthesis of cognitive load theory, four-component instructional design, and self-directed learning. Educ Psychol Rev. 2009;21:55–66.
45Prins FJ, Sluijsmans DM, Kirschner PA. Feedback for general practitioners in training: Quality, styles, and preferences. Adv Health Sci Educ Theory Pract. 2006;11:289–303.
46Epstein RM. Mindful practice. JAMA. 1999;282:833–839.
47Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69:861–871.
48Sargeant J, Mann K, Sinclair D, van der Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract. 2006;13:275–288.
49Lowe MM, Bennett N, Aparicio A; American College of Chest Physicians Health and Science Policy Committee. The role of audience characteristics and external factors in continuing medical education and physician change: Effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135(suppl 3):56S–61S.
50Boor K, Scheele F, van der Vleuten CP, Teunissen PW, den Breejen E, Scherpbier AJ. How undergraduate clinical learning climates differ: A multi-method case study. Med Educ. 2008;42:1029–1036.
51Teunissen PW, Scheele F, Scherpbier AJ, et al. How residents learn: Qualitative evidence for the pivotal role of clinical activities. Med Educ. 2007;41:763–777.