The concept of self-assessment remains a challenge, as evidenced by a literature of contradictions. Its importance as a foundation of self-regulation is widely embedded in statements of educational bodies,1,2 yet research findings concerning individuals' ability to assess their own performance accurately have consistently revealed widespread flaws.3–5 The literature suggests that, to be valid, self-assessment must be informed by external information from a variety of sources.4,6 Others have reported that although feedback may be sought and provided, several factors influence its acceptance and utility in improving performance.7
We use the term informed self-assessment to describe the set of processes through which individuals use external and internal data to generate an appraisal of their own ability. Although we use that term in this report to distinguish it from other conceptualizations of self-assessment,8 conceptualizing self-assessment as an activity that must be informed by external information is not novel. Writing in the wider education literature, Boud9 described self-assessment as a process that requires drawing on both internal and external data about one's performance and comparing these with a standard to make a judgment about one's performance. The process is aided when engaged facilitation guides learners in interpreting and using data appropriately.1,6,7 Epstein et al10 similarly described performance self-assessment within clinical settings as an externally and internally informed process of interpreting data about one's performance and comparing it with an explicit or implicit standard. Difficulty remains, however, with understanding the interplay between external data and internal perceptions and which contextual variables influence the relative dominance of one form (or source of data) over the other. This is an important issue because simply stating that there is a need to inform self-appraisal with external data underrepresents the extent to which the response to such data will be guided by the self-appraisal they are meant to inform.
Our interest in informed self-assessments as an integral part of professional development and practice, combined with awareness of the contradictions and complexities surrounding its effectiveness, led to a multinational study11 of informed self-assessment that explored the external and internal forces at play.8,12 That study aimed to understand how learners and practicing physicians perceived and used informed self-assessment in clinical learning and practice. A model of informed self-assessment emerged, consisting of five dynamically interacting components; sources of information, interpretation of information, responses to information, external and internal conditions influencing each of these steps, and tensions arising from competing data and influences.11
The purpose of our present report is to present findings resulting from in-depth exploration of the tensions component of the informed self-assessment model described above. We describe the nature, role, and effects of tensions in informed self-assessment, and situate those findings in relation to current understandings of learning.
This qualitative study employed principles of grounded theory.13,14 We held 17 focus groups (January to May 2008) involving learners at the undergraduate and postgraduate levels, as well as practicing physicians, from eight medical education programs in five countries. We used purposive sampling, believing that participants experienced with structured self-assessment activities would be more conversant with the concepts and provide richer data. Thus, we selected programs available to us that used formal curricular activities to promote reflection and inform self-assessment about clinical performance. These formal activities included portfolios (UG, PG), personal learning plans (UG, PG), defined outcome competencies (UG), multisource feedback (PG, physicians), audit and feedback (PG, physicians) and facilitated practice-based small-group (PBSG) learning with long-standing groups (physicians). Table 1 lists the participating programs and their formal self-assessment activities. Administrators of the eight identified programs invited participants in their respective programs by letter to participate in one of two 1.5-hour focus groups for their program. Participation was voluntary; no compensation was provided. Physicians in the two PBSG groups were invited to participate as a group, as they function in an integral way; all other invitations were individual.
Semistructured interview questions addressed participants' perceptions of the ways in which they self-assess their clinical performance generally, the data they draw on, the contributions of formal and informal activities and resources for informing their self-assessments, and the factors influencing their usefulness. In the grounded theory tradition, the interview questions were revised as analysis progressed to explore and clarify emerging concepts. One member of the research team (J.S.) facilitated all group discussions, with assistance from at least one other team member who was familiar with each program's self-assessment activities and context. The cofacilitator assisted by clarifying questions or comments, participating in the discussion to enable greater exploration of concepts if required, and recording field notes. Discussions were audio-recorded and transcribed.
Initial themes were identified by reading and discussing the transcripts, and were expanded, revised, and modified through subsequent and iterative analysis, discussion, and interrogation of the data over a period of about 18 months. Two team members (J.S., E.L.) independently coded the first transcript and compared their interpretations of themes, which were then reviewed and verified by another team member (J.L.) and then through group discussion. Each transcript was read and discussed by at least three other team members in a series of teleconferences, continually informing revision of the coding structure. One team member (E.L.) computer-coded the data. The team met face-to-face for two days to critically discuss analyses of related data conducted by team subgroups, resulting in a preliminary conceptual diagram of informed self-assessment that integrated the major themes. Themes were interpreted through constant comparison and their interrelationships explored. Team subgroups refined the components of the conceptual model, verifying that they were grounded in the data. Matrices, theoretical memos, and conceptual mapping aided this process. Discussion and reexamination of transcripts and coded data were used to resolve differences of interpretation.
The research ethics boards of all participating institutions approved the study.
In all, there were 134 participants, including 53 undergraduate learners in three programs (in the United Kingdom, The Netherlands, and Belgium), 32 postgraduate learners in two programs (in the United Kingdom and the United States), and 49 practicing physicians in three programs (in Canada and the United States).
The central category,13 or theme, of tensions in informed self-assessment became apparent during the analysis as the team worked through analyses of the other four components of the model (sources of information, interpretation of information, responses to information; and conditions influencing these). Although each aspect of the model interacted dynamically with other components, the tensions component was notable, as it pervaded all other model elements and was present across all levels of learners, including physicians.
The tensions that emerged, while influencing one another, can be grouped as follows: tensions within self, tensions between people, and tensions inherent in the environment. A summary list of the tensions we identified is shown in Table 2. In the following paragraphs, quotations illustrating the concepts are labeled with an alphanumeric code. Each letter represents a specific program; the number is the code assigned to a specific participant.
Tensions within self
The process of informing one's self-assessment was accompanied by two ongoing internal tensions: (1) wanting feedback yet fearing disconfirming information, and (2) recognizing the need to use feedback yet struggling to do so because of incongruence with one's self-appraisal.
Feedback was perceived by learners across all groups as essential to knowing how one was doing and how to improve. A postgraduate participant commented:
[T]o count on yourself to say what you think you're doing well—I mean that's an important part of the exercise. But it's really not the most important. The most important part is getting some other feedback that's realistic and not cushy. (Q8)
Nonetheless, for learners at all levels, wanting feedback was always in tension with fearing feedback information that would be negative or critical about their performance. The quotes below from postgraduate and undergraduate learners illustrate this tension.
It's, like, are you scared about being unusual, like, are you going to be commended for trying to bring up something unusual or hard, or are you too scared of being marked for not managing it, as it should have been. (Postgraduate O7)
H2: But I think it's also important to take your own initiative. You should do it.
Facilitator: You should do what?
H1: Ask for feedback and say, “Is this what we should do [for the patient] now?”
H7: Yes, but then they are really negative, then they really think like “Oh, my god, there she is again.” That can also be a result of it. (Undergraduates)
The second tension within self concerned overcoming the difficulty created by feedback that is incongruent with the individual's self-appraisal, thus making even feedback that is received and valued difficult to hear, accept, and use to improve. Disconfirming feedback required reconciling the information received with one's own view of one's performance and often required moving past the emotional reactions that disconfirming feedback can evoke. Physicians relate below how disconfirming feedback can elicit shock and tension and, eventually, result in changed self-perception.
J4: I always thought that I was very good at communicating. And I got this feedback from the families [and] the nurses working in long-term care, [they] thought ... that I wasn't all that good at communicating with the family of my patients in long-term care. I was just blown away! ... [I]t really made me think, well, I'm not cluing in.
Facilitator: It sounds [like] that was feedback ... you could incorporate ... into what you were doing.
J4: Yeah. At first I thought, Who are these guys? I know what I'm doing. I thought about it like that....
J3: I think you are right. There is this shock when negative [feedback] comes back....
J2: But [that] is the point, right? ... [Y]ou really do get that bit of a shock. But enough that you delve into it in your brain a little bit more in terms of what would be the potential explanations and/or what can I do differently.
Tensions between people
The tensions between people highlighted the relational nature of informed self-assessment. An overarching tension was the need to be autonomous, yet to be part of a network or group of colleagues. Within this theme, three specific tensions focused on factors influencing the seeking, provision, and receipt of information about one's performance. These tensions overlapped with and extended the tensions identified within self, and included (1) wanting to be able to question others and learn from feedback, yet not wanting to look incompetent or share areas of deficiency, (2) wanting feedback, yet not being able to pursue it or not trusting feedback that is received, and (3) needing a positive and safe relationship to give and receive feedback, yet worrying about damaging the relationship with genuine feedback.
The need to be autonomous yet also to be part of a network or group of colleagues was a tension expressed at all levels of learning and practice. For practicing physicians, who we found often lacked access to formal feedback based on explicit standards to assess their practice,15 a particularly strong theme was the value of having colleagues with whom to discuss difficult patients, review actions, learn with and from, and whose practices could serve as a mirror or check for one's own practice. They viewed these interactions as providing valuable feedback, thus informing self-assessment and providing opportunities for improved patient care. A physician explained:
What we see among [ourselves when working together] really tells us what is going on. When we see things happening in emergency—our patients are getting the wrong diagnoses or coming back for the second time—you are missing something. You talk to your colleagues, you find out what happened, and you say, “Geez, what happened here? Why am I not doing this test? What is holding me back?” Looking among your peers to help you change. We use that a lot here. (Physician M3)
An undergraduate learner expressed a similar view of the value of peer interaction:
There is one student at one side of the ward and [another student] at the other side of the ward. We often talk to each other, how you experienced this or how did you do things. And it really helps you to reflect on yourself. Like, “Do I do things well, am I a normal student?” or “Am I very stupid?” It helps you to [share] experience. (Undergraduate I5)
At the same time, however, tension emerged between wanting to question and learn and yet not wanting to look incompetent. Physician participants spoke of how difficult it was to admit to others that they did not know something. Students and residents spoke of reluctance to ask questions. They described feeling inadequate in such situations compared with peers who conveyed an attitude of superior knowledge. Some participants seemed to have resolved this tension with maturity and, importantly, had cultivated a group of trusted colleagues or consultants who could provide help with patient care and inform self-assessment in a safe and trusted manner. As one physician participant, a long-term member of a formalized continuing education small group, described:
[W]e can say anything to anybody. Whereas before in ... my first few years out, I just found I was a little more reserved and not too forthcoming with information. Didn't want to look like you were missing something, or didn't want to look like you were not knowing something you should or this type of thing. And now, I'll ask anybody what seems like the stupidest question, and “Gee, I don't know that. Well, tell me this, if I don't know something....” It is just that overall comfortable feeling about knowing that we are all in this together and we are all sort of at the same level and working through our problems and how can we even go on from there.... I think it has been a very valuable experience of learning, a way of learning much more valuable than I would have done any number of years prior to that. (Physician M4)
Yet, even within the group, with well-known and trusted colleagues, the seeking of data to inform self-assessment could be left unspoken. Individuals noted inwardly to themselves the need to reconsider an approach or update their knowledge as they compared their knowledge with others'. A member of a PBSG noted:
But we are physicians, so I think whenever we hear something we have our own internal feedback often.... Like, “Gosh, I didn't think about that.” Or, “My learning base isn't strong enough in that”... You don't know what you don't know, right?... Until somebody knows it and you say, “I don't know that.” (Physician M1)
Tensions between people also resulted from the desire to seek feedback that could potentially inform self-assessment, yet being precluded from doing so. For learners, feedback from supervisors was especially challenging to obtain and was not received as frequently as desired. Asking for feedback from supervisors runs the risk that one will be perceived as not performing autonomously enough, as a resident related:
You know, you can hound people and say, “So, how do you think I did? How do you think I did?” [Laughter] But after a while they get a bit annoyed with you. So you know there has to be a balance between just doing things and getting feedback. (Postgraduate D6)
However, when feedback was received, it was sometimes not perceived as credible, especially if the supervisor had not actually had the opportunity to observe the performance on which the assessment was needed or to observe the supervisee closely enough over time. Referring to consultants' feedback, an undergraduate remarked, “[T]hey've not known enough about me to give any sort of feedback that I would value” (C6). Although feedback was desired, feedback that lacked credibility was not used.
The third tension between people concerned the nature of relationships, their influence on the provision of feedback, and the implications of feedback for the ongoing relationship. Feedback from other colleagues, be they peer learners, practice colleagues, or specialists, was definitely thought to be enhanced when it occurred in a relationship where mutual trust and respect were present. These characteristics had often formed because of familiarity and relationships of long duration. Such feedback was perceived as intended to be helpful and credible. Paradoxically, however, trust and respect in the relationship might lessen the likelihood of receiving constructive feedback: Feedback-giving sometimes was tempered by the desire not to harm or disrupt the relationship. A resident observed, “There's a great deal of sensitivity to ... people's feelings. And that sometimes gets in the way” (Postgraduate Q1).
In fact, however, the failure to give feedback could be perceived to be as damaging to the relationship as the act of giving feedback. For example, participants described situations where a coworker (e.g., a nurse) declined to provide feedback through multisource feedback. This reluctance was perceived by participants as indicating potentially negative feedback, raising questions about what standard(s) they had failed to meet: Absence of direct feedback increased, rather than diminished, the tensions between people.
Tensions in the learning or practice environment
Individuals do not operate separately from the environment they are part of; the influence of the learning environment on processes of informed self-assessment was expressed in two tensions: (1) tensions between the stated curriculum and the “curriculum-in-action,” and (2) tensions between engaging in authentic activities that inform one's self-assessment and “playing the evaluation game.”
Learners reported perceiving a tension and some dissonance between the explicit feedback received through the stated curriculum and the implicit feedback received through the curriculum- in-action. This presented as a disconnect between what they as learners were asked to do and what they saw their teachers do. An undergraduate made this comment about an OSCE:
We're, like, doing the proper checklist of everything that we should do, the gold standard, whereas in reality a lot of doctors miss out things because they're too busy or can't be bothered for whatever reason. (C2)
The result was that individuals could not resolve the gaps between espoused and enacted practices. This was further complicated by standards that were inconsistent and by lack of a means to find a stable set of benchmarks. Hierarchical relationships and power dynamics complicated resolving this tension.
A final tension emerged between the benefits expected of seeking to inform one's self-assessment and concerns about the authenticity of self-assessment processes, which suggests that the use of informed self-assessment strategies can be helpful or counterproductive depending on their implementation and learner engagement. For example, when there were many requirements for reflective activity and/or feedback, some questioned whether it was helpful or a meaningless “game” that had to be played to satisfy others:
I know you have got to play the game, but that's how I always feel when I'm doing it. I'm playing the game. I don't feel like I'm learning and evolving and becoming a better person because I've done it. (Postgraduate E2)
Both prelicensure learners and foundation (postgraduate medical learners in the United Kingdom) trainees described similar examples of tension in being required to create portfolios and engage in formal reflection and self-assessment. The responses revealed tension between forced and spontaneous reflection, and between learning from reviewing one's reflection and a feeling of “doing it for someone else,” exemplified by the quotes below.
[I]t seemed like it was quite forced upon us that we had to reflect, and it didn't seem like something that we'd want to do. It was sort of like “You must do this, otherwise you'll fail your portfolio” rather than, “If you feel like this was significant, then reflect on it and it'll help you.” It was ... I don't know, I'd prefer it if it was more about us. (Undergraduate B8)
So it's just about “Ahh, ok, then, I have to make something up for it”.... The quantity is more [valuable] than the quality. (Undergraduate H7)
[T]he thing that I find difficult about doing reflective practice is ... [you] have to make sure that when you reflect back, you're using the good doctor guidelines and the words they want to hear. (Postgraduate E3)
Yet, despite these tensions, some found that, however reluctantly they had approached keeping a portfolio, they considered it useful for their learning and self-assessment. As one learner commented, “[T]his is what I've kind of woken up to this last couple of months, is that it's good practice for your own benefit” (Postgraduate E3).
We grouped the tensions that emerged as one component of our model of informed self-assessment into three categories: within the individual, between people, and in the learning environment. However, tensions discovered were diffuse, often tacit, and not readily defined or observed in the learning and practice settings, findings similar to those identified earlier in resident and student activities related to feedback-seeking16 and asking for assistance.17,18 The tensions were pervasive and detected across the education continuum and across learning and practice sites and countries from which the participants came. They were also dynamic and interrelated, influenced by multiple personal and contextual factors and, in turn, influencing the processes of acquiring and interpreting external data to inform self-assessment.
Both Teunissen et al16 and Kennedy et al17,18 describe a complex and dynamic interplay of influences on trainees' behavior in feedback-seeking and asking for help. Although not labeled directly as tensions by these authors, the relationships described by Teunissen et al between residents' learning and performance orientations and their seeking of and perceptions of the costs and benefits of feedback seem to echo the tensions that we identified within individuals between wanting feedback yet fearing its content. Similarly, Kennedy and colleagues ' description of the risks to professional credibility in asking for help reflected the tension in our findings between being a learner and being an evolving professional. What is striking is that the tensions these two author groups identified among trainees were also found present to varying extents in practicing physicians in our study.
Although our study did not yield longitudinal data, we were interested to explore whether tensions were sustained across learner and physician stages, and whether they changed over time, were resolved, or simply were no longer relevant. Based on our findings, most tensions did not seem to change over time; moreover, they seemed to require renegotiation each time they were encountered and their impact was undiminished, even with experience. For example, the tension between asking questions to learn yet desiring to appear competent was present afresh each time, and the risks had to be reassessed in the context of each encounter. Last, even though feedback was desired, resolution of feedback that was incongruent with one's self-assessment presented an ongoing challenge to one's professional identity.
Two theoretical perspectives
We sought to understand the robust nature of the tensions, their endurance over time, and their apparent resistance to a permanent resolution through two theoretical perspectives. These were social cognitive theory (SCT), as described by Bandura,19 and situated learning in communities of practice.20,21
Social cognitive theory.
SCT offers some insight into learning at an individual level. First, this theory places central importance on the social nature of learning with and from others; it also posits a continuous dynamic reciprocal interaction between the individual and the learning environment. SCT emphasizes individual agency, viewing individuals as having an innate capability to set goals and to reflect on their performance and progress toward their goals; it also emphasizes the fundamental importance to learning of feedback, particularly its role in helping learners to monitor progress toward their goals.
In our study, participants continually highlighted the tensions inherent in their interactions with the environment and their dynamic nature. They also noted the key importance of feedback in their ability to monitor their progress toward meeting their learning goals and the challenges in obtaining feedback to inform their self-assessment.
Second, SCT also emphasizes the vicarious learning that occurs through observation of others' actions and interactions, along with their consequences. In our study, values such as appearing competent may be reinforced by what learners and physicians observe and see is valued in the profession and in the workplace. Learners may also see that although certain values are espoused, they are not enacted in practice.22 Indeed, as in Kennedy and colleagues'17,18 work through watching interactions in the environment, learners may determine that some feedback-seeking behaviors are risky. As our participants reported, although the importance of feedback was espoused, demonstrated behaviors suggested otherwise, and role models for effective feedback-seeking and provision were few.
Third, the construct of self-efficacy, or the perceptions that persons hold of their ability to execute a specific task or set of tasks, is integral to SCT.19 Although two internal tensions—of wanting feedback yet fearing disconfirming information, and recognizing the need for feedback yet struggling to use it because of its incongruence with one's self-appraisal—were articulated separately, the concept of self-efficacy may underlie them both. Learners are building a sense of self-efficacy in and for the practice setting, which, given its social nature, may set up these tensions.
Situated learning and communities of practice.
A second helpful theoretical perspective is that of situated learning and communities of practice as described by Lave and Wenger20 and Wenger,21 which focuses on learning at a collective as well as an individual level. These theoretical approaches originate in sociocultural theory and are based on the premise that learning is inseparable from the context in which it occurs. In this view, learners may not only acquire the skills to inform their self-assessments but may also acquire the tensions extant in the community.
A key tenet of situated learning is that learning occurs through the learner's participation in, and becoming an increasingly responsible part of, the community. Learners participate in the community's activities and practices, learning both “from talk” and “to talk.” As Bleakley23,24 has noted, the activities and practices, or the discourse, of the community help to create as well as to understand the world. In this case, “talk” in the community may imply that evaluation is a “game,” or that provision of genuine feedback is not valued. Our participants strove to become part of the community and did so through listening and learning how to talk about assessment and how to seek feedback both tacitly and explicitly.
Last, in Lave and Wenger's20 view, newer community members also contribute through their participation, and therefore they may participate in creating and recreating the existing tensions. They may help to explain how certain tensions are perpetuated, and may even become invisible and unspoken.
Together, these two theoretical approaches provide complementary perspectives within which to frame an understanding of both individual and collective construction and negotiation of the tensions in informed self-assessment. They may also hold promise for understanding how to effect change.
Implications of our findings for practice
The fact that tensions permeate education and practice is important. Several implications of our findings may offer insight into self-assessment, improving the quality of feedback and the learning environment. Although speculative, we propose the following conditions that might support the provision of an effective environment for informed self-assessment.
- For tensions within the self, (1) there is explicit recognition of the emotional aspect of receiving feedback, and (2) the inherent difficulty in reconciling self-appraisal with contradictory feedback is recognized.
- For tensions between individuals, (1) teacher–learner and peer–peer relationships are respectful and supportive, (2) feedback that is provided is informed and specific to the learner's needs, and (3) the adoption of feedback that seeks to enable learning is supported.
- For tensions in the learning environment, (1) a climate of mutual and collective learning is supported, (2) there is appropriate modeling of giving and receiving feedback, and (3) benefits of informed self-assessment are made explicit to teachers and learners.
The perceived quality of feedback will likely have an important effect on the likelihood that individuals will use that feedback to inform their self-assessments and improve their practices. Perhaps most critical is the importance of meaningful feedback, related to the learner's needs, and given thoughtfully and constructively with the aim of improving clinical performance. Much emphasis has been placed on improving skills of feedback-giving; less attention has been focused on the skills needed for effective feedback-seeking and -receiving and to the interactive dynamic, relational context in which feedback is exchanged and self-assessment occurs.7,25 Emphasis on these aspects and on the skills required to seek, receive, and process feedback, and to use it for improvement, could significantly diminish some of the tensions expressed in our study. Reflection also serves as a basis both for reflecting on progress and for integrating the content and affective aspects of feedback.26,27 Such skills could empower learners in the learning situations in which they find themselves and would enable greater confidence in seeking feedback.
The work environment embodies the values and culture of the organization, the professionals who work there, and the activities and practices that occur there. Special attention may be required to understand the activities and practices of the specific community setting and the implicit and explicit messages that are embedded in them. This is no small undertaking. Often, the messages deter honest self-assessment and true learning rather that supporting them. The intent of such an exploration would be to increase the frequency, credibility, and safety of seeking and receiving feedback. Increasing safety would mean aligning the explicit messages with those that are unstated. Examples from our study would suggest the importance of making it safe to ask for feedback and of providing careful feedback rather than a cursory comment. Both of these activities would improve the quality of the feedback provided, of the data available to inform self-assessment, and of the teacher–learner relationships. Credible feedback is based on the perception of genuine concern for the learner and relationships of mutual respect. Feedback can occur effectively within a network of relationships that recognize learners as important junior members of the community and support their growth within that community and, similarly, that recognize and support physicians' growth within their community of practice.
A limitation of this study is the small number of programs and participants included in the study, the effect of which may be overcome by the consistency of findings across programs, sites, and level of participant. Rigor of the findings was also contributed to by the research team, who brought diverse theoretical and research perspectives to the study design and analysis and regularly questioned the emerging themes, the data, and their colleagues throughout the analysis process.14
Perhaps understanding the significance of the tensions in informed self-assessment and how they are learned can provide some avenues for improving the educational process in ways that may mitigate some of those tensions. Informed self-assessment should not be seen as an end in itself but, rather, as an essential route to developing, maintaining, and improving performance. The tensions we have described may slow the development of informed self-assessment, which could impede learning and, therefore, ultimately have an impact on the learner's achievement of greater competence.
Understanding and developing interventions at the individual level may provide benefit, but that benefit will be decreased if the larger collective and organizational elements remain unexamined. Creating a culture in which the formal, informal, and hidden curricula about self-assessment are aligned will require a deeper examination of shared norms and values and of the way these are embodied in the discourse of the community and the profession.
The authors thank Mieke Embo, academic coordinator, University College Arteveldehogeschool, Ghent, Belgium, for facilitating participation of students at the Arteveldehogeschool; Lonneke Bokken, PhD (University of Maastricht, Maastricht, The Netherlands), for research support and cofacilitation of the Maastricht and Ghent focus groups, and for verifying transcripts; and Tanya Hill, MSc (Dalhousie University, Halifax, Nova Scotia, Canada), for providing administrative support. We also thank the reviewers of an earlier version of this report, whose careful and constructive comments helped so much to improve the report.
The Medical Council of Canada, the American Board of Internal Medicine, and Dalhousie University, Faculty of Medicine, Office of Continuing Medical Education (CME) provided funding for the study. The following participating organizations provided in-kind support: Office of CME, Faculty of Medicine, University of Calgary, Alberta, Canada; Faculty of Medicine, Manchester University, Manchester, United Kingdom; Department of Educational Development and Research, University of Maastricht, Maastricht, The Netherlands; and University College Arteveldehogeschool, Ghent, Belgium.
Ethical approval was granted by the Dalhousie University Health Sciences Research Ethics Board, Halifax, Nova Scotia, Canada; the University of Calgary Conjoint Health Research Ethics Board, Calgary, Alberta, Canada; the New England Institutional Review Board, Wellesley, Massachusetts; and the Stockport Research Ethics Committee, Stockport, Greater Manchester, United Kingdom. The Research Ethics Committees of the University of Maastricht, Maastricht, The Netherlands, and the University College Arteveldehogeschool, Ghent, Belgium did not require ethical approval.
Abstracts of earlier versions of this article were presented at the Canadian Conference on Medical Education, St. John's, Newfoundland, Canada, May 2010, and at the Association for Medical Education in Europe Conference, Glasgow, Scotland, September 2010.