Secondary Logo

Self-assessment, Self-direction, and the Self-regulating Professional

Regehr, Glenn; Eva, Kevin

Clinical Orthopaedics and Related Research: August 2006 - Volume 449 - Issue - p 34-38
doi: 10.1097/01.blo.0000224027.85732.b2
SECTION I: SYMPOSIUM I: C. T. Brighton/ABJS Workshop on Orthopaedic Education
Free
SDC

One of the cornerstones of autonomy for any profession is the claim to self-regulation. To be effectively self-regulating, the profession generally depends on the individual practitioner to self-regulate his own maintenance of competence activities. This model of individual self-regulation, in turn, depends on the practitioner's ability to self-assess gaps in competence and willingness to seek out opportunities to redress these gaps when identified. The literature relevant to these processes, however, would suggest this model of individual self-regulation is overly optimistic. We review the literature and describe several difficulties associated with the traditionally held model of individual self-regulation. In particular, research demonstrates repeatedly that 1) self-assessment is not an effective mechanism to identify areas of personal weakness and that 2) even when areas of weakness are obvious to the adult learner, we often avoid engaging in learning in these areas because such learning often takes more energy and commitment than we are willing to expend. Implications of these difficulties for the current model of self-regulation are explored.

From the *Wilson Centre for Research in Education, University Health Network; the †Department of Surgery, University of Toronto, Toronto, Ontario, Canada; the ‡Department of Clinical Epidemiology and Biostatistics; and the §Program for Educational Research and Development, McMaster University, Hamilton, Ontario, Canada.

One of the authors (GR) has received funding as the Richard and Elizabeth Currie Chair in Health Professions Education Research.

Correspondence to: Glenn Regehr, PhD, Wilson Centre for Research in Education, 1E565-200 Elizabeth Street, Toronto, Ontario, Canada, M5G 2C4. Phone: 416-340-3615; Fax: 416-340-3792; E-mail: g.regehr@utoronto.ca.

One of the cornerstones of autonomy for any profession is the claim to self-regulation.4 Although not well-articulated in the field of medicine, self-regulation in a professional sense has two mutually supported meanings. In the first sense, self-regulation refers to authorities in a professional field responding to problematic members of their community by first assessing the individual's situation, then taking the responsibility to mete out sanctions, place limitations on practice, and/or provide support for reeducation as necessary. Such actions are intended to ensure a minimum standard of performance among the profession's members and thereby to ensure the safety of the public. In short, the profession takes responsibility for constructing mechanisms to police its own members and, in this sense, the profession as a whole is self-regulating.

However, for many vocations, most health professions, and certainly the medical profession in North America, the profession has ceded an important part of this responsibility for self-regulation to the individual practitioner in the field. That is, part of the expectation of being a professional is a demonstrated willingness to engage in personal, individualized efforts to maintain a minimum level of competence. Although some low rate of random testing of members may be enacted by the professional body, most programs aimed at professional maintenance of competence are limited to documenting learning activities by the individual professional. In fact, there is generally little or no effort on the part of the professional bodies to make an external determination of whether these learning activities have been successful or even whether they are directed at the right areas of competence for an individual. Mere documentation of having participated in some learning activity is sufficient.

The rationale for this individual model of professional self-regulation is built on an archetype of the self-regulating professional and his approach to daily practice. This self-regulating model of daily practice involves the following set of continuously enacted steps: (1) through ongoing monitoring of and/or retrospective reflection on daily practice, the self-regulating professional self-assesses his daily performance; (2) through this self-assessment process, the self-regulating professional identifies certain areas of personal knowledge or skills that seem to have dropped below professional (or personal) standards of practice; (3) this recognition of the “gap” in knowledge or skill leads to a decision to seek opportunities to improve in these areas; (4) the appropriate learning opportunities are taken advantage of, such that the self-regulating professional learns (or relearns) the knowledge or skills necessary to perform well (at or above the minimum professional or personal standard); (5) The new knowledge or skills are put into action; (6) performance is reassessed to ensure the self-regulating professional has achieved (at least) the minimum standard of practice in this area of performance; and (7) this process is repeated as needed where needed. An excellent and elaborated discussion of this model can be found in an article by Handfield-Jones et al.12

However, empirical support for the various steps of this model of self-regulation in the literature is questionable at best. In this paper, we review the literature in adult education, medical education, and cognitive psychology in an effort to highlight two key flaws in this idealized process. First, we challenge the assumption that adult learners, when faced with the identification of a learning gap, will spontaneously seek out opportunities to redress those gaps. Second, we challenge the assumption that the process of self-assessment as conceptualized in this model can lead to the identification of gaps in skill or knowledge.

Back to Top | Article Outline

Questioning the Motivation to Fill Gaps in Learning

Support for the current model of the self-regulating professional has been drawn from several theoretical frameworks in education and psychology, including Knowles' model of the adult learner,14 Bereiter and Scardamalia's model of the expert practitioner,1 and Ericsson's model of expertise.6 Each of these models describes how adults approach learning opportunities and push themselves to become better at what they do. However, while each of these models is valuable for their own purpose, they may be less relevant to the situation described above.

For example, the models of Bereiter and Scardamalia and of Ericsson describe the individual who is already very good in a particular domain and getting still better at that domain. They describe individuals on the cutting edge of the field, those who are pushing the boundaries of excellence. Interestingly for our purposes here, such individuals must choose to sacrifice many areas of general ability to be able to invest the time and energy necessary to achieve the level of expertise for which they are aiming (one tongue in cheek definition of an expert is one who knows more and more about less and less until he knows everything about nothing). It is a separate issue to assess whether these hyper-experts have allowed themselves to drop below the minimum professional standard in the areas they have chosen to ignore and whether this is appropriate for the hyper-expert in a domain like medicine. However, for our purposes here, the point is both these models focus on the areas where the expert has chosen to invest energies for improvement, their areas of hyper-expertise. As a result, these models of learning and discovery may be less relevant to the individual who has allowed an area to drop below acceptable standards and must exert the energy necessary to raise performance back above the minimum.

The difference likely lies in the motivation for learning. In the expertise models of learning, the individual has already selected a content domain in which he excels. The reward for learning involves pushing oneself to the edges of the field, to become incrementally better in an area where one has already demonstrated clear interest, aptitude and, in fact, a capacity for excellence. Interestingly, Ericsson has raised questions about the motivation that drives even those individuals who are continuing to try to improve when already at the perimeter of their field.7 He has suggested learning or practice for real improvement is not generally fun. Yet, the expert keeps practicing, likely because the expert keeps getting better. Thus, one possibility for describing the difference between the virtuoso who is willing to invest 10,000 hours of practice and the merely excellent individual who is willing to invest only 4,000 hours is that the virtuoso keeps seeing continued incremental benefits despite already being excellent, while the person who asymptotes in performance will eventually lose motivation to keep trying. Again, however, it seems unlikely this reward of incremental improvement at the upper edge of the field is a relevant motivation for the individual who is struggling not to be substandard.

By contrast, Knowles' model of the adult learner appears to be particularly relevant to recreational learning. That is, it seems an excellent model of the individual who is seeking out areas of personal interest (eg, learning photography) or who is expanding the exploration of preexisting areas of personal interest. Such an individual needs only to satisfy himself or herself regarding the level of knowledge or skill developed. One can choose to continue learning in areas that are of personal interest, which, therefore, have entertainment value, and can choose not to pursue learning in domains that are insufficiently rewarding given the required energy for learning. Here the learning process can be a reward even if one's knowledge or skill level would only rank as amateur status. These descriptions are likely relevant for the professional who is choosing to improve in domains that are of personal interest (perhaps this description of adult learning represents the early stages of what Ericsson and Bereiter and Scardamalia are describing). But, again, such a motivation for learning may not be a relevant description of how practitioners react to the recognition they are particularly weak or even substandard in an area of practice they must redress.

Thus, the models of learning that are often referenced when describing the individually self-regulating professional are built around the description of an individual who is seeking to improve for the joy of being even better, likely in domains where the individual demonstrates clear aptitude or, at least, clear interest. They appear particularly relevant to the situation in which the person has the luxury of choosing those areas in which she wishes to learn and choosing the extent to which she wishes to learn. However, these conditions simply do not hold for the practitioner who discovers he is substandard in a particular domain. For this individual, the substandard domain is, by definition, a domain he has not chosen to keep up with merely for the sake of personal interest or excitement. Under such circumstances, the rewards of learning may very well be outweighed by the amount of energy actually required to learn, and the result of the cost-benefit equation may be avoidance rather than action. In fact, anecdotally (at least personally) it appears humans will spend huge amounts of energy covering up gaps rather than simply addressing them. Again, this speaks to an issue that is inadequately acknowledged and perhaps underappreciated by those who are trying to apply models of adult learning and expertise to the self-regulating professional in practice: the internal motivation to learn may not always outweigh the inertia of maintaining current practice. In fact, the most obvious difference between adult learners and child learners may be the fact that you can make a child sit at a piano and do scales (anyone who has tried this knows one must put up with a lot of whining-another example of the extent to which humans in general will exert more energy in avoiding a learning activity than they would expend simply doing the learning-but you can make them do it). This is often not true with adults, a fact that is attested to by all the musical instruments and exercise or sporting equipment collecting dust in adults' closets. As adults, we often ask ourselves wistful questions like, “Wouldn't it be fun to learn how to play a guitar?” However, unless one has an unusually strong innate aptitude for the new skill, the answer will likely be, “No!” It would be fun to be able to play a guitar. Learning how to play a guitar is just hard work. And whether or not we are able to persist past the initial challenge of learning will likely depend more on the short term rewards of initial success than on the longer term goal of wanting to play the instrument well.

As an academic community, we have been misled by our focus on studies that ask why adults choose to learn rather than why they choose not to learn.3 Anecdotally, we have been misled by reflection on our own experiences regarding the learning opportunities we have taken (usually because they were personally rewarding) rather than trying to understand why we did not follow up on the learning opportunities that we ignored (usually because they were uninteresting or too much work). As a result, we as an academic community must repeatedly remind ourselves of the studies showing physicians tend to attend continuing education activities that reinforce what they already know rather than focusing mostly on learning opportunities in domains where they are weak.16 But the reason for this seemingly anomalous behavior may be a simple truism that has not been fully articulated for many of us: learning itself isn't fun. Learning fun things is fun, but learning hard things is hard, and learning boring things is boring. Thus, it requires a great deal of internal self-control to make one learn in areas where one is weak (or bored). In fact, in many circumstances, it may require more self-control than many of us have. Additionally, it gets harder with time, not because the learning is harder (although this may also be true9), but because the older we get the less energy we are willing to exert to learn something new (and the fewer the number of people who have the authority to tell us we must learn it).

In short, the places where we as professionals identify serious gaps in our skills and knowledge are exactly the places where traditional adult learning motivations and incentives are most likely to have failed us. As a result, the response to an identified gap may not automatically be a headlong rush to fill it. Of course, there are a variety of circumstances where this is appropriate. As one example described earlier, the process of professional specialization is, by definition, the process of focusing on some things and letting other things slide (knowing more and more about less and less). Sometimes the right response to a recognition of gaps is to decide one does not do that anymore (and needs to refer or at least to get a consultation). However, there are also a variety of circumstances where allowing one's knowledge and skills to drop below an acceptable standard is not appropriate. There is likely, for any profession (or specialty or subspecialty), a base of knowledge and skills that must be maintained. Further, for any profession, there is an evolving state of the art with which the competent professional must keep up. Under such circumstances, the professional must engage in the learning of these new (or neglected) skills. And for many of us (especially as we get older) this is exactly the place where learning is not fun, it is just plain hard.8 Thus, it is here where the adult learning models (and certainly our naïve models of the self-regulating individual) may be overly optimistic in their assumptions that people, upon recognizing their gaps, will possess sufficient self-control to raise themselves to the minimum standard. And if this is true, then in circumstances where learning must take place in a self-regulating profession, that profession must be more vigilant in ensuring the activity does take place rather than relying entirely on the self-regulating professional's internal motivation to do so.

Back to Top | Article Outline

Questioning the Ability to Self-Identify Gaps in Learning

The previous discussion describes a potential concern regarding the motivation for responding appropriately in the circumstance where an individual detects a gap or weakness in his/her abilities. Importantly, however, it presumes the individual has been able to identify the gap or weakness. Therein is the second aspect of the model of individual self-regulation that may be problematic. That is, the literature on self-assessment suggests the capacity to self-identify such gaps in one's knowledge and ability may itself be limited. The study of self-assessment inside and outside the domain of healthcare has been active for several decades. An overwhelming number of the authors producing studies in this domain begin their papers with some paraphrase of the claim that accurate self-assessment is vital to the self-regulating professional, then are forced to conclude on the basis of their data that self-assessment is in fact quite poor. Extensive reviews of the literature5,10,11,17 reveal three consistent patterns. First, there is little or no relationship between actual performance or ability and self-rated performance or ability (correlations between objectively assessed performance and self-assessed performance usually lie in the 0.3 range). Second, the vast majority of individuals rate themselves to be above average in performance, with all but the highest performers overestimating their level of performance. Third, and perhaps most critical for this discussion, the worst offenders are those in the lowest quartile of performance, those most in need of remediation.

Kruger and Dunning15 have suggested this overestimation by the poorest performers should not be surprising. Rather, it is a natural consequence of being a poor performer. They argue the skills necessary to perform well are intimately tied to the skills necessary to know what a good performance is. Thus, incompetence not only robs people of the ability to perform well, but also robs them of the ability to realize how poorly they are performing. As evidence for this position, Kruger and Dunning described a study in which college students were given a 20-item written test of grammar, with questions taken from the National Teacher Examination preparation guide. Each question contained a sentence with a specific portion underlined and participants were to indicate whether the underlined section was grammatically correct or should be changed to one of four different wordings displayed. Based on actual test scores, participants were divided into quartiles of performance. Consistent with other studies in the field, this study found those performing in the highest quartile tended to underestimate their performance slightly while those performing in the lowest quartile tended to overestimate their performance greatly (with an average estimated percentile rank of 66.8 compared with the actual average percentile rank of 12.5 for this lowest-performing group). The researchers then asked individuals from these two extreme groups to look over and score a selected set of tests from across the range of performance and, after this activity, to reassess their own test. After the opportunity to see others' tests, the highest performing group recalibrated their own self-assessments appropriately upward. It appears upon seeing the actual range of performance by others, members of this group recognized themselves to be comparatively better than they had originally thought. Distressingly, however, the lowest performing group, if anything, also adjusted their self-assessed scores upward from their original estimation. This group was completely unaware of how poorly they were performing. Hodges et al13 have since replicated this pattern of results in the context of family medicine residents performing and self-evaluating a difficult patient interview.

As an aside, most teachers have had the experience of trying to work with a student who simply cannot learn a particular concept or skill. No amount of explanation and no number of examples seem to clear the confusion. The student has a form of content-specific perceptual deficit in that he cannot comprehend the difference between a good performance and a poor performance in this particular domain. The dimensions of comparison are simply incomprehensible to him. He cannot see why he is performing poorly, so he cannot see he is performing poorly. Such a situation is usually a source of great frustration to the teacher, for whom the difference is painfully obvious. Of course, as Brookfield has pointed out,2 the teacher has become a teacher of this domain because it is a domain in which the teacher has always excelled. Thus, ironically, teachers of a domain almost by definition have their own form of perceptual deficit in that they cannot know what it is like to not get the concept easily.2 As the rock group Jethro Tull eloquently stated, “…wise men don't know how it feels to be thick as a brick.” Brookfield2 has suggested, in fact, every year all teachers should try to learn something they are not good at as a way of reminding themselves what it feels like to be on the other side (a recommendation that may be hard to enact given the discussion on motivation above).

In the discussion of the student above and in Kruger and Dunning's15 data, the implication is self-assessment ability is not a universal skill. Rather, it is tied to ability in specific domains. The person who is bad at grammar is likely not the person who is bad at math or humor or driving. As a result, the excellent grammarian will probably recognize his grammar is good but will likely be oblivious to the fact that he is a lousy driver. The excellent mathematician will likely know her math skills are high, but be unaware that she has a poor sense of humor. Thus, as was the case with motivation to learn, it is exactly in the areas where the learning need is greatest that the individual's ability to recognize a learning need is most likely to let him down. The motivation to fill gaps in knowledge or skill, therefore, is not only undermined by the perceived magnitude of the effort required to learn when a gap is identified, it is also undermined by the difficulty in recognizing a gap even exists. Thus, again, any self-regulating profession that wishes to see these gaps identified must enact mechanisms to search for them rather than depending on the self-regulating professional's ability to do so himself.

Back to Top | Article Outline

DISCUSSION

Any profession whose claim to self-regulation relies heavily on the individual self-regulating professional is likely overly optimistic regarding the extent to which the underlying model is a valid description of professional activity, generally. Ironically, it may function well in areas where the self-regulating professional is already strong, generating the illusion of sufficiency. But it is exactly in the places where self-regulation is most needed that the model of individual self-regulation is most likely to fail. And no one is immune to those places where it fails. Each of us has areas of skill or knowledge we know could stand some improvement, but we haven't quite gotten around to addressing them yet. Each of us knows of new techniques we should look at, but find the way we do it now is working just fine. Each of us has areas that may be below our own personal standard, but, we believe, are certainly not below the minimum acceptable standard. Such are the justifications by which we maintain our self-concept as safe and competent professionals. But it is important to remember for any given skill, 25% of us are in the bottom quarter of performance. And if Kruger and Dunning's data are generalizable, those of us who are in that bottom quarter believe we are above average. So in the context of self-regulating professions, whose job is it to tell us, and whose job is it to convince us we should do something about it?

Back to Top | Article Outline

References

1. Bereiter C, Scardamalia M. Surpassing Ourselves: An Inquiry into the Nature and Implications of Expertise. La Salle, Ill: Open Court; 1993.
2. Brookfield S. Through the lens of learning: How experiencing difficult learning challenges and changes assumptions about teaching. In: Richlin L, ed. To Improve the Academy. Vol 15. Stillwater, Okla: New Forums Press; 1996:3-15.
3. Courtney S. Why Adults Learn: Towards a Theory of Participation in Adult Education. London, UK: Routledge; 1991.
4. Cruess SR, Johnston S, Cruess RL.“Profession”: A working definition for medical educators. Teach Learn Med. 2004;16:74-76.
5. Dunning D, Heath C, Suls JM. Flawed self-assessment: Implications for health, education and the workplace. Psychological Sci Public Interest. 2004;5:69-106.
6. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79 (10 Suppl):S70-S81.
7. Ericsson KA, Krampe RT, Tesch-Romer C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993;100:363-406.
8. Eva KW. The aging physician: Changes in cognitive processing and their impact on medical practice. Acad Med. 2002;77 (10 Suppl): S1-S6.
9. Eva KW. Stemming the tide: Cognitive aging theories and their implications for continuing education in the health professions. J Contin Educ Health Prof. 2003;23:133-140.
10. Falchikov N, Boud D. Student self-assessment in higher education: A meta-analysis. Rev Educ Res. 1989;59:395-430.
11. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991;66: 762-769.
12. Handfield-Jones RS, Mann KV, Challis ME, Hobma SO, Klass DJ, McManus IC, Paget NS, Parboosingh IJ, Wade WB, Wilkinson TJ. Linking assessment to learning: a new route to quality assurance in medical practice. Med Educ. 2002;36:949-958.
13. Hodges B, Regehr G, Martin D. Difficulties in recognizing one's own incompetence: Doctors who are unskilled and unaware of it. Acad Med. 2001;76:S87-S89.
14. Knowles M. The Adult Learner: A Neglected Species (3rd Ed.). Houston, TX: Gulf Publishing. 1984.
15. Kruger J, Dunning D. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77:1121-1134.
16. Miller SH. American board of medical specialties and repositioning for excellence in lifelong learning: Maintenance of certification. J Contin Educ Health Prof. 2005;25:151-156.
17. Ward M, Gruppen L, Regehr G. Research in self-assessment: Current state of the art. Adv Health Sci Educ Theory Pract. 2002;7: 63-80.
© 2006 Lippincott Williams & Wilkins, Inc.