According to Health Canada, the proportion of Canadians aged 65 and over rose from 4.8% in 1921 to 12.3% in 1998.1 This cohort is projected to grow further before reaching a plateau at more than 20% of the population by the year 2026. This trend is not unique; similar predictions have been made for both industrialized and non-industrialized countries worldwide.2 Not surprisingly, a great deal of effort has been expended attempting to predict the impact of this demographic drift on the provision of health care.3 Less examined, however, has been the effect of this trend on health care providers themselves. Already in Ontario, Canada, however, one in four family practitioners and one in three specialists are over 55 years of age.4 While the rate of physicians' retirements has been increasing, these older physicians are going to become an ever-more-valuable resource as the ratio of physicians to population declines.5
Most of the medical education research that has used age as an independent variable has been performed in the context of physician-review programs. This literature suggests that aging induces cognitive changes in the way that diagnosticians approach clinical cases. There are discrepant findings, however, in terms of whether clinical performance improves or declines with aging. In this paper we systematically examine the small amount of evidence available in medical education that highlights the issue of aging and attempt to reconcile contradictory findings by drawing on the much larger psychological literature on pre-senile aging. Finally, we identify some of the specific implications for continuing education.
Medline, ERIC, and the Research and Development Resource Base in Continuing Medical Education were used to search for articles focused on physicians' competence, physician assessment, and continuing competence. “Age” was used as a keyword in all searches and articles were included if they made reference to the relationship between age and performance. After a series of articles had been identified, the reference lists were examined and experts in this field were consulted to find relevant papers that had been missed. In parallel, a similar search was performed using PsychLit to identify articles that focused on age-related changes in cognitive processing. Articles were excluded if they focused on clinical conditions unless they allowed insight into the psychological mechanisms affected by pre-senile aging.
The Negative Relationship between Age and Performance
Since the 1970s, physician review programs have begun to flourish. Much of this work has been performed in Canada,6,7 but other countries, including England8,9 and the United States10,11 have also turned to assessments of this type as a way to ensure physicians' competence and advocate performance enhancement. While it is only one example of a variety of strategies that have been implemented, the Physician Review and Enhancement Program (PREP), based at McMaster University, typifies programs of this nature by using a battery of assessment tools to evaluate the abilities of practicing physicians.12,13 The process has evolved over the years, but the current battery consists of a multiple-choice-question test of medical knowledge, encounters with four standardized patients, and chart-stimulated recall. Skills evaluated with the standardized-patient encounters include communication, diagnosis, and data gathering. During chart-stimulated recall, the physician's own charts are reviewed and used as the basis for discussion between the assessors and the physician being assessed. This phase is meant to test problem solving, patient-management skills, and record-keeping practices. PREP evaluations are currently based primarily on referrals received from quality assurance committees operating within the College of Physicians and Surgeons of Ontario (CPSO),14 but during the validation phase of the program PREP also evaluated a criterion group of randomly selected physicians.15
Attempting to identify predictors of competence, Norman et al. entered a series of variables into a regression analysis and discovered that age was most predictive of three variables (the other two being Canadian versus foreign education and certification status) that significantly predicted performance.15 Older physicians performed less well than did younger physicians. Further (unpublished) analyses of these data confirm that this negative relationship is not an artifact of including CPSO-referred physicians in the study sample; the correlation between age and performance was stronger in the criterion group (r = -.50) than in the CPSO-referred group (r = -.36). More recent work shows this trend is not directly linked to neuropsychological impairments. Turnbull et al. administered a neuropsychological test battery on 27 physicians at the end of the regular PREP testing.14 The correlation between age and performance was strong (r = -.57) and became even stronger after removing from the analysis the 13 physicians who scored in the moderate-to-severe range on any of the neuropsychological tests (r = -.80).
This inverse relationship between age and competence is consistent with previous work by McAuley and Henderson,16 who audited the practices of 391 randomly selected physicians, and with Norcini et al.'s17 analysis of the knowledge bases of practicing internists using the American Board of Internal Medicine's (ABIM) recertification exam, as well as with more recent work by Sample et al.,18 who examined patient-management skills using a computer-based case simulation. A problematic omission, however, if remediation is to be effective, is an answer to what is causing this inverse relationship.
One possibility, supported by the work of Day et al.,19 is that older practicing physicians are less likely than are their younger colleagues to have up-to-date knowledge bases. Scores achieved on the ABIM recertification examination increased as the time since residency decreased when questions tested medical knowledge that had changed over the preceding 30 years, but showed no effect of the time since residency when questions tested medical knowledge that had been stable over the same period of time. While this is a seemingly straightforward explanation for the inverse relationship observed between age and competence, it does not appear to be sufficient. More recent work by Caulford et al.20 suggests that the nature of the problems encountered by failing physicians extends beyond a reduced tendency to assimilate new knowledge. Using the written reports generated for the PREP program, two assessors were asked to identify the specific deficiencies noted for each physician's performance. These deficiencies were categorized as problems of knowledge, history taking, problem solving, physical examination, patient management, communication skills, and record keeping. In addition to knowledge deficits, however, the identified deficiencies occurred across all eight categories of problems. Interestingly, for reasons that will become clear shortly, the most prevalent errors identified within many of the categories tended to be errors that would be expected to correlate with premature closure; the interviews tended to be abrupt with many interruptions, history taking was not comprehensive, data gathering was incomplete, important management strategies were not considered, and important details were left out of patient records. These results could, understandably, lead to pessimism regarding the ability of older physicians to practice medicine. However, there is an aspect of diagnostic practice that has been shown to improve with age, thereby yielding (a) further insight into the psychological mechanism whereby competence declines and (b) optimism regarding the potential to tailor continuing education efforts to the specific abilities/deficiencies of individual clinicians.
The Positive Relationship between Age and Preliminary Diagnostic Accuracy
The inverse relationship between age and performance seems counter-intuitive given the emphasis that most educators place on experience. Physicians who have been practicing longer have more experience and, as a result, should be better positioned to make accurate diagnostic decisions. This claim is especially central to the nonanalytic instance-based frameworks that have been proposed as models of medical knowledge.21 These frameworks argue that diagnosis is based, in part, on a rapid and unconscious matching of current patients to previous clinical encounters.22 The greater the number of cases one has seen, the more prior examples one should have available to draw upon. Consistent with this framework, diagnostic accuracy in the context that would be expected to elicit decisions based primarily on nonanalytic processes has been shown to increase with age.
To examine the influence of experience on the generation of diagnostic hypotheses based solely on contextual factors, Hobus et al.23 presented research participants with short case histories consisting of a patient's picture, previous disease history, and presenting complaint. Family doctors produced 36% more correct hypotheses (12.11/32) than did medical students (8.88/32). When this procedure was repeated with 28 physicians whose years of experience ranged from four to 32, a strong positive correlation was found between experience and diagnostic accuracy (r = .68).24 This is noteworthy because previous work has shown that the accuracy of the first hypothesis is predictive of the accuracy of the final diagnosis,25 and a negative correlation has been observed between “accuracy” and “time required to raise diagnostic hypotheses.”26
These findings run counter to those mentioned in the preceding section. Physicians' diagnostic skills seem to improve with age, at least when the diagnostic information available is minimal. The important question then becomes “Why are older physicians more likely to be labeled incompetent by physician-assessment programs when they are better at generating diagnostic hypotheses based on contextual information alone?” A potential explanation is that as individuals age, nonanalytic diagnostic strategies remain strong (allowing the positive relationship between years of experience and accuracy of early hypotheses), but the use of analytic confirmation strategies declines (causing older physicians to score less well when dealing with conflicting data within comprehensive patient descriptions).
A less complicated answer to the question posed would be that older physicians are better diagnosticians, but they (a) exert less effort toward the testing procedures utilized in review programs, or (b) have deteriorated testmanship skills as a result of having been out of school longer than younger colleagues. These possibilities seem unlikely. First, given the threat of losing a license to practice medicine, any physician who is called to a PREP review has tremendous motivation to perform at his or her highest level (even though the physician is unlikely to be happy doing so). Second, Day et al.19 observed that older physicians perform just as well as younger physicians on examinations as long as the questions are directed at knowledge that has not changed since they were trained. Still, the general point remains that age is confounded with many other variables when studied in the context of physicians' performances. To avoid these complications, the rest of this paper examines this issue by reviewing the psychology literature on aging in the hope that it will yield insight into (a) diagnostic decision making, (b) medical expertise, and (c) remediation strategies that might be considered for failing physicians. To set the context, a brief review of the literature on diagnostic expertise is first presented.
Models of Diagnostic Reasoning
Beginning in the late 1970s, thanks in large part to the work of Elstein, Shulman, and Sprafka,27 researchers in the field of medical education began to switch their views of diagnostic expertise from one in which problem-solving skills were emphasized to one in which the mental representation of medical knowledge became predominant.21 Central to Elstein et al.'s claims was the hypotheticodeductive model of diagnostic reasoning —namely, that when faced with a new case, physicians generate a set of hypotheses that they later use to test against the data presented. While this model elicited criticism in the mid-1980s,28,29,30 it created fertile ground on which numerous frameworks of knowledge representation were developed. An important divide between these various models is that of analytic (i.e., prototype-based) and nonanalytic (i.e., instance-based) knowledge frameworks. In reviewing the various models, Custers, Regehr, and Norman21 concluded that attempts to prove any one framework correct will likely be fruitless as a number of investigations have suggested that many forms of knowledge are available to act as independent sources of information that can be coordinated during diagnosis.31,32,33 Some authors have proposed that the relative importance of a particular mental representation changes as expertise develops, typically suggesting that expert diagnosticians become expert by virtue of their becoming able to rely on prior episodes.34,35,36 To date, however, models of diagnostic expertise have not incorporated changes in the relative contributions of different knowledge representations that might explain the decline in performance physician review programs detect.
Hobus and Schmidt's24 finding suggests that physicians with more experience are better positioned to take advantage of the features of a clinical presentation that are expected to enable accurate nonanalytic processing. It is possible, however, for these initial hypotheses to be incorrect, as evidenced by the fact that the experts in Hobus et al.'s23 study generated the correct diagnosis only 38% (12/32) of the time. Therefore, diagnosticians must avoid relying too heavily on this type of information. Three lines of research on aging (human memory, reasoning tendencies, and comprehension) suggest, however, that nonanalytic processing becomes increasingly dominant in adulthood due to a decline in the analytic contributions of knowledge.
Age-related Changes in Cognitive Processing
Human memory. In the literature on human memory, there is general agreement that performance losses are age-related, but that the losses do not occur in all memory tasks.37,38 Younger adults tend to outperform older adults when memory is tested directly (e.g., with instructions to recall all words presented in a list). Similar age effects are observed upon asking participants to manipulate information that is held in memory while continuing to deal with incoming stimuli (i.e., working memory tasks). For example, participants might be asked to sum a series of numbers while also remembering the second digit of every number. In contrast, young and old tend to perform equally well on less effortful tasks such as picture recognition and on indirect tests of memory such as word-fragment-completion tasks. On these tasks, participants are not told to use their memory for studied words, but are simply given a word fragment (e.g., “b-n-”) and asked to complete the word. When the word “bend” is included in a preceding study list, both young and old adults are equally likely to use “bend” to complete the word fragment “b-n-,” even though older adults do not recall having seen the word in the initial list. This dissociation between direct and indirect tests of memory suggests that age-related deficits tend to be deficits of analytic (i.e., controlled) processing, whereas nonanalytic (i.e., automatic) processing remains intact.
As Jacoby39 correctly pointed out, however, direct versus indirect tests of memory do not provide pure measures of different memory processes. For example, one could complete the word fragment “b-n-” using either an automatic, familiarity-based process or a conscious recollection of having seen the word in the study list. As a result, Jacoby created a process-dissociation procedure with which the relative contributions of analytic and nonanalytic processes can be teased apart by developing experimental conditions that place these processes in opposition to one another. Consider the following example, published by Hay and Jacoby.40 A series of word pairs (e.g., organ—piano) can be presented to participants, young and old, in such a way that a habitual response to the first word in the pair can be created. The word “organ” might be paired with both “piano” and “music,” but if “organ” is presented with “piano” 75% of the time it is shown, participants will come to expect the word “piano” when the word “organ” is presented. After such a training phase, participants are shown a series of word pairs that are to be remembered for a later memory test. “Organ—music” might be one such pairing. At test, Hay and Jacoby presented the stimulus “organ———” and asked participants to recall the word pair they had been asked to study during Phase 2 of the experiment. The word “piano” should come to mind based on nonanalytic availability due to the creation of a habitual response. However, “piano” should be offered as a response only when the correct answer (“music”) is not recalled. In this way, habit and recollection are placed in opposition to each other and can be compared with a condition in which both are placed in concert (by pairing “organ” with “piano” during Phase 2). Older adults are less likely than are younger adults to respond with the habitual response in the “in-concert” condition, but more likely than are younger adults to respond with the habitual response in the opposition condition. Taken together, these results suggest that older adults are able to utilize the habitual memory response, but are less able to take advantage of the controlled analytic component of memory. This conclusion is consistent with many other studies.41,42,43
Converging evidence from the medical domain can be found in a recent study of medical expertise that compared the memory performance of practicing physicians with that of residents using multiple memory tasks.44 This study, combined with the results of this review, calls into question whether the intermediate effect of Schmidt and Boshuizen45 might be related to age differences across level of expertise. Perhaps encapsulation occurs, at least in part, because older adults (experts) tend to base their decisions on general, gist-based46 processing (as opposed to paying careful analytic attention to specific details) to a greater extent than younger adults (intermediates).47
Reasoning tendencies. One could argue that the age-related decline in analytic processing observed using memory tasks is a weak indicator of the clinical reasoning that diagnosticians employ, but cognitive research focusing on reasoning abilities and task switching has led to similar conclusions. Numerous studies have shown that measures of fluid intelligence (reasoning: e.g., determining the most efficient way through a maze) tend to show an age-related decline, whereas measures of crystallized intelligence (accumulated knowledge: e.g., vocabulary tests) show little effect of aging.48,49 Day et al.'s19 finding that age is not related to competence on static knowledge questions provides a medical example of the stability of crystallized knowledge.
The aspect of fluid reasoning that is most relevant to this review is the ability to overcome first impressions by recognizing that alternative solutions are plausible, thereby avoiding prematurely closing a clinical case. An astute physician might have more correct first impressions than someone less experienced, but there will still be cases (the majority of cases according to Hobus et al.'s23 results) in which the clinician will need to re-orient to other possibilities. If aging reduces one's tendency to consider alternative solutions, this might account for the declining competence scores observed by review programs.
Several studies have suggested that older adults have more difficulty updating task requirements than do young adults.50,51,52 As one example, Collins and Tellier attempted to test cognitive flexibility in old and young adults by using the Visual Verbal Test.53 This test involved the presentation of four stimuli that shared traits along multiple dimensions (e.g., size, color). A participant's task was first to identify three stimuli that shared a common dimension and then to repeat the same task for a second dimension. First responses of old and young adults were equally accurate, but older individuals tended to make a greater number of second-response errors than did younger adults.53 Furthermore, participants' performance correlated with the number of perseverative errors on the Wisconsin Card Sorting Task (WCST), thereby suggesting that older adults were less able to reorient to new problem solutions than were younger adults. Further work with the WCST by Hartman, Bolton, and Fehnel54 suggests that this difficulty to reorient might arise, at least in part, due to the age-related decrements in working memory described in the previous section.
Evidence in the medical education literature also suggests that older physicians are particularly influenced by the information they encounter early on in a case (i.e., the information that is likely to set the context in which nonanalytic processes operate). Cunnington et al.55 presented diagnosticians with clinical cases that were indicative of two distinct diagnoses and manipulated the order in which the features consistent with each diagnosis were presented within the fictitious patient's written history. Their data revealed that clinical information that is presented earlier in a case history is more influential in medical diagnosis than that presented later—a primacy effect. Interestingly, however, among the two groups that can be considered to have developed a minimal level of expertise (i.e., residents and internists), the primacy effect was larger within the group of older participants.
Finally, it has been shown that older adults have a greater tendency to infuse personal experience into problem representations—a trend that would be expected if older adults rely more heavily on nonanalytic knowledge representations. Klaczynksi and Robinson,56 for example, presented 18 reasoning problems drawn from two social domains—religion and social class—to adults of various ages. Using religion as an example, one third of the vignettes contained religion-favorable conclusions (i.e., conclusions that were favorable toward the participant's religion), one third contained religion-unfavorable conclusions, and one-third contained religion-neutral conclusions. For each problem, participants rated the strength of the conclusion and the persuasiveness of the argument. Bias scores were then created by subtracting the ratings assigned to the unfavorable vignettes from those assigned to the favorable vignettes. All age groups showed a bias toward viewing favorable arguments as more persuasive and stronger. The amount of bias was not correlated with intellectual ability, but it was correlated with age; older adults were more biased than young adults.
Comprehension. Finally, it should be noted that older adults tend to have more difficulty comprehending messages than do younger adults. The magnitude of this difference gets larger when the length or complexity of sentences increases57 and when inferential processing based on the message being delivered is required.58 Furthermore, older adults rely to a greater extent on the context within which the speech takes place,59 thereby further supporting the idea that older adults are more readily influenced by automatically activated knowledge. As one example, Schneider and colleagues have shown that it is easier to report the final word of a sentence when the final word is predictable from the sentence's context (e.g., “Tree trunks are covered with bark”) than when it is not (e.g., “Mary has considered the bark”). The magnitude of the difference is greater in the elderly than in the young regardless of whether the sentence is read60 or spoken aloud.61
It would be remiss to not point out that some age-related effects, including those involved in comprehension of the messages patients emit, might be due to declines in perceptual acuity. Sensory functioning declines across the lifespan at a rate parallel to that of many cognitive tasks.62 In fact, 93% of the age-related variance in intelligence tests performed on individuals aged 70 to 103 years can be explained by decreases in sensory acuity.63 Furthermore, the performance of younger adults on cognitive tasks is more comparable to that of older adults when vision is degraded to the extent that is experienced by the elderly, and the performance of Alzheimer's patients is improved substantially when the contrast of visual stimuli is enhanced.64 However, age-related perceptual degradation does not detract from the hypothesis that older adults tend to rely more heavily on nonanalytic processing. On the contrary, declining visual or auditory capacities could facilitate a reliance on gist.
Can Experience Counteract the Effects of Aging?
As alluded to earlier in this paper, it is possible that the experiences derived from practicing a skill could help counteract age-related decrements in performance, thereby lessening their impact on everyday life or work. To study this issue, a number of researchers have asked experts and non-experts of varying ages to perform tasks that resemble the activities required to succeed in the domain of expertise. Salthouse65 had a group of engineers and a group of computer programmers interpret two-dimensional drawings of three-dimensional objects—a task commonly undertaken by the engineers. He found negative age-related effects in both groups, but no interaction between age and expertise. Similarly, Tsang and Shaner66 asked pilots and non-pilots to perform a series of time-sharing tasks in which more than one task had to be performed simultaneously. Of the 21 measures used, 20 showed an age effect and 14 showed an expertise effect, but only one showed the anticipated “age × expertise” interaction. Studies in music have found similar results.67 More recently, Masunaga and Horn68 have studied the effects of aging and expertise in a Japanese game called “go”. These authors detected an age × expertise interaction, but only when professional players were included in the analysis (as opposed to when the analysis was performed solely on the data from multiple levels of amateurs). This could suggest that only “true expertise,” as defined by Ericsson and Charness,69 will allow one to overcome age-related decrements in performance, but it should be noted that the maximum age of the professionals in Masunaga and Horn's sample was 20 years less than the maximum age of any of the amateur groups, thereby suggesting that the apparent buffering effect of expertise in “go” performance might be an artifact. Assuming for the moment that the interaction is real, this finding, combined with the fact that most physicians are not “true” experts using Ericsson and Charness'69 strict criteria, might suggest ways in which elderly physicians whose skills are deteriorating could be remediated.
Summary and Discussion
The current paper systematically reviewed the medical education literature that has used age as an independent variable. At first glance conflicting findings exist; physician-review programs15 consistently report a strong negative correlation between performance and age, whereas Hobus and Schmidt24 observed an equally strong positive correlation upon asking physicians to perform a diagnostic task in which only contextual information was provided. One interpretation of these findings that appears to be supported by much of the psychological literature on aging is that analytic processing tends to decline with age whereas nonanalytic processing remains stable. Further work needs to be performed to test this dissociation (and its impact) more directly within a medical context, but research on human memory, reasoning tendencies, and comprehension across the lifespan is consistent with this interpretation. The implications of this conclusion include the possibility to improve remediation efforts by directing them toward the specific characteristics of physician performance that decline with age.
Before considering this and other implications in more detail, however, we should be clear regarding what is not advocated by this review. First, the work described in this paper does not support ageist discrimination. In fact, one of the more robust findings in aging research is that the variability across the scores individuals receive tends to increase with age, thereby suggesting that strong individual differences exist. Although the average performance tends to be lower, many older individuals perform at levels equal to (or above) those of their younger colleagues. As such, decisions regarding continuing competence should be made at the individual level rather than instituting mandatory retirement policies. Second, the competence problems induced as a result of declining analytic capabilities should not be viewed as evidence for the inadequacy of nonanalytic diagnostic strategies. On the contrary, Norman et al.70 have shown quite convincingly that nonanalytic approaches to problem solving can be superior to analytic approaches. The problem arises when these nonanalytic approaches are not tempered by additional analytic consideration of clinical cases. Finally, changes in the relative contributions of analytic and nonanalytic processes should not be assumed to be the only factor contributing to the declining performance of older physicians.71 It might be true that older physicians are less likely to keep up-to-date with the medical literature, as Day et al.'s19 findings suggest. However, it might also be the case that this tendency is caused by the older physicians' increased reliance on nonanalytic processing; the more individuals rely on their prior experience, the less of a tendency there will be to critically incorporate novel conflicting information.
One of the difficulties with the conclusion outlined here is determining how to utilize the knowledge that older individuals rely more heavily on nonanalytic processes to improve the clinical practices of those physicians whose performances appear to be slipping. As one gains experience through residency, the opportunity to draw on prior instances increases and might, therefore, be accompanied by increasing confidence in one's diagnostic ability, making it less likely that individuals will recognize slips in their analytic abilities. As a result, simply instructing older physicians to be more careful with each case will likely prove useless if it simply slows the same approach. Directed instruction to be more analytic has proven to be successful at reducing heuristic-induced biases in medical students,72 but this strategy requires use of precisely the components of knowledge that appear to be declining in older physicians. Nonetheless, the research that has been reviewed in this paper does suggest some interventions that might prove successful in remediating older physicians, each of which should be tested more thoroughly.
- External supports—In everyday situations, such as remembering to take one's medication, the memory performance of elderly adults is commonly improved through the use of external supports. Obviously there is no diagnostic equivalent to a pill container that indicates days of the week, but some of the research reported in this review suggests that reducing the number of resource-demanding distractors might maximize the extent to which the analytic processing components available to older physicians can be utilized. At a facile level, this might involve providing simple forms on which diagnostic information can be recorded, simplifying any reports that need to be filled in, and reducing the number of commitments impinging on older physicians. These suggestions might seem simplistic, but it has been shown that tasks as automated as walking have a greater deleterious effect on cognitive processing in older adults than in young adults.73 At a more involved level, changes might be made to diagnostic stimuli (e.g., increasing the contrast on radiographic slides) that could allow easier interpretation of diagnostic features despite age-related sensory declines. Work in this area is taking place74; providing assistance to older physicians might prove an important application.
- Deliberate practice—Masunaga and Horn's68 finding that only professional “go” players showed no decline in performance with aging suggests that prolonged and deliberate practice might prove to be the secret to minimizing the detrimental affects of aging. This result, combined with the current review, suggests that such practice should focus more specifically on the analytic components of medical diagnosis, as nonanalytic processing seems to remain intact. Determining the optimal way to implement such practice strategies might prove to be a challenge,75 but continuing education efforts should be tailored toward age-specific tendencies.
- Education and testing—Finally, the current review underlines the importance of physician-review and enhancement programs. While the validity of self-assessment tends to be poor in general,76 any age-related increase in the reliance on nonanalytic components of knowledge will reinforce the difficulty physicians will have in determining when their skills are deteriorating. As a result, peer review of physicians' performance is critical, because motivation to participate in the enhancement aspect of PREP will be low until the individual requiring remediation recognizes that (a) his or her performance is slipping, and (b) a specific diagnosis can be assigned to the problem.
2. United Nations. United Nations Population Division Homepage, 〈http://www.un.org/esa/population/unpop.htm
〉. Accessed 8/21/02.
3. Reagan PA, Brookins-Fisher J. Community Health in the 21st Century. San Francisco, CA: Benjamin—Cummings, 2002.
4. Kralj B. Physician human resources in Ontario: the crisis continues. Ont Med Rev. 2001;68:19–28.
5. Ontario Physician Human Resources Data Centre. Physicians in Ontario. McMaster University, Hamilton, ON. [Various years]
6. Page GG, Bates J, Dyer SM, et al. Physician-assessment and physician enhancement programs in Canada. Can Med Assoc J. 1995;153:1723–8.
7. Cunnington JPW, Keane D. The Physician Review Program (PREP). Pedagogue. 1996–1997;7:1,3–6.
8. Southgate L, Cox J, David T, et al. The assessment of poorly performing doctors: the development of the assessment programmes for the General Medical Council's performance procedures. Med Educ. 2001;35(1 suppl):S2–S8.
9. Southgate L, Cox J, David T, et al. The General Medical Council's performance procedures: peer review of performance in the workplace. Med Educ. 2001;35(1 suppl):S9–S19.
10. Aring CD. Recertification. JAMA. 1975;233:1063–4.
11. Meskauskas JA, Webster GD. The American Board of Internal Medicine Recertification Examination: process and results. Ann Intern Med. 1975;82:1–5.
12. Davis DA, Norman GR, Painvin A, Lindsay E, Ragbeer MS, Rath D. Attempting to ensure physician competence. JAMA. 1990;263:2041–2.
13. Cunnington JPW, Hanna E, Turnbull J, Kaiga TB, Norman GR. Defensible assessment of the competency of the practicing physician. Acad Med. 1997;72:9–12.
14. Turnbull J, Carbotte R, Hanna E, et al. Cognitive difficulty in physicians. Acad Med. 2000;75:177–81.
15. Norman GR, Davis DA, Lamb S, Hanna E, Caulford P, Kaigas T. Competency assessment of primary care physicians as part of a peer-review program. JAMA. 1993;270:1046–51.
16. McAuley RG, Henderson HW. Results of the peer assessment program of the College of Physicians and Surgeons of Ontario. Can Med Assoc J. 1984;131:557–61.
17. Norcini JJ, Lipner RS, Benson JA, Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination. Ann Intern Med. 1985;102:385–9.
18. Sample L, LaDuca T, Leung C, et al. Comparing patient-management skills of referred physicians and non-referred physicians on a computer-based case-simulation examination. Acad Med. 2001;76 (10 suppl):S24–S26.
19. Day SC, Norcini JJ, Webster GD, Viner ED, Chirico AM. The effect of changes in medical knowledge on examination performance at the time of recertification. Proc Annu Conf Res Med Educ. 1988;27:139–44.
20. Caulford PG, Lamb SB, Kaigas TB, et al. Physician incompetence: specific problems and predictors. Acad Med. 1994;69(10 suppl):S16–S18.
21. Custers EJFM, Regehr G, Norman GR. Mental representations of medical diagnostic knowledge: a review. Acad Med. 1996;71(10 suppl):S55–S61.
22. Norman GR, Brooks LR. The non-analytical basis of clinical reasoning. Adv Health Sci Educ. 1997;2:173–84.
23. Hobus PPM, Schmidt HG, Boshuizen HPA, Patel VL. Contextual factors in the activation of first diagnostic hypotheses: Expert—novice differences. Med Educ. 1987;21:471–6.
24. Hobus PPM, Schmidt HG. 1993. In: Schmidt HG. The encapsulation framework in the presentation of physicians' recall of clinical cases. Presented at the annual meeting of the American Educational Research Association. Seattle, WA, 2001.
25. Barrows HS, Norman GR, Neufeld VR, Feightner JW. The clinical reasoning of randomly selected physicians in general medical practice. Clin Invest Med. 1982;5:49–55.
26. Norman GR, Rosenthal D, Brooks LR, Allen SW, Muzzin LJ. The development of expertise in dermatology. Arch Dermatol. 1989;125:1063–8.
27. Elstein AS, Shulman LS, Sprafka SA. Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, MA: Harvard University Press, 1978.
28. Groen GJ, Patel VL. Medical problem-solving: some questionable assumptions. Med Educ. 1985;19:95–100.
29. Bordage G, Zacks R. The structure of medical knowledge in memories of medical students and practitioners: categories and prototypes. Med Educ. 1984;18:406–16.
30. Gale J, Marsden P. Medical Diagnosis from Student to Clinician. Oxford, U.K.: Oxford University Press, 1983.
31. Regehr G, Brooks LR. Perceptual manifestations of an analytic structure: the priority of holistic individuation. J Exp Psychol Gen. 1993;1222:92–114.
32. Regehr G, Cline J, Norman GR, Brooks L. Effect of processing strategy on diagnostic skill in dermatology. Acad Med. 1994;69(10 suppl):S34–S36.
33. Regehr G, Brooks L, Norman GR, Cline C, Whittlesea B. Coordinating analytic and episodic processes in visually based medical diagnosis. [Unpublished].
34. Dreyfus HL, Dreyfus SE. Why skills cannot be represented in rules. In: Starkey NE (ed). Advances in Cognitive Science. Chichester, U.K.: Ellis Horwood, 1986:315–35.
35. Schmidt HG, Boshuizen HPA. On acquiring expertise in medicine. Educ Psychol Rev. 1993;5:205–21.
36. Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Acad Med. 1990;65:611–21.
37. Craik FIM, Jennings JM. Human memory. In Craik FIM, Salthouse TA (eds). The Handbook of Aging and Cognition. Hillsdale, NJ: Erlbaum, 1992.
38. Tulving E, Craik FIM (eds). The Oxford Handbook of Memory. New York: Oxford University Press, 2000.
39. Jacoby LL. A process dissociation framework: separating automatic from intentional uses of memory. J Mem Language. 1991;30:513–41.
40. Hay JF, Jacoby LL. Separating habit and recollection in young and older adults: effects of elaborative processing and distinctiveness. Psych Aging. 1999;14:122–34.
41. Jacoby LL, Jennings JM, Hay JF. Dissociating automatic and consciously controlled processes: implications for diagnosis and rehabilitation of memory deficits. In: Herrman DJ, McEvoy CL, Hertzog C, Hertel P, Johnson MK (eds). Basic and Applied Memory Research: Theory in Context. Mahwah, NJ: Erlbaum, 1998.
42. Dywan J, Jacoby LL. Effects of aging on source monitoring: differences in susceptibility to false fame. Psychol Aging. 1990;5:379–87.
43. Jacoby LL, Kelley C, McElree BD. The role of cognitive control: early selection versus late correction. In: Chaiken S, Trope Y (eds). Dual Process Theories in Social Psychology. New York: Guilford, 1999.
44. Eva KW, Norman GR, Neville AJ, Wood TJ, Brooks LR. Expert/Novice Differences in Memory: A Reformulation. [Unpublished].
45. Schmidt HG, Boshuizen HPA. On the origin of the intermediate effects in clinical case recall. Memory and Cognition. 1993;21:338–51.
46. Reyna VF, Brainerd CJ. Fuzzy trace theory: an interim thesis. Learn Individ Diff. 1995;7:1–75.
47. Koutstaal W, Schacter DL. Gist-based false recognition of pictures in older and younger adults. J Mem Language. 1997;37:555–83.
48. Park DC, Cognitive aging, processing resources, and self-report. In: Schwarz N, Park DC, Knauper B, Sudman S (eds). Cognition, Aging, and Self-Reports. Philadelphia, PA: Psychology Press, 1999.
49. Schaie KW. Intellectual Development in Adulthood: The Seattle Longitudinal Study. New York: Cambridge University Press, 1996.
50. Van der Lindan M, Bredart S, Beerten A. Age-related differences in updating working memory. Br J Psychol. 2000;85:145–52.
51. Kray J, Lindenberger U. Adult age differences in task switching. Psychol Aging. 2000;15:126–47.
52. Sciafa CT, Ho G, McPhee L, Dennis W. Age differences in triple conjunction visual search. Paper presented at the annual meeting of the Canadian Society for Brain, Behavior, and Cognitive Science. Laval, Quebec, Canada, 2001.
53. Collins B, Tellier A. Differences in conceptual flexibility with age as measured by a modified version of the Visual Verbal Test. Can J Aging. 1994;13:368–77.
54. Hartman M, Bolton E, Fehnel SE. Accounting for age differences on the Wisconsin Card Sorting Test: decreased working memory, not inflexibility. Psychol Aging. 2001:16;385–99.
55. Cunnington JPW, Turnbull JM, Regehr G, Marriott M, Norman GR. The effect of presentation order in clinical decision making. Acad Med. 1997;72(10 suppl):S40–S42.
56. Klaczynski PA, Robinson B. Personal theories, intellectual ability, and epistemological beliefs: adult age differences in everyday reasoning biases. Psychol Aging. 2000;5:400–16.
57. Obler LK, Fein D, Nicholas M, Albert ML. Auditory comprehension and aging: decline in synctactic processing. Appl Psycholinguistics. 1991;12:433–52.
58. Wingfield A. Comprehending spoken questions: effects of cognitive and sensory change in adult aging. In: Schwarz N, Park DC, Knauper B, Sudman S (eds). Cognition, Aging, and Self-Reports. Philadelphia, PA: Psychology Press, 1999.
59. Laver GD, Burke DM. Why do semantic priming effects increase in old age? A meta-analysis. Psychol Aging. 1993;8:34–43.
60. Speranza F, Daneman M, Schneider BA. How aging affects the reading of words in noisy backgrounds. Psychol Aging. 2000;15:253–8.
61. Pichora-Fuller K, Schneider BA, Daneman M. How young and old adults listen to and remember speech in noise. J Acoustic Soc Am. 1995;97:593–608.
62. Baltes PB, Lindenberger U. Emergence of powerful connection between sensory and cognitive functions across the adult lifespan: a new window to the study of cognitive aging? Psychol Aging. 1997;12:12–21.
63. Lindenberger U, Baltes PB. Sensory functioning and intelligence in old age: a strong connection. Psychol Aging. 1994;9:339–55.
64. Gilmore GC, Thomas CW, Klitz T, Persanyi MW, Tomsak R. Contrast enhancement eliminates letter identification speed deficits in Alzheimer's disease. J Clin Geropsychol. 1996;2:307–20.
65. Salthouse TA. Age and experience effects on the interpretation of orthographic drawings of three-dimensional objects. Psychol Aging. 1991;6:426–33.
66. Tsang PS, Shaner TL. Age, attention, expertise, and time-sharing performance. Psychol Aging. 1998;3:323–47.
67. Meinz EJ, Salthouse TA. The effects of age and experience on memory for visually presented music. J Gerontol Psychol Sci. 1998;53B:60–9.
68. Masunaga H, Horn J. Expertise and age-related changes in components of intelligence. Psychol Aging. 2001;16:293–311.
69. Ericsson KA, Charness N. Expert performance: its structure and acquisition. Am Psychologist. 1994;49:725–47.
70. Norman GR, Brooks LR, Colle CL, Hatala RM. The benefit of diagnostic hypotheses in clinical reasoning: experimental study of an instructional intervention for forward and backward reasoning. Cogn Instruct. 2000;17:433–48.
71. Lockyer JM. Physician performance: the roles of knowledge, skill, and environment. Teach Learn Med. 1992;4:86–6.
72. Eva KW. The influence of differentially processing information on diagnostic decision-making [dissertation]. McMaster University, Hamilton, Ontario, Canada, 2001.
73. Lindenberger U, Marsiske M, Baltes PB. Memorizing while walking: increase in dual-task costs from young adulthood to old age. Psychol Aging. 2000;15:417–36.
74. Hamstra SD. Technical skills training in surgery: transfer of learning and the specification of essential constructs. Presented at McMaster University, February 20, 2002.
75. Guest CB, Regehr G, Tiberius RG. The lifelong challenge of expertise. Med Educ. 2001;35:78–81.
76. Eva KW. Assessing tutorial-based assessment. Adv Health Sci Educ. 2001;6:233–46.