Dualtrack CME: Accuracy and Outcome


Section Editor(s): Cosgrove, Ellen MD

Academic Medicine:
Papers: Assessing Practitioners
Author Information

Correspondence: Jocelyn M. Lockyer, Continuing Medical Education, University of Calgary, 3330 Hospital Drive, NW, Calgary, AB T2N 4N1, Canada; e-mail: 〈lockyer@ucalgary.ca〉.

The authors thank Pfizer Canada for the funding they provided to develop and disseminate the Alzheimer's and Other Dementias Course across Canada, especially Derek Sadko, Josee Ann Maurais, and Dale Yakutchik. They also thank Cynthia MacDermott, University of Calgary, for her work assigning physicians into groups and handling the pre/post data processing.

Article Outline

Although no single view of teaching or learning dominates,1 “many of those involved in the teaching of adults think first about the learners as they plan, conduct, evaluate and reflect on their teaching.”2 By focusing on the learner and not the content, the learner is supported in his or her quest to obtain knowledge and skills at the level of sophistication required. Research in continuing medical education (CME) has shown group learning can be individualized and made more personally relevant to physician—learners by needs assessments, case-based discussions, and question-and-answer periods.3

Focusing on the learner can be difficult, particularly when groups are heterogeneous or large. Carrying out pre-course testing in a timely fashion and ensuring that faculty are able to interpret and respond to these data has proven to be challenging.4,5 Similarly, making substantive changes to a course's content or design while teaching ordinarily is beyond the expertise of many teachers.

Tracking, or the process of separating students by ability, interest, and skill, has been applied in K—12 education. While rarely used for practicing physicians, it may be helpful to consider when physicians vary in their expertise. In these circumstances, it may ensure physicians are taught at an appropriate level with methods likely to help them attain their goals. For example, whereas a novice might benefit from introductory material and appreciate lectures and activity with feedback, an advanced learner may benefit more from a student-directed discussion.6

The management of Alzheimer's disease (AD) and other dementias by family physicians is an area of variability in clinical practice.7 Some family physicians have many elderly patients with dementia. Others have relatively few elderly patients in their practices and limited ability to diagnose and manage AD. The advent of new pharmaceutical agents with the surrounding media activity has increased physician and patient interest in AD.

The purpose of this study was to examine the impact of an intensive educational program, on the diagnosis and management of AD and other dementias, in which physicians were separated into two groups based on a pre-course assessment. Our research had two questions: Did we accurately assign physicians? What outcomes did we achieve?

Back to Top | Article Outline


We developed two MAINPRO-C™8 courses on AD. Both courses were 6.5 hours in length. Both contained three 30-minute lectures on diagnosis, treatment, and driving issues. The introductory course (track 1) focused on the fundamentals of AD, including definition, differential diagnosis, pharmacotherapy, and driving. It used roleplaying to teach the mini-mental state examination (MMSE) and case-based discussion. It was designed for physicians who were not seeing many patients with dementia and/or did not use a standardized approach to diagnosis. The advanced course (track 2) was casebased, focusing on pharmacotherapy, management of moderate to severe stages of dementia, behavioral problems, and assessing competency and capacity. It was designed for physicians who had greater knowledge, skill, and experience.

Both courses conformed to the College of Family Physicians of Canada requirements for MAINPRO-C courses8 and received accreditation as such. Thus, the courses were relevant to family medicine, had objectives, and included an evaluation. They also were required to have a pre-course and a post-course assessment. The majority of course time (two thirds) was to be spent in small groups of ten physicians. The courses had to be delivered in a manner consistent with the ethical guidelines established by the Canadian Medical Association.

The two courses were delivered in parallel on the same day by two or more of the 28 trained facilitators. Participants were assigned to one of the two tracks.

Our pre- and post-course (three months later) assessments were identical and consisted of six dimensions. Physicians provided data from their own practices on the numbers of MMSEs administered in the preceding six months, the numbers of patients they had cared for with AD, and the numbers of patients they had cared for with dementia. There was a scale for their usual involvement in patient management (involvement level), and scores for both knowledge and comfort. The involvement-level scale was 1 = “I refer after a minimum of history taking”; 2 = “I take a complete history, make a tentative diagnosis, then refer for diagnosis confirmation, treatment and ongoing care”; 3 = “I take a complete history, make a tentative diagnosis, then refer for diagnosis confirmation and treatment. I provide ongoing care after a treatment plan has been established”; 4 = “I provide complete care of the uncomplicated dementia patient, including make a tentative diagnosis, then develop a treatment plan”; to 5 = “I provide complete care of the ‘uncomplicated’ and ‘complicated’ patient.” The knowledge score was derived from totaling the correct answers to a series of true/false and multiple-choice questions to examine knowledge; 14 was the best possible score. The comfort score was derived from 13 items to assess comfort with management (scored on a 1-to-3 scale with 1 being low); a score of 39 was the highest possible score. Cronbach's alpha analyses were run for both the knowledge and the comfort items to assess reliability.

Selected pre-course data were used to separate participants for the courses. The number of MMSEs done within a defined period of time was identified as a proxy for experience with AD and reflected the extent of clinical contact with these patients. Accordingly, we examined the number of MMSEs administered in the preceding six months (with the cut point being six or more for the advanced track). We then verified that the physician had at least 20 patients in his or her practice for whom he or she had a working diagnosis of AD. If not, the physician was assigned to track 1. If the physician met the first two criteria, but was not involved with patients at level 4 or 5 (i.e., assuming a high level of responsibility for the patients), he or she was also placed in track 1. Faculty for each course were provided with the pre-course data and the tracking recommendation.

The appropriateness of the course assignment was examined retrospectively using discriminant-function analysis. This analysis allowed us to take into account the six dimensions of the pre-course assessment to determine the group to which the physician should have been assigned. Physicians were included in this analysis if they provided data for all six dimensions. Discriminant-function analysis is a statistical technique that is used for classification purposes to separate groups maximally. It tests which of the independent variables (i.e., the six dimensions) account for the difference between the groups. Thus, it gives the “best” prediction, in the least-squares sense, of the correct group membership for each physician.

We assessed course outcome by comparing the pre-test data with corresponding post-course data (i.e., using matched pairs). The pre- and post-knowledge scores, comfort scores, and involvement levels were compared within tracks using a paired Student t-test. Effect-size measures (standard mean differences, d) were computed. A d value shows the difference between two means divided by their pooled standard deviation. Results were considered significant if p values were < .05. Generally, effect sizes of < .2 are considered small or negligible, those around .5 moderate, and those > .8 practically important.8 Effect sizes were considered in conjunction with the statistical tests in arriving at conclusions about the theoretical and practical significances of the relationships pre- and post-course. ANOVA was used for the between-tracks comparison so we could assess whether there were differences between the two groups before and three months after the course.

Back to Top | Article Outline


A total of 637 physicians participated in 48 offerings of the courses between February 1999 and June 2000. Of these, 413 physicians provided complete data for all six dimensions and were included in the discriminant-function analysis. Of the 637 total participants, 628 (98.6%) provided pre-tests, 422 (66.2%) provided post-tests. Of the 628 physicians who provided pre-tests, 382 (60.8%) took the track 1 course and 246 (39.2%) were in track 2. Both pre- and post-course assessment data were provided by 413 (64.8%), and their data were used for the matched-pairs outcome assessment.

Cronbach's alpha assessed the reliability of the knowledge and comfort scales. The Cronbach's alpha for the pre-test for cognitive knowledge was .5605, that for the post-test was .5614. For comfort, that for the pre-test was .8837 and that for the post-test was .9316.

The discriminant-function analysis (see Table 1A,Table 1B,Table 1C) indicated that 83.1% of physicians were assigned correctly. The number of MMSEs administered in the preceding six months accounted for most of the variance (.821).

The results of the ANOVA between-tracks comparisons showed significant differences between tracks for the pre-course assessments of knowledge (F = 19.43, p < .001), comfort (F = 66.72, p < .001), and involvement level (F = 60.61, p < .001). The results of the ANOVA between-tracks comparisons showed significant differences between the tracks for the post-course data for comfort (F = 17.37, p < .001) and involvement (F = 28.76, p < .001), but not knowledge (F = .451).

The t-test and effect-size analysis data shown in Table 2 suggest that track 1 physicians improved moderately in knowledge, comfort, and involvement based on both the t-test and the effect-size analyses. Track 2 physicians also improved significantly, but the effect size was small to negligible.

Back to Top | Article Outline


Our retrospective comparison of data with a more comprehensive model for tracking showed we were able to accurately assign over 80% of the physicians to the appropriate tracks based on the numbers of MMSEs administered in the previous six months and the numbers of patients managed with dementia. The between-tracks comparison further confirmed the pre-course differences between the two tracks.

The t-test and effect-size analyses indicated that track 1 (introductory course) participants improved moderately when their pre- and post-course data were compared. The data related to involvement in patient care for track 1 were particularly encouraging, as the effect size (d = .7) suggested that we had achieved practical as well as theoretical significance. Physicians were assuming more responsibility for their patients. While their involvement pre-course was one in which they took a complete history, made a tentative diagnosis, then referred for diagnosis confirmation and treatment, three months after the course more were providing complete care of the uncomplicated dementia patient, including making a tentative diagnosis and developing a treatment plan.

The effect size for track 2 (advanced course) physicians, while significant, suggested a small or negligible change in their knowledge, comfort, and involvement in patient care. As Tinsley and Brown noted, “large sample studies produce less biased estimates of population effects than small sample studies, but they also yield negatively biased estimates of population effects because measurement error is always present.”10 This type of result could be anticipated with the track 2 physicians, who had higher knowledge, comfort, and involvement levels pre-course, as a ceiling effect is more likely to occur with higher initial scores. Furthermore, the advanced course was designed to provide an opportunity for physicians to discuss cases, thus building incrementally on existing expertise. This is in contrast to the introductory course, which taught basic management skills related to AD. Our assessment tools, which detected differences in track 1, may not have been sufficiently refined to detect changes for track 2. Furthermore, the Cronbach's alpha for knowledge is somewhat lower than the .7 to .8 regarded as satisfactory for comparing groups,11 which may have affected the analysis.

Separating physicians into two courses on the same topic requires that educational programs be designed so that they are sufficiently different in content and focus. It also requires more administrative work on the part of the course provider and sufficient lead time to get data analyzed and then provided to teachers. As other studies have found,4,5 it is difficult to get pre-tests back in sufficient time that faculty can adjust their presentations in meaningful ways. We believe that tracking allowed the participating physicians to take courses that were more appropriately matched to their levels of clinical expertise. The teachers did not have to adapt their teaching in major ways. Physicians beginning to develop skills in the management of AD were able to learn the basic principles of diagnosis and management with other inexperienced physicians. Conversely, physicians who had developed expertise could use their CME time to discuss more complex patients.

We did not have complete data for all course participants. Pretests were not provided by nine (1.4%). Post-tests were not provided by 215 (33.8%). However, almost two thirds of the physicians provided both pre- and post-course assessments, which is a higher proportion than we obtained in earlier studies.5,12 There were significant differences between those who completed both pre/post data and those for whom we had only pre-course data. Physicians who provided pre/post data had been in practice fewer years (a mean of 19.41 vs. 22.59, p = .000); were more likely to be certificants of the College of Family Physicians of Canada or fellows of the Royal College of Physicians and Surgeons of Canada vs. being a GP (75% and 76.5% vs. 36%, p = .000); were more likely to be Canadian/U.S. graduates versus international graduates (68.4% vs. 52.8%, p = .000); and had higher mean knowledge scores on the preassessment (mean 8.75 vs. 8.14, p = .003). There was no significant difference between those who provided pre/post data and those who provided pre-test data only for comfort and involvement. It must be noted that the course was designed as a MAINPRO-C course for physicians who needed study credits and wanted to improve their skills as family physicians; this group would be more likely to be compliant with course requirements.

We encourage additional research into course designs with multiple tracks. We believe this can facilitate learner-centered teaching and ensure that participants can participate in a course better matched to their knowledge and experience levels. We were able to take advantage of a clinical area that lent itself to introductory and advanced-level courses and the separation of physicians into two groups based on their clinical work. While onerous at times to obtain the necessary information to separate physicians in a timely fashion, it was possible to do so with reasonable accuracy and to demonstrate that the introductory-course participants had improved scores at three months.

Back to Top | Article Outline


1. Pratt DD, Arseneau R, Collins JB. Reconsidering “good teaching” across the continuum of medical education. J Cont Educ Health Prof. 2001;21:70–81.
2. Pratt DD, et al. Five Perspectives on Teaching in Adult and Higher Education. Malabar, FL: Krieger Publishing, 1998:8.
3. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA. 1995;274:700–5.
4. Ward J, MacFarlane S. Needs assessment in continuing medical education: its feasibility and value in a seminar about cancer for general practitioners. Med J Aust. 1993;159:20–3.
5. Ward R, Fidler H, Lockyer J, Toews J. Physician outcomes and implications for planning an intensive educational experience on attention-deficit hyperactivity disorder. Acad Med. 1999;74(10 suppl):S31–S33.
6. Merriam S. Andragogy and Self-directed Learning: Pillars of Adult Learning Theory. The New Update on Adult Learning Theory. San Francisco, CA: Jossey—Bass, 2001.
7. Patterson C, Gauthier S, Bergman H, et al. The recognition, assessment and management of dementing disorders: conclusions from the Canadian Consensus Conference on Dementia. Can J Neurol Sci. 2001;28(suppl 1):S3–16.
8. Anonymous. MAINPRO, Website for College of Family Physicians of Canada, 〈http://www.cfpc.ca/cme/mainpro/mainpro.asp〉. Accessed 1/25/02.
9. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum, 1987.
10. Tinsley HEA, Brown SD. Multivariate Statistics and Mathematical Modeling. In: Tinsley HEA, Brown SD. Applied Multivariate Statistics and Mathematical Modeling. San Diego, CA: Academic Press, 2000:26.
11. Bland JM, Altman DG. Statistics notes: Cronbach's alpha. BMJ. 1997;314:572.
12. Ward R, Fidler H, Lockyer JM, Basson RJ, Elliott S, Toews J. Physician outcomes following an intensive educational program on erectile dysfunction. J Sex Educ Ther. 2001;26:358–62.
© 2002 Association of American Medical Colleges