Jochemsen-van der Leeuw, H.G.A. Ria MD; van Dijk, Nynke MD, PhD; Wieringa-de Waard, Margreet MD, PhD
During their clinical training, medical interns and residents (hereafter called trainees) learn professional and competent behavior by working alongside clinical trainers.1–5 A clinical trainer is any physician who supervises trainees in clinical practice.
Trainees take an active role in improving their clinical competence through educational sessions during which clinical trainers act as teachers, as well as through actively responding to the feedback they receive (with regard to their clinical skills) from clinical trainers acting as mentors. In addition, trainees grow into their future role as physicians by imitating clinical trainers acting as role models.6–8 In order to become a positive role model, a clinical trainer should be competent in three domains: personal, clinical, and teaching qualities.9–11 We have, therefore, defined role modeling to be a combination of personal characteristics (Heart), professional patient care (Hands-on), and teaching that involves continuously making the implicit explicit (Head).1,10,12 Being a role model, as opposed to being a teacher or a mentor when the moment calls for it, implies that the clinical trainer integrates the “3Hs” as a unity all the time and everywhere.
Wright and Carrese (2002)12 emphasize that identifying with and emulating multiple role models is important for medical trainees. Yet, in many small clinical settings (e.g., general practitioner [GP] training practices), only one or two clinical trainers are available. The absence of multiple role models, whom a trainee can compare against one another, may increase the trainee’s risk of imitating negative role models. To recognize which of their trainers’ role model behaviors they should imitate, trainees need an apperception11 assessment tool. Such a tool could allow trainees to discriminate between positive and negative role modeling. In addition, an educational manager (e.g., residency director) could also use a role modeling assessment tool to evaluate and provide feedback to individual clinical trainers about their role modeling, thereby enabling them to examine and improve their role-modeling behaviors.12,13
Many tools have already been developed for assessing clinical trainers as medical educators; these include the Cleveland Clinic Teaching Effectiveness Instrument (CCTEI),14 the System for the Evaluation of Teaching Qualities instrument,15 and the Clinical Teaching Assessment Instrument.16 Two tools (the Evaluation and Feedback For Effective Clinical Teaching instrument17 and the Maastricht Clinical Teaching Questionnaire18) even address specifically whether a clinical trainer should be considered a role model. Because they do not assess particular attributes of role modeling (as opposed to teaching or mentoring), these tools do not identify negative role modeling as such and cannot help trainees avoid imitating negative behaviors.12,19–22 Furthermore, these tools offer no possibilities for providing specific feedback on which aspects of the role model function could be improved.
This study therefore aims to develop and validate a tool for assessing various aspects of the clinical trainer as a role model.
In order to work independently as a GP in the Netherlands, physicians who have graduated from a bachelor–master curriculum at a university must complete three years of additional specialty training. During the first and third years of this training, each of these doctors works as a trainee in a primary care practice four days each week, supervised by at least one GP, the clinical trainer. In the second year, each physician works in a variety of clinical settings, usually with more than two clinical trainers. Central curriculum sessions, which cover all CanMEDS skills and topics, are organized at one of the eight Dutch institutes for GP specialty training. GPs and behavioral scientists (both called “teachers”—not to be confused with trainers) provide the instruction. The sessions are held one day each week for all GP trainees and eight days each year (for faculty development) for all GP trainers.
We conducted this study among GP trainees at four of the Dutch institutes for GP specialty training. These four, located in Amsterdam, Nijmegen, Maastricht, and Leiden, represented the Netherlands geographically, and also provided a good number of different GP training practices and settings. The potential participants (n = 328) comprised the first and third-year GP trainees attending one of the central curriculum sessions during the first four months of 2012. We invited all the trainees who were present on the day we distributed the questionnaires.
We instructed the participants to complete the questions keeping their primary clinical trainer in mind. We informed all of the GP trainees about the nature of the study, and we assured them that participation was voluntary and that the questionnaires would be coded to protect anonymity. We offered no incentives. Each participating trainee signed an informed consent form. We stored all informed consent forms and other identifiers (separately from the forms) in a locked cabinet under the control of the head of the research department (M.W-dW.). We obtained ethical approval for this study from the ethical review board of the NVMO (Dutch Association for Medical Education).
Development and administration of the instrument
We developed the survey instrument in three stages. First, we conducted a systematic review of all available original studies published through May 2011 that provided attributes of clinical trainers as role models.11 We organized and ranked our list of attributes (e.g., “Knowledge acquisition and clinical thought processes”) according to both the frequency of their occurrence in the included studies, and their importance as indicated in these studies.
In the second stage, we combined like attributes and transformed them into questionnaire items. They were then translated into Dutch (based partly on translations obtained from a Dutch article),23 adapted to cover all of the identified attributes, and assessed by experts: two clinical trainers, two teachers of GP trainees, two faculty developers of GP trainers, and two researchers (N.v.D., M.W-dW.). These eight experts performed a critical assessment of whether collectively the potential questionnaire items accurately summarized and characterized the original attributes identified through the systematic review. They also considered whether the items were clearly worded and whether they were sufficiently relevant to warrant inclusion in the instrument.
Finally, second-year trainees pretested the resulting concept instrument in a focus group setting. In one of two focus groups, the trainees, instructed to consider one of their past clinical trainers, completed the instrument using a five-point Likert scale, whilst freely expressing and discussing their thoughts. Their evaluation did not interfere with the actual testing of the instrument, as the second-year trainees were not working in GP training practices at the time of this study. After adjusting the items according to the second-year trainees’ feedback, we compiled the instrument for testing.
After we finalized the instrument, we asked first- and third-year trainees to complete it during one of their central curriculum sessions. A teacher (who, as mentioned, is not the clinical trainer) proctored the session to ensure that trainees would work independently as they assessed their clinical trainers as role models, and a research assistant collected the questionnaires.
We applied the three-step development process described above in order to evaluate the content validity24 of the role model assessment instrument. We performed other tests, as described below, to examine the relationship between items, reliability, convergent validity, and known-group comparison on the instrument. We performed all data analyses using SPSS 16 (SPSS Inc, Chicago, Illinois).
To test for relationships between items, we conducted a principal component analysis. We verified the sampling adequacy by calculating the participant-to-item ratio and using the Kaiser–Meyer–Olkin measure. We assessed multicollinearity examining the determinant (R) of the R-matrix, and we evaluated the correlation between variables, using the significance of the Bartlett's test.
We extracted components using oblique rotation (direct oblimin method). Using a scree plot, we established the number of components that should be retained, and we ran an analysis to obtain components with eigenvalues greater than 1, based on the Kaiser's criterion. We described the relative variance accounted for by each factor. Adopting alpha scores greater than 0.7 as an indication of good reliability, we calculated the Cronbach's alpha scores in order to assess the internal consistency of the retained components.25
We also asked the trainees to complete the CCTEI14 on the same clinical trainer they assessed with our instrument. We used the CCTEI because it contains items that do not interfere with the attributes for role modeling (e.g., “Coaches me on my clinical/technical skills”); because, like our instrument, it was unfamiliar to the respondents; and because it was developed to evaluate clinical faculty and to provide feedback to clinician–educators so they might improve their teaching. The known internal consistency of the CCTEI is high (Cronbach's alpha = 0.97).
Using descriptive statistics, we elaborated the scores for both instruments. In comparison with the CCTEI, we determined the convergent construct validity with the Spearman's rank correlation test. We expected a high correlation between the CCTEI and the Role Model Apperception Tool (RoMAT) because both instruments evaluate the competence of the clinical trainer for the learning process of the trainee.
According to the method of known-groups comparison, we evaluated the instrument’s ability to discriminate between the less and more experienced trainees using nonparametrical testing (Mann–Whitney U test). Previous studies showed that trainees who recently started their training were more likely to report the importance of personal characteristics and the learning environment, whereas more experienced trainees emphasized the clinical skills and the ability to transfer them.2,8,10
We also thought, given previous studies,8,12 that we might detect differences between male and female trainees. Whereas one previous study had identified a significant difference between men and women with regard to ratings for important role model characteristics,8 another study12 reported no differences.
Of all 328 trainees, 279 (85%) completed instruments; 49 (15%) of the trainees were absent because of illness, holidays, or having to work in the training practice. Tables 1, 2, and 3 provide an overview of the personal characteristics of the respondents, the clinical trainers they assessed, and the gender concordance of the trainees and trainers.
Content and external validity
Our eight experts organized and reviewed role model attributes that we had identified through a systematic review of the literature,11 reducing the original set of 21 to 20 items, including 3 that they revised. Two focus group sessions involving, respectively, 10 and 8 second-year trainees (13 women and 5 men), resulted in the elimination of 3 items, the acceptance of 10 items with no change, the reformulation of 4 items, and (to increase their clarity) the addition of a few words to 3 other items. This three-step development process yielded a final instrument consisting of 17 items, comprising 5 items on personal (heart) qualities, 6 on clinical (hands-on) qualities, and 6 on teaching (head) qualities—all of which are scored using a five-point Likert scale. Appendix 1 shows the final Role Model Apperception Tool (RoMAT).
Role Model Appercept...Image Tools
We established external validity by testing the tool on groups from different training institutes in different parts of the country.
Figure 1 shows the mean scores ± the standard deviation (SD) scores for all of the RoMAT items. Individual scores ranged from 1 to 5. Despite the possibility that trainees would provide only socially desirable answers because of fears of negative evaluations by their GP trainers, the respondents used the full range of options.
With a participant-to-item ratio of 16:1 (> 10:1)26 and a Kaiser–Meyer–Olkin measure of 0.940, we found the sample size adequate.25 The determinant of the R-matrix was 4.24E-005 (necessary value > 0.00001),25 thus reducing the risk of multicollinearity. A Bartlett's test showed a χ2 (136) = 2713.737, P < .001, indicating that correlations between variables were large enough to perform a principal component analysis.25 We based the choice for oblique rotation on a high correlation of 0.582 between the two factors after extraction and oblique rotation of the variables.26 Based on the scree plot and Kaiser's criterion, two components were extracted, together explaining 56.6% of the variance. The first component (Caring Attitude) clusters items that reflect characteristics of the relationship of trainers to their patients, trainees, and others. The second component (Effectiveness) represents items relating to the ability of trainers to provide their patients and trainees with what they need. Both components include the same number of items for each of the 3Hs (see Table 4). Figure 1 shows the mean scores and SDs for both of the two components.
Convergent validity and reliability
The reliability of the two components was high. The Cronbach's alpha score for Caring Attitude was 0.919, and the score for Effectiveness was 0.843 (see Table 4). The CCTEI score indicated significant positive correlation coefficients of 0.662 for Caring Attitude (P = .01) and 0.779 for Effectiveness (P = .01).
Known-group comparison (see Table 5)
We detected no significant difference in the scores of first- and third-year trainees or in the Effectiveness scores between trainees who had worked only a few years as a physician before starting GP training and those with many years of experience; however, the Caring Attitude scores of trainees with fewer than 2.5 years of previous experience were significantly higher than those of trainees with more than 2.5 years of experience (P = .03). In addition, the scores of first-year trainees were higher than were those of third-year trainees, although this difference was not significant. We detected no significant difference between the scores of males and those of females.
We developed and tested a valid, reliable tool for assessing and evaluating the role model behavior of clinical trainers: the RoMAT. The RoMAT consists of two components—Caring Attitude and Effectiveness—both of which include an equal number of items addressing personal, teaching, and clinical qualities, indicating that competence in all 3Hs is a condition for positive role modeling.
Although the RoMAT may serve as an instrument to identify educators who exemplify positive role models, characteristics of the trainees have to be taken into account. Our analyses revealed a significant difference in the Caring Attitude scores between less and more experienced trainees. Previous researchers2,8,10 have also observed this difference; just-starting trainees tend to focus on the personal qualities of their clinical trainers and the learning climate, whereas more experienced trainees tend to concentrate on the professional competence and teaching skills of their clinical trainers. Nevertheless, the greater attention just-starting trainees give to personal characteristics does not fully explain the higher Caring Attitude scores these trainees assigned their trainers. Given that GPs in the Netherlands are free to choose to work as clinical trainers, those who do elect this option are likely to have a caring attitude toward just-starting trainees. This process of natural selection could have caused the relatively high scores on the instrument in general, and on the Caring Attitude score in particular. In addition, given their uncertainty regarding what is to be expected, it seems obvious that just-starting trainees may be very sensitive to their learning environment, as well as to the attitude of their trainers toward others. This sensitivity is in contrast to that of trainees who have more working experience and may focus on their future as independent physicians, thus experiencing a greater need for and attending more to professional skills.
Elzubeir and Rizk8 conducted a study in a medical school in the United Arab Emirates and discovered that men assign more importance than women to “friendliness” (which is one item in the Caring Attitude component) as a characteristic for a role model. Elzubeir and Rizk anticipated this gender difference because female trainees in Middle Eastern culture are not likely to give their predominantly male role models a high score for friendliness. In their study, women rated nine characteristics (which collectively fall both within the Caring Attitude and the Effectiveness component) as more important than men rated them.
Even though the majority of trainees in our study were female and the majority of trainers were male (Tables 1, 2, and 3), we detected no sex differences in how trainees rated their trainers. Our results are consistent with those of Wright and Carrese,12 who also found that gender has no influence on the identification of an excellent role model.
Despite the high correlation between the scores on the RoMAT and the CCTEI (r = 0.765), the CCTEI scores (mean 3.89 ± SD 0.502) are lower than the RoMAT scores (mean 4.19 ± SD 0.509). This difference is significant (t test, P < .001), indicating that the trainees can distinguish between their trainer as a teacher and their trainer as a role model on a trainer assessment, which in turn indicates that the RoMAT may serve as a possible tool for evaluating role model behavior. Furthermore, the trainees evaluated their clinical trainers with varied scores on the RoMAT items; their use of the full range of choices (from 1 to 5 on the Likert scale) indicates the RoMAT’s capacity to reveal positive role modeling.
Given that the CCTEI measures teaching skills, trainees could be expected to assign high scores on the CCTEI for trainers who had attended teach-the-teacher courses. The GP clinical trainers assessed in our study included both new and very experienced trainers (Table 2). As demonstrated by Wright and Carrese,12 the completion of more teaching-training courses is not a condition for being identified as an “excellent” role model. Moreover, as observed by Paice et al,27 regardless of teaching qualities, being a role model depends largely on what happens when clinical trainers are doing other things (i.e., not explicitly teaching). Because the RoMAT assesses qualities other than those addressed by the CCTEI, the scores can differ significantly.
We initially expected that the RoMAT would discriminate among three domains of qualities: personal, teaching, and clinical.9–11 As shown in Table 4, however, the three domains are equally divided between two components (Caring Attitude and Effectiveness), thus demonstrating that competence in all 3Hs is a condition for positive role modeling.
Because completing the RoMAT was voluntary, selection bias is a possibility. The nonparticipation of any trainees who were not satisfied with the role-modeling behavior of their clinical trainers and who refused to participate because they feared the consequences of their evaluations could have had a positive influence on the total scores; however, the scores, reflecting the full range of 1 to 5 on the Likert scale, suggest that this was not the case.
We tested the RoMAT in a primary care setting where only one or two clinical trainers are available for each trainee. All of the GPs in our study voluntary chose to become clinical trainers. Additional research could reveal whether the RoMAT scores of trainees in other clinical settings, especially where multiple role models are available or where trainers have priorities other than working with trainees, would result in other components or in lower or more diverse and discriminating scores.
Implications for research
As shown in a previous study,13 role-modeling behavior can improve after the role model receives feedback on role model assessments. The RoMAT could be similarly used for evaluating and providing feedback to clinical trainers, thus helping them to improve their own role-modeling behavior. Further research is necessary to determine whether improvement would be represented with higher scores on the RoMAT. Wright and colleagues4,12 have suggested that education or training, adjusted to address the identified attributes, could improve the clinical trainer’s role model behavior. Therefore, further research may show if integrating targeted role model behavior training into a curriculum for clinical trainers would, as evaluated by trainees using the RoMAT, result in higher scores.
The RoMAT has proven to be a valid and reliable instrument for assessing the role model behavior of clinical trainers. Educational managers, such as program directors, can use the RoMAT to improve clinical trainers’ awareness of their work as role models. Trainees can also use the RoMAT to identify which aspects of their clinical trainers’ professional behavior they should emulate.
Acknowledgments: The authors are sincerely grateful to Nienke Buwalda, Minou Snijders Blok, and Gerscha Jochemsen for their contributions in collecting and processing the data for this study.
1. Park J, Woodrow SI, Reznick RK, Beales J, MacRae HM. Observation, reflection, and reinforcement: Surgery faculty members’ and residents’ perceptions of how they learned professionalism. Acad Med. 2010;85:134–139
2. Yazigi A, Nasr M, Sleilaty G, Nemr E. Clinical teachers as role models: Perceptions of interns and residents in a Lebanese medical school. Med Educ. 2006;40:654–661
3. Maker VK, Curtis KD, Donnelly MB. Are you a surgical role model? Curr Surg. 2004;61:111–115
4. Wright SM, Kern DE, Kolodner K, Howard DM, Brancati FL. Attributes of excellent attending-physician role models. N Engl J Med. 1998;339:1986–1993
5. Côté L, Maheux B, Beaudoin C. Role models of residents graduating in family medicine and in different specialties in Quebec [in French]. Can Med Physician. 1997;43:907–913
6. Wear D, Skillicorn J. Hidden in plain sight: The formal, informal, and hidden curricula of a psychiatry clerkship. Acad Med. 2009;84:451–458
7. Weissmann PF, Branch WT, Gracey CF, Haidet P, Frankel RM. Role modeling humanistic behavior: Learning bedside manner from the experts. Acad Med. 2006;81:661–667
8. Elzubeir MA, Rizk DE. Identifying characteristics that students, interns and residents look for in their role models. Med Educ. 2001;35:272–277
9. Wright S. Examining what residents look for in their role models. Acad Med. 1996;71:290–292
10. Fromme HB, Bhansali P, Singhal G, Yudkowsky R, Humphrey H, Harris I. The qualities and skills of exemplary pediatric hospitalist educators: A qualitative study. Acad Med. 2010;85:1905–1913
11. Jochemsen-van der Leeuw HG, van Dijk N, van Etten-Jamaludin FS, Wieringa-de Waard M. The attributes of the clinical trainer as a role model: A systematic review. Acad Med. 2013;88:26–34
12. Wright SM, Carrese JA. Excellence in role modelling: Insight and perspectives from the pros. CMAJ. 2002;167:638–643
13. Maker VK, Curtis KD, Donnelly MB. Faculty evaluations: Diagnostic and therapeutic. Curr Surg. 2004;61:597–601
14. Copeland HL, Hewson MG. Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Acad Med. 2000;75:161–166
15. Lombarts MJ, Arah OA, Busch OR, Heineman MJ. Using the SETQ system to evaluate and improve teaching qualities of clinical teachers [in Dutch]. Ned Tijdschr Geneeskd. 2010;154:A1222
16. Nation JG, Carmichael E, Fidler H, Violato C. The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis. Med Teach. 2011;33:e290–e296
17. Fluit C, Bolhuis S, Grol R, et al. Evaluation and feedback for effective clinical teaching in postgraduate medical education: Validation of an assessment instrument incorporating the CanMEDS roles. Med Teach. 2012;34:893–901
18. Stalmeijer RE, Dolmans DH, Wolfhagen IH, Muijtjens AM, Scherpbier AJ. The Maastricht Clinical Teaching Questionnaire (MCTQ) as a valid and reliable instrument for the evaluation of clinical teachers. Acad Med. 2010;85:1732–1738
19. Quill TE. Medical resident education. A cross-sectional study of the influence of the ambulatory preceptor as a role model. Arch Intern Med. 1987;147:971–973
20. Maudsley RF. Role models and the learning environment: Essential elements in effective medical education. Acad Med. 2001;76:432–434
21. Côté L, Leclère H. How clinical teachers perceive the doctor–patient relationship and themselves as role models. Acad Med. 2000;75:1117–1124
22. Jochemsen-van der Leeuw HG, van Dijk N, Wieringa-de Waard M. Attitudes towards obesity treatment in GP training practices: A focus group study. Fam Pract. 2011;28:422–429
23. Shabo G, Van der Wijngaart H, Sana S, Scherpbier AJ, Gemke RJBJ. National study among Dutch paediatric trainees to examen supervisors’ competencies and aspects of good role modelling [in Dutch.] Netherlands J Med Educ. 2010;29:237–244
24. Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35:382–385
25. Field A Discovering Statistics Using SPSS. 20093rd ed Los Angeles, Calif SAGE:627–682
26. Wetzel AP. Factor analysis methods and validity evidence: A review of instrument development across the medical education continuum. Acad Med. 2012;87:1060–1069
27. Paice E, Heard S, Moss F. How important are role models in making good doctors? BMJ. 2002;325:707–710