In our Q-sort game, participants start with 22 playing cards representing the milestones and place them in the appropriate column for priority. For example, the participant in Figure 1 prioritized the milestone coded as PC/F1 as the most important and thus placed it in column 7; she placed milestones PC/A1, PC/C1, and PC/B1 in column 6, denoting the second most important position (all items in a column are considered to be of equal importance). She placed the four milestones she prioritized as least important in columns 1 and 2. Participants move the milestones around on the board until they are satisfied with the rank order. When the game is complete, they tape the milestones to their boards and give the final prioritization schemes to the facilitator. The top eight milestones are those placed in columns 7 to 5, which denote milestones prioritized higher than neutral.
Q-sort game participants and procedure
IM faculty at the 13 E-ROC programs participated in the EsAMB Q-sort exercise over the period May 2010 to December 2010. Each program’s PI invited all ambulatory faculty to a 90-minute workshop at which she or he provided directions for the exercise using the faculty development slide set and instructions we created. We distributed the game electronically to the site PIs, who had them printed locally with minimal cost to their institutions. At one institution, the game was played electronically but faculty received the same instructions. Other variations included the setting and timing of the Q-sort activity. Some programs performed the exercise during a broader, time-protected program educational day, and others did it during short, directed sessions. Some programs gathered large groups of faculty, whereas others involved only core faculty.
Faculty, working in faculty units of one to four individuals, were asked to rank the 22 milestones on the basis of their importance in deciding whether a trainee had reached the EsAMB landmark. Site PIs collected the completed game boards from participants and reported the numerical values on each gameboard. All programs’ Q-sort results were returned to us by e-mail for collation and analysis.
We analyzed the Q-sort results by calculating the mean rank order of milestones, both by faculty units and by program. We also counted how many faculty units and programs prioritized each milestone in their top eight choices (i.e., those prioritized higher than neutral on the game board). For each milestone, we calculated a program rank order mean by averaging across the program’s faculty unit rank orders; we also calculated a mean for each milestone by averaging across all faculty units. We identified the eight milestones with the highest faculty means as the top priorities. We calculated a percent agreement for each of the high-priority milestones based on the number of programs that had individually ranked them in their top eight; we repeated this analysis using faculty units as the unit ofanalysis.
Each participating institution obtained institutional review board approval or an educational exempt status for this faculty exercise (see Table 1).
The Q-sort game was distributed to a total of 149 faculty units (approximately 250 faculty) at the 13 participating IM residency programs. Six faculty units from different programs reported data incorrectly, and their rankings could not be analyzed. Thus, our analysis was based on data from 143 faculty units at 13 programs (Table 1). Table 2 presents the 22 milestones in order of priority for EsAMB. The top 8 milestones are clustered in the Patient Care (PC) and Professionalism (P) competencies, and 2(25%) of them focus on recognition oflimits.
Among their top 8 milestones, 13 (100%) programs chose PC/A1, PC/F1, and P/F5, whereas 12 (92%) programs chose PC/B1, PC/C1, and PC/C2 (Table 3). Thus, 6 (75%) of the top 8 milestones had at least 92% agreement. For the seventh- and eighth-priority milestones, there was a drop in the level of agreement of both faculty units (80 [56%] and 59 [41%], respectively) and programs (9 [69%] and 4 [31%], respectively). This drop is explained by how few faculty and programs prioritized other milestones in their top 8: 11 of the 14 lower-priority milestones were prioritized in the top 8 by 2 to 31 (0.1%–22%) of the faculty units and 0 to 1 (0%–8%) of the programs. Interestingly, one program chose 3 unique milestones, clustered in the Interpersonal and Communication Skills competency, in its top 8 (IPC/A2, IPC/A3, IPC/A4). Across faculty units, there was greater than 75% agreement for 5 (63%) of the top 8 milestones: PC/F1, PC/A1, P/F5, PC/C1, and PC/C2.
Although not formally studied, the qualitative experience of the participating faculty was reported by the site PIs. They indicated that the Q-sort game was educationally productive and engaged faculty in lively, active discussions about the order of importance as they prioritized the milestones. Faculty completed the Q-sort exercise expressing more confidence in the behaviors they would like to directly observe in order to advance their residents in EsAMB. They also expressed a better understanding of steps along the way to competence as a result of having to differentiate between skills in EsAMB and in more complex ambulatory settings. The PIs felt that the process of sharing perspectives and ideas enhanced the educational experience and thus the exercise was more valuable as faculty development when it was conducted as a group rather than an individual activity. As noted above, one program conducted the Q-sort as an individual activity using an electronic format; this had the benefit of reaching more faculty members by eliminating the need to attend a meeting to perform the exercise. Overall, faculty expressed interest in performing the game again with a new landmark or another group of faculty. Finally, faculty described the exercise as “useful” and “fun,” indicating strong intellectual and creative engagement with these educational concepts.
Milestones and entrustable professional activities (EPAs) are receiving considerable scrutiny in medical education, moving residency training from a “dwell-time,” apprenticeship model to an observed, entrustment-based compilation of targets for competence. To meet the ACGME’s requirement that IM residency programs use reporting milestones4 for the NAS beginning July 2014, programs are charged with developing a structure for implementation. Engaging faculty in assessment using milestones can sometimes be difficult, however. As this study demonstrates, using Q-sort—a standardized research method for the study of subjectivity—can actively engage faculty in prioritizing curricular milestones2 for training landmarks and achieve concordance between programs and participants. Following our Q-sort exercise, site PIs reported that faculty members gained a sense of optimism for direct observation, milestones, andCBME.
One limitation of this study relates to the uncertainty of the initial published set of curricular milestones.2 Milestones identified as priorities in a Q-sort for a landmark (or an EPA) are dependent on the initial milestone set to be sorted. Using an exclusion process such as the Q-sort raises the question of whether other milestones could have been included in our priority set of 22 milestones, perhaps drawn from other sources. In addition, the inclusion of other categories of participants (e.g., nurses or patients) might have influenced the prioritization. Whereas the results of a Q-sort can vary depending on the starting set, participants, and framing issue, we found the general method to be a useful construct to prioritize any subjectively held perspectives.
Another limitation is that all participating programs were EIP programs, chosen by the ACGME in 2006 as programs of excellence for innovation. EIP programs are, therefore, adept at and have institutional support for educational innovation, are willing to take more risks, and have an incentive to produce positive educational outcomes.14 Although these factors may be more apparent in EIP programs, we assert that most residency programs are similarly focused on improving their means of trainee assessment, which may be affected by the implementation of the NAS.
Finally, this research is limited as a tightly controlled study because it is a complex intervention.15 Complex interventions must be adjusted to the local educational environment to be adopted successfully, so we allowed for variability within the overarching guidelines we set for this exercise. This study design intentionally gave local control to the site PI to meet the goal of collaboration in a real-world educational setting. This decision does, however, increase the ecological validity of this study.
The next phase of the EsAMB milestones study is to apply the high-priority curricular milestones to the educational setting. Using the top eight EsAMB milestones, we constructed a feedback tool for faculty to record observations of trainees in the ambulatory setting. Five E-ROC programs (29 faculty) piloted the feedback tool in the ambulatory setting from December 2010 to February 2011. (For the final EsAMB feedback tool, see Supplemental Digital Appendix 1, http://links.lww.com/ACADMED/A137.) We are currently studying faculty and resident acceptance of this evaluation method and its correlates with trainee performance.
Our Q-sort activity is one practical and engaging step toward the development of a new training model. This exercise can be adapted to engage faculty in prioritizing milestones for other EPAs. It is unclear whether there will be the same agreement for other EPAs and whether other EPAs will be as conducive to this type of prioritization method. Similar Q-sort exercises may also be useful along the continuum of physician training from undergraduate education to recertification. Although this study provides a set of eight milestones that faculty ranked as high priority for EsAMB, it is unclear whether these are the correct milestones for evaluation of competence in this landmark. Further study is needed to understand the relevance of Q-sort for applying milestones to assessment.
Many IM programs are actively applying curricular milestones to EPAs such as “discharging a patient safely,” “leading a code effectively,” and “managing a health care team efficiently.” Caution is advised as CBME has not yet been proven to measure distinct attributes that people can demonstrate in their actual work.16 Other important education principles, such as those of andragogy, may be of equal value.17 The paradigm shift to CBME using milestones is sound educational theory driven by public accountability and, thus, may fulfill an overall social mission of training excellence and patient care.16,18
The application of milestones in the real-world educational setting is a new challenge to residency programs. A Q-sort exercise is simple to distribute, cost-effective, and engages faculty in examining relationships between curricular milestones and landmarks/EPAs. As our results show, our Q-sort game enabled diverse programs to prioritize milestones for EsAMB with interprogram and interparticipant consistency. Q-sort is a playful way to address milestones in medical education and may provide a practical first step toward using milestones in the real-world educational setting.
Acknowledgments: The authors wish to thank Dr. William Iobst for guiding the early iterative process and Dr. Terry Albanese and Sarah Hood for research and project support. They would also like to thank the following principal investigators from the corresponding programs: Drs. Biana Ieybishkis and Mark Gennis, Aurora Healthcare; Drs. Sam Ives and Anne Pereira, Hennepin County Medical Center; Drs. Jason Post and Denise Dupras, Mayo Clinic College of Medicine; Dr. David Shaw, Scripps Mercy Hospital; Drs. Siegfried Yu and Andrew Varney, Southern Illinois University; Dr. Rebecca Shunk, University of California, San Francisco, School of Medicine; Drs. Bradley Mathis and Eric Warm, University of Cincinnati; Drs. Mary Thompson and Kathleen O’Connell, University of Wisconsin; and Dr. Christopher Nabors, Westchester Medical/New York Medical College.
Other disclosures: None.
Ethical approval: Ethical approval was granted or the study was determined to be exempt as per each participating institution’s internal review board.
Previous presentations: Data were previously reported at a Workshop for Associate Program Directors Internal Medicine Spring Meeting, Atlanta, Georgia, April 24–25, 2012.
* During the course of this study, the term entrustable professional activity (EPA) gained acceptance for use in the assessment of residents in training. EPAs are the critical activities in a profession that a member of the profession needs to perform competently.7 Our use of the term landmark is likely synonymous with EPA, but for the integrity of the study, we use landmark in this article.
1. Nasca T. The CEO’s first column—The next step in the outcomes-based accreditation project. ACGME Bull. May 2008:2–4
2. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine. J Grad Med Educ. 2009;1:5–20
3. Varney A, Todd C, Hingle S, Clark M. Description of a developmental criterion-referenced assessment for promoting competence in internal medicine residents. J Grad Med Educ. 2009;1:73–81
4. Caverzagie KJ, Iobst WF, Aagaard EM, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158:557–559
5. Meade LB, Borden SH, McArdle P, Rosenblum MJ, Picchioni MS, Hinchey KT. From theory to actual practice: Creation and application of milestones in an internal medicine residency program, 2004–2010. Med Teach. 2012;34:717–723
6. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—Rationale and benefits. N Engl J Med. 2012;366:1051–1056
7. ten Cate O, Scheele F. Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice? Acad Med. 2007;82:542–547
8. Caverzagie K, Hood S, Iobst W. Windows to competence: A framework for implementing milestones to visualize resident competence. October 2010Poster presented at: 31st Annual Alliance for Academic Internal Medicine Conference San Antonio, Tex
9. Kennedy TJ, Regehr G, Baker GR, Lingard L. Point-of-care assessment of medical trainee competence for independent clinical work. Acad Med. 2008;83(10 suppl):S89–S92
11. Brown SR. Q methodology and qualitative research. Qual Health Res. 1996;6:561–567
12. Valenta AL, Wigger U. Q-methodology: Definition and application in health care informatics. J Am Med Inform Assoc. 1997;4:501–510
14. Mladenovic J, Bush R, Frohna J. Internal medicine’s educational innovations project: Improving health care and learning. Am J Med. 2009;122:398–404
15. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew MMedical Research Council Guidance. . Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ. 2008;337:a1655
16. Lurie SJ, Mooney CJ, Lyness JM. Commentary: Pitfalls in assessment of competency-based educational objectives. Acad Med. 2011;86:412–414
17. Hinchey KT, Rothberg MB. Can residents learn to be good doctors without harming patients? J Gen Intern Med. 2010;25:760–761
18. Medicare Payment Advisory Commission. . Graduate medical education financing: Focusing on educational priorities. Report to the Congress: Aligning Incentivesin Medicare. 2010 Washington, DC Medicare Payment Advisory Commission http://medpac.gov/documents/Jun10_EntireReport.pdf
. Accessed May 3,2013
Supplemental Digital Content
© 2013 by the Association of American Medical Colleges