Li, Su-Ting T. MD, MPH; Paterniti, Debora A. PhD; Tancredi, Daniel J. PhD; Co, John Patrick T. MD, MPH; West, Daniel C. MD
Self-directed lifelong learning is widely recognized as an integral component of medical professionalism1–3 and of practice-based learning and improvement (PBLI).4 Many countries require documentation of self-directed lifelong learning during residency training as well as for board certification and maintenance of certification.5,6 Despite the importance of self-directed learning, many physicians do not have the skills necessary to engage in it effectively.7
In graduate medical education, the use of individualized learning plans (ILPs) has the potential to help trainees develop lifelong learning skills by creating a structure for them to assess their individual learning needs, create personal learning goals, and measure their own progress toward achieving these goals.8 ILPs could be particularly useful for competencies that are difficult to teach and evaluate,9,10 such as PBLI and systems-based practice (SBP),11 by allowing learners the opportunity to develop and work on learning goals targeting those competencies. The Review Committee for Pediatrics of the Accreditation Council for Graduate Medical Education (ACGME) has underscored the potential importance of ILPs by requiring all pediatric residency and fellowship programs to use them.12
Although ILPs show great promise, little is known regarding the best ways to design and use ILPs. We previously developed a model describing the critical elements of successful self-directed learning,13 but the relationship between type of learning goal and the likelihood of success has not been determined.
Using the six ACGME general competencies4 as our framework, we examined which learning goals residents choose to pursue in their ILPs, which goals they identify as being the most important, and how the goal characteristics relate to residents' progress toward successful goal completion. By improving understanding of these areas, we hope to help educators identify better ways to train physicians to engage in effective self-directed learning, which may lead to better-trained and more-reflective physicians and, ultimately, to higher-quality patient care.
We performed a national, cross-sectional, Web-based survey of U.S. pediatric and combined pediatric (e.g., medicine/pediatric, pediatric psychiatry) residents in academic year 2008–2009. At the 2008 Association of Pediatric Program Directors (APPD) annual spring conference and through the APPD's listserv, we invited pediatric program directors to allow their residents to participate in this study. We described the study methods previously.7,13
The survey asked for demographic information and posed a mix of multiple-choice and open-ended questions about residents' experiences with ILPs. Residents who responded “yes” to the question “Have you ever written an individualized learning plan (ILP) for yourself?” were asked to identify (1) their “most important” goal and the amount of progress they had made on it, (2) the goal on which they had made the “most” progress, and (3) the goal on which they had made the “least” progress. For the purposes of this study, we considered residents to have completed the survey if they responded to at least one of these three qualitative questions. Residents rated their progress on the most important learning goal using a five-point Likert-type scale (1 = no progress, 2 = a little progress, 3 = some progress, 4 = a lot of progress, and 5 = met goal).
Of the 196 pediatric residency training programs in the United States, 46 (23%) participated in the survey. This study was approved by the institutional review board (IRB) at the University of California, Davis (UC Davis). The majority of residents at participating institutions were recruited by our research team under the UC Davis IRB's approval; however, 13 programs received additional IRB approval from their home institution to permit program directors to recruit their own residents into the study.
Using inductive iterative review, two study investigators (S.T.L., D.A.P.) systematically analyzed the responses to the following open-ended question: “For your most recent ILP, what would you consider your most important learning goal?” The inductive content analysis involved three phases: organizing existing data, generating lists of recurring patterns, and categorizing redundant patterns into larger themes based on the ACGME general competencies.14
Responses were entered into an Excel spreadsheet to facilitate iterative review and comparative coding (organization). When more than one learning goal was listed by a resident, we considered only the first listed goal in the analysis. We analyzed the first listed goal because we reasoned that it was the primary goal the resident thought of when answering the question. To avoid double-counting participants, we did not consider subsequent goals in the analysis.
S.T.L. and D.A.P. reviewed each response and noted predominant and recurring themes in residents' learning goals (list generation). These two investigators discussed emerging themes until reaching consensus about the themes that provided the most parsimonious and illustrative interpretation of the data. Frequency distributions were calculated for each of the recurring themes found in the data. Finally, redundant patterns were grouped into larger categories that included the ACGME general competencies—medical knowledge (MK), patient care (PC), interpersonal and communication skills (ICS), PBLI, professionalism (Pr), and SBP.
S.T.L. and D.A.P. repeated this process of thematic review and analysis for responses to the other two questions: “For your most recent ILP, what goal did you make the most progress toward achieving?” and “For your most recent ILP, what goal did you make the least progress toward achieving?”
We determined the number and relative frequency of types of goals residents considered to be the most important, types of goals residents reported the most progress toward achieving, and types of goals residents reported the least progress toward achieving. We used the Fisher exact test to examine whether learning goal type was associated with gender or with level of training.
To understand the relationship between the type of learning goal and progress on goal achievement, we performed several different analyses. For analyses involving residents' reports of most and least progress on goals, we reasoned that if a goal type were not related to goal achievability, it would be equally likely to be reported in the most and least progress categories. Hence, for each goal type, we counted the number of goals of that type among residents' reports of the most and the least progress and then computed the ratio of these two numbers, which we refer to as the (most:least) progress ratio or odds, depending on the context. We restricted the statistical analysis of these ratios to residents who provided valid reports for both the most and the least progress items. In addition, we excluded reports from residents whose most and least progress goals were of the same type, because concordant reports do not contribute useful information for the statistical testing of the progress ratios. We report goal-type-specific progress ratios with exact binomial 95% confidence intervals (CIs). The exclusion of “1” from the 95% CI indicates statistical significance at the 5% level. For example, when the CI for a goal type lies entirely above 1, goals of that type are statistically significantly more likely to be reported as most progress than as least progress goals, indicating greater achievability.
We used conditional logistic regression to compare the goal-type-specific progress ratios, analyzing the most and the least progress goals from each resident as a matched pair. This approach is analogous to the analysis of matched case–control data. In our application, each resident serves as his or her own match, with most progress goals analogous to cases and least progress goals analogous to controls. Conditional logistic regression is ideal for this purpose because it statistically adjusts odds ratios (ORs) and CIs for the main effects of all resident characteristics (both measured and unmeasured) by treating each resident as his or her own stratum. Each resident contributed two observations, with the dependent variable coded “1” (“0”) for the goal with the most (least) progress. For the independent variable, dummy variables were used to indicate the type of goal on which residents reported having made either the most or the least progress. We report an adjusted OR for each type of goal, using MK goals (the most frequently identified goal type) as the reference category. Therefore, a goal type whose OR is statistically significantly greater (less) than 1 is associated with relatively more (less) progress on achieving goals of that type than on MK goals.
To estimate the effect of goal type on the amount of progress residents reported on their most important learning goal, we performed multilevel linear regression, adjusting for resident- and program-level factors—drawn from our previously published model7—that we thought might be associated with progress on learning goal achievement. Resident-level factors included resident demographics, confidence in self-directed learning abilities, learning style, propensity for lifelong learning, attitude toward ILPs, and ILP-related factors. Resident demographics included gender, level of training, and career interest (generalist, specialist, undecided). Learning style was measured by the previously validated 12-item Kolb Learning Style Inventory, which categorizes learners into four learning styles (converging, accommodating, diverging, and assimilating).15 Propensity for lifelong learning was measured by the previously validated 14-item Jefferson Scale of Physician Lifelong Learning.16 ILP-related factors included the resident's thinking that an ILP meets his or her learning needs, the resident's tracking progress on achieving the learning goal,16 time elapsed since the resident last wrote an ILP, and number of learning goals the resident included in his or her last ILP.
Program-level factors included residency program size (small, medium, large)15; setting (university affiliated, community based); support for residents for ILPs (written directions, sample completed ILP, conference introducing ILPs, protected time to meet with advisor); and support for faculty advisors for ILPs (conference introducing ILPs, no faculty support for ILPs).7 We did not include SBP-type goals in the model because too few residents (n = 6) identified an SBP goal as their most important goal. All analyses were conducted using SAS version 9.2 (SAS Institute Inc., Cary, North Carolina).
Participating institutions and residents
Of the 1,739 residents at the 46 participating institutions, 992 (57%) completed the survey.7,13 The majority (78%; 772/992) of residents who completed the survey had participated in the ILP process prior to the survey. A total of 668 residents (87% of the 772 residents who had previously completed an ILP and 38% of the 1,739 eligible residents)13 responded to at least one of the three open-ended questions regarding their experience with the ILP process. This group of 668 residents served as the sample for this study. These respondents' residency programs included pediatric and combined pediatric programs (e.g., medicine/pediatrics, pediatric psychiatry) and were representative of all U.S. pediatric residency programs.7
As we reported previously, the demographic distribution of the 992 respondents reflected those of all U.S. pediatric residents in terms of postgraduate year, gender, and subspecialty interest distribution.13,17 Characteristics of the 772 residents who had previously completed an ILP were also similar to those of all U.S. pediatric residents.13
We categorized residents' responses regarding their most important learning goal and the goals on which they reported the most and least progress into seven goal types: the six ACGME general competencies plus future practice. When possible, goals assigned to each type were subcategorized. The learning goal types, subcategories, and examples are shown in Table 1. Few residents (n = 13) reported more than one learning goal as being the most important. Even fewer residents reported more than one learning goal as the one on which they had made the most progress (n = 11) or the one on which they had made the least progress (n = 1).
Residents most frequently identified MK (327/609; 53.7%) and PC (158/609; 25.9%) goals and least frequently identified Pr (9/609; 1.5%) and SBP (6/609; 1.0%) goals as being the most important (Table 2). Similarly, compared with other types of goals, residents more frequently reported the most progress on goals related to PC (192/504; 38.1%) and MK (171/504; 33.9%) and less frequently reported the most progress on goals related to Pr (15/504; 2.9%) and SBP (4/504; 0.8%). No residents identified an SBP goal related to patient safety or quality improvement as their most important goal or the goal on which they had made the most progress. There was no difference in distribution of goal types by gender or by level of training (data not shown).
Progress ratios comparing the number of times each goal type appeared in the most progress category and the number of times it appeared in the least progress category are shown in Table 3. For this analysis and the analyses that follow, we analyzed two of the PBLI subcategories—evidence-based medicine and teaching—separately because they were so clearly different from each other. We did not include the third PBLI subcategory—self-reflection—in the analyses because of insufficient numbers. We found that PC (128/186; 69%) and PBLI-teaching (40/54; 74%) goals were more likely to be reported in the most progress category than in the least progress category. SBP (3/27; 11%) and MK (94/237; 40%) goals were less likely to be reported in the most progress category than in the least progress category. In our conditional regression model, when compared with MK learning goals, residents reported significantly greater progress on PC goals (OR: 2.20; 95% CI: 1.57–3.09) and PBLI-teaching goals (OR: 2.99; 95% CI: 1.59–5.63), and they reported significantly less progress on SBP goals (OR: 0.16; 95% CI: 0.05–0.56). Reported progress on other types of learning goals was not significantly different compared with progress on MK goals.
After accounting for factors we had previously hypothesized to be associated with progress on learning goals,7 we found that residents with PBLI-teaching goals as their most important goal reported less progress on achieving their goal (β = −0.19; 95% CI: −0.37 to −0.01) compared with those who reported MK goals as their most important. No other goal type was significantly associated with more or less progress toward goal achievement. The relationships of other factors from our previous model to progress on the most important goal were essentially identical to our previous findings (data not shown).8 Importantly, residents' use of a system to track progress on achieving learning goals was the most significant factor associated with greater progress, independent of learning goal type. Propensity for lifelong learning, greater confidence in self-directed learning abilities, and more time elapsed since last ILP written were also significantly associated with greater progress. Being undecided on a specialty was associated with less progress on achieving the most important learning goal.
In this study, we found that the progress residents made on achieving self-directed learning goals was dependent on the type of learning goals they selected. Our conditional logistic regression model showed that, compared with an MK goal, the odds of a resident making the most (versus the least) progress were two times greater on a PC goal and nearly three times greater on a PBLI-teaching goal. By contrast, the odds of a resident making the most progress on an SBP goal were more than six times lower than the odds on an MK goal. Based on these findings, we conclude that residents make more progress on goals related to everyday duties and tasks, such as PC and PBLI-teaching, than they do on goals that are less integrated into everyday training, such as SBP.
Residents' progress on their most important learning goal was less dependent on goal type than it was on their progress on other learning goals. When we added goal type to our previously reported regression model,7 we found that goal type was no longer associated with progress on the most important learning goal except for goals of the PBLI-teaching type. Residents who stated that their most important goal was a PBLI-teaching goal reported relatively less progress than did residents who stated that their most important goal was an MK goal. This finding is particularly striking because, in our conditional logistic regression model, residents were more likely to report the most progress on PBLI-teaching goals compared with MK goals. This difference would be useful to explore in subsequent studies to determine how the learning goal's importance to the individual learner influences the learner's achievement of that goal. Importantly, the addition of goal type to our regression model did not change our previous findings that using a system to track progress toward goal achievement was strongly associated with more progress.7 Future studies should explore the strategies residents use to track their progress successfully, including formal tracking systems (e.g., checklist, calendar, daily planner) or informal methods (e.g., frequent informal check-ins with advisor) as well as external (e.g., faculty, peer) or internal monitoring.
Residents were more likely to identify MK and PC learning goals as most important compared with SBP or Pr goals. In fact, no resident identified an SBP goal related to patient safety or quality improvement as their most important goal or as a goal on which they had made the most progress. This observation is particularly significant given the increased emphasis on quality improvement, patient safety, and professionalism as expectations for maintaining certification18 and for adequate residency training.4,11 It could be that residents value MK and PC learning goals because these goals are more closely related to what they need to know to care for patients effectively on a daily basis. Training programs may also provide greater acknowledgment of MK and PC goals than of SBP goals, either through the purposeful design of curricula or the hidden curricula19 of residency training. Regardless, future studies should examine why residents value some learning goals over others and what strategies might be used to increase the prioritization and likelihood of successful completion of certain goal types, such as SBP.
Because goal development is such a critical step in effective, self-directed lifelong learning, it is important that residents acquire the skills to identify learning goals during training. Evidence from other studies suggests that traditional medical training may not adequately prepare physicians to formulate their own learning goals or provide them with the skills to accomplish them effectively.20,21 For example, in one study, third-year medical students preferred choosing from a list of goals provided to them rather than creating their own goals.20 In another study, residents reported lacking confidence in their ability to formulate and follow through with a plan to accomplish their learning goals.21 Our findings suggest that this lack of confidence may be related to the type of goal that a learner is trying to accomplish.
We included only pediatric and pediatric combined residents in this study, so it is possible that learners from other disciplines (e.g., surgical subspecialties) or levels of medical education (e.g., medical students, practicing physicians) would prioritize different learning goals or report different degrees of progress on goals than those found in our study. In addition, our outcome measure was based on resident self-report of progress on learning goals rather than external measures of goal progress. Although this is a limitation, we chose this approach for practical reasons because it allowed execution of a large, multi-institution, survey-based study. Further exploration through in-depth semistructured interviews could provide more nuanced understanding of factors related to goal achievement. In addition, better clarification of best practices for goal development and achievement might suggest avenues for trainees and practicing physicians to engage in more-effective self-directed learning activities that could ultimately lead to better patient care.8,13
Residents' progress on self-directed learning activities is dependent on the type of learning goal that residents choose to pursue. In our study, residents made significantly more progress on goals related to day-to-day patient care compared with other types of goals. Training programs should therefore enhance the opportunities for residents to integrate their learning goals into day-to-day practice. Less traditional competencies, such as SBP, seem to be the most difficult targets for learning goal development and achievement, which suggests that educators should make special efforts to reframe these competencies in ways that are relevant to clinical practice. Beyond the impact of goal type on successful ILP completion, our findings reinforce our previous recommendation that training programs should provide systems for residents to track their progress toward achieving learning goals.
Future research should focus on more directly testing ILPs' effectiveness in residents' development of self-directed learning skills and establishing the relationship of physicians' self-directed learning skills to the quality of patient care provided. If ILPs were to be shown effective in improving self-directed learning skills, and if improved self-directed learning skills were found to be associated with higher-quality patient care, such findings would provide compelling evidence to support the expansion of ILPs to training programs in all medical specialties and to maintenance of certification activities.
The authors thank the residents and program directors who participated in this study. The authors also thank Mohammadreza Hojat, PhD, for permission to use the Jefferson Scale of Physician Lifelong Learning, and HayGroup for permission to use the Kolb Learning Style Inventory.
Funding for this study was provided in part by a special projects grant from the Association of Pediatric Program Directors. The funders had no role in the concept and design, analysis, interpretation of data, or drafting or revising of the manuscript.
Institutional review board exemption was granted by the University of California, Davis. In addition, the institutional review boards at 13 additional institutions approved this study: Bronx–Lebanon Hospital Center, Creighton University/University of Nebraska Medical Center, Connecticut Children's Medical Center, Kaiser Permanente Southern California, Medical College of Georgia, Meridian Health, Mount Sinai School of Medicine, New York Methodist Hospital, Orlando Regional Healthcare, Saint Louis University, University of South Alabama, University of Utah, and Weill Cornell Medical College.
This study was presented in part at the Pediatric Academic Societies Annual Meeting; May 1–4, 2010; Vancouver, British Columbia, Canada.