Scholarship is an important element of both undergraduate and graduate medical education (GME), and as the pediatric community has worked to fully integrate a competency-based assessment model into its residencies and fellowships, we discovered a lack of tools for evaluating trainees’ level of performance in scholarship. Participation in scholarly activity is highly encouraged for residents in all specialties but is a requirement for pediatric subspecialty training. As noted in the Accreditation Council for Graduate Medical Education (ACGME) requirements for each of the three-year pediatric subspecialties, “Each fellow must design and conduct a scholarly project … with the guidance of the fellowship director and a designated mentor.”1 Scholarly activity is also a requirement for board eligibility for all American Board of Pediatrics (ABP)-certified pediatric subspecialties.2
Despite the importance of scholarship across all specialties, data are limited regarding the assessment of scholarship both within medicine and in other fields. Furthermore, the medical community has not defined the specific competencies and behaviors for progressive levels of performance in scholarship or the appropriate level of supervision for a given performance level.
Recently, the ABP collaborated with educational leaders within the pediatric community to develop entrustable professional activities (EPAs), which represent the important observable professional activities required to independently practice a specific specialty or subspecialty. EPAs are a key element of an outcomes-based approach to assessment, as they define the context for competency and milestone assessment and unify the approach to assessment across the full continuum of medical education. Through an ABP-sponsored consensus conference, seven EPAs that cross all pediatric subspecialties were identified.3 One of these common EPAs is based on scholarship, given that pediatric subspecialty fellows have a requirement to perform a scholarly activity.4 In creating the scholarship EPA, the conference participants relied on Boyer’s5 definition of scholarship, which encompasses not only traditional research but other academic efforts including quality improvement, health services, and education-based research. Thus, the scholarship EPA recognizes scholarly activity using this same broad definition. While the other six common subspecialty EPAs were easily mapped to the existing pediatrics competencies and milestones, the group struggled to do the same for the scholarship EPA because the range of skills needed for scholarly activity had not been defined.
As the authors began approaching the problem of a lack of tools to assess scholarship, we felt that using a framework different from that used for the other EPAs would run the risk of either over- or undervaluing research as a professional activity. Hence, we felt strongly that assessment of scholarship, through incorporation into pediatrics’ existing framework of EPAs, competencies, and milestones, was the best implementation strategy. Additionally, although we could assess the scholarship EPA using level of supervision scales that did not link to competencies and milestones, we felt that doing so would mean giving up the shared mental model of performance levels that the milestones bring to assessors and the learning road map that they provide to trainees. We believed that the loss of either of these elements would weaken the assessment process and outcomes. Our aim in this project was to develop competencies and milestones for the scholarship EPA to provide a framework for the assessment of this important professional activity across the educational continuum, while also developing a level of supervision scale consistent with those that exist for the other common pediatric subspecialty EPAs.
Initially, in May 2015, we performed a literature search that revealed three potential paradigms6–8 in which the scholarship EPA could be conceptualized. The first of these was from Glassick,6 who proposed six standards for scholarship applicable to all of the forms of scholarship proposed by Boyer5; however, this framework did not include performance levels. CanMEDS, which includes competencies specific to the physician–scholar, provided a second potential framework but did not provide a comprehensive set of milestones across the educational continuum.7 The third framework, the Vitae Researcher Development Framework (RDF),8 was the one that best served as a template for this effort. The RDF was created to support the training of investigators in the United Kingdom.8 It was constructed based on interviews with researchers and includes four domains: (1) knowledge and intellectual abilities; (2) personal effectiveness; (3) research governance and organization; and (4) engagement, influence, and impact. Within each domain are the equivalent of competencies, and distinct stages of developmental narratives, which are similar to milestones. Although the RDF had an extensive framework from which to guide a traditional research career, it was lacking in other aspects. For example, the RDF encompassed many areas in the general domain of research but had an insufficient number and inadequate description of milestones or performance levels, with some competencies only having three levels. Thus, we used parts of the RDF as a template for developing the competencies and milestones for the scholarship EPA, modifying the terminology to encompass all forms of scholarship. The intent was to create five performance levels within each competency using the Dreyfus and Dreyfus model of skill acquisition, as was done for the majority of the other pediatric milestones.9
In developing the assessment tools (Tables 1 and 2), we started in June 2015 by identifying key competences for the scholarship EPA, including those skills required by the ACGME1 and described in the ABP’s scholarship curriculum standards,2 such as performing a literature review and understanding study design, how to apply for funding, biostatistics, and the ethical principles involved in research. We then created specific milestones based on the progression from novice to expert. Using a modified Delphi approach, we addressed the content validity of the proposed competencies and milestones by circulating drafts for feedback in an iterative process to members of the Council of Pediatric Subspecialties, a group of pediatricians from 14 different subspecialties who have expertise in various types of scholarship as well as to specific investigators highly skilled in traditional, medical education, quality improvement, health services, or implementation science research. Beginning in September 2015, we distributed three distinct drafts for comment and modified the drafts on the basis of feedback after each round; broad agreement was achieved with the third draft. We also broadly circulated the last version to pediatric fellowship program directors because their position requires that they both perform scholarly activity and supervise that of trainees.1 After we created the final version of the competencies and milestones in August 2016, the Steering Committee of the Subspecialty Pediatrics Investigator Network created a five-point level of supervision scale, modeled after one it had previously developed for the other common pediatric subspecialty EPAs, in October 2016.10
In the final version of the assessment instruments, we identified eight competencies important in making entrustment decisions related to scholarship (Table 1). These competencies largely mirror those suggested by Glassick,6 providing additional evidence for the content validity of this tool. For each competency, we created five milestone levels and, to be consistent with the other pediatric milestones, these levels span the continuum from novice to expert. We also aligned the milestone levels vertically so as to maintain consistency across the eight competencies. Although the RDF, which is a research framework, served as the template to develop the instrument, we modified the original milestone descriptions from the RDF to be applicable to all types of scholarly activity. We purposely did not include the word “research” because this term tends to be associated only with more traditional hypothesis-driven research efforts. Thus, to be as inclusive as possible, we used terms such as “scholarship” in place of “research,” “scholarly project” rather than “research project,” and “question or intervention” as opposed to “hypothesis.” We also used the term “publication” in a broad sense that includes not only traditional research articles but also knowledge dissemination to the public through other means, such as online portals, conference proceedings, and the development of smartphone applications. We included both publication and peer review as part of knowledge dissemination given the crucial nature of the peer review process in ultimately determining what information becomes available to the public.
Six of the eight competencies are explicitly addressed in requirements stipulated by the ACGME and ABP.1,2 The two that are not, collaboration and mentoring in scholarship, were added on the basis of input from the pediatric experts that emphasized the critical need to incorporate these competencies for one to be productive in scholarly activities.
The level of supervision scale for the scholarship EPA that the Steering Committee of the Subspecialty Pediatrics Investigator Network developed has five progressive levels of entrustment (Table 2). It was modeled after one of the scales created for the other common pediatric subspecialty EPAs, in which the levels progress from observation only, to coaching at the institutional level, to leading at the regional and/or national level. We gathered validity evidence to support the use of the scale for the other common pediatric subspecialty EPAs in a previous study.10
The level of supervision scale that the Steering Committee of the Subspecialty Pediatrics Investigator Network developed for the scholarship EPA is based on a common framework used for the other common pediatric subspecialty EPAs and adapted to specific strategies for the supervision of trainees performing this specific EPA. A study of the scales for the other six common pediatric subspecialty EPAs provided strong validity evidence, which lends some support to the use of this level of supervision scale.10 Next steps for this EPA will involve a study of this scale’s validity and to determine the correlation between milestone and supervision levels. We recently administered a survey to all pediatric fellowship program directors asking them to choose a targeted supervision level expected for graduation from fellowship and a targeted level that equates with entrustment for unsupervised practice for the scholarship EPA, as well as for all EPAs relevant to that subspecialty. The next steps for this project will be to follow a cohort of fellows over their three years of training to study actual performance in all common and subspecialty-specific EPAs. Our plan is to use the results of these and other ongoing studies to inform the setting of national standards for the performance of all pediatric subspecialty EPAs, including the scholarship EPA.
Curricular components for all the subspecialty EPAs, including the scholarship EPA, have been completed and are available on the ABP website for program directors and learners to use as a guide during training.3 We focus on the scholarship EPA in this report on the basis of (1) the scarcity of assessment strategies for scholarly skills, which are required in some residencies and fellowships; and (2) the road map for learning and the substrate for feedback that competencies and milestones can provide to trainees. The competencies and milestones that we created provide the framework for this EPA and resemble the other ACGME competencies and milestones in that they are context independent. This makes them suitable for adoption by other disciplines and professions in which scholarly activity is a training requirement.
We believe that this work can help to fill the gap on scholarship competencies and milestones in medical education, training, and practice because they were constructed to be applicable across this broad continuum. In pediatric fellowships, trainee progress in scholarship is monitored by a scholarship oversight committee. Ideally, this EPA and the associated competencies could be used by this committee to follow the growth of fellows via the attainment of milestone levels and the level of supervision required, in the same manner that clinical competency committees use this information to assess other EPAs, competencies, and milestones. In creating the milestones, we have no expectation that trainees, and even some faculty, will achieve a high level for any specific competency.
This type of assessment will hopefully serve as a road map for trainees interested in scholarly pursuits during undergraduate medical education and GME, as well as during the transition from training to a junior faculty position. Implementation and study by others outside of pediatric GME settings will address the instruments’ potential for generalizability across disciplines, professions, and the educational continuum. The use of the Dreyfus and Dreyfus model for creating the progressive performance levels from novice to expert facilitates the use of these milestones to judge professional improvement during early cycles of maintenance of certification (MOC). That is, advancing from competent to proficient to expert or requiring less supervision to perform scholarly activities provides a possible option for earning MOC credit as it demonstrates self-improvement. Especially important is the potential role that these instruments could play in the assessment of physician–scientists, including MD/PhD students, who begin their PhD work in medical school and continue their professional research development throughout GME and into practice. Nonetheless, while these tools do provide a new opportunity to address the assessment of a broad variety of scholarly activities across the educational continuum, further investigation is needed to determine their true value.
The authors greatly appreciate the feedback provided by Laurel Leslie, MD, MPH, Alan Schwartz, PhD, Robert Englander, MD, MPH, and Joseph St. Geme III, MD, as well as the feedback from the Steering Committee of the Subspecialty Pediatrics Investigator Network, members of the Council of Pediatric Subspecialties, and the pediatrics community.
1. Accreditation Council for Graduate Medical Education. Pediatrics: Program requirements and FAQs. http://www.acgme.org/Specialties/Program-Requirements-and-FAQs-and-Applications/pfcatid/16/Pediatrics
. Accessed May 30, 2018.
2. American Board of Pediatrics. General criteria for subspecialty certification. https://www.abp.org/content/general-criteria-subspecialty-certification
. Accessed May 30, 2018.
3. American Board of Pediatrics. Entrustable professional activities for subspecialties https://www.abp.org/subspecialty-epas
. Accessed May 30, 2018.
4. American Board of Pediatrics. Curricular components for common subspecialty EPA. https://www.abp.org/sites/abp/files/pdf/scholarly_activity_combined.pdf
. Accessed May 30, 2018.
5. Boyer EL; Carnegie Foundation for the Advancement of Teaching. Scholarship Reconsidered: Priorities of the Professoriate. 1990.New York, NY: Jossey-Bass.
6. Glassick CE. Boyer’s expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000;75:877880.
7. Royal College of Physicians and Surgeons of Canada. CanMEDS guide. http://canmeds.royalcollege.ca/guide
. Accessed May 30, 2018.
9. American Board of Pediatrics. The Pediatric Milestones Project. https://www.abp.org/sites/abp/files/pdf/milestones.pdf
. Accessed May 30, 2018.
10. Mink RB, Schwartz A, Herman BE, et al.; and the Steering Committee of the Subspecialty Pediatrics Investigator Network (SPIN). Validity of level of supervision scales for assessing pediatric fellows on the common pediatric subspecialty entrustable professional activities. Acad Med. 2018;93:283291.