Since 1983, the Academic Pediatric Association (APA) has engaged pediatric educators nationwide in the collaborative development of three successive editions of educational guidelines for residency training. Each edition of these guidelines was shaped in purpose and scope by the changing environment of clinical medicine and medical education.1–3 Pressure to develop a better-articulated and more structured approach to residency training led to revised accreditation standards for residencies. In 1997, the Accreditation Council for Graduate Medical Education (ACGME) mandated the use of written goals and objectives for residency curricula.4 Later, the ACGME focused on strengthening the evaluation of residents and required programs to certify residents’ competence in six competency domains by the completion of their training.5–7 These changes challenged programs in all medical disciplines to improve their curricula.
The APA addressed these challenges by developing a curriculum resource for pediatric residencies that evolved over three editions to become a comprehensive and flexible set of tools. In 1985, the APA published the first edition of the guidelines, Educational Guidelines for Training in General/Ambulatory Pediatrics,1 which outlined a minimum core of pediatric knowledge, skills, and attitudes that should be taught in ambulatory experiences and normal newborn rotations for medical students and residents.8 In 1996, the APA completely rewrote the 1985 edition and published an expanded Educational Guidelines for Residency Training in General Pediatrics.2 This edition provided the first comprehensive set of learning goals and objectives for the education of general pediatricians across all three years of postgraduate training. Finally, in 2004, updated APA Educational Guidelines for Pediatric Residency were published on an interactive Web site.3 This edition offers residency programs a resource for building their own customized, competency-based curricular documents, using interactive tools and a comprehensive database of goals and objectives.
In this article, we describe the seven-step national consensus development process used to create the 1996 and 2004 Guidelines and develop a community of Guidelines users. The Guidelines developers were influenced by a national climate of collaboration around shared curriculum development that was visible in projects reported by several disciplines in the 1990s. The Society of Teachers of Family Medicine (STFM) created a collaborative family medicine clerkship curriculum in 19909; a pediatric clerkship curriculum was published in 1995 by the Council on Medical Student Education in Pediatrics (COMSEP)10–11; and an internal medicine clerkship curriculum was published in 1995 by the Clerkship Directors in Internal Medicine.12–14 Collaborative curriculum development is a common method of the STFM, which resembles the APA in its practice of networking around national projects.15–18 Use of a highly collaborative process for the Guidelines was appropriate because the goal was to develop a flexible product that all residency programs could readily adapt to their own needs. Creation of a standardized, prescriptive curriculum resource was not considered a useful end point. This same strategy of flexibility was adopted by the Clerkship Directors in Internal Medicine for their curriculum.14
Carole Bland and colleagues,19 in an influential article published in 2000, described 35 features of successful curricular change in medical schools which were drawn from a careful review of the educational and business literature. Her process includes numerous elements that resemble our collaborative process—for instance, creation of a cooperative climate, broad participation, strong communications, formative evaluation, training support, and effective guiding leadership—but she was not describing a curriculum development process that crossed institutional borders. Our project was particularly ambitious because it was designed to serve more than 200 residency programs and to address the full scope of three years of residency training. None of the national collaborative projects mentioned above have described their development process in clear steps with sufficient detail to help other groups implement their methods.
Therefore, we wrote this article to describe a national consensus development method that might serve as a useful model for other disciplines. We also demonstrate wide utilization of the Guidelines, and we hypothesize that the systematic collaborative development of this shared resource and its customizable format have enhanced its acceptance, usefulness, and broad dissemination.
The Collaborative Development Process: Key Steps and Lessons Learned
The collaborative development processes used to develop the 1996 and 2004 editions of the Guidelines were similar. The seven steps of this process are described below and summarized in Table 1. In this article, we will focus mainly on development of the 2004 edition, which updated the content of the 1996 edition and added subspecialty goals and objectives. This edition is built on a large database of 334 goals with objectives that can be accessed dynamically on the Web; lists of goals and objectives can be selected for every residency experience and downloaded as customizable documents. The Web site also includes onscreen instructions, curricular tools such as customizable evaluation forms and templates for rotation planning, and six tutorials on how to build competency-based curricula that are downloadable for local adaptation.
Step 1: Centralized, national leadership combined with coordinated, disseminated authorship
All editions of the Guidelines were official projects of the APA Education Committee. The Guidelines project director for the 1996 and 2004 editions (D.K.) was chair of the education committee from 1992 to 1995. For both editions, about 50 educators were divided into writing and review subcommittees. Hence, the Guidelines were “owned” from the start by a large group of leaders in pediatric education, who not only contributed their expertise in the development and refinement process but also were able to assist with dissemination of the final product.
In 2000, substantial funding to produce the 2004 edition enabled formation of a national advisory board to garner the support of pediatric subspecialists and to facilitate an efficient nationwide collaborative process. The core team recruited 10 section editors from the APA Education Committee to manage writing and review of content revisions. Other contributors were members of the APA and/or other organizations, or they were paid consultants on competency-based accreditation, project evaluation, and computer programming.
While the revision process was intensely collaborative, centralization of control was essential. The core team developed templates and instructions for the section editors and reviewed and combined all the documents to ensure consistency in format and language across all sections. This was especially important for the complex tasks of Web site design, database construction, and development of new curricular tools. These jobs were most efficiently conducted by a small, focused group.
Step 2: Clear definition of targeted users and repeated assessment of their needs
Needs assessment surveys were conducted before and during the development of both the 1996 and 2004 Guidelines editions, to ensure that the content served the needs of a broad group of potential users and to enhance end-user acceptance.
Surveys of pediatric program directors conducted in 1993, 1996, and 1999 pointed out discrepancies between residency program performance and current accreditation requirements. These data helped the team define the utility, content, and scope of the resources under development. For example, the 1999 survey on programs’ uses of the 1996 Guidelines identified demand for a more flexible, easy-to-customize document that could be accessed online. Hence, the 2004 edition was designed to be interactive at its point of access, so programs could select needed tools and customize them after downloading. The survey also showed that the previous edition was more useful for generalist than subspecialty rotations, so extensive subspecialty content was added. Finally, because the survey showed that new ACGME requirements were still very challenging to many programs, tutorials and program planning and evaluation tools were added.
The core team also used workshops to gather continuing needs assessment data. To guide development of the 2004 edition, workshops were conducted in a computer classroom, so users could try out the Web site interface, sample objectives in the Guidelines database, and suggest ways to make the resource more useful.
Cycles of needs assessment were essential to make the product as responsive as possible to the needs of educators “in the trenches,” especially because they were working in the context of rapid change and new demands. Workshops served as focus groups and taught the development team more than mailed surveys did about specific user needs and reactions to individual Guidelines components.
Step 3: Incorporation of up-to-date information from the literature and national experts
During the 1996 Guidelines development, the editorial team conducted an extensive review of the literature on curriculum content, drawing on expert opinion about medical content and educational methodologies. For the 2004 edition, the core team and its consultants also reviewed literature on educational change, competencies, evaluation processes, and faculty development. Further content refinement resulted from the extensive expert review process described in Step 5, below.
This step is essential to earn credibility for a curricular resource. The amount of content to include in the final product was an issue that we and our targeted users debated extensively. We decided, as did the internal medicine clerkship curriculum task force,14 that making our resource comprehensive rather than abbreviated would facilitate local adaptation, even though the large volume of information could be overwhelming.
Step 4: Responsive consultation with the national Pediatric Residency Review Committee on the latest accreditation requirements
Throughout the development process for both editions, members of the national Pediatric Residency Review Committee (RRC) informed the APA and other organizations about anticipated changes in accreditation requirements. This knowledge enabled the Guidelines team to address current and upcoming faculty needs, even before revised RRC requirements were published. For example, in developing the 2004 edition, learning objectives from the 1996 edition were revised by modifying the verbs to reflect performance (e.g., “analyze” and “manage” rather than “discuss” and “explain”). Customizable evaluation tools were developed that could be built around a program’s selected list of learning objectives and, thus, be specific and competency-based. Templates for program and rotation planning were also created to help program directors organize their responses to new ACGME requirements.3
Consultations with the RRC were essential to adjust the Guidelines to users’ needs. However, the team learned that keeping some distance from the RRC was important, given the frustrations felt by educators during those years of rapidly changing ACGME requirements. We carefully disclaimed a prescriptive intent for the Guidelines and consistently encouraged local customization of the resources provided.
Step 5: Wide distribution for prepublication review, to obtain broad organizational buy-in and end-user acceptance
The APA Board and the core Guidelines team recognized the value of a national consensus-building effort around development of the Guidelines. The 1996 edition went through three drafts, and at each step broad input was sought from APA members and other groups, including subspecialists and academic leaders. A prepublication draft was distributed in advance to participants at a national workshop, generating valuable practical feedback. A formal external review by all the major pediatric organizations involved in education took place prior to publication. The APA explored the need for formal endorsement of the document from leadership organizations within pediatrics, but these groups deemed formal approval to be unnecessary, because they had been involved in the development process, and because the product was intended to be a flexible tool for local adaptation, not a prescription for curriculum change.
For development of the 2004 edition, a similar national consensus process for content review was formalized through the national advisory board. A new challenge was developing users’ acceptance of and comfort with the web-based platform. Guidelines section editors served as alpha-testers, and we recruited beta-testers at annual workshops and computer laboratory demonstrations. Beta-testing was conducted in cycles throughout two years as new functions were completed. Feedback from reviewers and workshop attendees led to significant revisions of some Web site components and the addition of several new tools.
Prepublication review has been included in the development of many national curricular resources. We found that it was an effective way to test innovations, garner support, and enlist future users. This process is relatively inexpensive when conducted using workshops and e-mail queries facilitated by links to online materials to review. Modern information technology has greatly reduced the time and expense of this activity.
Step 6: Intensive dissemination and faculty development through national workshops
The 1996 and 2004 Guidelines have been distributed free of charge and widely publicized in 14 well-attended national workshops. In 2000 and 2007, pediatric residency programs were invited to showcase their own innovative curriculum development activities that implemented the Guidelines. In 2003 and 2004, live demonstrations using a portable computer laboratory gave participants the opportunity to explore new Web site functions and give instant feedback, and helped the core team recruit beta-testers for more extensive explorations.
Faculty development is one of the cornerstones of educational change. While users like self-directed online tutorials,20 interactive workshops encouraged educators to use the Guidelines creatively and share what they had learned. These low-cost workshops informed potential users about the resource, taught them how to use it, helped to win their acceptance, and gathered their feedback so the tools could be optimized to meet educators’ needs. We believe that the personal contact achieved in workshops was instrumental in making the resource more “friendly” to users.
Step 7: Careful evaluation of utilization and user feedback
During development of the 1996 and 2004 Guidelines, we surveyed users repeatedly to measure their use of the Guidelines and gather information on satisfaction, barriers to use, and suggestions for improvement. The results are summarized in List 1. These evaluations were approved by Dartmouth Medical School’s IRB.
To prepare for the 2004 edition, questionnaires mailed to educators at all 195 ACGME-approved pediatric residency programs (in 1999) yielded 170 responses (program response rate, 87%). Among all programs, 131 (77%) reported that they had used the Guidelines; most users were pediatric generalists. The major limitations to use of the Guidelines were reported to be lack of time, resources, and faculty support. Although few considered the format (14; 8%) or content (2; 1%) to be a limiting factor, written comments indicated that the document was intimidating in volume, and many respondents suggested online publication.
In October 2005, 18 months after publication of the 2004 Guidelines, members of the APA, Association of Pediatric Program Directors (APPD), and Society of Adolescent Medicine were surveyed using a commercial Web survey tool (SurveyMonkey; http://www.SurveyMonkey.com; accessed May 21, 2009). Replies were received from 582 respondents, who represented 171 of 204 (84%) ACGME-approved residency programs. The data showed that 149 of training programs (73%) were aware of the Guidelines and that 106 (62%) had both logged onto the Web site and used the Guidelines; many said they were likely to return at a later time for further use. The majority of respondents did not report significant barriers in using the Web site, but 145 (about 25%) of users commented on long downloaded documents and difficulty manipulating the tables.
In December 2007, we extended this preliminary survey by evaluating online use of the 2004 Guidelines by registered users between July 2005 and December 2007. A total of 1,747 registered Web site users came from 47 states and 33 foreign countries and represented all pediatric residencies approved by the ACGME in 2008. In all, 8,754 files had been downloaded by 188 of a total of 194 residency programs (97%). Our companion study by Beach and colleagues,20 published in this issue of Academic Medicine, provides more detail on how these users implemented the online resources.
Evaluation data collected from users provided critical information to help us improve the Guidelines during development. An online survey tool vastly simplified the process of gathering these data, compared with mailed surveys, but data collected directly from the Web site were far more representative of users. Getting overloaded program directors to respond to surveys has become increasingly difficult over the past decade. We decided that a combined approach to data collection worked best: we harvested utilization data from the Web site, and supplemented these data with evaluation of specific Guidelines tools by those who had used them.20
Model of collaboration
The collaborative development of the Guidelines is consistent with organizational practices first introduced in the business world. Peters and Waterman,21 in In Search of Excellence: LessonsFrom America’s Best-Run Companies, emphasize the importance of staying close to the customer, listening to users, and promoting intense communications within an organization—all principles that were reflected in our collaborative process. They also describe the value of “simultaneous loose-tight properties,” that is, a combination of central control and disseminated freedom of action, which also typified our process. Day22 wrote that the value of a business should be anchored in value offered to the customer, and Kotter’s23 eight steps for the change process emphasize the importance of coalition building and communication. Our goal was to build not only a set of tools and resources but also a community of Guidelines users who initially contributed to the development of document content and Web site design and who later shared ideas about how to implement the Guidelines in their programs. In our view, our most effective community-building technique was conducting national workshops; these simultaneously monitored user concerns, elicited formative feedback, and provided faculty development.
Our collaborative process also drew on methods used by several concurrent national curricular development projects. Development of the national curricular guidelines for family medicine clerkships10 was funded by the Bureau of Health Professions (BHPr) and published in 1991. This project balanced central control, provided by the BHPr, with disseminated review, facilitated by an advisory committee of representatives from national organizations with an interest in family medicine education. We also drew inspiration from the COMSEP curriculum, which was also supported by the BHPr and published in 1995. This project employed an advisory committee, conducted two national surveys, and used an iterative process of review by its future constituency.10–11 A 1995 clerkship curriculum for internal medicine was also developed collaboratively.12 This BHPr-funded development project, like ours, included a collaborative process, an advisory board, and national surveys of clerkship directors to help define curriculum content.13–14 Although these projects, and others published more recently,24–25 collectively included methods which we adopted or adapted, none of them has published a well-articulated model to disseminate their approaches to other groups.
Model of developmental flexibility
The project’s emphasis on flexibility in curriculum design reflects developmental approaches to innovation and the evaluation of innovations that have evolved in industry during the past 20 years. Peters and Waterman,21 and also Collins and Porras,26 advocate methods that avoid restrictive traditions and hierarchical power structures so that organizations can adapt to unexpected environmental challenges and opportunities. The Guidelines were created in a rapidly and radically changing world of education and clinical practice. We designed them to help residency programs deal with evolving changes by (1) developing the document by a collaborative process that engaged the intended users in a dynamic fashion, and (2) making the document flexible and adaptable, rather than prescriptive. For the 2004 edition, technological innovations enabled us to create a highly flexible resource that surpasses many available Web-based educational tools by offering users choices for selecting the content and the format of curriculum documents. For example, educators can build a customized list of learning objectives for a rotation and then insert those objectives into custom-formatted evaluation tools, in order to meet the requirements of competency-based education. We believe that in today’s challenging world of health professions education, developmental and interactive, user-driven approaches to innovation have great value to offer.
Another factor that may have enhanced dissemination and utilization of the Guidelines was our use of a systematic, scholarly approach for development, implementation, and evaluation. Our process was consistent with all six of Glassick’s27 criteria for the evaluation of educational scholarship. The project team
- set clear goals for the project, carefully identifying our purpose and our targeted end-users in advance;
- made adequate preparation by conducting needs assessments and literature reviews and consulting with RRC members and other experts;
- used appropriate methods for team building, collaborative development and review, evaluation, faculty development, and dissemination;
- demonstrated significant results through surveys, focus groups, and online data collection;
- used effective communications, through workshops and publications, to keep targeted users apprised of project status, new online functions, evaluation results, and implementation ideas; and
- engaged in reflective critique to examine our work before, during, and after publication of the Guidelines.
This process led to many enhancements of the Guidelines as we worked to build innovative tools to meet evolving needs—inventing, testing, and refining at each stage of the development process.
Although our usage data are impressive—188 programs (97%) have used the 2004 edition—we recognize that self-reported use of the tools via surveys and Web site usage data cannot measure how programs actually applied the tools they obtained from the Guidelines. Nor can these data tell us how well the Guidelines helped faculty integrate competencies into their programs. To address such key questions, more comprehensive outcome studies will be needed. Residents’ competence at the end of training and after entry into practice will be the best measure of the real effectiveness of our educational resource, the basis of an important study that is beyond the scope and time frame of this project.
The urgent need for curricular resources to meet changing RRC requirements probably drove the extensive use of the 1996 and 2004 Guidelines, but we would argue that our collaborative development process, our customizable design, and our scholarly approach were probably additional important contributors to the widespread use of the product.
A causal connection between collaborative development and widespread use of the Guidelines may be implied by these associations, but it cannot be proven by the data available at this time. However, evaluation data from other national collaborative projects suggest similar associations. In pediatrics, for example, the collaborative COMSEP curriculum for medical students10 was used by 90% of all U.S. medical schools.11 The Web-based CLIPP project,28–29 which recruited more than 100 faculty nationwide to author and peer review 31 teaching cases for pediatric clerkships, is now licensed by more than 75% of U.S. medical schools and used by more than 12,000 students (Leslie H. Fall, MD, associate professor of pediatrics, Dartmouth Medical School, personal communication, May 2008). Many educators recognize the “not invented here” phenomenon, which typically limits the dissemination of educational innovations across institutions. That these three educational projects have all succeeded in overcoming this parochialism may be attributable at least in part to their collaborative development.
The 1996 and 2004 editions of the Guidelines were created with generous support from a large number of pediatric educators from within and outside the APA. We believe that our effort garnered national participation and enjoyed wide acceptance at least in part because it was carefully planned and conducted with extensive input from intended users and, therefore, met their immediate need to move toward competency-based educational models with flexible tools suitable for local adaptation. Our collaborative model, with its emphasis on developmental flexibility and customizable products, attention to faculty development, and adherence to the scholarly criteria of Glassick, may prove useful in other settings and for other disciplines.
The authors of this article acknowledge the generous assistance of individuals representing many pediatric organizations in refining the 1996 and 2004 editions of the Guidelines. These include the American Academy of Pediatrics, APA, APPD, American Board of Pediatrics, Association of Medical School Pediatric Department Chairs, COMSEP, the Federation of Pediatric Organizations, Pediatric RRC, Society of Adolescent Medicine, and Society for Developmental and Behavioral Pediatrics. The authors also wish to thank the hard-working chapter editors of the 1996 Guidelines as well as the many contributors and reviewers. Special thanks are offered to our consultants for the 2004 online edition; members of the Guidelines’ national advisory board, who provided guidance throughout development of this edition, under the able leadership of Kenneth Roberts; and the loyal and indispensable section editors for the 2004 Guidelines.
The projects described in this article were supported by the Academic Pediatric Association (1992–2007); the DHHS, Bureau of Health Professions, Contract 103HR940857 (1994–1995); the American Board of Pediatrics Foundation (1996); the Pfizer Foundation (2000–2001); and the Josiah Macy Jr. Foundation (2002–2004).
1 Education Committee of the Ambulatory Pediatric Association. Educational Guidelines for Training in General/Ambulatory Pediatrics. McLean, Va: Ambulatory Pediatric Association; 1985.
2 Kittredge D, Baldwin CD, Bar-on M, et al, eds. Educational Guidelines for Residency Training in General Pediatrics. McLean, Va: Ambulatory Pediatric Association; 1996.
3 Kittredge D, Baldwin CD, Bar-on ME, Beach TS, Trimm RF, eds. APA Educational Guidelines for Pediatric Residency. MedEd PORTAL. Available at: (http://www.aamc.org/medportalID=1736
). Accessed July 2, 2009.
4 Accreditation Council for Graduate Medical Education. Program requirements for residency education in pediatrics. In: Graduate Medical Education Directory, 1996–1997. Chicago, Ill: Accreditation Council for Graduate Medical Education; 1996;200–206.
5 Accreditation Council for Graduate Medical Education, Outcome Project 2002–2011. Available at: (http://www.acgme.org/Outcome
). Accessed May 21, 2009.
6 Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: From Flexner to competencies. Acad Med. 2002;27:361–367.
7 Sectish TC, Zalneraitis EL, Carraccio C, Behrman RE. The state of pediatrics residency training: A period of transformation of graduate medical education. Pediatrics. 2004;114:832–841.
8 Sahler OJZ, Lysaught JP, Greenberg LW, Siegel BS, Caplan SE, Nelson KG. A survey of undergraduate pediatric education: Progress in the 1980s? Am J Dis Child. 1988;142:519–523.
9 National curricular guidelines for third-year family medicine clerkships. The Society of Teachers of Family Medicine (STFM) Working Committee to Develop Curricular Guidelines for a Third-Year Family Medicine Clerkship. Acad Med. 1991;66:534–539.
10 Ambulatory Pediatric Association; Council on Medical Student Education in Pediatrics. General Pediatric Clerkship Curriculum and Resource Manual. McLean, Va: Ambulatory Pediatric Association; 1995.
11 Olson AL, Woodhead J, Berkow R, Kaufman NM, Marshall SG. A national general pediatric clerkship curriculum: The process of development and implementation. Pediatrics. 2000;106(1 pt 2):216–222.
13 Bass EB, Fortin AH, Morrison G, Wills S, Mumford LM, Goroll AH. National survey of clerkship directors in internal medicine on the competencies that should be addressed in the medicine core clerkship. Am J Med. 1997;102:564–571.
14 Goroll AH, Morrison G, Bass EB, et al. Reforming the core clerkship in internal medicine: The SGIM/CDIM Project. Ann Intern Med. 2001;134:30–37.
16 Kligler B, Gordon A, Stuart M, Sierpina V. Suggested curriculum guidelines on complementary and alternative medicine: Recommendations of the Society of Teachers of Family Medicine Group on Alternative Medicine. Fam Med. 1999;31:30–33.
19 Bland CJ, Starnaman S, Wersal L, Moorhead-Rosenberg L, Zonia S, Henry R. Curricular change in medical schools: How to succeed. Acad Med. 2000;75:575–594.
20 Beach PS, Bar-on M, Baldwin CD, Kittredge D, Trimm RF, Henry R. Evaluation of the use of an interactive, online resource for competency-based curriculum development. Acad Med. 2009;84:1269–1275.
21 Peters TJ, Waterman RH. In Search of Excellence: Lessons From America’s Best-Run Companies. New York, NY: Harper and Row, Publishers, Inc.; 1982.
22 Day GS. Market Driven Strategy: Processes for Creating Value. New York, NY: The Free Press; 1999.
23 Kotter JP. Leading Change. Boston, Mass: Harvard Business School Press; 1996.
24 Carroll RG. Design and evaluation of a national set of learning objectives: The medical physiology learning objectives project. Adv Physiol Educ. 2001;25:2–7.
25 Bowen JL, Clark JM, Houston TK, et al. A national collaboration to disseminate skills for outpatient teaching in internal medicine: Program description and preliminary evaluation. Acad Med. 2006;81:193–202.
26 Collins JC, Porras JL. Try a lot of stuff and keep what works. In: Built to Last. New York, NY: HarperCollins Publishers, Inc.; 1997.
27 Glassick CE. Boyer’s expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000;75:877–880.
28 Fall LH, Berman NB, Smith S, White CB, Woodhead JC, Olson AL. Multi-institutional development and utilization of a computer-assisted learning program for the pediatrics clerkship: The CLIPP Project. Acad Med. 2005;80:847–855.
29 Computer-assisted Learning in Pediatrics Program (CLIPP). Available at: (http://www.clippcases.org
). Accessed May 21, 2009.