Medical schools constantly revise their curricula to keep up with societal changes, such as shifting demographics and new health care delivery systems, as well as with advances in medicine and education. Nevertheless, medical schools are often reluctant to initiate major curricular changes, particularly changes that emphasize primary care, because such modifications are difficult and expensive and often fail to result in enduring and desired outcomes. Further, medical school administrators may fear that successful curricular changes will come at the cost of lowering students' preparation, faculty's research productivity, and the institution's reputation.
In light of these fears, we wondered whether major curricular change resulting in desired outcomes can occur without significant negative impact on students, faculty, and institutions; whether these changes and outcomes can endure; and what institutional characteristics predict success of new curricular initiatives. This research addressed these questions in the context of a cluster evaluation of the W. K. Kellogg Foundation's initiative, the Community Partnerships and Health Professions Education program (CPHPE). This five-year (1991 to 1996), nationwide initiative was designed to produce more primary care health care providers (e.g., family physicians, general internists, general pediatricians, and primary care nurses) by modifying health professions' curricula to include: (1) multidisciplinary instruction, (2) a shift in training from hospitals to at least 20% in community-based academic health centers, (3) over-sight by a governing board made up of community and institutional members, (4) monitoring and facilitation by the Kellogg Foundation, and (5) changing national and state policies to sustain these changes. Participating institutions included 27 health education schools at seven sites in seven different states (Georgia, Hawaii, Massachusetts, Michigan, Tennessee, Texas, and West Virginia). At a minimum, each CPHPE site included a medical school and a nursing school, but schools of social work, public health, and dentistry, for example, also took part.
The overall evaluation of the CPHPE was conducted by an interdisciplinary team called the cluster evaluation team. As part of the overall evaluation, the team tracked 12 process factors that are correlated with successful and enduring curriculum change (e.g., cooperative climates, policy support, institutional member involvement), and one set of outcome indicators (e.g., the number of new courses developed, quality of students' performance, and students' specialty selections) over the five years of the project. The occurrence of negative outcomes was also assessed in areas such as faculty reward systems, faculty research productivity, and faculty and student attitudes about primary care.
The process factors associated with enduring curriculum change were identified through a review of the literature on implementing and sustaining curricular change and on collaboration.e.g.,1–3 The 12 factors tracked in this study were placed into three categories: institutional design features, institutional process features, and curriculum process features, as outlined in List 1. The list we include is based on a 1991 literature review. For the updated version (2000), see the related article in this issue on page 575. Also in List 1 are the specific curriculum outcome indicators assessed; of these, the ones reported here include the competency levels of students participating in the revised curriculum, the number of new or revised primary courses offered (particularly those with a multidisciplinary emphasis and with a community training component), and the number of students choosing primary care specialties. For medical students, two definitions of primary care were tracked: (1) primary care internal medicine, family practice, primary care pediatrics, and medicine—pediatrics; and (2) those four specialties plus internal medicine and pediatrics.
In addition to collecting information about the curricular change process and outcomes, the evaluation team gathered information about other evaluation questions in areas such as financing, leadership in the projects, multidisciplinary strategies used, and related public policies. Information about all the evaluation questions, including the ones reported here, were gathered in the following ways: pre- and post-surveys administered to all faculty and students at the participating institutions, annual site visits involving a standard interview protocol, a survey on leadership sent to all members of each site's partnership governing board and its major committees, standardized annual reports submitted by project directors, a focus-group study of leadership involving the project directors and a major community leader (who was a member of the partnership board), a clinical site survey, and national medical residency match data. The majority of the data reported here were derived from the pre- and post-surveys, site visits, and annual reports. In year 2, responses to the pre-surveys were received from 1,926 faculty members (396 of whom were directly involved with the curricular change initiative via developing or teaching in new or revised courses) and from 3,338 students (of whom 601 directly participated in the new or revised courses). The response rates to these two surveys were 64% and 48%, respectively. In year 5, responses to the post-surveys were received from to 1,599 faculty (359 involved) and 2,709 students (738 involved), for response rates of 53% and 39%, respectively. (Some sites did not provide sufficient information about the numbers of surveys sent in year 5 to calculate an accurate response rate. In such cases, the number of surveys sent in year 2 was used to estimate a response rate for those sites in year 5.)
The scales, each of which consisted of two to eight items, were constructed from the survey items. Table 1 lists the average ratings for each scale. For example, the scale “overall climate for faculty” lists the average ratings from poor (1) to excellent (4) for four items: the administration at this institution, the quality of life at this institution, the intellectual environment at this institution, and the sense of community at this institution. These scales were constructed conceptually when the survey was developed and finalized via factor analysis using varimax rotation. The reliabilities of the scales ranged from .312 to .990, with most scales having reliabilities above .500. The scales with the fewer items (e.g., two) had the lower reliabilities.
Each site was visited annually by two of the cluster evaluators, one of whom was the primary evaluator for that site and was present at every visit for that site. The site visits involved two to three days of interviews with predetermined participants using a standard list of interview questions. Each year the questions were designed by the cluster team to address all the evaluation areas and to provide more in-depth information about data gathered through less qualitative means, such as surveys or annual reports.
First we present data related to some of the process factors likely to result in enduring curricular change. These data also reveal the impact of the curricular change on important and valued school aspects, such as perceived reputations and climate of cooperation. For each feature, we state how it is associated with successful curricular change and then describe the CPHPE sites with regard to the feature. Second, we present the impacts of the curricular change on three curriculum outcome measures: courses developed or revised, participating students' abilities, and medical students' specialty choices.
Institutional Design Features
Mission, goals, and history of change. Curricular change is more likely to be successful when it matches the mission of the institution and when the institution has had past success with change. Among other criteria, the CPHPE sites were initially chosen because they had missions that supported primary care. At the beginning of year 2, site-visit interviews and a survey of faculty at project sites indicated that the institutions had: (1) transformed themselves quickly in the past to adapt to changing circumstances, (2) emphasized primary care curricula over the previous ten years, (3) existed in communities that influenced the schools' missions, and (4) attempted to match services and training to community needs. In year 5, a second survey of the faculty found that faculty perceived that over the life of the project there had been a significant increase in these institutional characteristics, in particular in the emphasis placed on primary care and the involvement of community and in overall climate for faculty (see Table 1, scales 1 and 4).
In addition to a mission that advocates primary care, these institutions had goals associated with an overall quality of clinical care and a strong academic reputation. When asked in year 2 about these areas, faculty were some-what more positive about their institutions' clinical services than about their academic reputations. There was an overall increase in perceived quality of both the clinical services and academic reputation over the five years of the project. Moreover, this positive change was greatest among those faculty involved in the CPHPE (see Table 1, scales 2 and 3).
Cooperative climate. A culture of collegial and institutional support is associated with successful curricular change. Fortunately, the CPHPE institutions had an initial climate of cooperation, as evidenced by: (1) overall climate for faculty, (2) positive faculty perceptions of administration, and (3) a high level of institutional support for the curriculum project (see Table 1, scales 4–6). At year 5, faculty continued to perceive high institutional support for the project. Also, during site-visit interviews in year 5, respondents reported (with some variation) that the CPHPE had increased both the formal and the informal mechanisms to facilitate collaboration across the major units within institutions. Several respondents also stated that one of the major contributions of the CPHPE program had been an increase in collaborations across institutions and schools.
Political structure and policies. Relevant policy changes are important indicators that institutions are engaging in permanent change. At the end of the five years, the sites reported in their annual reports that approximately 63 policy changes had been made within the participating institutions to sustain the key elements of the CPHPE initiative. The majority of these changes (40) were curricular, but changes were also made in policies related to faculty roles (14), in policies related to multidisciplinary efforts (12), and in admissions and other student policies (7).
Specifically with regard to faculty appointments, the data in Table 2 show that about 30% of the faculty surveyed in year 5 thought there had been a positive change over the preceding three years in support of primary care and community-based teaching, research, and service. With regard to the evaluation and reward systems linked specifically to faculty activities in primary care and community-based teaching, research, and service, of the faculty who knew about such policies, most perceived that no change had occurred. When looking to the next five years, however, of the faculty who ventured an opinion, 27% to 29% thought that there would be positive changes in these systems to reward community-based and primary care activities.
Another indication of the continued support for this initiative was the extent to which involved faculty sensed that they were being fairly rewarded for their efforts. On the year 2 survey, faculty indicated that their rewards and burden—including their teaching loads, salary levels in general, and their own individual salaries—were “fair” on a scale from poor (1) to excellent (4). This perception showed a statistically significant increase by year 5, although the average rating was still closer to “fair” than to “good” (see Table 1, scale 7).
Relationships with external environments. Maintaining effective relations with external constituencies (e.g., community and accrediting bodies) is also important to successful curricular change. The CPHPE projects were very community-oriented, and in all cases the site-visit interviews revealed improved community relations and community participation at all levels from project governance to contact with students and faculty. The positive impact of community involvement was strikingly demonstrated in a separate study of the project leadership that found that the sites that had over 51% of the members of the project governing board from the community were also the ones with the most positive outcomes.4 In regard to relationships with external professional bodies, with one exception, CPHPE project participants reported in site interviews that involvement in the curricular initiative either had not changed the relationships of the institutions with their accreditation bodies or had had positive impacts on recent accreditation proceedings. One medical school reported that there had been a question during the accreditation review about the value of putting medical students in the same classes with nurse practitioners and physician assistants.
Institutional Process Features: Participation and/or Buy-in of Institutional Members
Meaningfully involving institutional members in the many decisions that occur in a major curricular change is essential for success. This is accomplished via a highly participative leadership approach by institutional administrators, and the engagement of faculty. With regard to leadership approach, faculty in year 2 of the CPHPE project perceived their institutions' and departments' leadership approaches to be halfway between autocratic and democratic (see Table 1, scale 5). Surprisingly, in the post-survey, faculty reported that the institutional and departmental leadership had significantly moved to be less democratic. This perception was most strongly held by faculty not in the project. Project leaders' governance approach was also tracked in the leadership study mentioned above, which looked at their use of organizational power, coalition-building behaviors, value-shaping behaviors, and participative governance behaviors. The use of participative leadership behaviors was found to be highly positively correlated with desired outcomes tracked in that study, such as participants' commitment to and satisfaction with the project.4
Faculty buy-in is also suggested by the lack of anxiety expressed by faculty about their involvement or students' involvement. Their concern about this remained low over the five years of the project (see Table 1, scales 8 and 9).
Faculty involvement is also increased when influential faculty participate in the project. There was initial concern that the faculty involved in the project might be less influential or be mostly junior faculty. However, faculty demographic data gathered in the first faculty survey indicated that many long-established faculty who then served or had served on key institutional committees were involved in the initiative.
Curriculum Process Features: Curricular Features and Changes
Perhaps most important to the success of curriculum change is the perception that it is needed and will advantage (and not disadvantage) students and faculty. Data collected in the year 2 survey indicated a perception shared by both faculty and students that there was a critical need to produce more primary care professionals (mean = 3.41 on a four-point scale), although there was a significant difference between the perceptions of faculty who were involved in the CPHPE project and those who were not; faculty not involved were less positive (see Table 1, scale 10). By year 5 of the project, the perceived importance of primary care training was still high, although somewhat decreased from year 2 (mean = 3.20). On average, in year 2, the faculty perceived primary care practitioners as adequately trained, and capable as specialists. They maintained this perception in year 5 (mean = 3.06). This positive attitude about primary care strengthened during the project for those faculty involved in the CPHPE, but decreased slightly for un-involved faculty (see Table 1).
There was an initial concern by some administrators and faculty that being involved would negatively impact research productivity. Table 3 shows that both involved and uninvolved faculty slightly increased their research productivity over the years of the project. Of particular note is the increase in primary care articles by both groups of faculty. It is striking, however, that these increases in research productivity occurred over a period of declining time spent on scholarly activities. In the year 5 survey (as compared with year 2 data), faculty on average reported spending less time on teaching and research, more time in communities, and significantly more time in clinical care (see Table 4). This could have resulted from a shift in faculty roles to include more clinical work or the addition of community faculty.
In the first survey, students expressed slightly more anxiety about the impact of the program on student education than did faculty (see Table 1, scale 15). Involved faculty did not express a great deal of concern that changing the curriculum to be more community-based and multidisciplinary would have a negative impact on students' ability to pass boards or licensing examinations, get into graduate programs, or learn about clinical practice or health care delivery. Most faculty concerns were centered around the ability of students to travel to and from the community sites. Year 2 and year 5 survey data indicate that both faculty and students agreed that students received adequate training in community-based settings (see Table 1, scales 12 and 16).
We now move from the features of successful curricular change to the impacts of the curricular change on three outcome measures: courses developed or revised, participating students' abilities, and medical students' specialty choices.
A total of 199 new or revised courses were developed during the CPHPE initiative. Of these, 138 were offered in the community, 141 involved multidisciplinary faculty, and 80 had multidisciplinary students. Compared with year 2, faculty and students interviewed in year 5 expressed less skepticism about multidisciplinary experiences. About a third of CPHPE faculty responding to the year 5 survey indicated that this project had increased their abilities to work with multidisciplinary teams. However, curricular issues remain, particularly for nursing and medicine, with regard to implementing multidisciplinary experiences. For example, medical students generally have less structure than do nursing students (depending on the level of student), medical schools typically structure curricular time differently than nursing schools (e.g., week rotation versus quarter course), and beginning medical students are usually older and more advanced than are beginning nursing students (although this is reversed when the nursing student is in a graduate program).
A significant component of the new curriculum was moving students' experiences from the hospital/campus setting into the clinic/community setting. During the first four years of the project, a total of 61 community-based academic health centers (CBAHCs) were used to provide student training. Students who participated in the new curriculum indicated in the year 5 survey that approximately 27% of their time was spent in Kellogg activities. Further, the majority of their Kellogg time (68%) was spent in community health centers or in other community locations.
During the year 5 site-visit interviews, the quality of student training was assessed by asking faculty, administrators, and students to compare the skills of CPHPE students with those of students not in this program. Faculty, administrators, and students said resoundingly that the CPHPE curricular changes had led to better training of the students in primary care and team practice. The students involved were also reported to be as well trained as non-CPHPE students in non—primary care areas. Project directors reported no significant difference in test or board scores between these two groups. Similarly, the involved students reported very favorably on their training. In the year 5 survey, a majority of these students indicated that their training had prepared them somewhat (or well) to practice primary care (80%) and to work in multidisciplinary teams (76%).
The ultimate outcome of this project was to increase the number of students who entered primary care practice. The average percentage of medical students selecting primary care residencies (narrow definition) from the project schools increased between 1991 and 1997 from 27.5% to 44.1%, a larger increase than that in medical schools in general during this period (23.5% to 33.0%) (see Table 5). Following the project periods in 1998 and 1999, the percentages of graduates from project schools selecting primary care residencies were 30.0% and 38.6%, while the national averages of graduates during the same years were 32.5% and 31.4% (see Table 5). The specialty match data using the broad definition reveal a different pattern, where the match results of project schools and national average are quite similar in 1991 and 1996 to 1999 (see Table 6). This suggests that the curricular changes impacted decisions regarding primary care specialty selection when narrowly defined, but not when broadly defined. (See Figure 1).
In spite of the fact that major curricular change is always difficult, this study found that these institutions did make major curricular changes. Further, they either possessed or developed the institutional, curricular design, and curriculum features that not only facilitate curricular change, but also increase the likelihood that the change will endure. In addition, these changes were accomplished without negative impact on other important aspects of the health professions schools such as reputation, research productivity, and institutional climate. Most important, the major curricular change in each school achieved its desired outcomes. In this case, those outcomes included more training in the community settings (as opposed to in the hospital or on campus), more training in multidisciplinary settings, increased primary care training, and increased primary care specialty selection, narrowly defined. The increase in the percentage of graduates from project schools selecting narrowly defined primary care rather than broadly defined primary care is not surprising. The new curricula all emphasized primary care that worked closely with the community, in community clinics, and with multidisciplinary teams. The specialties that most closely match this type of practice are the narrowly defined primary care ones.
However, one should cautiously interpret how much these curricular changes influenced specialty choice decisions. During the CPHPE project, there was tremendous external pressure on health professions schools to reform their curricula and to increase primary care output. Also, other curricular change initiatives to increase primary care specialty choice were occurring in other schools, such as those funded by The Robert Wood Johnson Foundation and the United States Department of Health and Human Services, Bureau of Health Professions. Thus, it is difficult to separate the impact of the external environment from that of the initiative on specialty choice. Still, the larger increase in the narrowly defined primary care outputs of the CPHPE schools, compared with the average primary care output of other schools in the nation, suggests that the initiative had an impact over and above the common external forces influencing all schools.
Some limitations of this study should be considered when trying to generalize to other institutions. The survey response rates were less than ideal. Studying the CPHPE, a major initiative that involved several schools and states, was a challenge. We were fortunate to have all the schools agree to a common pre/post survey that they helped develop and administer and to which they added their unique items. In return, the cluster team coordinated the development of the survey, prepared the final version, did the data entry, and provided each site with a file of its own data. Still, there were difficulties in getting the surveys administered and returned, as evidenced by the lower-than-ideal response rates and some schools' not keeping track of the total numbers of surveys sent out in year 2.
Another limitation to the generalizability of the study is that, in addition to external forces such as national associations' and managed care organizations' calling for the production of more primary care providers, these projects each received about $7 million from the W. K. Kellogg Foundation to help accomplish curricular change. Further, most projects leveraged these funds to gather additional similar amounts of money and/or in-kind support from their institutions, communities, and/or states. As a consequence, it is unclear whether successful curricular change, such as reported here, is achievable without significant new resources.
The above issue led the cluster-team evaluators to hypothesize that the strategic success of curricular initiatives such as the CPHPE would be most noticeable when the funding that drove the implementation ended. A follow-up interview study of key leaders (i.e., project directors, medical and nursing school deans, community representatives) conducted two years after the end of the Kellogg funding (during academic year 1998–1999) indicated that overall the curricular changes launched as a part of the CPHPE initiative had been sustained at all seven sites. The percentages of curricular changes still in place varied from school to school within each of the seven projects, with nursing schools being more apt to maintain their levels of involvement than medical schools. At all of the sites, enhancements or expansion to multidisciplinary education similar to those in the original CPHPE initiative have occurred. The struggles of nursing and medicine to create avenues for professional interaction and student learning have created practical mechanisms that have provided enhanced opportunities for other professional schools (i.e., ways to schedule multidisciplinary, community-based courses; ways to schedule faculty and students; and ways to handle community faculty appointments). At all sites there have been increased numbers of students from different disciplines involved in primary care health education. Public health, audiology, human ecology, social work, psychology, education, respiratory therapy, and speech therapy students who were not part of the original program are now involved at various sites. There has also been an increase in the number of multidisciplinary programs, certificates, or majors that have been created. When asked to identify the one major long-term positive outcome of their involvement in the CPHPE initiative, the respondents most often indicated that it had been the collaborative efforts among disciplines, universities, and communities.5
Finally, as mentioned in the beginning of this article, the study reported on here was part of a large cluster evaluation of the CPHPE initiative. The cluster evaluation addressed three broad components: academic institutions, reported here (including curriculum, faculty, and students); community partnership organizations; and public policy change. We would like to alert readers to publications describing other findings and lessons learned from the cluster evaluation.
This article and one other by Harris et al.6 report the evaluation of the academic institution component. The Harris et al. article, which focused on the multidisciplinary education outcomes of the initiative, found that students sang the praises of multidisciplinary education. However, four difficulties in implementing multidisciplinary curricula were also identified: curricula structures of different disciplines make coordination difficult (e.g., lengths of rotations, emphases on classroom instruction as opposed to practical experience); professional identities of students can be inhibited; turf issues among faculty, especially those from different disciplines or specialties, can decrease collaboration; and uses of teaching strategies that differ by discipline (e.g., projects, courses, shadowing role models) can result in conflict.
The community partnership organizations mandated by this initiative were intended to oversee the implementation of the curricula in their communities and promote a cooperative and supportive interaction between academic institutions and communities. A survey of community partnership board members revealed that respondents from projects whose boards had community-majority memberships expressed greater satisfaction with their partnerships as a whole, as well as with their progress and function. When compared with projects with non-community majorities, they also indicated leadership was shared to a greater extent, and they received more frequent and better information. A study of the leadership of the projects revealed that leaders of more successful projects used a consistent set of leadership behaviors that were not as frequently used by leaders of less successful projects.4 In spite of the extra energy required to use a community/university board, a follow-up survey conducted three years after funding ended revealed that all but one of the seven boards were still overseeing the community-based curricula.7
Public policy change was thought to be essential to the success of the CPHPE because without it many of the community-based academic health centers could not be sustained as teaching sites. The evaluation assessed how public policy changes in the seven states and nationwide affected long-term sustainability. In all except one of the seven CPHPE states, the projects achieved substantial awareness of the CPHPE among key policymakers. Further, in all CPHPE states some form of relevant policy change did occur, although it was impossible to tell to what extent the initiative had informed these changes. Four CPHPE states were successful in achieving direct funding from governments or foundations to help sustain the programs. Two CPHPE states were intending to redirect portions of graduate medical and nursing education funding to support community-based education. One CPHPE state incorporated the project into a larger state effort to expand community-based primary care.8–12
In short, one need not fear that curricular change resulting in changed outcomes will have a negative impact on other valued features of a medical school, such as research productivity, school reputation, and student board scores. In fact, this curricular change initiative not only accomplished its desired outcomes without negative impact, it also accomplished an additional outcome that many project participants would argue, in the final analysis, is more important than influencing specialty choice. That is, now, primary care, community-based, multidisciplinary instruction is available for every CPHPE student, whether he or she selects a primary career or not.