Secondary Logo

Journal Logo

Performance of Medical Students in a Nontraditional Rural Clinical Program, 1998–99 through 2003–04

Schauer, Roger W., MD; Schieve, Dean, PhD

doi: 10.1097/01.ACM.0000232409.20271.7a
Innovative Curricula

Purpose To compare knowledge acquisition as measured by test scores for students in nontraditional clinical clerkships to scores for students in traditional urban hospital-based clerkships. Interdisciplinary and continuity-of-care clerkships in rural areas are the focus of the study.

Method All the students’ Medical College Admission Test (MCAT) scores, National Board of Medical Examiners (NBME) subject exam scores, and United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores over a six-year period, 1998–99 to 2003–04, were compared for third-year students in nontraditional and traditional clerkships at the University of North Dakota School of Medicine and Health Sciences. Cohorts were 29 students in our Rural Opportunities in Medical Education (ROME) program and 296 students in traditional third-year clerkships. NBME subject exam scores were those in pediatrics, internal medicine, surgery, and obstetrics and gynecology. The exam used for family medicine is not standardized to national standards, but controlled within the Department of Family Medicine. MCAT and USMLE Step 1 scores were used as a means of controlling for prior academic achievement and ability.

Results There were no significant differences (p ≥ .05) in MCAT scores, Step 1 scores, subject exam scores, or Step 2 scores between the two groups. In contrast, students from ROME scored higher (p ≤ .05) on the internal medicine clinical preceptor assessments than did students from the traditional track.

Conclusions These findings suggest that students in remote, rural, longitudinal, integrated learning environments can attain fund-of-knowledge scores comparable to the scores of students in traditional clerkships, and may, as in this study, receive higher ratings for clinical proficiency.

Dr. Schauer is director for predoctoral education, Department of Family Medicine, and is affiliated with the Office of Medical Education, University of North Dakota School of Medicine and Health Sciences. He is course director for Rural Opportunities in Medical Education (ROME) and course director for the state-wide third-year Family Medicine clerkship.

Dr. Schieve is an evaluation consultant to the office of Medical Education and Health Sciences, and is a principal with the firm DMD Consulting in Grand Forks, North Dakota.

Correspondence should be addressed to Dr. Schauer, Department of Family Medicine, Grand Forks, ND 58203-9037; e-mail: (

Many medical schools are seeking to offer both traditional and nontraditional clinical learning opportunities to their students. Promising nontraditional clinical clerkships include those in ambulatory practice settings and with underserved populations; those that emphasize interdisciplinary training, continuity-of-care training, or remote and diverse learning environments; and combinations of any of these. Schools utilizing remote and diverse learning environments must meet standards of the Liaison Committee for Medical Education (LCME) regarding equivalency or comparability between geographically separated programs.1

Back to Top | Article Outline

Previous Studies of Nontraditional Programs

Previous studies have found either similar or better examination outcomes for students placed in a variety of settings other than traditional academic centers.2–9 Verby2 reported in 1988 that students in the nine-month Rural Physician Associate Program (RPAP) in Minnesota had outcomes similar to those of non-RPAP students for Part I and Part II of the National Board of Medical Examiners (NBME) exam, now called Step 1 and Step 2 of the United States Medical Licensing Examination (USMLE). Ogrinc et al.3 in an extensive literature review of outcomes of longitudinal ambulatory experiences in the United States from 1966 to 2000, found little or no difference in knowledge acquisition between students in longitudinal and block clerkships. Ogrinc also commented on the paucity of reported studies that employ rigorous evaluation methods.

Worley and his Australian colleagues4 reported significantly higher test scores for their students in a 12-month primary care integrated community-based program at remote rural sites than for students who completed their clinical studies at either a secondary care facility (Royal Darwin Hospital) or the tertiary-care university hospital system in Adelaide. White and Thomas5 at the Medical College of Georgia reported that pediatrics clinical exam scores were significantly higher for community practice site (CPS) students than for students based at an academic medical center (AMC), but found no significant difference between the NBME subject exam scores of these two groups. In their discussion these authors point to greater numbers of patient encounters and higher mean clinical grades for the CPS students than for the AMC students. Williams et al.6 at Wayne State University School of Medicine reviewed their surgery clerkship outcomes for both NBME subject exams and objective structured clinical exams (OSCEs), comparing the performances of students in academic settings, community hospitals, or a combination of the two. While there were no statistically significant differences, community-based students received the highest scores in the OSCE and combination program students received the highest scores on the subject exam. No correlation existed between OSCE and subject exam scores.

McKendry et al.7 studied examination outcomes for family medicine residents completing their training in urban or rural postgraduate family medicine training programs in Ontario, Canada. They found no significant differences in the performance of the students in either type of program on the Medical Council of Canada Qualifying Examinations Part II and the College of Family Physicians certification examination. Markert et al. at Wright State University School of Medicine8 found similar outcomes for NBME Part III (USMLE Part 3) scores and supervisor ratings after the first year of residency for students in five different combinations of sites for surgery and medicine clerkships.

Outcome studies regarding programs developed to meet rural needs may be difficult to interpret because of a number of limitations. Study limitations identified by Mennin et al.9 include self-selection of participants in these programs, the effect of participants’ residency experiences, participants’ family issues (such as employment for spouses), perceived quality of education opportunities for participants’ children, as well as economic issues and government policies. Their study compared outcomes of students in the 1980s who had participated in a primary care curriculum (PCC) at the University of New Mexico School of Medicine instead of the conventional track. At the time of the study, the study participants had been in practice up to ten years. While they took note of the limitations, they found that PCC graduates were more likely to be in practice in medically underserved areas, felt better prepared for practice, were likely to spend more time in community activities, and were nearly five times more likely to be involved as preceptors for students. The authors also noted that women tend to predominate in the PCC track.

There has been limited research regarding outcomes for students in nontraditional learning environments. However, the literature currently available does support the notion that students in those nontraditional settings perform as well as or better than their counterparts in traditional clinical education settings. Ogrinc et al.3,p.692 stated that “while support for superior learning outcomes in longitudinal rotations is sparse, the logic of educating students in the context of their future practice setting is unassailable.” This principle is important for those students who may be considering a future practice in a rural setting, but also should benefit any future physician who might provide collaborative patient care with rural-based physicians.

Back to Top | Article Outline

Our School’s Rural Education Program

“Future practice settings” was one context considered when faculty at our state-supported medical school began the discussion regarding nontraditional medical education for our students at the University of North Dakota School of Medicine and Health Sciences (hereafter, “our school”). Rural Opportunities in Medical Education (ROME) was developed as an alternative clinical education option for third-year medical students in concert with a school-wide curriculum renewal process to replace traditional course-centered biomedical science education with patient-centered learning. ROME, an integrated seven-month longitudinal rural-based option for clinical education, was approved for third-year students’ education beginning in 1998.

In preparation for change the task force assigned by the dean reviewed existing successful programs, particularly the Rural Physician Associate Program (RPAP) developed by the University of Minnesota Medical School—Twin Cities in 1971–72.2 RPAP was developed in response to concerns about the shortage of primary care physicians in rural Minnesota, similar to concerns voiced by rural communities in North Dakota. RPAP has demonstrated success in meeting those health care needs in Minnesota. To establish the ROME program, we sought and received grants from the Bureau of Health Professions Academic Administrative Unit for program development and to purchase interactive video equipment for the rural sites to allow ROME students immediate access to campuses and their colleagues.

Back to Top | Article Outline

Curriculum implementation

A steering committee was charged by the dean of our school with developing general curricular objectives for ROME. Those objectives were aggregated clerkship objectives for traditional rotations for third-year students and addressed the competencies required by the Accreditation Council for Graduate Medical Education. All six core third-year clinical clerkship directors, campus deans, and the associate dean for medical education participated in program development. The original clerkships were internal medicine, pediatrics, obstetrics–gynecology, surgery, family medicine, and neuroscience. After the first two years it was determined that course objectives for neuroscience could be better met during campus clerkships. Neuroscience is now available only as a two-month on-campus experience during the third year. The rural continuity experience for the last five years of the report is seven months, one month less than originally planned.

Students in the ROME program are expected to meet all clerkship requirements for surgery and family medicine (eight weeks each) during their seven months in the rural community, and to make significant progress in meeting requirements for the three other core clerkships(four weeks each). After the seven-month rural experience students return to their campus for the remaining five months of their third year, to complete one-month clerkships in internal medicine, pediatrics, and obstetrics–gynecology, and the two months of neuroscience. The ROME Steering Committee determined that peer support and team learning were important elements of the experience; thus, two students are teamed in each rural community participating in ROME.

Back to Top | Article Outline

Community selection

Prerequisites for community participation in the ROME program included the availability of at least three board-certified family physicians and one board-certified surgeon, and the willingness of the hospital system to host and support teams of two students each year. That support includes availability of a medical library and housing for students and their families. Four rural communities around the state, as distant as 435 miles from the main campus, were identified as initial ROME sites. The physicians and hospital systems in those communities had been actively involved with medical student education for over two decades. ROME coordinators in each community are board-certified family physicians, but students work with all physicians and other health care providers in that community. Contact is maintained between the isolated rural students and other campus students via e-mail and interactive video conferencing for lectures, conferences, and group meetings.

Back to Top | Article Outline

Student selection

Students self-select to participate in ROME, but must complete an application and approval process. Early in the autumn of their second year, an information session is provided to inform all students of the benefits and challenges of ROME. Current ROME students and a number of previous ROME students participate in that session via interactive video to discuss their experiences. Personal learning styles and preferences, as well as family concerns such as moving children in and out of schools, limit the number of students who apply for ROME. The application process includes written responses to questions addressing the perceived benefits of the ROME experience for that student, the commitment of the candidate to self-directed learning, potential challenges and barriers and how the student will address those issues, and priority for community assignment and preferred learning partner.

Professed interest in primary care is beneficial, but not mandatory. Any student at our school in good academic standing is eligible to apply for ROME. A subcommittee of the ROME steering committee reviews the applications, sometimes requests interviews with individual applicants, and makes final recommendations to the steering committee. For demographic information about students at our school who did and did not select ROME from 1998–99 through 2003–04, see Table 1.

Table 1

Table 1

Back to Top | Article Outline

The ROME experience

Prior to moving to their ROME community, students attend a two-day orientation session during which all course directors address learning expectations and resources. Lectures during ROME are limited to periodic course lectures delivered via interactive video. Group meetings and peer presentations occur during the seven months, sometimes only with other ROME students, but when appropriate, with the entire class. The ROME director and each course director make at least one visit to each ROME site annually to assess student progress, receive oral case presentations from ROME students, and meet with physicians involved in teaching ROME students. These meetings address student progress and challenges and allow for continuing on-site faculty development.

During their seven months in a rural setting students learn about health care problems encountered in rural primary care delivery, including routine health maintenance, commonly encountered health care problems and chronic disease management, medical and trauma emergencies, and rare and unusual problems. ROME students become involved as first-contact providers and rapidly become part of the health care team in those communities. Continuity-of-care learning occurs as students meet the patient, then follow that patient through outpatient or inpatient care, consultant care or referral, home care, nursing home care, or hospice care. Continuity is especially important in maternal–child care, as students become involved with the mother-to-be during prenatal care, and follow up with the mother, newborn, and family through delivery and infant care. In the continuity process students meet the extended family and become acquainted with the entire health care system, including community resources and barriers. Economic issues and practice management concerns are part of the learning.

Clinical encounters, including problems and/or procedures, are recorded on a personal digital assistant (PDA). Those data, which are regularly downloaded to a central database, are monitored by the ROME director and each course director to ascertain the students’ breadth of experience. Clinical rotations during the additional five-month campus clerkships are designed by each clerkship director, using site-specific information provided by ROME clinical preceptors and the patient encounter data. For example, one ROME site might provide extensive prenatal care and delivery opportunities, but no gynecological surgery experience. The obstetrics–gynecology clerkship director would have those students focus on gynecology experiences when returning to campus.

In the rest of this report, we describe a study we carried out to examine whether ROME students were incorporating appropriate academic learning.

Back to Top | Article Outline


We initiated this study to compare the knowledge acquisition, measured by test scores, of the ROME students and those at our school who are in traditional clerkships based in urban hospitals. Standards of the LCME call for equivalence or comparability between geographically separated programs (Part 1, p.17, ED 39–45).1

To compare performance outcomes, USMLE Step 2 scores and NBME subject exam scores in pediatrics, internal medicine, surgery, and obstetrics– gynecology were collected for all 325 third-year medical students, comprising 296 in traditional clerkships and 29 in the ROME program, for the program years 1998–99 to 2003–04. Only scores from the past three years of the family medicine exam were included in the analyses; until three years ago the testing instrument in family medicine was an internally generated exam. The “ExamKit,” developed by the authors of Essentials of Family Medicine,10 has replaced the previous exam, thus the smaller number for that cohort. MCAT and USMLE Step 1 scores from the groups were compared to control for possible differences in academic achievement and/or ability prior to entering the third-year experience. MCAT scores used in the study were calculated by adding the physical science, biological science, and verbal subtest scores. All data were compared using Student’s t-test with a p ≤ .05 level of significance.

Back to Top | Article Outline


Average MCAT scores were 27.0 for nontraditional students (ROME) and 27.2 for traditional students; average USMLE Step 1 scores for the two groups were similar, 211.2 ± 21.2 for ROME students and 210.7 ± 22.2 for traditional students (p ≥ .05). The average USMLE Step 2 scores for ROME students was 213.7 ± 22.9 compared to 214.7 ± 23.3 for traditional students (see Table 2; p ≥ .05). NBME subject exam scores for internal medicine, surgery, pediatrics, and obstetrics–gynecology were also included in the analysis. While ROME students had higher average scores than traditional students in three of the subject exams, none of the differences was significant (p ≥ .05). Because only the students from the past three years (2001–02 to 2003–04) had taken the ExamKit family medicine exam, the sample size is reduced from 296 to 142 traditional and 29 to 15 ROME students in that comparison. There were no significant differences between the two groups regarding the mean ExamKit scores (p >.05).

Table 2

Table 2

Clinical outcomes are somewhat more difficult to measure than are academic outcomes. Our school’s Department of Internal Medicine, in an unpublished study11 compared internal medicine subject exam scores, preceptor scores of clinical proficiency, and total scores for a subset of ROME students and traditional third-year students who studied on campus. That study looked at outcomes for four classes, from 2001–02 to 2004–05, and involved 18 ROME and 189 traditional campus students. Internal medicine subject exam scores were only slightly lower for ROME students (29.5 ± 2.24) than for traditional students (30.0 ± 2.23) for this campus cohort (p = .383). However, the ROME students scored significantly better in the clinical proficiency component than did their traditional campus colleagues (Table 3); 31.4 ± 1.66 for ROME students compared to 30.4 ± 1.79 for traditional students (p = .034).

Table 3

Table 3

Survey comments by ROME students include: “… independence with clinical decision making and helping to understand your transition into the role of a physician were significant benefits of the ROME program; I certainly do not think the ROME program was a detriment to my shelf [subject] and USMLE scores,” “… independence was a strength that translated well into my intern year.” “The one-on-one interactions with attending and supervising MDs was a strength,” “… develop working relationships with doc/nurses – part of the team,” “The OB portion was quite good, in that we were able to follow patients on a longitudinal basis as well as be present when they delivered. . . .”

To date, 18 (62%) of the 29 ROME graduates selected primary care residencies (family medicine, internal medicine, or combined med–peds) compared to 107 (36%) of our 296 traditional program graduates. A total of 11 (38%) of the ROME graduates selected family medicine compared to 47 (16%) of traditional students; six (21%) of ROME students selected general internal medicine vs. 33 (11%) for traditional students. No ROME graduates selected pediatrics residencies, while 29 (10%) of traditional graduates selected that specialty. Of the ROME graduates who have completed residency, three have returned to North Dakota to establish practices, and one additional ROME graduate has signed a contract pending completion of her Family Medicine residency. All are or will be in communities of less than 25,000.

Back to Top | Article Outline


Our research found that academic outcomes for ROME participants are comparable to outcomes for students in traditional clerkships. These findings are consistent with those from other institutions reporting outcomes of nontraditional programs. Clinical proficiency scores were significantly better for ROME than for traditional students, although our study was limited to one campus and one clerkship cohort. Available data do not provide an explanation for the higher clinical scores; one hypothesis might be that positive learning outcomes occur when students have a sense of team membership and a larger view of the problem(s) brought by the patients because of continuity of patient care opportunities, in addition to the side-by-side continuity experience with practicing clinicians. White and Thomas5 alluded to the potential benefit of greater numbers of patient encounters and higher clinical scores for community practice site students compared to academic medical center students. Feedback from ROME graduates has been extremely positive regarding continuity of care and its benefit on their learning. To date, every ROME graduate has indicated that he or she would repeat the ROME experience, given the same options.

Point-of-service learning in rural areas should foster increased awareness of needs and opportunities as well as awareness of lifestyles in those rural settings.2,9 The need to train physicians who will ultimately establish a practice in a medically underserved community is goal of ROME, and also speaks to the mission of a state-supported institution. While acknowledging that the ROME program is new, the number of participants is small, and self-selection to ROME is likely a contributing factor, we can be encouraged by the emerging pattern of primary care residency selection and practice decisions of graduates from the first two years of the program.

Future research should include an analysis of the clinical encounter data regarding issues such as level of involvement in patient care for ROME students compared to those in the traditional clerkships. Other outcome measures could include clinical proficiency in all clerkships and/or OSCE performance. Collaborative research, on a national or even an international scale, could be readily facilitated because of electronic communications. Finally, graduates from ROME and similar programs might be worthy subjects for research to further define characteristics of individuals who ultimately choose to provide health care for rural and other underserved populations.

Back to Top | Article Outline


Bureau of Health Professions Academic Administrative Unit grants were used for program development and purchase of interactive video capabilities at the ROME sites.

Back to Top | Article Outline


1 Liaison Committee on Medical Education. Curriculum Management of Geographically Separated Programs in LCME Accreditation Standards. In: Functions and Structure of a Medical School (Standard D. 2.) (
2 Verby JE. The Minnesota Rural Physician Associate Program for medical students. J Med Educ. 1988;63:427–37.
3 Ogrinc G, Mutha S, Irby DM. Evidence for longitudinal ambulatory care rotations: a review of the literature. Acad Med. 2002;77:688–93.
4 Worley P, Esterman A, Prideaux D. Cohort study of examination performance of undergraduate medical students learning in community settings. BMJ. 2004;328:207–9.
5 White CB, Thomas AM. Students assigned to community practices for their pediatric clerkships perform as well or better on written examinations as students assigned to academic medical centers. Teach Learn Med. 2004;16:250–54.
6 Williams M, Ambrose M, Carlin AM, Tyburski JG, Steffes CP. Evaluation of academic and community surgery clerkship at a mid-western medical school. J Surg Res. 2004;116:11–13.
7 McKendry R, Busing N, Dauphinee D, Brailovsky C, Boulais A. Does the site of postgraduate family medicine training predict performance on summative examinations? A comparison of urban and remote programs. CMAJ. 2000;163:708–811.
8 Markert RJ, Barnes V, Dunn MM, Goldenberg K, Hennessey JV. Comparing clerkship sites in a community-based medical school by evaluating students’ undergraduate and postgraduate performances. Acad Med. 1993;68:298–300.
9 Mennin SP, Kalishman S, Friedman M, Pathak D, Snyder J. A survey of graduates in practice from the University of New Mexico’s conventional and community-oriented, problem-based tracks. Acad Med. 1996;71: 1078–89.
10 Sloane PD, Slatt LM, Ebell MH, Jacques LB. Essentials of Family Medicine. 4th ed. New York: Lippincott Williams & Wilkins, 2002.
11 Newman WP, Swann BL. Internal medicine clerkship evaluation data, Southeast Campus [unpublished data]. Grand Forks, ND: Department of Internal Medicine, University of North Dakota School of Medicine and Health Sciences.
© 2006 Association of American Medical Colleges