After the founding of the University of Arizona College of Medicine in Tucson in 1967, the state's leaders in academic medicine began to envision a four-year medical school for the Phoenix area as well. In 2007, to improve training opportunities and address a statewide physician shortage, the University of Arizona College of Medicine realized this vision by establishing a new four-year medical school campus in Phoenix (COM-P) in partnership with Arizona State University.1,2 At the same time, a new biomedical informatics (BMI) department was created by the Arizona State University on the same campus. Since its inception, the COM-P curriculum, unlike that of the Tucson campus, has included significant instruction in BMI across all four years of the medical school curriculum. In this article, we address the importance of BMI in today's practice environment and make the case for its purposeful and skillful inclusion in the curriculum. We then describe how BMI is integrated into the COM-P curriculum and, finally, discuss our observations and lessons learned after more than four years of experience.
Shortliffe and Cimino3 define BMI as “the scientific field that deals with biomedical information, data, and knowledge—their storage, retrieval, and optimal use for problem solving and decision making.” Increasingly, physicians and policy makers are turning to electronic storage, retrieval, sharing, and use of biomedical data to support biomedical research and to improve the U.S. health care system. The landmark Institute of Medicine report To Err Is Human suggests that electronic health records (EHRs) with decision support and physician order entry are important pillars to improve the safety of medical care.4 Through its Clinical and Translational Science Awards, the National Institutes of Health has made BMI a key component of how academic research institutions will manage translational research on-site.5
Currently, the Office of the National Coordinator for Health Information Technology within the U.S. Department of Health and Human Services is leading many initiatives aimed at improving health information technology. Thanks in large part to the 2009 Health Information Technology for Economic and Clinical Health Act,6 the Office of the National Coordinator for Health Information Technology received significant funding to serve as a resource to the entire U.S. health system, supporting the adoption of health information technology and the promotion of nationwide health information exchange and quality improvement in health care.7
The development of an information infrastructure has tremendous potential to support clinical care and accelerate clinical and translational research throughout any institution or health system. However, this potential will only be realized if clinicians and researchers employing these systems are sufficiently knowledgeable to recognize their benefits and limitations and understand the broader societal risks and benefits of accessible, high-quality health information. Several organizations and individuals have emphasized the growing need for a well-trained workforce of biomedical informaticians to develop, deploy, and support this infrastructure.8
The importance of the instructional component of the effort to incorporate BMI in health care has long been recognized by the informatics community.9 However, it is only relatively recently that biomedical informaticians have had the opportunity to educate medical students before they begin clinical practice.10–15 In many medical schools, the content identified as BMI has been primarily derived from the Association of American Medical Colleges Medical School Objectives Project (MSOP) report on medical informatics and population health16 and focused largely on information literacy topics, including literature searching, biostatistics, and critical appraisal.17–19 Implementation and evaluation of this content are often inconsistent, as noted in an analysis assessing the degree to which the MSOP medical informatics and population health educational objectives have been incorporated into undergraduate medical curricula in the United States.20 In this analysis, the authors reported that
while many of the medical informatics concepts relevant to the clinician, research, and manager roles were being addressed in the curricula, when broken down by those concepts that required health information technology interaction, only a few schools had stated objectives and fewer assessed the competencies.20
Here, we discuss the development, implementation, evolution, and evaluation of a sequenced curriculum comprising 45 hours of required BMI instruction at COM-P. Given the likely emergence of informatics as a core component of medical education, we anticipate that our experience will inform the development of other informatics curricula elsewhere.
Creation of the COM-P Curriculum
The curriculum that emerged from the COM-P curricular design work in 2006 and 2007 consisted of system-oriented blocks and includes several innovative curricular elements (see List 1). Viewed as relevant to future medical practice and supported by the new Arizona State University BMI department, a significant focus on BMI was also incorporated into the curriculum, and a team began work on the COM-P BMI curriculum in August 2005. This team was initially made up of one of the authors (H.S.) and local clinical informaticians pending hiring of full-time Arizona State University BMI department faculty (including T.C. and D.F.). During this initial phase, we accomplished two key tasks. First, the team updated and revised the MSOP BMI educational objectives.16 Information literacy learning objectives were compiled as a separate list and were coordinated and taught by a team of faculty with expertise in BMI, public health, and library sciences. Second, each learning objective was mapped to be taught in one or more of the specific curricular elements in List 1.
The director of each longitudinal theme depicted in List 1 met with each system block director during the design of his or her specific block to integrate and sequence theme content within and across blocks. When working with block directors to design specific blocks, the longitudinal theme directors assigned theme-based learning objectives that most closely corresponded with the material presented in the specific block under consideration in an effort to maximize integration between longitudinal themes and system blocks. For example, as a result of this collaboration, the BMI session on process redesign was taught when students learned pulmonary physiology and was focused on the use of rule-based systems to improve pneumonia and influenza vaccination rates. A similar process was ensured that longitudinal theme content was incorporated into the cases under development for weekly, case-based instruction sessions (e.g., adding a decision support learning objective to a case on clotting mechanisms and the diagnosis and treatment of deep vein thrombosis).
In addition to incorporating BMI content in blocks, we developed a 1-week, 13-hour, first-year BMI block. Initially, this block was focused on an overview of BMI, privacy and security issues, and the use of online and handheld clinical decision support calculators and tools. A 1-week, second-year BMI block involved 19 hours of instruction in decision making, decision analysis, and clinical decision support. This block was taught in lectures and labs in which student groups developed decision trees using TreeAge Pro decision analysis software (TreeAge Software Inc., Williamstown, Massachusetts). Students were tested on all BMI content via United States Medical Licensing Examination (USMLE)-style questions, which were included in the regular and BMI block examinations. In the second-year BMI block, students were also graded on small-group decision tree projects.
Evolution of the COM-P BMI Curriculum
In 2008, the BMI curriculum was reviewed and revised to ensure a comprehensive and properly sequenced presentation of longitudinal theme content and to minimize content repetition and overlap. We developed a list of key topics and compared it with other available listings, such as the Clinical Informatics Core Competencies promulgated by the American Medical Informatics Association.21 After comparing these key topics to the original BMI learning objectives, we made relatively minor additions, deletions, and revisions to the list. The full list, which is available from the authors, includes such topics as knowledge representation; clinical data standards; BMI's role in the generation of evidence, including simulated clinical trials; and workflow and process redesign.
Our experience was that BMI content did not always link naturally to the basic science content, and thus the presentation of BMI content had to be modified to create a closer integration with basic science content. This distorted the desired BMI focus and created unnecessary confusion among students. It also resulted in significant difficulties in properly sequencing the BMI instruction. On the basis of experience and student feedback, in 2009 we redesigned the first-year BMI block to focus on data acquisition, organization, manipulation, and visualization. Lectures provided core content and concepts, and subsequent small-group work focused on the creation and use of data schemas for specific diseases representing gene, patient, and population data elements.
In 2010, Arizona State University withdrew from the formal partnership with COM-P, an unfortunate consequence of widespread economic difficulties in Arizona and resulting contractions of university budgets. After the withdrawal and the resultant challenges of identifying suitable faculty to teach the labs associated with the first-year BMI block, we discontinued this block and migrated the core lecture content to the second major block at the beginning of the first year. This instruction was delivered in two 3-hour sessions in a single week dedicated to BMI content. Short lectures were coupled with in-class labs focusing on the creation, merging, linking, and querying data schemas based on clinical cases presented earlier in the year. A detailed description of this instruction is provided in Chart 1. This change was well received by students.
As a result of this redesign, the flow of BMI instruction is more logical, sequenced, comprehensive, and coordinated. It extends across all curricular elements over the first three years of the medical school curriculum, and the one-week, second-year block and the BMI content assigned to other blocks and curricular elements are integrated at a much higher level. The first year is now centered on data (acquisition, storage, manipulation, and visualization), the second year focuses on decision making, and the third year incorporates both data and decision making into sessions on efficiency, safety, and quality. The instruction moves progressively over these three years from BMI “basic sciences” to the application of BMI principles in clinical reasoning and clinical systems. A one-month BMI elective is offered in the fourth year.
The BMI curriculum is in its fifth year of teaching, and we have collected student evaluation data using standard evaluation processes with oversight by the COM-P Educational Policy Committee. Additionally, this committee carried out formal reviews of the BMI curriculum in 2008 and 2009, focusing on BMI instruction in the first and second years.
Because student evaluations of blocks are optional, the annual response rates have been relatively low. In 2009, there were 16 responses from a class of 47 students (34%) evaluating the first-year BMI block, and 17 responses from a class of 45 students (38%) evaluating the second-year BMI block. These block evaluation data reflected a bimodal distribution of views on the value of BMI within the curriculum, with some students finding relevance in the instruction and others feeling the opposite. It is difficult to discern whether this represents a sampling bias given the low response rates. One student commented that “I don't think I learned anything in this block that I'll be able to apply in my career,” whereas another found the “exposure to important, yet rarely addressed, aspects of clinical medicine” a positive aspect. There were numerous positive responses and comments regarding the decision analysis tree group project in the second-year BMI block (7 of the 11 specific comments were positive). Students commented that they found using the decision analysis software very useful and mastered its use quickly.
In addition to the COM-P formal evaluation processes, we administered a 10-question self-assessment to COM-P students. This self-assessment was originally derived from the first-year BMI block learning objectives and used a four-point Likert scale (1 = strongly agree, 2 = agree, 3 = disagree, 4 = strongly disagree). We administered it before and after the first-year BMI block in 2007 (students from the Class of 2011) and in 2008 (students from the Class of 2012), and we discontinued it after the redesign of the first-year BMI block in 2009. Although these data were originally gathered strictly for educational evaluation purposes, we obtained approval from the University of Arizona human subjects protection program to further analyze and publish these results.
In 2010, we readministered this self-assessment to all 25 Class of 2011 COM-P students (response rate: 25/25; 100%) at the end of their third year as well as to 32 Class of 2011 third-year students who had completed their first two years of training in Tucson and migrated to the Phoenix campus for their third and fourth years (response rate: 16/32; 50%). The students from the Tucson campus did not have any formal BMI instruction during their preclinical years.
The data demonstrate improvements in self-assessment scores between the pre and post first-year block scores for the Class of 2011 (Table 1). Data gathered for the Class of 2012 in 2008 demonstrated similar post block improvements, but these data are not included in Table 1 for the sake of clarity. Additionally, this positive change from the pretest baseline persisted through the third year for the Class of 2011 (as depicted in Table 2) with only minimal falloff in self-assessment scores relative to their first-year posttest scores.
The comparison between the Phoenix and Tucson cohorts depicted in Table 2 reveals statistically significant differences between these two groups in performance on 5 of the 10 questions. These questions reflect content that was specifically presented to Phoenix (but not Tucson) students during their first and second years covering BMI definition, relevance, barriers, mobile decision support, and tools and techniques.
The remaining five questions did not show a statistically significant difference between Phoenix and Tucson students and reflect content that is less specific to the BMI curriculum and generally taught in most medical schools. These included legal and ethical issues, information literacy, and virtual teamwork.
In this section, we discuss key lessons that emerged from analysis of our experience in developing and teaching the curriculum and its evaluation. Although some of these insights emerged from formal evaluation of the students enrolled in the curriculum, in many cases these are drawn from our experience in constructing and teaching the curriculum and are consequently anecdotal in nature. We have included them in this paper because they provide, we feel, essential information for any faculty contemplating or initializing a similar program elsewhere. Certainly, these lessons would have informed the design of our program had we encountered them beforehand.
Computer use versus informatics competency
We have frequently heard the opinion from both faculty and students that medical students already possess all the basic skills necessary to function effectively in an information-rich environment. This is similar to assuming that a student proficient in the use of a word processing program knows how to write an award-winning novel. Although students might be able to quickly learn to navigate an EHR, our experience in the BMI sessions and labs is that they are generally lacking in the ability to serve as capable implementers and informed users of health care information technology.22
Longitudinal student assessment of BMI instruction is challenging
Students are very responsive to learning material when they know they will be tested on it. When theme content including BMI topics was first introduced, there was no mechanism in place to test this material because it was interspersed within system blocks. Additionally, assessing students during a one-week block proved to be challenging. Thus, students perceived BMI content to be outside the normal scope of knowledge required of medical students, and they were less motivated to learn the material.
In response, we designed and implemented relevant, informatics-specific student evaluation measures, including both project-related assessments in the second-year BMI block and standardized questions following USMLE templates, which were incorporated into standard medical knowledge examinations administered in all blocks.
To create additional incentives for both medical students and medical school faculty to embrace the broader inclusion of BMI content material within medical school curricula, it will be very advantageous to incorporate BMI content material on Step 1 and Step 2 USMLE examinations.
Clinically trained BMI faculty are crucial for content creation and teaching
Identifying BMI faculty to develop appropriate instructional materials and teach these sessions is a significant challenge. To effectively develop and teach BMI content in a medical school curriculum requires more of faculty than a generalized expertise in BMI. Faculty trained in both medicine and in BMI can achieve a higher level of relevance and integration of BMI topics into the medical school curriculum. Although nonphysician BMI faculty trained in computer science were capable of providing topic-specific lectures in BMI, these faculty needed to work closely with faculty who have both BMI and clinical expertise to develop teaching sessions and ensure the appropriate level and relevancy for medical students at different levels of training.
Although some of the content covered in the medical school curriculum was drawn from faculty members' experience in teaching BMI graduate students, it proved difficult to use these topics and lectures directly. In some cases, students lacked the necessary prerequisite information, which had to be incorporated into the curriculum. In other cases, the examples and exercises used in existing BMI curricula were not geared to the medical student audience or educational level. Arizona State University BMI graduate students, for whom existing BMI curricula had been developed, often had more technical expertise than the medical students, so exercises in complex knowledge representation or computationally intensive approaches to data categorization needed to be adapted when used to teach medical students.
There are very few existing comparable BMI curricula to learn from because this type of instruction is not yet common in medical schools. Successful evolution of the content and appropriate sequencing required continuous revision over the span of several years. We expect future evolution of the curriculum to focus on a number of specific activities, which we describe in this section.
Comprehensive longitudinal evaluation
The evaluation methods we currently use rely primarily on nonvalidated pre/post student assessments and standard student evaluations of faculty and sessions. We recognize the need to now shift our attention from curriculum formulation and implementation to student, program, and faculty evaluation with particular emphasis on more robust evaluation methods that would allow us to look at both the short- and long-term effects of the BMI curriculum. Evaluating key BMI competencies by incorporating them into existing objective structured clinical examinations would provide valuable student assessment information to be used both individually and in the aggregate. In our setting, the third- and fourth-year students from the Tucson campus could function as a “built-in” comparison group because the Tucson preclinical curriculum is fairly similar to that of the COM-P students, albeit with minimal BMI instruction.
Impact of the pending subspecialty of clinical informatics
Suggested core content21 and program requirements23 were recently published for a proposed subspecialty of clinical informatics. This new subspecialty has now been approved by the American Board of Medical Specialties (ABMS) and will be certified by an examination administered by the American Board of Preventive Medicine and available to physicians who have primary specialty certification through the ABMS.24,25
The emergence of fellowships leading to this certification will offer a unique opportunity to engage fellows and fellowship faculty in the BMI instruction of medical students. This may also increase the motivation of students to see BMI as a relevant and exciting career path.
Access to training EHR lab
The availability of an educational EHR would provide a powerful tool to familiarize preclinical students with the use of EHR systems.26 In particular, an EHR used as a platform for delivering clinical information during a case-based instruction session would provide students the opportunity to interact with the simulated patient's record in a commercial EHR system adapted for educational use. In so doing, students would gain familiarity with electronic clinical documentation, data viewing, and ordering procedures. Use of the EHR by students would also allow tracking of what students “order” during the unfolding of a simulated case, thus allowing more precise evaluation of individual and aggregate student performance. Such a system could also accommodate interprofessional education by allowing health science students from a variety of disciplines to work in small interprofessional groups to care for virtual groups of patients through live seminars, online instruction, and team-based use of the EHR.
We postulate that early development of EHR fluency coupled with instruction in BMI will expand students' abilities to leverage the use of data and EHRs in the service of improving the quality and safety of care they provide to patients throughout their careers. Properly designed and used, the EHR will become as important in the 21st century as the stethoscope.
Preparing Medical Students for the Near Future
The practice of medicine is becoming increasingly dependent on information technology to support safe, efficient, high-quality health care, and on informatics to support the creation, validation, and dissemination of knowledge. Consequently, training medical students to understand and use informatics and its tools and to recognize the strengths and limitations of these tools and approaches is important if they are to be better equipped to address the information management challenges they will face as physicians. The curriculum we have described represents an effort at addressing the need for such training. We trust that our successes, challenges, and lessons learned will inform the development of similar programs at other institutions.
The authors wish to thank Dr. Robert Greenes for his invaluable assistance in editing the manuscript.
Approval for data review and analysis was obtained from the University of Arizona institutional review board / human subjects protection program.
This article was prepared while Dr. Fridsma was employed at Arizona State University. The opinions expressed in this article are the author's own and do not reflect the view of the Office of the National Coordinator for Health Information Technology, the Department of Health and Human Services, or the U.S. government.
1 Joiner KA, Schloss EP, Malan TP, Flynn SD, Chadwick JA. Phoenix rises, with Tucson's help: Establishing the first four-year allopathic program in the nation's fifth largest city. Acad Med. 2007;82:1126–1138.
2 Greenes RA, Panchanathan S, Patel V, Silverman H, Shortliffe EH. Biomedical informatics in the desert—A new and unique program at Arizona State University. Yearb Med Inform. 2008:150–156.
3 Shortliffe EH, Cimino JJ. Biomedical Informatics Computer Applications in Health Care and Biomedicine. New York, NY: Springer Science Business Media, LLC; 2006.
4 Kohn LT, Corrigan JM, Donaldson M, eds; Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. Washington, DC: Institute of Medicine; 2000.
5 National Institutes of Health Clinical and Translational Science Awards. About CTSAs. http://www.ctsaweb.org/index.cfm?fuseaction=home.aboutHome
. Accessed September 27, 2011.
6 U.S. Department of Health and Human Services. Fiscal year 2010 budget in brief. http://dhhs.gov/asfr/ob/docbudget/2010budgetinbriefw.html
. Accessed September 27, 2011.
7 U.S. Department of Health and Human Services. The Office of the National Coordinator for Health Information Technology (ONC). http://healthit.hhs.gov/portal/server.pt?open=512&objID=1200&mode=2
. Accessed September 27, 2011.
8 Hersh W, Williamson J. Educating 10,000 informaticians by 2010: The AMIA 10x10 program. Int J Med Inform. 2007;76:377–382.
9 Shortliffe EH. Biomedical informatics in the education of physicians. JAMA. 2010;304:1227–1228.
10 Fleiszer DM, Posel NH. Development of an undergraduate medical curriculum: The McGill experience. Acad Med. 2003;78:265–269.
11 Keenan CR, Nguyen HH, Srinivasan M. Electronic medical records and their impact on resident and medical student education. Acad Psychiatry. 2006;30:522–527.
12 Moehr JR, Grant A. Medical informatics and medical education in Canada in the 21st century. Clin Invest Med. 2000;23:275–280.
13 Morrow JB, Dobbie AE, Jenkins C, Long R, Mihalic A, Wagner J. First-year medical students can demonstrate EHR-specific communication skills: A control-group study. Fam Med. 2009;41:28–33.
14 Stead WW, Searle JR, Fessler HE, Smith JW, Shortliffe EH. Biomedical informatics: Changing what physicians need to know and how they learn. Acad Med. 2010;86:429–434.
15 Zimmerman JL. Defining biomedical informatics competency: The foundations of a profession. Adv Dent Res. 2003;17:25–28.
16 Association of American Medical Colleges Medical School Objectives Project. Report II: Contemporary Issues in Medicine: Medical Informatics and Population Health. Washington, DC: Association of American Medical Colleges; 1998.
17 Gjerde CL, Pipas CF, Russell M. Teaching of medical informatics in UME-21 medical schools: Best practices and useful resources. Fam Med. 2004;36(suppl):S68–S73.
18 Scott CS, Schaad DC, Mandel LS, Brock DM, Kim S. Information and informatics literacy: Skills, timing, and estimates of competence. Teach Learn Med. 2000;12:85–90.
19 Seago BL, Gyore R, Williams S. Assessment of the integration of AAMC medical informatics objectives into the medical school curriculum. Med Ref Serv Q. 2007;26:1–14.
20 McGowan JJ, Passiment M, Hoffman HM. Educating medical students as competent users of health information technologies: The MSOP data. Stud Health Technol Inform. 2007;129(pt 2):1414–1418.
21 Gardner RM, Overhage JM, Steen EB, et al. Core content for the subspecialty of clinical informatics. J Am Med Inform Assoc. 2009;16:153–157.
22 Centre for Information Behaviour and the Evaluation of Research. Information Behaviour of the Researcher of the Future. http://www.jisc.ac.uk/media/documents/programmes/reppres/gg_final_keynote_11012008.pdf
. Accessed September 27, 2011.
23 Safran C, Shabot MM, Munger BS, et al. Program requirements for fellowship education in the subspecialty of clinical informatics. J Am Med Inform Assoc. 2009;16:158–166.
24 American Board of Medical Specialties. ABMS “In the News.” January/February/March 2010. http://www.abms.org/News_and_Events/MediaMentions/ABMS_JanFebMarch_InTheNews_Web.pdf
. Accessed September 27, 2011.
25 American Medical Informatics Association. The clinical informatics subspecialty. www.amia.org/clinical-informatics-medical-subspecialty
. Accessed September 27, 2011.
26 Otto A, Kushniruk A. Incorporation of medical informatics and information technology as core components of undergraduate medical education—Time for change! Stud Health Technol Inform. 2009;143:62–67.