In this report, we describe key components of a faculty development project in informatics and evidence-based medicine carried out at our institution. We discuss how participation by faculty members in the project increased their medical informatics competency. Finally, we report how we applied what we learned to the initial development of a residents' curriculum in informatics and EBM, designed as a follow-up step to the project.
As the American health care system makes the transition to the 21st century, health care providers are increasingly taking on new and expanded roles. Among the most important of these roles is that of information manager. Providing information to patients and their families about up-to-date treatment and management options, using computerized patient records, assessing and improving quality in clinical practice, and using electronic mail are becoming parts of the physician's day-to-day job.
For primary care physicians, who are called on to diagnose and manage a virtually limitless domain of medical problems, these responsibilities are particularly daunting. It has been estimated that the amount of medical knowledge doubles every two years.1 It is clearly impossible for an individual physician to keep up with this explosion of new information. Computers have made storing, sorting, and retrieving information more manageable because they can accomplish it much more quickly and dependably than humans. However, despite the clear efficiencies computers present, many practicing primary care physicians do not use information technologies, hereafter referred to as medical informatics.* Medical informatics is intricately linked to the evolving clinical emphasis on practicing evidence-based medicine (EBM), which relies heavily on access to the most up-to-date information in a quick and efficient manner, allowing application of that information during the patient encounter.
In academic primary care, both faculty members and residents need a foundation in computer skills on which to apply informatics and EBM. Surprisingly, while they have access to information technologies, faculty physicians in primary care residencies often lack the proficiency needed to utilize these tools.2 Ironically, these same faculty members are asked to provide informatics training to their residents. An obvious conclusion can be drawn from these facts: faculty members require their own training before they can be expected to instruct residents.2–4 Despite the clear need for faculty development, the process of designing and implementing such programs has proceeded slowly. Limitations to faculty education have included a lack of time for physicians to develop skills and the lack of quick and easy access to local informatics experts.2,5–9
The need for faculty development in information technology has been confirmed by our own experience. We at the Maine Medical Center Family Practice Residency Program (MMCFPRP) in Portland conducted a pilot research project from 1994 to 1997 designed to assess physicians' and patients' reactions to a shared decision-making computer software program. The study involved residents' using the computer during patient encounters in the ambulatory setting. The project was unsuccessful. This was due in large part to a lack of faculty and preceptor support for the residents who were using the computer. When surveyed, the faculty reported their lack of confidence with computers as a major barrier to modeling and teaching these skills with residents.
Based on the premise that if informatics instruction for residents is to be successful, then faculty members must first be trained, the MMCFPRP carried out a faculty development project in informatics and EBM from 1999 to 2001. This project was funded by a grant from the Health Resources and Services Administration (HRSA). Thirteen physicians and one behavioral science faculty member participated in the project. The rest of this report describes that project and its ramifications.
An important goal of this project was to change the underlying culture of the department of family practice at our institution. Prior to the project, the physicians in the department agreed that the concept of practicing EBM was sound, however, it was not being applied clinically. The challenge was how to get a group of physicians who were oriented to textbooks and specialists as sources of medical information to turn to computers and the Internet for up-to-date answers to clinical questions. In particular, we wanted faculty members to feel comfortable enough that they would go to the computer as a first resource during precepting and do it in front of learners. A multifaceted approach was used to reach this goal.
An informatics team was recruited to direct the project. The team consisted of a physician faculty member, an education specialist with a medical technology and nursing background, and a project assistant with strong computer skills and a background in research. A multidisciplinary task force monitored the faculty development project and provided ideas and feedback. Two workshops, each led by a national expert in the topic, were held to train faculty in key aspects of informatics and EBM. Grant funds were used to purchase a laptop computer for each faculty member who wanted one. Laptops were equipped with secure modems allowing access to the hospital computer network. Each faculty member completed a project, of her or his choice, that focused on the application of informatics to the faculty member's work as a teacher. Informatics team members provided mentoring for faculty members in the use of EBM during clinical precepting. The combination of formal training, additional hardware, and on-site experts available to reinforce new concepts led to enhancement of the technical infrastructure of the department and to increased use of information technology in its day-to-day work.
Basic Computer Training
The first step in the project involved collecting baseline information on faculty members' knowledge, comfort, and skills with informatics and EBM. This information allowed the project leaders to tailor basic computer training to individual needs. Self-assessment revealed that faculty computer competencies ranged from inability to use basic Windows applications to the ability to design a personal digital assistant (PDA) application. This broad range of skill levels necessitated creative approaches to teaching from the informatics team. Instructional strategies included one-on-one and small-group teaching with guided practice, purchase of computerized tutorials, and “curbside” consults. Everyone had access to a desktop computer at work and to a computer at home. According to faculty members, access to technologic resources as well as the flexibility and affability of the informatics team was crucial to the success of this project.
Two half-day faculty workshops offered instruction in the two main themes of the project—medical informatics and EBM. Each workshop featured a family physician presenter with nationally recognized expertise in his area. Workshops were conducted in a computer training room, allowing each participant the opportunity for hands-on experience.
The first workshop focused on medical uses of the Internet and basic aspects of Internet technology such as servers, listservers, uniform resource locators (URLs), cookies, bookmarks, security, and virus protection. Topics related to medical uses of the Internet included discussion of ethical issues and visits to a sample of Web sites with information designed for medical practitioners. Hands-on exercises included searching the Internet, downloading a plug-in, and downloading a file. This workshop served to jump start reluctant faculty members, and to motivate the entire group to increase their skills in the use of these resources.
The second workshop focused on EBM, especially as it relates to teaching residents. The workshop emphasized “just-in-time” information sources, those that can be quickly accessed while seeing patients or precepting residents. Additional topics included critical appraisal of review articles and barriers to incorporating EBM into residents' training and medical practice. Hands-on exercises and speaker—participant interaction were also featured in the second workshop. Faculty feedback indicated that both workshops provided inspiration and confidence to incorporate these skills into teaching and day-to-day clinical work.
The next step in the faculty development process was working with faculty members as they provided clinical precepting to residents. Members of the informatics team spent time in the precepting room with every faculty preceptor at least once a month, assisting them in answering clinical questions posed by residents. Team members helped faculty preceptors identify and access useful Web sites with information specific to topics such as diagnosis, treatment, and prevention. Faculty members' skills in using the computer and the Internet were improved by this process.
A systematic approach was used to speed the process of information retrieval. The informatics team organized into a chart those sites that they had determined were most likely to return useful, evidence-based information. The team also bookmarked recommended Web sites on computers used for precepting and updated faculty members regularly with information about new computerized resources for answering clinical questions. The process of finding useful Web sites and working with faculty during precepting continued for the duration of the project.
Each member of the faculty was required to design and carry out an independent project to reinforce and expand his or her personal skills in computer use and EBM. Faculty members chose topics related to their own interests, generally relating to their work as teachers. They were relieved of clinical precepting duties two half-days per month for four months to give them scheduled work time for thier projects. Members of the informatics team were available during scheduled work times to assist faculty as necessary. In fact, the informatics team was quite pro-active in encouraging faculty members to complete their projects; they made informal visits to offices during work time, asked for formal plans for projects, and got regular updates of faculty members' progress. The natures of the projects were quite varied; examples are included in Table 1. A grand-rounds poster session, described below, allowed the faculty to share their work.
The ultimate goal of this faculty development project was to have a permanent impact on the culture of the residency program. One of the strategies designed to accomplish this was enhancing the residency's “information systems” infrastructure. This was done by purchasing both hardware and software as well as by advocating the use of information technology as an integral part of residency training.
Faculty members were given the option of obtaining a laptop computer with remote access to the hospital network. This was done to encourage them to use the computer regularly in their work and to allow Internet access for those faculty members who did not otherwise have it at home. Additionally, it allowed the novices in the group to gain comfort with computing at their own speed. Recognizing the potential pitfall of encouraging faculty members to blur the lines of work and personal life, we concluded that the benefit of greater access and flexibility to schedule work time would outweigh the risks involved.
In addition to laptop computers, faculty members were offered the opportunity to obtain personal digital assistants (PDAs). A digital camera was purchased for the residency program using grant funds and is being used to document dermatologic findings and improve presentations given during resident teaching sessions. A digital data projector was obtained using departmental funds. This gives faculty members the opportunity to use PowerPoint regularly in their educational presentations. The informatics team provided one-on-one and group tutorial sessions to introduce each piece of the new hardware and software. The ready availability of personalized help encouraged reluctant members of the department to move forward in improving their skills and utilizing the available tools, and allowed for the large variety of starting points, learning styles, and interests of department members.
Maine Medical Center has an intranet behind a firewall, accessible from any networked computer within the hospital system. The Department of Family Practice has become an active user of this resource. Our departmental site includes links to EBM Web sites, local presentations, newsletters, residents' curriculum resources, and physician and conference schedules. Online evaluation of residents is under development at our intranet site.
Informatics Task Force
A task force was convened to oversee the implementation of the faculty development project. Members of the task force represent a variety of constituents of our medical community: the informatics team, the residency program director, other faculty members, resident representatives, a community family physician who teaches in the residency, a faculty member from the internal medicine residency program within our institution, a medical librarian, a representative of the hospital Information Services Department, and the director of the hospital's community patient education centers. Opening communication among the various departments of the hospital and community proved to be valuable to both the project and committee members.
Monthly task force meetings included updates of information technology activities within the institution. The meetings were also used for educational presentations, including Internet security, a description of the services provided by the patient education center, the institutional intranet, and the Cochrane Library.
The task force was also involved in the development of a residents' curriculum in informatics and EBM. The task force will continue to meet quarterly to keep the residency program informed about new opportunities and activities related to information technology.
Results of faculty projects were disseminated in a poster session that was part of the family practice department's grand rounds. Each faculty member who completed an independent project produced a poster representing his or her work. Following grand rounds, the posters were included in a hospital-wide research conference. These sessions allowed faculty members to share with each other as well as with other physicians from the community the nature of their work and demonstrate how they applied medical informatics and EBM.
A description of methods and results of the faculty development project itself was disseminated in several venues. The informatics team produced a poster about the project that was included in the family practice grand-rounds poster session and the institution-wide research symposium. A seminar at a regional meeting of the Society of Teachers of Family Medicine helped disseminate information about the project to faculty from other residencies in the Northeast. The plenary session of the annual meeting of faculty from the four family practice residencies programs in Maine, held in November 2000, featured a presentation of the project by the informatics team. Finally, a poster describing the project was displayed at the 2001 Annual Spring Conference of the Society of Teachers of Family Medicine.
Residents' Curriculum in Informatics and EBM
As a result of the project, informatics and EBM are regularly incorporated into resident teaching. The informatics team introduces these subjects to new family practice residents during orientation and to medical students on month-long rotations with our department. A monthly teaching conference on critical appraisal of the literature is held for all residents. Presenters of weekly teaching conferences on core clinical concepts are expected to incorporate summaries of the best available evidence on their topics. A medical librarian regularly attends morning report for the inpatient service and assists with Medline searching to answer questions that arise.
With faculty members adequately prepared to take on the task of residents' instruction, the informatics team completed the final element of the project: the drafting of a formal curriculum in informatics and EBM. The informatics task force helped us by identifying core competencies for the family practice physician and recommending experiences to attain those competencies. Curriculum development continued through the residency program's stepwise process, resulting in a pilot residents' curriculum in place for academic year 2001–2002. In addition to the educational activities described above, all residents are now required to document their use of EBM to answer clinical questions, find computer-based patient education resources, compare treatment of patients with established clinical guidelines through a chart audit, and use CD ROM and Internet-based resources for self-learning. Residents who don't demonstrate competency in informatics and EBM knowledge and skills are required to participate in a two-week informatics rotation individually designed to address their deficiencies.
WHAT THE PROJECT ACCOMPLISHED
Faculty members were asked to self-assess their knowledge or comfort with 20 different informatics skills before and after the faculty development project. Each skill was rated on a six-point Likert scale, with 1 being “not at all knowledgeable or comfortable” and 6 being “a great deal of knowledge or comfort.” The items included use of software such as word processing and spreadsheets, Internet skills such as searching and downloading, and skills necessary for incorporating EBM into practice, such as knowledge of EBM web sites. Fourteen faculty members took part in project activities; 11 completed both a pre-assessment and a post-assessment. The informatics team distributed paper copies of the form for the pre-assessment. For the post-assessment, 13 of 14 faculty members were able to access and complete the form electronically; attached to an e-mail message.
Table 2 shows the mean pre- and post-project knowledge or comfort level of the family practice faculty members. The differences between the scores for the two assessments were analyzed using the Wilcoxon signed-rank test, a nonparametric test of differences among pairs. The null hypothesis would state that there is no consistent difference between the pre- and post-project assessments for each skill for each faculty member. “Project skills” were the primary foci of this project (e.g., use of PowerPoint, use of the Cochrane Library) and “additional skills” were those not specifically addressed (e.g., shared decision-making tools, use of electronic medical records). The p-values for the differences in levels for both sets of skills are shown in Table 2.
Twenty informatics and EBM skills were assessed. Analysis of the responses indicates statistically significant improvement (p < .05) for each of the ten “project skills.” Responses also indicated such improvement in six of the ten “additional skills.”
Participants reported that their greatest gains in knowledge as a result of this faculty development effort were in downloading information from the Internet, use of presentation hardware, and accessing medical information and EBM sites on the Internet, all areas targeted by the project. The participants reported being knowledgeable and comfortable with generalized Medline searching before the project, but rated themselves low on more specialized resources such as use of the Cochrane Library. Gains in “additional skills” can be attributed to a general increase in computer literacy and self-confidence. Observation by the informatics team and informal feedback from residents indicates that faculty members are also applying these skills to their teaching of residents and their practice of family medicine.
There is agreement in the psychology, education, and medical literature that self-report is a valid method of assessment.10 We believe that our assessment is an accurate reflection of skill levels before and after our faculty development project. We base this on observations by EBM mentors, comments made by residents, and analysis of the results. However, we acknowledge the possibility that some faculty members may have under- or overstated their skill levels. This is at least partly due to the realization, which can occur as one learns a computer application, that there are layers of sophistication in its use that were not initially apparent.
EBM has been a focus in our residency program since the mid-1990s. Faculty members had been trained in basic critical appraisal skills before this project began. This may have led to their higher ratings of comfort with practicing and teaching EBM than with their abilities to access up-to-date information via the computer and the Internet.
Two forms of assessment were used to evaluate this project. One was designed to measure changes in individual learners' knowledge and comfort relative to the use and acceptance of informatics described above. The second strategy was evaluation of the project itself, used to determine faculty participants' perceptions of the value of key components and of the project overall.
All participants indicated that the faculty development project met or exceeded their expectations. The component rated “most significant” was interaction with the informatics team. Other highly rated components were the faculty workshop on EBM, one-on-one help during the individual projects, EBM mentoring during resident precepting, and completion of an individual project. The components noted to be least significant were purchase of laptop computers, installing remote access on the laptops, and one-on-one basic tutoring before the workshops. Participant comments included suggestions for additional components such as more EBM coaching, better collaboration with the hospital Information Services Department, and more training on PDAs.
This project has had a positive impact on the faculty members in our program, increasing their ability to employ information technology in individual and group teaching sessions as well as their ability to practice EBM. Our main goal was realized: the culture within the residency program has been changed to one of utilizing computers and the Internet as principal resources for up-to-date information. The true test of success, however, will be the test of time as we see how residents' education is affected over the long term and whether the culture change in the use of informatics is permanent.