Secondary Logo

Journal Logo

Graduate Medical Education

Using Direct Observation, Formal Evaluation, and an Interactive Curriculum to Improve the Sign-Out Practices of Internal Medicine Interns

Gakhar, Bhavna MD; Spencer, Abby L. MD, MS

Author Information
doi: 10.1097/ACM.0b013e3181da8370


The safe transfer (handoff) of the responsibility for a patient's care from one physician to another requires that health care facilities have rigorous sign-out systems and that physicians develop effective communication skills. Incomplete exchange of information about patients during transitions in their care is a serious problem1–3 that can result in adverse outcomes in both outpatient4–8 and inpatient9–13 settings. For this reason, the education and evaluation of interns and residents with regard to their sign-out behavior have become priorities for internal medicine residency training programs.

Concerns About Patient Safety

Over the past several years, multiple groups within and outside the medical field have become increasingly aware of problems related to transitions of care, and they have demanded that health care systems measure and guarantee patient safety.14 In response, the Joint Commission made the standardization of handoff communications a key focus of the 2009 National Patient Safety Goals.15 In addition, the Accreditation Council for Graduate Medical Education (ACGME) called for improvements in the exchange of information among health care providers.16 Moreover, the ACGME initiated the Outcome Project to ensure that residency educators fulfill their obligations to the public by teaching specific competencies that would produce measurable outcomes and improvements in the residents and their residency programs.14 Clearly, the transfer of individual patient care is a process that is critical to the delivery of safe and satisfactory care. Moreover, it is a process that lends itself to competency-based education and evaluation that are designed to produce reportable programmatic and individual outcomes. At roughly the same time as it initiated the Outcome Project, the ACGME adopted and implemented duty hours restrictions. Although the restrictions were meant to serve as a parallel safety initiative, there were concerns that they would have a negative impact on safety by disrupting the continuity of care and increasing the frequency of handoffs. In fact, soon after the duty hours restrictions were implemented, hospital and residency program staff began to notice some negative effects at our institution, Allegheny General Hospital, a large teaching hospital located in Pittsburgh, Pennsylvania.

In our institution, the chief residents traditionally introduced interns to the rules for sign-out during their orientation. Then the senior residents informally taught the interns how to complete preformatted sign-out sheets and how to transfer care to the covering intern at the conclusion of each shift. The interns wrote key information on the sign-out sheet, filling in sections under headings for the patient's identification data, problem list, medication list, allergies, and code status and the intern's task list. During sign-out rounds, the interns discussed each patient with the covering intern and physically handed that intern the sign-out sheet. At the beginning of the next shift, the covering intern updated the sign-out sheet and performed a spoken sign-out with the primary intern. However, as the duty hours restrictions took effect and sign-outs became more frequent, the interns began to take shortcuts. In some cases, rather than preparing a new sign-out sheet each day, interns amended the information on the previous day's sheet by covering it over with sticky notes and correction fluid and making multicolored notes in the margins, thereby making the form confusing and illegible. In other cases, they failed to add new information. Moreover, they began shortening or skipping the spoken sign-out and simply handing their sign-out sheets to the covering intern.

We considered several potential reasons why the interns were not meeting our expectations. First, interns may not understand the importance of accurate and up-to-date sign-out sheets. Second, they may not recognize the value of the spoken communication that accompanies handoffs. Third, they may be unfamiliar with the literature on the dangers of inadequate sign-out. Fourth, the program leaders may have failed to make their expectations for proper sign-out clear. Fifth, a standardized system to facilitate efficient preparation and delivery of sign-out sheets was not in place.

As shown by a national survey of internal medicine residency program administrators,17 problems with sign-out were not limited to our institution. This survey showed that very few of the programs had comprehensive handoff systems in place or provided residents with formal training concerning sign-out procedures. Indeed, Allegheny's internal medicine residency program was among the programs without any formal training for residents in the knowledge, attitudes, and skills necessary to facilitate safe patient handoffs. Clearly, Allegheny needed to develop and institute a process to standardize patient transfers, educate residents about the principles of safe patient transfers, and evaluate individual and program performance. We believed that such a process would readily allow for the implementation of competency-based education and evaluation principles to produce outcomes that could be expressed in the language of the competencies.

Development of a Program to Improve Sign-Out Practices

Although improving the quality and safety of patient handoffs and promoting residents' professional development are institutional goals, our program's approach to solving the problem of poor sign-out practices was more of what the ACGME calls a “bottom-up” (resident- and program-initiated) approach than a “top-down” (institution-initiated) approach.18 This project gave us the opportunity to integrate our goals of improving safety with the ACGME goals of teaching residents practice-based learning and improvement (PBLI) and systems-based practices (SBP).

In this report, we describe the methods we used in our effort to improve and standardize our system of patient handoffs, with the ultimate goal of better patient care, and we discuss the results of the project. We developed a formal system for evaluating the interns' current sign-out practices and beliefs and for gaining a clearer definition of the problem. We then used the results to design a structured sign-out curriculum and new sign-out system. After implementation of the new curriculum, we reevaluated the sign-out practices of our interns and also evaluated the sustainability of the intervention. The institutional review board of Allegheny General Hospital judged this study to be exempt.

Formal evaluation of the current sign-out system

First, to determine key components of an ideal sign-out, we reviewed the literature on improving patient handoffs. We then generated questions for a survey of our interns' current sign-out practices. The questions used a five-point Likert scale to assess the interns' agreement or disagreement with statements about their formal training, satisfaction, and comfort with these practices. To ensure the face validity of the survey, we asked the program director, senior residents, and others to review the questions and help us revise them. Our program administrators distributed the survey to all of the interns at conferences and asked them to complete it anonymously.

Second, we developed a method to evaluate the current spoken sign-out process via direct observation and completion of a rating sheet. For purposes of evaluation, we used a seven-item checklist of important components of a proper spoken sign-out. The checklist was based on the Yale University “SIGN-OUT” mnemonic: sick or “do not resuscitate” status (S), identification data (I), general hospital course (G), new events of the day (N), overall health (O), upcoming possibilities (U), and tasks to do (T).19 Our associate program director trained two observers (a third-year resident and a second-year resident) to carry out the evaluation by reviewing the checklist with them and conducting practice sessions in which the observers would watch and rate our associate program director as she “signed out” multiple patients to them. In the practice sessions, the observers achieved 100% agreement, which indicated interrater reliability of the method and the observers' readiness to begin the evaluation. During the evaluation sessions, the observers watched 14 randomly chosen interns on a ward rotation as they signed out four to eight patients each, for a total of 100 observations. For each patient presented, the observers recorded whether the intern did or did not complete each item on the list. They then determined the percentage of times that each item was covered and the percentage of times that all seven items were jointly covered during spoken sign-out rounds.

Third, to evaluate the completeness of the current written sign-out process, we collected written sign-out sheets from the same 14 interns and checked whether the sheets were legible and whether they included the following seven components: the attending physician's name; the patient's identification data, code status, problem list, medication list, and allergy list; and the intern's task list. We scored each sheet from 1 to 8, according to the number of items the intern included and whether the information was legible. After scoring the sheets for all patients signed out by an intern, we calculated the intern's average score. We also tallied the number of complete written sign-outs (i.e., legible sign-out sheets that included all items).

Fourth, to assess the accuracy of the current written sign-out process, we randomly selected the sign-out sheets for 28 patients (2 patients each for 14 interns) and reviewed the corresponding patients' charts within 30 minutes after collecting the sheets. We specifically verified the accuracy of the identification data, code status, medication list, and allergy list. Because of time limitations and obvious logistical problems, we did not attempt to verify other items, such as the problem list, task list, and legibility.

Fifth, we assessed the results of the pretraining survey (Table 1). At the time of the survey, we had 25 interns (21 internal medicine interns plus 4 emergency medicine interns rotating in medicine). All 25 (100%) interns responded. Of that group, only 9 (36%) reported that they had ever been taught how to do a proper sign-out, and even fewer reported that a resident or attending physician had ever given them feedback. It is interesting that, whereas 22 (88%) reported that they routinely give a complete and accurate sign-out to their peers, only 12 (48%) reported that they routinely receive a complete and accurate sign-out. Most of the respondents felt that the sign-outs they receive usually consist only of a list of tasks to complete, and they reported that they are told what to do with the results of laboratory studies only about half the time. Not surprisingly, only 11 respondents (44%) believed that the sign-outs they receive help reduce unexpected overnight events.

Table 1
Table 1:
Results of a Survey of Interns Before Training in the Correct Conduct of Spoken or Written Sign-Outs, Allegheny General Hospital, 2007–2008

Sixth, we assessed the results of the skills evaluation (Table 2). During spoken sign-outs, the residents frequently failed to present important components of the patient's history, such as code status, identification data, new events of the day, overall health status, and potential overnight occurrences. Of 100 written sign-out sheets, only 16 (16%) included all eight components that were requested, and the average number of components included was 5.8. Of the 100 sign-out sheets, only 4 had an accurate medication list, 64 had accurate identification data, 82 had accurate code status, and 96 had an accurate allergy list.

Table 2
Table 2:
Comparison of Results of Pretraining and Posttraining Assessments of Sign-Out Skills Among Interns at Allegheny General Hospital, 2007–2008

Curriculum design and implementation

Our results indicated that we needed interventions to address individual-level and system-level problems. Specifically, we needed to provide interns and residents with more training and feedback about proper sign-out, help them gain a better insight into the importance of proper sign-out, and help them improve their spoken and written sign-out skills. We also needed to develop a user-friendly, efficient, electronic sign-out system to address issues of legibility and time. Therefore, we designed a formal sign-out curriculum with multiple components, including a didactic component for all residents and an interactive component for interns, and we created a Web-based program for completing the sign-out sheets. The Web-based sign-out sheets retained the same template of formatted headings under which the interns now type (rather than write) the patient's identification data, problem list, medication list, allergies, and code status and the intern's task list. The sign-out sheets are accessible from any hospital computer, are updated daily, and are printed out and handed to the covering intern during sign-out rounds.

We initially presented our curriculum to residents during a noontime conference session. The curriculum began with a 30-minute didactic lecture and PowerPoint presentation reviewing the literature on the dangers of improper sign-out and outlining the important components of proper sign-out. It emphasized patient safety and the minimization of errors. To improve spoken sign-out skills, we reviewed the SIGN-OUT mnemonic19 and provided specific examples of ideal and poor sign-outs. The examples included cases involving common clinical problems and cases in which patients had complex and multiple medical problems that could increase the likelihood of errors and difficulties in the sign-out process. To improve written sign-out skills, we reviewed the eight-item checklist discussed above and emphasized the importance of completeness and accuracy. In addition, we presented the results of our pretraining assessment and discussed the myriad errors that could have occurred as a result of the inaccuracies and problems that the assessment revealed. We also described how to access and use the new sign-out template on the Web-based system, and we provided examples of complete and incomplete (acceptable and unacceptable) electronic sign-out sheets.

The didactic component was followed by an interactive, 30-minute, small-group practice session led by faculty and chief residents. In the practice session, we asked each intern to use the new format to orally sign out a series of patients. Each intern then received feedback from peers and session facilitators. We emphasized the skills needed both to deliver and to receive a proper sign-out. For example, if one intern handed off the task of checking a partial thromboplastin time for a patient receiving a heparin drip, we prompted the receiving intern to ask why the patient was receiving heparin and to specify the patient's targeted partial thromboplastin time. We emphasized that the sign-out step is an active practice that involves skills in handing off and receiving information. We also gave each intern a printed version of the SIGN-OUT mnemonic19 to keep in the pocket of his or her white coat.

Of the 21 internal medicine interns, 15 (71%) participated in the curriculum; the remainder either were on vacation or had a day off. We also presented the curriculum later at a housestaff meeting. To account for residents' days off and vacation days, we now present the training session at intern orientation as well as at “rising-resident orientation,” which is a teaching and leadership workshop for incoming senior residents.


Posttraining assessment

The posttraining assessment included two components that were designed to assess the beliefs and skills of interns who had participated in the curriculum. The first component was a 10-item survey that was distributed immediately after implementation of the new curriculum. The survey included questions about whether the interns were satisfied with the course and whether they thought it would help them give and receive better sign-outs and would better prepare them to take care of overnight issues when they were cross-covering patients for their colleagues. This survey also used a five-point Likert scale to record responses, and participants completed it anonymously. Of the 15 interns who participated in the curriculum, 12 (80%) responded to the posttraining survey (Table 3). Of that group, 11 (92%) felt that the curriculum would change how they give spoken sign-outs and how they prepare sign-out sheets, and 11 (92%) said they would recommend that incoming interns go through the curriculum.

Table 3
Table 3:
Results of the Posttraining Survey of Sign-Out Procedures Among Interns at the Allegheny General Hospital, 2007–2008

The second component consisted of an objective assessment of spoken and written sign-out skills, took place about eight weeks after the curriculum was presented, and followed the same methods that were used in the previously described pretraining assessment. In this case, to assess spoken sign-out skills, we watched as 12 randomly chosen interns on ward rotations signed out four to six patients each, for a total of 61 observations. To assess “written” sign-out skills on the new electronic sign-out sheets, we used the same eight-item pretraining checklist to evaluate the completeness of these sign-out sheets for 74 patients. Finally, to evaluate accuracy, we randomly selected the sign-out sheets for 28 patients (2–3 patients each for 12 interns) and reviewed the corresponding patients' charts within 30 minutes after collecting the sheets. We again specifically verified the accuracy of the identification data, code status, medication list, and allergy list.

Comparison of pretraining and posttraining data

We used chi-square tests to compare the results of pretraining and posttraining assessments (Table 2). We considered a P value of < .05 to be significant.

After the interns participated in the curriculum, both their spoken and written sign-out skills improved. In spoken sign-outs, they more frequently reported each of the seven items in the SIGN-OUT mnemonic (P < .001 for six of the items and P = .02 for the remaining item). The percentage of complete written sign-out sheets rose from 16% to 77% (P < .001), and the score for completeness on an eight-point scale rose from 5.8 to 7.6 (P < .001). With regard to the accuracy of written sign-outs, the interns showed significant improvement from pretraining to posttraining scores in three of the four items evaluated: identification data (64% versus 89%), code status (82% versus 100%), and medication list (4% versus 79%) (P < .001 for each item).


One of the principal objectives of the 2009 National Patient Safety Goals15 and the ACGME Outcome Project14,20 is to improve the quality and safety of patient handoffs. Parallel objectives of the ACGME include competency-based education and evaluation, specific attention to teaching quality improvement, and the implementation of standardization principles that will prepare residents for lifelong practice. We believe that the activities reported here are helping us meet these objectives.

The pretraining survey of our interns indicated that most of them had never been formally trained in performing sign-outs. Although the survey results also indicated that they were confident about their sign-out skills, our formal evaluation of their skills identified serious deficiencies related to the completeness and accuracy of their written and spoken sign-out data. These results spurred us to design and implement a targeted curriculum that encompassed several of the ACGME core competencies, including those related to patient care, interpersonal and communication skills, professionalism, PBLI, SBP, and medical knowledge. Because the 2009 National Patient Safety Goals for handoffs emphasize the importance of “interactive communication that allows the opportunity for questioning between the giver and receiver of patient information,”15 our curriculum included experiential training sessions to teach interns how to sign out patients and receive patient handoffs and to allow the interns to practice these skills in a safe environment with real-time feedback.

In addition to education for individuals, improvements in the safety of patient handoffs required system-wide changes. In our case, we transformed our written sign-out templates to electronic, Web-based templates. Our actions were in keeping with the results of studies indicating that electronic sign-out systems decrease the incidence of adverse events,21,22 improve information content,23,24 and improve accuracy by eliminating problems related to legibility and by allowing for more efficient and valid updates to sign-out sheets.21,25,26

We believe that the substantial improvements we found in the interns' spoken sign-out skills were primarily attributable to the curriculum, whereas the improvements in the accuracy of the sign-out sheets were due in large part to our system-level change. For this reason, we now present the curriculum to all interns and teach them how to use the electronic sign-out system before they start fulfilling any of their clinical duties. Moreover, because the sign-out sheets are now Web based, we are able to regularly evaluate and review them to ensure that all of the essential components continue to be included and addressed accurately. In fact, checklists for formal grading of interns' sign-out sheets are now part of residents' evaluation portfolios as one measure of competency in interpersonal skills and communication.

In the future, we may want to expand the electronic sign-out system to allow attending physicians and residents on other medical services to use it. We may also want to explore the possibility of developing a computer program that would automatically extract up-to-date clinical information from electronic medical charts and present it in an organized, user-friendly format. This would reduce the time it takes interns to prepare their sign-out sheets. However, even with this change in the sign-out process, the quality of the bidirectional spoken communication during handoffs would remain of the utmost importance, as would the ability to formally evaluate it.

Limitations of the project

Our project initiatives have at least three limitations that deserve mention. First, our results might not be generalizable to other teaching institutions. Even so, we believe that our methodology would be helpful to other institutions in performing their own local needs assessments and that it could easily be adapted to additional uses. For example, we are using our methodology to evaluate our interns' and residents' skills in completing discharge summaries, discharge instructions to patients, and transfer notes/orders for patients transitioning to long-term care facilities, and we are using the results of these evaluations to develop new teaching modules. Second, the Hawthorne effect (defined as a change in a subject's behavior in the presence of an observer in response to the subject's awareness of being observed) might have led our interns to perform better during the assessments than at other times, as a result of their awareness of an observer's presence. If that were the case, however, we would expect the interns to perform better in both the pretraining and posttraining assessments. Third, our interns' work schedules and vacations prevented us from including all of them in all aspects of the initiatives described here. Now that we are incorporating the curriculum into the orientation for interns and residents, we have eliminated this problem.

Despite these limitations, we believe that the initiatives reported here have several major strengths. First, they add to the literature on housestaff sign-out practices by providing a feasible mechanism for formally evaluating the sign-out process, not just for teaching it. Our pretraining survey allowed us to determine how many of our interns had been trained in sign-out procedures and to gauge how aware they were of their deficiencies in skills. Our multifaceted skills assessment included the identification of skill deficiencies via the collection and checking of sign-out sheets for completeness and accuracy and also via the direct observation of the sign-out process. This step is especially important, given that the 2009 Residency Review Committee guidelines mandate that residents' improvements in clinical competencies be demonstrated by measures that include direct observation of skills.20 Second, our interactive curriculum was targeted to the needs of interns. It incorporated didactic and experiential learning, capitalized on key tenets of adult learning theory, and focused on the communication skills required of the givers and receivers of sign-out data. Third, our evaluation of posttraining skills was rigorous, again incorporating direct observation as well as the evaluation and validation of sign-out sheets. Fourth, and most important, we believe that the multifaceted processes that we developed will lend themselves to modification and use by other residency programs, not only to improve their handoff systems but also to teach and demonstrate SBP and PBLI initiatives to their residents.


Teaching the tenets of SBP and PBLI to residents can be challenging. SBP is closely linked to PBLI because it is often through analysis of the practices of individuals that system-level issues are identified. Indeed, residents' illegible, sticky-note- and correction-fluid-ridden sign-out sheets cried out for improvements in technology and efficiency. The ACGME calls for residents to demonstrate the ability to participate in identifying system errors and implementing system-based solutions, as well as the ability to effectively transmit the clinical information necessary to ensure safe and proper care of patients. A quality-improvement initiative on patient handoffs epitomizes the teaching of these competencies. Similarly, the ACGME calls for residency programs to evaluate residents' ability to coordinate their patients' care (including the transition of care), to work in teams, to identify system problems, and to support and participate in quality-improvement activities. Observing handoffs and other patient-care transitions and creating and using checklists to “grade” transition communication can set the stage for reliable observation and evaluation of the ACGME competencies. Residents need to develop abilities in SBP and PBLI, not only to provide safe and effective care for their patients but also to enable themselves to act as effective physicians within a variety of different settings. Participating in initiatives such as ours promotes resident education about how to work in, learn from, and continually improve health care delivery systems and about how to develop and apply quality-improvement methods throughout their careers.


The authors are grateful to Dr. Sandy Sekhon for her assistance in gathering data, to Prem Kumar and John Kolodziej for their statistical assistance, and to Dr. John Cinicola for helping to facilitate the Web-based system. The authors are especially grateful to Dr. James J. Reilly for his wisdom, guidance, and insight into the ACGME Outcome Project and its implications for residency education.



Other disclosures:


Ethics approval:

Not applicable.


1 Coleman EA, Berenson RA. Lost in transition: Challenges and opportunities for improving the quality of transitional care. Ann Intern Med. 2004;141:533–536.
2 Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. Communication failures in patient sign-out and suggestions for improvement: A critical incident analysis. Qual Saf Health Care. 2005;14:401–407.
3 Horwitz L, Moin T, Krumholz H, Bradley E. Consequences of inadequate sign-out for patient care. Arch Intern Med. 2008;168:1755–1760.
4 Cabana MD, Jee SH. Does continuity of care improve patient outcomes? J Fam Pract. 2004;53:974–980.
5 Moore C, Wisnivesky J, Williams S, McGinn T. Medical errors related to discontinuity of care from an inpatient to an outpatient setting. J Gen Intern Med. 2003;18:646–651.
6 Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143:121–128.
7 Van Walraven C, Seth R, Austin PC, Laupacis A. Effect of discharge summary availability during postdischarge visits on hospital readmission. J Gen Intern Med. 2002;17:186–192.
8 Wasson JH, Sauvigne AE, Mogielnicki RP, et al. Continuity of outpatient medical care in elderly men: A randomized trial. JAMA. 1984;252:2413–2417.
9 Petersen LA, Brennan TA, O'Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121:866–872.
10 Laine C, Goldman L, Soukup JR, Hayes JG. The impact of a regulation restricting medical house staff working hours on the quality of patient care. JAMA. 1993;269:374–378.
11 Beach C, Croskerry P, Shapiro M. Profiles in patient safety: Emergency care transitions. Acad Emerg Med. 2003;10:364–367.
12 Chassin MR, Becher EC. The wrong patient. Ann Intern Med. 2002;136:826–833.
13 Gandhi TK. Fumbled handoffs: One dropped ball after another. Ann Intern Med. 2005;142:352–358.
14 Accreditation Council for Graduate Medical Education. Outcome Project. Available at: Accessed November 30, 2009.
15 The Joint Commission. 2009 National Patient Safety Goals Hospital Program. Available at: Accessed on March 10, 2010.
16 Accreditation Council for Graduate Medical Education. Advancing Education in Interpersonal and Communication Skills: An Educational Resource From the ACGME Outcome Project. Available at: Accessed November 30, 2009.
17 Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards. Arch Intern Med. 2006;166:1173–1177.
18 Philbert I. Accreditation Council for Graduate Medical Education (ACGME). Council for graduate medical education and institute for healthcare improvement 90-day project. Involving residents in quality improvement: contrasting “top-down” and “bottom-up” approaches. Available at: Accessed on March 10, 2010.
19 Horwitz L, Moin T, Green M. Development and implementation of an oral sign-out skills curriculum. J Gen Intern Med. 2007;22:1470–1474.
20 Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Resident Education in Internal Medicine. Available at: Accessed November 30, 2009.
21 Van Eaton EG, Horvath KD, Lober WB, Rossini AJ, Pellegrini CA. A randomized, controlled trial evaluating the impact of computerized rounding and sign-out system on continuity of care and resident work hours. J Am Coll Surg. 2005;200:538–545.
22 Petersen LA, Orav EJ, Teich JM, O'Neil AC, Brennan TA. Using a computerized sign-out program to improve continuity of inpatient care and prevent adverse events. Jt Comm J Qual Improv. 1998;24:77–87.
23 Kannry J, Moore C. MediSign: Using a Web-based SignOut System to improve provider identification. Proc AMIA Symp. 1999:550–554.
24 Ram R, Block B. Signing out patients for off-hours coverage: Comparison of manual and computer-aided methods. Proc Annu Symp Comput Appl Med Care. 1992:114–118.
25 Van Eaton EG, Horvath KD, Lober WB, Pellegrini CA. Organizing the transfer of patient care information: The development of a computerized resident sign-out system. Surgery. 2004;136:5–13.
26 Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: Strategies for a safe and effective resident sign-out. J Hosp Med. 2006;1:257–266.
© 2010 Association of American Medical Colleges