Secondary Logo

Journal Logo

Special Report

The Double-Edged Sword of Checklists

Shaw, Gina

doi: 10.1097/01.EEM.0000616440.05701.7f
Special Report

Checklists have become nearly as ubiquitous in medicine as scrubs.

The juggernaut began in 2006 when Peter Provonost, MD, PhD, and colleagues at Johns Hopkins University published the results of a study at several Michigan hospitals showing that using a paper checklist of infection-control procedures contributed to a dramatic decrease (up to 66%) in catheter-related bloodstream infections. (N Engl J Med. 2006;355[26]:2725.) The World Health Organization also recommended that all hospitals adopt its Surgical Safety Checklist after two pilot studies showed that death rates declined from 1.5 percent to 0.8 percent after surgical checklists were implemented at eight hospitals around the world. (N Engl J Med. 2009;360[5]:491.)

Harvard surgeon and noted author Atul Gawande, MD, was also an advocate for the method, writing in his book The Checklist Manifesto in 2011 that “...checklists seem able to defend anyone, even the experienced, against failure in many more tasks than we realized. They provide a kind of cognitive net. They catch mental flaws inherent in all of us—flaws of memory and attention and thoroughness.”

Checklists soon swept through many aspects of clinical care. There were rounding checklists in the ICU, discharge checklists for hospitalists, and, of course, plenty of checklists for emergency physicians, such as EMCrit's for intubation. (http://bit.ly/33zwFBU.) In complex, broad, and time-pressured specialties such as emergency medicine, these cognitive aids help ensure that no components of patient care are omitted and that interventions are completed when required. “Checklists are effective in managing unpredicted emergency situations,” wrote Stephen Hearns, MBChB, an emergency physician at Royal Alexandra Hospital in Paisley, Scotland. (Emerg Med J. 2018;35[9]:530.)

Back to Top | Article Outline

Not a Panacea

But a growing body of evidence and opinion suggests that checklists in and of themselves aren't the panacea for medical errors. Other researchers have found it difficult to replicate the success of that surgical safety checklist: An analysis of more than 200,000 procedures at 101 hospitals in Ontario, Canada, for example, found that surgical safety checklists yielded no significant decreases in complications or deaths. (N Engl J Med. 2014;370[11]:1029; http://bit.ly/2MCKybs.)

And an effort by British hospitals to replicate Dr. Provonost's results with central line infections also led to no significant changes in infection rates. (BMJ Qual Saf. 2013;22[2]:110; http://bit.ly/2nQiV6w.)

Although a recent study from critical care medicine was not performed in an ED, it has relevance for emergency medicine, finding that the verbal performance of a written, preprocedure checklist did not prevent hypoxemia or hypotension during endotracheal intubation of critically ill adults compared with usual care.

“We surveyed over 20 airway experts and asked them what the preparation items are that they always or almost always do when intubating a patient,” said lead author David Janz, MD, an assistant professor of medicine in pulmonary/critical care at Louisiana State University School of Medicine in New Orleans. “We derived a checklist from that survey and randomized patients to the checklist versus usual care. We saw that the checklist improved the rate at which these items were being performed such as evaluation for a difficult airway. But in the end, use of the checklist did not result in better procedural or patient outcomes, perhaps because lots of these checklist items were already being performed fairly frequently at baseline.”

That finding illustrates a broader theme about the efficacy of checklists. “It appears that their utility and effectiveness are most strikingly seen in clinical environments where the items on that checklist are not being commonly performed in usual care,” Dr. Janz said. “The evidence seems to bear that out. Studies in which checklists are associated with improved outcomes are usually done in environments where those items are never or rarely being performed at baseline.”

Back to Top | Article Outline

Checklists in Context

“In the right context, a checklist can work very well,” agreed Ken Catchpole, PhD, the SmartState Endowed Chair of clinical practice and human factors in the department of anesthesia and perioperative medicine at the Medical University of South Carolina.

He gave the example of crisis checklists, which are especially germane to emergency physicians. “If you have situations that might be high workload and high difficulty that you only rarely encounter, having a checklist to go through certain items can help,” Dr. Catchpole said.

“Compare it with driving a car. If you had to go through a checklist every time you left your driveway—do I have enough gas, is my seat belt on, are the lights working—it would be frustrating and wouldn't improve your driving,” he said. “But what if you had to change a tire on your car? That's something you probably don't do very often, but it has a number of key steps to remember that will definitely improve safety and performance, such as making sure the parking brake is on before you jack up the car.

A similar example might be found in surgery, Dr. Catchpole said. “Think about conversion from laparoscopic surgery to open. In the old days, surgeons performed many more open surgeries...but today our trainees mostly do laparoscopy. About one out of every ten laparoscopies must be converted to open surgery, and in those circumstances, a checklist might help with that.”

Checklists that take time but don't add value can be worse than useless, he said. “In most situations, spending time is just money, which is bad enough, but in other situations, going through a needless checklist with a patient who is in a risky situation can put them at greater risk.”

Back to Top | Article Outline

Checklists for Procedures

In other cases, checklists can be a double-edged sword. Emergency physicians at the Icahn School of Medicine at Mt. Sinai in New York developed a checklist of possible etiologies for syncope to be provided alongside ECGs to help emergency medicine residents identify ECG patterns more accurately. (Am J Emerg Med. 2019 Mar 29. doi: 10.1016/j.ajem.2019.03.048.)

They found no difference in test scores between those who read ECGs with a checklist and those who read them without. A post-hoc analysis found that those who used checklists were more likely to recognize Brugada, long QT, and heart block than those without a checklist, but they were also more likely to overread normal ECGs, finding pathology where there was none.

“ECGs are something that come up all the time that you have to look at and assess very quickly, and can also be an interruption in the rest of care, as they are often brought to residents and attendings while they are in the midst of another task,” said Jillian Nickerson, MD, an emergency physician who was the lead author of the study at Mt. Sinai. “It's an interpretation that could easily come with errors because of distraction. On top of that, it's an area where there is a significant difference in learning between junior residents, senior residents, and attendings. When you've seen thousands of ECGs, it's much easier to identify pathology because of pattern recognition. We were thinking about ways we could assess and address that.”

The study randomized 165 residents; 39 percent were interns, 23 percent second-year residents, and 38 percent were third- or fourth-years. The sample size wasn't large enough to do a subgroup analysis to assess whether the checklist might have been more effective with junior trainees, Dr. Nickerson said.

“I'm a big believer in checklists, and I think they're really useful in procedural situations, particularly in times when people might forget a step,” she said. “But what was eye-opening about this project is that while there is a potential to impact cognitive load and differentials by using checklists, I don't think they are as simple a solution as they might be for a more procedural situation.”

Back to Top | Article Outline

Designing Effective Checklists

Not all checklists are created equal. Design is essential to an effective checklist, Dr. Catchpole said. “First, you need to work out what problem it is that you're trying to solve. Look at outcome measures, safety incidents, observations of the work, and the challenges people have in conducting their jobs, which would be reasons why you might have bad outcomes. This involves systems analysis, looking at different ways in which human performance is affected by the system. Don't jump straight to a checklist; try to design a system where you don't need one. If you can't, then a checklist might be effective.”

If a checklist seems needed once the systems analysis is conducted, understand the tasks at hand to develop the checklist items. It should be comprehensive enough to address the problem, but short enough so that it is relatively easy to execute, Dr. Catchpole said. “This works relatively well with things like complex devices and discrete tasks that need to be remembered,” he said. “A lot of checklists have tons of words, which end up being instructions and not checks. ‘If this indicator is red, then go and talk to this person; if it's green, then go and talk to that person.’ That's not a checklist. It should be ‘Is this indicator green?’ assuming that if it's not, the person knows how to move forward.”

Dr. Catchpole also recommends timing how long it takes to complete a checklist and how many people are involved, considering those tasks within the broader context of everyday practice. How many checklists is it reasonable for an individual clinician to use in a day?

Dr. Catchpole advised listening to the people who are most negative in designing a new checklist. “It's a mistake to associate checklist objections with people who don't like change,” he said. “Sometimes resistance to change is why people push back on a checklist, but it also may be because they have a really genuine concern and will tell you something important that will make your checklist better.”

Ultimately, checklists in medicine are only effective if end users believe in them enough to employ them in more than mindless ways. “Checklists work and work really, really well when people believe that they improve results,” said Charles Bosk, PhD, a professor of sociology and of anesthesiology and critical care at the Perelman School of Medicine at the University of Pennsylvania. “It's easier to provide examples of where they don't work. The thing about checklists is that they always promise more than what they deliver, in part because an overuse of them forgets that the operators are human.”

Message overload, particularly overwhelming in the emergency department, can wear down the effectiveness of a checklist over time. “The first time you go through a checklist, you pay a lot of attention, but when you've done it multiple times a day over years, it's easy to skip over. And it loses its value,” Dr. Nickerson said. “That's the challenge with much of quality improvement today. Reminders within the medical record, pop-ups that say, ‘Are you sure you want to do this?’ ultimately yield a lot of clicking without thinking, and they don't always yield what they are meant to.”

Dr. Bosk agreed, noting that checklist erosion occurs, making a once-meaningful checklist an exercise in ticking boxes. “It's not enough to just invent a checklist; you have to reinvent it. It has to be a living thing,” he said.

Share this article on Twitter and Facebook.

Access the links in EMN by reading this on our website, www.EM-News.com.

Comments? Write to us at emn@lww.com.

Ms. Shawis a freelance writer with more than 20 years of experience writing about health and medicine. She is also the author of Having Children After Cancer, the only guide for cancer survivors hoping to build their families after a cancer diagnosis. You can find her work atwww.writergina.com.

Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.