Experience, confidence, and competence are all distinctly different qualities when it comes to medical practice — at least, that is what a new study published online first in the June 6 issue of Neurology, suggests.
When neurology residents at different stages of training were tested on their ability to perform lumbar puncture (LP), most did not meet predetermined mastery standards — even though they had a high level of previous experience and expressed confidence in their abilities. Moreover, the way they were trained might have played a role in their performance, the study authors suggest.
The investigators compared two training methods in different trainee groups at Northwestern University and university-affiliated medical centers in Chicago. Using the simulation-based mastery learning (SBML) model, first-year internal medicine residents were trained to perform LPs on a simulator modeled on a patient's lower torso, which provided realistic resistance to a spinal needle. They then completed a three-hour education session (including a video on LP, an interactive LP demonstration, and practice on the simulated model with direct feedback from supervisors). Immediately afterwards, the residents were required to meet a minimum passing score on a 21-item checklist — and were tested and retested until this was accomplished, which is “the key feature of mastery learning,” the authors wrote.
They compared their scores on the 21-item checklist with those of neurology residents who had undergone traditional training methods, attending lectures, reading journals, observing other residents and clinicians performing LPs on patients and doing the procedure on patients, as well.
The 58 internal medicine residents scored significantly better for LP than the 36 neurology residents on nearly all checklist items. The internal medicine residents improved from a mean of 46.3 percent to 95.7 percent (p < .001) and all met the minimum passing score at the final test. The neurology residents scored significantly lower, with a mean score of 65.4 percent (p < .001) and with only 6 percent meeting the minimum passing score. All residents were blind to checklist items.
The main takeaway point here, study author Diane B. Wayne, MD, professor of medicine and director of the Internal Medicine Residency Program at Northwestern University in Chicago, IL, told Neurology Today, “is that standard clinical training does not always deliver the outcomes that we expect. It's surprising with something as basic as lumbar puncture because when you ask neurology residents, they feel very comfortable doing this procedure — they believe that they are competent, but when you actually test them, there are serious gaps.” For example, more than half of neurology residents could not correctly identify the anatomic location for an LP and were unable to list routine tests ordered on CSF, the study authors noted.
“When you learn [a procedure] on the clinical service — as neurology residents traditionally do, it's possible that you learn it incorrectly as people potentially pass down mistakes and inaccurate techniques from one resident to another,” Dr. Wayne said. There's no gold standard in this way, she added, but when you teach the procedure in a simulation classroom, you control what is being taught and then ensure learning outcomes with a rigorous exam at the end.
“These results illustrate the limitations of traditional clinical training and suggest a role for simulation training during neurology residency,” the study authors wrote. Given the findings, “we believe a procedural standard should be set and documented for all residents prior to performing an LP in actual clinical care,” they wrote. [See “Other Opportunities for Simulation Training in Neurology.”]
EXPERT COMMENTARY, FEASIBILITY
Victoria S. Pelak, MD, associate professor and director of the residency program in the neurology department at the University of Colorado-Denver School of Medicine, who was not involved in the study, said: “The old method of apprenticeship [traditional residency training] which sometimes is referred to as 'See One, Do One, Teach One' is truly not tenable in today's health care environment. Study after study has shown the utility of simulation in education, and these types of rigorous methods really can no longer be ignored when the stakes are so high from the first time a resident lays hands on a patient.”
This is a really good example of where we should be headed in the educational training of our residents — “the use of [tangible] measures to show that the methodology used for training really incurs good outcomes,” she said.
Dr. Pelak said clinicians at her own hospital at the University of Colorado perform about 1000 LPs a year, and many of those are likely done by residents. “Even if you consider half are performed by residents, and if we have a 40 percent success rate (as some studies suggest for resident-performed LPs), that's potentially 200 spinal taps per year — meaning 200 patients per year — who have gone through a procedure that did not allow for a successful diagnosis, and that is a large number. To do something at very minimal cost, such as the procedure set up in this study, that would allow us to save those numbers of unsuccessful spinal taps would be a significant improvement.”
If you're considering cost-efficiency of these simulation models, she said, you have to account for the delay in diagnosis an unsuccessful LP can cause, not to mention costs of fluoroscopy suites that may not be reimbursed, additional spinal tap kits, and the cost of having another physician perform the procedure.
Jeffrey H. Barsuk, MD, lead study author and associate professor of medicine (hospital medicine) a Northwestern University Feinberg School of Medicine, said that they have not yet demonstrated improvement in patient outcomes from LPs using these trainings, but “we have shown it for a central venous catheter insertion mastery learning program, where not only did we reduce mechanical complications but we also reduced infections in our ICU. The cost savings we found for that — and it was a very conservative estimate, taking into account space, faculty, time, simulators, ultrasound — were approximately 700,000 dollars for the hospital in one year.”
“Next, the plan is to figure out if our LP programs improve clinical outcomes and save money as well.” Dr. Barsuk said he expects to see similarly significant numbers. At some institutions, he added, they may refer patients to interventional radiology for an LP, which can increase hospital expenses and reduce value for patients. Factors such as post-LP headache or need for hospital admission because of unsuccessful LPs also currently add to hospital costs, he said.
Dr. Barsuk told Neurology Today, “Adequately training residents to do these procedures competently is no doubt in my mind going to improve patient outcomes and save health care dollars on the back end. It's well-worth the investment you put in.”
As well as teachers, doctors, and educators, we are also patients, Dr. Pelak said, “and so knowing that this individual has mastered a skill in a simulated environment before they're laying hands on you really is much more comforting than what they do now.”
Victoria S. Pelak, MD, associate professor and director of the residency program in the neurology department at the University of Colorado-Denver School of Medicine, talks about the important strides being made in simulation-based education, and why there is no time like the present to start using these training methods: http://bit.ly/AqMXdf.
©2012 American Academy of Neurology
• Barsuk JH, Cohen ER, Wayne DB, et al. Simulation-based education with mastery learning improves residents' lumbar puncture skills. Neurology
2012; E-pub 2012 June 6.
Neurology Today Quick Links