Technical or practical skills deficits upon graduation from medical school and during the first year of internship are prevalent.1–5 It has been recognized that the first month of internship represents an error-prone period, with a need for research aimed at reducing errors by recently graduated medical students.6 The deficits in practical skills of junior doctors have prompted criticisms and reevaluation of the traditional “apprenticeship” model of medical education and training and have prompted researchers to evaluate the efficacy of alternative methods of clinical training.7–11
Current evidence suggests that the use of medical simulation in combination with deliberate practice constitutes best practice.12 Simulation-based learning is considered to have many advantages over the traditional teaching model including increased patient safety; potential for trainee exposure to a myriad of illnesses, physical symptoms, and medical complications; shortened learning curves; the capacity to adjust the degree of difficulty of the simulated task or to repeat learning tasks; and the ability to set learning objectives and a required degree of mastery or competence.13 It has also been demonstrated to be a more effective method of teaching with students who receive simulation-based training, outperforming those who receive traditional training on a variety of practical skills including colonoscopy performance, critical care skills, and surgical performance.14–16
The use of deliberate practice involves the repeated performance of a target skill coupled with frequent assessment of performance and the provision of specific corrective feedback in a controlled setting.17 Deliberate practice has been demonstrated to lead to the acquisition and maintenance of targeted skills and to promote expert performance in a variety of domains.18 The combination of simulation and deliberate practice has been shown to be a highly effective method of teaching important procedural and practical skills such as advanced cardiac life support skills,19 cardiology bedside skills,13 lumbar puncture skills,20 handoff communication,21 and a variety of other basic clinical skills including cardiac auscultation, paracentesis, the management of acutely unwell patients, and appropriate patient communication.22 Research has also demonstrated that procedural skills acquired through deliberate practice in a simulated setting generalize to the natural setting.15,16,23 A recent meta-analysis revealed a large effect size of simulation-based medical education incorporating deliberate practice as compared with traditional clinical medical education.24
Recent research has examined the outcomes of intensive, simulation-based training courses for preinterns intended to improve procedural skills and to facilitate the transition from medical student to trainee doctor.1,22,25 Given that providing appropriate documentation, completion of forms, and prescriptions has been identified as a key weakness among medical interns,26 the current article sought to prospectively examine the impact of a simulation-based, hemovigilance learning program, incorporating deliberate practice, on the ability of incoming interns to request blood products, as compared with a group of interns who did not receive this training.
METHODS
Study Design
This study used a quasi-experimental between-groups design. Postintervention outcomes were assessed by analyzing all blood product requests completed by interns (and more experienced nonconsultant hospital doctors) in an Irish teaching hospital during the first 13 weeks of internship commencing on the July 9, 2012. An audit was performed of the blood product requests at the teaching hospital in which the interns worked, and whether the request was accepted or rejected was recorded. In Irish hospitals, blood product requests are processed by the hospital’s blood bank laboratory after receipt of an accurately completed blood product request document and blood sample by a doctor. A correctly completed document contains (1) patient name, hospital number, date of birth, sex, location, and consultant; (2) printed name, hospital contact number, and signature of the person taking the specimen; (3) date and time of specimen collection; (4) information on the blood products requested (including volume/amount); (5) any special requirements for blood/blood products (eg, irradiated blood); (6) date and time upon which blood products will be required; (7) patient clinical details including information on diagnosis and/or indication for the transfusion; and (8) transfusion history including information on blood group, previous transfusions, reactions, and marrow transplants. Items 1 to 3 must also be replicated on the blood sample specimen. Inaccurate completion of the blood product requesting documentation and blood sample is typically identified by laboratory technicians, which results in the disposal of the specimen, notification of the physician who made the erroneous request, the need to retake the specimen form the patient, and a delay in patient treatment. In more serious cases, inaccurate completion of the documentation can result in the provision of an incorrect blood product to a patient.
Participants
This study was reviewed and approved by Galway University Hospital’s Research Ethics Committee before its commencement. Fifty-nine interns commencing work at an Irish teaching hospital participated in this study. Of these, 28 had participated in an intensive, 3-week training course after graduation from a large Irish medical school (known as school A in this article). This program included extensive simulation-based hemovigilance training, which incorporated deliberate practice. These interns were considered the “trained group.” The trained group was composed of 19 female and 9 male interns. Only one participant in this group was a non–European Economic Area (EEA) citizen. The 31 interns in the “untrained group” did not receive formal training in blood product prescribing before beginning internship. The untrained group was a nonrandomized comparison group consisting of 20 interns who had graduated from other medical universities and 11 graduates from school A but who had not attended the additional 3 weeks of training. The group included 14 female and 17 male interns, while 21 of the participants were EEA citizens and 10 were non-EEA citizens. Among the untrained group participants, 26 had attended an Irish medical school, 4 had attended a medical school in a different EEA country, and 1 participant had attended a non-EEA medical school.
Data were also collected from 104 nonconsultant hospital doctors postinternship who were working in the teaching hospital associated with school A. These participants were included as a nonidentical comparison group to compare the performance of interns with their more established senior peers.
Training
The trained group undertook the intensive training program offered by school A that has been described previously by Byrne et al.1 This 3-week preparation for clinical practice program is offered to all final-year students at school A upon completion of their undergraduate degree. Delivered by a multidisciplinary team of specialists, this training program seeks to provide additional training in the practical skills that have been identified as lacking among new medical school graduates by previous research.2–5 The modules delivered in the program include infection control and policy, hemovigilance and anticoagulation, documentation and clinical note taking, communication and professional development, medicines and prescribing, radiology, and basic core technical procedures (eg, cannulation). Shadowing an experienced intern on the ward and completion of a logbook are also part of the program. The positive impact of this training course on intern preparedness has been reported by Byrne et al.1
The current study is an evaluation of the outcomes of the 3-hour hemovigilance module delivered as part of this training program and additional repetitive practice opportunities throughout the remainder of the training program. The hemovigilance module is taught by 2 hemovigilance officers and a consultant hematologist. The module begins with brief instruction in the protocols for requesting blood products, which focuses on the accurate completion of the documentation and inclusion of all pertinent clinical data. After this, learners are presented with a series of clinical vignettes told by simulated computer-based patients. All vignettes were based on real clinical situations and were developed by the hemovigilance team. Each vignette provides information on the patient’s presenting complaint, background medical history, reason for admission, and reason for requiring blood products. Patient demographics are supplied at the bottom of the screen. Laboratory results, such as hemoglobin or platelet levels, are available to learners as required. Learners are subsequently instructed to request blood products for each simulated patient by completing the hospital documentation, patient wristband, and blood sample bottle correctly and legibly to the standard required by the hospital laboratory, as per the criteria described previously. Upon completion of the assigned task, a group discussion takes place, the correctly completed documents are displayed on screen, and each student’s attempt is reviewed by a hemovigilance officer who provides substantial individual feedback. This process is repeated with vignettes and patient histories of increasing complexity. For example, the initial vignettes describe patients who are undergoing elective surgical procedures and require blood products as part of the preoperative preparation. Later, more complex cases such as those involving patients with previous transfusion reactions, pregnant patients, and immunocompromised patients requiring irradiated blood are used. Learners complete a total of 6 cases during the hemovigilance module, each requiring 10 to 15 minutes for completion, depending on complexity. After this, the learners continue to practice and repeat blood product, requesting an additional 6 times as part of longer simulated patient scenarios delivered during other modules on the program. In these scenarios, learners are required to request the blood products as part of the admission process for the simulated patient while attending to other admission duties such as consent, note taking, and requesting consults.
The key principles of deliberate practice are incorporated in the current training in a manner similar to that described by Issenberg et al13 and Ericsson et al18 in their simulation- and deliberate practice-based training in cardiology bedside skills. We begin the training with simple, straightforward learning activities commensurate with the lack of experience of blood product requesting reported by the learners (see Byrne et al1) and progressively increase the complexity of learning trials as the learners’ performance improves. During the training, learners repeatedly perform the structured learning activities in a controlled setting, their performance on each learning trial is assessed, and individualized corrective feedback is provided to each learner after a learning trial. Improvements in learners’ performance are assessed during each learning trial and in a summative assessment upon completion of the training program.
Assessment
All learners are required to complete a summative assessment at the end of the training program. As part of this assessment, learners must complete the documentation, specimen label, and wristband required for a blood product request, in response to a novel clinical vignette that resembles the more complex clinical scenarios presented after completion of the hemovigilance training program. Further training and practice opportunities are available for learners who do not pass the initial assessments. Learners may retake the assessment after this remediation process.
The outcomes of the simulation-based hemovigilance training program were also assessed in the clinical setting through the analysis of all blood product requests completed by participants in the trained and untrained groups in the first 3 months of internship. The protocols for processing a blood product request were informed by the hospital’s hemovigilance team and the national guidelines from the Irish Blood Transfusion Board.27 Any inaccuracies in the blood product requesting documentation, on the blood bottle, or patient wristband resulted in the blood sample not being processed. The doctor who performed the request was then contacted by laboratory personnel and required to perform a repeat request. Details of rejected specimens and requests, requesting doctor, and the reasons for rejection of the request were compiled over the 3-month period for analysis.
Statistical Analysis
The risk ratio (RR) was used to compare the probability of a sample being rejected between the 2 groups under consideration (eg, trained interns vs. untrained interns). The rationale for using the RR, as opposed to the odds ratio (OR), is that it has been found that ORs can be hard to understand and susceptible to misinterpretation.28 When the probability of an outcome is low (<0.10 or 0.20), as is the case in this study, the RR and OR can be used interchangeably.29 Therefore, as suggested by Grimes and Schultz,29 the RR will be reported in the interests of promoting better understanding of the findings.
RESULTS
All of the learners who attended the hemovigilance training module successfully completed the summative assessment at the end of the training program on their first attempt. All 59 interns (28 trained and 31 untrained) completed the first 3 months of internship. To ensure that a small number of interns were not committing the majority of errors, the data were screened for potential outliers. An outlier was defined as an intern who had committed a number of errors more than 1.5 times the interquartile range above the third quartile. This screening resulted in the data from 2 interns being removed from further analysis (one from the trained group and one from the untrained group). The percentage of rejected samples is shown in Table 1. Of the 70 cases of rejected samples from the untrained group, 27 (39%) were from graduates of school A and 43 (61%) were graduates of other medical schools.
;)
TABLE 1: Number of Accepted and Rejected Samples for Trained Interns, Untrained Interns, and Nonconsultant Hospital Doctors Postinternship
No significant differences were found in the probability of having a sample rejected between untrained school A graduates (rejected rate, 10.5%) and untrained graduates of other medical schools (rejected rate, 9.5%; RR, 1.11; 95% confidence interval [CI], 0.70–1.75; z = 0.45; P > 0.05). There was a significant difference in the rate of rejection between those school A graduates who had not attended the training (rejected rate, 10.5%) and those school A graduates who had attended the training (rejection rate, 3.4%; RR, 0.37; 95% CI, 0.24–0.57; z = 4.52; P < 0.00001).
The RR was used to compare accepted and rejected samples for trained interns versus untrained interns, trained interns versus nonconsultant hospital doctors postinternship, and nonconsultant hospital doctors postinternship versus untrained interns (Table 1 presents the number of accepted and rejected samples for each group). A significant difference was found between untrained interns and trained interns (RR, 0.35; 95% CI, 0.24–0.52; z = 5.14; P < 0.0001), between trained interns and nonconsultant hospital doctors postinternship (RR, 0.55; 95% CI, 0.38–0.78; z = 3.34; P < 0.001), and between untrained interns and nonconsultant hospital doctors postinternship (RR, 1.56; 95% CI, 1.20–2.01; z = 3.35; P < 0.001).
The percentage of rejected samples for each week during the first 13 weeks of internship is shown in Figure 1. It can be seen that the percentage of errors made for untrained interns decreased during the first 13 weeks of internship. As can be seen in Figure 1, an improvement in the performance of all 3 groups was observed across the study. There are a number of factors that may have contributed to these gains. Although the increasing experience levels of the trained and untrained interns may explain the improvements in their blood product prescribing and requesting, this could not account for the improvements in the nonconsultant hospital doctor’s performance as these doctors had all been working in the hospital for an extended period. However, the hospital’s hemovigilance team is responsible for identifying interns or doctors who repeatedly make errors in the requesting of blood products. In such cases, this team intervenes and provides additional training. It is likely that such intervention contributed to the improvements in the performance of each of the groups.
FIGURE 1: Number of rejected samples during the first 13 weeks of internship.
The RR of accepted versus rejected samples for trained and untrained interns for 3 specific periods as internship progresses is demonstrated in Table 2. As would be expected from Figure 1, there was a significant RR for weeks 1 to 5 and weeks 6 to 9. However, although the percentage of errors was higher for untrained interns compared with that of trained interns for weeks 10 to 13 (4.60% and 2.51% respectively), the RR was not significant.
TABLE 2: Number of Accepted and Rejected Samples and RR Separated for the Beginning, Middle, and End of the Study Period
The primary causes of interns’ errors in blood product requests are presented in Table 3. The most frequent source of error involved the blood bottle or patient wristband. A significant portion of all groups had multiple errors involving a given sample. The most frequent errors involved misspelling of names, errors in dates, and omitted details. Omitted details included transfusion history, previous transfusion reactions, and prescriber information. Errors in ordering the correct blood product did not occur. The entire blood product prescribing errors (100%) analyzed in our series led to near-miss events, identified by laboratory staff, with none contributing to the 4 major blood transfusion events reported in our institution during the study period.
TABLE 3: Analysis of Types of Errors Made in Rejected Samples
DISCUSSION
The current study sought to evaluate the efficacy of a short simulation-based hemovigilance training program, which incorporated deliberate practice on the effects of trainee doctors’ blood product requesting during the first 3 months of internship. It was found that those interns who had not received the training were 3 times more likely to have a sample rejected than those interns who had received the training.
The results suggest that between 6 and 13 learning opportunities during the training program were sufficient for the trained interns to be performing at a level of more senior doctors. Moreover, it took 10 to 13 weeks before the untrained interns were not having significantly more samples rejected than the trained interns (although even then the percentage of rejected samples by the untrained interns was almost twice as high as that of the trained interns). The reduction in the probability of a rejected sample observed across each of the groups during the intervention is likely attributable to the intervention provided by the hospital’s hemovigilance team for those who repeatedly make errors in blood product requesting. However, the statistically significant advantage of the trained group over both the untrained comparison group and the nonconsultant hospital doctor suggests that the intervention offered a significant advantage over the routine interventional procedures. These findings are important given the recognized need to reduce medical errors and the poor level of satisfaction reported by junior doctors with regard to transfusion training.30,31 These results also support previous findings that highlight the poor practical competencies of new medical interns and the importance of preinternship training for patient safety and the reduction of medical errors.22
The rate of medical errors in blood product requesting among interns in our study is higher than those observed by Chow et al.6 However, this is likely because the study of Chow et al underestimated the number of errors as the result of the reliance on voluntary near-miss reports. Nevertheless, both studies demonstrate a decrease in errors across time as interns became more experienced in the clinical setting.
Previous research has emphasized the importance of assessing the clinical benefits of simulation-based training programs.15 In the current study, the generalization of the training effects to the real environment suggests that the use of simulation and deliberate practice methodologies may be an effective way to train interns. The significant differences in the competence of trained and untrained interns are consistent with previous studies, which found that simulation-based medical education with deliberate practice results in significantly better procedural skills than traditional medical education.32
There were several limitations to the current study. All data were collected in a single teaching hospital with a relatively small sample of medical interns over a relatively short period. It was also not possible to randomly assign interns to the trained and untrained group. Therefore, the results may be effected by the lack of random assignment and the potential differences in the medical education received by the trained group at school A and the untrained group at other universities. However, although this is a possibility, it seems unlikely given the significant differences in performance noted between the trained school A graduates and the untrained school A graduates and the lack of significant differences found between the untrained school A graduates and the untrained participants who graduated from other universities. A further limitation to this study was the paucity of available information on participant demographics. As a result of this lack of information, it was not possible to provide information relating to participants’ age and to state with certainty that the groups were equivalent on this variable at the outset of the study. Finally, this study was not designed or powered to comprehensively assess the effect of the intervention on adverse event rates in the institution from errors in the requesting of blood products. Therefore, it remains to be determined whether the decrease in the rate of medical errors by interns will translate into a reduction in the rate of adverse events. Although the errors reported in our study did not result in patient transfusion-related events, we do stress the clerical, medical, and financial implications of near-miss events. Improvements in practice will require further analysis on the cause of errors, incorporation of new technology to reduce the complexity of the process, and regular monitoring of hemovigilance performance standards.
Given the limitations of the current study, our findings must be considered as preliminary. It is therefore imperative that future research examines the replicability of the outcomes observed in this study in other settings to confirm the validity of the findings observed in this study. Future research should seek to elucidate the optimal process of incorporating deliberate practice and simulation into undergraduate medical education through the use of more rigorous experimental designs to evaluate the impact of the intervention. Furthermore, the contribution of our results could be strengthened by extending the analysis to other parameters and skills taught in our deliberate practice and simulation program. Finally, there is limited information on the learning curves associated with the acquisition of procedural skills by junior doctor. Future research should assess the number of times a procedure should be performed before competency is achieved and identify the particular step(s) in a procedure where errors are most likely to occur.
CONCLUSIONS
To our knowledge, the current study is the first to examine the effects of deliberate practice- and simulation-based training on the blood product requesting skills of medical interns. The results suggest that the use of a deliberate practice- and simulation-based training methodology led to significantly less errors in ordering blood products as compared with interns who had not received the training. Although these findings must be considered preliminary, in combination with the multiple other studies that have highlighted the utility of simulation- and deliberate practice-based practical and procedural skills training, the recommendation for the more frequent use of these teaching practices in undergraduate medical curricula may be made. This type of training may ensure that medical students are adequately prepared for the challenges of clinical practice and protect patients from unnecessary risk.
ACKNOWLEDGMENTS
The authors would like to acknowledge the contribution of all the hemoviligance team at Galway University Hospital, including Dr Margaret Murray, Dr Amjad Hayat, and Dr Ruth Gilmore, consultant hematologists, and Martina O’Connor, chief hemovigilance officer. They would also like to acknowledge the contribution of the School of Medicine, National University of Ireland, Galway, in the provision of intern training.
REFERENCES
1. Byrne D, O’Connor P, Lydon S, Kerin M. Preparing new doctors for clinical practice: an evaluation of pre-internship training.
Ir Med J 2013; 106 (5): 328–330.
2. Finucane P, O’Dowd T. Working and training as an intern: a national survey of Irish interns.
Med Teach 2005; 27 (2): 107–113.
3. Tallentire VR, Smith SE, Skinner J, Cameron HS. Understanding the behaviour of newly qualified doctors in acute care contexts.
Med Educ 2011; 45 (10): 995–1005.
4. Glover ML, Sussmane JB. Assessing pediatrics residents’ mathematical skills for prescribing medication: a need for improved training.
Acad Med 2002; 77 (10): 1007–1010.
5. Potts MJ, Phelan KW. Deficiencies in calculation and applied mathematics skills in pediatrics among primary care interns.
Arch Pediatr Adolesc Med 1996; 150 (7): 748–752.
6. Chow K, Szeto C, Chan M, Lui S. Near-miss errors in laboratory blood test requests by interns.
QJM 2005; 98 (10): 753–756.
7. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room.
Am J Surg 1999; 177 (1): 28–32.
8. Cation LJ, Durning SJ. Procedure skill competence and certification in internal medicine residency training.
Teach Learn Med 2003; 15 (3): 175–179.
9. McCashland T, Brand R, Lyden E, De Garmo P. The time and financial impact of training fellows in endoscopy.
Am J Gastroenterol 2000; 95 (11): 3129–3132.
10. Club TSS, Moore MJ, Bennett CL. The learning curve for laparoscopic cholecystectomy.
Am J Surg 1995; 170 (1): 55–59.
11. Sheehan D, Bagg W, de Beer W, et al. The good apprentice in medical education.
N Z Med J 2010; 123 (1308): 89–96.
12. Motola I, Devine LA, Chung HS, Sullivan JE, Issenberg SB. Simulation in healthcare education: a best evidence practical guide. AMEE Guide No. 82.
Med Teach 2013; 35 (10): e1511–e1530.
13. Issenberg SB, McGaghie WC, Gordon DL, et al. Effectiveness of a cardiology review course for internal medicine residents using simulation technology and deliberate practice.
Teach Learn Med 2002; 14 (4): 223–228.
14. Gordon JA, Shaffer DW, Raemer DB, Pawlowski J, Hurford WE, Cooper JB. A randomized controlled trial of simulation-based teaching versus traditional instruction in medicine: a pilot study among clinical medical students.
Adv Health Sci Educ Theory Pract 2006; 11 (1): 33–39.
15. Park J, MacRae H, Musselman LJ, et al. Randomized controlled trial of virtual reality simulator training: transfer to live patients.
Am J Surg 2007; 194 (2): 205–211.
16. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study.
Ann Surg 2002; 236 (4): 458.
17. McGaghie WC. Research opportunities in simulation-based medical education using deliberate practice.
Acad Emerg Med 2008; 15 (11): 995–1001.
18. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition of expert performance.
Psychol Rev 1993; 100 (3): 363.
19. Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice.
J Gen Intern Med 2006; 21 (3): 251–256.
20. Barsuk JH, Cohen ER, Caprio T, McGaghie WC, Simuni T, Wayne DB. Simulation-based education with mastery learning improves residents’ lumbar puncture skills.
Neurology 2012; 79 (2): 132–137.
21. Sawatsky AP, Mikhael JR, Punatar AD, Nassar AA, Agrwal N. The effects of deliberate practice and feedback to teach standardized handoff communication on the knowledge, attitudes, and practices of first-year residents.
Teach Learn Med 2013; 25 (4): 279–284.
22. Cohen ER, Barsuk JH, Moazed F, et al. Making July safer: simulation-based mastery learning during intern boot cAMP.
Acad Med 2013; 88 (2): 233–239.
23. Ewy GA, Felner JM, Juul D, Mayer JW, Sajid AW, Waugh RA. Test of a cardiology patient simulator with students in fourth-year electives.
Acad Med 1987; 62 (9): 738–743.
24. McGaghie WC, Issenberg SB, Cohen MER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence.
Acad Med 2011; 86 (6): 706.
25. Laack TA, Newman JS, Goyal DG, Torsher LC. A 1-week simulated internship course helps prepare medical students for transition to residency.
Simul Healthc 2010; 5 (3): 127–132.
26. Illing J, Morrow G, Kergon C, et al.
How Prepared Are Medical Graduates to Begin Practice? A Comparison of Three Diverse UK Medical Schools. Final report to GMC April 2008. 2008.
27. Service IBT, National Blood Users Group (NBUG).
Guidelines for the Administration of Blood and Blood Components. Dublin. 2001. Available at:
http://www.giveblood.ie/Clinical_Services/Haemovigilance/Publications/Guidelines_for_the_Administration_of_Blood_and_Blood_Components.pdf.
28. Holcomb Jr. WL, Chaiworapongsa T, Luke DA, Burgdorf KD. An odd measure of risk: use and misuse of the odds ratio.
Obstet Gynecol 2001; 98 (4): 685–688.
29. Grimes DA, Schulz KF. Making sense of odds and odds ratios.
Obstet Gynecol 2008; 111 (2 pt 1): 423–426.
30. Berwick DM, Leape LL. Reducing errors in medicine.
BMJ 1999; 319 (7203): 136–137.
31. Graham J, Grant-Casey J, Alston R, Baker P, Pendry K. Assessing transfusion competency in junior doctors: a retrospective cohort study.
Transfusion 2014; 54 (1): 128–136.
32. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations.
Acad Med 2011; 86 (11): e8–e9.
