There has been strong demand for the development and implementation of tools to assess the competency of postgraduate medical trainees.1 Simulation is a valid tool for competency assessment that can be poorly assessed with typically used forms of assessment such as oral or written examinations.2
The use of simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. Simulation-based medical education with deliberate practice has been shown to be superior to traditional clinical medical education in achieving specific clinical skill acquisition goals within the realms of emergency-stabilization.3
Emergency-stabilization of patients is an area where successful treatment relies on excellent therapeutic actions and communication skills. Recent study has reported that postgraduate medical education programs would benefit from a robust process early in residency for training followed by assessment of competence in critical care.4 In particular, postgraduate first year (PGY-1) residents, who are directly responsible for giving primary care, are expected to perform efficiently in emergency stabilization of critical patients. Simulation-based objective structured clinical examination (OSCE) is considered an idea tool for assessment of residents’ resuscitation competency.5 One way of ensuring the necessary expertise is to test whether PGY-1 residents meet the required standards for proficiency of emergency-stabilization in practical exams such as the simulation-based OSCE. Rotation of formative OSCE by group is referred to as GOSCE, and has been found to motivate the self-reflection, self-directed learning, and peer learning of participants.6–10 Peer learning is a form of learning that involves informal dynamic relationships within a group of individuals who are similar in experience and rank. Because of the equality among group members, relationships are generally personal and mutual, and ideally, each participant has something of value to contribute and gain.11,12 Accordingly, this study implements a new Trios-OSCE model to enhance the PGY-1 residents.
The research questions of this study are as follows: (1) Do PGY-1 residents, who receive formative Trios-OSCE, in addition to common workshop, show better emergency-stabilization skills than a PGY-1 resident used exclusively as a control single-OSCE? (2) Are they more satisfied with the training? (3) In comparison with pretraining score, do they have higher post-training self-efficiency scores?
2.1. Baseline advanced cardiac life support certification
All PGY-1 residents were required to complete their advanced cardiac life support (ACLS) courses before entering training. In addition to pass the skills certification, writing knowledge tests were completed by all PGY-1 at the end of ACLS certification.
2.2. Background for implementation of the “emergency-stabilization”-enhanced simulation course
An annular survey of physicians and nurses of the postgraduate educational committee revealed that emergency-stabilization is the top subcompetency of patient care competency that needs to be trained first at the beginning of the PGY-1 residency program (Supplementary Table 1). In fact, program directors and experts considered that the ACLS course by itself is sufficient to ensure trainees’ proficiency in emergency-stabilization during clinical rotation. Thus, additional simulation courses, including workshops and OSCE, were organized to improve the emergency-stabilization subcompetency of PGY-1 residents’ patient-care competency.
2.3. “Emergency-stabilization”-enhanced simulation course
The simulation course was held in the first 2 months of PGY-1 training. This 4-hour simulation course began with 1-hour of classroom didactics to introduce the core elements for the emergency-stabilization subcompetency. Meanwhile, the Queen’s simulation assessment tool (QSAT) was introduced to all enrolled PGY-1 residents. All PGY1 residents watched video of real examples of good practice in the therapeutic action and communications skill domains of QSAT in simulation. In the second hour, trainees interacted through role-play, discussions, and scenario creation, and they practiced the demonstrated scenario from the training video and their newly created scenario.
In the training video, the example provided was a case of an acute subarachnoids hemorrhage that requires superior levels of therapeutic actions including (1) neuroprotective rapid-sequence intubation; (2) elevation of head of bed; (3) blood pressure monitoring and control; (4) drugs (mannitol, hypertonic saline, rapid sequence intubation medications, prothrombin complex concentrate [Octaplex], vitamin K, anti-HTN); and (5)ventilator settings (hyperventilation). On the contrary, perfect communication included the following: (1) introduction of self and explanation of clinical situation; (2) clear and concise orders and directions; (3) prioritization of tasks and anticipation of further steps; (4) demonstration of leadership in managing crises; (5) appropriate specialist consultation (neurosurgery); and (6) requests for family presence. For assessment, the performance of trainees in therapeutic actions and communication domains were divided as follows: inferior (delayed or incomplete performance of all criteria), novice (delayed or incomplete performance of many criteria), competent (delayed or incomplete performance of some criteria), advanced (competent performance of most criteria), superior (competent performance of all criteria); the overall performance were also divided as the following: inferior (all skills require significant improvement), novice (most skills require moderate or significant improvement), competent (some skills require moderate improvement), advanced (some skills require minor improvement), and superior (few, if any skills require moderate or significant improvement).
Initially, the QSAT was conceptually designed to assess the competence of a senior-level resident.13 Nonetheless, our study used QSAT to train junior PGY-1 residents. Accordingly, in our test scenarios, the prespecific information required for the primary assessment and diagnostic actions domains were routinely provided to trainees by a senior standardized nurse and a standardized intern. In other words, among the four domains of the original version of QSAT, our study focused on training for the therapeutic actions and communication domains of emergency-stabilization subcompetency. For facilitators, standardized nurses, and trainees, the assessment of overall performance of each trainee was based on their performance in the two aforementioned domains rather than four domains.
In the third hour, there was an introduction by the facilitator in the individual OSCE station to orient all PGY-1 residents to the objectives of this formative assessment, simulation environment, and the structure/scoring of OSCE. This included a question and answer session to address trainee concerns. It was emphasized that each PGY-1 resident should treat the high-fidelity SimMan 3G simulator and SP as a real patient and family in a clinical setting. The PGY-1 residents were also notified that simulation-based assessment would start as soon as they entered the station. They were expected to demonstrate their emergency-stabilization subcompetency according to the evolving scenario by interacting with the standardized senior nurse, standardized intern, and family. Then, 3-case-OSCEs, either in Trios-OSCE or Single-OSCE groups, were undertaken in the last hour of the emergency-stabilization-enhanced simulation course.
In the Trio-OSCE long-station, each PGY-1 resident received four-graded performance levels in the four aspects of assessor-evaluated therapeutic actions, assessor-evaluated communication, facilitator-evaluated overall performance, and standardized senior nurse-evaluated overall performance during their turn to exercise one scenario. In contrast, in the regular Single-OSCE group, an assessor, facilitator, and standardized senior nurse all observed the performance of each PGY-1 resident in three stations from the control room and gave a final rating for each PGY-1 resident’s performance. Accordingly, the graded performance levels of Trios-OSCE were determined by randomly exercising one of the three preselected scenarios by a PGY-1 resident, whereas the graded performance level of each Single-OSCE trainee was determined from three exercises of all three scenarios. The assessments were completed immediately at the end of either Trios-OSCE or Single-OSCE by the assessor, facilitator, and standardized senior nurse.
2.4. Study design
This prospective interventional study was conducted between January 2016 and October 2016. After the common workshop, PGY-1 residents were randomly divided into intervention group-A (n = 18), regular group (n = 18), and control group (n = 15) (Fig. 1 and Supplementary Table 2). Within the OSCE station with standard simulation setting, interventional Trios-OSCE and regular single-person OSCE were arranged over two consecutive days, as shown in Supplementary Tables 3 and 4. Unlike the regular format of Single-OSCE in the regular group, PGY-1 residents were randomly divided into groups of three to rotate through the OSCE long-station in the intervention group. Additionally, one control group of trainees, who observed their peers’ performance and gave feedback in the debriefing phase, was included for comparison. Ethical approval (IRB 2015-12-015BC) was obtained from the ethics committee of our institution, and care was taken to apply the World Medical Association Declaration of Helsinki principle of research.
2.5. Intervention group
This simulation course was held with triplicate circuits of one 3-case-long-OSCE according to the availability of the high-fidelity simulation setting in our OSCE center. In Trios-OSCE, three scenarios in one OSCE long-station were completed sequentially by three PGY-1 residents. In the first 10 minutes of long-station, when the first PGY-1 resident acted as the treating physician in the first emergency-stabilization scenario, the other two PGY-1 residents were peer observers. In the next 20 minutes, the first PGY-1 resident in turn observed the other two performances for the second and third scenario. There was a warning bell at five minutes and the last two minutes of each scenario. There were two minute periods between the three scenarios in order to change the clinical setting. As a formative assessment, the assessor, facilitator, senior standardized nurse, and control group made specific feedback to each PGY-1 resident’s performance, according to the QSAT framework in therapeutic actions, communication, and overall performance (Fig. 1). Notably, each PGY-1 resident in Trios-OSCE group exercised three different scenarios in rotation of the long-station.
2.6. Regular group
For the regular exam OSCE, each PGY-1 resident rotated three stations independently to complete the therapeutic actions and communication domains in three scenarios. The setting and individuals involved in this Single-OSCE were similar to Trios-OSCE.
2.7. Control group
With similar background training from the common workshop, this group served as the control to analyze the advantage of simulation-based OSCE for trainees. In addition to the common workshop, PGY-1 residents in the control group observe the OSCE and gave feedback to their peers without being involved in the OSCE stations. This group aimed to demonstrate the efficacy of simulation-based OSCE on trainees (Supplementary Table 2).
2.8. Preparation of OSCE
Three emergency-stabilization-focused scenarios from a previous study13 were revised by the educational committee. They were as follows: first, a simulation of hyperkalemia with bradycardia; second, a simulation of ST-segment elevation myocardial infarction with ventricular fibrillation; and finally, a simulation of upper gastrointestinal bleeding with pulseless electrical activity. Each scenario included scripted roles and clear instructions for the standardized nurse and the intern as well as the human simulator operator and clinical setting staff. For each scenario, a customized QSAT was used on the generic QSAT but included specific anchors relating to clinical presentation and desired observable behaviors. Each of the scenarios and assessment tools were piloted and revised in the simulation laboratory through practice sessions. For either the Trios-OSCE or Single-OSCE, the scenario ended when the allotted time expired regardless of actions performed by the PGY-1 resident or the condition of the patient, an acceptable practice in the context of an assessment.
2.9. Time-point of assessments
To standardize assessments, the qualified assessors, experienced facilitators, and standardized senior nurses received a two-hour orientation training session for each OSCE on the use of the QSAT for specific scenarios. The assessors, facilitators, and standardized senior nurses were the same across similar OSCE stations. Immediately after the OSCE, each PGY-1 resident completed the questionnaire about course-value and self-efficacy listed in Supplementary Table 5. In particular, the self-efficacy scores for emergency-stabilization were rated by trainees themselves before and after the simulation course (Fig. 1).
2.10. Statistical analysis
A reliability analysis (Cronbach α) was calculated for the questionnaires and assessment instruments. Continuous variables (eg, mean age or the score knowledge test in ALCS) were compared by using ANOVA, whereas categorical variables (eg, gender distribution or the percentage of who had finished one month of emergency medicine rotation) were compared by using Chi-square test. In this study, the Chi-squared test of homogeneity was used to test whether the trainees of two groups had the same distribution of performance levels in terms of therapeutic actions, communication, and overall aspects.
3.1. Reliability of assessment
In the postworkshop OSCE, the inter-rater reliability for the QSAT in each of the three emergency-stabilization-enhanced scenarios was estimated using Cohen’s kappa agreement coefficient. Good inter-rater reliability for three scenarios with acceptable kappa values (0.72, 0.69, and 0.7) were noted.
3.2. Basal characteristics
Also notable was that the age, gender distribution, and the percentage of trainees who had completed their one-month emergency-medicine rotation and knowledge test score in ACLS were comparable among intervention, regular, and control groups (Table 1). Notably, all aforementioned parameters were the same among groups.
3.3. PGY-1 residents perform better in Trios-OSCE than in Single-OSCE
In comparison with the regular group, a higher proportion of the intervention group’s trainee’s performances were rated as better than competent levels “competent + advanced + superior” on the communication and overall performance domains by qualified assessors, an experienced facilitator, and the standardized senior nurse (Table 2). Additionally, a higher proportion of the interventional group’s trainees were rated at the “superior” level than those in the regular group. For the therapeutic actions domain, a similar proportion of the intervention-group’s trainees’ performances were rated as better than “competent” level in comparison to those in the regular-group. In general, although they had similar written knowledge score in their previous ACLS courses of both groups, a higher proportion of intervention group trainees were rated as superior level compared to the regular group trainees. For therapeutic actions, our study suggests that the observation of peers’ performance helped the intervention group’s trainees perform better in Trios-OSCE than the regular-group in Single-OSCE. Notably, the proportion of overall performance of trainee’s who were rated by the experienced facilitator as being above competent level was significantly higher in intervention group A than those in regular group B. Nonetheless, the differences between the other parameters of performance between groups did not reach significance in Table 2.
3.4. Additional simulation course increases self-efficacy and receives high course value scores from PGY-1 residents
Basically, precourse average self-efficacy scores were similar for the intervention, regular, and control group trainees. In general, the postcourse self-efficacy scores were higher than precourse scores among trainees of the intervention, regular, and control groups (Table 3). The degree of increase in self-efficacy scores was higher among the intervention group than the regular and control groups. In the follow-up stage, the trend of increasing self-efficacy scores was noted in both the intervention and regular groups. For all trainees among the three groups, high postcourse and follow-up course value scores confirm the well-constructed course meets the needs of trainees, as well as the success of the motivational factors of self-directed learning and self-reflection in Trio-OSCE. At the postcourse stage, a comparably larger increase in self-efficacy scores was noted in observation control group C trainees than in the other two groups. This result indicated the effectiveness of the common workshop as well as the benefits of observation and giving feedback that raised trainees’ proficiency in emergency-stabilization in our program (Table 3).
3.5. Trios-OSCE-based simulation course increases the transfer of learned skills to clinical practice
Figure 2 reveals that a higher proportion of the interventional group’s trainees applied the therapeutic actions and communication skills learned during the course in practice, a result which is represented by higher “always” and “always + frequently + often” proportions observed in the intervention group than in the other two groups. Again, the proportion of clinical application of learnt skills in the control group was comparable to the regular group. This supports the importance of peer observation and giving feedback in our new model. On the contrary, this result indicated that the pre-OSCE common simulation course is crucial for motivating trainees to apply learned skills in clinical practice.
Training for emergency-stabilization subcompetency is critical for patient safety and is an essential part of postgraduate medical education. In acute critical condition, delayed action may have severe consequences. Thus, good situational confidence (self-efficacy) is important to ensure that individuals have the abilities to succeed in a given challenge due to their actual actions.14 In the shift toward competency-based medical education, simulation-based OSCEs have been reported as an effective competency-based training and assessment tool. In particular, simulation-based OSCEs can assess specific ACGME competence or subcompetence effectively. The new simulation course in our study was characterized by the inclusion of common pre-OSCE workshop and Trios-OSCE to audit and train PGY-1 residents’ for emergency-stabilization subcompetency. In comparison to Single-OSCE in our study, good self-efficacy and clinical transfer of trained skills were found in PGY-1 residents receiving the new Trios-OSCE simulation course.
In comparison with individual-based diagnosis of patients, higher accuracy by team-based diagnosis indicated the importance of training junior residents’ for group collaboration.15 In our study, all three intervention group’ PGY-1 residents rotated the Trios-OSCE long-station together, a typical formative GOSCE. Compared with the regular group, better examination results in the intervention group can be explained by the additional opportunities to observe their peers’ performances in the same simulation OSCE station during examination. With regards to the clinical transfer in our study, group training and auditing by the Trios-OSCE simulation course benefited the intervention group more than the regular group with single-person rotation.
Peer learning is the main theoretical basis for the design of Trios-OSCE in our study. The additional opportunity to observe the performance of peers before one’s turn with hands-on practice might increase PGY-1 resident’s performance in Trios-OSCE. Nonetheless, there is no significant difference in the distribution of proportion of the assessor’s, facilitator’s, and standardized senior nurse’s graded performance levels of first, second, and third practicing PGY-1 residents in Trios-OSCE by subgroup analysis. These results might be explained by the small sample size (n = 6 of first, second, and third practicing PGY-1 residents, separately) in this pilot study as well as the dominant effects of the common pre-OSCE workshop on trainees’ performance in Trios-OSCE. Accordingly, it is mandatory to clarify this issue in a future large-scale study.
It has been reported that hands-on learning is as efficient as vicarious learning in the acquisition of complex manual skills such as emergency-stabilization.16 It has also been suggested that video watching enhances trainees’ observational powers, improves their ability to integrate different information, and motivates them to learn.17,18 Progressively, activities including classroom didactics, discussion, scenario creation, video watching, and role-playing built the emergency-stabilization subcompetency of all trainees in our study.
Self-assessment is the first step in the feedback process and represents the person’s ability to self-assess for a particular task. Specifically, active observation and provision of feedback further enhance the effectiveness of vicarious learning.19,20 Thus, it is reasonable to find that our simulation course has comparable benefits of self-efficacy and clinical transfer of learned skills for the control group, whose members were observers and feedback givers rather than OSCE examinees.
Our results show that the effect of vicarious learning extends beyond knowledge and skills acquisition. Notably, it increases self-efficacy of PGY-1 residents and ultimately encourages the clinical transfer of learned skills. This provides a convenient opportunity for educators to increase the training group size to be more economical.
Although the number of PGY-1 residents was limited, different group trainees did not differ from those of other groups on important characteristics and may thus be considered as being representative of their group. In addition, assessment from qualified assessors, experienced facilitators, and standardized senior nurse simultaneously support the effectiveness of this Trios-OSCE simulation course for training the emergency-stabilization subcompetency.
The limitation of our study is that the comparison of OSCE scores is not under the same standard between the two groups because the trainees of the interventional group received only one score from the single scenario exercised, while those in the regular group received three or more scores from three different stations. In fact, our study did not assess the subjective effects on patient’s outcomes. Nonetheless, few emergency care courses have used patient outcomes as an end-point.21 Meanwhile, due to the frequent changing of the rotation ward characteristics of PGY-1 residency courses, it is difficult to ask mentors to assess trainees’ performances at bedside. A strong positive association exists between the hours of practice on simulation and learning outcomes in medical education.22 Although the intervention was a single four-hour course in our study, a significantly higher degree of increase in self-efficiency and clinical transfer of trainees was observed. The results show that experience in workshops, Trios-OSCE (intervention), Single-OSCE (regular) or observation (control) groups are promising for fostering PGY-1 residents’ emergency-stabilization subcompetency.
In conclusion, the strength of this work includes the following: (1) a follow-up design and significantly better confidence in the intervention group, which suggests better learning retention with the new training method; and (2) an increased training capacity (nine instead of three trainees at the same time), which makes the new training method more economical per-trainee. In particular, the emergency-stabilization simulation offers opportunities for PGY-1 residents to build therapeutic action and communication skills. In a typical simulation course including workshop and OSCE (especially Trios-OSCE), PGY-1 residents’ self-efficacy increased with training, and the improvement was associated with positive response to the course and a high degree of clinical transfer.
APPENDIX A. SUPPLEMENTARY DATA
Supplementary data related to this article can be found at https://links.lww.com/JCMA/A19.
1. Steadman RH, Huang YM. Simulation for quality assurance in training, credentialing and maintenance of certification.Best Pract Res Clin Anaesthesiol2012263–15
2. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality.Acad Med201388872–83
3. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence.Acad Med201186706–11
4. McMurray L, Hall AK, Rich J, Merchant S, Chaplin T. The nightmares course: a longitudinal, multidisciplinary, simulation-based curriculum to train and assess resident competence in resuscitation.J Grad Med Educ20179503–8
5. Hall AK, Pickett W, Dagnone JD. Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents.CJEM201214139–46
6. Biran LA. Self-assessment and learning through GOSCE (group objective structured clinical examination).Med Educ199125475–9
7. Ludwig AB, Raff AC, Lin J, Schoenbaum E. Group observed structured encounter (GOSCE) for third-year medical students improves self-assessment of clinical communication.Med Teach201739931–5
8. Ledford CJ, Seehusen DA, Canzona MR, Cafferty LA. Using a teaching OSCE to prompt learners to engage with patients who talk about religion and/or spirituality.Acad Med20148960–5
9. Yusuf IH, Ridyard E, Fung THM, Sipkova Z, Patel CK. Integrating retinal simulation with a peer-assessed group OSCE format to teach direct ophthalmoscopy.Can J Ophthalmol201752392–7
10. Shah I, Mahboob U, Shah S. Effectiveness of horizontal peer-assisted learning in physical examination performance.J Ayub Med Coll Abbottabad201729559–65
11. Noonan M, Ballinger R, Black R. Peer and faculty mentoring in doctoral education: definitions, experiences, and expectations.Int J Teach Learn Higher Educ200719251–62
12. Martinez J, Harris C, Jalali C, Tung J, Meyer R. Using peer-assisted learning to teach and evaluate residents’ musculoskeletal skills.Med Educ Online20152027255
13. Hall AK, Dagnone JD, Lacroix L, Pickett W, Klinger DA. Queen’s simulation assessment tool: development and validation of an assessment tool for resuscitation objective structured clinical examination stations in emergency medicine.Simul Healthc20151098–105
14. Woods JL, Pasold TL, Boateng BA, Hensel DJ. Adolescent health care and the trainee: roles of self-efficacy, standardized patients, and an adolescent medicine rotation.Simul Healthc20138359–67
15. Hautz WE, Kämmer JE, Schauber SK, Spies CD, Gaissmaier W. Diagnostic performance by medical students working individually or in teams.JAMA2015313303–4
16. Stegmann K, Pilz F, Siebeck M, Fischer F. Vicarious learning during simulations: is it more effective than hands-on training?Med Educ2012461001–8
17. Chan LK, Patil NG, Chen JY, Lam JC, Lau CS, Ip MS. Advantages of video trigger in problem-based learning.Med Teach201032760–5
18. Balslev T, de Grave WS, Muijtjens AM, Scherpbier AJ. Comparison of text and video cases in a postgraduate problem-based learning format.Med Educ2005391086–92
19. Chi MT, Roy M, Hausmann RG. Observing tutorial dialogues collaboratively: insights about human tutoring effectiveness from vicarious learning.Cogn Sci200832301–41
20. Grierson LE, Barry M, Kapralos B, Carnahan H, Dubrowski A. The role of collaborative interactivity in the observational practice of clinical skills.Med Educ201246409–16
21. Teteris E, Fraser K, Wright B, McLaughlin K. Does training learners on simulators benefit real patients?Adv Health Sci Educ Theory Pract201217137–44
22. McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. Effect of practice on standardised learning outcomes in simulation-based medical education.Med Educ200640792–7