Triage is an essential process in most emergency departments (ED) owing to overcrowding and impossibility of taking care of every patient immediately. Usually, triage decisions are made by ED nurses, according to pre-established criteria. Several triage scales are available for hospital triage. The Australasian Triage Scale, the Canadian Triage and Acuity Scale, the Manchester Triage System, and the Emergency Severity Index have shown moderate to good reliability and validity when used by triage nurses in ED [1–4]. The Swiss Emergency Triage Scale (SETS), which is available in French, Italian, and German, has high reliability and is recommended by the Swiss Society of Emergency and Rescue Medicine and used in Switzerland, France, and Belgium .
These scales are usually used in the ED to determine treatment priorities among patients. In the prehospital setting, no general ED triage scale has ever been validated, and data supporting the use of triage instruments by paramedics are scarce , except for specific situations such as stroke or trauma [9–12]. Nevertheless, the use of a triage scale in the prehospital setting may not only allow to determine the level of priority of a given patient but might also facilitate his/her orientation from the field to the most appropriate hospital as both acuity and suspected diagnosis may determine required resources and specific orientations.
To improve care of elderly patients requiring urgent evaluation, a specific Emergency Geriatric Admission Unit (EGAU) was opened in late 2016 in Geneva County. This unit, located a few miles away from the main ED building, has less intense medical and technical resources. Therefore, elderly patients aged older than 75 years who present with nonlife threatening conditions should be directly diverted by paramedics to EGAU using a triage scale.
The primary objective of this study was to assess the inter-rater reliability in prehospital triage among a group of trained paramedics using the SETS. The secondary objectives were (a) to evaluate the accuracy of triage performed by paramedics compared with the emergency levels attributed by experts, (b) to assess the reliability among paramedics regarding the triage of simulated patients older than 75 years, and (c) to evaluate the accuracy of orientation of patients older than 75 years to a dedicated emergency unit decided by paramedics.
Patients and methods
This study was a prospective cross-sectional study among a convenient sample of paramedics exposed to 28 clinical scenarios using an interactive computerised triage simulator.
The Geneva University Hospitals is a 1800-bed primary and tertiary urban teaching hospital. The adult ED admits more than 65 000 patients annually and is the only public ED in Geneva County, Switzerland. Elderly patients (>75 years old) represent 35% of patients admitted by ambulance. All patients admitted to the ED are triaged using the SETS, a four-level triage scale with high inter-rater and intra-rater reliability . Chief complaints, objective parameters (vital signs), and key questions are used to stratify the risk in four categories: emergency level 1 – life-threatening emergencies requiring immediate care, emergency level 2 – urgent conditions requiring medical evaluation within 20 min, emergency level 3 – semiurgent conditions, requiring medical evaluation within 2 h, and emergency level 4 – nonurgent conditions.
Selection of participants
In Geneva County, six ambulance companies take care of 25 000 patients each year. In Switzerland, paramedics receive a 3-year education program including didactic courses and field internship. All of the 23 certified paramedics working in the largest ambulance company in Geneva were eligible for participation. Informed consent was required for each participant. The study was approved by the ethics committee of Geneva University Hospitals (CER 15–083, clinicaltrials.gov #NCT02559531).
All participants were trained to the principles and the use of SETS during a 4-h teaching session, which is the standard to train ED nurses to the use of SETS in our center. This training was provided by two nurses and two emergency physicians certified in emergency medicine and specialized in ED triage. Paramedics were also specifically taught that patients of at least 75 years should be primarily admitted to EGAU if (a) their chief complaint was part of a pre-established list of complaints considered compatible with EGAU technical resources and (b) their attributed emergency level was 3 or 4. If both conditions were not fulfilled, those patients should be primarily admitted to the main adults ED.
A mobile version of the SETS available on tablet PCs was used to facilitate its use in the ambulance. After the initial teaching session, all participants used the SETS in the field during 2 months to become familiar with the scale, and they received systematic feedback by the principal investigators on their triage decisions.
After the training period, all participants were asked to triage 28 standardized scenarios ‘played’ by an interactive triage computerized simulator. This simulator integrated 28 clinical scenarios based on cases abstracted from real situations and representative of most common situations at the ED (Table 1). The program simulated the triage process as close as possible to real-life conditions. For each scenario, a short description was presented, including the patient’s age, sex, and presenting symptom. The interactive simulator offered the participant the possibility to ask questions, to obtain vital signs, and additional clinical features to gather information for the triage decision. A list of 273 different answers (including patient’s history and clinical signs) was available for each scenario. The participants were not limited in the number of questions they could ask and were required to proceed as they would do under usual practice conditions. The scenarios were performed in a random order, and the participants were not allowed to communicate with one another during the triage. At the end of their simulated evaluation, each paramedic assigned a chief complaint and an emergency level (1–4) using the SETS criteria. They also determined whether the patients more than or equal to 75 years should be admitted to EGAU or to the main adults ED.
The primary outcome was the reliability of the triage level performed by the 23 participants. Interpretation of intraclass correlation coefficient (ICC) was made as follows: ICC between 0.61 and 0.8 reflected ‘substantial’ reliability and ICC above 0.81 ‘excellent’ reliability. Because we used a convenient sample of 23 participants and 28 predefined clinical vignettes, we estimated that the expected variance for a targeted ICC between 0.80 and 0.90 would be between 0.0006 and 0.002 .
Secondary outcomes were the accuracy of triage compared with the expert-based level, the reliability of triage level in the vignettes presenting simulated patients of at least 75 years, and the adequate orientation of these patients to EGAU versus ED. A triage was considered as correct if the paramedics assigned the same emergency level as the level attributed by a panel of multidisciplinary experts (two nurses and two emergency physicians certified in emergency medicine and specialised in ED triage). Overtriage and undertriage were defined respectively as all overestimation and underestimation of the emergency level by the participant compared with the expert-attributed level. The orientation of patients of at least 75 years was considered as correct if the paramedics dispatched the patient adequately to the main ED or to the EGAU. Orientation was classified as ‘overtriage’ if participants chose ‘main ED’ instead of ‘EGAU’ and ‘undertriage’ if they selected ‘EGAU’ instead of ‘main ED’.
For each participant, we obtained information on age, sex, and time experience in prehospital care in years (< 2, ≥2 to <4, ≥4 to <8, and ≥ 8 years).
Continuous variables were presented by their median and interquartile ranges. Categorical variables were presented by their frequency and relative proportions.
To measure the reliability in the triage assessment among the 23 participants, we calculated ICC with its 95% confidence interval (CI) using a two-way random-effect model to take into account that the 28 scenarios were rated by the same set of 23 independent raters.
We assessed the factors associated with the accuracy of triage by performing a generalized linear mixed model using a logit function and two non-nested random effects on the intercept (one on the scenario and the other on the rater). All models were adjusted for the following variables: age of the patient presented in the scenario (< or ≥ 75 years), sex of paramedics, and time experience spent in prehospital care (< 2, ≥2 to <4, ≥4 to <8, and ≥ 8 years). For each variable, we reported an adjusted odds ratio (OR) with its 95% CI and the random-effect estimates of the scenario and the rater as SD. We assessed the factors associated with undertriage after exclusion of the scenario presenting level 4 (23 observations). Similarly, we assessed the factors related to overtriage after exclusion of all scenarios presenting a level 1 (n = 5 scenarios and 115 observations). For the two outcomes (undertriage and overtriage), we used again a generalized linear mixed model with a logit function and two non-nested random effects on the intercepts (one on the scenario and the other on the rater). We adjusted for the same variables as in the first model assessing the accuracy of triage.
Finally, we assessed the factors associated with the accuracy of orientation to EGAU (vs. main ED) in the subgroup of scenarios presenting patients older than 75 years. Again we used a generalized linear mixed model with a logit function and two non-nested random effects on the intercept (one on the scenario and the other on the rater). We adjusted for the following variables: sex of paramedics and time experience spent in prehospital care (< 2, ≥2 to <4, ≥4 to <8 and ≥ 8 years).
All analyses were performed using Stata version intercooled 14 (Stata Corp., College Station, Texas, USA). Statistical significance was defined as P value of less than 0.05 (two sided)
All 23 eligible paramedics completed the evaluation of the 28 scenarios (644 triage decisions). The participants were mostly men (73.9%), with a median age of 31.3 (interquartile range: 28.8–33.6) and a median prehospital care experience of 4 years (interquartile range: 2–8).
Overall ICC was 0.84 (95% CI: 0.77–0.99). Correct emergency level was assigned in 89% of the 644 triage decisions. Overtriage rate was 4.8% and undertriage rate was 6.2%. In the multivariate model, correct triage was not associated with any of the variables included in the model (sex and experience of the paramedic and age of the simulated patient). Overtriage was lower in male paramedics (OR in males 0.34, 95% CI: 0.14–0.85, P = 0.021) but was not associated with the age of the simulated patient or the experience of the paramedic. No variables were significantly associated with undertriage (Table 2).
Fourteen of the 28 clinical scenarios concerned patients older than 75 years. In this subset of elderly patients, six were expected to be primarily addressed to EGAU and eight to the main adults ED. In this subset of patients, ICC assessing the reliability of answers among participants regarding orientation was 0.76 (95% CI: 0.61–0.89). Correct orientation was chosen by participants in 93% of cases, 3% were overtriaged (admission to the main ED instead of EGAU), and 4% undertriaged (admission to EGAU instead of the main ED). Neither sex nor time experience spent in prehospital care was associated with the secondary outcome.
In this study, the reliability of the SETS used by paramedics was evaluated with an ICC of 0.84, reflecting excellent inter-rater agreement. Moreover, emergency levels attributed by paramedics were correct in 89% of cases and orientation concerning the subset of elderly patients was correct in 93% of cases.
Other triage instruments have been evaluated for the triage of patients at the door of hospital EDs. These have shown moderate to good inter-rater reliability between nurses trained in triage [1–41516]. Among these hospital-based triage scales, only the Canadian Triage and Acuity Scale and the Emergency Severity Index have been tested in a prehospital setting. Both these instruments have shown low to moderate concordance between paramedics and research nurses triage assignments . Although our study used expert opinion rather than research nurses triage as a reference, it suggests that the SETS reliability is comparable or better than reliability of other validated triage scales when used by paramedics for prehospital triage. This was obtained after minimal training and field experience with the instrument. Moreover, this excellent reliability in the convenient sample tested was also associated with a low proportion of undertriage or overtriage, suggesting that reliability was consequent to efficient triage of the simulated patients.
To deliver the right care to the right patient in the right place, ED have specialized their services. Trauma centers or stroke-dedicated units are examples of this specialization. In Switzerland, paramedics decide the orientation of patients in most cases, a prehospital physician being involved only for life-threatening situations. Paramedics are responsible to choose the most appropriate hospital, taking into account resource intensity, specific procedure availability, and proximity. Therefore, paramedics have a pivotal role in dispatching the good patient to the most appropriate hospital structure. In trauma, efforts have been made to develop prehospital triage criteria predicting the need for primary transfer to trauma center but so far with low sensitivity and specificity . In stroke, implementation of prehospital stroke triage policies routing ambulance primarily to dedicated centers increased the percentage of patients admitted to stroke centers and the percentage of patients receiving thrombolysis . These prehospital pathways used disease-specific triage scales such as the Cincinnati Prehospital Stroke Scale score or the Los Angeles Prehospital Stroke Screen. Vicente et al.  explored the benefit of prehospital triage by nurses for specific geriatric conditions based on a decision support system tailored to elderly patients. They showed that nurse triage in an ambulance is feasible and may help to better dispatch elderly patients . In contrast, the SETS is a nonspecific general triage scale that is applicable to every clinical condition. This has the potential to tailor prehospital triage to the specificities of various healthcare environments. In our study, we showed that elderly patients could be easily and reliably oriented by paramedics toward the appropriate center based on predetermined triage criterias.
The way to optimally evaluate the reliability of triage instruments is source of debate. These evaluations should be as close as possible to real life. Although some authors suggest that real-life evaluations should be performed by different users, this has almost never been done . The most common method is the use of written vignettes displaying all required items to the evaluator. This does not reflect the variability of the triage process and may overestimate reliability. The use of a computerized interactive triage simulator is far more realistic, as it requires active search of relevant items by the evaluator . As this process is closer to real-life triage, the observed reliability of SETS is at least as good and credible as the one observed with other validated triage scales. Similarly, the accuracy of triage with SETS by paramedics was very high and at least comparable to other hospital-based triage instruments [1–415].
Our study has some limitations. First, this study was performed on a limited number of scenarios and was limited to a single ambulance company, which may preclude the generalizability of the results. Nevertheless, the chosen ambulance company was the largest in Geneva, and the scenarios were created to represent a wide and typical spectrum of emergencies. Second, the number of participants was limited to a convenient sample of paramedics. Despite this limitation, the number of evaluations gave us a higher than 80% probability that the lower limit of the one-sided 95% CI of the ICC would be at least 0.70, considered as substantial reliability when the anticipated value is 0.84. This study was considered as a pilot study before a wider use of the SETS in prehospital setting. Third, the education of paramedics differs between countries and continents which may limit the generalizability of our results. However, in our country, paramedics receive a 3-year education which is very comparable to US programs and the use of a general triage scales may be of interest worldwide. Fourth, the evaluation of clinical scenarios was not performed in real-life conditions. Nevertheless, our computerized triage simulates real-life triage as close as possible, allowing the evaluators to interact with the simulated patients as they would do in the field. Fifth, this study evaluated a computerized version of the SETS that determines mandatory levels of emergency according to main complaints, vital signs, and key questions. Therefore, results cannot be generalized to other types of support. Details regarding the computerized SETS have been published previously . Finally, we had a limited number of variables to adjust our statistical models for all potential confounders, in particular to assess the association between sex and outcomes. The large majority of our participants were males, which further limits the interpretation of this association. However, association between triage officers’ sex and overtriage was also reported in a recent study by Vigil et al.  where women nurses were more likely than their male colleagues to attribute higher emergency severity index to emergency patients.
This study shows that the SETS, a general triage instrument, has excellent reliability when used by paramedics and that triage was done with low proportion of undertriage or overtriage. This suggests that the SETS could be used safely in the prehospital setting to determine the level of emergency and orientate patients to the most appropriate hospital in Switzerland.
The authors thank Emmanuel Durand and Frederic Monjo for the development of the computer simulator software and mobile SETS application.
Conflicts of interest
There are no conflicts of interest.
1. Ebrahimi M, Heydari A, Mazlom R, Mirhaghi A. The reliability of the Australasian Triage Scale: a meta-analysis. World J Emerg Med 2015; 6:94–99.
2. Mirhaghi A, Heydari A, Mazlom R, Ebrahimi M. The reliability of the Canadian Triage and Acuity Scale: meta-analysis. N Am J Med Sci 2015; 7:299–305.
3. Mirhaghi A, Heydari A, Mazlom R, Hasanzadeh F. Reliability of the emergency severity index: meta-analysis. Sultan Qaboos Univ Med J 2015; 15:e71–e77.
4. Parenti N, Reggiani ML, Iannone P, Percudani D, Dowding D. A systematic review on the validity and reliability of an emergency department triage scale, the Manchester Triage System. Int J Nurs Stud 2014; 51:1062–1069.
5. Rutschmann OT, Hugli O. Recommendations from the Swiss Emergency and Rescue Medicine Society for triage in Swiss Emergency Departments. Bull Med Suisses 2009; 90:1789–1790.
6. Rutschmann OT, Hugli OW, Marti C, Grosgurin O, Geissbuhler A, Kossovsky M, et al. Reliability of the revised Swiss Emergency Triage Scale
: a computer simulation study. Eur J Emerg Med 2017; [Epub ahead of print].
7. Smith DT, Snyder A, Hollen PJ, Anderson JG, Caterino JM. Analyzing the usability of the 5-level Canadian Triage and acuity scale by paramedics
in the prehospital
environment. J Emerg Nurs 2015; 41:489–495.
8. Buschhorn HM, Strout TD, Sholl JM, Baumann MR. Emergency medical services triage using the emergency severity index: is it reliable and valid? J Emerg Nurs 2013; 39:e55–e63.
9. Hasegawa Y, Sasaki N, Yamada K, Ono H, Kumai J, Tsumura K, et al. Prediction of thrombolytic therapy after stroke-bypass transportation: the Maria Prehospital
Stroke Scale score. J Stroke Cerebrovasc Dis 2013; 22:514–519.
10. Newgard CD, Richardson D, Holmes JF, Rea TD, Hsia RY, Mann NC, et al. Physiologic field triage criteria for identifying seriously injured older adults. Prehosp Emerg Care 2014; 18:461–470.
11. Tirschwell DL, Longstreth WT Jr, Becker KJ, Gammans RE Sr, Sabounjian LA, Hamilton S, et al. Shortening the NIH Stroke scale for use in the prehospital
setting. Stroke 2002; 33:2801–2806.
12. Wasserman EB, Shah MN, Jones CM, Cushman JT, Caterino JM, Bazarian JJ, et al. Identification of a neurologic scale that optimizes EMS detection of older adult traumatic brain injury patients who require transport to a trauma center. Prehosp Emerg Care 2015; 19:202–212.
13. Rutschmann OT, Kossovsky M, Geissbuhler A, Perneger TV, Vermeulen B, Simon J, et al. Interactive triage simulator revealed important variability in both process and outcome of emergency triage. J Clin Epidemiol 2006; 59:615–621.
14. Shoukri M, Asyali M, Donner A. Sample size requirements for the design of reliability study: review and new results. Stat Methods Med Res 2004; 13:251–271.
15. Esmailian M, Zamani M, Azadi F, Ghasemi F. Inter-rater agreement of emergency nurses and physicians in Emergency Severity Index (ESI) triage. Emerg (Tehran) 2014; 2:158–161.
16. Dalwai MK, Twomey M, Maikere J, Said S, Wakeel M, Jemmy JP, et al. Reliability and accuracy of the South African Triage Scale when used by nurses in the emergency department of Timergara Hospital, Pakistan. S Afr Med J 2014; 104:372–375.
17. Fuller G, McClelland G, Lawrence T, Russell W, Lecky F. The diagnostic accuracy of the HITSNS prehospital
triage rule for identifying patients with significant traumatic brain injury: a cohort study. Eur J Emerg Med 2016; 23:61–64.
18. Prabhakaran S, O’Neill K, Stein-Spencer L, Walter J, Alberts MJ. Prehospital
triage to primary stroke centers and rate of stroke thrombolysis. JAMA Neurol 2013; 70:1126–1132.
19. Sanossian N, Liebeskind DS, Eckstein M, Starkman S, Stratton S, Pratt FD, et al. Routing ambulances to designated centers increases access to stroke center care and enrollment in prehospital
research. Stroke 2015; 46:2886–2890.
20. Vicente V, Svensson L, Wireklint Sundstrom B, Sjostrand F, Castren M. Randomized controlled trial of a prehospital
decision system by emergency medical services to ensure optimal treatment for older adults in Sweden. J Am Geriatr Soc 2014; 62:1281–1287.
21. Farrohknia N, Castren M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, et al. Emergency department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resusc Emerg Med 2011; 19:42.
22. Nissen L, Kirkegaard H, Perez N, Horlyk U, Larsen LP. Inter-rater agreement of the triage system RETTS-HEV. Eur J Emerg Med 2014; 21:37–41.
23. Vigil JM, Coulombe P, Alcock J, Stith SS, Kruger E, Cichowski S. How nurse gender influences patient priority assignments in US emergency departments. Pain 2017; 158:377–382.