To the Editor:
We are writing to express our concerns about the study methodology and conclusions drawn by Mosier et al (1) published in the recent issue of Critical Care Explorations. Although publication of negative findings is of critical importance, and we have no objections to the publication of negative findings, a negative study must still be properly performed. We have significant concerns about the study by Mosier et al (1) data and the limitations of the retrospective design resulting in their conclusions. Our concerns include the following: 1) inappropriate inclusion of erroneous data, 2) unclear definition of point-of-care ultrasound (POCUS) in the study, 3) inappropriate data mining techniques, and 4) incorrect conclusions based on flawed data.
We perform quality assurance review of all POCUS examinations stored in Qpath ultrasound database (2). Since we share the exact same data sources as the investigators and found the results inconsistent with years-long quality assurance review, we queried the same data during the same time frame. Our retrospective query during the identical time period of the study by Mosier et al (1) varies considerably with that presented in the article. The number of examinations were several hundred less than what is listed in the study. In addition, when we specifically queried for POCUS examinations performed for hemodynamic instability, the number of examinations we found were only 10% of what is listed in the study by Mosier et al (1). Hence, we found the original data to be flawed.
The flaw in this retrospective study design was that it did not accurately reflect POCUS utilization at our institution. The POCUS program at our institution is a very robust program that provides education to approximately 78 residents from three different programs per year and ongoing graduate medical education for physicians who may lack POCUS skills. As part of our POCUS educational system, an attending physician or resident may perform educational ultrasound examinations which are not used for medical decision-making and are thus educational only, yet these are logged and stored on the same ultrasound database. POCUS examinations that are used for medical decision-making are done under supervision of an Emergency Medicine attending credentialed by the institution to perform and integrate POCUS results into clinical decisions. If POCUS findings are used for medical decision-making, then they would be clearly documented in the ultrasound report, with the report archived in the patient’s electronic medical record.
After comparison with more refined database queries (educational studies vs studies used for medical decision-making), it is clear that our colleagues included educational POCUS studies in their sample. They did not check if POCUS examinations were used for medical decision-making or whether or not the faculty who are listed on the examinations were credentialed to perform these studies. From our query, it is evident that authors included several hundreds of educational POCUS studies in cohorts 2 or 3, thus misclassifying exams performed for diagnostic or therapeutic purposes.
Even within the completely flawed ultrasound sample the authors used, it is also unclear what type of POCUS study was included. POCUS in the emergency medicine setting can mean one of 12 different applications, and one of five different modes (resuscitative, diagnostic, monitoring, therapeutic, or symptom/sign-based). The article does not define how POCUS was performed in each case. An ultrasound performed to simply look at the inferior vena cava collapsibility provides very different information than an advanced POCUS performed at the bedside. It is clear that the investigators did not review the POCUS studies themselves to determine whether or not or how the POCUS findings changed medical management. Although cardiac and pulmonary POCUS can direct interventions, POCUS also guides advanced procedures such as central line placement or abscess identification for source control. Are we to assume that a POCUS-guided central line (long recognized as the safest method to perform a procedure that limits patient morbidity and mortality) is now associated with increased mortality?
Also not identified in the methods was that the ICU consultants began performing POCUS ultrasound examinations on all potential ICU admissions, inadvertently contributing to cohort 3 ultrasound examinations and skewing the temporal relationship between examinations and interventions in the emergency department (ED). Thus without removing many included subjects from their cohorts, it is erroneous to conclude that any association exists between mortality rate and POCUS since interventions had already been performed without the information provided by the POCUS examination.
Prior studies using POCUS in hypotensive, dyspneic, or hemodynamically unstable patients have used strict protocols that define the equipment, training, sonographic windows, and findings (3, 4). In the study by Mosier et al (1), there is no clear definition of the sequence and type of POCUS applications. There was no protocol in place to assess patients presenting with hemodynamic instability with POCUS during the study period at our institution. There were no specific documentation instructions or check boxes in the medical records to document if POCUS was used to direct interventions. This leads one to question the role of POCUS in the chain of causality. Was POCUS the cause of mortality or a confounder in the complex relationship between critical illness and mortality? Since use of vasopressors and antibiotics were also associated with increased mortality (with a stronger relationship in these data), how would the authors respond if readers concluded that they too are harmful? What is a plausible mechanism for POCUS effect on mortality? Did POCUS result in such mismanagement as to divert from a successful therapeutic pathway? Did physicians withhold IV fluids or antibiotics or delay a diagnostic test? The most likely explanation for these discrepancies is likely to be found in how the data were collected and modeled. Although the model included many important variables several were absent. Given the observational nature of these data, how is the reader to pick and choose which signal matters? Imputation of missing values is also a significant limitation and problem with the study by Mosier et al (1), which easily could have influenced the results. Imputation for outcome values seem extremely problematic in a study that has so much ambiguity.
The ED POCUS literature is slowly becoming inundated with studies that lack clear methodology defining interventions and outcomes. Physicians, with fixed mindsets about which diagnostic tests best serve their patients, are continuously allowed to deviate from protocols when POCUS is randomized resulting in confusing conclusions. Investigators loosely define exposure to POCUS in observational designs muddying any meaningful associations. One of the challenges to Emergency Care research is that patient management relies on several physician groups (emergency physicians, intensivists, and surgeons) each with particular mindsets on the importance of POCUS. Unless POCUS studies begin to enforce group assignment and specify clear outcomes, conclusions about the potential benefits (or harms) will continue to remain opaque at best and suspect at worst.
In summary, our colleagues included flawed data in the study by Mosier et al (1). A majority of POCUS examinations included in the study by Mosier et al (1) were educational studies, which were not used for medical decision-making or to direct any interventions and had no temporal relationship to interventions. We feel strongly that this fatally flawed study which provides misleading results regarding a potentially lifesaving diagnostic application will cause more harm to our critically ill patients than good.
Richard Amini, MD
Elaine Hua Situ-LaCasse, MD
Josie Acuña, MD
, Department of Emergency Medicine, The University of Arizona, Tucson, AZ;
Daniel Theodoro, MD
, Division of Emergency Medicine, Washington University School of Medicine, St. Louis, MO;
Michael Blaivas, MD
, Department of Medicine, University of South Carolina School of Medicine, Columbia, SC;
Vivek Tayal, MD
, Department of Emergency Medicine, Carolinas Medical Center, Charlotte, NC;
Srikar Adhikari, MD, MS
, Department of Emergency Medicine, The University of Arizona, Tucson, AZ
1. Mosier J, Stolz U, Milligan R, et al. Impact of Point-of-Care Ultrasound in the Emergency Department on Care Processes and Outcomes in Critically Ill Nontraumatic Patients.Crit Care Expl20191e0019
2. Adhikari S, Amini R, Stolz L, et al. Implementation of a novel point-of-care ultrasound billing and reimbursement program: Fiscal impact.Am J Emerg Med201432592–595
3. Jones AE, Tayal VS, Sullivan DM, et al. Randomized, controlled trial of immediate versus delayed goal-directed ultrasound to identify the cause of nontraumatic hypotension in emergency department patients.Crit Care Med2004321703–1708
4. Jones AE, Craddock PA, Tayal VS, et al. Diagnostic accuracy of left ventricular function for identifying sepsis among emergency department patients with nontraumatic symptomatic undifferentiated hypotension.Shock200524513–517