Course Corrections for Clinical AI

Introduction

The translation of artificial intelligence (AI) from a promising technology to a routine clinical tool is already well underway, but its benefits to patients and their physicians are not assured, requiring additional guidance from clinicians and industry regulators. In this perspective article, we review the evidence that clinical AI benefits or harms patients and their medical providers. We argue that although clinical AI has already created positive change in medicine, stakeholders such as physicians, hospital administrators, and regulators must demand higher standards of evidence, improved auditing capabilities, and greater attention toward workers’ quality of life if we are to maximize this technology’s potential.

Recently, a boom in AI in medicine has occurred, driven by modern machine learning (ML) methods, whose ability to automatically learn patterns from large amounts of data has enabled them to widely surpass earlier AI systems that required more manual programming by human experts. Although the idiosyncrasies of ML are occasionally relevant to clinicians, in this article we more broadly define AI to include both ML and other computational approaches that automate intellectual tasks typically reserved for human intelligence. In this way, we consider AI from an operational perspective, namely, in terms of its effect on health care workers and patients. For any type of medical AI system, the salient clinical questions remain the same: how does AI effect patients? How does it affect physicians? And what should we do to make it better?

How Does AI Affect Patients?

AI affects patients by informing diagnosis or treatment decisions through the processing of medical images, physiologic measurements, or health records; these technologies, known as “clinical decision support systems,” may decrease cost and improve patient outcomes. Dozens of AI systems have been approved by the United States Food and Drug Administration (FDA), the majority of which analyze radiologic images, whereas the next most common subset monitors cardiac function. Miscellaneous other systems analyze other medical images, monitor other physiologic signals, act as digital assistants, or provide automated therapy (¹).

To assess the potential benefits of these systems, we reviewed publicly available FDA application summaries and peer-reviewed publications and found a few devices show benefit in prospective clinical trials. In evaluating these devices, attention to high-quality evidence, namely, prospective data, ideally from controlled trials and concerning clinically meaningful end points, is critical, because the de facto standard for assessment of AI systems in the ML community is evaluation of predictive performance (e.g., accuracy at disease classification) on retrospective data, which may overestimate real-world performance and, more importantly, disregards patient outcomes. In a success case, a prospective clinical trial of intracranial hemorrhage detection software that triages computed tomography scans demonstrated reduced time to diagnosis of intracranial hemorrhage (²), and a number of similar software devices for triage in radiology have since come to market. Similarly, a randomized clinical trial of 68 patients in a single center demonstrated that an AI-based early warning system for hypotension events reduced the frequency and severity of hypotensive episodes during surgery (³). A small prospective trial of three nurses with 30 patients each evidenced that an AI sonography assistant can enable nurses who were not trained in sonography to acquire satisfactory ultrasound images (⁴), which could perhaps improve access to this imaging modality. Finally, in a randomized clinical trial of 142 patients, an ML-based sepsis prediction tool reduced hospital length of stay and mortality (⁵); although larger, multicenter trials will be necessary to confirm the generalizability of this result, this study provides rare evidence of meaningful improvements in patient outcomes. Although the ultimate effect of the remainder of these systems on patient outcomes remains to be seen (e.g., does the faster time to stroke diagnosis correspond to lower National Institutes of Health stroke scale scores?), these trials, which examine not just the predictive accuracy of AI systems but rather more meaningful clinical end points, support that AI is already offering improvements to medicine.

However, despite the promise of clinical decision support systems, high-quality empirical evidence for patient benefit is scarce. Most AI systems are approved via Section 510(k) of the Food, Drug and Cosmetic Act (¹), which requires only that the device sponsor prove substantial equivalency with an already marketed, “predicate” device. Although potentially sufficient to ensure device safety, equivalency studies often lack high-quality evidence for benefits of AI; the performance data are frequently retrospective (⁶) and may indicate as little as correlation with the predicate device’s output (⁷). The lower costs and straightforward requirements of section 510(k) premarket notifications incentivize this mechanism over the de novo and premarket authorization pathways, which require a higher standard of evidence, implying the regulatory structure is partially responsible for the dearth of high-quality evidence.

Even when high-quality studies are available, the evidence may not indicate substantial benefit for AI. An AI-based tool for triage of traumatic brain injury via electroencephalogram was examined in a multisite prospective observational trial of 720 patients, and failed to meet its predefined primary end points for sensitivity and specificity (⁸); the system’s usefulness at the lower-than-target sensitivity and specificity remains unclear. Similarly, when a widely used, proprietary sepsis prediction model (the Epic Sepsis Model), which is based on a simple form of ML, was evaluated in a retrospective cohort study of 27,697 patients who were hospitalized, this model fell far short of its developer’s reported performance (⁹), questioning whether such an inaccurate model could retain net benefit relative to the alert fatigue it generates. Another AI system, which assists physicians in dosing insulin for pediatric patients with type 1 diabetes, was demonstrated in a multinational clinical trial of 108 patients to be noninferior for maintaining the desired blood glucose levels to physician guidance alone (¹⁰); the study indicates not that the device provides benefit, but rather that use of the device does not hinder physicians, although providing only weak evidence (via Likert scales) that the device decreases physician workload. A prospective clinical trial of 900 patients in ten centers determined that an AI-based screening system for diabetic retinopathy achieved prespecified end points for sensitivity and specificity (¹¹); although we laud the prospective design, which is greatly needed in AI testing, it disregarded meaningful clinical effects such as cost or patient outcomes, and a later study on human factors associated with a similar system discovered shortcomings in usability and actionability that would necessitate additional refinement (¹²). This lack of evidence of benefit implies additional high-quality studies will need to be performed before physicians should be confident using these devices with their patients.

Such caution is appropriate with the introduction of AI technologies because, in addition to their potentially inefficient use of financial resources, in worst-case scenarios they could not only fail to improve quality of care, but indeed reduce quality of care. For instance, computer-aided detection for screening mammography was one of the earliest forms of clinical AI, but multiple studies encompassing hundreds of thousands of patients concluded that computer-aided detection fails to improve—and may actually impair—both sensitivity and specificity, while costing upwards of US$1 billion annually and potentially harming patients through additional, ineffectual biopsies (^1314–15). In another example, a prospective clinical trial of an electronic health record popup that alerts of AKI demonstrated the alerts were associated with increased mortality (¹⁶), casting doubt on ML-based alert systems for AKI that are under development (¹⁷,¹⁸). Although the simple, formula-based alert from this clinical trial lacks the predictive abilities of modern AI systems to anticipate future AKI, plausible mediators of the observed increase in mortality, including alert fatigue or increased prescription of nonindicated interventions (¹⁶), may persist regardless of a shift toward more technically complex AI. These counterproductive instances of AI reinforce that health care providers and hospital administrators should consider not purchasing or using a clinical AI system outside clinical trials until high-quality evidence of benefit is available, and likewise they should consider taking part in clinical trials of promising AI to help improve the availability of evidence.

How Does AI Affect Physicians?

Predictions of the effect of AI on physicians’ work vary widely, ranging from the assertion that radiologists no longer need to be trained to the more optimistic forecast that AI will make all physicians more efficient and productive, rather than replacing them (¹⁹). Although our review of the literature identified no technology sufficiently advanced to supplant any medical provider in the near future, we also remain skeptical that AI’s incorporation will improve physicians’ work experience in the absence of attentive guidance toward this goal.

One area where AI is poised to influence medical providers’ work experiences is the optimization of scheduling. For instance, AI systems better predict the duration of surgical procedures than standard techniques (²⁰), implying they could improve operating room scheduling (²¹). However, published evaluations of automated scheduling systems frequently retrospectively measure surrogate outcomes, such as the distribution of allocated shifts, rather than prospectively measuring physician satisfaction (²²,²³). Furthermore, a metareview of interventions to reduce physician burnout demonstrated that schedule adjustments had a mixed effect on physician stress and job satisfaction (²⁴). Given that workload and scheduling are reliable predictors of emotional exhaustion in physicians (²⁵), AI scheduling systems could improve or deteriorate medical providers’ work experience according to the extent they reflect worker preferences.

Clinical AI systems may also indirectly affect scheduling by altering the rates of medical procedures. For instance, an AI system that diagnoses a disease more frequently than physicians (²⁶) may increase the rate of a corresponding intervention, changing the demands on physicians. If not addressed with appropriate staffing changes, such an increase in diagnoses, whether these diagnoses are correct or overdiagnoses, may contribute to physician burnout (²⁷).

Beyond effects on scheduling and procedure rate, which directly modify when and how much medical providers work, details of any clinical AI’s implementation may contribute to improvements or declines in the work experience, as evidenced by existing instances of medical automation. The use of electronic health record systems is associated with lower physician satisfaction and increased burnout (²⁸), whereas improved electronic health record usability mitigates burnout (²⁹), implying the effects of AI on medical providers’ work experiences may similarly hinge not on the goal of an AI system, but rather its usability. AI voice recognition services may decrease costs relative to traditional dictation services but shift burdens from medical transcriptionists to physicians (³⁰), illustrating the organizational tradeoffs in implementing an AI system may also encumber clinicians. Automated dispensing systems in pharmacies may decrease job stress of pharmacy workers overall but elicit negative responses from pharmacy technicians who work most directly with these systems (³¹), illustrating how AI systems too could deteriorate the work experience of employees when the burdens of a system are shared unequally. These preliminary studies on the likely effect of AI on physician practice have strengthened the concern that AI systems may increase technological fatigue or inefficiently shift occupational burdens. Examining this issue on a more systematic basis, a recent review of AI systems for medical imaging found that although AI had potential for patient benefit, the vast majority of AI studies (>86%) were likely to increase physician workload due to an increase in both postprocessing and interpretation time (³²). Therefore, medical providers should consider an active role in guiding the introduction of these technologies to encourage an improved working environment.

What Changes Should Be Made To Maximize the Positive Effect of AI?

Clinical decision support systems that are under development frequently neglect clinical needs, and development of tools to address the most pressing clinical issues may require closer collaboration between medical professionals and AI researchers. For instance, an AI system for detecting skin cancer from photographs of a patient’s skin was hailed by its developers for achieving “dermatologist-level” performance (³³), but the absence of a commercial version of this system four years later suggests that accurate predictions may not equate to actionable information when bereft of a well-defined clinical role. Perhaps such an AI system could aid in screening, but crucial questions remain: who should be screened, and when? How should physicians incorporate additional information, such as lesion history and patient risk factors? What specific actions should a physician recommend in response to a positive screen? Ideally, translational questions such as these must be considered from the outset of the design process, when AI experts and clinicians collaborate to find the intersection of the technically feasible and medically useful. In another example, an AI-based screening system for diabetic retinopathy was deployed as part of a clinical trial, but patients opted out of the trial or did not follow up on the AI system’s recommendations, due to lack of transportation (¹²). Thus, even when a tool’s clinical role is well defined, neglect of patient factors may impair its utility. Given the increasing ease with which highly accurate ML systems are developed, these translational challenges may now be the bottleneck in the utility of a system.

Another opportunity for positive change is the improvement of safeguards that address the frequently “brittle” performance of modern AI that is high in some scenarios but unexpectedly and precipitously lower in others, which may threaten its utility or pose dangers to patients. Much of modern AI relies on ML, in which algorithms automatically infer patterns from large amounts of data, implying the quality of the system depends on the quality of the data. Oftentimes, performance is measured on a “held-out” portion of the training dataset, which exaggerates performance relative to prospectively collected test data, because ML is prone to “overfitting” and memorizing peculiarities of the training data that do not persist in prospective data (³⁴). For instance, multiple studies developing ML-based risk scores for AKI and for CKD progression only examine the score’s performance on held-out data from the same source as the training data (¹⁸,³⁵,³⁶), such that we would expect the true performance on data from an external (or prospective) source to fall considerably short of the reported values. Frequently, data may contain spurious associations that enable the AI system to learn “shortcuts” (³⁴): for example, AI may think a patient is sick simply because they come from a hospital where more patients are ill (³⁷,³⁸), or identify a pneumothorax on the basis of the chest tube that was placed to fix it (³⁹). Even if AI learns genuine, rather than spurious, associations, dataset shift between the population of patients used for training the algorithm and the population of patients for whom the tool is used in practice may degrade the model’s performance (⁴⁰). Training data may also poorly represent some subgroups, leading to inequitable performance. For instance, audits of commercial facial recognition classifiers revealed poor performance in individuals with darker skin (⁴¹), and longstanding disparities in melanoma care among patients with darker skin combined with high potential for racially biased training data have heightened concerns that similar biases may arise in clinical AI systems (⁴²). To safeguard against these idiosyncratic failure modes, regulatory bodies might require prospective evaluation or retrospective evaluation on “external” data, and principled subgroup analyses to avert reliance on shortcuts and ensure equitable performance.

Compounding the “brittleness” problem of AI, these systems may be challenging to audit, given their complex “black-box” nature, their frequently proprietary status, and the poor availability of training data. In principle, an understanding of an AI system’s decision process may enable users or auditors to predict failure modes, but many modern AI systems are far too complex to understand without specialized tools. Fortunately, numerous tools to understand complex AI systems, which are known as “explainable AI,” have been developed. These techniques range widely and include highlighting the important parts of an image, identifying which of a patient’s variables contribute most to a prediction, displaying similar cases from the training data, and showing altered, “counterfactual” versions of the input that would have produced different predictions. Explainable AI has been useful for auditing medical ML models, for example revealing that some medical imaging classifiers take the “shortcuts” mentioned above. Our review of FDA approvals identified infrequent use of explainable AI, and more widespread use before deployment may help safeguard against these failures.

Although difficulty understanding the complex decision process of AI systems may be mitigated with explainable AI, other factors that limit auditing of AI elude technological solutions. Commercial AI systems, which constitute valuable intellectual property, typically are not available to audit. If the medical community wishes to better ensure the robustness of medical AI systems by requiring their availability to independent auditors, regulatory action or concerted consumer pressure may be required. Even when AI is not guarded by business interests, patient privacy concerns may limit access to the raw data needed to audit models, predict failure modes, and improve AI. The vast libraries of data, such as electronic health records or medical images, used to train AI systems are frequently withheld even from academic publications where data deposition is otherwise the norm, due to institutional roadblocks or fears that patient privacy may be breached. However, the availability of a number of public datasets evidences that privacy concerns are surmountable; increased institutional willingness to publish raw data could go a long way toward academic research on the safety and reliability of medical AI models.

Outlook

Because AI methodologies are now maturing sufficiently to translate into the field of medicine, medical providers have a unique opportunity to shape their future. AI is a powerful tool for achieving accurate predictions, but to improve the lives of patients and their providers, clinical expertise is required to guide this technology toward the most exigent applications. Clinicians may involve themselves at each step of the process, by (1) collaborating with AI developers, (2) participating in clinical trials of promising AI, (3) critically evaluating the evidence behind AI devices before use, and (4) advocating for their patients and themselves at the administrative and regulatory levels. With thoughtful guidance, AI may transform medicine for the better.

Disclosures

All authors have nothing to disclose.

Funding

This work was funded by the National Science Foundation (CAREER DBI-1552309 to S.-I. Lee) and the National Institutes of Health (R35 GM 128638 and R01 AG061132 to S.-I. Lee).