Artificial intelligence in clinical settings has long promised to reduce diagnostic error and ease cognitive burden on frontline physicians — but whether those promises survive contact with real emergency department conditions is another matter entirely. This prospective pilot study from a tertiary emergency department offers one of the most ecologically valid tests yet of a large language model (LLM) clinical decision support system operating in a high-stakes, high-volume care environment.
The study tracked the integration of the LLM decision support tool across four weeks in a live ED setting, monitoring both patient safety outcomes and clinician usage patterns. The headline safety finding was reassuring: the system showed no adverse safety signals during the pilot period, suggesting that, at minimum, LLM-generated clinical guidance did not introduce measurable harm when layered into existing workflows. However, the adoption trajectory told a more complicated story — clinician engagement with the tool declined steadily and consistently across the four-week window, raising substantive questions about real-world utility beyond controlled research conditions.
This pattern of initial uptake followed by erosion is well-documented in health informatics literature and is sometimes termed 'alert fatigue' or workflow friction — phenomena that have plagued earlier generations of clinical decision support tools, from rule-based systems to early machine learning alerts. What makes this finding particularly noteworthy is that LLMs were broadly anticipated to overcome these barriers through more naturalistic, conversational interfaces. The declining adoption curve suggests that interface design alone may be insufficient if the tool's outputs don't demonstrably alter or improve clinical decisions in real time. The study is a single-site pilot, limiting generalizability, and four weeks may be too short a window to distinguish true disengagement from onboarding variability. Nevertheless, for health systems investing in AI clinical infrastructure, this finding is a meaningful caution: safety is a floor, not a ceiling — sustained clinical value requires demonstrating workflow integration that physicians actually find indispensable.