As AI tools become routine fixtures in clinical workflows, the question of which systems can actually be trusted at the point of care is no longer academic — it is a patient safety issue. A direct head-to-head evaluation of three AI platforms across obstetrics, gynecology, and urogynecology reveals meaningful performance gaps that carry real implications for how clinicians should approach AI-assisted decision-making.
The study applied a validated quality-assessment instrument — the Expert-Adapted DISCERN (EA-DISCERN) tool — to score responses from ChatGPT (GPT-5), Gemini (Gemini 2.5), and OpenEvidence across 24 structured clinical questions, each rated by two blinded OB-GYN specialists across 12 quality domains on a five-point scale. OpenEvidence, a retrieval-augmented generation (RAG) platform that grounds responses in indexed clinical literature, achieved a mean total score of 54.0 out of a possible 60, versus 50.3 for Gemini and 48.7 for ChatGPT — statistically significant differences (p < 0.001). Critically, the gaps were most pronounced in domains directly tied to clinical risk: guideline consistency, evidence transparency, and completeness.
This finding matters because it exposes a structural limitation of purely generative large language models when applied to high-stakes clinical contexts. Standard LLMs like ChatGPT and Gemini are trained on broad corpora with knowledge cutoffs and no live tethering to clinical guidelines, making them susceptible to confident-sounding but outdated or unanchored responses. RAG architectures partially mitigate this by retrieving and citing current evidence before synthesizing an answer. The study is limited by its cross-sectional design, a relatively small question set, and evaluation within a single specialty cluster, so generalizability to primary care or other high-complexity domains remains untested. Gemini's intermediate ranking over ChatGPT in rationale explanation suggests that frontier generalist LLMs are improving, but the performance ceiling for guideline-consistent clinical reasoning may require purpose-built retrieval infrastructure. This work is incremental but practically useful — it offers clinicians a framework for differentiating AI tools by architecture, not just brand.