Accurate extraction of patient-reported outcomes from clinical records is a bottleneck that quietly shapes the quality of every retrospective health study — and by extension, the clinical guidelines that follow. When that extraction is inconsistent or slow, the downstream science suffers. A new concordance study probes whether large language models can perform this foundational task as reliably as trained human reviewers, without custom training on clinical data.
The retrospective study examined 26 patients who received low-dose radiation therapy for osteoarthritis. Both human reviewers and a HIPAA-compliant generative pretrained transformer (GPT-based) AI system independently extracted pain scores using two validated instruments: the Numeric Rating Scale (NRS, 0–10) and the von Pannewitz Score (VPS, 0–4) at baseline, end of treatment, and first follow-up. Concordance metrics included exact match rates, intraclass correlation coefficients for the NRS, and weighted Cohen's kappa for the VPS. The AI operated without any fine-tuning on study-specific data, reflecting how such tools would realistically be deployed. Abstraction time for each method was also recorded and compared.
The study's most consequential contribution is not that AI performed well — that has been shown in larger, better-resourced settings — but that an off-the-shelf, privacy-compliant language model achieved meaningful concordance without domain-specific training. This matters enormously for smaller research teams and community health settings that cannot afford bespoke AI development. The critical limitation here is the small cohort of 26 patients, which constrains statistical power and generalizability. Osteoarthritis pain documentation may also follow relatively formulaic clinical note patterns, potentially making extraction easier than in more complex or ambiguous clinical contexts. This should be viewed as a proof-of-concept rather than a practice-ready validation. As AI abstraction tools scale, human oversight will remain essential for edge cases — but this incremental evidence nudges the field toward hybrid workflows that could meaningfully accelerate real-world outcomes research.