Locally Deployed Large Language Models for AI-Assisted Outpatient Prescription Review: Crossover Study

The useful part

Pharmacists' prescription review is key to medication safety, but rising numbers of outpatient prescriptions and expanding formularies leave less time per case, increasing error risk. This study aimed to evaluate the feasibility and usefulness of a locally deployed, knowledge-augmented LLM as a decision-support tool for pharmacist-led outpatient prescription review. The open-source Qwen3-14B model was deployed on a hospital intranet server using the Ollama framework.

How it works

• A 2-period crossover design was used: 2 pharmacists independently reviewed 213 outpatient prescriptions under both unaided and AI-assisted conditions, yielding paired unaided and collaborative results for...
• Plain AI review and knowledge-augmented AI review were also run on the same 213-prescription test set to quantify hallucination rates, and stand-alone AI performance was assessed against the reference standard.
• The reference standard was established by independent consensus between two supervising pharmacists, with disagreements adjudicated by a deputy chief pharmacist.
• Accuracy, sensitivity, and specificity were compared between conditions using paired McNemar tests, and review time was compared using the Wilcoxon signed-rank test.