paper
arXiv cs.AI
October 31st, 2026 at 4:00 AM

Benchmarking decision models against generative large language models on medical multiple-choice questions

arXiv:2610.38448v1 Announce Type: cross Abstract: Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.

#ai
#llm

Get stories like this every Monday

The week's most important AI news in one email. Free, unsubscribe anytime.

Score: 2.94

Engagement proxy: 0

Canonical link: https://arxiv.org/abs/2610.38448