The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models
arXiv:2603.09989v2 Announce Type: replace-cross Abstract: Large language models routinely produce factually incorrect or misleading outputs, yet conventional automated benchmarks overlook how users perceive reliability during interaction. We introduce the System Hallucination Scale (SHS), a lightweight instrument modeling the famous System Usability Scale (SUS) and System Causability Scales (SCS). SHS measures perceived reliability across five dimensions: factual accuracy, source reliability, logical coherence, deceptiveness of presentation, and responsiveness to user guidance, using ten balanced Likert-scale items that provide a directional score and consistency diagnostic. In a supervised validation study with 193 participants (386 evaluations across 47 trained experimenters), SHS demonstrated strong internal consistency and high clarity, completing in an average of 4.2 minutes. Participants rated source verifiability and presentation deceptiveness worse than factual accuracy, indicating perceived vulnerabilities lie primarily in traceability and framing. As a proof of concept, SHS assesses human-perceived reliability rather than ground-truth correctness, providing open-source multilingual implementations and an interactive calculator.
Get stories like this every Monday
The week's most important AI news in one email. Free, unsubscribe anytime.
Score: 2.94
Engagement proxy: 0
Canonical link: https://arxiv.org/abs/2603.09989