Decomposing Yes-No Bias in LLM Moral Judgment: Stance, Position, Word, and Verdict
arXiv:2607.05552v2 Announce Type: replace Abstract: Large language models (LLMs) increasingly issue binary verdicts - safety gates, judge decisions, value probes - and the answer is read as the model's judgment. A settled judgment should be coherent across paired phrasings of the same question: yes to "do you approve?" should mirror no to "do you oppose?". LLMs often violate this coherence, and in a yes/no question the verdict, the answer word, and the printed position coincide, so a conventional flipped-question audit yields one contrast that cannot say what moved. We introduce a crossed psychometric battery that extends the flip to full 2x2 and 2x2x2 crossings of question polarity, printed order, and answer label, separating stance from position, word, label, and verdict-role effects; an independent graded instrument estimates each model's stance $\theta$ with no binary format; and a two-parameter fit (framing susceptibility $m$, decisiveness $s$) separates robustness from saturation that hides susceptibility. Applying the battery to twenty moral dilemmas - 1.35 million trials over ten frontier models from the Claude, GPT, and Gemini families in 23 reasoning configurations - we find response-form bias widespread but never uniform. The largest apparent yes-no biases are almost purely surface (word and position), while several of the newest models show the reverse: near-zero flip contrasts concealing verdict attachment up to +0.42, a component no surface counterbalance can expose, replicated across label alphabets. Requested reasoning reshapes these channels rather than removing them. When subjects state decisions in free text and a separate model transcribes them, two judge families agree on 99.3% of jointly scored transcriptions with zero opposing committed labels, and the transcribed stance converges with the graded ruler (median $r=0.87$): response form is part of the instrument, and the judgment can be measured apart from it.
Get stories like this every Monday
The week's most important AI news in one email. Free, unsubscribe anytime.
Score: 2.88
Engagement proxy: 0
Canonical link: https://arxiv.org/abs/2607.05552