paper
arXiv cs.LG
November 2nd, 2026 at 5:00 AM

Instrument-Anchored LLM Judges Transfer Expert Counseling-Quality Ratings Across Three German Corpora

arXiv:2610.08055v2 Announce Type: replace-cross Abstract: Automatic assessment of communication quality in counseling conversations is bottlenecked by data: expert-rated corpora are small and expensive to grow. Therefore we study cross-domain prediction of the expert's overall impression across three German simulated-counseling corpora from one university (two general-practice medical, one parent-teacher; $n=195$ sessions). Features are construct scores that four small LLM judges assign from the transcript, drawn from the experts' own rating instruments, plus a compact nonverbal block; all models are open-weight and run locally, preserving data protection. Trained on the other two corpora, this stack predicts the held-out corpus at Spearman $\rho = .533$; in-domain training on the same sessions falls about a fifth short ($.420$; paired gap $+.113$, 95% CI $[+.067, +.157]$), and the gap holds at matched training size and on identical test sessions. The signal lies in what is said: the four judges alone transfer at $.523$, and the nonverbal block adds $+.01$ (CI includes zero). Asking a judge for instrument constructs instead of generic dialogue qualities raises its transfer by about 30% ($.32$ to $.42$); restoring speaker attribution where per-speaker audio was lost is worth $+.08$ to $+.10$ on that fold. Where a program's own ratings are scarce, other programs' data is a viable training set and language carries it.

#ai
#llm
#research

Get stories like this every Monday

The week's most important AI news in one email. Free, unsubscribe anytime.

Score: 2.75

Engagement proxy: 0

Canonical link: https://arxiv.org/abs/2610.08055