---
_id: '7217'
abstract:
- lang: eng
  text: Typed decision models (TDMs) return a probability for each allowed answer
    instead of generating text, which makes them candidates for cheaply replacing
    LLM judges that compare two answers. Whether they share the biases of LLM judges
    is untested. We run the CALM benchmark's bias tests on eight TDMs and seven LLM
    judges of 2024 and 2026. Every TDM is biased, in three to eight of twelve tests.
    Emotional tone affects them most, and rewriting the better answer in such a tone
    lowers the accuracy of a typical TDM from 99% to 42%. TDMs are more biased than
    2026 LLM judges. All eight change verdicts when the answers swap places, which
    the 2026 LLM judges hardly do, and several side with an answer that most people
    allegedly prefer or that a famous model allegedly wrote. Yet both kinds of judge
    are tripped up by the same prompts. The TDMs' probabilities show that most tests
    mainly flip verdicts the TDM was unsure of, and that good calibration does not
    protect against bias. As TDMs are more consistent and 37 times cheaper than 2026
    LLM judges, our results suggest they can be attractive judges if their biases
    are tested for the task.
author:
- first_name: Fabian
  full_name: Lukassen, Fabian
  last_name: Lukassen
- first_name: Christoph
  full_name: Weisser, Christoph
  id: '264138'
  last_name: Weisser
  orcid: 0000-0003-0616-1027
  orcid_put_code_url: https://api.orcid.org/v2.0/0000-0003-0616-1027/work/230070257
- first_name: Alexander
  full_name: Silbersdorff, Alexander
  last_name: Silbersdorff
citation:
  alphadin: '<span style="font-variant:small-caps;">Lukassen, Fabian</span> ; <span
    style="font-variant:small-caps;">Weisser, Christoph</span> ; <span style="font-variant:small-caps;">Silbersdorff,
    Alexander</span>: <i>Cheap, Consistent, Calibrated, but Biased? Typed Decision
    Models as Judges</i>, 2026'
  ama: Lukassen F, Weisser C, Silbersdorff A. <i>Cheap, Consistent, Calibrated, but
    Biased? Typed Decision Models as Judges</i>.; 2026.
  apa: Lukassen, F., Weisser, C., &#38; Silbersdorff, A. (2026). <i>Cheap, Consistent,
    Calibrated, but Biased? Typed Decision Models as Judges</i>.
  bibtex: '@book{Lukassen_Weisser_Silbersdorff_2026, title={Cheap, Consistent, Calibrated,
    but Biased? Typed Decision Models as Judges}, author={Lukassen, Fabian and Weisser,
    Christoph and Silbersdorff, Alexander}, year={2026} }'
  chicago: Lukassen, Fabian, Christoph Weisser, and Alexander Silbersdorff. <i>Cheap,
    Consistent, Calibrated, but Biased? Typed Decision Models as Judges</i>, 2026.
  ieee: F. Lukassen, C. Weisser, and A. Silbersdorff, <i>Cheap, Consistent, Calibrated,
    but Biased? Typed Decision Models as Judges</i>. 2026.
  mla: Lukassen, Fabian, et al. <i>Cheap, Consistent, Calibrated, but Biased? Typed
    Decision Models as Judges</i>. 2026.
  short: F. Lukassen, C. Weisser, A. Silbersdorff, Cheap, Consistent, Calibrated,
    but Biased? Typed Decision Models as Judges, 2026.
date_created: 2026-10-15T07:09:21Z
date_updated: 2026-10-15T08:13:47Z
language:
- iso: eng
status: public
title: Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges
type: working_paper
user_id: '261203'
year: '2026'
...