---
res:
  bibo_abstract:
  - Typed decision models (TDMs) return a probability for each allowed answer instead
    of generating text, which makes them candidates for cheaply replacing LLM judges
    that compare two answers. Whether they share the biases of LLM judges is untested.
    We run the CALM benchmark's bias tests on eight TDMs and seven LLM judges of 2024
    and 2026. Every TDM is biased, in three to eight of twelve tests. Emotional tone
    affects them most, and rewriting the better answer in such a tone lowers the accuracy
    of a typical TDM from 99% to 42%. TDMs are more biased than 2026 LLM judges. All
    eight change verdicts when the answers swap places, which the 2026 LLM judges
    hardly do, and several side with an answer that most people allegedly prefer or
    that a famous model allegedly wrote. Yet both kinds of judge are tripped up by
    the same prompts. The TDMs' probabilities show that most tests mainly flip verdicts
    the TDM was unsure of, and that good calibration does not protect against bias.
    As TDMs are more consistent and 37 times cheaper than 2026 LLM judges, our results
    suggest they can be attractive judges if their biases are tested for the task.@eng
  bibo_authorlist:
  - foaf_Person:
      foaf_givenName: Fabian
      foaf_name: Lukassen, Fabian
      foaf_surname: Lukassen
  - foaf_Person:
      foaf_givenName: Christoph
      foaf_name: Weisser, Christoph
      foaf_surname: Weisser
      foaf_workInfoHomepage: http://www.librecat.org/personId=264138
    orcid: 0000-0003-0616-1027
    orcid_put_code_url: https://api.orcid.org/v2.0/0000-0003-0616-1027/work/230070257
  - foaf_Person:
      foaf_givenName: Alexander
      foaf_name: Silbersdorff, Alexander
      foaf_surname: Silbersdorff
  dct_date: 2026^xs_gYear
  dct_language: eng
  dct_title: Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges@
...