Reinforcement Learning for Calibrated Decisions (RLCD)

Canonical version: Reinforcement Learning for Calibrated Decisions (RLCD).

RLCD is how TypeSafe AI says it trains Jev. The name stands for Reinforcement Learning for Calibrated Decisions, and the pitch fits in one line from their docs: RLCD "trains TypeSafe to return decisions and calibrated probabilities instead of generated text."

The goal is what they call "epistemically honest probabilities". When the model says 0.8, the thing should happen about 80% of the time. Not "the model sounds confident", not "the answer is usually right", but the NUMBER itself is trustworthy. The launch post has the sentence that explains why they care: "If a model can do a task 95% of the time but doesn't say when it's in the 5%, it can't automate that task."

TypeSafe hasn't published how RLCD works. No paper, no reward function, no reliability diagram. Almeida calls it a "novel, unpublished technique." So this note separates what's known from what people guess.

The third branch of post-training

TypeSafe's primer presents three ways to post-train a pretrained model:

• RLHF "turned pretrained models into chatbots. It trains models to produce responses people prefer"
• RLVR "created reasoning models that are strong at tasks such as mathematics, but slower and more expensive"
• RLCD returns "decisions and calibrated probabilities instead of generated text"

The launch post's comparison table sums up the contrast: existing LLMs are optimized with RLHF/RLVR for "human preference: writeups and chat responses that human raters prefer", output strings token by token, and tend "toward overconfidence". Jev is optimized with RLCD for "calibrated decisions", outputs "type-safe structured values" with probabilities, and samples everything in parallel in one query.

RLCD names a target

This is the most important thing to understand, and it comes from Almeida himself on the Latent Space podcast (21 September 2026). When he says RLHF, he means the task of instruction following: "It's not about the PPO. That part doesn't matter." DPO "also does RLHF, despite not using the algorithm." RLCD is framed the same way: "RLCD is this new task... here's another North Star."

So don't picture a clever new variant of PPO. Picture a different answer to the question "what are we rewarding?". RLHF rewards approval. RLVR rewards correctness. RLCD rewards honest probabilities over a bounded, typed answer space.

What we actually know

• The output contract. No text. The model returns decisions (a choice among up to 255 options, a yes/no probability, a score) with probabilities. Higher probability should mean a higher chance of being right, measured over groups of predictions, never as a promise about one answer
• The data is 100% synthetic and task-dependent (Almeida, Latent Space). They deliberately avoid user data so the model doesn't overfit to "the present day". Customer data isn't used for training, and there's no fine-tuning or LoRA for customers. See Synthetic Data
• Data over compute. "Data matters way more than compute." "Data is so unbelievably complicated, and that is what gets nines." TypeSafe calls itself "a data lab, not a model lab", and a data team hunts down and "surgically" fixes jaggedness in the model's abilities (Jagged Intelligence)
• Typed outputs on purpose. Almeida says calibration is "total poison into the probability distributions of strings". My reading: calibrating a distribution over all possible sentences is poorly defined, so they constrain the answer space first and calibrate inside it
• Consistency over determinism. There's no seed parameter. The target is similar outputs for semantically similar inputs (they test it by injecting random UUIDs into prompts)

What people guess (all speculative)

• Proper scoring rules. The obvious candidate is to reward the model with a strictly proper scoring rule (log loss or Brier score) on its output distribution. Those are the only rewards where reporting your true belief is the best strategy. Archer Hume guessed this after ~10,000 API calls; he also notes you can't tell training-time calibration from post-hoc temperature scaling through the API. [SPECULATIVE] See Proper Scoring Rules
• Plackett-Luce plus Brier. Di Zhang hypothesizes a "schema-conditioned Plackett-Luce objective" (a multi-way generalization of the Bradley-Terry model used in RLHF reward models) plus a Brier calibration term, i.e. the reward model becomes the product. [SPECULATIVE]
• Why "RL" at all? If you enumerate the answers and know the outcomes, you can minimize a proper scoring rule with plain supervised cross-entropy. Calling it RL may mean the targets come from a process (synthetic environments, programs in the loop), or it may be positioning. Almeida's "the task, not the algorithm" framing is consistent with both. [SPECULATIVE]

Prior art

Training models for calibration isn't new, and I think it's fair to say so:

• RLCR (Damani et al., MIT, arXiv:2507.16806, July 2025) adds a Brier score term to a binary correctness reward for reasoning models, and improves calibration with no loss in accuracy
• Rewarding Doubt (Bani-Harouni et al., TU Munich, arXiv:2503.02623) turns confidence into a betting game rewarded with the logarithmic scoring rule
• Kadavath et al. (Anthropic, arXiv:2207.05221, 2022) trained a "P(IK)" head, a scalar probability that the model knows the answer
• Leng et al. (arXiv:2410.09724) showed RLHF reward models are biased toward high-confidence answers

The difference, as far as anyone can tell: RLCR and Rewarding Doubt calibrate a number written inside generated text. Jev appears to calibrate its native output distribution over typed answers. That's closer to training a classic classifier with proper scoring rules than to anything chat-shaped.

Watch out for a name collision too: "RLCD" already meant Reinforcement Learning from Contrastive Distillation (Yang, Klein, Celikyilmaz, Peng and Tian, arXiv:2307.12950, ICLR 2024), an unrelated alignment method that builds preference pairs from contrasting prompts. Search results mix the two up.

How well does it work?

Independent numbers are mixed, and calibration is always relative to a data distribution:

• Archer Hume: expected calibration error of 0.031 on 1,200 MMLU items. Slightly overconfident at the top (predicted 98.7%, observed 96.3%), bigger gaps in the middle bins
• Rajesh Beri: ECE 0.107 on synthetic support tickets. Noul (yes/no) answers were underconfident; Choice and Score answers overconfident. On an unknowable task, Jev was right 44.7% of the time at a mean probability of 0.74
• Turing Post noticed that TypeSafe's own adapter returns different confidence values than the hosted API

TypeSafe itself publishes no calibration curve, uses its own "workflow evals" where the reference answers are the average of GPT-6 Astra and Claude Fable 5.1, and Almeida is "extremely anti-public benchmarks". The docs do list the limits, though. The Jev 1.13 jaggedness page says Score levels are "weak in numerical calibration", and a question and its negation asked as two separate Nouls can sum to 1.19.

Criticism

• Unfalsifiable as shipped. No paper, no ablation, no ECE, terms that forbid public benchmarking. You're asked to trust the word "calibrated"
• Calibration doesn't compose. Anthony Maio's point: "Individually calibrated judgments do not automatically compose into a calibrated workflow once you run them through thresholds, weights, and branches." And distribution shift can "quietly undermine probabilities that were reliable last month"
• Architecture or algorithm? Maio thinks the strongest case for Jev is "architectural rather than algorithmic". I mostly agree: typed answer spaces, shared-state batching and synthetic data per task shape could explain a big share of the results without any exotic RL

My take

I think RLCD's intellectual core is old and solid: reward probabilities with a proper scoring rule (Brier did it for weather forecasts in 1950). The genuinely new part is the product shape around it, and the courage to stop optimizing for human approval. The evidence that preference training damages calibration comes from OpenAI and Anthropic themselves (see AI Model Calibration), so the premise is well grounded.

But "calibrated" is a claim about a distribution, and the distribution that matters is YOURS. Before trusting any Jev threshold, I'd measure a reliability curve on a few hundred of my own labeled examples, or put a conformal layer on top (see routing on confidence). Trust, then verify.

References

• Introducing System One Models & Jev (TypeSafe AI)
• TypeSafe AI primer
• Jev 1.13 jaggedness (TypeSafe docs)
• Jev: System One models for Prod, not God, with Diogo Almeida (Latent Space)
• Jev's Architecture Unmasked (Archer Hume)
• TypeSafe Jev: calibration, decomposition, shadow eval (Rajesh Beri)
• Jev, the language model that won't talk (Anthony Maio)
• What is RLCD? (Di Zhang)
• What is Jev and RLCD (Turing Post)
• RLCR (arXiv:2507.16806)
• Rewarding Doubt (arXiv:2503.02623)
• Language Models (Mostly) Know What They Know (arXiv:2207.05221)
• Taming Overconfidence in LLMs: Reward Calibration in RLHF (arXiv:2410.09724)

Related

• Jev
• TypeSafe AI
• Diogo Almeida
• AI Post-Training
• Reinforcement Learning From Human Feedback (RLHF)
• Reinforcement Learning with Verifiable Rewards (RLVR)
• AI Model Calibration
• Proper Scoring Rules
• Mode Collapse
• System One Models
• Decision Models (DMs)
• Confidence-Gated Routing
• Synthetic Data

About Sébastien

Ready to get to the next level?