Reduced my Jev judge’s calibration error [D]

I benchmarked Jev on TRIVIA+ dataset using an untouched 645-example test set.

Before calibration:
ECE: 0.0982
After learning from human-labelled examples:
ECE: 0.0313

That’s a 68.1% reduction in calibration error.
But hallucination-detection F1 only moved:
0.5833 → 0.5877

So what improved?
Not the judge’s ability to classify examples.
Its confidence became much more aligned with reality.

And that distinction matters.
If a judge score is only being displayed on a dashboard, maybe not much.
But if confidence > 0.8 means “automatically approve this answer,” or < 0.4 means “escalate to a human,” then badly calibrated scores can quietly become a production problem.

That’s one of the reasons I’ve been building calibration into Typed Evals instead of treating raw judge confidence as trustworthy by default.

Benchmark/methodology report:
https://github.com/TrustifAI/typed_evals/blob/main/docs/BENCHMARK.md

Would be interested to know how people here pick production thresholds for LLM/Jev judges.