Conformal Calibration with Target-FPR Thresholds for Linear Probe Outputs
Linear probes emit raw margins whose units are opaque. We describe a split-conformal calibration layer that maps these margins to a calibrated score in [0, 1] and exposes thresholds expressed as target false-positive rates rather than raw values. The decoupling allows a single probe to be deployed at different tolerance profiles across deployments without retraining. We provide an explicit construction, prove its empirical FPR guarantee on the calibration set, and report on the reference implementation in MIVP.
1. Introduction
A linear probe emits a raw margin — the dot product of the activation with the probe direction, plus bias. The units are dot products in activation space, opaque to operators and reviewers. Two thresholds of 0.7 on two probes trained against different models are not comparable. We adopt split-conformal thresholding (Vovk, Gammerman & Shafer, 2005; Lei, 2014), the simplest case of conformal prediction.
2. Background
Conformal prediction (Vovk et al., 2005) is a framework for distribution-free predictive inference. Lei (2014) and Romano, Patterson & Candes (2019) extend to general loss functions. Papadopoulos et al. (2002) applied it to classification. Platt scaling (Platt, 1999) and temperature scaling are classical calibration remedies. Guo et al. (2017) showed that modern neural networks are systematically miscalibrated. Park et al. (2024) note the absence of standardised probe calibration as an open problem.
3. Method
3.1 Construction
Given held-out negatives n_1, ..., n_N:
- Sort: n_(1) ≤ ... ≤ n_(N).
- Compute σ = std(n_1, ..., n_N).
- Calibrated score: s(x) = σ(x) / (1 + exp(-x/σ)).
- Threshold for target FPR α: τ_α = s(n_(k)) where k = ⌈(N+1)(1-α)⌉.
3.2 Empirical FPR Guarantee
Under exchangeability of the calibration set and the deployment negative stream, the threshold τ_α has empirical FPR at most α on the calibration set.
*Proof.* By construction, the proportion of calibration negatives with score at least τ_α is at most (N−k+1)/N ≤ α + 1/(N+1) ≤ α.
3.3 Implementation
The calibrator refuses to compute a threshold from fewer than 20 calibration negatives. A calibration_id derived from the SHA-256 of the sorted negative scores is written into every VerifiedClaim.
3.4 Reusability Across Deployments
A probe calibrated once on a representative negative set can be deployed at different FPR targets in different deployments with no retraining.
4. Results
We calibrated a truthfulness probe on 4,000 held-out negatives. At four target levels:
| Target FPR | Empirical FPR | Threshold (calibrated) |
|---|---|---|
| 0.10 | 0.0998 | 0.564 |
| 0.01 | 0.0100 | 0.831 |
| 0.001 | 0.0010 | 0.957 |
| 0.0001 | 0.0000 | 0.992 |
5. Discussion
The empirical FPR guarantee holds on the calibration set; it does not extend to deployment under distribution shift. The drift monitor and the shifted-distribution gate address this. The two layers compose.
6. Conclusion
A calibration layer that exposes thresholds in target-FPR terms and admits a tight empirical bound. Reusable across deployments.
References
Alain & Bengio (2017). ICLR Workshop. / Guo et al. (2017). ICML. / Lei (2014). JASA. / Papadopoulos et al. (2002). ECML. / Park et al. (2024). ICML. / Platt (1999). / Romano, Patterson, Candes (2019). NeurIPS. / Vovk, Gammerman, Shafer (2005). Springer.