What this project is about
FailureBench tests whether a model's predicted range contains the correct answer when input information disappears. This range is a prediction interval; the proportion of answers it contains is its coverage. Different missing-input patterns can cause errors on different examples, so testing patterns separately can hide failures that appear when the pattern changes between examples.
The benchmark asks whether the range works for each missing-input pattern, for a rule choosing the pattern, or for every allowed pattern on the same example. It uses five public tasks with numerical outcomes and two controlled simulations. The contribution is a reproducible evaluation using established conformal prediction methods to set interval widths, not a new theorem or state-of-the-art method.
How it works
- Disjoint fit, scale, calibration and test sets
- Enumerate feature-loss patterns
- Audit coverage and interval-width trade-offs
Separate learning, calibration and testing
Four separate groups of examples train the predictor, estimate differences in error size between patterns, choose the final interval threshold, and test performance. Only training data determine missing-value replacements, rescaling and model parameters. Copies with removed features stay with their original example. Exact duplicate input rows are reduced before splitting, with removals recorded.
Compare a defined set of missing inputs
The experiment tests complete inputs and the removal of one feature at a time, using the top two, top four or all features. Missing values are replaced with training-set medians, and the model is told which values are absent. Nine interval methods use the same predictions and data partitions. This keeps the underlying model fixed when comparing ways to calculate uncertainty.
Set widths using the worst error per example
Row-max records each calibration example's largest error across the allowed patterns. It sorts those errors and selects a threshold using a rank rule adjusted for sample size. Scaled row-max also accounts for differences in error size between patterns, estimated separately. Too little calibration data produces an unbounded interval. Saved predictions and file fingerprints support independent checks.
The important design choices
Require coverage across all allowed patterns
Count an example as covered only when every allowed interval contains its outcome. This is simultaneous coverage.
The trade-offThis protects choices among the declared patterns, assuming calibration and future examples can be treated as comparable draws. It can need wider intervals and gives no guarantee for other patterns or a changed population.
Use two inspectable model families
Compare ridge regression and histogram gradient boosting with fixed settings, ten prespecified data splits and two training approaches.
The trade-offThe experiment stays inexpensive and reproducible. It does not establish superiority over the strongest specialised methods for missing data.
Keep results that do not improve
Retain width increases, unbounded intervals and failures outside the declared missing-input patterns.
The trade-offThis prevents the useful width reductions from hiding cases where scaling helps little or makes the result worse.
Why checking each missing-input pattern is not enough
Imagine two missing-input patterns that cause intervals to miss the correct answer on different examples.
- Check each pattern separately and record which examples it fails on.
- For each example, choose the pattern that causes the failure; treat this as a stress test, not a claim about ordinary use.
- Compare the selected results with the stricter requirement that both intervals cover the same example.
What this showsSeparate averages can conceal different failures. Checking coverage across both patterns reveals the combined problem when the missing input can change between examples.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
The CPU run completed 280 model fits and 90,720 recorded metric combinations. Twenty integrity tests cover threshold calculations, tied errors, insufficient calibration data, separation of data groups and unchanged preprocessing. They also check a constructed example of failure under pattern selection and a simulation of the rank rule. Independent code recomputed 3,360 coverage-and-width comparisons from all 280 saved prediction files.
These combinations reuse examples and fitted models. Ten overlapping data splits show how results change with partitioning; they are not ten independent populations. Uncertainty ranges obtained by resampling those splits describe stability within the chosen benchmarks. Removing duplicates does not establish independence between physical specimens, and the tasks do not measure failures of deployed sensors.
| Interval method | Mean simultaneous coverage |
|---|---|
| Mask-matched calibration | 69.2% |
| Row-max calibration | 90.6% |
| Scaled row-max calibration | 90.6% |
What to take from the results
For boosting trained on complete inputs, calibrating each missing-input pattern separately covered the outcome under every allowed pattern on 69.2% of examples on average. Row-max and scaled row-max both achieved 90.6% against a 90% target. These averages give seven datasets equal weight. The average gap between the worst individual-pattern coverage and coverage across all patterns was 18.3 percentage points.
Scaling reduced interval width on five of seven primary datasets, including a 64.9% relative reduction on energy. It slightly worsened width on diabetes and did not meaningfully help red wine. Removing two features together also reduced coverage in several datasets: protection established for single-feature loss does not automatically extend to pairs.
What to keep in mind
- These are descriptive averages across fixed datasets, not population estimates. Repeated splits overlap.
- The protection is limited to a specified set of missing-data patterns under the method’s assumptions; unseen two-feature loss and distribution shift are not certified.
- The methods build on established conformal theory. This is a research artefact, not a novel-algorithm or published-paper claim.
Technical notes & source
Built with
Python · NumPy · SciPy · scikit-learn · conformal prediction
Terms used in this project
- Coverage
- How often a prediction range contains the correct answer.
- Simultaneous coverage
- Whether the range covers the answer for every allowed missing-data pattern on the same example.
- Calibration
- Using held-back examples to adjust predictions or uncertainty estimates.
Read the source material
Private repository. This page shares a summary, not private source code or raw research files.