What this project is about
TelemetryUQ asks whether a monitor can still predict an AI agent's failure when confidence measurements become unavailable. Recorded messages, tool calls and outcomes stay unchanged. Only the monitor's evidence changes: measurements indicating how uncertain the agent was when producing an answer or tool call.
The audit reuses 556 AgentUQ runs across 278 tasks, each attempted by two agent models in three simulated customer-service settings. The source authors generated those runs. This project prepares their data, removes selected measurements, compares forecasts and verifies calculations. It analyses existing runs using established methods; it is not a new agent or a deployed warning system.
How it works
- 556 source trajectories / 278 paired tasks
- Task-disjoint forecasts and paired masking
- Compare calibration, transfer and simple baselines
Check measurements against the recorded conversation
Preparation uses a fixed source release and verifies file fingerprints. It matches confidence measurements to messages, checks speakers and token counts, and reconstructs published summaries. Compact summaries allow local reruns without new model calls. Hidden reasoning text is excluded, although Kimi's numerical measurements can still reflect reasoning and formatting tokens.
Use only information available at the checkpoint
At fixed action checkpoints, the forecast combines visible interaction features with confidence summaries: how surprising generated tokens were and how uncertain the model's choices appeared. Data preparation is learned only from training tasks. Separate calibration tasks adjust the probabilities to match observed failure rates. Both agents' versions of a task stay in one data partition, preventing the same task from appearing in training and testing.
Hold the forecast model fixed when measurements disappear
The main test hides confidence measurements from messages containing tool calls. A random control hides the same number of measured messages. The original forecast model and probability adjustment stay fixed, isolating sensitivity to missing information. Other runs retrain for reduced measurements. Reserving a whole service domain for testing also examines transfer to an unfamiliar setting.
The important design choices
Predict the eventual outcome
Use the observed conversation so far to forecast whether its final benchmark reward falls below one, the study's failure definition.
The trade-offThis does not locate the first unsafe action. It applies to runs still continuing after the checkpoint, not every task or every moment.
Check useful probabilities and useful rankings
Report Brier score and log loss for probability error, AUROC for separating failures from successes, and both false alarms and the proportion of failures detected.
The trade-offGood ranking does not necessarily mean accurate probabilities or a useful warning threshold. The tested cutoff has no formal guarantee on future error rates.
Include a simple comparison after review
A separately labelled follow-up asks whether the agent model and service domain's training failure rates already predict as well as the richer monitor.
The trade-offThis analysis was added after inspecting results, so it is exploratory. It checks whether conversation features contribute beyond information already known about the setting.
Separate missing measurements from failed actions
Take an illustrative recorded conversation whose tool calls and final outcome are fixed.
- Forecast failure using the available confidence measurements.
- Hide measurements from messages containing tool calls without changing the conversation or reward.
- Apply the unchanged monitor and compare its probability with the original forecast.
What this showsA changed probability shows sensitivity to the monitor's information. It does not show that the agent acted differently or that a tool failed.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
Nineteen integrity tests check that forecasts exclude future messages, tasks stay grouped, data preparation uses training examples, and missing-measurement tests and statistics behave as intended. Source checks cover 556 runs and 12,192 measurement groups. Independent verification recalculates metrics from 302,200 saved prediction rows and separately checks the added simple baselines.
These rows reuse recorded runs; they are not independent agent experiments. Five overlapping task splits show sensitivity to partitioning. Resampling-based uncertainty ranges apply to the fitted model, selected test group and particular removal pattern. Matching the number of removed messages does not match their length: random removal hid 66.46% of available tokens on average, versus 53.36% for tool-message removal in the eligible Kimi cases.
| Calibrated predictor | Both-agent Brier | Kimi Brier |
|---|---|---|
| Original combined forecaster | 0.2269 | 0.1940 |
| Model/domain failure-rate baseline | 0.2125 | 0.1722 |
What to take from the results
After four actions, hiding tool-message confidence worsened the Kimi monitor's average probability error: Brier score rose from 0.194 to 0.246, where lower is better. Retraining and adjusting probabilities for those missing measurements achieved 0.208. This test used separate tasks from familiar domains, called the IID setting. Random removal had a similar effect; testing on unseen telecom tasks reversed the direction.
The simple baseline changed the interpretation. Using only model/domain failure rates achieved Kimi Brier 0.1722 versus 0.1940 for the richer monitor, and 0.2125 versus 0.2269 across both agents. The paired uncertainty ranges for split-11 include zero, so a wider performance advantage is not established. Added value from conversation features remains unproven; human-handoff benefits and saved inference costs were not measured.
What to keep in mind
- The five splits reuse tasks and are not independent populations.
- Random masking had similar effects, and the telecom transfer comparison reversed direction. A universal effect is not established.
- This is secondary analysis of existing runs, not a new agent system or a published paper.
Technical notes & source
Built with
Python · NumPy · SciPy · pandas · calibration · paired bootstrap
Terms used in this project
- Telemetry
- Measurements recorded while a system runs.
- Brier score
- A score for probability predictions that penalises confident mistakes; lower is better.
- Baseline
- A simpler reference method used to judge whether extra complexity helps.
Read the source material
Private repository. This page shares a summary, not private source code or raw research files.