itsloganmann/rare-failure-eval
Reproducible agent-evaluation experiment: rare failures, selection errors, and regret under fixed budgets - itsloganmann/rare-failure-eval
0 stars · 0 forks
README
Rare Failure Eval An agent evaluator can make more wrong picks and still lose less utility. A reproducible Python experiment on rare failures, fixed evaluation budgets, and the gap between selection accuracy and the cost of a mistake. Results · Full comparison · Protocol · Raw data
Results Same budget. Same tasks. Same 1,024 simulated worlds.
At 25,600 draws per policy per world Uniform, full budget Variance allocation
Concentrated risk: wrong picks 455 / 1,024 539 / 1,024
Concentrated risk: mean utility lost 0.0268 0.0193
Diffuse risk: wrong picks 541 / 1,024 602 / 1,024
Diffuse risk: mean utility lost 0.0208 0.0229
Report both frequency and cost of mistakes. Under concentrated risk, the variance-based policy made more wrong picks but lost less utility on average. Under diffuse risk, it did worse on both metrics. This is a tradeoff inside a synthetic generator, not a general win for adaptive evaluation. Try it Python 3.13. CPU only. No model API, credentials or private data. git clone https://github.com/itsloganmann/rare-failure-eval.git cd rare-failure-eval python3.13 -m venv .venv .venv/bin/python -m pip install -r requirements.txt .venv/bin/python run_experiment.py --mode smoke --out outputs/smoke Read outputs/smoke/REPORT.md to inspect the smoke run. Use a new output directory for each run. The published findings come from the full run, not smoke mode.
Tests, full replication and independent verification .venv/bin/python -m pytest -q .venv/bin/python run_experiment.py --mode primary --approved-primary --out outputs/replication .venv/bin/python run_experiment.py --verify --out outputs/replication .venv/bin/python independent_audit.py outputs/replication .venv/bin/python make_figures.py outputs/replication/summary.json --out outputs/figures The primary flag is an execution gate, not evidence of peer review. To verify the distributed run without resampling: mkdir -p outputs/published cp results/*.json results/REPORT.md outputs/published/ gzip -dc results/results.jsonl.gz > outputs/published/results.jsonl .venv/bin/python run_experiment.py --verify --out outputs/published .venv/bin/python independent_audit.py outputs/published The manifest hashes the frozen study code and outputs. The separate arithmetic audit reconstructs weighted estimates, rankings, ties, selection error and regret from saved sufficient statistics.
What is inside Four synthetic agents, eight tasks, three risk settings, two budgets and four sampling policies. Utility is reward - 800 * failure, using fixed task weights. Regret is the true best agent's expected utility minus the selected agent's.
Policy Allocation
Uniform, full budget Every draw goes into balanced estimation
Uniform, pilot matched A pilot followed by balanced independent estimation
Variance allocation A four-draw-per-cell pilot sets a frozen Neyman allocation, with a uniform floor
Follow pilot failures Allocate toward observed failures; otherwise use uniform sampling
The full run contains 24,576 records and 393,216,000 simulated draws. All conditions · Paired intervals · Audit Scope
Synthetic, exploratory replication. Fresh seeds after a 128-world pilot; not publicly preregistered and not a real-agent benchmark. No universal improvement claim. Results depend on setting and budget. Risk settings do not match total event mass, so their difference is not causal. Rare failures are easy to miss. Four draws detect a 0.001-probability event only 0.4% of the time. No observed failures does not establish safety. Expected regret is not a safety constraint. Paired bootstrap intervals are descriptive and unadjusted. One-look risk bounds are not anytime guarantees.
Background Inspired by Active Evaluation of General Agents, Lanctot et al. (2026). This independent experiment studies rare-loss utility and sampling allocation; it does not reproduce that paper's Elo or Soft Condorcet algorithms. No employer data or institutional endorsement is involved.