Benchmarks

Every number on this page comes from a script in the experiments/ directory of the ARC repository, so you can re-run it. Results measured on ARC 5.2.0 come first. Older results come from ARC 5.1 code and are marked as such until they are re-run.

Fault-Injection Recovery (ARC 5.2.0)

Protocol: 5 seeds, CPU. Each trial trains the same MLP on the same data twice, once unprotected and once with arc.Controller, and injects the same fault at step 60 of 150. Both runs are judged by the same checks at the end: no exception, every weight finite, and held-out loss no more than 2× its pre-fault value.
Script: experiments/recovery_benchmark.py

GPU Overhead (ARC 5.2.0)

Protocol: GPT-2 (124M, random init), batch 4, sequence 128, AdamW. 400 steps in interleaved blocks of 50, with and without ARC, after 20 warm-up steps each. The figure is the median step time.
Script: python experiments/controller_overhead.py --steps 400 --batch 4 --seq 128

Snapshots are copied to host memory on a background thread right after optimizer.step(), so the copy overlaps the next forward and backward pass. Gradient and weight checks use one host sync per step. ARC 5.1 measured 40.7% (GPT-2) and 33.4% (GPT-2 Medium) on a Tesla T4 with a different engine and settings, so those numbers aren't directly comparable. A100, H100, and multi-billion-parameter runs have not been profiled yet.

Results from ARC 5.1 (pending re-measurement)

Baseline Comparison

Protocol: 4 methods × 5 failure types × 5 seeds = 25 scenarios per method.
Script: experiments/baseline_comparison.py

Detection Rate

Billion-Scale Rollback (H200)

Protocol: NVIDIA H200, PyTorch 2.12. A 40 to 50 step fine-tune with one NaN loss injected at step 30, using the legacy WeightRollback. Each model is run healthy, unprotected with the failure, and ARC-protected with the failure.
Script: experiments/billion_scale_benchmark.py

The unprotected run is stopped at the NaN by the script. This shows that rollback fits and runs at 2.65B parameters on one GPU. It is not a recovery rate.

Real-Dataset Recovery

These experiments check whether a recovered model reaches the same quality as one trained without interruption. All ran on an NVIDIA Tesla T4.

CIFAR-10, ResNet-18 (3 seeds)

10 epochs of SGD. A NaN loss was injected at epoch 5, step 100.

GPT-2 (124M) Fine-Tuning on WikiText-2

300 steps with AdamW. A NaN loss was injected at step 150.

Gradient Explosion, ResNet-18 on CIFAR-10 (lr = 10, 5 seeds)

Realistic Training Bugs

Protocol: ResNet-50 on CIFAR-100 for 10 epochs on a CUDA GPU. Each run injects a common real-world bug instead of a synthetic NaN. The healthy run reaches 32.66% top-1 accuracy.
Script: experiments/industrial_benchmark.py

In the same benchmark, unprotected GPT-2 and BERT fine-tuning runs crashed mid-training. The ARC-protected runs completed with one rollback each.

Organic Failures

Protocol: 8 failures caused by real training conditions. None of them assigns NaN to the loss directly. Each runs for 60 steps with and without ARC. Data and loss faults start at step 20 and persist. Measured on ARC 5.1.2, CPU.
Script: experiments/organic_failure_test.py

Large-Model Stress Tests

Protocol: 20 stable steps, then one catastrophic failure, with recovery checked within 10 steps.
Script: experiments/validate_claims_phase2.py

See Billion-Scale Rollback for models from 355M to 2.65B parameters.

Failure Prediction (research)

Protocol: 4 architectures × 5 failure types × 5 seeds × 2 labels = 200 simulated training curves, 5-fold cross-validation.
Script: experiments/prediction_200_v2.py

v1 to v2 Improvement

Lyapunov Early Warning

Protocol: The learning rate ramps linearly from stable to explosive, and we record the step at which each detector first fires. Averaged over 5 seeds. Measured on ARC 5.1, before velocity was normalized by learning rate.

The lead is largest under gradual destabilization, which is the hardest case for threshold-based detectors. These lead times have not been re-measured on 5.2.

Fisher-Aware vs. Naive Rollback

Protocol: CNN on CIFAR-10. The top third of parameters by Fisher importance are corrupted, then recovered with either a uniform rollback or a Fisher-weighted rollback. Test accuracy is measured after 40 recovery steps and averaged over 5 seeds.

Ablation Study

Protocol: ResNet-18, 5 seeds, ARC 5.1. The metric is the total number of rollbacks needed to survive each scenario, so lower means more efficient recovery. The three scenarios are (A) a gradient explosion every 10 steps, (B) 60% of weights zeroed at step 25, and (C) a NaN every 5 steps for the whole run. Every configuration survives every scenario.

Overhead Breakdown (ARC 5.1, CPU)

Protocol: Median of 100 iterations, time.perf_counter, on a 288K-parameter CNN.
Script: experiments/overhead_measurement.py

Added Time by Model Scale (CPU)

ARC's added time per step next to the baseline step time.

Relative overhead falls as models grow, because monitoring cost grows more slowly than the forward and backward passes. 5.2 fuses the weight checks into one host sync, so the weight-statistics share above is an upper bound.

Known Limitations