ARC detects training failures, rolls back to the last healthy state, and keeps training. One controller, one call per step. In a symmetric fault-injection benchmark it recovered all 70 transient faults, where unprotected training recovered from 20.
pip install arc-training
import arc
controller = arc.Controller(model, optimizer)
for batch in dataloader:
optimizer.zero_grad()
loss = model(batch)
loss.backward()
action = controller.step(loss)
if action.should_skip:
continue
optimizer.step()A single NaN gradient at hour 47 of a 48-hour run can destroy days of compute. Engineers waste enormous time adding manual checkpointing and babysitting long runs.
The loss becomes NaN or Inf with no warning and corrupts the run.
Vanishing or exploding gradients stop learning without any alert.
Every failure you can't recover from burns GPU-hours you already paid for.
Engineers watch runs overnight and write their own recovery scripts and checkpointing logic.
Checks every step for NaN/Inf, loss explosions, gradient explosions, and weight corruption. Thresholds auto-calibrate to your model during the first 50 steps.
Lyapunov stability and trend-drift detectors flag rising instability before loss-based checks fire. Velocity is measured per unit of learning rate, so a warmup schedule isn't mistaken for divergence.
Restores model, optimizer, and RNG state from the last validated snapshot, then cuts the learning rate. A failure counts as recovered only after 10 healthy steps. If a fault keeps coming back from the data, ARC says so and marks the run unrecoverable instead of looping.
One entry point, arc.Controller, replaces five legacy APIs. Rollback, learning-rate backoff and recovery, scheduler sync (including OneCycleLR), GradScaler support, and resumable checkpoints.
Online, learning-rate-normalized Lyapunov exponent estimation on a fixed parameter sample, with FFT oscillation detection.
Sliding-window regression on loss, gradient, and weight norms catches slow numerical drift that threshold checks miss.
Learns loss and gradient thresholds from your model's first 50 steps, so you never have to guess what "too high" means.
When a fault lives in the data (a NaN row, log(0), a zero-variance batch), ARC stops claiming recovery, raises one warning, and names the likely cause.
A hash-chained log of every intervention (HMAC-keyed with ledger_key), exportable as JSON. Savings in GPU-hours, dollars, and carbon are measured against your own checkpoint interval, per run, team, and project.
SHA-256 and HMAC signing on every storage strategy, verified on restore, plus a disaster-recovery drill that proves a checkpoint restores correctly.
Declarative remediation rules over failure predictions with a dry-run approval gate, and a severity-stacked alert router with de-duplication.
Detectors for KL blowup, entropy collapse, reward hacking, advantage explosion, mode collapse, and n-gram repetition in rollouts.
Straggler detection across ranks, cross-replica gradient checks for silent data corruption, and bit-reproducibility reports.
Snapshots copy to host memory on a background thread right after optimizer.step(), overlapping the next forward and backward pass. Frequency adapts to how stable training is.
PINN stabilizer, conformal uncertainty, adversarial detection, and EWC continual learning ship in the same package.
arc/
core/ Controller (canonical API), recovery ledger
signals/ Lyapunov, trend, gradient, weight, RL + rollout health
features/ Feature extraction and buffering
prediction/ Failure predictor, remediation policy engine
intervention/ Rollback, OOM, hardware, DDP, model heuristics
checkpointing/ Adaptive + signed checkpoints, DR drill
distributed/ DDP rollback, straggler detection
evaluation/ Reliability metrics, run comparison, chargeback
security/ SDC detection, health baseline, adversarial
api/ Alert router, enterprise report, callbacks
introspection/ Hessian + Fisher analysis (experimental)
physics/ PINN stabilization
uncertainty/ Conformal prediction
learning/ Failure meta-model, EWCController.step(loss) runs every detector in order and returns a StepAction with health status, failure type, and what ARC did about it.
Lyapunov, trend, gradient, and weight-health monitors switch on or off independently in ControllerConfig, without touching recovery logic.
Every intervention lands in a hash-chained ledger you can export, verify, diff across runs, and roll up per team.
Requires PyTorch, NumPy, and SciPy. Optional integrations for Lightning, Hugging Face, and Weights & Biases.
Exercised in ARC's test scripts on GPT-2 (124M to 1.5B), a 2.65B-parameter custom transformer, BERT, ResNet-18/50, VGG11, ViT-Base, DINOv2, YOLOv11, Llama-style models, and Stable Diffusion UNet.
ClearBound Labs is a research institution building infrastructure that keeps intelligent systems reliable, observable, and secure. ARC is its flagship open-source project, released under AGPL-3.0, with 9.71K downloads on PyPI since its first release in March 2026.
Install it in seconds and add one call to your training loop.