Frequently Asked Questions

ARC (Autonomous Recovery Controller) is an open-source Python library that makes PyTorch training self-healing. It wraps your existing training loop, checks every step for numerical failures, predicts instability before the loss diverges, and recovers automatically. It rolls back to the last validated snapshot and lowers the learning rate, with no restart needed.

ARC is the flagship project of ClearBound Labs, a research institution in New Delhi founded by Aryan Kaushik and Mitul Gupta. ClearBound Labs builds research and infrastructure to keep intelligent systems reliable, observable, and secure. ARC has been downloaded 9.71K times from PyPI since its first release in March 2026.

The Controller catches NaN/Inf losses, loss explosions (absolute and relative spikes), NaN/Inf weights, weight sparsity explosion and magnitude collapse, NaN/Inf gradients behind a finite loss, gradient explosions, Lyapunov divergence (predictive), and gradual trend drift. Separate modules cover silent failures such as dead neurons and optimizer-state corruption, RL instabilities (KL blowup, entropy collapse, reward hacking), rollout mode collapse, slow ranks, and silent data corruption across replicas.

Yes. ARC is released under the AGPL-3.0 license and is free for academic and commercial use, subject to the license terms.

ARC supports PyTorch natively. It also integrates with Lightning through LightningCallback and with the Hugging Face Trainer through HuggingFaceCallback (both in arc.api.integrations), and with Weights & Biases through controller.log_to_wandb(). TensorFlow and JAX are not supported.

Python 3.10+, PyTorch 2.4+, NumPy 1.23+, and SciPy 1.9+. Optional extras: pip install "arc-training[hf]" for the Hugging Face callback, "arc-training[lightning]" for the Lightning callback, and "arc-training[dev]" for tests.

Clone the repository and install in development mode: git clone https://github.com/a-kaushik2209/ARC.git && cd ARC && pip install -e .

5.2.0, released October 25, 2026. It changes some defaults (no gradient clipping, confirmed recoveries, honest savings fields), so read the changelog before upgrading.

For vanilla PyTorch, a few: create controller = arc.Controller(model, optimizer), call action = controller.step(loss) after loss.backward(), and skip optimizer.step() when action.should_skip is true. For Lightning or the Hugging Face Trainer, add LightningCallback() or HuggingFaceCallback() to the trainer's callbacks.

Use Controller. Since v5.0 it is the single canonical entry point. Arc, ArcV2, ArcCallback, and BulletproofTrainer are still supported. Use ArcV2 when you need its EWC continual-learning, conformal, or adversarial helpers.

No. ARC wraps your training loop and leaves your model untouched. Your architecture, loss function, optimizer, and scheduler stay the same. If you pass your scheduler to the Controller, ARC scales its base_lrs on each LR reduction so the scheduler cannot undo it. On healthy steps ARC changes nothing: it doesn't clip gradients unless you set enable_baseline_clipping=True.

Any PyTorch model. ARC has been validated on GPT-2 Medium, Large, and XL, a 2.65B-parameter custom transformer, BERT, Llama-style models, NanoGPT, ResNet-18/50/101, Wide ResNet, VGG11, ViT, DINOv2, YOLOv11, Stable Diffusion UNet, and RNNs.

Yes. Set ControllerConfig(gradient_accumulation_steps=N) so ARC only runs full checks on complete accumulation steps. For fp16, pass your scaler: arc.Controller(model, optimizer, scaler=scaler). ARC unscales the gradients, judges them at true scale, and leaves the scaler's routine overflow steps alone. See Mixed Precision. If overflow comes from the inputs on every step, ARC marks the run unrecoverable instead of looping.

With DDP, yes. Create the Controller the same way on every rank. Ranks agree on every rollback through one all-reduce per step, so replicas never diverge (tested with 2 processes). FSDP and DTensor models are refused with a clear error. Straggler detection and cross-replica silent-data-corruption checks ship too. They have unit tests but have not yet been validated on a live multi-node cluster.

Rollback only repairs the model. If the fault is a NaN row in your data, an unguarded log(0), or a zero-variance batch, it comes back after every rollback. After 3 escape cycles (max_persistent_failures), ARC sets action.unrecoverable, raises one RuntimeWarning, and says whether the likely cause is the data or loss (weights still finite) or corrupted checkpoints. Stop the run and fix the input pipeline.

On GPT-2 (124M) on an RTX 3050, ARC 5.2.0 adds 2.2% per step. Snapshots are copied on a background thread that overlaps the next forward and backward pass. Larger GPUs and multi-billion-parameter models have not been profiled yet; run experiments/controller_overhead.py on your own hardware. See Benchmarks.

No trained predictor ships with the package, so FailurePredictor reports risk_level="unknown" until you load your own model. In research experiments on 200 simulated training curves, a 12-feature MLP reached 97.5% accuracy. That measures the classifier on synthetic data, not ARC on real runs. Separately, in ARC 5.1 experiments the Lyapunov monitor flagged instability 26 to 101 steps before loss-based detection.

2.65B parameters on a single NVIDIA H200, with ARC 5.1. GPT-2 Medium (355M), Large (774M), and XL (1.56B) and a 2.65B custom transformer each rolled back once after an injected NaN and finished the run. That shows rollback fits and runs at this scale; it is not a recovery rate. Snapshots live in host RAM, so plan for max_checkpoints (3) copies of the model plus optimizer state.

Barely, in our tests. ResNet-18 on CIFAR-10 with a NaN injected mid-run finished at 77.42% vs. 77.43% clean. GPT-2 fine-tuning on WikiText-2 finished at 16.4 perplexity vs. 15.8 clean, because the rollback discards about 5 steps and the LR cut slows convergence slightly.

Yes. controller.export_recovery_ledger() writes a hash-chained JSON log of every intervention, and verify_ledger_integrity() detects tampering. Set ledger_key to make the chain an HMAC that can't be rebuilt without the key. Every rollback records the work it threw away. To report savings, tell ARC how often you save durable checkpoints with baseline_checkpoint_interval and set gpu_hourly_cost, gpu_power_watts, and carbon_intensity_kg_per_kwh; reliability_roi_report() then reports GPU-hours, dollars, and CO₂ saved compared with restarting. Without the interval, savings are reported as unknown. Tag runs with team/project and roll them up with arc.evaluation.chargeback.

Enable sign_checkpoints=True with a signing_key in AdaptiveCheckpointConfig. Every save gets a SHA-256 hash and HMAC signature, and every restore verifies them. All of ARC's loaders use torch.load(weights_only=True) and never fall back to unsafe unpickling on their own. Run verify_recoverability() as a disaster-recovery drill: it restores into a throwaway model and compares outputs with the live model.

AlertRouter sends alerts to any callables you register per severity and suppresses duplicates within a time window. PolicyEngine turns failure predictions into actions from a JSON policy file, with a dry-run approval gate. You supply the Slack or PagerDuty function. Vendor-specific adapters are not built in.

RLStabilityMonitor flags KL blowup, entropy collapse, advantage explosion, and reward hacking (reward rising while your true objective falls). RolloutHealthMonitor checks generated batches for mode collapse, n-gram repetition, and length drift. Coordinated rollback of policy, reward, and reference models is not built yet.

PINNStabilizer provides adaptive loss weighting for physics-informed neural networks with multiple competing loss terms. It balances the losses automatically and stabilizes gradients to prevent chaotic training dynamics.

EWC prevents catastrophic forgetting during continual learning through the ArcV2 API. After a task, call arc.consolidate_task(loader) to compute Fisher information. On the next task, add arc.get_ewc_loss() to your loss before backward().

ConformalPredictor provides distribution-free coverage guarantees. Instead of a single prediction, it returns a prediction set guaranteed to contain the true label with a chosen probability (e.g. 90%).

With the Controller, raise lyapunov_check_frequency and weight_health_check_frequency, or turn off signals you don't need (enable_lyapunov=False, enable_trend=False). Keep async_checkpoint=True and adaptive_frequency=True. For the legacy API, use Config.low_overhead().

That is auto-calibration learning loss and gradient thresholds for your model. NaN/Inf losses are still caught during this phase. Change the length with calibration_steps, or set auto_calibrate=False to use fixed thresholds.

Since 5.2.0 the Controller doesn't clip unless enable_baseline_clipping=True. Earlier versions clipped to 1.0 by default, so upgrade or set enable_baseline_clipping=False. If you use the Hugging Face or Lightning trainer, the clipping may come from the trainer itself (max_grad_norm or gradient_clip_val).

Pass verbose=False to Controller (or Arc), or quiet the logger with logging.getLogger("arc").setLevel(logging.WARNING). ARC writes through the standard arc logger, so its output follows your logging setup. Interventions are still recorded in controller.intervention_log and get_stats().

This is a legacy Arc API error. Call arc.attach(model, optimizer) before on_epoch_end(), or use ArcV2.auto(model, optimizer), which attaches automatically. The Controller takes the model and optimizer in its constructor and has no attach step.

Yes. controller.save_checkpoint(path) writes the model, optimizer, scheduler, scaler, RNG, and ARC's own state in one atomic file, and load_checkpoint(path) restores it and returns the step to resume from. state_dict() and load_state_dict() let you store ARC's state inside your own checkpoints. The in-memory rollback buffer itself is not saved. The legacy API has arc.save_state(path) / load_state(path), and ArcV2.save(path) / load(path) also store EWC state.

Open an issue describing your proposed change and wait for it to be assigned. Then fork the repo, create a feature branch, and open a focused PR that references the issue. Before a larger contribution, email hello@clearboundlabs.com with the subject "ARC".