Quick Start

ARC (Autonomous Recovery Controller) makes PyTorch training self-healing. It watches every training step, catches numerical failures such as NaN losses, exploding gradients, and corrupted weights, and rolls the run back to a healthy state without you restarting it. ARC is open source (AGPL-3.0) and built by ClearBound Labs.

Installation

From PyPI

pip install arc-training

From Source

git clone https://github.com/a-kaushik2209/ARC.git
cd ARC
pip install -e .

Optional Dependencies

pip install "arc-training[hf]"          # Hugging Face Trainer callback
pip install "arc-training[lightning]"   # Lightning callback
pip install "arc-training[dev]"         # tests and build tools

Requirements: Python 3.10+, PyTorch 2.4+, NumPy 1.23+, SciPy 1.9+. The latest release is 5.2.0 (October 25, 2026).

First Integration

import torch
import torch.nn as nn
import arc

model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

controller = arc.Controller(model, optimizer)

for epoch in range(100):
    for batch_x, batch_y in dataloader:
        optimizer.zero_grad()
        loss = nn.functional.cross_entropy(model(batch_x), batch_y)
        loss.backward()

        action = controller.step(loss)
        if action.should_skip:
            continue          # ARC already rolled back; skip this update

        optimizer.step()

controller.summary()          # end-of-run report: failures, rollbacks, LR changes

Where to Call step()

Call controller.step(loss) after loss.backward() and before optimizer.step(). Gradient checks need the gradients to exist, and the optimizer must not apply an update that ARC rejected. Inside a framework hook where the loss is not available, call controller.step(None): gradient, weight, and dynamics checks still run.

Mixed Precision (fp16 and GradScaler)

Hand the scaler to the controller. ARC unscales the gradients itself, checks them at their true scale, and treats the scaler's routine overflow steps as a warning, not a failure. Pass the unscaled loss and don't call scaler.unscale_() yourself.

scaler = torch.amp.GradScaler("cuda")
controller = arc.Controller(model, optimizer, scaler=scaler)

for batch_x, batch_y in dataloader:
    optimizer.zero_grad()
    with torch.autocast("cuda", dtype=torch.float16):
        loss = nn.functional.cross_entropy(model(batch_x), batch_y)
    scaler.scale(loss).backward()

    action = controller.step(loss)
    if not action.should_skip:
        scaler.step(optimizer)
        scaler.update()

bf16 needs no scaler: use the plain loop. When a framework manages its own scaler, set external_grad_scaler=True so non-finite gradients are left to it. The Hugging Face and Lightning callbacks do this for you.

How ARC Works

Each call to Controller.step(loss) runs the detectors in order and stops at the first failure:

Controller.step(loss)
    |
    +-- warm-up: auto-calibrate thresholds (first 50 steps; NaN/Inf still caught)
    +-- NaN / Inf loss                    immediate
    +-- loss explosion                    absolute threshold + 10x spike vs recent mean
    +-- NaN / Inf weights, weight health  every 10 steps: sparsity, magnitude collapse
    +-- NaN / Inf gradients               true scale (GradScaler-aware)
    +-- gradient explosion                norm threshold (clipping only if enabled)
    +-- Lyapunov divergence               predictive, every 5 steps
    +-- trend drift                       OLS slope over a sliding window
    +-- DDP: every rank agrees            one all-reduce, so replicas never diverge
    |
    +--[FAILURE]--> restore snapshot (model + optimizer + RNG), skip the step
    |               reduce LR x0.5 (undone after 500 healthy steps)
    |               5 rollbacks within a few steps -> escape: oldest snapshot,
    |                   perturb weights, 2 extra LR cuts, skip 5 steps
    |               3 escapes with no progress -> unrecoverable (warns once)
    |
    +--[HEALTHY]--> snapshot copied async after optimizer.step(),
                    kept once the next step passes; adaptive frequency

Detectors

The original signal collectors (GradientCollector, ActivationCollector, WeightCollector, OptimizerCollector, LossCollector, CurvatureCollector) are still shipped for the legacy Arc API and custom pipelines. See the API reference.

Failure Prediction

ARC looks for trouble before the loss shows it in two ways.

• Lyapunov early warning. An online Lyapunov exponent over a fixed per-tensor parameter sample rises when nearby trajectories start to separate. Velocity is measured per unit of learning rate, so a warmup schedule isn't mistaken for divergence. In ARC 5.1 experiments with learning-rate ramps it fired 26 to 101 steps before loss-based detection. That lead time has not been re-measured on 5.2.
• Learned classifier (research). FailurePredictor scores 12 signal features. No trained model ships with the package, so it returns risk_level="unknown" until you load your own with model_path. On simulated training curves a two-layer MLP reached 97.5% accuracy. That figure measures the classifier on synthetic data, not ARC on real runs.

Predictions can drive actions through the policy engine. Policies never fire on an unknown prediction.

Recovery and Rollback

When a detector fires, ARC:

1. Restores model weights, optimizer state (including momentum buffers), and RNG state from the most recent validated snapshot, and skips the poisoned step. If that snapshot is unusable, it falls back to an older one.
2. Reduces the learning rate by lr_reduction_factor (0.5), up to max_lr_reductions (8) times. Each snapshot records the reduction in force when it was taken, so a restore never applies a cut twice. An attached scheduler's base_lrs (and OneCycleLR's max_lrs) are scaled too, so the scheduler cannot undo the cut.
3. Clears the loss and gradient history and resets the Lyapunov and trend detectors, so the restored state is not judged against corrupted history.
4. If 5 rollbacks happen within a few steps of each other, it escapes the loop: it restores the oldest snapshot, adds small noise to the weights (the same noise on every DDP rank), applies two extra LR cuts, and skips the next 5 steps.
5. Counts the failure as recovered only after recovery_confirm_steps (10) healthy steps. A rollback alone is not a recovery.
6. After lr_recovery_steps (500) further healthy steps, undoes one LR reduction at a time. OneCycleLR and CyclicLR keep the reduction.
7. Logs every event to the hash-chained recovery ledger.

Snapshots live in host memory. Right after optimizer.step(), a background thread copies the new state on a side CUDA stream, overlapping the next forward and backward pass. A snapshot becomes a rollback target only once the following step passes every check, and a failed copy is dropped with a warning without stopping training. With adaptive_frequency on, ARC snapshots every 5 steps when training is unstable and every 100 steps when it is calm. Plan host RAM for max_checkpoints (3) copies of the model plus optimizer state.

Unrecoverable Failures

Rollback can only repair the model. A NaN row in the dataset, an unguarded log(0) in the loss, or an fp16 overflow in the inputs comes back on every step no matter how far ARC rewinds. Since 5.1.2, after max_persistent_failures (default 3) escape cycles ARC stops claiming recovery:

• It sets action.unrecoverable and controller.unrecoverable. The flag stays set, so you cannot miss it on a skipped step.
• It raises a single RuntimeWarning.
• It names the likely cause. If the weights are still finite, the fault is in the data, the loss, or numeric precision. If they are not, every retained checkpoint is probably already corrupted.

for batch in dataloader:
    ...
    action = controller.step(loss)
    if action.unrecoverable:
        print(action.message)   # names the likely cause
        break                   # fix the input pipeline, then restart

Save and Resume

In-memory snapshots disappear with the process. For a durable checkpoint, save_checkpoint() writes the model, optimizer, scheduler, scaler, RNG, and ARC's own state (thresholds, LR reductions, counters) in one atomic file that loads with weights_only=True.

controller.save_checkpoint("run/latest.pt")        # atomic write

# later, in a new process
controller = arc.Controller(model, optimizer, scheduler=scheduler)
start_step = controller.load_checkpoint("run/latest.pt")
...
controller.close()   # detach ARC's optimizer hooks when you are done

state_dict() and load_state_dict() return and accept ARC's state as plain Python values, if you'd rather store it inside your own checkpoint.

ControllerConfig

from arc import Controller, ControllerConfig

config = ControllerConfig(
    checkpoint_frequency=20,
    calibration_steps=50,
    lr_reduction_factor=0.5,
    enable_lyapunov=True,
    enable_trend=True,
    gradient_accumulation_steps=4,   # only check full accumulation steps
    verbose=False,
)
controller = Controller(model, optimizer, scheduler=scheduler, config=config)

Defaults

Auto-Calibration

Most users don't know what loss_explosion_threshold=100 means for their model. During the first calibration_steps, ARC records loss and gradient norms, then sets each threshold to mean + 10 × max(std, 0.1 × mean). The loss threshold is capped at 100× the warm-up mean (and 10⁶) so a few noisy warm-up steps cannot switch detection off. Once five losses of history exist, the relative spike check runs regardless of where the absolute threshold lands.

Legacy Config

The original Config system used by Arc, ArcV2, and ArcCallback is still supported:

from arc import Config

config = Config.low_overhead()     # production: 5% sampling, no curvature
config = Config.high_accuracy()    # research: 30% sampling, curvature proxy

config.thresholds.loss_explosion_factor = 10.0
config.thresholds.vanishing_grad_threshold = 1e-7

json_str = config.to_json()
config = Config.from_json(json_str)

Recovery Ledger

Every intervention is appended to controller.intervention_log. Each entry stores the step, reason, loss, action, restored step, steps lost, new learning rate, stability score, and GPU-hours lost. With baseline_checkpoint_interval set, it also records the steps, GPU-hours, and dollars saved compared with restarting from your last durable checkpoint. Without it those fields are null, because ARC can't know what the rollback saved you. Confirmed recoveries are logged as separate recovery_confirmed entries.

Each entry's hash covers its own fields plus the previous entry's hash, so editing or deleting any past entry breaks the chain. The hash is SHA-256 by default and HMAC-SHA256 when ledger_key (or ARC_LEDGER_KEY) is set, so someone without the key can't rebuild a valid chain.

controller.export_recovery_ledger("ledger.json")   # strict JSON, NaN-safe
assert controller.verify_ledger_integrity()         # tamper check

ROI and Chargeback

from arc import Controller, ControllerConfig
from arc.evaluation.chargeback import aggregate_ledger_files
from arc.api.enterprise_report import EnterpriseReport

config = ControllerConfig(gpu_count=8, gpu_hourly_cost=2.50, gpu_power_watts=700,
                          carbon_intensity_kg_per_kwh=0.4,
                          baseline_checkpoint_interval=1000,   # you save to disk every 1000 steps
                          team="research", project="llm-pretrain")
controller = Controller(model, optimizer, config=config)
# ... train ...
print(controller.reliability_roi_report())          # GPU-hours, $, kg CO2 saved

rollup = aggregate_ledger_files("ledgers/*.json")   # per team/project totals

EnterpriseReport(controller).save(json_path="report.json", html_path="report.html")

EnterpriseReport combines the ledger, ROI report, policy decisions, and routed alerts into one JSON file and one self-contained HTML page with inline charts. Nothing needs to be hosted.

Checkpoint Integrity

AdaptiveCheckpointer picks a storage strategy (full CPU, quantized, incremental delta, streaming to disk, or sharded) based on available memory. Quantized strategies keep a per-tensor scale and never quantize optimizer state unless you set quantize_optimizer=True, and restores keep each tensor's original dtype. Loads use weights_only=True; set allow_unsafe_load=True only for checkpoints you produced yourself. With signing enabled, every strategy records a SHA-256 content hash and an HMAC signature, and restore() raises ValueError on a mismatch. verify_recoverability() is a disaster-recovery drill. It restores a checkpoint into a throwaway model and checks that the outputs match the live model, without touching the live model.

import os
from arc.checkpointing import AdaptiveCheckpointer, AdaptiveCheckpointConfig
from arc.checkpointing.dr_drill import verify_recoverability

ckpt = AdaptiveCheckpointer(model, optimizer, AdaptiveCheckpointConfig(
    sign_checkpoints=True, signing_key=os.environ["ARC_SIGNING_KEY"]))
ckpt.save(step=0)

drill = verify_recoverability(
    ckpt,
    model_factory=lambda: MyModel(),
    optimizer_factory=lambda m: torch.optim.Adam(m.parameters()),
    sample_input=sample_batch,
)
print(drill.passed, drill.max_abs_diff)

Policies and Alerts

PolicyEngine applies declarative remediation rules to FailurePrediction output. Policies load from JSON. A policy with dry_run: true records its decisions without running any notifier, which serves as an approval gate. AlertRouter sends each alert to every channel at or below its severity and suppresses repeats of the same dedup_key within a time window.

[
  {"name": "high-risk", "min_overall_risk": 0.7, "min_confidence": 0.5,
   "actions": ["checkpoint", "notify_slack"], "dry_run": true}
]

from arc.prediction.policy import PolicyEngine
from arc.api.alerting import AlertRouter, Alert, Severity

engine = PolicyEngine.from_config("policies.json",
                                  notifiers={"notify_slack": post_to_slack})
decisions = engine.evaluate(prediction)     # FailurePrediction from FailurePredictor
engine.export_log("decisions.json")

router = AlertRouter(channels={Severity.WARNING: [post_to_slack],
                               Severity.CRITICAL: [page_oncall]},
                     dedup_window_seconds=300)
router.route(Alert(Severity.CRITICAL, "NaN loss", "Rolled back to step 1200",
                   dedup_key="nan_loss"))

Run Comparison

from arc.evaluation.run_comparison import RunComparator
from arc.evaluation.reliability_metrics import compute_reliability_metrics
from arc.security.health_baseline import HealthBaseline

report = RunComparator().compare_files("last_good_ledger.json", "ledger.json")
print(report.has_regression, report.to_dict())

metrics = compute_reliability_metrics(controller.intervention_log)  # recovery rate, MTBF

baseline = HealthBaseline(zscore_threshold=3.0)
for past in past_ledgers:
    baseline.add_run(past)
print(baseline.check(current_ledger).is_anomalous)

RunComparator flags higher rollback rates, worsening stability trends, and failure modes that did not appear in the baseline. HealthBaseline learns what normal looks like from your own past runs and flags a new run that falls outside that range.

RL and Generation Stability

from arc.signals.rl_stability import RLStabilityMonitor
from arc.signals.rollout_health import RolloutHealthMonitor

rl = RLStabilityMonitor(kl_threshold=0.1, entropy_floor=0.01)
report = rl.update(kl=kl, entropy=entropy, reward=reward,
                   true_objective=eval_score, advantage=adv_mean)
if not report.is_healthy:
    print(report.messages)   # KL blowup, entropy collapse, reward hacking, advantage explosion

rollouts = RolloutHealthMonitor(ngram_n=3, repetition_threshold=0.3)
health = rollouts.check_batch(generated_token_ids)   # mode collapse, repetition, length drift

Reward hacking is flagged when the reward keeps rising while the true objective falls.

Distributed Signals

from arc.distributed.straggler_detection import StragglerDetector
from arc.security.sdc_detection import CrossReplicaGradientCheck
from arc.security.reproducibility import ReproducibilityTracker

stragglers = StragglerDetector(window=20, zscore_threshold=4.0)
report = stragglers.record_step({rank: step_seconds for rank, step_seconds in timings.items()})

sdc = CrossReplicaGradientCheck(relative_deviation_threshold=3.0)
alert = sdc.check({rank: grad_vector for rank, grad_vector in per_rank_grads.items()})

repro = ReproducibilityTracker(check_every_n_calls=100)
repro.maybe_check(model, fixed_batch)
print(repro.report().reproducible_ratio)

PINN Stabilizer

from arc import PINNStabilizer

stabilizer = PINNStabilizer(model, n_loss_terms=3)

for epoch in range(1000):
    pde_loss, bc_loss, ic_loss = compute_losses(model, inputs)
    total = stabilizer.get_stabilized_loss([pde_loss, bc_loss, ic_loss])
    optimizer.zero_grad()
    total.backward()
    stabilizer.stabilize_step()
    optimizer.step()
    stabilizer.update(epoch, total)

Continual Learning (EWC)

Elastic Weight Consolidation is available through the legacy ArcV2 API, and ArcV2.auto() turns it on by default. Add the EWC penalty to your loss before backward(). ArcV2.step() only records it.

from arc import ArcV2

arc = ArcV2.auto(model, optimizer)

arc.begin_task("mnist")
train(model, mnist_loader, arc)
arc.consolidate_task(mnist_loader)

arc.begin_task("fashion_mnist")
for epoch in range(10):
    for batch in fashion_loader:
        optimizer.zero_grad()
        loss = compute_loss(model, batch) + arc.get_ewc_loss()
        loss.backward()
        optimizer.step()
        arc.step(loss)
    arc.end_epoch(epoch)

Uncertainty Quantification

from arc import ConformalPredictor

cp = ConformalPredictor(model, alpha=0.1)
cp.calibrate(calibration_loader)

result = cp.predict(test_input)
print(result.prediction, result.confidence, result.set_members)

Adversarial Defense

from arc import AdversarialDetector

detector = AdversarialDetector(model)
detector.fit(clean_data_loader)

alert = detector.detect(suspicious_input)
print(alert.is_adversarial, alert.confidence)

PyTorch Lightning and Hugging Face

import lightning as L
from arc.api.integrations import LightningCallback

trainer = L.Trainer(callbacks=[LightningCallback()])

from transformers import Trainer
from arc.api.integrations import HuggingFaceCallback

trainer = Trainer(model=model, args=args, train_dataset=ds,
                  callbacks=[HuggingFaceCallback()])

Both callbacks build an arc.Controller when training starts and run it in the hook just before the optimizer step (on_pre_optimizer_step in Transformers, on_before_optimizer_step in Lightning). A poisoned update is skipped and the weights are rolled back. Install the extras with pip install "arc-training[hf]" or "arc-training[lightning]".

Weights & Biases

wandb.init(project="my-run")
for step, batch in enumerate(dataloader):
    ...
    action = controller.step(loss)
    controller.log_to_wandb(step)   # arc/stability_score, arc/current_lr, arc/total_rollbacks, ...

log_to_wandb() does nothing if wandb is not installed or not initialized. For other tools, controller.get_stats() returns the same numbers as a dict.

Docker

FROM pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime
RUN pip install arc-training
COPY . /app
WORKDIR /app
CMD ["python", "train.py"]

Changelog

v5.2.0 (October 25, 2026)

A correctness and honesty release from a line-by-line audit. Several defaults change.

• fp16 GradScaler support. Runs were rolled back for no reason because ARC judged scaled gradients. Pass scaler=; ARC unscales, checks true-scale gradients, and lets the scaler skip overflow steps.
• NaN gradients behind a finite loss are now caught as nan_gradients.
• No gradient clipping by default. Clipping changes training, so it is opt-in with enable_baseline_clipping=True.
• Recovery is confirmed only after 10 healthy steps (recovery_confirm_steps). LR cuts are undone one at a time after 500 healthy steps (lr_recovery_steps).
• Honest savings. The ledger records steps_lost; steps, GPU-hours, and dollars saved need baseline_checkpoint_interval and are null without it. HMAC-keyed ledger with ledger_key.
• DDP replicas stay identical: every rank agrees on each rollback.
• Fixes: double LR reduction on restore, a rollback loop on stale history, a step-2 false positive, quantized checkpoints that zeroed Adam state, silent unsafe unpickling, Hugging Face and Lightning callbacks that acted after the optimizer step, and a predictor that scored risk with untrained weights.
• New APIs: step(None), save_checkpoint() / load_checkpoint(), state_dict() / load_state_dict(), close().
• Snapshots copy to host memory on a background thread right after optimizer.step(). Measured overhead: 2.2% on GPT-2 (124M). import arc loads in about 3 s, down from 13 s.
• Python 3.10+ and PyTorch 2.4+. A new symmetric recovery benchmark replaces the 5.1 stress suites. 251 tests.

v5.1.2 (August 2026)

• LR reductions now accumulate across rollbacks. Restoring a checkpoint used to restore the old learning rate. Four rollbacks now step 5e-4 → 2.5e-4 → 1.25e-4 → 6.25e-5.
• Unrecoverable terminal state. When the fault is in the data or the loss, ARC sets StepAction.unrecoverable, warns once, and names the likely cause instead of looping forever.
• New ControllerConfig.max_persistent_failures (default 3), and an 8-scenario organic-failure harness. 169 tests.

v5.1.1 (August 2026)

• Fixed the shipped examples/arc_demo.py import after the IntrospectiveArc rename.

v5.1.0 (August 2026)

• Enterprise reliability layer: recovery ledger, ROI and chargeback reporting, tamper-evident audit log, EnterpriseReport.
• Checkpoint integrity: SHA-256 and HMAC signing enforced on all storage strategies, plus a disaster-recovery drill.
• Auto-remediation policy engine with a dry-run approval gate. Also an alert router with de-duplication.
• RL and generation stability: KL blowup, entropy collapse, reward hacking, and rollout mode-collapse detectors.
• Straggler detection, silent-data-corruption checks, health baselines, and reproducibility reports.
• Fixed checkpoint signing, which silently did nothing for 4 of 5 strategies, and an incremental-delta restore that dropped intermediate changes. Tests grew from 89 to 165.

v5.0.0 (June 2026)

• Unified Controller API, one entry point replacing 5 legacy APIs.
• Lyapunov stability monitoring and trend-based drift detection.
• Auto-calibration, async CPU-offloaded checkpointing, and LR-scheduler coordination.
• 10/10 on the failure gauntlet: NaN, Inf, weight corruption, gradient explosion, loss spike, persistent failure.

v4.1.0 to v4.2.2 (March 2026)

• 97.5% classifier accuracy on simulated training curves (up from 86.5%) with the 12-feature MLP.
• Every paper claim backed by an experiment script.

v4.0.0 (January 2026)

• 100% recovery on all numeric failure types. OOM recovery across all training stages.
• Quantized FP16 checkpointing (50% memory reduction). Silent-failure detection.

Full Changelog on GitHub