Modeled layer outputs retained for backward.
Activation checkpointing lowers training memory by retaining selected layer outputs and rebuilding the missing activations during backward. Explore the exchange rate between memory, compute, and batch capacity.
Inspect the training graphOrdinary backpropagation stores intermediate activations so gradients can reuse them. Checkpointing keeps only selected boundaries. During backward, each discarded segment runs forward again. That often unlocks longer sequences or larger batches, but the extra work changes throughput.
Modeled layer outputs retained for backward.
Reduction versus storing every layer output.
Approximate recomputation added to one training step.
Relative activation-limited batch headroom.
A checkpoint is a boundary activation. If the interval is four, the lab stores every fourth layer output plus the input. Smaller intervals retain more tensors and reduce repeated work.
Backward needs intermediate values. For each segment, the system reruns forward from its nearest stored boundary, reconstructs the missing activations, computes gradients, then releases temporary tensors.
Real peak memory includes parameters, gradients, optimizer states, attention intermediates, allocator fragmentation, distributed buffers, and framework-specific checkpoint behavior. Measure the deployed stack with a profiler before choosing production batch sizes.