Save activations or recompute them later.

Activation checkpointing lowers training memory by retaining selected layer outputs and rebuilding the missing activations during backward. Explore the exchange rate between memory, compute, and batch capacity.

Inspect the training graph

Memory is not free. Neither is recomputation.

Ordinary backpropagation stores intermediate activations so gradients can reuse them. Checkpointing keeps only selected boundaries. During backward, each discarded segment runs forward again. That often unlocks longer sequences or larger batches, but the extra work changes throughput.

Transformer activation map

Forward · storing every layer
forwardbackward + recompute
Stored activation memory3.22 GB

Modeled layer outputs retained for backward.

Memory saved83%

Reduction versus storing every layer output.

Extra forward work75%

Approximate recomputation added to one training step.

Batch capacity index5.9×

Relative activation-limited batch headroom.

What gets stored?

A checkpoint is a boundary activation. If the interval is four, the lab stores every fourth layer output plus the input. Smaller intervals retain more tensors and reduce repeated work.

What gets recomputed?

Backward needs intermediate values. For each segment, the system reruns forward from its nearest stored boundary, reconstructs the missing activations, computes gradients, then releases temporary tensors.

What the simple estimate omits

Real peak memory includes parameters, gradients, optimizer states, attention intermediates, allocator fragmentation, distributed buffers, and framework-specific checkpoint behavior. Measure the deployed stack with a profiler before choosing production batch sizes.

Primary reading

Training Deep Nets with Sublinear Memory Cost Chen et al.torch.utils.checkpoint PyTorchCurrent and New Activation Checkpointing Techniques PyTorch Foundation