Token-local operations can shard cleanly.
Layer normalization, dropout, and residual operations act independently over token positions. Each device can own a contiguous sequence slice.
Activation shard map
Activation memory/device0 GB
Collective traffic0 GB
Layer time0 ms
Memory reduction1x
Partition token-local work, then restore the tensor boundary.
Tensor-parallel regions need collectives.
Reduce-scatter creates sequence shards after a row-parallel projection; all-gather restores the full sequence before operations that require it.
Memory falls with shard degree.
Token activations divide approximately by the number of sequence-parallel workers.
Long context raises the payoff.
Activation memory grows linearly with sequence length, hidden width, and microbatch size.
Overlap hides only compatible traffic.
Collectives can overlap independent compute, but exposed communication remains on the critical path.