Teacher forcing
100%
Previous ground-truth tokens are supplied. Optimization is stable, but corrupted prefixes are absent from training.
Autoregressive training laboratory
Train with the answer. Deploy without it.
Teacher forcing gives a model perfect previous tokens during training. At inference, the model must consume its own outputs. Change the handoff policy and watch one mistake alter every step that follows.
Two sequence lanes, one model
correcterrorteacher token
Training looks clean because every prefix is repaired. Inference receives no repairs, so errors can compound.
Training token accuracy--Measured under the selected handoff policy
Inference token accuracy--Always self-fed, exactly like deployment
Exposure gap--Training accuracy minus inference accuracy
Teacher forcing changes the distribution of prefixes a model sees. Scheduled sampling mixes clean and model-generated prefixes. That can reduce the train-deploy gap, but it does not guarantee a statistically consistent objective.
Scheduled sampling
0–100%
Ground-truth and model tokens are mixed. The model practices recovery, while the objective itself changes.
Self-feeding
0%
The model trains on its own prefixes. This matches deployment exposure but makes credit assignment and optimization harder.
What is exposure bias?
It is the mismatch between the clean prefixes commonly provided during maximum-likelihood training and the model-generated prefixes encountered during autoregressive inference. A single wrong token can move later predictions into unfamiliar states.
Does scheduled sampling solve it?
It can expose the model to its own mistakes, but Bengio et al.'s practical proposal has an important caveat: Huszár showed that the resulting objective can be statistically inconsistent. Treat it as a training intervention to evaluate, not a universal cure.
What should an evaluation measure?
Evaluate complete self-fed rollouts, not only next-token loss under clean prefixes. Track sequence success, recovery after the first error, and the gap between teacher-forced and deployed performance.