In a sequence generation task, an auto-regressive model generates tokens step-by-step, in a left-to-right fashion (or whatever order is specified):
- It starts by predicting the first token.
- It then uses this first token as part of the input to predict the second token.
- This continues token by token, with each new token prediction based on all previously predicted tokens.
So, each step is conditioned on the previous steps, making the process sequential and self-dependent, which is the essence of being “auto-regressive.”