Special Tokens

In depth

Reserved tokens that mark structure rather than text, such as beginning-of-sequence (BOS), end-of-sequence (EOS), padding, and BERT-style [CLS] and [SEP] markers. Chat models add more, like <|im_start|> or <start_of_turn>, to show who is speaking and where a turn ends. The tokenizer inserts them automatically when it is set up to. Getting them wrong, by leaving out an EOS or adding a second BOS, quietly makes output worse or stops generation in the wrong place.

Primary source: huggingface.co