Byte Pair Encoding (BPE)

In depth

A tokenization algorithm that builds its vocabulary by starting from single characters (or bytes) and repeatedly merging the most frequent adjacent pair into a new token. In 2015 Sennrich, Haddow and Birch adapted it from a data-compression algorithm so neural translation models could handle rare and unseen words as sequences of subword units. Byte-level versions power the GPT family and most current LLM tokenizers. That is why a common word is usually one token while a rare name splits into several.

Primary source: arxiv.org