Special Token And Control Sequence Handling

tiktoken is a fast BPE tokeniser for use with OpenAI's models

Unique: Maintains a curated registry of OpenAI's special tokens per encoding scheme and handles them as atomic units rather than splitting them into subword tokens. This ensures chat prompts with <|im_start|>, <|im_end|>, and other control sequences are tokenized identically to how OpenAI's servers tokenize them.

vs others: More accurate for chat models than generic tokenizers because it explicitly recognizes OpenAI's special tokens and prevents them from being split into subword pieces, matching OpenAI's internal tokenization exactly