ttok 1.0 Released with GPT-5/GPT-6 Tokenizer Default
· via Simon Willison
The developer released version 1.0 of ttok, upgrading the default tokenizer from GPT-4 to GPT-5/GPT-6. OpenAI has not officially confirmed the tokenizer compatibility between GPT-6 and GPT-5, but an experiment by William Liu suggests they are the same. The new version ensures consistency across GPT models, matching token counts and fixtures.
Read the full article
Continue reading at Simon Willison →This is an AI-generated summary. Read the original for the full story.