RC RANDOM CHAOS

ttok 1.0 Released with GPT-5/GPT-6 Tokenizer Default

· via Simon Willison

Original source

ttok 1.0

Simon Willison →

The developer released version 1.0 of ttok, upgrading the default tokenizer from GPT-4 to GPT-5/GPT-6. OpenAI has not officially confirmed the tokenizer compatibility between GPT-6 and GPT-5, but an experiment by William Liu suggests they are the same. The new version ensures consistency across GPT models, matching token counts and fixtures.

Read the full article

Continue reading at Simon Willison →

This is an AI-generated summary. Read the original for the full story.