javirandor/anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream

9 best points · 1 comments · 2 Hacker News posts · 173 GitHub stars · View on GitHub

Hacker News discussions

• Approximation of the Claude 3 tokenizer by inspecting generation stream (9 points, 1 comments, by tosh, 2024-03-22)
• The Worst (But Only) Claude 3 Tokenizer (2 points, 0 comments, by dpaleka, 2024-03-12)