1rgs/tokenwiz

A clone of OpenAI's Tokenizer page for HuggingFace Models

3 best points · 1 comments · 2 Hacker News posts · 77 GitHub stars · View on GitHub

Hacker News discussions

• OpenAI's Tokenizer Page for OSS Models (3 points, 1 comments, by rgs224, 2023-11-12)
• Open-Source Tokenizer Visualizer (3 points, 0 comments, by pierremenard, 2023-11-13)