marcelroed/gigatoken

Language model tokenization at GB/s

620 best points · 119 comments · 1 Hacker News post · 4,153 GitHub stars · View on GitHub

Hacker News discussions

• GigaToken: ~1000x faster Language model tokenization (620 points, 119 comments, by syrusakbary, 2026-07-22)