zurawiki/tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

2 best points · 1 comments · 1 Hacker News post · 410 GitHub stars · View on GitHub

Hacker News discussions

• Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer (2 points, 1 comments, by zurawiki, 2023-03-18)