bminixhofer/tokenkit

A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.

1 best points · 0 comments · 1 Hacker News post · 72 GitHub stars · View on GitHub

Hacker News discussions

• Show HN: Tokenkit – Convert LLMs to new tokenizers (incl byte-level Llama/Gemma) (1 points, 0 comments, by bminixhofer, 2025-04-23)