jakecyr/omni-tokenizer

Node based tokenizers for open source LLMs hosted on HuggingFace.

2 best points · 1 comments · 1 Hacker News post · 2 GitHub stars · View on GitHub

Hacker News discussions

• Node.js Open Source LLM Tokenizer (2 points, 1 comments, by jakecyr, 2024-03-22)