alasdairforsythe/tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
10 best points · 0 comments · 2 Hacker News posts · 624 GitHub stars · View on GitHub
Hacker News discussions
- TokenMonster: Ungreedy tokenizer, outperforming tiktoken by 35% (10 points, 0 comments, by tosh, 2023-06-04)
- Tokenmonster: Determine tokens to optimally represents a dataset (1 points, 0 comments, by anotherpaulg, 2023-06-09)