jmaczan/bpe-tokenizer

Byte-Pair Encoding tokenizer for training large language models on huge datasets

5 best points · 0 comments · 1 Hacker News post · 6 GitHub stars · View on GitHub

Hacker News discussions