kuprel/minbpe-pytorch

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA

19 best points · 9 comments · 1 Hacker News post · 43 GitHub stars · View on GitHub

Hacker News discussions

• 55x Speedup of Andrej Karpathy's Minbpe LLM Tokenizer with PyTorch/CUDA (19 points, 9 comments, by kuprel, 2024-02-22)