karpathy/minbpe

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

81 best points · 31 comments · 1 Hacker News post · 11,018 GitHub stars · View on GitHub

Hacker News discussions

• Code for the Byte Pair Encoding algorithm, commonly used in LLM tokenization (81 points, 31 comments, by magoghm, 2024-02-17)