AviSoori1x/makeMoE

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

88 best points · 14 comments · 1 Hacker News post · 821 GitHub stars · View on GitHub

Hacker News discussions

• Implementation of mixture of experts language model in a single file of PyTorch (88 points, 14 comments, by avisoori1x, 2024-03-18)