turboderp/exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

322 best points · 125 comments · 1 Hacker News post · 4,651 GitHub stars · View on GitHub

Hacker News discussions

• Exllamav2: Inference library for running LLMs locally on consumer-class GPUs (322 points, 125 comments, by Palmik, 2023-09-13)