epolewski/EricLLM

A fast batching API to serve LLM models

3 best points · 0 comments · 1 Hacker News post · 188 GitHub stars · View on GitHub

Hacker News discussions

• Show HN: Made a batching LLM API for a project. Mistral 200 tk/s on RTX 3090 (3 points, 0 comments, by muttled, 2023-12-26)