microsoft/sarathi-serve

A low-latency & high-throughput serving engine for LLMs

2 best points · 0 comments · 1 Hacker News post · 535 GitHub stars · View on GitHub

Hacker News discussions

• Sarathi-Serve: A low-latency and high-throughput serving engine for LLMs (2 points, 0 comments, by tonyabracadabra, 2024-06-12)