distantmagic/paddler

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

151 best points · 20 comments · 2 Hacker News posts · 1,689 GitHub stars · View on GitHub

Hacker News discussions

• Show HN: Open-source load balancer for llama.cpp (151 points, 20 comments, by mcharytoniuk, 2024-06-01)
• Paddler – Stateful load balancer custom-tailored for llama.cpp (3 points, 0 comments, by throwaway888abc, 2024-05-18)