SivagurunathanV/token-aware-balancer

L7 reverse proxy in Go that routes LLM requests by in-flight token count instead of connections. -12% P99 latency under contention.

2 best points · 0 comments · 1 Hacker News post · 3 GitHub stars · View on GitHub

Hacker News discussions

• Token-Aware LLM Load Balancer route by inflight tokens,not connections (2 points, 0 comments, by sivagurunathanv, 2026-04-06)