nickaggarwal/nvidia-triton-llm-streaming

Integrating SSE with NVIDIA Triton Inference Server using a Python backend and Zephyr model. There is very less documentation how to use Nvidia Triton in Streaming use-cases ( hard to find in their docs ), hence this should be helpful for people who want to deploy streaming with Triton

3 best points · 0 comments · 1 Hacker News post · 10 GitHub stars · View on GitHub

Hacker News discussions

• Real-Time Streaming Apps with Nvidia Open Source Triton Inference (3 points, 0 comments, by agcat, 2024-06-05)