vitoplantamura/OnnxStream

Lightweight inference library for ONNX files, written in C++. It can run Stable Diffusion XL 1.0 on a RPI Zero 2 (or in 298MB of RAM) but also Mistral 7B on desktops and servers. ARM, x86, WASM, RISC-V supported. Accelerated by XNNPACK. Python, C# and JS(WASM) bindings available.

510 best points · 307 comments · 5 Hacker News posts · 2,087 GitHub stars · View on GitHub

Hacker News discussions

• Running Stable Diffusion XL 1.0 in 298MB of RAM (510 points, 154 comments, by Robin89, 2023-10-03)
• Running Stable Diffusion in 260MB of RAM (293 points, 64 comments, by Robin89, 2023-07-20)
• OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2 (109 points, 87 comments, by Robin89, 2023-12-14)
• OnnxStream running TinyLlama and Mistral 7B, with CUDA support (17 points, 2 comments, by Robin89, 2024-01-14)
• Show HN: OnnxStream running TinyLlama and Mistral 7B, with CUDA support (2 points, 0 comments, by vitoplantamura, 2024-01-14)