Inspired by antirez's ds4, my goal was to see if a dedicated engine, built from scratch for exactly one model architecture and one GPU class, could offer a significant performance and usability advantage over generic runtimes.
Benchmarks (measured back-to-back with the GPU capped at 400W) show some good results, along with the expected trade-offs:
- Where it shines (Prefill): The tensor-core FlashAttention prefill reaches 13.4k t/s (pp2048) and 7.6k t/s (pp90k) — 21% to 43% faster than llama.cpp on the same hardware. - Where it stands (Decode): Pure decode speed is 270+ t/s, which currently trails llama.cpp by 1% to 9% at depth. Decode at batch1 is memory-bandwidth-bound — the model reads ~2.5 GiB of active weights per token, which caps any engine at ~656 t/s on this card (q36 runs at 41% of that physical limit); the remaining gap to llama.cpp is their more mature flash-decode attention kernels at long context.
Inspired by antirez's ds4, my goal was to see if a dedicated engine, built from scratch for exactly one model architecture and one GPU class, could offer a significant performance and usability advantage over generic runtimes. Benchmarks (measured back-to-back with the GPU capped at 400W) show some good results, along with the expected trade-offs: - Where it shines (Prefill): The tensor-core FlashAttention prefill reaches 13.4k t/s (pp2048) and 7.6k t/s (pp90k) — 21% to 43% faster than llama.cpp on the same hardware. - Where it stands (Decode): Pure decode speed is 270+ t/s, which currently trails llama.cpp by 1% to 9% at depth. Decode at batch1 is memory-bandwidth-bound — the model reads ~2.5 GiB of active weights per token, which caps any engine at ~656 t/s on this card (q36 runs at 41% of that physical limit); the remaining gap to llama.cpp is their more mature flash-decode attention kernels at long context.
[flagged]