Citiverse è uno spazio aperto a tutte le comunità. Se vuoi aprire un gruppo locale o una sezione per la tua organizzazione, puoi contattare gli amministratori: pagina dei contatti.
Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU
-
- Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller is a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget.
- Jinja chat template helps use fewer thinking tokens without noticeably affecting quality, which is great for speed.
- And the final ingridient is beellama.cpp engine which supports the
kvarnKV cache types needed for this optimization.
Here's a command to start the server, the magic is in the
kvarncache settings and the tail precision:"$LLAMA_DIR"/llama-server \ -m "$MODEL_PATH" \ -a "$MODEL_NAME" \ --port 11434 \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --parallel 1 \ --n-gpu-layers 99 \ --batch-size 1024 \ --ubatch-size 256 \ --flash-attn on \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --cache-type-k kvarn5 \ --cache-type-v kvarn4 \ --kv-tail-tokens 1024 \ --ctx-size 100000 \ --fit-ctx 100000 \ --jinja \ --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \ --chat-template-file "$MODEL_JINJA" \ --no-mmproj-offload \ --threads 7 \ --threads-batch 8 \ --metrics \ --verbosity 3 \ --perfThis should give you around 50 tok/sec using just under 16GB of VRAM.
The
kvarn5(K) /kvarn4use thekvarntypes from beellama to balance memory and quality. Using speculative decoding--spec-type draft-mtpwith 2 draft tokens gives another speed boost. The--kv-tail-tokens 1024precision tail is key for keeping recent tokens at higher precision to preserve output quality. Finally, the near-losslesskvarnquantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible. -
- Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller is a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget.
- Jinja chat template helps use fewer thinking tokens without noticeably affecting quality, which is great for speed.
- And the final ingridient is beellama.cpp engine which supports the
kvarnKV cache types needed for this optimization.
Here's a command to start the server, the magic is in the
kvarncache settings and the tail precision:"$LLAMA_DIR"/llama-server \ -m "$MODEL_PATH" \ -a "$MODEL_NAME" \ --port 11434 \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --parallel 1 \ --n-gpu-layers 99 \ --batch-size 1024 \ --ubatch-size 256 \ --flash-attn on \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --cache-type-k kvarn5 \ --cache-type-v kvarn4 \ --kv-tail-tokens 1024 \ --ctx-size 100000 \ --fit-ctx 100000 \ --jinja \ --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \ --chat-template-file "$MODEL_JINJA" \ --no-mmproj-offload \ --threads 7 \ --threads-batch 8 \ --metrics \ --verbosity 3 \ --perfThis should give you around 50 tok/sec using just under 16GB of VRAM.
The
kvarn5(K) /kvarn4use thekvarntypes from beellama to balance memory and quality. Using speculative decoding--spec-type draft-mtpwith 2 draft tokens gives another speed boost. The--kv-tail-tokens 1024precision tail is key for keeping recent tokens at higher precision to preserve output quality. Finally, the near-losslesskvarnquantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible.Hi, thanks for posting this guide!
I tried it on a modern laptop with 32GB of unified memory. It starts at 40 tok/s when loading the context, but the output drops to 3 tok/s.
Could I ask you which GPU you are using for these tests? What context size would a 16GB GPU give?
-
Hi, thanks for posting this guide!
I tried it on a modern laptop with 32GB of unified memory. It starts at 40 tok/s when loading the context, but the output drops to 3 tok/s.
Could I ask you which GPU you are using for these tests? What context size would a 16GB GPU give?
I saw the guide posted on reddit originally, so not sure which card it was tested with unfortunately.
Ciao! Sembra che tu sia interessato a questa conversazione, ma non hai ancora un account.
Stanco di dover scorrere gli stessi post a ogni visita? Quando registri un account, tornerai sempre esattamente dove eri rimasto e potrai scegliere di essere avvisato delle nuove risposte (tramite email o notifica push). Potrai anche salvare segnalibri e votare i post per mostrare il tuo apprezzamento agli altri membri della comunità.
Con il tuo contributo, questo post potrebbe essere ancora migliore 💗
Registrati Accedi
Citiverse è un progetto che si basa su NodeBB ed è federato! | Categorie federate | Chat | 📱 Installa web app o APK | 🧡 Donazioni | Privacy Policy