Here's a command to start the server, the magic is in the kvarn cache settings and the tail precision:

"$LLAMA_DIR"/llama-server \
    -m "$MODEL_PATH" \
    -a "$MODEL_NAME" \
    --port 11434 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --presence-penalty 0.0 \
    --repeat-penalty 1.0 \
    --parallel 1 \
    --n-gpu-layers 99 \
    --batch-size 1024 \
    --ubatch-size 256 \
    --flash-attn on \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    --cache-type-k kvarn5 \
    --cache-type-v kvarn4 \
    --kv-tail-tokens 1024 \
    --ctx-size 100000 \
    --fit-ctx 100000 \
    --jinja \
    --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
    --chat-template-file "$MODEL_JINJA" \
    --no-mmproj-offload \
    --threads 7 \
    --threads-batch 8 \
    --metrics \
    --verbosity 3 \
    --perf

This should give you around 50 tok/sec using just under 16GB of VRAM.

The kvarn5 (K) / kvarn4 use the kvarn types from beellama to balance memory and quality. Using speculative decoding --spec-type draft-mtp with 2 draft tokens gives another speed boost. The --kv-tail-tokens 1024 precision tail is key for keeping recent tokens at higher precision to preserve output quality. Finally, the near-lossless kvarn quantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible.