Alibaba · Qwen
EXL3-XPU ENGINE · DENSE 27B · THE LIVING RECIPE
Qwen3.8-27B EXL3 4bpw + MTP3 + SC5 mixed-K
ExLlamaV3 trellis on a vLLM-XPU fork — primary route since the 2026-09-27 re-measure
Tested on 1× B70 (32GB)
Best for: the longest context — full 256K on one card.
Loading current benchmark…
The only measured route that reaches the full 262,144-token context on one card — and the only quantized arm that passes the exact-answer battery. Every W4A16/GPTQ export of this model answers "30" on in-head arithmetic; the BF16 base answers 14. OpenVINO int4 is the steady-150W alternate.
Stack & configuration
Image: ghcr.io/0xsero/exl3xpu@sha256:21412bdd… · vLLM 0.26.1.dev (separate engine from mainline vLLM; isolated dockerd required)
Artifact: turboderp/Qwen3.8-27B-exl3 rev 113cf7ab (~15.7 GiB, trellis 4.00bpw, vision tower)
Context: full native 262,144 on one card (fp8 KV pool 287,040 tokens at util 0.94) — 131K completion proven
KV cache: fp8 only (bf16 rejected) · prefix caching ON by default (~5.4× on repeated 10K prefixes)
Power: most cap-sensitive arm: +38–42% sustained decode at 230 W vs 150 W
Mixed-bit variant: SC_5.00bpw_H6 (K 4–7) on ghcr.io/sergiiob/exl3xpu-mixedk@sha256:7bb0c5cf… — the quality pick: lower KL vs bf16 than plain 4.0, ~0.89× decode (matched n=5 ABBA). Validated to ~8K prompts; launcher model qwen38-27b-sc5
Closed alternative: W4A16/GPTQ-family exports (GPTQModel, AutoRound compressed-tensors, local auto_gptq) — deterministic "30" defect, ~15 tok/s, 256K unreachable
Alibaba · Qwen
VLLM XPU · MOE 35B (3B ACTIVE)
Qwen3.6-35B-A3B Native MTP 1/2/4
MTP-preserved community GPTQ, full 128K window
Tested on 1× B70 (32GB)
Best for: everyday chat and coding — fast and cool-running.
Loading current benchmark…
MoE decode reads only the ~3 GB of active experts per token, so it self-limits to ~140 W: run it at a 150 W cap — same speed, 80 W less heat. MTP2 beats MTP4 deep in context (85.8% acceptance at 128K).
Stack & configuration
Image: vllm/vllm-openai-xpu@sha256:2c427ef4… · vLLM 0.26.1rc1.dev457 · kernels 0.1.12
Artifact: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GPTQ-Int4
Context: 131,072 · scheduler 8,192 (16,384 buys +17.6% prefill, +12.0% decode) · util 0.85 · fp16 KV
Patches: mtp_nightly then mtp_boundary
Cold input: client-proxy input rate, flat across context depth — see
benchmarks
Alibaba · Qwen
VLLM XPU · DENSE 27B · FP8 KV
Qwen3.6-27B Dense + mandatory fp8 KV
MTP-preserved community GPTQ, heretic-v2 base
Tested on 1× B70 (32GB)
Best for: long sessions that re-read the same document — follow-ups get much faster.
Loading current benchmark…
The KV cache decides this recipe: FP16 KV needs ~9.5 GiB at 128K and does not fit; fp8 halves it. In a resident 32K document session, follow-up TTFT drops from 38.2 s cold to ~4.1–4.5 s on 91% prefix hits.
Stack & configuration
Image: 2c427ef4 digest (same as the MoE) · vLLM 0.26.1rc1.dev457
Artifact: llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-GPTQ-Int4
Context: 131,072 safe ceiling — 200K loads with <5 MiB free (abort zone) · util 0.88 for MTP4
Power: dense prefill scales +52% from 165 W to 230 W; 180 W is the sustained sweet spot
NVIDIA
VLLM XPU · MAMBA2 + LATENTMOE (3B ACTIVE)
Nemotron-3.5-Lightning-30B-A3B DFlash n=7
GPTQ-INT4 G64 target + NVFP4→BF16 DFlash draft
Tested on 1× B70 (32GB)
Best for: sustained throughput on long prompts.
Loading current benchmark…
Native MTP does not work on this stack; the converted DFlash draft is the working speculator. The capacity campaign and two upstream fixes are documented in the full recipe.
Stack & configuration
Image: vllm/vllm-openai-xpu@sha256:1da0a954… · vLLM 0.26.1rc1.dev668 · kernels 0.1.12.3
Artifacts: target SergiioB/…-GPTQ-INT4-G64-sym (~18 GB) + draft …-DFlash-BF16 (~1.7 GB)
Serving: max-model-len 120,000 · prefix caching ON (production policy) · measured cells: max_num_seqs 1, prefix cache off, 150 W cap
Power draw: 149–160 W cells, 179.3 W peak interval, 68 °C max
Headline rule: use only the representative catalog record; do not select the fastest cell from a wide family
Open weights · local GPTQ
VLLM XPU · HYBRID GDN MOE
Ornith-1.5-35B-A3B MTP1 + DraftINT4
qwen3_5_moe: 30 GDN + 10 full-attn layers, 256 experts × 8 active
Tested on 1× B70 (32GB)
Best for: trying the newest hybrid MoE — self-reported results, check the evidence.
Loading current benchmark…
There is no official GPTQ for this architecture, so the target is a local experts-only quant with the MTP layer left in BF16. MixedCal-v2 calibration cuts RTN fallback from 24.76% to 10.37% — a conversion-quality win at speed parity. Self-reported E2 with raw evidence.
Stack & configuration
Image: f01e24f6 digest (same generation as Qwen3.8) · vLLM 0.27.2rc1.dev77 · WNA16 MoE backend
Artifact: SergiioB/Ornith-1.5-35B-A3B-GPTQ-Int4-sym-G128-MTP-BF16-MixedCal-v2 (~22.77 GiB, 6 shards)
Serving: default MTP1 + DraftINT4 · cache off · fp16 KV (only 10 full-attn layers need it)
Context ladder: 65,536 / 131,072 / 262,144 context ladder, loads C1 — see
benchmarks
Power A/B: 150 W holds decode, 230 W buys prefill — see
benchmarks
Meta
LLAMA.CPP SYCL · DENSE MULTIMODAL 30B
Muse-Glimmer-30B Vision + DFlash n2
UD-Q4_K_XL + mmproj + dflash GGUF trio
Tested on 1× B70 (32GB)
Best for: images + text — vision on llama.cpp, no Docker.
Loading current benchmark…
The llama.cpp path supports the reasoning channel, vision through mmproj, and block-diffusion drafting. The -DGGML_SYCL_F16=ON build flag is mandatory for usable prefill.
Stack & configuration
Engine: llama.cpp d2f83055d · IntelLLVM 2026.0 · GGML_SYCL_F16=ON
Artifacts: unsloth UD-Q4_K_XL 15.88 GB · mmproj-kquant 1.40 GB · dflash-kquant 1.63 GB
Serve: -c 131072 --parallel 1 -fa on -ctk q8_0 -ctv q4_1 -ub 8192, draft n-max 2
Failures logged: n8 aborts at 128K; n5–7 collapse to 0.30–0.37 acceptance; vLLM path still experimental
Intel · OpenVINO
OPENVINO GENAI · INT8-OV IR · NATIVE MTP5
Qwen3.8-27B INT8-OV + MTP5 Short-context champion
Hub INT8 IR · VLMPipeline · byte-identical greedy output
Tested on 1× B70 (32GB) · 230 W cap
Best for: simplest install (pip, no Docker) and the fastest short-context decode.
Documented capability on 1× B70 at the 230 W cap — MTP5 decode with greedy parity; not a ranked catalog cell methodology →
The production OpenVINO route on one B70. Native MTP5 gives 3.65× over no-MTP with greedy parity holding at every depth. In the 2026-09-15 INT4-class ranking, OpenVINO INT4-GDN8 + grafted MTP5 leads short context at 79.5 tok/s @512; vLLM stays the serving leader and llama.cpp owns the full 256K. OpenVINO's protocol ceiling is 128K.
Stack & configuration
Engine: OpenVINO GenAI 2026.5.0.0-3412 · VLMPipeline (the int8-ov export is a VL split — LLMPipeline dies on input_ids)
Artifact: OpenVINO/Qwen3.8-27B-int8-ov · draft openvino_mtp_model.xml · 25.8/32 GB resident
Serving: ChatML rendered manually (apply_chat_template=False) · num_assistant_tokens=5 · prefix caching OFF (MTP requires it)
Context ladder: 48K clean MTP+f16 KV · 64K with u8-main/f16-draft KV · 96K no-MTP paths (13.3–14.5)
Power: 202–211 W mean at the 230 W cap
Alibaba · QwenVLLM XPU · FP8 · TENSOR PARALLEL 2
Qwen3.8-27B FP8 TP2 2× B70
One model across two cards; requires worker affinity and oneCCL settings.
Tested on 2× B70 (64GB installed)
Best for: two-card rigs — one big model split across both B70s.
Follow the model-specific image and patch matrix. Multi-card memory is not automatically pooled; the topology and context limit belong to this recipe.
Intel · OpenVINO
HUB INT4 EXPORT · ARTIFACT FAILED
Qwen3.8-27B INT4-OV OpenVINO IR
Official Hub export OpenVINO/Qwen3.8-27B-int4-ov
Tested on 1× B70 (32GB) · 150 W
Quality gate failed (greedy token id 0). Speed is not a ranking cell.
The engine is production — the working OpenVINO card above runs the INT8 IR with quality gates passed. This specific Hub all-INT4 export is broken: greedy decode emits token 0 on GPU and CPU alike (openvino.genai#4467). FastDraft companion IR was never published. Do not download this artifact; use INT8 or the mixed GDN8 IR.
What was tested
Load: VLMPipeline GPU.0 LATENCY + plugin cache. generate() must use generation_config= (positional cfg is treated as images)
Decode: 128 greedy tokens run; every id is 0. Do not quote wall tok/s
CB: SchedulerConfig linear-attn cache OOMs or exceeds cache_size
Tokenizer: convert_tokenizer required on OpenVINO 2026.2/2026.3 Intel IRs
Community Labs
CASCADIA 0.2.3 · RESEARCH · NOT PRODUCTION
Cascadia on Qwen3.8-27B ov-genai + qwen35
Rust OpenAI-compatible worker wrapping OpenVINO
Tested on 1× B70 (32GB) · 150 W
ov-genai SIGSEGV. qwen35 1-stage loads, HTTP content empty.
Cascadia is not a second GPU stack. ov-genai is FFI over GenAI; qwen35 is IR surgery for Qwen3.5/3.8. The 0.2.3 Linux binary ships OpenVINO 2026.2 and crashes on the official VLM IR. After cascadia shard --num-stages 1 the worker serves, engine log ~5 tok/s, usage.completion_tokens 0.
What was tested
ov-genai: SIGSEGV in Tokenizer::setup_tokenizer / VLMPipeline CB (libopenvino.so.2620)
qwen35: 1-stage shard loads in ~46 s. --ov-* flags ignored. Prefix cache works
Quality: shard validator logits max_abs=0. Same token-0 as Python GenAI
Not tested yet: FastDraft (no IR), --cb at C≥4, dual-card --num-stages 2
MiniMax
COMFYUI 0.33.0 · PYTORCH 2.13.0+XPU · T2V
MiniMax H3 FL2VA Dual B70
GPU0 UNet+VAE (~26.9 GB) · GPU1 CLIP (~17.0 GB)
Tested on 2× B70 (64GB)
Best for: text-to-video clips — needs two B70s.
ComfyUI T2V wall times per act, OmniXPU ESIMD+FP8 kernels recipe ↗
Two cards, two roles — a role split without tensor parallelism. Native canvas 1344×768; 124 frames ≈ 5 s, 243 ≈ 10 s, longer pieces concatenate from acts. OmniXPU kernels buy 1.15× s/it (grainier); the turbo LoRA cuts sampling to 8 steps.
Stack & configuration
Engine: ComfyUI v0.33.0 checkout · PyTorch 2.13.0+xpu venv · omni_xpu_kernel 0.1.0b8.dev0 · host venv, no Docker
Weights: six files ≈ 40 GiB — UNet FP8 19.5 · CLIP (Qwen3-VL 32B) NVFP4 14.6 · video VAE 4.9 · audio VAE 0.6 · turbo LoRA 0.35 · people LoRA 0.13 GiB
Recipe: 1344×768 · 124f @ 24 fps · er_sde 8 steps · sigma shift 12+3 · turbo 0.75 · 180 W both cards
Resident runs: act 1 ≈ 650 s cold load, acts 2–5 ≈ 444 s each (H3_SKIP_UNLOAD=1)
Energy: 5-act run totals — GPU0 406 kJ, GPU1 113 kJ, ~94× realtime factor
Rendered on this host
GALLERY · N=1 WALL TIMES · 180 W
Every clip, with its wall time
Scene is finished clip length; generate is ComfyUI wall from prompt to saved file. Single observations at 180 W, 8 steps, turbo 0.75 — planning numbers, published medians need repeats.