LLM Inference on RTX PRO 6000 Blackwell

First time setting up the RunPod pod with these GPUs. Immediately ran into a hang — the RTX PRO 6000 reports SM120 compute capability, and vLLM's detection logic (120 // 10 = 12) doesn't recognize it as SM100 family. So a lot of optimized kernels just don't load: FlashInfer, VLLM_CUTLASS, some DeepGEMM variants — all gated behind is_device_capability_family(100). The kind of thing you only find out by running into it.

Tried to get MiniMax M2.5 NVFP4 (~126 GB) and Step 3.5 Flash running. MiniMax loaded fine.

Step 3.5 Flash couldn't run at all on this hardware/vLLM combo:

• NVFP4 — not supported for MoE models (vLLM issue #31782)
• GGUF Q4 — vLLM dequantizes to BF16, exceeding VRAM
• FP8 — weights alone exceed 192 GB capacity

This one was a mess. The MiniMax quality scores came out terribly wrong and it took a while to figure out why. Turned out to be three bugs stacking on top of each other.

Speed Results (Valid)

Original Quality (Broken)

After Fix

With the proxy fixed and max_gen_toks increased from 256 to 4096:

Tried running gpt-oss-120b (117B MoE, 128 experts, 4 active) on SM120. This model uses OpenAI's Harmony response format (reasoning_content field) instead of <think> tags, which avoids the whole think-tag mess.

Backend Compatibility on SM120

FP8 backend cascade on SM120:

1. FLASHINFER — SM100 only, unavailable
2. DEEPGEMM — disabled or produces garbage
3. TRITON — selected but produces garbage tokens
4. MARLIN — crashes with thread config errors

Quality Results (MXFP4)

Speed

Working Configuration

NCCL_P2P_DISABLE=1 vllm serve openai/gpt-oss-120b \
    --tensor-parallel-size 2 \
    --enforce-eager \
    --tool-call-parser openai \
    --reasoning-effort low

Ran mratsim/MiniMax-M2.5-FP8-INT4-AWQ (~92.5 GB) through the full benchmark suite. Found a weird perplexity artifact baked into the AWQ weights.

Quality with Reasoning Parser

Ran the same models on a B200 (SM100) to see if the kernel issues were Blackwell-wide or specific to SM120. Turns out SM100 supports different and often better kernel paths. So the problems I hit are SM120-specific — workstation Blackwell, not data center Blackwell.

I found a vLLM PR (#31089) that enables Triton on SM120, so I tested it head-to-head against the stock Marlin backend using gpt-oss-120b. The Triton patch technically works, but it completely crashes the performance.

Question 1: Is the AWQ artifact backend-dependent?

All variants show ~350 perplexity. The artifact is in the model weights, not the backend.

Question 2: Reasoning parser per-benchmark effect

This was the experiment I was most curious about. Same model family, two architectures: MoE (35B total, only 3B active per token) vs Dense (27B, all active). The MoE model worked way better than I expected.

Speed Comparison

Both models served on 2x RTX PRO 6000 (TP=2) via vLLM. “rate=N” means N new requests arriving per second. Throughput is total output tokens per second across all in-flight requests. TPOT (time per output token) is per-request latency—how long a user waits between each streamed word. TTFT (time to first token) is how long until the first word appears after sending a prompt.

Simultaneous Full-Context Sessions

After loading weights, remaining VRAM determines how many 131K-token conversations can run at the same time. MoE activates fewer parameters per token, producing smaller KV cache entries and leaving more room for concurrent sessions.

The quality scores from Project 9 were suspiciously bad (~29% IFEval, ~9% MATH Hard). Turns out thinking mode was silently corrupting all the benchmark results. I'm still not fully sure if this is a fundamental limitation of the benchmark tooling with reasoning models, or just a configuration problem — but disabling thinking fixed everything.

The Problem

1. Chat template injects <think> as a prefix (not generated by the model)
2. Model only generates </think> as a closing tag
3. vLLM's --reasoning-parser qwen3 expects both tags → fails silently
4. Thinking content remains in content field, corrupting extraction
5. 96% of MATH samples had untagged reasoning that received no stripping
6. Greedy decoding with 1024-token cap causes thinking to exhaust budget before answer

The Fix

--default-chat-template-kwargs '{"enable_thinking": false}'

Results

Good to know: Qwen3.5-35B-A3B handles images natively (early-fusion, same weights, no separate adapter). I tested vision on 1 GPU vs 2 to see if splitting helps.

Text Quality (No Degradation)

Vision Quality

Speed

Compared BF16, FP8, and AWQ-4bit of Qwen3.5-35B-A3B on a single card to see how much speed you gain and how much quality you lose.

Quality

*Scores collapsed due to thinking mode being enabled. FP8 perplexity is essentially identical to BF16 (9.58 vs 9.56).

Speed

Re-ran all models with consistent methodology: vllm bench serve with 512 input / 256 output tokens, same gpu_memory_utilization=0.90 and max_model_len=131072 across the board. Added Qwen3.5-122B-A10B (the bigger MoE) to the lineup.

Speed (512in/256out, 2x RTX PRO 6000)

Quality

*MATH Hard failed for Qwen3.5-35B variants due to missing math-verify dependency (installed mid-run).

KV Cache & Concurrency

At gpu_memory_utilization=0.90 and max_model_len=131072, how many concurrent 128K-token conversations fit in the remaining VRAM after loading weights:

Key Takeaways

Two-part investigation: (1) why SM120 is stuck on the Triton fallback for FP8 MoE, and (2) whether NVFP4 quantization unlocks better performance via native SM120 kernels.

SM120 vs Datacenter Blackwell

The RTX PRO 6000 (SM120) shares the Blackwell brand with B200/B100 (SM100) but has a fundamentally different instruction set:

SM120 has FP8 and FP4 tensor cores but can only access them via mma.sync (SM89-era instruction), not the high-throughput wgmma or tcgen05 used by optimized backends.

Backend Feasibility for FP8 MoE on SM120

The NVFP4 Path Forward

SM120 has one capability SM90 does not: FP4 tensor cores. FlashInfer CUTLASS already ships SM120-specific FP4 MoE kernels that use mma.sync with FP4 data types and fit within 99 KB shared memory.

NVFP4 Backend Comparison (Qwen3.5-122B-A10B)

Tested two open-source AI coding agents — opencode and hermes-agent — against local models on a practical task: clone a real project, understand it, install dependencies, and investigate vLLM integration.

Round 1: Small Models

Round 2: Qwen3.5-122B-A10B (2x GPU)

*opencode hit context window limits (32K output default consumes most of 65K context).

FP8 vs NVFP4 for Agentic Use