AI Benchmarks — dooner.tech

ai01 · AMD Ryzen Threadripper PRO 9965WX · 24 cores / 48 threads · 125 GiB RAM

RTX PRO 6000 Blackwell

384 GB

VRAM class total

450 W

persistent cap per GPU

480,000

active max context

The active service is GLM-5.2 TP4+DCP4 on all four GPUs at port 5001. DeepSeek results below are retained as labeled historical configurations.

Current Deployment

GLM-5.2 753B Hybrid

active on ai01 · GPUs 0-3 · TP4+DCP4 · :5001 · verified 2026-07-14

activeMXFP8NVFP4 / NF3 hybridvLLM v1.3

753B

model parameters

active GPUs

480K

max input tokens

131K

max output tokens

Current profile controls:

MODEL=GLM-5.2 MODEL_BASE=madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid ENDPOINT=http://192.168.0.213:5001/v1 GPUS=0,1,2,3 TP=4 DCP=4 MAX_MODEL_LEN=480000 MAX_OUTPUT_TOKENS=131072

All four RTX PRO 6000 Blackwell cards serve the hybrid GLM-5.2 profile. LiteLLM exposes it through the stable alias ai01-glm52.

Historical TP2 v10 Results

Archived DeepSeek DSpark results: Lucifer CUTLASS, two GPUs at 450 W, FP8 KV, graph capture 192, and 15-second sustained-decode cells after warmup.

1M-capable DSpark profile

llm-inference-bench 0.4.30 · fully shared-prefix decode matrix

226.1

zero-context C1 tok/s

1,105.5

zero-context C16 tok/s

1,655.3

zero-context C32 tok/s

308.7

coding median tok/s

Context	C1	C8	C16	C32
0	226.1	734.2	1,105.5	1,655.3
8K	268.7	875.0	1,193.4	1,692.7
16K	248.7	855.8	1,330.9	1,861.5
32K	250.4	776.7	1,225.2	1,673.4

11,656

8K cold prefill tok/s

10,755

64K cold prefill tok/s

9,687

128K cold prefill tok/s

1,149,077

reported KV token budget

The decode matrix used 100% shared context. It measures prefix-sharing service throughput, not 32 independent 32K contexts.

Raw matrix JSON → Methodology and caveats →

Validated near-million-token cold prefill

1,044,482 actual prompt tokens · one accepted cold sample per mode

Mode	Prompt tokens	TTFT	Prefill tok/s
Standard MTP2	1,044,482	261.157 s	3,999
DSpark	1,044,482	285.578 s	3,657

These prove both profiles can ingest a real near-1M prompt. They are not a matched speed comparison: memory settings, graph profile, and speculative mode differ.

Three-GPU TP3 Engineering

Validated on three RTX PRO 6000s at 450 W each. These results use the pinned v9 stack and a 524,288-token profile.

Standard MTP2 + B12X A16 at TP3

matched production validation at the time · FP8 KV · B12X only at exactly 64 KiB

213.2

C1 decode tok/s

1,284.0

C16 decode tok/s

1,875.3

C32 decode tok/s

2,689.5

C64 decode tok/s

11,645

8K cold prefill tok/s

10,580

64K cold prefill tok/s

3,287,424

KV capacity tokens

228.5

coding median tok/s

TP3 architecture → Raw JSON →

B12X odd-world-size all-reduce

TP3 transport fix · eager, CUDA graph, and multistream coverage

23,808

TP3 launches in soak

11.97 s

soak runtime

64 KiB

final one-shot route

2,745.4

routing confirm C64 tok/s

B12X minimum	Zero-context C1 tok/s
16 KiB	210.7
32 KiB	222.6
64 KiB	227.0

Root cause: the signal slab was published through CUDA IPC before zero-fill. A fast rank could post its first barrier arrival and a slower rank's late memset could erase it.

Read the all-reduce investigation → Download the minimal patch →

DSpark N5 vs standard MTP2 at TP3

matched host, power, scheduler, KV dtype, graph mode, and all-reduce routing

Zero-context decode	C1	C8	C16	C32	C64
DSpark N5	219.7	813.4	1,130.0	1,460.9	2,282.2
Standard MTP2	213.2	899.1	1,284.0	1,875.3	2,689.5

+30.0%

DSpark coding median

-8.9%

DSpark KV capacity

-22.1%

DSpark C32 decode

-15.1%

DSpark C64 decode

DSpark won the sequential coding probe. MTP2 retained more KV and won the high-concurrency service workload, so production returned to MTP2 after the A/B.

Full TP3 A/B and correctness fixes →

Two-GPU v9 Backend Comparison

B12X A16 vs Lucifer CUTLASS

TP2 · standard MTP2 · 450 W per card · 20 seconds per decode cell

Cold prefill	8K	64K	128K	256K
B12X A16	10,925	9,963	9,123	7,851
Lucifer CUTLASS	11,910	10,672	9,178	7,346

+9.0%

Lucifer 8K prefill

+7.1%

Lucifer 64K prefill

+11.1%

Lucifer 8K C16 decode

-6.4%

Lucifer 256K prefill

There was no universal winner. Lucifer led short-to-medium prefill and several C16 cells; B12X was stronger through much of the 128K/256K matrix.

Full 20-cell decode matrix →

Measurement Rules

What counts as reusable

warm kernels, explicit context sharing, actual token counts, and raw artifacts

1. Wait for every model endpoint. 2. Settle 30 seconds after the final server reports ready. 3. Run unreported C1/C16/C32/C64 and prefill warmups. 4. Settle another 30 seconds and mark the log boundary. 5. Reject measured phases with JIT compilation or cache misses. 6. Report shared vs unique context percentage. 7. Publish prompt tokens, TTFT, sample count, launch profile, and raw JSON.

Two early 1M attempts reported 3.3M and 6.9M tok/s with roughly one-second TTFT and no server-validation samples. They are retained as rejected artifacts, not benchmark records.

Long-context methodology → Artifact hashes and rejected-run labels →

Prior Platform

PVE03 PCIe topology study

retired host · retained as historical hardware data · 2026-07-04

+44%

P2P memcpy after PIX move

+52%

ring bandwidth

+25-35%

prefill gain

+25-40%

C1 decode gain

This host no longer serves the active DeepSeek alias. Its NODE-to-PIX experiment remains useful evidence that physical PCIe placement and ACS routing can dominate a two-card inference result.

Earlier PVE03 write-ups →

Current Architecture

OpenWebUI / agents | v LiteLLM stable model aliases | v ai01 :5001 GPUs 0-3 GLM-5.2 753B hybrid / TP4+DCP4 All four RTX PRO 6000 Blackwell cards: persistent 450 W cap