One Qwen Model, Two Speeds: What Smart Routing Bought Us
Aug 28, 2026 · Updated Sep 1, 2026
Ramjet routes short and long responses across two Qwen3.8 replicas while preserving prompt-cache locality, improving measured throughput by up to 79%.
Our eight-GPU Qwen3.8 server runs two copies of the same model. Until now, both copies used the same serving configuration. That was simple, but our tests showed that no single configuration was fastest for every request.
So we gave the replicas different jobs. Engine A predicts three tokens ahead with multi-token prediction (MTP3). Engine B uses standard decoding. Ramjet sits in front and chooses between them while keeping related requests close to the prompt cache they can reuse.
The result is not “speculation is always faster.” For outputs capped at 256 tokens, MTP3 increased aggregate decode throughput at every measured concurrency: from 79.1% at concurrency 1 to 7.5% at concurrency 32. But in our longer 512-token test at concurrency 32, standard decoding reached 1,844.1 tokens per second versus 1,759.2 for MTP3—about 4.8% faster.
The reversed result for longer outputs is exactly why routing matters.
Cache first, profile second
Ramjet does not blindly send every short request to A and every long request to B. A warm prompt is often worth more than a small decoding advantage, so cache locality stays ahead of the profile preference. Current load comes next. Output length breaks the remaining ties: requests for up to 256 output tokens prefer MTP3; larger or unspecified requests prefer standard decoding.
We also closed three costly gaps around that decision:
- Once a request starts generating its answer, Ramjet reduces the load it reserves for that request. The engine is no longer treated as if it were still reading the whole prompt.
- Concurrent requests with the same new prefix follow the first request to one engine. This “single-flight” step gives the cache time to become visible instead of scattering identical work across both replicas.
- A restarted engine must complete three successful health checks and remain stable for 30 seconds before it receives traffic. Reachable is not the same as ready.
Ramjet’s topology view during qualification. The two model copies have different decoding profiles but remain visible as one serving system. This screenshot is operational telemetry, not a benchmark result.
For session traffic, Ramjet can now consume each engine's live cache events and route from exact token-block residency. We rolled that mode from 1% to 10% and then 100%. In the final 100-request qualification, all requests were tokenized successfully and the exact decision agreed with the safe approximate route every time. That test proves the new path did not regress placement; it does not invent a cache-speedup number where the workload created no disagreement.
What we gained
The direct, measured upside is:
| Change | Measured result |
|---|---|
| MTP3 for 256-token work | +79.1% / +35.0% / +27.0% / +7.5% throughput at c1 / c8 / c16 / c32 |
| Standard decoding for the 512-token c32 cell | +4.8% over MTP3 |
| Mixed MTP3 + standard cache capacity | +6.9% aggregate key-value cache token capacity versus two MTP3 replicas |
| Ramjet proxy overhead in the earlier same-engine test | 0.03% at c1, 0.24% at c16 |
The less tidy upside is avoiding wasted work: fewer duplicate prompt reads, fewer requests sent to a cold cache, and no traffic sent to a replica that is technically alive but still warming up. Those gains depend on the mix of prompts and response lengths, so we will not compress them into one universal percentage.
The GPU view keeps burst shape and per-device behaviour visible. A single aggregate throughput number cannot show whether one replica is idle, saturated or waiting on a different part of the serving stack.
The useful lesson is simple: the fastest model server is not always the one with the fastest single setting. It is the one that can keep several good settings available and choose between them without throwing away cached work.
The rollout is live on our two-replica, 8× RTX PRO 6000 server. The Ramjet source, Qwen deployment, and full experiment journal are public. For the model and hardware setup, see our earlier post, Qwen3.8-Flash-Next on Eight GPUs.
Measured on two TP4 vLLM replicas of Qwen/Qwen3.8-Flash-Next-FP8 across 8× RTX PRO 6000 Blackwell Server Edition GPUs. Results describe this machine, model revision and workload; they are serving measurements, not general model-quality claims.