My benchmark said 42 ms. The math said 945 ms.

What I learned comparing vLLM and TensorRT-LLM on one RTX 3090 — mostly about measuring, not about engines.

The first run of my benchmark reported a time-to-first-token (TTFT) of 42 ms for a 4,096-token prompt.

It looked great. Then I did the math.

Do the math first

Prefill — processing the prompt before the first token appears — is limited by compute. For a dense model it takes about 2 × parameters × tokens operations. For Qwen3-8B and a 4,096-token prompt:

2 × 8.19e9 × 4096 / 71e12 FLOP/s  ≈  945 ms

71 TFLOPS is the RTX 3090's FP16 tensor-core peak. No software can do more math than the hardware allows. So if prefill must take about 945 ms, a 42 ms TTFT does not mean a fast engine. It means a broken measurement.

The cause was simple. My test sent the same prompt every time, and vLLM's prefix caching served the prefill from cache. The server log showed it: Prefix cache hit rate: 75.3%. Every TTFT in that run measured a cache lookup, not real work.

The fix: every request now starts with a unique ID, so no two prompts share a prefix, and the server runs with --no-enable-prefix-caching. After that, prefill measured 962.8 ms — 98% of the roofline.

The same trap exists in TensorRT-LLM: its KV cache block reuse is on by default.

Both engines are already at the limit

Decode — generating one token at a time — is limited by memory bandwidth, because every step reads all the weights again:

16.38 GB / 936 GB/s  ≈  17.50 ms per token

Both engines reach 83–98% of what the hardware allows. There is almost no room left for one engine to beat the other. That explains the next result.

"They are equally fast" is the wrong conclusion

The gap is about 2%. It is easy to write "the engines are equally fast" and move on.

But I ran each engine twice, alternating A-B-A-B so that drift over time could not favor either one. Running the same engine twice changed throughput by a median of 5.4%. The gap between engines is smaller than the gap between two runs of the same engine.

So the honest conclusion is not "they are equally fast." It is "this setup cannot tell them apart." Measuring the noise first is what makes that statement meaningful. (The likely cause of the noise is heat: GeForce cards cannot lock their clock speed.)

The backend I was not actually measuring

My second mistake became the most interesting result.

My first write-up described TensorRT-LLM as an ahead-of-time compiler. But the version I tested (1.2.0rc1) had two backends. The default, pytorch, does not compile ahead of time. The compiled path was --backend tensorrt. I was explaining one thing while measuring another.

So I measured both. That makes ahead-of-time compilation the only difference:

And the compiled engine was not faster. It got slower as concurrency went up:

At batch 1 the compiled engine is 2.4% faster — inside the noise. At batch 32 it is 18.5% slower — far outside it.

My best explanation is scheduling, not kernels. The pytorch backend adds new requests to the running batch right away. An engine compiled for fixed shapes has less freedom to reshape work at runtime. Under load, scheduling wins over specialized kernels.

Since then, newer TensorRT-LLM releases have deprecated the --backend flag of trtllm-serve, and tensorrt is no longer one of its options. My numbers show one reason that direction makes sense.

Which engine should you pick?

On this hardware and this model, the engine cannot change a number the hardware has already fixed. The choice is operational: startup time, how easy it is to deploy, and how fast you can iterate.

I would expect TensorRT-LLM to pull ahead only outside this setup:

• FP8 on Ada or Hopper GPUs, which Ampere cannot run natively
• INT4 / AWQ weights, which move decode off the bandwidth limit, so kernel quality starts to matter
• Multi-GPU tensor parallelism, where optimized communication matters
• Datacenter GPUs with locked clocks, which would shrink the 5.4% noise enough to see a 2% gap

Four lessons

1. Do the math before you trust a measurement. Five minutes of roofline math caught a 20× error.
2. Measure your noise. A difference smaller than run-to-run variance is not a difference.
3. Check that you measured what you are describing. I was explaining ahead-of-time compilation while benchmarking the backend that does not do it.
4. Change one thing at a time. Comparing the two TensorRT-LLM backends only worked because everything else stayed the same.

Code, raw results, and charts — everything reruns with one command: github.com/wu840407/llm-inference-benchmark

Setup: RTX 3090 (24 GB, 936 GB/s), Qwen3-8B in BF16, vLLM 0.27.1, TensorRT-LLM 1.2.0rc1, WSL2. Three warmup rounds, then five measured rounds per setting; two alternating runs per engine.