Real latency and throughput data from production-grade benchmarks on a single GPU. Pick your workload, set your traffic, get a clear answer.
Different workloads stress different parts of the model. Pick yours to see which model fits best.
Drag the slider to your expected concurrent users. Cards show each model's predicted performance at that load.
Every point is a measured value. Latency lines break at levels where error rates exceeded 50% — those points are excluded as misleading. Throughput lines span the full range.
The moment text starts streaming. Knee = where latency spikes.
Full round-trip. Matters for non-streaming API calls.
Total generation rate across all users. Higher = more efficient.
How fast text streams once it starts. Lower = smoother UX.
One row per concurrency level. Best value per row highlighted. Error % = real HTTP failures. Incomplete % = streaming when window closed (not failures). Cells showing — had >50% errors — latency values excluded as misleading.
Auto-generated for the selected workload. Updates when you change anything above.
GuideLLM concurrent sweep — holds exactly N requests in-flight at all times,
stepping through 4, 8, 16, 32, 64, 128, 256. Each level runs for 90 seconds with 30 s cooldown between levels.
Standard OpenAI-compatible /v1/chat/completions with SSE streaming.
Each prompt is unique (random synthetic text) to defeat vLLM's automatic prefix caching.
All models run on a single NVIDIA A10 (24 GB) via vLLM with FP16 precision.
Same GPU, same serving framework, same benchmark harness — apples-to-apples.
TTFT measured from request send to first SSE token arrival (network included). E2E from request send to final token. TPOT = inter-token gap at decode. All percentiles computed across all requests in the level.