AI Gateway Benchmarks - Anthropic
Benchmark AI gateway latency and token throughput, measured against a direct-to-Anthropic baseline. Inspired by this benchmark from Ronny Badilla.
Powered byProvider Leaderboard
"Anthropic (Direct)" is a no-gateway control, not a competing gateway — it isolates how much latency each gateway adds on top of the underlying model provider.
Provider VarianceComposite Score
"Anthropic (Direct)" is a no-gateway control, not a competing gateway.
Delta vs. the Anthropic (Direct)'s composite score. Positive is better than baseline; negative is worse.
Performance Over Time
Composite Score
Detailed Metrics
Provider | Score | Cold E2E (med) | Warm TTFT (med) | Tokens/sec (med) | DNS (med) | TCP (med) | TLS (med) | Success |
|---|---|---|---|---|---|---|---|---|
| Anthropic (Direct)Baseline | 89.8 | 561ms | 504ms | 53 | 1.20ms | 1.02ms | 4.08ms | 100% |
| OpenRouter | 89.7 | 590ms | 497ms | 53 | 1.33ms | 1.12ms | 3.90ms | 100% |
| Cloudflare AI Gateway | 89.4 | 653ms | 508ms | 51 | 1.21ms | 0.99ms | 6.41ms | 100% |
| Pydantic AI Gateway | 89.1 | 646ms | 624ms | 50 | 1.22ms | 1.79ms | 4.74ms | 100% |
| Vercel AI Gateway | 89.1 | 692ms | 556ms | 54 | 0.96ms | 3.38ms | 13.47ms | 100% |
| Concentrate AI | 89.1 | 729ms | 620ms | 51 | 1.04ms | 1.06ms | 5.68ms | 100% |
| Blazerail | 88.7 | 738ms | 595ms | 50 | 12.96ms | 1.58ms | 3.54ms | 100% |
| ngrok AI Gateway | 88.3 | 695ms | 617ms | 49 | 1.20ms | 13.88ms | 17.13ms | 100% |
| LLMgateway | 88.1 | 801ms | 759ms | 46 | 1.20ms | 1.74ms | 6.09ms | 100% |
| LLM API | 87.2 | 1177ms | 775ms | 44 | 1.15ms | 87.08ms | 87.72ms | 100% |
| Neon | 85.6 | 1228ms | 1052ms | 41 | 1.38ms | 13.85ms | 31.40ms | 100% |
Want to see a gateway added?
Methodology
Every run executes on a GitHub Actions runner hosted by Namespace, physically located in Northern Virginia, US. Since results are a property of the network they're measured from, treat rankings as relative to that vantage point rather than a universal number — see Limitations in the full methodology.
What We Measure
Every gateway is hit directly with the same prompt and model (Claude Haiku 4.5), addressed the way each API expects it to be addressed — no gateway is proxied through another. Each request is either cold (a brand-new TCP+TLS connection, opened from scratch) or warm (a second request reusing an already-open keep-alive connection).
Cold requests are broken into DNS, TCP, and TLS phases (real socket timestamps, not estimates), plus time to first response byte and first streamed token. Every request also records output tokens/sec and success/failure.
This family's no-gateway control is a direct call to Anthropic's own API, not a competing participant — it isolates how much latency each gateway adds on top of the underlying model provider. See the other target-model families on the AI Gateway Benchmarks overview. Full methodology, including how "cold" and "warm" are defined and their limitations, is documented in the public benchmarks repo.
How We Score
The Composite Score combines cold end-to-end latency and warm time-to-first-token — the short-lived-process case and the steady-state case — with generation throughput, all multiplied by success rate.
- • Cold E2E median: 30%
- • Warm TTFT median: 30%
- • Cold E2E P95: 15%
- • Warm TTFT P95: 15%
- • Tokens/sec: 10%