Every ComputeSDK benchmark runs the same workload on every provider, repeats it on a schedule, scores outcomes over best cases, and publishes the raw results so anyone can audit or reproduce it.
The rules every benchmark follows, regardless of the provider or family.
The same script, payload, and model hit every provider, addressed the way its own API expects. Nobody gets a tuned or special path.
Every suite runs many iterations per provider, automated on a recurring schedule, so results reflect steady state rather than one lucky request.
Every family has its own scoring function, but none reward a single fast run. Latency is blended from median and tail percentiles, completion is counted in phases, and coverage or quality is measured where those matter. All scored families multiply by success rate so providers that fail often lose real points.
Every run is committed as JSON to the public benchmarks repo, next to the code that produced it. Nothing is cherry-picked or held back.
What we measure today, grouped by category. For the latest providers and exact workload, visit the live benchmark.
A real-world developer workload: install system packages and Node.js, download and unpack Bun, shallow-clone the opencode repo at a pinned commit, run bun install, then bun typecheck. The benchmark records the seven phases and reports median durations for each.
Scoring: The composite score is the median number of the seven phases completed, shown as a fraction like 7/7. Per-phase durations are reported separately as median seconds.
Time to Interactive from compute.sandbox.create() to the first successful runCommand(). Each run launches 100 sandboxes concurrently and scores latency against a fixed 10-second ceiling.
Scoring: Each TTI percentile is scored against a 10-second ceiling and weighted 60% median, 25% P95, 15% P99, then multiplied by success rate.
Upload/download latency and throughput for object storage. The benchmark uploads and downloads files of configurable size, capturing upload latency, download latency, and throughput in Mbps. Results shown use the 10 MB file size.
Scoring: Download (55%: median 35%, P95 15%, P99 5%), upload (40%: median 25%, P95 10%, P99 5%), and throughput (5%) are converted to a 0–100 scale—latency against a 30-second ceiling, throughput on a 1–1000 Mbps scale—then multiplied by success rate.
Bucket branch operations. A small dataset is seeded, then the benchmark times snapshot creation, fork from snapshot, fork from live, and the first read from the fork to confirm time-to-usable. All resources are torn down in a finally block.
Scoring: Each operation's median latency is scored against a 60-second ceiling with weights 40% snapshot create, 35% fork from snapshot, 15% fork from live, 10% first read, then multiplied by success rate.
Headless browser session startup. Each iteration provisions a browser, connects over CDP, navigates to a target page, and releases the session, measuring create, connect, navigate, release, and total end-to-end latency.
Scoring: Latency percentiles are scored against a 10-second ceiling with weights 40% total median, 30% create median, 20% total P95, 10% total P99, then multiplied by success rate.
Per-action throughput inside a running session. Each session executes ten sequential actions on Wikipedia — navigate, waitForSelector, screenshot, textContent, click, goBack — and records individual action times and actions per second across 100 sessions.
Scoring: Actions/sec median (40%), task duration median (25%), task duration P95 (20%), and screenshot median (15%) are combined, then multiplied by success rate.
Cold and warm LLM request latency. Every gateway is hit with the same prompt and model per family, measuring cold end-to-end latency, warm time-to-first-token, output tokens per second, and real DNS/TCP/TLS socket phases. A direct-to-provider baseline is included as a control.
Scoring: Cold E2E median (30%), warm TTFT median (30%), cold E2E P95 (15%), warm TTFT P95 (15%), and output tokens/sec (10%) are combined, then multiplied by success rate.
The numbers we use when they fit the benchmark. Not every family reports every metric.
For families that can be reduced to a single comparable number, the composite score ranges from 0–100. Most latency-driven families convert measurements to a 0–100 scale against a fixed ceiling, blend them with explicit weights, and multiply by success rate. DAX reports the fraction of the seven phases completed (e.g., 7/7). Families that are better compared as tables or pass/fail matrices are shown that way instead.
The middle value of the iteration distribution. Half the runs were faster and half were slower, making it the primary signal for the typical experience.
The value below which 95% of observations fall. It captures the tail latency that users experience during busy or noisy periods.
The value below which 99% of observations fall. It measures extreme outliers and how consistent a provider is under repetition.
The fraction of iterations that completed without error. For scored families, a provider that fails one in ten requests loses 10% of its composite score. For other families it is reported as a reliability signal.
Context-aware speed: Mbps for storage transfers, actions per second for browser sessions, and output tokens per second for AI gateways. Higher is better and it is blended into the composite where relevant.
Every benchmark is built to be reproducible. The harness code, configuration, and raw JSON outputs all live in the public benchmarks repository.