How we model the cost to run a query

The formula

In plain terms: a rented GPU costs a fixed amount per hour; divide that by how many output tokens it produces per hour (its throughput × utilization), and you get the compute cost per token. Multiply up to a million tokens, then scale by an overhead multiplier (≥ 1) for the non-GPU infrastructure (host, networking, storage, load-balancing). The overhead is an assumption, presented as an editable slider, not a cited constant.

Input tokens cost less to serve

Generating output (decode) is the expensive part. Reading the prompt (prefill) is compute-bound and highly parallel, so each input token costs roughly 6× less to serve than an output token — an assumption, not a measured constant. The calculator prices a full query as output-token compute plus this cheaper input-token compute, and compares it to the published price for the same input:output token mix. Input-heavy workloads (RAG) therefore cost providers less per query than an output-only view implies.

The per-GPU throughput caveat

Throughput here is the aggregate output rate of a single GPU at high batch size (FP8), not a whole 8-GPU node and not the speed one user sees. This is the easiest figure to get wrong by an order of magnitude: One H100 ≈ 2,000 tok/saggregate; an 8×H100 node is ~17,000–24,000 tok/s; a single user's stream is ~89 tok/s. The calculator's default and every figure below are per GPU.

Shared assumptions

These three inputs are the same across all model classes.

• GPU rental$3/GPU-hr (range $2/GPU-hr–$7/GPU-hr)

  Per-GPU on-demand; ~$2 neocloud floor, ~$7 AWS. Spot ~$1.50.

  RunPod/Lambda/AWS on-demand H100 (2026) ↗
• Utilization30% (range 20%–50%)

  Production duty cycle; 40% gives ~2× the cost/token of 80%.

  NVIDIA Perspectives, inference cluster economics (2026) ↗
• Overhead1.4× (range 1.3×–1.5×)

  Non-GPU infra (host, networking, storage, load-balancing). An assumption, not a measured constant.

  Assumption (no canonical source) ↗

Per-class figures

Only throughput and the published price ranges vary by model class.

• Frontier (GPT-5.5, Claude Opus 4.x-class)
  Throughput: 2000 tok/s per GPU (1500–3000)
  Output price: $15–$30/1M · Input price: $2.5–$10/1M
• Mid-tier (GPT-5.4-mini, Claude Haiku 4.5-class)
  Throughput: 2400 tok/s per GPU (1800–3500)
  Output price: $4–$8/1M · Input price: $0.75–$3/1M
• Small / efficient (GPT-5.4-nano, Gemini Flash-Lite-class)
  Throughput: 3000 tok/s per GPU (2200–4500)
  Output price: $0.5–$2.5/1M · Input price: $0.1–$1/1M
• Open-weight (DeepSeek V4, GLM-4.6, Kimi K2, MiniMax-class)
  Throughput: 3000 tok/s per GPU (2200–4500)
  Output price: $0.3–$2.3/1M · Input price: $0.14–$0.6/1M