Quick verdict
Production GPU cost is an utilization and traffic-shape problem, not a single $/GPU-hour sticker. The short version Cipher uses when sizing inference:
- Mid-range NVIDIA (A10G / L4 class) on AWS g5/g6: roughly $0.80–$1.20 / GPU-hour on-demand class (region and size dependent). Fine for many vision, smaller LLM, and media jobs.
- H100-class on specialist clouds (RunPod / Modal and peers): often about $1.50–$4 / GPU-hour depending on availability and commitment, vs AWS p5 on-demand often in the ~$6–$7 / GPU-hour class. Specialist clouds win sticker price; AWS wins when you need VPC, IAM, and account-level residency already paid for.
- Spot / interruptible: deep discounts with a kill risk. Great for batch; painful for user-facing p95 unless you design failover.
- Serverless GPU: you trade idle cost for a cold-start tax (seconds to tens of seconds loading weights). Fine for bursty tools; bad for chat that needs sub-second first token unless you keep warm capacity.
- Managed APIs (Bedrock / OpenAI / Anthropic) usually beat self-host for spiky chat, small models, and teams with no residency or weight-ownership need. Self-host wins for steady high utilization, custom weights, or media pipelines where $/hour maps cleanly to jobs.
Cipher has shipped an AI ad-video production pipeline on commodity cloud GPU at about $0.40/hr typical run cost (A6000-class rental, job-shaped, not always-on). See the case studies page (AI ad-video production pipeline).
Last updated: August 2026. Hourly bands are directional. Re-verify AWS EC2 on-demand, RunPod pricing, and Modal pricing before budgeting.
Directional hourly bands (re-verify)
| Class | Typical SKUs | Directional on-demand / public rate | Planning note |
|---|---|---|---|
| Mid GPU (A10G / L4) | AWS g5 / g6 family | ~$0.80–$1.20 / GPU-hr | Common default for vision, TTS, smaller open weights |
| Upper mid (A6000 / A40-class rentals) | Specialist clouds | Often ~$0.40–$1.00 / hr depending on market | Cipher video pipeline case sat near ~$0.40/hr job cost |
| A100 / H100 specialist | RunPod, Modal, peers | ~$1.50–$4 / H100-hr class | Best sticker for training bursts and large open models |
| H100 hyperscaler on-demand | AWS p5 (and peers) | ~$6–$7 / GPU-hr class | Pay for AWS-native networking, IAM, enterprise path |
| Spot / interruptible | AWS Spot, community clouds | Often 50–90% off on-demand (volatile) | Batch and checkpointed jobs; not naive real-time |
These bands exclude model license fees, storage of weights, egress, orchestrator markup, and engineer time. For the full product stack (LLM API + vector + hosting), see What it actually costs to run an AI product in 2026. For Bedrock token meters when APIs win, see AWS Bedrock pricing explained.
The three taxes people forget
1. Idle always-on
A pod at $2/hr that sits warm 24×7 is about $1,440/month before anyone asks a question. If useful utilization is 10%, your effective inference cost is 10× the sticker. Always-on only wins when queue depth or latency SLO requires warm weights.
2. Cold-start (serverless / scale-to-zero)
Loading a 7B–70B model (or a ComfyUI graph with large checkpoints) can take seconds to minutes. Serverless GPU removes idle burn and adds a latency tax. Product chat with tight first-token SLOs usually needs a warm pool or a managed API. Batch and “generate when the operator clicks” media jobs tolerate cold starts well.
3. Ops and glue
Self-host means CUDA drivers, vLLM/TGI/TensorRT-LLM, autoscaling, health checks, and weight distribution. That engineering time is part of TCO. Day-one prototypes on RunPod are cheap; production SLOs on customer AWS accounts are a different project. Related runtime ownership questions: AgentCore vs Daytona vs E2B.
When managed APIs beat self-host GPU
Choose Bedrock / OpenAI / Anthropic (or similar) when most of these are true:
- Traffic is spiky (chat, agents, internal tools) and average GPU utilization would be low.
- Models are small-to-mid and quality gates pass on hosted SKUs; you do not need custom weights.
- You have no hard residency or weight-ownership requirement that forces private serving (or Bedrock in-region already satisfies it).
- Team size cannot absorb CUDA/serving ops yet.
Choose self-host / rented GPU when:
- Utilization is high and steady (or jobs are batchable overnight).
- You run custom fine-tunes, diffusion/video graphs, or models not offered (or not priced sanely) on APIs.
- Per-token API spend at your volume already exceeds a warm mid-GPU or specialist H100.
- Data residency or IP rules require weights and prompts inside your VPC / chosen cloud.
Worked shapes: real-time vs batch vs always-on
Directional monthly planning shapes in USD. Plug your own hours and region rates; these are conversation starters, not quotes.
| Workload | Traffic shape | Typical compute pattern | Directional monthly band | Planning note |
|---|---|---|---|---|
| Real-time API (chat / agents) | Spiky daytime; quiet nights | Managed API tokens, or warm small GPU + autoscale | API: tens–thousands by token volume; GPU warm pool: often $300–$2,000+ if you keep mid-GPUs hot | Cold serverless rarely meets chat SLOs; APIs usually win early |
| Batch overnight | Jobs queued; latency hours OK | Spot / specialist cloud / scheduled g5–H100 | $50–$800 for many founder-scale nightly jobs (hours × band) | Best place to use spot and H100 rentals; checkpoint everything |
| Always-on pod | Steady queue or low-latency SLO | 1× mid GPU or 1× H100 24×7 | Mid GPU ~$0.9/hr → ~$650/mo; H100 at $3/hr → ~$2,200/mo; p5 on-demand much higher | Only if utilization or SLO justifies warm capacity |
| Job-shaped media pipeline | Operator-triggered generations | Rent A6000/A10-class per job; scale to zero | Cipher case: ~$0.40/hr while running; monthly = hours used | Cost tracks output, not calendar. See case studies |
Example math (illustrative)
- Overnight batch: 4 hours/night × 30 nights × $2.00/hr H100 rental ≈ $240/month GPU (plus storage). Same work always-on would be ~$1,400+.
- Always-on g5-class: 730 hours × $1.00/hr ≈ $730/month for one GPU before replicas.
- Spiky chat at low volume: Bedrock/OpenAI token bill often lands below a warm GPU until you have a proven floor of concurrent generations.
Cipher cutover: how we size GPU vs API
- Measure a week of real traffic (or a honest synthetic load) before buying always-on capacity.
- Map jobs to bands: interactive tokens → API or warm small GPU; overnight evals/embeds → spot/batch; media graphs → job-shaped mid GPU (our video pipeline pattern).
- Put cost alarms on the account you own. Client-owned AWS or cloud accounts remain Cipher’s default so spend and residency stay with the customer.
- Re-check stickers quarterly. Specialist H100 and AWS on-demand move; last month’s band is not a contract.
Choose X if…
- Choose managed APIs (Bedrock/OpenAI/Anthropic) if traffic is spiky, models are hosted SKUs, and you do not need private weights.
- Choose AWS g5/g6 if you already live in AWS, need VPC/IAM, and mid-GPU VRAM is enough.
- Choose RunPod/Modal (or peers) if you want H100/A100 sticker efficiency for bursts, experiments, or batch, and can accept their operational model.
- Choose spot/interruptible if jobs are checkpointed and latency can stretch.
- Choose always-on pods if you have a proven utilization floor or a hard warm-latency SLO.
- Do not self-host GPUs yet if you are still changing models weekly and average utilization would be single-digit percent.
Related: AI product cost breakdown · AWS Bedrock pricing · AgentCore vs Daytona vs E2B · Applied AI Engineering
FAQ
Is H100 on RunPod always cheaper than AWS? On sticker $/GPU-hour, specialist clouds often are. Total cost depends on data gravity, egress, compliance, and engineering time. If your system of record is already AWS with residency constraints, p5 or g6 inside the account can still win.
How bad is serverless cold start? Enough to break many chat SLOs when large weights load from scratch. Fine for operator-triggered media and batch. Measure your checkpoint load time; do not assume “serverless = free latency.”
When does a $0.40/hr GPU beat an API? When work is job-shaped (minutes to hours of compute per asset), APIs do not expose the model/graph you need, or per-token pricing at that volume exceeds rented hours. Cipher’s ad-video pipeline is the pattern: pay while generating, idle at zero.
Should Australian teams prefer AWS GPUs for residency? Often yes when data must stay in a specific AWS Region and account. Specialist clouds can still fit non-regulated or carefully architected workloads. Treat residency as a first-class constraint, not a footnote.
Who can help design a GPU vs API cost shape? Cipher Projects ships production AI pipelines (agents, GPU workloads, and synthetic media) for Australian and Singapore teams under infrastructure you own.
Conclusion
Budget GPU inference in bands and traffic shapes: mid-GPU hours, specialist H100, hyperscaler on-demand, and managed APIs each win a different utilization curve. Prefer job-shaped and batch where you can, warm capacity only when SLOs demand it, and re-verify public rate cards before you lock a forecast.