Self-hosting is not a cheaper API — it is a different cost curve with an engineering team attached. Start with the memory maths, because that decides whether your plan fits on one card at all.
Someone else owns the GPUs, the uptime and the model updates. You pay per token and scale to zero. The default, and usually the right answer.
Cost shape: purely variable. Zero at zero traffic.
Dedicated GPUs or a private deployment on an inference platform. You get isolation, predictable throughput and data controls, without owning a cluster.
Cost shape: fixed per hour. Wastes money when idle.
Rented pods, a colocated box, or a rack in your office. Full control over data, weights and versions — and full responsibility for everything that breaks.
Cost shape: fixed, plus the people who keep it running.
Weights are the part everyone calculates and the part that matters least once context gets long. The estimator adds the KV cache, which is what actually runs you out of memory.
Or type parameters below
≈4.6 KB per token per B params at FP16
Weights = parameters × bytes-per-parameter × 1.15 for runtime overhead. KV cache is a rule of thumb for grouped-query-attention transformers — mixture-of-experts models scale KV with layers and heads, not total parameters, so treat their estimate as rough. Add 2 GB for CUDA context and activations. Real deployments need headroom on top for batching.
| GPU | VRAM | Per hour | Per month (24/7) | Verdict |
|---|
On-demand dedicated pods. Preemptible and reserved capacity run cheaper; contract pricing is a sales call.
| GPU | VRAM | Per hour | Per month (24/7) | Note |
|---|
Pick by workload, not by benchmark. They are all free; they differ in how much of your life they cost.
| Engine | Best at | Notes | Ops load |
|---|
Halving memory is not free. The loss is small on most tasks and obvious on maths and long-form reasoning.
| Format | Bytes / parameter | What it costs you |
|---|