When Self-Hosting an Open-Weight Model Is Actually Cheaper Than an API

Last updated: August 2026

Most people arrive at this question with the arrow pointing the wrong way. Open weights do not mean cheap inference. They mean optional inference. Moonshot publishes the Kimi K3 weights and charges $3.00 per million input tokens and $15.00 per million output for its own API. DeepSeek publishes V4 Flash and charges $0.14 and $0.28. Artificial Analysis scores those two 60 and 52 on its Intelligence Index, K3 at max effort against V4 Flash 0731 at max reasoning effort. Treat those two numbers as soft: the same pair read 57 and 50 in an Artificial Analysis post from July, because the score moves with both the effort setting and index revisions. All read 11 August 2026. None of it tells you what the weights cost to run on your own metal.

That is a separate problem, and it splits hard by model size. The K3 checkpoint alone is 1.56 TB, so the minimum vLLM states is a whole 8x B300 node. GLM-5.2 ships BF16 weights that come to roughly 1.5 TB. Meta's Muse Glimmer 30B fits in a 17 GB file on one workstation card. That is the segmentation worth handing a client. Buy the trillion-parameter tier through an API. Start the on-prem conversation at the 30B tier.

Every break-even figure circulating on this topic disagrees with the others by five to ten times, because almost nobody shows the working. Here is the working. Published GPU hourly rates, published token prices, the cost lines that never make the spreadsheet, and a calculator you can point at your own traffic.

Open weights price the download, not the inference

A model's license and its API price have nothing to do with each other. Every model below ships open weights, and they span roughly 40x on blended price. The most expensive one is also the only one here whose license carries a revenue gate. What the vendors charged on 11 August 2026, read off their own pages: platform.kimi.ai, api-docs.deepseek.com, together.ai/pricing, openrouter.ai.

Blend those at the 7:2:1 cache-hit to cache-miss to output mix Artificial Analysis uses and K3 comes out at exactly $2.31 per million tokens, DeepSeek V4 Flash at $0.058. Close to a 40x spread between two open-weight models eight index points apart. If your reason for looking at self-hosting is the bill, changing model is the bigger lever, and it takes an afternoon rather than a quarter.

One caveat on DeepSeek. Their API docs carry a standing notice that they plan to raise overall pricing in the near future, with a significant increase expected. That is the vendor's own wording, and it has sat there for months without a date. Do not build a three-year case on $0.14.

What the trillion-parameter tier actually costs to serve

Kimi K3 ships in native MXFP4 with MXFP8 activations. The Hugging Face repo is 96 safetensors shards, about 1,561 GB decimal, 1.42 TiB. That single number decides your topology, and topology decides everything else.

• One 8x H200 node is 1,128 GB. 8 x 141 GB. The weights do not fit, and it is not close. Every blog post that casually suggests an H200 box for K3 has not done this subtraction.
• One DGX B200 node is 1,440 GB per NVIDIA's spec sheet, 8 x 180 GB. Still short, before a single token of KV cache.
• One 8x B300 node is 2,304 GB. 8 x 288 GB. This is the floor vLLM actually publishes: at least one 8x B300, or a GB300 NVL72. Not two H200 nodes, which is the number that circulates.
• 16x B200 is 2,880 GB and vLLM supports that too, but you are now across two nodes and need an all-to-all backend to shuttle activations between the 896 experts: --all2all-backend deepep_v2 over RDMA, or flashinfer_nvlink_one_sided over NVLink. That is a fabric problem, not a GPU problem, and it is the part nobody prices.

Price the cheap version. Eight B300s at RunPod's Secure Cloud on-demand rate of $7.39 per GPU-hour, read 11 August 2026, is $59.12 an hour, about $43,158 a month. Against K3's blended $2.31 per million tokens you need roughly 18.7 billion tokens a month to cover the rent alone. The 16x B200 route is worse: $5.89 a card is $94.24 an hour, about $68,795 a month, and 29.8 billion tokens to break even. No engineer costed in, no fabric costed in, and you cannot generally rent a coherent multi-node allocation at single-GPU on-demand prices anyway.

GLM-5.2 is the same story one notch down. The Hugging Face card's safetensors metadata totals 753B parameters; vLLM's own recipe describes the model as roughly 743B total with 39B active. Either figure puts it near 1.5 TB at BF16 and roughly 750 GB at FP8, which does fit the single 8x H200 node the vLLM recipe targets. Eight H200s at $4.39 is $35.12 an hour, about $25,638 a month.

The tier where self-hosting is a real conversation

Below about 40B parameters the hardware story changes completely. One GPU. No interconnect, no multi-node scheduler, no fabric to debug at 3am. Three models landed in that band during 2026.

The Muse Glimmer degradation figures come from Meta's model card, which is a good template for how this should be published: both quants named, 0.2% and 1.0% against BF16 averaged over 15 benchmarks, and the memory envelope each targets. Vendor claim, not independent measurement, but falsifiable. On accuracy, Meta publishes 76.0 on SWE-bench Verified and 83.5 on GPQA Diamond for Glimmer; Google publishes 85.2 on MMLU-Pro and 84.3 on GPQA Diamond for Gemma 4 31B. Both self-reported, neither independently replicated at the time of writing, and none of it is a throughput number, which is the only figure your cost model actually needs.

This is the tier where I would let a client run the numbers. Everything above it, I would not.

The break-even, with the working shown

You want both sides from one vendor, otherwise you are pitting somebody's loss-leader token price against somebody else's list-price GPU. Together AI publishes both: Gemma 4 31B serverless at $0.39 in and $0.97 out per million tokens, and dedicated capacity at $5.49 per H100 per hour. The specialist clouds set a cheaper floor.

All read 11 August 2026, on-demand only. Spot rates are excluded: they move weekly and you cannot run production on capacity that evaporates mid-request. Take the cheapest credible H100 line, RunPod at $2.99. Assume a 10:1 input-to-output mix, roughly what a retrieval-heavy app or a coding agent looks like.

The engineer is the line that flips the verdict, and it is the line that gets left out. On GPU rent alone the break-even sits near 1,800 input tokens a second sustained, which is a plausible ask of one H100 serving a 31B model at FP8. Measure it on your own traffic before believing it, including mine. Add a fifth of one person and you need 4,400 a second, forever, at 100% duty cycle. Add the second GPU that keeps a 2am driver crash from being an outage and you need 6,100 through one active card. That is not an inference platform, it is a dare.

Picture 11.6 billion input tokens a month. At 8,000 tokens of context per call that is 1.45 million calls, roughly 48,000 a day. For a coding agent chewing 40,000 tokens a turn it is 290,000 turns. If your client is not doing that, no amount of GPU shopping changes the answer.

The cost lines that never make it into the spreadsheet

Four of these, roughly in order of surprise.

• Utilization below 100%. Idle time does not change the break-even token count, it shrinks the window to hit it in. If load only arrives 50 hours a week you pay for 730 and use 217, so 4,400 tokens a second becomes 14,900 during business hours. That usually ends the conversation.
• Idle overnight and at weekends. Scaling to zero sounds like the fix. For a 31B model it is not: a cold start pulls tens of gigabytes onto the card, so the first request after an idle window waits minutes. You keep a replica warm, which is the capacity you were trying not to pay for.
• The fraction of a senior engineer who owns the serving stack. Somebody upgrades vLLM, chases a CUDA version mismatch, retunes --max-num-seqs when traffic shifts, and gets paged when the endpoint OOMs. I have never seen it come in under 20% of one person in production. Twenty percent of a $200k fully loaded engineer is $3,333 a month, more than the GPU.
• Redundancy. One GPU is a single point of failure with no SLA behind it. A serverless API comes with a fleet you are not billed for. Care about uptime and the GPU line doubles while each card runs at half duty.

A fifth item is not a cost but behaves like one: you give up changing model with a config edit. When something better ships in November, the API user edits a string. You re-run capacity planning.

The KV cache is what breaks capacity plans

Weights are the number everybody quotes. The KV cache decides how many users fit on the card. It scales with concurrency times context length, it does not shrink when you quantize, and at long context it can be several times larger than the model.

The formula is simple: 2 (for K and V) x layers x KV heads x head dimension x 2 bytes, per token. What varies wildly is what you multiply. Values below read from the published configs.

Both concurrency figures assume weights resident: Gemma at FP8 (30.7 GB), Glimmer at its 17 GB 4-bit quant, 4 GB for CUDA graphs, activations and allocator fragmentation. Sixteen times the concurrency on identical hardware, from two models about a billion parameters apart. Gemma 4 buys long-context quality with 16 KV heads and a 256-wide head dimension. Muse Glimmer buys throughput with 2 KV heads and aggressive sliding windows. Size a box from parameter count and you will be wrong by 16x, in production.

The cache also stays in BF16, so quantizing weights to 4-bit does nothing for it. You can switch on FP8 KV cache in vLLM and halve it, but run your evals against that first. It is a real accuracy tradeoff, not a free win.

#!/usr/bin/env python3
"""KV cache VRAM straight from a model config.json.
Values below were read from google/gemma-4-31B-it and
meta-models/Muse-Glimmer-30B on 11 August 2026."""

GB = 1e9   # decimal, to match how vendors quote VRAM


def kv_bytes_per_seq(n_layers, kv_heads, head_dim, ctx,
                     sliding_window=None, sliding_ratio=0, dtype_bytes=2):
    """sliding_ratio = sliding layers per full-attention layer, so 5 encodes
    the 5:1 pattern in Gemma 4. dtype_bytes stays at 2: the KV cache is BF16
    regardless of how hard you quantized the weights, unless you explicitly
    enable FP8 KV and accept the accuracy cost."""
    per_layer_per_token = 2 * kv_heads * head_dim * dtype_bytes   # K and V
    full_layers = n_layers // (sliding_ratio + 1)
    sliding_layers = n_layers - full_layers

    total = full_layers * per_layer_per_token * ctx
    if sliding_window:
        total += sliding_layers * per_layer_per_token * min(ctx, sliding_window)
    else:
        total += sliding_layers * per_layer_per_token * ctx
    return total


# layers, kv_heads, head_dim, sliding_window, sliding_ratio, max_ctx, weights
MODELS = {
    "Gemma 4 31B (FP8)":     (60, 16, 256, 1024, 5, 262144, 30.7 * GB),
    "Muse Glimmer 30B (4b)": (52,  2, 128, 2048, 3, 131072, 17.0 * GB),
}

H100_80GB = 80 * GB
RUNTIME_OVERHEAD = 4 * GB   # CUDA graphs, activations, allocator fragmentation

for name, (L, kvh, hd, win, ratio, max_ctx, weights) in MODELS.items():
    for ctx in (32768, max_ctx):
        per_seq = kv_bytes_per_seq(L, kvh, hd, ctx, win, ratio)
        free = H100_80GB - weights - RUNTIME_OVERHEAD
        print(f"{name:23s} ctx={ctx:>6}  {per_seq/GB:6.2f} GB/seq  "
              f"{int(free // per_seq):>4} concurrent on one 80GB H100")

# Gemma 4 31B (FP8)       ctx= 32768    6.21 GB/seq     7 concurrent on one 80GB H100
# Gemma 4 31B (FP8)       ctx=262144   43.79 GB/seq     1 concurrent on one 80GB H100
# Muse Glimmer 30B (4b)   ctx= 32768    0.52 GB/seq   113 concurrent on one 80GB H100
# Muse Glimmer 30B (4b)   ctx=131072    1.83 GB/seq    32 concurrent on one 80GB H100

4-bit stopped being a community afterthought

The reason the numbers above are not far worse is that frontier open models now ship quantized by the lab that trained them. K3's MXFP4 weights are quantization-aware trained from the supervised fine-tuning stage onward, not converted afterwards. DeepSeek's V4 Flash card puts its MoE experts in FP4 and most other parameters in FP8. These are not community GGUFs made after the fact.

The consequence is arithmetic. 2.8 trillion parameters at BF16 would be 5.6 TB. At four bits plus block scales, K3's checkpoint is 1.56 TB. That is the difference between a deployment that fits one Blackwell node and one that would need three. It also retires the reflex of penalizing quantized weights on quality: this is what the lab trained and what the lab benchmarked.

The old caution still applies to community quants of models that shipped BF16. Unsloth's 2-bit UD-IQ2_M of GLM-5.2 is a 239 GB file wanting about 245 GB of combined RAM and VRAM, and Unsloth's own figure puts it near 82% of the full model's accuracy. That is a usable number and an honest one, but it is still a different model from the one on the benchmark table, and nobody has run your evals against it.

A calculator you can run on your own numbers

Take the structure, not my scenario, and put your own traffic in it. This reproduces every figure in the break-even table above.

#!/usr/bin/env python3
"""Break-even: a rented GPU vs serverless tokens.
Every price hard-coded below was read from a vendor page on 11 August 2026.
Change them. They move."""

HOURS_PER_MONTH = 730                      # 365 * 24 / 12
SECONDS_PER_MONTH = HOURS_PER_MONTH * 3600 # 2,628,000


def monthly_selfhost(gpu_hourly, gpu_count, engineer_fraction,
                     engineer_loaded_annual, platform_fixed):
    gpu = gpu_hourly * gpu_count * HOURS_PER_MONTH
    people = engineer_fraction * engineer_loaded_annual / 12
    return gpu + people + platform_fixed


def breakeven_tokens(monthly_cost, price_in_per_m, price_out_per_m, out_ratio):
    """out_ratio = output tokens per input token. 0.10 is a 10:1 RAG-ish mix."""
    cost_per_m_input = price_in_per_m + out_ratio * price_out_per_m
    m_input = monthly_cost / cost_per_m_input
    return m_input * 1e6, m_input * out_ratio * 1e6


def sustained_rate(tokens_per_month, active_gpus=1):
    return tokens_per_month / SECONDS_PER_MONTH / active_gpus


if __name__ == "__main__":
    # Gemma 4 31B on Together AI serverless: $0.39 in / $0.97 out per 1M tokens.
    API_IN, API_OUT, OUT_RATIO = 0.39, 0.97, 0.10
    GPU_HOURLY = 2.99      # H100 SXM 80GB, RunPod Secure Cloud, on-demand

    scenarios = [
        ("GPU rent only",        1, 0.00),
        ("+ 20% of an engineer", 1, 0.20),
        ("+ failover GPU",       2, 0.20),
    ]

    for label, gpus, eng in scenarios:
        cost = monthly_selfhost(
            gpu_hourly=GPU_HOURLY,
            gpu_count=gpus,
            engineer_fraction=eng,
            engineer_loaded_annual=200_000,  # fully loaded. Plug in your own.
            platform_fixed=150,              # registry, logs, metrics, images
        )
        t_in, t_out = breakeven_tokens(cost, API_IN, API_OUT, OUT_RATIO)
        print(f"{label:22s} ${cost:9,.2f}/mo   "
              f"{t_in/1e9:5.2f}B in + {t_out/1e9:4.2f}B out   "
              f"{sustained_rate(t_in):6,.0f} input tok/s sustained, 24/7")

# GPU rent only          $ 2,332.70/mo    4.79B in + 0.48B out    1,823 input tok/s sustained, 24/7
# + 20% of an engineer   $ 5,666.03/mo   11.63B in + 1.16B out    4,427 input tok/s sustained, 24/7
# + failover GPU         $ 7,848.73/mo   16.12B in + 1.61B out    6,133 input tok/s sustained, 24/7

The one input you cannot read off a pricing page is your own throughput, and it is the one everybody guesses. Published tokens-per-second figures are third-party and workload-dependent, shifting with the vLLM version, the card revision, the sequence length distribution and the batch shape. SGLang and vLLM trade the lead depending which you hold fixed.

# Serve Gemma 4 31B at FP8 on a single H100 and find out what the box does.
vllm serve google/gemma-4-31B-it \
  --quantization fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 8 \
  --enable-prefix-caching

# Now measure it. Do not trust anyone else's tokens/sec figure, mine included.
# Set the lengths from your own traffic. 512/512 flatters every serving stack.
vllm bench serve \
  --model google/gemma-4-31B-it \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 800 \
  --num-prompts 200 \
  --request-rate 4

# The only number that matters falls out of that run:
#   cost per 1M output tokens = monthly_gpu_bill / (output_tok_s * 2628000 / 1e6)
# 2628000 is the number of seconds in a 730-hour month.
# Compare it to the API price. If you are not at least 2x cheaper, buy tokens.

The four honest reasons to self-host

Cost is rarely one. These four are, and I have seen each justify a deployment the spreadsheet argued against.

• Data residency and contractual isolation. A German insurer, a UK NHS trust, a UAE bank under central bank guidance. When the answer to where does the prompt go has to be a machine you can point at, the API is not an option at any price.
• Latency control. Not average latency, tail latency. A shared serverless endpoint hands you a p99 you do not control and cannot get explained. A dedicated box hands you one you own. Inside a synchronous interaction that is worth real money.
• Portability and exit. Weights you hold cannot be deprecated out from under you, repriced overnight, or quietly changed in a point release. That matters most for anything you have evaluated and certified.
• Fine-tuning you intend to keep. With a proprietary dataset and a LoRA that beats the base model on your task, serving it yourself beats negotiating custom hosting, and a tuned 30B can replace a much larger general model.

None of those is the bill. A client who leads with cost is usually about to spend six months of engineering to save nothing. A client who leads with residency is worth talking to on day one.

What I would actually build

For a mid-market client with a normal workload the recommendation is negative. Do not self-host. Route to a serverless open-weight model, keep the provider abstraction thin enough that swapping takes an afternoon, and revisit in two quarters with real volume data. That advice costs me revenue and it is still right.

If residency forces your hand, start at the 30B tier and nowhere else. Muse Glimmer 30B on a 32 GB card when agentic tool use and screenshot reading are in scope. Gemma 4 31B when you need 256K context and can live with the KV arithmetic above. Qwen3.6-35B-A3B when coding throughput is the priority. One GPU, vLLM or SGLang, prefix caching on, and a hard --max-model-len set from your measured p99 prompt length rather than the model's maximum. That last setting is the biggest lever on how many users fit, and almost nobody touches it.

And if somebody wants to self-host Kimi K3 because the weights are free: they are 1.56 TB, vLLM's minimum serving unit is a whole 8x B300 node, and Moonshot sells the same model at $3.00 and $15.00 per million tokens with a 1,048,576-token window and a flat rate at any context length. Free weights, roughly $43,000 a month of rent.