Token Math at Warehouse Scale: What an LLM Job Actually Costs

Last updated: August 2026

Every LLM pricing page prices a conversation. Cost per chat, cost per user, cost per agent run. Fine if you are building a chatbot. Useless if somebody just asked what it costs to classify eleven million support tickets, because the finance person across the table does not buy conversations. They buy a table getting a new column.

The framing that works in a business case is cost per row, cost per partition, cost per nightly run. The arithmetic is easy. The order of operations is what people get wrong: they pick a model, find a price on a marketing page, multiply by a guessed average token count, and land 3x off in either direction. Measure first, then price.

Below is the method, a worked example at 1M, 10M and 100M rows across three tiers, the levers that move the number ranked by effect, and the failure mode that turns a clean estimate into a two-week project. Prices are dated August 2026 and taken from the vendors' own pricing pages. They will move. The method will not.

The unit is the row, and there are only four terms

Strip everything else away and a warehouse enrichment job has one cost formula:

total = rows x (prefix_tokens + row_tokens) x input_rate + rows x output_tokens x output_rate

Four terms, all of them yours. prefix_tokens is the fixed instruction block, taxonomy and few-shot examples, paid once per row because it is resent on every request. row_tokens is the variable payload. output_tokens is whatever the model writes unless you stop it. The two rates depend on which endpoint you call, not only which model.

Warehouse compute sits outside the formula and is usually rounding error against it. A Small warehouse for a few hours does not change a four-figure decision. Put it in the estimate anyway, because the first question in the review will be whether you forgot it.

What the formula leaves out is the people cost of rows that come back wrong. At ten million rows, that term decides whether the project lands.

Step one: measure input tokens on a real sample

The usual mistake is to estimate from a mean token count. Text in warehouse tables is not normally distributed. Support tickets, product descriptions and clinical notes all have a long right tail, and the tail is where the money goes. Pull twenty thousand rows, tokenize them properly, look at the percentiles.

-- Snowflake. Profile input size on a real sample before you quote a number.
-- Averages lie: it is the p95 tail that decides your truncation policy.
-- AI_COUNT_TOKENS, not SNOWFLAKE.CORTEX.COUNT_TOKENS. The latter is legacy and
-- Snowflake has it slated for deprecation by the end of 2026.
WITH sampled AS (
    SELECT ticket_id, ticket_body
    FROM   support.public.tickets
    SAMPLE (20000 ROWS)
),
measured AS (
    SELECT
        LENGTH(ticket_body)                                          AS chars,
        AI_COUNT_TOKENS('ai_complete', 'claude-sonnet-5', ticket_body) AS row_tokens
    FROM   sampled
    WHERE  ticket_body IS NOT NULL
)
SELECT
    COUNT(*)                                          AS sampled_rows,
    ROUND(AVG(row_tokens))                            AS mean_row_tokens,
    APPROX_PERCENTILE(row_tokens, 0.50)               AS p50_row_tokens,
    APPROX_PERCENTILE(row_tokens, 0.95)               AS p95_row_tokens,
    APPROX_PERCENTILE(row_tokens, 0.99)               AS p99_row_tokens,
    MAX(row_tokens)                                   AS max_row_tokens,
    ROUND(AVG(chars) / NULLIF(AVG(row_tokens), 0), 2) AS chars_per_token
FROM measured;

-- Two things this tells you that an average never will:
--   1. p99 / p50. Above 5 and you have a long-tail problem: truncate at p95
--      instead of paying for the tail.
--   2. chars_per_token. Use the measured value in the projection below.
--      Do not use 4. It is wrong for JSON, code, URLs and non-English text.

Three tokenizer caveats that will bite you:

• Snowflake's token counters read low. The docs are explicit that COUNT_TOKENS ignores the managed system prompt Cortex AI functions prepend, so billed tokens exceed what it returns. AI_COUNT_TOKENS, the replacement, does account for labels and function scaffolding, but the docs say structured output on Claude models generates extra request content that is billed and not estimated, so the gap can still be material. AI_CLASSIFY counts your labels, descriptions and examples as input tokens on every record, not once per call.
• Tokenizers changed mid-generation. Anthropic's pricing docs state that Claude 4.7 and later use a newer tokenizer producing roughly 30% more tokens for the same text than Sonnet 4.6 and earlier. Calibrate on an older model, price a newer one, and your input estimate is a third low before you write any code.
• Count for free where you can. Anthropic's /v1/messages/count_tokens endpoint costs nothing and is rate-limited separately from the Messages API, so tokenize the sample exactly instead of approximating from characters.

With a measured chars_per_token for your corpus, project the whole table. Truncate at p95, not p99: the last four percent of the distribution often doubles the input bill for rows no human would finish reading either.

-- Project the invoice over the full table. Every constant here is measured,
-- not assumed. Prices are Claude Sonnet 5 batch rates as of August 2026, which
-- are introductory rates: from September 1, 2026 use 1.50 and 7.50.
SET prefix_tokens   = 1150;   -- fixed instruction block, counted with the same tokenizer
SET chars_per_token = 3.70;   -- calibrated from the sample query above
SET output_expected = 45;     -- observed mean output under max_tokens = 64
SET retry_uplift    = 1.05;   -- 5% for retries and sample drift
SET price_in        = 1.00;   -- USD per 1M input tokens
SET price_out       = 5.00;   -- USD per 1M output tokens

WITH est AS (
    SELECT
        COUNT(*) AS rows_to_process,
        SUM($prefix_tokens + LEAST(LENGTH(ticket_body), 4000) / $chars_per_token) AS input_tokens
    FROM  support.public.tickets
    WHERE ticket_body IS NOT NULL
      AND enriched_at IS NULL
)
SELECT
    rows_to_process,
    ROUND(input_tokens)                AS input_tokens,
    rows_to_process * $output_expected AS output_tokens,
    ROUND(input_tokens / 1e6 * $price_in * $retry_uplift, 2)                        AS input_usd,
    ROUND(rows_to_process * $output_expected / 1e6 * $price_out * $retry_uplift, 2) AS output_usd,
    ROUND((input_tokens / 1e6 * $price_in
           + rows_to_process * $output_expected / 1e6 * $price_out)
          * $retry_uplift, 2)          AS total_usd,
    ROUND((input_tokens / 1e6 * $price_in
           + rows_to_process * $output_expected / 1e6 * $price_out)
          * $retry_uplift / NULLIF(rows_to_process, 0) * 1000, 4) AS usd_per_1000_rows
FROM est;

Step two: cap the output, because output bills at 5x to 8x input

The ratio is consistent across vendors as of August 2026. Claude Sonnet 5 and Opus 5 bill output at exactly 5x input, the GPT-5.6 family at 6x, GPT-5-nano at 8x, Gemini 3.5 Flash-Lite at about 8.3x. A model that writes a friendly paragraph where you asked for a label is not costing you slightly more. It is costing you multiples.

In the worked example below, output averages 45 tokens under max_tokens = 64. Let it run to 320, which is what "explain your reasoning" buys, and the Sonnet 5 bill per million rows goes from $1,625 to $3,000. The input side never moved.

Three things, in this order:

• Set max_tokens to the smallest value your schema fits in. Count the tokens in your longest legitimate response, add 20%. On Anthropic models a generous cap carries no rate-limit penalty: max_tokens does not factor into the output-tokens-per-minute limit, only tokens generated do.
• Constrain the shape, not only the length. OpenAI's structured outputs with strict: true on a json_schema constrain decoding at the API level, so the model cannot emit an undeclared field or an out-of-list enum value. That deletes a whole class of parse failure instead of catching it downstream.
• Drop the reason field unless somebody reads it. A free-text justification runs 30 to 60 output tokens per row. At ten million rows on Sonnet 5 batch rates, 40 extra tokens per row is 400 million output tokens, or $2,000. If a reviewer needs it to audit disagreements, keep it only on rows you route to review.

The worked example: cost per million rows across eight models

The job: classify support tickets into a 40-label taxonomy. Fixed prefix of 1,150 tokens covering instructions, the label list and six few-shot examples. Median ticket body of about 260 tokens after truncating at p95. Call it 1,400 input and 45 output tokens per row. All rates below are the vendors' published batch rates as of August 2026.

Two notes before you copy a row out. Claude Sonnet 5 is on introductory pricing of $2/$10 standard through August 31, 2026, then $3/$15, which takes the batch rate to $1.50/$7.50 and the per-million-row total to $2,437.50. If your business case runs past September, price September. Second, the spread from cheapest to most expensive is 95x, far wider than any accuracy gap you will measure on classification with a well-written taxonomy.

Scaling to 10 million and 100 million rows

Linear in rows, so the interesting comparison is batch against online serving on the same model. Same 1,400 input and 45 output tokens per row, August 2026 rates.

Find your row. The honest version of the 10M slide carries three numbers, not one, because the choice of tier is the decision. Vendor barely registers next to it.

Lever one: the batch endpoint

OpenAI, Anthropic and Google all publish the same discount for asynchronous batch processing: exactly 50% off input and output. Vendor blog posts claiming savings of 5x, 10x or 30x are not reproducible from any rate card. The rate card says half. Put half in the business case.

The stronger argument for batch is throughput, also from published numbers. On Anthropic's Scale tier, Claude Sonnet 5 gets 10,000,000 uncached input tokens per minute on the Messages API, in its own bucket separate from the Sonnet 4.x pool. At 1,400 input tokens per row that is about 7,140 rows a minute, so ten million rows is roughly 23 hours of pinning your organization's entire rate limit for that model, assuming no 429s and no other traffic. Batch usage does not draw from that pool at all, and Anthropic's docs say most batches finish inside an hour.

Constraints to design around, from each vendor's docs in August 2026:

When I would not use batch: anything a human waits on, and anything where the source row can change inside the 24 hour window. Edit a ticket after submission and you write a label derived from text that no longer exists. Store a content hash with the submission and refuse to merge a result whose hash no longer matches.

Lever two: prompt caching, and the break-even that decides it

Caching a stable prefix is the lever most often asserted rather than calculated. The arithmetic is one line. With W as the cache write multiplier and R as the read multiplier, both against base input price, caching wins once the requests sharing that prefix exceed (W - R) / (1 - R).

At warehouse scale you clear all of those on row two, so caching always pays if it fires. Two reasons it will not fire:

• Your prefix may be too short to cache at all. Anthropic's minimum cacheable length is per model: 512 tokens on Opus 5, 1,024 on Sonnet 5, 4,096 on Haiku 4.5. A 1,150 token taxonomy caches on Sonnet 5 and Opus 5 and does nothing at all on Haiku 4.5. No error is raised. You check cache_creation_input_tokens and cache_read_input_tokens, see two zeros, and wonder where the saving went. OpenAI's floor is 1,024 tokens on GPT-5.6 and later, and 1,024 to 2,048 depending on the model on earlier families.
• Batch and caching do not cooperate. Anthropic's pricing page says the discounts stack, and they do. The caching docs also say a cache entry written during batch processing would likely expire before the follow-up request runs, which is why pre-warming with max_tokens: 0 is rejected inside a batch. You do not control batch scheduling, so you cannot guarantee rows land inside a 5 minute window. Keep cache savings out of a batch business case.

Caching earns its place on the online path: the tiered escalation below, and any interactive scoring endpoint sharing the taxonomy. Price that path at standard rates, not the batch rates in the table above. On Sonnet 5 at $2 in and $10 out, a million rows is $3,250. Cache 1,150 of the 1,400 input tokens and the input side falls from $2,800 to $730, because the cached slice reads at $0.20 per million instead of $2.00. Total $1,180, a 64% cut, and the cache write is one row's worth of tokens.

Lever three: tier the models and route only the ambiguous rows

Biggest effect, least attention, and the place I will state an opinion plainly: for classification and extraction over a warehouse table, a cheap model with a strict schema plus a review queue for low-confidence rows beats a frontier model on everything, usually by more than an order of magnitude on cost, and the accuracy gap is much smaller than the invoice gap.

At ten million rows: GPT-5.6 Luna over the full set is $1,670. Route the lowest-confidence 8% to Claude Opus 5 for a second pass, 800,000 rows at $0.0040625 each, and that is $3,250. Total $4,920 against $40,625 for Opus 5 on everything. Cheaper by 8.3x, and the 92% that never reached the expensive model were rows both models agreed on.

Two things make or break it. Self-reported confidence is not calibrated, so use it as a ranking signal, not a probability: label 1,500 rows by hand, sort by reported confidence, pick the threshold that fits the escalation budget, then measure accuracy on the rows you kept. And the escalation prompt should be a different prompt. Hand the second pass the first answer and ask it to confirm or correct. Easier task, shorter output.

When I would not tier: when the label feeds a regulated decision and you need one auditable model in the path; when the cheap model's accuracy sits below roughly 80%, because then you escalate half the table and pay twice; and under about 200,000 rows, where building the router costs more than it saves. This is a routing decision, not a knowledge-injection one. If the model simply does not know your domain, start with RAG vs fine-tuning for enterprise AI instead.

The 0.5 percent problem, which is really the whole problem

The tradeoff that bites in production is not cost per token. A 0.5% failure rate at ten million rows is 50,000 rows. Reprocessing them on the cheap tier costs about eight dollars. Nobody cares about the eight dollars. Somebody has to care about the 50,000 rows, and if you did not design for them, that somebody is you.

Failures are not one thing, and strict schemas kill only one category:

• Truncated output. The response hit max_tokens mid-object. Detectable exactly, via stop_reason == "max_tokens", and always a quarantine rather than a parse attempt.
• Per-request failures inside a successful batch. Individual requests come back errored, canceled or expired while the batch reports finished. A job that checks batch status and not per-request status silently loses these rows.
• Refusals and empty content. Valid JSON, no useful label. Schema enforcement does not catch it because the output is well-formed.
• Out-of-enum labels. Without constrained decoding, the model invents a plausible label that is not in your taxonomy. Validate against the enum in code, not just in the prompt.
• Semantic garbage. Right shape, right enum, wrong answer. Only sampling catches it. Budget 500 human-reviewed rows per run and track the rate, because a prompt that was fine last month drifts when the model version rolls.

The design that survives: custom_id is the primary key and results join on it, never positionally, because batch results do not come back in submission order. Failures land in a quarantine table with a reason code and an attempt counter, and a row that fails twice goes to human review rather than an infinite retry loop. Downstream has to run with half a percent unlabeled, so the label column is nullable and every dashboard has an explicit unclassified bucket. Retry and quarantine design belongs in the cost model, not the cleanup sprint.

Orchestration: submit and reap as two separate tasks

The scheduling pattern is the boring part and where most of these pipelines fail. Submit is fast and returns a batch id. Reaping happens whenever the provider finishes, ten minutes to 24 hours later. Write them as one task and you get an Airflow worker asleep in a poll loop, plus a scheduler restart that loses the batch id.

"""
Submit and reap a Message Batch as two separate scheduled tasks.
Never both in one task: a reap that polls in a loop pins an Airflow worker
for an hour to do nothing.

Limits verified against Anthropic's batch docs, August 2026:
  100,000 requests or 256 MB per batch, whichever comes first
  most batches finish inside an hour; anything unfinished at 24h expires
  results downloadable for 29 days
  in-queue cap is 200k (Start) / 300k (Build) / 500k (Scale) batch requests
"""
import json
import anthropic

client = anthropic.Anthropic()

MODEL       = "claude-sonnet-5"
MAX_TOKENS  = 64        # hard cap. Output bills at 5x input on this model.
CHUNK_ROWS  = 50_000    # half the per-batch cap, and a smaller blast radius

PROMPT_VERSION = "taxonomy_v7"
with open(f"prompts/{PROMPT_VERSION}.txt") as fh:     # frozen, in git
    SYSTEM_PREFIX = fh.read()

SCHEMA_HINT = ('Reply with one JSON object and nothing else: '
               '{"label": <exactly one label from the list above>, '
               '"confidence": <float 0.0-1.0>}')


def submit_chunk(rows):
    """rows: list of (row_id, text). Returns a batch id to persist immediately."""
    requests = [{
        "custom_id": str(row_id),          # your primary key. This is the join key.
        "params": {
            "model": MODEL,
            "max_tokens": MAX_TOKENS,
            # No temperature. Sonnet 5, Opus 4.7 and Opus 4.8 return a 400 on any
            # non-default temperature, top_p or top_k. Pin determinism in the prompt.
            "system": [{"type": "text", "text": SYSTEM_PREFIX}],
            "messages": [{"role": "user", "content": f"{text}\n\n{SCHEMA_HINT}"}],
        },
    } for row_id, text in rows]

    batch = client.messages.batches.create(requests=requests)
    return batch.id                        # write this to your control table now


def reap(batch_id, out_path, quarantine_path):
    """Idempotent. Returns False if the batch is not finished yet."""
    batch = client.messages.batches.retrieve(batch_id)
    if batch.processing_status != "ended":
        return False                       # let the next scheduled run try again

    ok = bad = 0
    with open(out_path, "a") as good, open(quarantine_path, "a") as quar:
        for item in client.messages.batches.results(batch_id):
            row_id, res = item.custom_id, item.result

            # errored | canceled | expired. Requeue, never silently drop.
            if res.type != "succeeded":
                quar.write(json.dumps({"row_id": row_id, "reason": res.type}) + "\n")
                bad += 1
                continue

            msg = res.message

            # The cap did its job but the JSON is truncated and unparseable.
            if msg.stop_reason == "max_tokens":
                quar.write(json.dumps({"row_id": row_id, "reason": "truncated"}) + "\n")
                bad += 1
                continue

            try:
                payload = json.loads(msg.content[0].text)
                label = payload["label"]
            except (json.JSONDecodeError, KeyError, IndexError):
                quar.write(json.dumps({"row_id": row_id, "reason": "malformed",
                                       "raw": msg.content[0].text[:500]}) + "\n")
                bad += 1
                continue

            good.write(json.dumps({
                "row_id":         row_id,
                "label":          label,
                "confidence":     payload.get("confidence"),
                "prompt_version": PROMPT_VERSION,
                "input_tokens":   msg.usage.input_tokens,
                "output_tokens":  msg.usage.output_tokens,
            }) + "\n")
            ok += 1

    rate = bad / max(ok + bad, 1)
    print(f"{batch_id}: {ok} written, {bad} quarantined ({rate:.2%})")
    if rate > 0.02:
        raise RuntimeError("quarantine rate above 2%, check the prompt version")
    return True

Around it you need three things in the warehouse. A control table holding batch id, chunk boundaries, prompt version, submitted timestamp and status, written before the API call returns so a crash mid-submit is recoverable. A reap task frequent enough that nothing sits past the 24 hour expiry, which means hourly. And a chunk size well under the hard cap: 50,000 against Anthropic's 100,000 leaves headroom on the 256 MB limit and halves the blast radius of a bad prompt version.

Freeze and version the prefix file, and stamp the version onto every output row. Six months from now somebody asks why two near-identical tickets got different labels, and the only answer that ends the conversation is taxonomy_v6 versus taxonomy_v7.

If you are running this inside Snowflake instead

Cortex AISQL changes the accounting, not the arithmetic. Snowflake bills AI functions in AI Credits, a separate currency from Platform Credits, flat-priced as of August 2026 at $2.00 per credit for global routing and $2.20 for regional, the same on Standard, Enterprise, Business Critical and VPS. AI_COMPLETE and friends burn AI Credits per million tokens at a per-model rate published in the Snowflake Service Consumption Table, and AI_COMPLETE bills both input and output.

Four things that move your estimate:

• Warehouse compute is separate and additional. Snowflake's docs state the cost of keeping a warehouse active still applies while a query calling a Cortex LLM function runs. Platform Credits on top of AI Credits.
• There is no 50% batch discount. Snowflake publishes no asynchronous tier for Cortex AI functions: one rate whether you pass one row or a hundred million, so a warehouse-native run is priced like online serving.
• Bigger warehouses do not help. Snowflake recommends no larger than MEDIUM for Cortex LLM functions; larger sizes do not improve performance and only add credits.
• Estimate then reconcile, do not estimate and trust. Every counter here understates for the reasons in step one, and structured output on Claude models adds billed request content nothing estimates. Take the token and credit numbers off CORTEX_FUNCTIONS_USAGE_HISTORY after run one and re-forecast from those.

The call I would make: if the job is large and can wait, unload the rows, run a provider batch endpoint, MERGE the results back. You trade in-database simplicity for half price and a queue that does not compete with interactive workloads. A few million rows or fewer, or governance says the text stays in the account, run it in Cortex and eat the online rate, because the unload-submit-reap-merge loop costs more to build than it saves. For what those functions can see and do, see our Snowflake Cortex AI guide.