Your Model Is Being Retired or Silently Upgraded. Here's the Runbook.

Last updated: August 2026

There are two ways you find out the model behind your product changed. The first is an email with a date on it. The second is a Tuesday where your tool-call failure rate goes from 0.3% to 4%, nobody merged anything, and you spend six hours reading your own diff before checking the provider's changelog.

The first case is annoying and the second is expensive, which is backwards from how most teams budget attention. A retirement notice gives you a deadline, a named replacement, and usually six months. A silent version bump gives you nothing, because from your side no event occurred. The endpoint kept returning 200. What moved was somebody else's deploy.

This is not a conspiracy theory about vendors. It is written down in their own docs, and this piece quotes them. The runbook below is what we put in front of clients running LLM calls on a customer-facing path, and the thesis is one sentence: a model version change is a schema migration, so give it the ceremony you give a schema migration.

Two Failure Modes, and the Dates That Matter Now

The loud failure mode is a published deprecation, and the notice periods are stated policy rather than goodwill. OpenAI commits to at least 6 months for generally available models, at least 3 months for specialized variants, and warns that preview models "may be retired with much shorter notice, such as 2 weeks". Anthropic commits to at least 60 days for publicly released models. Google is vaguer and more honest about it: its shutdown dates "indicate the earliest possible dates on which a model might be retired."

That is the manageable case. You get a date, and you get something to migrate to. Here are the live ones, read off each provider's own deprecations page on 11 August 2026 and worth rechecking quarterly.

The entry nobody puts on a deprecation dashboard is a price change. Anthropic's pricing page has Claude Sonnet 5 on introductory pricing of $2 per million input tokens and $10 per million output tokens through 31 August 2026, with standard pricing of $3 and $15 per million taking effect on 1 September 2026 (as of August 2026). Benchmark unit cost during the introductory window and every per-token figure in your model-selection spreadsheet is 50% low three weeks from now, with no deprecation notice attached to it.

The quiet failure mode has no date, because there is no event you can subscribe to. You pointed at a rolling alias, the provider shipped a new build behind it, and you experienced a vendor's release as an unannounced deploy into your production system.

The Silent Upgrade Is Documented, Not Theoretical

OpenAI publishes a model called chat-latest. Its own model page says it "points to the latest Instant model currently used in ChatGPT" and that "the underlying model snapshot will be regularly updated." The same page then tells you what to do about that: "We recommend leveraging GPT-5.6 for production API usage." The vendor is steering you off its own alias, and people ship it anyway because it is the cheapest way to track whatever ChatGPT is doing.

Here is what that subscription buys. On 24 June 2026 OpenAI shipped a new build of GPT-5.5 Instant, then the default model in ChatGPT, under the same name. The release notes describe a model that reads differently, paces its answers differently and reaches for bullet lists less. Every one of those is a change your prompt was tuned against. Nothing in your code could have detected it, because nothing in your code changed.

Benchmarks carry the ambiguity forward. The GPT-5.6 system card prints the same caveat twice, once against its model data and again against its challenging-prompt evals: "Note that comparison values from previously launched models are from the latest versions of those models, so may vary slightly from values published at launch for those models." So a number labelled with an older model's name in a new card came from whatever is currently served under that name, not from the build that shipped under it. Comparing your evals to a published table means comparing against a moving reference.

Anthropic goes the other way on identifiers, guaranteeing that a model ID maps to fixed weights, but it is blunt about the part it cannot pin: "Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed." Routers, safety classifiers and sampling logic all sit outside your version pin. Pinning is necessary and not sufficient, and a runbook that stops there is half a runbook.

What Actually Breaks, and It Is Never the 500s

Model drift is hard to catch because it fails inside a successful response. You get a 200, a well-formed body, a token count. The damage is semantic, and it lands in five places.

• Tool-call shape. The model starts batching three calls into one, or splitting one into three, or emitting an argument as "true" where your parser wants true. Your orchestration layer swallows it and retries.
• Schema adherence drift. Structured output goes from 99.8% valid to 97%. That 2.8% is invisible on a dashboard and lands entirely on the customers with the messiest inputs.
• Refusal boundaries. A prompt that was fine for eight months starts getting declined, usually in one narrow content category, usually the one your regulated customer cares about.
• Verbosity, which is your bill. Reasoning tokens are invisible in the response but, per OpenAI's reasoning guide, "are billed as output tokens." A default shift moves your unit economics without touching your prompt.
• Parameter rejections. On Claude Opus 4.7 and later, setting temperature, top_p or top_k to a non-default value returns a 400. That is a breaking change, and it is the good kind, because it fails at the door instead of drifting.

The tradeoff that actually bites is none of these. It is the unit. Anthropic's pricing page states that Claude 4.7 and later models use a newer tokenizer, and that it "produces approximately 30% more tokens for the same text" (as of August 2026). Sonnet 4.6 and earlier are on the old tokenizer, so a routine-looking hop from Sonnet 4.6 to Sonnet 5 crosses that boundary. Per-token prices did not rise. Your bill does, by roughly a third, on identical traffic. Every cost model, rate limit calculation and context-budget heuristic you wrote is denominated in a unit that changed size, and that line never makes it into the migration ticket.

Step 1: Pin to Snapshot IDs, and Check Your Provider's Convention

The old heuristic was simple: a name with a date is pinned, a name without one is a pointer. That heuristic is now wrong in both directions, which is worse than being wrong in one.

Anthropic changed convention with the 4.6 generation. Its docs address the misconception head on: dateless IDs such as claude-sonnet-4-6 do not "behave as evergreen pointers that route to the latest or best-performing version. That is not the case." For 4.6 and later, the dateless ID is the snapshot, and an updated model ships under a new ID. For models before 4.6, the short form is still a convenience alias resolving to the most recent dated snapshot. Same provider, opposite semantics, depending on generation. Meanwhile gpt-5.6 on OpenAI is documented as an alias that routes to gpt-5.6-sol.

The rule that survives: you cannot infer pinning from the shape of the string. Read each provider's versioning page once per model generation, then write the answer as a comment next to the constant in your code. Ten minutes of work, and it is the step that decides whether the other nine matter.

Step 2: Freeze an Eval Set Before You Need It

The ordering is the whole point. You need real requests, with the outputs your current model produced, captured while the current model is still current. Two hundred to four hundred examples sampled from production, committed as files, never regenerated.

The failure I see most often is a team building its regression suite after an incident, using the new model's outputs as the reference. That suite passes forever and tells you nothing, because it has enshrined the regression as the spec. Once the change lands, no amount of care recovers a clean baseline.

Assert in three layers, in order. Structural: does the JSON parse, does it validate, is the tool name in the allowed set, is the argument count right. Semantic: an LLM judge against a written rubric, fuzzy and slow and worth having anyway. Cost: output tokens per request as a ratio to baseline, not an absolute.

If you only have budget for one, take the structural layer. Two hundred examples with hard structural assertions catch more production incidents than five thousand scored by a judge, because structural failures are the ones that page somebody.

Step 3: Poll the Models API in CI

A retirement should fail a build, not a customer. Both providers expose a list-models endpoint. They return very different amounts of information through it, and your check has to be honest about which one you are talking to.

Anthropic's GET /v1/models returns, per model: id, display_name, created_at, max_input_tokens, max_tokens, and a capabilities object covering batch, citations, code execution, context management strategies, effort levels (low, medium, high, xhigh, max), image input, PDF input, structured outputs and thinking types. You can genuinely assert "xhigh effort is still supported" or "the input window did not shrink" against that.

OpenAI's GET /v1/models returns four fields: id, object, created and owned_by. No context window, no capabilities. So the check reduces to existence plus a creation timestamp. That sounds weak until you remember existence is exactly what changes on a shutdown date, and a red build the morning a model disappears beats a clever check you never wrote.

# model_guard.py - fail CI when a pinned model disappears or changes shape.
# Run on every PR and nightly on a schedule. Commit models.lock.json to the repo.
import json, os, sys, urllib.request

PINNED = {
    "anthropic": ["claude-opus-5", "claude-sonnet-4-6"],
    "openai":    ["gpt-5.6-sol", "gpt-5.6-terra"],
}

def anthropic_models():
    req = urllib.request.Request(
        "https://api.anthropic.com/v1/models?limit=100",
        headers={"x-api-key": os.environ["ANTHROPIC_API_KEY"],
                 "anthropic-version": "2023-06-01"},
    )
    body = json.load(urllib.request.urlopen(req))
    out = {}
    for m in body["data"]:
        caps = m.get("capabilities", {})
        # This provider publishes limits and a capability tree, so assert on both.
        out[m["id"]] = {
            "created_at":        m["created_at"],
            "max_input_tokens":  m.get("max_input_tokens"),
            "max_tokens":        m.get("max_tokens"),
            "effort_xhigh":      caps.get("effort", {}).get("xhigh", {}).get("supported"),
            "structured_output": caps.get("structured_outputs", {}).get("supported"),
        }
    return out

def openai_models():
    req = urllib.request.Request(
        "https://api.openai.com/v1/models",
        headers={"Authorization": "Bearer " + os.environ["OPENAI_API_KEY"]},
    )
    body = json.load(urllib.request.urlopen(req))
    # This endpoint returns id, object, created and owned_by. That is the whole
    # schema, so existence plus created is all you can assert. Existence is the
    # one that matters: a retired model stops appearing here.
    return {m["id"]: {"created": m["created"]} for m in body["data"]}

def main():
    live = {"anthropic": anthropic_models(), "openai": openai_models()}
    observed = {p: {mid: live[p].get(mid) for mid in ids} for p, ids in PINNED.items()}

    lock = "models.lock.json"
    if not os.path.exists(lock):
        with open(lock, "w") as fh:
            json.dump(observed, fh, indent=2, sort_keys=True)
        print("wrote baseline", lock)
        return 0

    with open(lock) as fh:
        locked = json.load(fh)

    drift = []
    for provider, models in observed.items():
        for mid, now in models.items():
            was = locked.get(provider, {}).get(mid)
            if now is None:
                drift.append("%s/%s: GONE from the models API" % (provider, mid))
            elif was != now:
                drift.append("%s/%s: %s -> %s" % (provider, mid, was, now))

    if drift:
        print("MODEL DRIFT DETECTED")
        for d in drift:
            print("  " + d)
        return 1
    print("no drift across %d pinned models" % sum(len(v) for v in observed.values()))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it on pull requests and on a nightly schedule. The PR run catches you changing the pin; the nightly run catches the provider changing it for you. If you only pick one, pick nightly.

# What do the providers say about your pinned models right now? Run it weekly.

# Anthropic publishes limits and a capability tree per model.
curl -s "https://api.anthropic.com/v1/models?limit=100" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
| jq -r '.data[] | [ .id, .created_at,
                     (.max_input_tokens|tostring),
                     (.capabilities.effort.xhigh.supported|tostring),
                     (.capabilities.structured_outputs.supported|tostring) ] | @tsv'

# OpenAI's list endpoint returns id, object, created and owned_by. Nothing else.
# So the only honest assertion is "this ID is still being served".
curl -s https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
| jq -r '.data[] | select(.id | startswith("gpt-5.6")) | [ .id, (.created|todate) ] | @tsv'

Step 4: Shadow Traffic Before You Cut Over

Shadowing means the candidate model sees real production requests and its output goes nowhere near a user. You pay twice for inference on the shadowed slice and get the only dataset that matters: how the new model behaves on your real traffic distribution, including the ugly 5% of inputs nobody thought to put in the eval set.

Fire the shadow call as a background task and never await it in the request path. A shadow that adds latency to production gets turned off within a week, and then you have neither the data nor the safety.

# shadow.py - run the candidate model beside production and diff the contract.
# The candidate's output is never returned to a caller. It only produces metrics.
import asyncio, json, logging
from jsonschema import Draft202012Validator
import anthropic

client = anthropic.AsyncAnthropic()
log = logging.getLogger("shadow")

BASELINE  = "claude-sonnet-4-6"
CANDIDATE = "claude-sonnet-5"

VALIDATOR = Draft202012Validator({
    "type": "object",
    "required": ["order_id", "status", "line_items"],
    "properties": {
        "order_id":   {"type": "string", "pattern": "^ORD-[0-9]{8}$"},
        "status":     {"enum": ["open", "shipped", "cancelled"]},
        "line_items": {"type": "array", "minItems": 1},
    },
    "additionalProperties": False,
})

async def call(model, messages, tools):
    loop = asyncio.get_running_loop()
    t0 = loop.time()
    r = await client.messages.create(
        model=model, max_tokens=2048, messages=messages, tools=tools,
        output_config={"effort": "medium"},   # set it; never inherit the default
    )
    return {
        "model":         model,
        "latency_ms":    int((loop.time() - t0) * 1000),
        "output_tokens": r.usage.output_tokens,
        "stop_reason":   r.stop_reason,
        "tool_names":    sorted(b.name for b in r.content if b.type == "tool_use"),
        "text":          "".join(b.text for b in r.content if b.type == "text"),
    }

def schema_ok(text):
    try:
        return VALIDATOR.is_valid(json.loads(text))
    except json.JSONDecodeError:
        return False

async def handle(messages, tools, emit):
    live = await call(BASELINE, messages, tools)
    asyncio.create_task(compare(messages, tools, live, emit))   # fire and forget
    return live                                                 # user gets baseline

async def compare(messages, tools, live, emit):
    try:
        cand = await call(CANDIDATE, messages, tools)
    except Exception as exc:              # a shadow failure is itself a finding
        emit("shadow.error", 1, model=CANDIDATE, err=type(exc).__name__)
        return
    emit("shadow.schema_ok",     int(schema_ok(cand["text"])), model=CANDIDATE)
    emit("shadow.schema_ok",     int(schema_ok(live["text"])), model=BASELINE)
    emit("shadow.tools_match",   int(cand["tool_names"] == live["tool_names"]))
    emit("shadow.token_ratio",   cand["output_tokens"] / max(live["output_tokens"], 1))
    emit("shadow.latency_delta", cand["latency_ms"] - live["latency_ms"])

Five metrics are enough: schema validity rate, tool-name distribution, refusal rate, p95 latency, and output tokens per request as a ratio to baseline. Run it for a full weekly cycle before drawing conclusions. Monday morning traffic and Saturday night traffic are different products, and a model change frequently only hurts one of them.

Step 5: Canary by Route, Not by Percentage

This is where I disagree with how most teams do it. Percentage rollouts are the wrong primitive for a model change. Sending the new model to 5% of users spreads it across every route, giving you 5% exposure to every failure mode and significance on none of them. You stare at a dashboard for three days, see nothing conclusive, and ramp anyway.

Route-based canarying inverts that. Pick one route, ship 100% of it, and get a clean signal on a bounded surface within hours. Order routes by blast radius, not by traffic volume.

The one case for percentage rollouts is a single route carrying most of your volume that you genuinely cannot subdivide. Even then, split by tenant or customer tier rather than by hash of user ID, so that if it goes wrong the blast radius is a list of accounts you can name and call.

Step 6: Write the Rollback Trigger as a Number

Write it before you start, in the migration ticket, as a threshold with a window and an owner. "Schema validation rate below 99.2% over any 30 minute window" is a trigger. "If it looks bad we'll roll back" is a plan to hold a meeting at 2am.

• The threshold. A number and a window, derived from your frozen baseline rather than from a round figure that felt right.
• The owner. One named person per shift who can pull the trigger without asking permission.
• The command. One config change or one flag flip, tested in staging before the canary starts. If rollback needs a deploy, it is not a rollback.
• The retention window. How long the old snapshot ID stays callable. On an expiring model this is a hard date, not a preference, and it caps how long you can sit half-migrated.

The threshold does not have to be right. It has to exist, because its job is to move the decision from a moment when you are tired and invested to one when you are calm and have not yet spent a week on the migration.

The Reasoning-Effort Gotcha That Gets Blamed on the Migration

Reasoning effort is a per-model default, not a platform default. OpenAI's reasoning guide states that "defaults are also model-dependent rather than universal", and that gpt-5.5 "defaults to medium reasoning effort". Anthropic starts higher: high is the default, Claude Sonnet 5 "defaults to high effort on the Claude API and Claude Code", and the docs tell you to set effort explicitly on Sonnet 4.6 "to avoid unexpected latency", with medium as the recommended starting point. All of that is as of August 2026, and all of it moves per model.

So the same model ID, same prompt, spends materially different token budgets depending on whether anyone passed the parameter. Your notebook and your product disagree, and the disagreement gets blamed on whichever migration you happen to be running that week.

Surface differences are real too. A developer posted the API's own error text to the OpenAI forum on 11 July 2026: "Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'." Matching reports name terra and luna. That is a community report rather than vendor documentation, so confirm it on your own account before you plan around it. It is also the exact shape of failure that gets written up as a model regression when it is an endpoint constraint.

There is a second-order cost. Anthropic's docs note that changing effort between requests invalidates cached prefixes, because effort shapes the rendered prompt. Lowering effort mid-conversation to save money can raise the bill by destroying cache hits, and a cache read costs 10% of the standard input rate. Set effort explicitly per workload and hold it constant inside a cached session.

The Runbook as a Checklist

Put this in the migration ticket template and stop rediscovering it every quarter.

• 1. Pin every production call to a snapshot ID. Confirm from the provider's versioning doc whether that ID is a snapshot or an alias. The naming convention will not tell you.
• 2. Grep for aliases. Anything ending in -latest, any bare family name, any model string built by concatenation at runtime. Fail the build on them.
• 3. Freeze 200 to 400 real requests with their current outputs. Commit them, and never regenerate them after the change.
• 4. Add the models-API check to CI and run it nightly, not only on pull requests.
• 5. Set every behavioral parameter explicitly. Effort, max tokens, temperature where still permitted. Never inherit a default.
• 6. Shadow the candidate for one full weekly traffic cycle, diffing schema validity, tool names, refusal rate, p95 latency and token ratio.
• 7. Recheck your cost model in tokens, not dollars. Tokenizers change, and price per token is not the only way a bill moves.
• 8. Canary by route, cheapest blast radius first, with agentic routes that have side effects last.
• 9. Write the rollback trigger as a number with a window, an owner and a one-command action, before the canary starts.
• 10. Diarize the retirement date of the model you migrated to, on the day you migrate. There will be another one.

None of this is clever. It is the discipline you already apply to a column type change on a table three services depend on, pointed at a dependency that happens to be somebody else's weights. The teams that get hurt are not the ones running bad models. They are the ones who treated a model string as configuration instead of as a version.