When an AI model gets retired: keeping AI features alive after the vendor moves on

Here’s one I’m a little embarrassed about. One of our internal tools uses a small AI model to sort incoming items into projects. For a while, it just… didn’t. No errors on the dashboard, no alerts, the endpoint returned success every time. The feature was dead and nobody noticed.

The cause was simple. The model it was pinned to, a Llama 3.1 8B model on Cloudflare’s Workers AI, was no longer being served. Every request to it failed. And because the code caught that failure as a warning and moved on to the next item, nothing looked broken from the outside.

That’s going to happen to a lot of small-business AI features over the next year or two, so here’s what I learned and what I do now.

Models have expiration dates

AI providers retire older models. The big ones post deprecation notices ahead of time, and after the retirement date the old model stops answering. You can see it for yourself on Anthropic’s model deprecations page, OpenAI’s deprecations page, the Gemini API changelog and the Workers AI changelog.

That’s reasonable. Models improve and running old ones costs them money. BUT it means any AI feature is a little like a car: it needs scheduled maintenance, even if nothing looks wrong.

What went wrong, specifically

Three things lined up:

• The model name was hard-coded. Nobody had a reason to look at it again.
• The error was caught and logged as a warning, item by item. Warnings don’t wake anyone up.
• The feature’s success was measured by “the request returned 200,” not by “the items actually got sorted.”

Any one of those alone would have been fine. Together, a dead feature looked healthy for a long time.

What I build in now

• Check before pinning. Before a model goes into code, I list the models the provider currently offers and pick from that list, not from memory or an old example.
• Treat the first failure as the batch failure. If the first AI call in a batch fails, stop the batch and log it as an ERROR with the model name in the message, instead of grinding through a hundred identical warnings.
• Watch the outcome, not the status code. Measure the thing the feature is supposed to produce: emails sorted, fields filled, summaries written. If that number drops to zero, alert.
• Keep the model name in one place. A setting, not scattered through the code, so a swap is a one-line change.
• Keep a test set. Twenty or thirty real examples you can rerun on a new model in an hour. I talk about building one in the first 30 days.

A simple maintenance routine for owners

If you have AI features running in your business, whether we built them or someone else did:

• Make a list: each feature, which provider, which model.
• Twice a year, check each model against the provider’s deprecation page.
• When a model is on the list, rerun the test set on the replacement, compare, and switch well before the date.

That’s an hour or two a year for most small businesses. Much cheaper than finding out from a customer.

The upside

Retirement is annoying, but the replacement is often better, and the test set tells you quickly whether it’s safe to switch. It’s also a good moment to recheck whether the job could move to a smaller model, which I covered in Route AI work by how much judgment it needs.

For the bigger picture on what these models can do now, see New AI capabilities this fall.

If you have AI features running and you’re not sure which models they use, send me a note and I’ll help you make the list. Thanks!