Running AI on your own hardware: when a local model makes sense

We run open AI models on our own machines, right alongside Claude. Not because the hosted models aren’t good. They are. It’s because some jobs are repetitive enough, or private enough, that running them locally makes more sense. Here’s what we measured and where I think local fits for a small business.

Where the tokens actually went

Before deciding what to move local, I looked at where our hosted AI usage went. Over one week on one of our machines, weighting each kind of token by what it costs, here’s the split:

• Reading cached context: about 64 percent.
• Writing new context to the cache: about 26 percent.
• The model’s actual output: about 10 percent.

That surprised me. My instinct was to hand the writing to a local model, since writing feels like the expensive part. It’s the SMALLEST slice. The cost is in reading: the model re-reading big conversations over and over. I wrote about why in Context is the cost.

So the jobs worth moving local are the read-heavy ones: searching through files, answering a quick factual question about a document, sorting and tagging. Judgment stays on the hosted models.

The harness mattered more than the hardware

The second thing we learned was about HOW you run a local model. We tested the same coding task (make a set of failing tests pass, two files) on a laptop with 64 GB of memory, using an open model in the 30 billion parameter class.

• Run as an agent in a loop, where the model decides what to read and do next: 4 to 11 minutes per attempt, about 500,000 tokens processed, and only one of the two attempts passed.
• Run one-shot, where we named the files and the model made a single editing pass with the tests run automatically: 26 to 40 seconds, about 4,000 to 11,000 tokens, and it passed all four attempts. Small samples, but a VERY clear difference.

Same model, same machine. The loop was slow because each turn re-sent a growing conversation, and a small model at long context drifts. We also tried different engines for running the model and they were within about 20 percent of each other. Swapping the model made a bigger difference than swapping the engine.

The takeaway… a local model does best when a bigger model, or a person, plans the work and hands it one clear, small job.

Jobs a local model handles well

• Sorting and tagging documents or emails into known categories.
• Pulling specific fields out of consistent forms.
• Answering “where in these files is X” questions.
• First drafts of routine text from a template, reviewed by a person.
• Anything where the data really shouldn’t leave the building.

Jobs I keep on hosted models

• Anything that weighs competing constraints or needs careful judgment.
• Client-facing writing where tone matters.
• Long, open-ended tasks with many steps.

That split mirrors how I route hosted work by size, in Route AI work by how much judgment it needs. Local is just one more, cheaper lane.

What it takes

• A machine with a lot of memory. Model size is limited by memory more than anything else.
• Free software to run the model. Several good options exist and, in our testing, they performed similarly.
• Someone to set it up and keep it updated. Models and tools move fast.

There’s also a middle ground. Cloudflare’s Workers AI runs open models on their network, billed per use, so you get open models without owning the hardware (Cloudflare). We use it for some small jobs on our own site.

Should you?

For most small businesses, I’d start with a hosted business plan and good habits. Local makes sense when you have a high-volume, narrow job, a privacy reason to keep data in house, or someone on staff who enjoys tinkering. If that’s you, it’s a fun and useful project.

If you’d like to talk it through, give me a buzz. Please let me know if you have any questions, thanks!