Context is the cost: what we learned measuring our own AI token use

I use AI heavily in our own development work, so I wanted to know where the tokens actually go. Not a guess, a measurement. We logged the size of every request across our AI coding sessions and broke it down. What came back changed how I set up every AI workflow, ours and our clients’.

The number that mattered

Over five days in early September, across 168 sessions and roughly 120,000 requests, here’s what we found:

• At the START of a session, the median request carried about 30,000 tokens. That’s the instructions, the tool definitions, the project notes. The stuff people usually try to trim.
• On a TYPICAL request during the session, the median was about 305,000 tokens.
• About 27 percent of requests carried more than 500,000.

That gap, from 30,000 to 305,000, is the whole story. It wasn’t the instructions. It was everything the session had picked up and never put down: file contents, search results, command output. Every one of those got re-sent on every following request.

The biggest source by far was command output, a little over half of the growth. Most of it was ordinary stuff, a search here, a file listing there, about 1,300 tokens each. No single call was the problem. Thousands of small ones were.

Context is what you keep, not what you write

That’s the lesson in one line. A model re-reads the whole conversation every time. So the cost of a session is roughly the number of requests times how much is in the conversation on each one. Adding a big document early makes EVERY later request more expensive.

Once you see it that way, the fixes are pretty obvious:

• Start a fresh session for a new task, instead of continuing a long one.
• Pull in the part of a document you need, not the whole thing.
• Keep long output (logs, test results, data dumps) in a file and look up the lines that matter.
• Summarize what’s been decided and start over from the summary.

Prepared packets beat letting the AI explore

The second measurement surprised me more. In mid September we had a small, cheap model review code two different ways.

• The exploring way: the model was free to read whatever files it wanted. Each run took 45 to 83 requests and averaged about 3.1 million tokens.
• The packet way: a script gathered the relevant excerpts into one prepared file, and the model read it once and answered. 4 requests, about 72,000 tokens.

Same model, same kind of question. The packet run cost about 2 percent of the exploring run and finished in under a minute. The model wasn’t the difference. The number of round trips was, because each one re-sent the growing conversation.

To be fair to the exploring approach, the packet run had one weakness. It got the substance right but reported line numbers from the packet instead of from the real files, so every line number was off. We now stamp the real file and line on every excerpt. Cheap to fix, and a good reminder to check the specifics.

What this means for a business workflow

You probably aren’t running AI coding sessions, but the shape is the same in a support chat, a document assistant, or any workflow that keeps a running conversation.

• Don’t let a chat grow forever. After a few turns, summarize and carry the summary forward.
• Don’t paste the whole manual into every request. Retrieve the two sections that answer this question.
• Prepare the input. A clean, focused packet of exactly what the model needs beats letting it wander.
• Put the stable parts first so they can be cached. I covered that in Prompt caching for owners.

And then send the job to the smallest model that can do it, which is its own article.

Measure before you optimize

The most useful thing we did was measure before changing anything. My first guess was that the instructions were too long. They weren’t the problem at all. If you’re spending real money on AI, ask your builder for the median request size and how it changes over a session. That number will tell you more than the monthly total will.

If you’d like help reading those numbers for your own workflows, let me know. Thanks!