CASE 02 — COST & INSTRUMENTATION
Inference was running about $20 per student per term. At 300 students that is roughly $6,000 a term for a company operating on a $100K raise. The money was going somewhere specific, and it wasn't where anyone assumed.
Fig. 2 — Cache lifetime against real turn gaps
A tutoring session is not a chat. The student reads a hint, works the problem on paper, tries something, gets stuck, comes back. The gaps between turns are minutes long — and that turns out to be the whole story.
This is the kind of fact you only get from watching real sessions. It isn't in any benchmark.
Every turn sends a large static prefix — the system prompt plus the tool definitions, 14 to 36 thousand tokens depending on which agent is running. Prompt caching was already enabled, at the provider's default five-minute window.
Five minutes is shorter than a student thinking. The cache expired in most of the gaps, and the entire prefix went back over the wire at full price, turn after turn, for every student.
Cached reads bill at a tenth of base price in both windows — only the one-time write premium differs, 1.25× for five minutes against 2× for an hour. So the hour wins whenever a session outlives five minutes, which is the common case rather than the edge case.
While I was in there I audited which agents were marked at all. Two had never been cached and were paying full input price on every turn. One I deliberately excluded: its prompt is about 700 tokens, below the 1,024-token minimum cacheable prefix, so a marker there could never produce a hit.
I didn't want to claim a number I had estimated, so I built the ledger. One immutable row per call, with cache reads broken out as their own token bucket rather than summed into input — which is exactly what lets you see the hit rate directly instead of inferring it from a falling bill.
It also counts tutoring turns per student per day. That one isn't a cost metric; it's a teaching metric. You want it trending down across a term, because that means the scaffolding is working and students need the tutor less.
The whole change sits behind one environment variable. Set it back to five minutes and the request is byte-identical to the old one — an instant revert with no deploy, shipped at the same time as the optimisation rather than after something broke.
~$20 → ~$10 per student. ~$6,000 → ~$3,000 per term at 300 students, measured rather than estimated. A separate fix in the same pass removed a continuation API call that was firing on every Canvas lesson opener and producing zero characters — about 19,000 tokens and 2.2 seconds of dead time, on the most-run path in the product.