CASE 03 — AI SAFETY ARCHITECTURE
The product serves K-12 students, so some things a student types need a response the tutor is not qualified to give. The question is where that check lives — and the answer is: not inside the model it is supposed to constrain.
Fig. 3 — Where the check sits
The simplest version is a line in the teaching prompt: if you notice something concerning, stop the session and escalate. It works, most of the time, and it is what most products do.
It has one structural problem. The check and the thing being checked are the same model reading the same instructions — so a single prompt injection, jailbreak, or sufficiently insistent student defeats both at once. There is no second opinion, because there is no second reader.
So the check runs as a separate classifier, before the teaching model sees the text at all. It is a small purpose-built model with one job and a structured output.
The property that buys is specific and worth stating plainly: a compromised teaching prompt cannot suppress a check that already ran. Whatever a student talks the tutor into, they cannot talk it into un-seeing what the classifier returned, because the classifier is not listening to the tutor.
The teaching model's objective pulls against this task. Its whole purpose is to keep a student working, which biases it toward continuing — exactly the wrong bias for a check whose costly error is missing something.
A separate classifier can be tuned for the error that matters without fighting the teaching objective, because it has no teaching objective to fight.
The classifier returns a structural verdict — a flag, a level, a confidence — and that is what the orchestration layer receives. The raw utterance does not cross into the teaching model unless something explicitly decides it should.
That boundary is a data-handling decision as much as a safety one. Student records carry real obligations, and the narrower the set of systems that ever see sensitive free text, the smaller the surface you have to reason about.
A safety classifier you have not measured is a claim, not a control. This one has an eval harness next to it — a corpus of labelled utterances in version control and a runner that reports against it — so a prompt change is a measurable regression rather than a vibe.
On the ordinary chat path there is a second, cheaper layer: a keyword scan on the order of a millisecond, then a verification pass that filters the false positives a regex inevitably produces. Both run off the response path, fire-and-forget, so a student never waits on a safety check — and an alert reaches a teacher rather than being handled by the model.
Two independent layers, neither able to silence the other, and neither able to slow down the thing a student is actually there for.