Jailbreaks, Automated Red Teaming and Multi-Turn Attacks

On a public chatbot, a user builds an elaborate fictional role-play frame and induces the model to produce content its policy forbids. No third-party content is involved and no tools are called. This is a jailbreak: an authorised user subverting the model's policy through their own input channel. The distinction that matters, and that gets tested, is against indirect prompt injection, which requires attacker instructions to arrive inside data the model was asked to process on somebody else's behalf. That third party is what creates the confused-deputy problem, where the model acts with the user's authority on the attacker's instructions. In the role-play case there is no third party and no confused deputy: the person issuing the instructions is the person the model is serving.

Automated red teaming and the success rate it produces

Automated red teaming puts a generator model in front of the target and a judge behind it: candidate attacks are produced at scale, the target responds, and the judge scores whether the attack succeeded. It is genuinely valuable, mostly for regression testing and for covering breadth that humans cannot. It also produces a number that is easy to misuse.

Consider a report: 50,000 attack prompts, a 3 percent success rate, improved from 8 percent last quarter. Then the small print: the attacker model, the seed pool and the judge model were all upgraded between runs. The most important caveat is that attack success rate is a joint property of the target, the generator, the seed pool and the judge. Changing any one of them moves the number without touching the target's actual robustness. A weaker generator, a narrower seed pool or a stricter judge all produce an improvement that means nothing. Trend claims therefore require a frozen benchmark, run unchanged alongside the evolving one, so that you can separate a change in the target from a change in the measuring instrument.

Multi-turn escalation and the unit of evaluation

A guardrail classifies every user turn in isolation and blocks any turn that is individually unsafe. A tester reaches restricted output over twelve turns, none of which is unsafe on its own. The diagnosis is that the unit of evaluation is wrong. Escalation is a property of the conversation, not of any turn within it, so risk has to be assessed over accumulated context and trajectory. This gradual, turn-by-turn pattern, where each request is a small step beyond the last and none is refusable on its own, is commonly called a crescendo attack, and the name is worth carrying because you will meet it in tooling and in incident write-ups. Each of the alternative fixes fails in an instructive way. Lowering the per-turn threshold trades false negatives for false positives and still misses any path that is spread across enough turns, because the attacker simply takes smaller steps. Truncating the context window does not remove the escalation, it removes the product's usefulness, and the attacker re-establishes state by summarising it themselves. And calling this poisoning misclassifies it: nothing about the model has changed.

It is worth being candid: no current approach to multi-turn safety is robust. Trajectory scoring raises the cost of escalation and catches unsophisticated attempts; it does not close the class. Decomposition across sessions and accounts remains largely open. This is one of several places in this course where the professional answer is to reduce what the system can do when it is wrong, rather than to promise that it will not be wrong.