Module 8 · 6 min read

Shipping safely: evaluation gates, atomic versioning and rollback

How to catch behavioural regressions that unit tests cannot see, and why reverting the code does not revert the behaviour.

A team ships prompt changes daily. Unit and integration tests pass on every change, coverage over the tool-calling paths is high, and every pull request touching a prompt file needs two reviewers. Three weeks after a small wording tweak, someone notices that injection resistance has quietly collapsed. Nothing in the pipeline was capable of seeing it, because the regression was behavioural and every gate in the pipeline measured code paths. Higher coverage would not have helped: the code did exactly what it always did, with a different prompt.

The control that closes this gap is an evaluation gate: adversarial and regression suites that run on every prompt, model or tool-configuration change, with pass thresholds enforced as a merge and deploy blocker. That last clause is what makes it a gate rather than a dashboard. A suite that produces a score nobody is obliged to act on will be ignored on the day it matters, which is the day someone is shipping a fix under time pressure.

A workable suite has a few properties. It is versioned alongside the code, so you can see which change moved which metric. It keeps a held-back private portion that is not used during development, because a suite everyone iterates against becomes a target to be optimised for rather than a measure. It accounts for non-determinism by sampling each case several times and thresholding on a rate rather than on a single pass or fail, otherwise the gate flaps and people learn to re-run it until it goes green. It covers the classes that matter for your system: direct and indirect injection, tool misuse, data exfiltration attempts, refusal and over-refusal, and any domain-specific harm. And it grows: every incident and every red team finding becomes a permanent case, which is how the suite stays relevant as the system changes.

Check yourself

A team runs its adversarial suite on every change and iterates on prompts until the suite is green. Six months later, real attacks succeed against classes the suite nominally covers, while the suite still reports a near-perfect score. What most likely went wrong?

External red teaming remains valuable and the two are complementary rather than alternatives. A red team finds the classes of attack your suite does not contain; the suite makes sure those classes never come back. Treating a periodic engagement as the control, though, leaves the daily change surface unguarded between engagements.

Rollback fails when versioning is partial

An incident forces a rollback. The team reverts the application deployment and the behaviour does not come back, because the prompt lives in a database row that the code revert did not touch, and the model reference is a floating alias pointing at whatever the provider currently serves as its latest version. Two of the three inputs to the system's behaviour were outside the unit that was rolled back, so the revert restored a third of the system.

The principle violated is atomic versioning of the full behavioural surface. Behaviour is the joint product of code, prompts, the pinned model version, tool and retrieval configuration, and the evaluation thresholds that were in force. All of it must roll forward and back together as one revertible unit, with model references pinned to exact identifiers rather than to aliases that move underneath you. Note why the neighbouring answers are not the principle at stake: blue-green deployment and feature flags are useful release mechanisms, but switching traffic to an environment or disabling a flag still assumes there is a coherent previous state to return to, which is exactly what a partially versioned system lacks. Reversible migration scripts for the prompt table address one component of the same underlying gap.

One revertible behavioural unit

Application image pinned by digest
Prompts in version control, referenced by version from the release manifest
Model pinned to an exact identifier
Tool config, retrieval index version and eval thresholds in the same manifest
One command returns the whole surface to a known state

Behaviour scattered across systems

Prompts edited live in a database row
Model referenced by a floating alias that moves under you
Tool config maintained in a separate console
Retrieval index rebuilt on its own schedule
Reverting the code restores one component of four

yaml

# One release manifest: everything that determines behaviour, versioned together
release: assistant-2026.07.19-a
app_image: registry.internal/assistant@sha256:7d31...
prompts:
  system: prompts/system@v41          # in version control, not a mutable DB row
  tool_preamble: prompts/tools@v12
model:
  id: provider/model-name-2026-05-01  # exact pinned identifier, never an alias
  decoding: { temperature: 0.2, max_output_tokens: 1024 }
tools:
  config: tools/config@v9
  enabled: [lookup_order, issue_refund]
retrieval:
  index: support-kb@2026-07-18
evaluation:
  suite: evals@v27
  thresholds: gates/prod.yaml@v6
rollback_to: assistant-2026.07.12-c   # a complete, re-deployable prior state

Two operational habits complete the picture. Roll out behind a staged exposure with automatic rollback triggers on the signals that actually move first: refusal rate, tool-error rate, latency, spend per session and any safety classifier rate. And keep the previous pinned model version genuinely available, which means tracking provider deprecation timelines as a dependency risk, since a rollback target that has been withdrawn is not a rollback target. When something goes wrong at three in the morning, the only question that matters is whether there is one command that returns the entire behavioural surface to a known state.