Assurance in Operation: Evidence, Monitoring, Incidents and Metrics

Everything so far produces obligations. This module is about what you can hand over when someone comes to check, and what you must be running in order to have anything worth handing over. The organising idea is an evidence hierarchy: records generated as controls operate outweigh statements about how controls are meant to operate.

What an examiner weights

1. Contemporaneous, system-generated records tying controls to specific deployed versions: evaluation results per release with thresholds and outcomes, approval trails with dates and identities, monitoring alerts and their documented resolution.
2. Records of the control operating in practice, including the exceptions: overrides, escalations, failed gates, incidents and what was done about each.
3. Independent validation reports and their findings, with evidence that findings were tracked to closure.
4. Management assertions and executive attestations, which are evidence of accountability rather than of operation.
5. Policy documents, which sit last because they demonstrate intent only.

The evidence pack for one high-impact model

For a supervisory review of a single deployed high-risk model, the pack has to show three things: that the risk was assessed and consciously accepted, that the version actually running passed defined evaluations, and that oversight operated in practice. Concretely that means the risk assessment for this use case with its approval trail and the residual risks named and accepted; evaluation results for the exact deployed version, with the thresholds applied and the outcomes recorded; and records showing human oversight operating, meaning override logs, escalations, and how each was resolved.

1. Fix the versionEstablish the exact model version, prompt version and guardrail configuration running in production on the date under review. Everything else in the pack hangs off this.
2. Show the risk was acceptedProduce the risk assessment for the use case, its approval trail, and the residual risks named and accepted by an identifiable person.
3. Show that version passedProduce evaluation results for that exact version, with the thresholds applied and the outcomes recorded. A general evaluation of the model family is not the same claim.
4. Show oversight operatedProduce override logs, escalations and how each was resolved. The exceptions are the evidence: a clean absence of exceptions invites the question of whether the control ran at all.
5. Show you were watchingProduce the monitoring configuration and the alert history with resolutions, including alerts that turned out to be nothing. False positives demonstrate the monitor was live.
6. Close the loopProduce the change and version history with approvals, and the system documentation at the deployed version, so that the pack is traceable end to end by version and date.

Continuous monitoring versus point-in-time assessment

Suppose your hosted-model provider ships quality updates to the version alias you use, without a version bump you can detect and without a notice you would recognise as significant. Your governance regime is an annual model assessment. The annual review can be eleven months late. Six-monthly halves an unacceptable exposure window to a smaller unacceptable one, and still detects nothing in between. A vendor certification that no significant changes occurred rests entirely on the vendor's definition of significant, which is commercial, not yours. The public changelog reports only what the vendor chose to publish.

monitor: assistant-prod
target: hosted model alias in use, queried through the production path
schedule: nightly
battery:
  - refusal_consistency      # 120 held-out policy probes, expect >= 0.97 pass
  - task_quality             # 500 graded reference tasks, expect >= baseline - 2pts
  - format_compliance        # structured output conformance, expect >= 0.99
  - bias_probe               # matched-pair prompts across protected attributes
drift_metrics:
  - output_length_distribution vs 30-day baseline
  - refusal_rate vs 30-day baseline
  - embedding_centroid_shift on sampled responses
alerting:
  warn:
    - any battery metric below threshold on one run
  page:
    - any battery metric below threshold on two consecutive runs
    - refusal_rate moves more than 3 standard deviations from baseline
on_alert:
  - freeze promotion of dependent releases
  - open incident record, classify by harm category
  - notify accountable owner and independent validation function
retention: results and raw responses retained as audit evidence

Drift and post-deployment obligation

A fraud model passed validation two years ago and has not been formally examined since, although fraud patterns have plainly shifted. The model owner argues the original validation still stands because the binary is unchanged. The argument confuses the artefact with the claim. A validation attests to performance on a population, so when the population moves, the attestation decays even though the file is byte-identical. Deployment starts the obligation rather than discharging it: monitor input distributions, output distributions and outcome quality against the validation baselines, and let defined degradation thresholds trigger re-validation.

Incident classification for AI failures

A support assistant leaks fragments of one customer's account details into another customer's session. The on-call engineer notes that uptime was unaffected, records it as a quality bug, and closes the ticket. The failure here is in the taxonomy, not in the engineer. Classification must key on harm: this is a cross-customer exposure of personal data through model output, which triggers severity assessment, potential notification duties and root-cause work regardless of what the availability figures say. Escalating it to the vendor does not discharge your obligations either, since you are the party with the customer relationship and, usually, the notification duty.

An incident taxonomy inherited from infrastructure operations recognises outages, latency and errors, and will therefore systematically misfile the incidents AI actually produces. The scheme needs categories such as: harmful or policy-violating output; disclosure of personal or confidential data through output; systematically unfair outcomes affecting a group; unsafe or unauthorised action taken by an agentic system through its tools; material degradation of decision quality; and successful adversarial manipulation of the system. Each needs its own severity criteria and notification triggers, and none of them are visible on an availability dashboard. Waiting for a second occurrence to confirm a pattern before invoking incident response is not prudence; for a data exposure it is delay in a process that is usually time-bound.

Metrics that evidence control effectiveness

A board pack reports fourteen published AI policies, ninety-six percent staff completion of AI awareness training, and an AI ethics committee that met four times. Every number is true and every number measures the same thing: that controls exist and that activity occurred. Policies can be unread, training can be clicked through, and a committee can meet four times without deciding anything. None of these say whether the control changed an outcome. The remedy is not longer trend lines, nor peer benchmarks, nor a swap from lagging to leading indicators; it is a shift from measuring existence to measuring effect.