Module 1 · 7 min read
Why AI Risk Breaks the Classic IT Control Model
How probabilistic, drifting systems invalidate the assumptions underneath configuration baselines, change control and point-in-time assessment.
Classic IT risk management rests on an assumption so basic that most control frameworks never bother to state it: a system does what it was configured to do, and its behaviour changes only when somebody changes it. That assumption is what makes configuration baselines, change tickets and annual control assessment coherent as a set. Verify the configuration, control the changes, and you have bounded the behaviour until the next change. AI systems violate the assumption at the root, and almost every governance failure you will meet in practice traces back to a framework that was never rebuilt around that violation.
Consider a bank that governs a credit decision model with the control set it uses for its payment switch. There is a configuration baseline. There is a change ticket for every deployment. There is an annual control assessment, and it passes cleanly. Six months after go live, approval rates for one segment of applicants have moved sharply and the model separates good from bad borrowers noticeably worse in that segment. No control failure was recorded, because no control failed. The binary is the binary that was approved. The configuration matches the baseline. What changed was the world: the applicant population shifted, and the statistical relationship the model learned no longer holds as well. The control set was built to detect change events, and there was no change event.
Four assumptions that do not survive
- Determinism. A traditional application given the same input returns the same output. A model returns a prediction with an error rate, and generative systems may return different text on identical input. Correctness is a distributional property, so a single passing test proves very little.
- Change-triggered risk. In classic IT, risk moves when configuration moves. In AI, risk moves when the input population moves, when an upstream data source changes shape, when user behaviour adapts, or when a hosted model is updated under a version alias you do not control.
- Complete specification. You can write exhaustive acceptance criteria for a payment routing rule. You cannot write them for a model that generalises: the specification is a training objective plus an evaluation set, and behaviour outside that set is inferred rather than guaranteed.
- Locality of failure. A software defect fails on the code path that contains it. A model failure is distributed across a population, often invisible in any single case, and detectable only in aggregate or on a slice nobody thought to cut.
| Classic IT control assumption | What an AI system does instead |
|---|---|
| Behaviour is fixed until a change is made | Behaviour degrades as the input distribution moves, with no change made |
| A passing test demonstrates correctness | Evaluation gives a rate on a sample, not a guarantee on the population |
| Point-in-time assessment bounds risk to the next assessment | Risk between assessments is unobserved and can be substantial |
| Failures are discrete, attributable events | Failures are statistical and often silent until harm accumulates |
| The change record is the audit trail | The behavioural monitoring record is the audit trail |
- Distribution shift
- The live input population diverges from the population the model was trained and validated on. Also called data drift when the inputs move.
- Concept drift
- The relationship between inputs and the correct answer itself changes, so a model can be fed familiar-looking inputs and still be systematically wrong.
- Silent failure
- Degradation that produces no error, no alert and no outage, because the system is functioning exactly as engineered and simply predicting worse.
- Point-in-time assurance
- Evidence that a control operated at, or over, a defined past window. Useful, but it makes no claim about the period after the window closed.
Drift, in one tap
Select a card to turn it over.
Check yourself
An insurer's claims model receives an input population that is statistically indistinguishable from last year: same claim types, same amounts, same regions. Its accuracy has nonetheless fallen sharply, because organised claimants have changed how they construct fraudulent claims. What is the most precise description?
Data drift moves the inputs; concept drift moves the mapping from inputs to the correct answer. Monitoring input distributions alone would show nothing at all here, which is exactly why outcome quality has to be monitored as well as input shape.
Try it first
The bank proposes moving its control assessment from annual to quarterly. Why does that not solve the problem?
Because it changes the frequency of the wrong measurement. The assessment verifies configuration, and the configuration was never wrong. Quarterly assessment still leaves three months of unobserved behaviour, and it still has no instrument pointed at outputs. What is needed is a different kind of observation, not a shorter interval between repetitions of the same one.
None of this means discarding classic risk practice. Asset inventory, named ownership, segregation of duties, access control, change management and incident response remain the backbone, and an AI governance programme that neglects them will fail for entirely ordinary reasons. The point is that AI adds a behavioural dimension that the classic set has no instrument for. Practically, that means three additions: baselines expressed as behavioural thresholds rather than configuration hashes, monitoring that compares live behaviour against those baselines, and defined triggers that force re-validation when the comparison moves. Everything else in this course is a consequence of taking that seriously.
One implication is worth flagging now, because it shapes tiering, oversight and incident response alike. Because a model can fail without anyone touching it, the question a supervisor will eventually ask is not what did you approve, but what did you know, and when did you know it. Approval evidence answers only the first. Monitoring evidence is the only thing that answers the second, and a governance programme that cannot answer the second will find that its careful approval records make the gap more visible rather than less.