Module 6 · 8 min read
Data, Secrets, Shared Responsibility and the Audit Trail
The baseline operational controls every AI deployment needs: minimising personal data in prompts and outputs, keeping secrets out of the context window, knowing which duties a hosted provider does not take on, and logging enough to investigate.
The controls in this module are unglamorous and they are where most real incidents are decided. They share one idea: the context window is not a private space, model output is not a trusted artefact, and the boundary of your organisation is now wherever your prompts go. Getting these right does not require any AI-specific technology, only the discipline to apply ordinary data-protection thinking to a channel that does not feel like a data transfer.
Personal data in prompts and in outputs
A common pattern: support agents paste entire customer records into a third-party AI chat tool to draft replies. The moment that happens, personal data has left the systems where its retention, access control and deletion were managed and entered a service governed by someone else's terms, with its own logging and its own staff. The control that most directly reduces the risk is minimisation: redact or reduce each record to the fields actually needed for the task before it is sent. Confirming that the vendor encrypts data in transit protects the channel but not the disclosure; the data still lands in an external system. A banner telling customers that AI may be used is a transparency measure that changes nothing about exposure. And fine-tuning an internal model on the same records replaces a disclosure risk with a memorisation risk rather than removing anything.
Outputs need the same attention. Personal data can emerge from a model through memorisation of tuning data, through retrieval that was indexed without per-user access filtering, or simply because it was in the conversation earlier and is now being restated into a channel with a wider audience. Treat generated text that may contain personal data as personal data for the purposes of storage, retention and access control, because that is what it is.
Secrets
Teams sometimes place a live API key in the system prompt so the model can use it when calling tools. The principal flaw is not that the model will fumble a long random string, nor that the provider will train on it, nor that rotation becomes awkward. The flaw is categorical: anything in the context is disclosable output. Extraction techniques and injected instructions both aim at getting the model to restate its context, and the model has no notion of a value it must never emit. Secrets therefore belong server-side, in the tool layer, where the orchestrator attaches credentials to a call that the model merely requested.
# Wrong: the credential is in the context, so it is potential output.
system_prompt = (
"You can query the customer database. "
"Use API key sk_live_REDACTED_EXAMPLE when calling it."
)
# Right: the model names an intent; the orchestrator holds the secret.
def run_tool(name, args, acting_user):
if name not in allowed_tools_for(acting_user):
raise PermissionError(name)
validate_against_schema(name, args)
credential = vault.fetch(name, acting_user) # never enters the context
return TOOLS[name](args, credential)Secrets and personal data: do this
- Hold credentials in the tool layer and attach them server-side to a call the model merely requested.
- Reduce each record to the fields the task actually needs before it crosses into the AI system.
- Treat generated text that may contain personal data as personal data for retention and access purposes.
- Assume every token in the context is potential output, and design the context accordingly.
Not this
- Place a live API key in the system prompt so the model can use it when calling tools.
- Paste whole customer records in and rely on the vendor's transport encryption as the control.
- Assume the model will decline to repeat a value because you told it the value is confidential.
- Treat a transparency banner as a substitute for reducing what is disclosed.
Shared responsibility for hosted models
Consuming a hosted foundation-model API divides duties in a way that resembles other cloud services, and the same misunderstanding recurs: customers assume the provider's security work covers more of their risk than it does. The provider runs and patches the serving infrastructure, secures its data centres, defends the weights it hosts, isolates tenant traffic, and does the safety training and alignment of the base model. None of that touches the decisions that cause most application-level incidents.
| Typically the provider | Always the customer |
|---|---|
| Physical and infrastructure security of the serving estate | Deciding what data is placed into prompts and for how long transcripts are kept |
| Patching model-serving hosts and accelerators | Authenticating and authorising which people and services can reach the AI feature |
| Safety-training and aligning the base model | Validating and encoding model output before it reaches a browser, a database or a shell |
| Isolating one tenant from another | Scoping the credentials the tool layer holds and gating consequential actions |
| Protecting hosted weights from theft | Application logging, retention, and the legal basis for the processing |
Check yourself
A provider publishes an independent assurance report evidencing strong tenant isolation and a mature patching regime for its model-serving estate. A team cites this report to close a finding about customer data exposure through their own AI feature. What is wrong with that reasoning?
The report addresses provider-side duties. The exposure in question comes from decisions that never transfer: what the team places in prompts, who may reach the feature, and what happens to the output downstream. Provider assurance is necessary evidence about the provider, and no evidence at all about the customer's own controls.
Logging and auditability
When something goes wrong you will need to reconstruct what was asked, what was retrieved, what the model produced and which actions fired, for a specific caller at a specific time. That means logging prompts, responses and tool invocations along with caller identity and timestamps. It also means recognising that such a log is now one of the most sensitive stores you operate, because it contains every prompt anyone ever wrote. So it must carry access controls, retention limits, and the same protection you would give the source data.
- Caller identity, session identifier and timestamp on every request.
- The assembled prompt and the generated response, or a defensible reduction of them, with the retrieval sources cited.
- Every tool invocation with its arguments, the credential scope used, and the result status.
- Model and prompt-template versions, so behaviour can be tied to a configuration.
- Access control, retention limits and monitoring on the log store itself.
Try it first
It is Monday morning. A pricing document has appeared on a competitor's site and you suspect the assistant was involved. Before reading on: list the specific things you must be able to answer, and therefore what your logs had to contain on Friday.
You need to establish who was acting, what the model was given, what it produced, and what it did. Concretely that means: the caller identity and session for every request, so you can bind activity to a person or service; timestamps, so you can bound the window; the assembled prompt including which documents retrieval supplied, so you can find the payload; the generated response, so you can see what was proposed; every tool invocation with its arguments, result and the credential scope used, so you can see what actually executed; and the model and prompt-template versions, so you can tie behaviour to a configuration. None of that is reconstructable from aggregate token counts, and none of it is available from the provider, who has neither your identities nor your application context. Note also the second-order duty: a store containing every prompt anyone ever wrote needs its own access controls and retention limits, or your investigation capability becomes your next incident.