Secrets and shared infrastructure: prompt caches, logs and system prompts

Two habits put credentials into places they cannot be retrieved from. The first is putting a key in the system prompt with an instruction that the model must never reveal it. The instruction and the secret occupy the same channel that an attacker can influence, so the control is being enforced by the thing under attack, and extraction of system prompt content under sustained adversarial pressure should be treated as a matter of time rather than a possibility. Base64-encoding the secret first is not secrecy: it is an encoding anyone can reverse, and it survives casual extraction no better than plaintext because the extracted string is just as usable.

The sound pattern is that secrets never enter model context at all. Keep API keys server-side and have the platform attach them to outbound tool calls at execution time, outside the context window, and scope credentials per tool invocation so that a leaked value has minimal reach and a short life, as covered in the tool design module. The model asks for an action; the platform decides whether the action is authorised and supplies the credential. The model does not need to know a credential exists.

The same reasoning extends past the prompt. Secrets leak into tracing spans that capture full request payloads, into error messages that include the failing request, into evaluation datasets curated from production traffic, and into the transcripts that get pasted into bug reports. Redact at capture in each of those paths, and test the redaction, because a redaction rule written against last quarter's payload shape fails silently when a field is renamed.

Caching: a shared optimisation on a shared path

A multi-tenant inference gateway caches responses keyed on a hash of the normalised prompt, to cut cost and latency. Tenants sometimes embed customer records in their prompts. The hazard is direct: if tenant identity is not part of the cache key, then a prompt from tenant B that normalises to the same value as one from tenant A returns A's cached response verbatim. Whether B can produce such a prompt depends on how guessable the prompt is, and prompts built from templates over modest parameter spaces are very guessable. There is a second, quieter leak: response timing distinguishes a cache hit from a miss, so an attacker can confirm whether a particular prompt has been submitted before, which reveals what other tenants have asked even when the response is never returned.

The fix is to partition caches per tenant, and more precisely to include in the cache key every input that could change who is entitled to the response: tenant, principal where results are user-specific, the entitlements or document set used to build the context, the model version and the system prompt version. Semantic caches that return an answer for a near-match rather than an exact match deserve particular care, because they will happily serve one tenant's answer for another tenant's similar-but-not-identical question. Encryption of the cache store is the tempting non-answer: the leak occurs through legitimate reads on a correctly functioning system, not through theft of the disk. Cache staleness and hash collisions are real engineering concerns but neither is the cross-tenant security issue.

# Unsafe: two tenants that build the same prompt share an entry
key = sha256(normalise(prompt))

# Safe: every input that affects entitlement is bound into the key
key = sha256(b"|".join([
    tenant_id,                 # hard partition
    principal_id,              # if the response is user-specific
    entitlement_set_version,   # documents the caller may see
    model_version,             # pinned identifier, not a floating alias
    system_prompt_version,
    normalise(prompt),
]))

# And bound what caching can do at all:
#  - never cache responses derived from retrieval over restricted documents
#  - set a short TTL so a revoked entitlement stops mattering quickly
#  - add jitter or constant-time lookup where hit timing is itself sensitive