The Model and Data Supply Chain

Very few organisations train foundation models. Almost everyone assembles systems from artefacts other people produced: base weights from a public hub, adapters and fine-tunes from a research group, tokenizer and vocabulary files, embedding models, evaluation datasets, container images for the serving stack, client libraries, agent tool packages and prompt templates copied from a blog post. Each of those is a dependency with an author, a distribution channel and an integrity story, and collectively they are a supply chain that most software inventories do not yet cover.

The risk that surprises people most is that a model file can compromise the machine that opens it, before a single token is generated. Several widely used checkpoint formats are built on general-purpose object serialisation, and general-purpose object serialisation reconstructs arbitrary objects by invoking arbitrary callables recorded in the file. Deserialising such a file is therefore equivalent to running code supplied by whoever produced it. A developer who downloads a pretrained model to fine-tune internally has already lost if the file was crafted: loading is the compromise, and the model never has to be queried. This is why formats that carry only tensor data, with no capacity to describe executable objects, are strongly preferred, and why loading an untrusted artefact belongs in a sandbox on a machine with no credentials worth stealing.

# Intake policy expressed as pipeline configuration, not as guidance
model_intake:
  allowed_formats: [tensor_only]        # reject object-serialisation checkpoints
  require_pinned_revision: true         # exact revision, never a moving tag
  require_publisher_checksum: true
  load_for_scanning_in: isolated_sandbox
  record:
    - source_url
    - revision_hash
    - artefact_sha256
    - licence
    - date_retrieved
    - approver

1. Identify the artefact preciselyRecord publisher, exact revision or commit, file hash and format. A moving tag is not an identity: the bytes behind it can change without notice, and re-review has to be triggered when they do.
2. Reject formats that can carry codePrefer checkpoint formats that hold tensor data only. If an object-serialisation format is unavoidable, treat the file as untrusted executable content for every step that follows.
3. Verify integrity against the publisherCheck the hash the publisher provides, and prefer a signature where one exists. A hash proves the bytes match what was published; only a signature over a key you trust independently proves who published it. Neither proves that what was published is benign, so this is a necessary step rather than a sufficient one.
4. Load first in an isolated environmentDeserialise on a host with no production credentials, no secrets and no network path worth having. Loading is the moment of exposure for code-bearing formats, so it must not happen on a build server that holds deployment keys.
5. Evaluate behaviour before adoptionRun your own evaluation, including targeted probes for the behaviours you most need absent. Accept that this cannot rule out a well-hidden backdoor, and record that residual uncertainty explicitly.
6. Record the decision and the trigger for re-reviewCapture source, revision, hash, licence, approver and date in an inventory. Define what forces a fresh review: a new upstream revision is a new dependency, not a patch.

The data and annotation chain

Behind the weights sits a longer and murkier chain. Web-scale corpora are assembled by crawling sources that are open to public contribution. Curated datasets are redistributed, mirrored and remixed until the original provenance is difficult to reconstruct. Annotation and preference data are typically produced by contracted workforces. Every one of those links is an opportunity to influence the finished model without touching a line of your code. A supplier that systematically mislabels safety-relevant examples changes what the model refuses; a contributor who plants crafted samples in a scraped source can aim for a specific triggered behaviour; a dataset assembled without regard to lawful basis embeds personal data in weights from which it cannot simply be deleted.

If you fine-tune, this chain becomes partly yours. You now own the provenance of your tuning data, the deduplication that reduces memorisation, the vetting of anyone who can add examples, and the evaluation that would notice a behavioural shift. Fine-tuning is often described as customisation, but from a security standpoint it is participating in the training pipeline, with the persistence properties that implies.

What provenance you can actually establish

• An inventory of every model and dataset artefact in production, with source, exact revision, hash, licence and the person who approved its use.
• Lineage for anything you produced: which base model, which tuning data at which version, which evaluation run, which config.
• Integrity verification at deployment, so the artefact serving traffic is provably the artefact that was reviewed.
• Behavioural evaluation before adoption and after every retrain, including targeted probes for the behaviours you most need the model not to have.
• A documented re-review trigger: a new revision of an upstream artefact is a new dependency, not a patch.

Be honest about the ceiling here. For a model pretrained on web-scale data by a third party, full provenance is not obtainable by anyone, including the publisher. You can verify that the file you loaded is the file you reviewed, and you can characterise behaviour empirically, but you cannot enumerate what went into the corpus. That residual uncertainty is a real and largely irreducible risk, and the honest response is to record it as an accepted risk with compensating controls downstream, rather than to imply an assurance level that no one can currently provide.