Securing the AI model supply chain: signing, provenance and safe formats

The risks

Code execution on load. Hugging Face's own documentation warns that pickle, the default format for PyTorch weights, can enable arbitrary code execution when a file is loaded, and recommends safetensors as a safer alternative¹. Security researchers have reported malicious models on public hubs that used this to plant backdoors, and noted that pickle isn't the only risky format: some Keras models can run code through Lambda layers².

Scanners are imperfect. A 2026 study of malicious pickle-based models reported that existing scanners varied widely in how well they detected them on its dataset, which is a reason not to treat a scan as proof of safety⁷.

Tampering and substitution. Without signatures, there's no reliable way to know a model file is the one its publisher released, or that the copy in your registry hasn't changed since it was approved.

Unknown lineage. Fine-tuned models inherit everything about their base model and training data, including licence terms and possible poisoning. Without records, you can't answer basic questions in a security review or due diligence.

The building blocks

The checklist

1. Use an internal model registry. Production never pulls directly from public hubs. Models are imported, reviewed and stored internally.
2. Prefer safe formats. Accept safetensors or equivalent where possible; convert or reject formats that execute code on load. Load untrusted files only in isolated sandboxes.
3. Scan on import, while remembering scanners miss things. Scanning reduces risk; it doesn't certify safety.
4. Pin by hash, not by name. Reference models by immutable digest so "latest" can never silently change what runs.
5. Sign what you produce and verify what you run. Sign fine-tuned and approved models at the end of your pipeline, and verify signatures at deployment.
6. Generate provenance in your training pipeline, aiming for at least signed provenance from a managed build platform, in the spirit of SLSA Build L2.
7. Keep an ML-BOM per model: base model, datasets, licences, training configuration and evaluation results, linked to your AI system register.
8. Restrict runtime permissions. Model-serving containers need no outbound internet access and minimal privileges, enforced with the policies in Kubernetes admission policies.
9. Re-evaluate after every model change with your evaluation set, including safety and behaviour tests, not just accuracy.

Beyond the weights

The supply chain extends past model files. Datasets, tokenisers, prompt templates, agent tools and MCP servers all shape behaviour. MCP servers in particular deserve the same allow-listing and review, as covered in securing MCP servers. Google's announcement of model signing noted plans to extend the approach to datasets and other ML artefacts⁴, which is the right direction.

Why this matters now

Enterprise customers and regulators are starting to ask where models come from and how they are controlled. An ML-BOM, signed models and a registry answer those questions concretely, and support governance frameworks such as those compared in ISO 42001 and the NIST AI RMF. They are also inexpensive to adopt early and painful to retrofit after an incident.

How I can help

I design secure AI platforms, from model registries and signing to provenance, runtime controls and evaluation, for organisations running private and open-weight models. See my AI architecture work, or analyse your challenge and bring the result to a call.