In a pickle: Why every pickle defense in machine learning is bypassed the same way

The machine learning (ML) ecosystem recently shipped a series of defenses against malicious model files. PyTorch restricted what its loader will deserialize. Keras added a safe mode that refuses to execute embedded code. Vendors built scanners to inspect model files at rest and reject the dangerous ones.

But none of them held up to six documented CVE bypasses. These bypasses, often called “pickles,” let a malicious Python data file slip past security scanners undetected. And after the Sysdig Threat Research Team (TRT) pulled apart every public pickle bypass on record, a pattern emerged.

Before we explore that pattern, let’s first understand what a pickle actually is and why machine learning (ML) is full of them. Then, we’ll analyze which model formats are most dangerous, what the Sydig TRT measured when we ran malicious model loads against two detection engines, and how you can best deploy and protect susceptible models in your environment.

A pickle is a program

Pickle is a built-in Python capability for saving something in memory to file so you can load it back later exactly as it was. Many teams may be pickling and they don't even realize it. torch.save() or joblib are both pickle, and that hidden misunderstanding increases exposure and makes it easy for something to go wrong without anyone noticing or understanding why.

Most file formats are data. You parse them, get a structure back, and nothing inside the file gets a vote in what the program does next. A malformed JPEG produces an error, for instance, not a shell.

Pickle is not that kind of format. pickle.load() doesn’t parse; it executes. A .pkl file is a sequence of instructions for a small stack-based virtual machine built into CPython, and loading the file means running them. That’s not an implementation flaw or a parser bug that gets patched, but the inherent design.

The easiest way to believe it is to read one. Pickle's protocol 0 is human-readable by design, and this is a complete, working payload:

The snippet above includes five instructions, and it reads like assembly because functionally that is what it is:

• GLOBAL resolves and pushes a callable.
• MARK and TUPLE assemble an argument tuple.
• REDUCE calls the callable with those arguments.
• STOP ends the program.
• Swap "id" for something with more ambition, and nothing about the structure changes.

That’s the entire remote code execution (RCE) story, and it’s only three operational codes (opcodes). Import something, build an argument tuple, and call it. Everything else in the format — later protocol versions, the memo table, framing, out-of-band buffers — is optimization around that core.

And because the payload is only opcodes, there are no offsets to leak and no build to fingerprint. One that works on a given Python works on any Python that can parse the protocol.

You wouldn’t normally handwrite those opcodes, either. Python will emit them for you through a documented extension point:

__reduce__ is how an object tells pickle to reconstruct it: The method names a callable and its arguments, and pickle invokes them on load — which is how objects that cannot be naively serialized rebuild themselves on the other side. The attack does not subvert that. It uses pickle exactly as specified, and nominates os.system as the callable.

Look at this script, for example:

pickletools.dis disassembles the byte stream without running it, and the output reads like a listing: GLOBAL, MARK, SHORT_BINUNICODE, TUPLE1, REDUCE, STOP. Disassemble freely. Run pickle.loads on that payload only somewhere you are willing to lose.

The consequence carries big implications

If a pickle is a program, then a file that is invalid as a whole can still be perfectly effective in part. The VM executes opcodes as it encounters them. It doesn’t validate the stream up front and then run it because there is no up front. It runs until it finishes, or until it reaches something it cannot execute. Everything before that point has already happened.

Data would be parsed and then used, and a corrupted file would yield nothing. A program runs until it faults.

Attackers noticed, and we will come back to that in the third section.

Why is machine learning full of pickles anyway?

A pickle solves a real problem, and the way it solves that problem requires precisely the capability that makes it dangerous.

The problem is general object serialization. Writing a list of integers to disk is easy. Writing an arbitrary Python object, however, is not.

Objects hold references to other objects, to classes defined in modules that may not be imported yet, to state that is only meaningful once something reconstructs it. There is no universal declarative description of "how to rebuild this." The general answer is to record a procedure — call this thing, with these arguments — and run it at load time.

That is what __reduce__ is, and the standard library specifies it in exactly those terms. The first two items of a __reduce__ tuple are:

• A callable object that will be called to create the initial version of the object.
• A tuple of arguments for the callable object.

Read that as an attacker and the exploit writes itself. Read it as a language designer, though, and it is the only general answer available.

So arbitrary code execution is not a bug in pickle, but a feature. Python has never been coy about it, either. The warning sits at the top of the module documentation:

The docs even name the mitigation. A section titled "Restricting Globals" observes that for many applications the default behavior "is unacceptable as it permits the unpickler to import and invoke arbitrary code," and shows how to subclass the unpickler to allow only what you expect.

So how did machine learning end up built on pickles?

torch.save serializes with pickle. Saving a model is one call, and the artifact is a .pt, .pth, or .bin, none of which look like a pickle to anyone reading a file listing. scikit-learn's persistence guidance points to joblib, a pickle wrapper optimized for objects carrying large NumPy arrays, which is exactly what a fitted estimator is. NumPy's own .npy is a plain numeric container right up until an array contains Python objects. At that point it needs pickle, which is why the allow_pickle flag exists at all.

Nobody in that chain chose arbitrary code execution. They chose "save my Python object," which is the obvious API to want, and pickle is the batteries-included way to provide it.

The result is a supply chain that’s majority-pickle. In a survey of 705,991 models on a Hugging Face mirror, PyTorch's .pt, .pth, and .bin accounted for 335,893 models (51.0%), with explicit pickle variants (.pkl, .pickle, .joblib, .dill) adding another 26,696, or 4.1%. More than half the models you might download execute code the moment you load them.

Why "just fix pickle" isn’t on the table

You can’t make pickle.load both general and safe, because generality is the thing that requires the call. However, you can do one of two narrower things:

• Restrict the loader. Keep the format, but allow only a known-safe subset of callables. This is the standard library's own suggestion, and PyTorch's weights_only=True and Keras's safe_mode are that idea applied to model files.
• Change the format. Store only what a model actually needs, such as named arrays of numbers, in a container with no way to express a call at all. That’s safetensors.

Both these approaches are sound in principle. Their records in practice are the subject of the next section, and only one of them has survived contact.

Which models are actually vulnerable

Which model files should be treated as code? The short answer is: more of them than the name suggests, and almost none of them are called .pkl.

Format

Executes on load?

Notes

.pkl.pickle.dill

Yes

The plain case

scikit-learn .joblib

Yes

joblib wraps pickle

PyTorch .pt.pth.bin

Yes

A ZIP-wrapped pickle. weights_only=True since 2.6; see below

NumPy .npy.npz

Only with allow_pickle=True

Default False since CVE-2019-6446. Object arrays require it, which is how it gets turned back on

Keras/TensorFlow .h5.hdf5

Yes

Lambda layers, and the legacy path ignores safe_mode

Keras .keras (v3)

Yes, via bypass

See CVE-2025-9906

TensorFlow SavedModel

Yes, in effect

Graph ops plus custom layers

safetensors

Named tensors and a JSON header. No opcode, no callable

GGUF

No known path

Tensors and metadata for quantized inference

ONNX

No known path

Protobuf compute graph, predefined ops only

Note the phrasing on the last two: no known code execution path. These formats are younger and attacked less often than pickle, and the absence of published CVEs is much weaker evidence than a format that cannot express a call in the first place.

Strategy one: Restrict the loader, get bypassed

Keep the format, allowing only a known-safe subset of callables: This is the standard library's own advice. Both major frameworks took that advice, but were still bypassed.

PyTorch flipped torch.load's weights_only argument from False to True by default in version 2.6, explicitly on security grounds. The restricted unpickler permits tensors, storages, and basic collections, and refuses everything else.

Except that in 2.5.1 and earlier, weights_only=True was already bypassable. CVE-2025-32434 found that _legacy_load — the path handling older tar-based checkpoints — did not actually enforce the restricted unpickler on everything it processed. The flag was set and the code that ran didn’t honor it.

Then it happened again. CVE-2026-24747, published in January 2026 with a CVSS of 8.8, defeats the restricted unpickler's validation with crafted opcodes, corrupting memory to reach code execution. It affects every version before 2.10.0, which means the fix for the 2025 bypass was itself exploitable across the entire 2.6 through 2.9 range. If you pinned PyTorch 2.6 in early 2025 because it was the secure default, you were exposed for four minor versions.

Keras has the same shape and more iterations. Lambda layers carry marshalled Python and execute it on load (. Keras made safe_mode=True the default from 2.13, which refuses to deserialize them.

Then, with CVE-2025-9905, loading a legacy .h5 file ignores safe_mode. No warning, no error. An attacker only has to author the checkpoint in an older format, and the current library will load it and run the code anyway.

Finally, CVE-2025-9906, a crafted .keras archive whose config.json calls keras.config.enable_unsafe_deserialization(), and then drops a Lambda layer. The file disables the check that’s inspecting it.

In that same survey of 705,991 Hugging Face models, researchers found 91 malicious ones, and 15 of them used Keras Lambda layers.

Strategy two: Scan the file before you load it, miss malicious files anyway

Check the artifact at rest and reject it if it looks malicious. This is what Hugging Face runs on upload, and what most continuous integration (CI) gates use.

Sonatype disclosed four picklescan vulnerabilities in 2025, and three of them describe the same failure with different mechanics:

• CVE-2025-1889: Standard detections keyed on file extension rather than content. If attackers renamed the file, it was never scanned.
• CVE-2025-1944: A deliberately malformed ZIP makes picklescan crash with BadZipFile. PyTorch's more forgiving ZIP reader loads the same file without complaint.
• CVE-2025-1945: If an attacker modified ZIP flag bits, picklescan fails to detect and PyTorch still loads.

Then there is nullifAI, found by ReversingLabs on Hugging Face in February 2025. The models were compressed with 7z instead of PyTorch's default ZIP. Picklescan didn’t parse them.

One bug, six times

Vulnerability

The gap

PyTorch CVE-2025-32434

weights_only not enforced on the _legacy_load path

Keras CVE-2025-9905

safe_mode ignored on the legacy H5 path

Keras CVE-2025-9906

The file disables the check inspecting it

picklescan CVE-2025-1889

Scanner reads extensions; the loader reads content

picklescan CVE-2025-1944 and 1945

Scanner's ZIP parser is strict; the loader's is forgiving

nullifAI

Scanner expects ZIP; the loader accepts 7z

In every case, the thing doing the checking and the thing doing the loading disagreed about what the file was. The attacker did not defeat the check, but found a reading of the file on which the check did not apply.

This is structural, not simply a run of bad luck. A checker has to anticipate how every loader, at every version and down every legacy code path, will interpret a stream of bytes. The loader only has to interpret it one way, and that’s what leads to missed malicious files.

A pickle doesn’t need to be valid to work

The nullifAI models were broken on purpose.

Recall from the first section that the pickle VM executes opcodes as it encounters them. There is no validation pass. The payload was placed at the head of the stream, so by the time deserialization reached the corruption and threw an error, os.system had already been called. As ReversingLabs described it, opcodes execute "until all opcodes are executed or a broken instruction is encountered."

A scanner that parses the whole file and reports a parse error has reported nothing useful. The victim is already compromised. Data is parsed and then used, so a corrupted file yields nothing. A program runs until it faults, and everything before the fault has already happened. Treating a model file as data isn’t just a stylistic error.

Look back at the table in the “Which models are actually vulnerable” section. The formats in the bottom rows are not safe because they are well patched, but because there is nothing in a safetensors file that can express a call — no opcode, callable, or reconstruction procedure. The security property is a consequence of what the format cannot say, which is why it does not need a version number to be trusted.

That is the whole argument for conversion. But conversion is not always available, and plenty of production systems load pickles today. So the next question is what you can see when one of them runs.

Detection

There are two places to catch a malicious model: before it loads, and while it runs. They fail in different directions, and the difference matters.

Scanning at rest

Scanning is the primary control. It is cheap, it runs in CI, it blocks the artifact before anything executes, and against a textbook payload it works. A __reduce__ gadget calling os.system is exactly what modelscan and picklescan are built to find.

The CVEs listed in the third section are the reason that is not reassuring. Every bypass in that list defeated a scanner that was working correctly on the file it believed it was reading. This includes extension checks, ZIP parsing, flag bits, 7z, and a deliberately corrupted stream. None of them broke the detection logic, but they did change what the file was on the way in.

A scanner that catches the obvious case has only told you the obvious case is caught. It cannot tell you the file is safe, because "safe" would require it to have anticipated how every loader, at every version, down every legacy path, will read those bytes: a huge amount of combinations to reason through and one that make scanning alone unfeasible. That is the asymmetry.

So the honest question is not whether to scan, but what’s left standing when the scan is bypassed.

Watching it run

Runtime detection is the layer that gets proposed as the logical answer, so let’s measure it.

We built five hostile gadget families, each one a pickle whose __reduce__ performs a different post-exploitation action, and ran them against two engines: open source Falco 0.44.1 on its default ruleset and Sysdig Secure.

Both runs fired a positive control first. All five outcomes were predicted in writing before the run.

Gadget family

OSS Falco (default ruleset)

Sysdig Secure

Sensitive file read, via shell

FIRED · Read sensitive file untrusted

FIRED · 1.6s

Sensitive file read, direct, no shell

FIRED · same rule

FIRED · 1.9s

Sensitive file read, in-process, no spawn

FIRED · same rule

FIRED · 1.5s

Composite, read plus outbound

FIRED · same rule only

FIRED · + System procs network activity

.pth startup-hook persistence

Modify Python Startup Hook File

FIRED · 1.5s · Python .pth Persistence File Written by Non-Package-Manager

All five gadget families triggered Sysdig Secure rules, all within two seconds. Four of five triggered OSS Falco defaults. Falco’s open source default ruleset contains 26 rules versus the commercial product’s 278. This includes the Falco rule Modify Python Startup Hook File, which was created after this test.

Three findings in that table matter more than the raw score, however.

First, the spawn axis is irrelevant. We expected the gadget that forks a shell to look different from the one that never leaves the Python process, but the detections don’t differ. The shell-spawned gadget was caught in 1.6 seconds, the direct one in 1.9 seconds, and the entirely in-process one in 1.5 seconds The same rule caught all three. Whether a gadget forks is a fact about its syscall surface, not about whether it’s visible.

Second, runtime itself doesn’t solve the checker vs. loader problem, but it doesn’t need to. A runtime sensor never observes the deserialization at all, but the consequences. Nothing in that table detected a pickle being loaded. What fired was a file read, or an outbound connection, that happened to originate downstream of one.

That sounds like a technicality until you read the rules that look like they should have caught this. Shell Spawned with Inline Python Command keys on proc.pcmdline containing -c together with the literal source text; import os and os.system. Base64-encoded Python Script Execution does the same thing one encoding layer down. Both assume malicious Python is visible on the command line.

And third, a pickle load is python3 load_pickle.py model.pkl. The gadget is in the file. None of that text is ever on the command line, at any encoding layer, ever.

Best practices if you must deploy a susceptible model

As we’ve established, no single layer is foolproof on its own. But that’s not an argument for doing nothing. It’s a case for ordering the layers by how they fail.

The stack below is ordered by strength, each a part of a stronger approach to defense:

1. Convert the format, and stop having the problem.

Most models don’t need to serialize anything but named arrays of numbers. If that is true of yours, store it in a container that cannot express a call: safetensors for weights, GGUF for quantized inference, and ONNX where you need a compute graph.

The difference between those three matters more than it may seem. Safetensors is structurally safe , as it consists of tensors and a JSON header, with nothing in the format capable of expressing a callable. GGUF and ONNX have no known path, which is more of a claim about the current state of research than about the format's expressive limits. ONNX in particular carries a full operator graph and a history of parser CVEs. When deploying a new model, opt for the format that is safe because of what it can’t say.

Caveat: "No known path" isn’t "proven safe," as conversion isn’t always available. A model distributed only as a pickle has to be loaded once to be converted. Do that in the sandbox from step 5 below, not on a build host.

2. Pin your loaders past the bypasses, and verify the version actually running.

For PyTorch, the most secure approach means version 2.10.0 or later. The weights_only=True default arrived in 2.6 and was widely reported as the fix, but CVE-2026-24747 defeats the restricted unpickler itself and affects every version before 2.10.0. A team that pinned 2.6 in early 2025 on security advice stayed exposed for four further minor versions.

For Keras, run current and refuse legacy .h5 inputs rather than trusting safe_mode on that path. CVE-2025-9905 is the case where it is ignored with no warning. For NumPy, keep allow_pickle=False and treat any code that flips it as a finding.

Caveat: This layer has arguably the worst track record of anything on this list. Every entry above is a fix that was itself later bypassed. Pin it, but do not spend the trust it appears to buy.

3. Scan at rest, in CI, as a gate.

Run modelscan or picklescan on ingest and fail the build rather than the deployment. It’s cheap and it catches the commodity case, which is most of what you will actually encounter.

Caveat: Scanners are blocklist-based, and every CVE evaluated beat a scanner that was working correctly. A clean scan is the only absence of evidence. Never let it be the signal that promotes an artifact to production.

4. Establish provenance.

Track which model came from where. Sign what you build, verify what you consume, and give preference to publishers who do the same.

Caveat: A signature proves origin, not intent. The malicious models found on Hugging Face were authentically uploaded by the accounts that uploaded them. Provenance narrows who you have to trust, but it doesn’t tell you whether that trust is deserved.

5. Load inside a sandbox with an egress boundary.

This is the least glamorous control on this list, but the most reliably effective. Load models in a process with no credentials worth stealing, no write access to anything that persists, and an explicit egress allowlist.

This is the only control here that doesn’t depend on recognizing the attack; it doesn’t have to. It isn’t deciding whether the connection is malicious, only whether it’s permitted.

Caveat: A sandbox bounds the blast radius, but doesn’t prevent execution. The code still runs. You are choosing what it can reach.

6. Always assume breach, and use runtime detection as the backstop.

Everything above is a prediction about a file. Runtime is the only layer that observes what actually happened. Use runtime observability to verify what your code is actually doing and catch malicious activity.

Caveat: In this case, it’s silent on behavior that looks innocuous, and it never sees the load itself, only consequences.

Why that order matters

Read through the first four controls again, and there’s one distinct pattern: Each works by predicting how a file will be interpreted. The converter predicts the format. The pinned loader predicts which code path will run. The scanner predicts what the loader will do with those bytes. The signature predicts that a known publisher produced what they meant to produce.

Every bypass is an attacker finding an interpreter that disagreed with one of those predictions. And there is no reason to expect it to stop, because the asymmetry producing it never closes. A checker has to anticipate every loader. A loader only has to interpret the file one way, and it is the one that actually runs.

The sandbox and the runtime sensor are the two controls on this list that don’t predict. The sandbox doesn’t care what the file is. It constrains what the code can reach once it’s already running. The sensor doesn’t care either. It reports what the process did.

But that doesn’t make them sufficient alone. Runtime never observes the deserialization, the rules that look like they should catch malicious Python assume the malice is visible on the command line, and the correlation layer that ought to be the sophisticated tier never fired once.

Ultimately, the most comprehensive approach is a “both and” strategy. A scanner and a loader that disagree produce a bypass. A scanner and a sensor that disagree produce an alert.

A model file is a program. It shouldn’t move through a pipeline built for data, and you must assume that whatever you use to check the model file has a reading of those bytes that the thing loading them doesn’t share.

Check out the latest in threat research from the Sysdig TRT