Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/Prove it does a learned rule actually change what your ai assistant writes secur
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

Prove It: Does a Learned Rule Actually Change What Your AI Assistant Writes? – Security Loop Series, Part 4

Source: Cisco Blogs•November 16, 2026 at 03:46 PM

Prove It: Does a Learned Rule Actually Change What Your AI Assistant Writes? – Security Loop Series, Part 4

Same requests, identical workspaces, a blind judge. Without learned security rules the agent wrote an insecure trust check 8 of 8 times. With them, 0 or 1 of 4.

Part 3 of this series ended with a promise I hadn’t kept yet.

I showed you how Foundry turns a real finding into a Project CodeGuard rule, and how that rule lands in the developer’s IDE as a single file. “The prevention is the file,” I wrote. Nice sentence. But it’s also a claim, and in this series we don’t accept claims without evidence.

So this post is the experiment. Same feature requests, a real AI coding agent, workspaces that are identical except for the rules, and a blind judge. The short version: without the rules Foundry learned from real code, the agent wrote an insecure trust check in 8 of 8 attempts. With them, in 0 or 1 of 4. The generic rules made no measurable difference.

The experiment

I wanted a simple test, one that could prove me wrong.

Twelve ordinary feature requests. Each one is the kind of thing a developer asks an AI assistant every day, with no security hints at all. In each one, the assistant has to modify a small existing service:

  • Six of them are about the two weakness classes Foundry learned in Parts 2 and 3. Three are about trusting where a request came from: a localhost-only admin endpoint, per-IP rate limiting behind an AWS load balancer, and an office-network allowlist behind Cloudflare. The other three are about parsing IDs out of strings: a run ID from an artifact URI, a tenant ID from an object key, and a user ID from an API key header.
  • The other six are about classics that Project CodeGuard’s core rules already cover: command injection, SQL injection and path traversal.

Three arms, one difference. Every request runs in a fresh workspace. The only difference between workspaces is what sits in “.cursor/rules/”:

Each request runs twice in each arm, in random order. That’s 72 sessions with a Cursor agent running Claude Sonnet 4.6, the same model behind every number in this series.

A blind judge you’ve already met. All 72 outputs go into folders with random names. A script removes every comment and docstring, because agents love to explain themselves (something like “per the CodeGuard rule…”), and that would tell the judge which arm it’s looking at. Then the same Foundry pipeline from Part 2 evaluates all of them in one run, and its evidence gate decides what’s confirmed. The forge grades the guard.

First, prove the guardrail is in the room

Before you measure whether a rule works, you need to know whether it arrived. So every arm, including the control, gets one harmless extra rule: start every Python file you edit with the comment “# house-style: v2”. If that line shows up in the output, the rules reached the agent. If it doesn’t, the rest of the experiment means nothing. I removed the line before judging.

I call this the canary. Making it show up every time needed three things, and the harness now does all three:

  • Rules have to live in “.mdc” files with a leading --- block. Cursor reads project rules ONLY in that exact format.
  • Agents started from code don’t read project rules unless you tell them to. The Cursor SDK has a setting for which rule sources to load, and you have to turn project rules on.
  • Scoped rules attach to the files the agent touches. A rule for “**/*.py” reached the agent when it edited an existing Python file, but NOT when it created a new one in an empty folder.

With all three in place, the canary showed up in 72 of 72 sessions. Every rule in every arm reached the agent in every trial.

If you only remember one thing from this post, make it this one: test that the guardrail is in the room before you test whether it works. A rule file that gets ignored without any warning is worse than no rule, because everyone thinks they’re protected. The canary is just one line, and it’s now part of the experiment harness.

My bet was half wrong

Part 3 opened with a bet: ask an assistant for a localhost-only admin endpoint, and it will trust “X-Forwarded-For”, just like the Langflow code Foundry caught.

It didn’t. In all six sessions, across all three arms, the agent checked the real address of the connection and ignored the headers. The model already knows that one.

But the weakness didn’t go away. It moved to the cousins of that bug: requests where “who is really calling?” matters just as much, but the bug is less famous.

Rate limiting behind a load balancer. Two control sessions and two CodeGuard sessions took the first address in “X-Forwarded-For”. The client writes that value. So an attacker can send a different fake address with every request, and never hit the limit.

An office allowlist behind Cloudflare. All four sessions in those two arms wrote some version of this:

It looks fine, and many tutorials do it this way. But anyone who can reach the server without going through Cloudflare can send that header with an office address, and get access to protected info. Two of the sessions even wrote in a docstring that the server should be behind a firewall… and then wrote code that trusts the header no matter what.

The result

Here’s what the blind judge confirmed:

With the learned rules, every one of those proxy sessions checked who was really connecting before reading any forwarding header. On the Cloudflare task, both sessions first checked that the connection really came from Cloudflare’s published address ranges:

Notice something: the rule Foundry synthesized never mentions Cloudflare. It was learned from a Langflow function that trusted “X-Forwarded-For” for a localhost check. The agent applied the same principle to a different header, a different provider and a different feature. That’s the difference between a rule and a regex.

I audited the judge too

A blind judge is still a model, so I checked every verdict against the code. Three of them need a human decision:

  • Taking the last “X-Forwarded-For” entry behind a load balancer (one control session, one CodeGuard session). That’s correct if the server can only be reached through the load balancer, and spoofable if not. The judge flagged both as vulnerable.
  • One CodeGuard + Foundry session used a ready-made proxy helper from its web framework. The helper trusts one proxy hop, so it does exactly the same as taking the last entry. But the logic is hidden inside the library, so the judge didn’t flag it. If the other two count, this one should count too.
  • One control finding on URI parsing looks like a false positive: the code did check the index.

With the strict reading, that gives control 4, CodeGuard 4 and CodeGuard + Foundry 1 out of 24. With the more relaxed reading, 3, 3 and 0. On the two tasks where unprotected agents actually failed, that’s 8 of 8 sessions vulnerable without the learned rules, and 1 of 4 with them (strict), or 6 of 8 and 0 of 4 (relaxed). Both differences are statistically significant, even with so few sessions. Across all twelve tasks the trend is the same, but most tasks never failed in any arm, so those numbers alone are not significant. I’d call this a well-controlled pilot, not a benchmark.

What the core rules told me

The CodeGuard core rules showed no measurable effect, and that’s not their fault. In every arm, the agent wrote zero SQL injections, zero command injections and zero path traversals. Parameterized queries, safe subprocess calls and checked file paths are now so common in the code models learn from that a modern assistant writes them without being asked. For these classes there was nothing left for a rule to fix.

This changes how I see the value of the flywheel. Generic rules protect you from the mistakes everyone already knows about, and models are learning those fast. The mistakes that still get through are the subtle ones that depend on context: the ones your own code taught you. That’s exactly what Foundry’s rule-gap log captures, and nobody can give you that in a generic rule set.

How much it costs

The 72 agent sessions used about 39M tokens. There was no real difference between arms, so adding 25 rules didn’t make sessions more expensive. Judging all 72 outputs took about 600k tokens. That’s the price of measuring prevention once. Running the rules in your developers’ editors afterwards adds nothing you’d notice on a bill.

How customers and partners can use this

Everything is in my Foundry repo: the tasks, the experiment harness, all 72 generated files, the blind judge’s run and the results. The full design and audit are in “docs/PREVENTION.md”, and you can reproduce it with one command:

There’s also an offline mock mode, so you can test the setup for free.

But more important, you can use the same approach with your own rules:

  • Check delivery first. Whatever assistant and rule format your developers use, add a harmless canary rule and check that it shows up. It takes five minutes, and it’s the only way to know your rules are actually loaded.
  • Turn your incidents into test tasks. Every confirmed finding from a Foundry run, a pentest or a bug bounty can become a feature request where the agent could make the same mistake. Put your own rules in the experiment’s rules folder, add the tasks, and measure.
  • Back your rule reviews with numbers. Part 3 said every synthesized rule should go through a pull request. Now that pull request can include a number: how often the agent wrote the weakness with and without the rule.
  • For partners, “we deployed secure-coding rules” becomes something a customer can verify: rules learned from their own code, proven to change what their assistants write, with the evidence attached.

What I am not claiming

One model, one agent, small numbers. This is Claude Sonnet 4.6 through Cursor, with two sessions per task and arm. Other models and other assistants may behave differently, and CodeGuard’s translators make it easy to repeat the test with them.

I wrote the tasks, and I also ran the work that found the original bugs. The proxy tasks were designed to test the pattern Foundry found in Langflow. So this tests whether the learned rule prevents that pattern, not whether synthesized rules are good in general. The Cloudflare result is a good sign, but not proof.

The judge and the rules come from the same model family. Blinding hides the arm from the judge, and I checked every verdict by hand, but a fully independent judge would be better.

The parsing rule had nothing to prove here. The agent parsed strings safely in every arm, so we still don’t know how much the second learned rule helps.

Back to Tuesday

So now the guard has a track record. Antares tells you where to look, Foundry tells you what’s real and learns from it, and the rules it learns really change what an AI assistant writes. And now we have the numbers to show it.

There’s one thing left. So far, every piece of this series was built and tested on its own. But real security work doesn’t arrive in pieces. It arrives on a Tuesday morning, as an advisory about one weakness class, in a repo you weren’t planning to look at this week.

In Part 5, the season finale, the star, the forge and the guard work together as one loop for the first time. And we finally answer that Tuesday advisory.

Everything in this post is reproducible from : the prevention command, the twelve tasks in “experiments/prevention”, every generated file and the blind judge’s run in “out-prevention/”, and the full design, audit and limitations in “docs/PREVENTION.md”. Project CodeGuard lives at project-codeguard.org.

← All articles

What this article says

More in Hardware & Electronics

All →
PX-CSI Support for Talos Linux Coming in December
Pure Storage

PX-CSI Support for Talos Linux Coming in December

A World’s Best Workplace: Cisco!
Cisco

A World’s Best Workplace: Cisco!

CUPRA et Sennheiser dévoilent la New Tribe Edition Sennheiser™, une édition spéciale où le design rencontre le son immersive
Sennheiser

CUPRA et Sennheiser dévoilent la New Tribe Edition Sennheiser™, une édition spéciale où le design rencontre le son immersive

Synopsys at Supercomputing 2026
Synopsys

Synopsys at Supercomputing 2026

Harnessing AI and HPC to Push the Frontiers of Human Knowledge
Lenovo

Harnessing AI and HPC to Push the Frontiers of Human Knowledge

Marvell and Penguin Solutions Collaborate to Address Data Center AI Memory and Accelerate Inference at Scale
Marvell

Marvell and Penguin Solutions Collaborate to Address Data Center AI Memory and Accelerate Inference at Scale

More from Cisco

A World’s Best Workplace: Cisco!
Cisco

A World’s Best Workplace: Cisco!

BYOH: Bring Your Own Harness with Cisco IQ, for partners
Cisco

BYOH: Bring Your Own Harness with Cisco IQ, for partners

The Digital Networks Act: Building the connectivity infrastructure Europe needs for AI
Cisco

The Digital Networks Act: Building the connectivity infrastructure Europe needs for AI

Skills-to-Jobs: A Strategic Career Ecosystem to Close the Gap
Cisco

Skills-to-Jobs: A Strategic Career Ecosystem to Close the Gap

Cisco stock jumps 5% on earnings and revenue beatПресса
Cisco

Cisco stock jumps 5% on earnings and revenue beat

Rebuilding PubHub: A Cautionary Tale of AI-Driven Requirements, Part 4
Cisco

Rebuilding PubHub: A Cautionary Tale of AI-Driven Requirements, Part 4