A skill can fail without an error
In our own builds on commercetools for Builders, the two skills whose names matched the work were never loaded. Two unexpected ones did the heavy lifting instead. Nothing crashed, and the answers looked fine. (We cover this inBuilding with AI: Best practices for guardrails.)
That raised two questions, one for each side of the table.
- For you, the builder: How do you ensure the right skill is used in your workspace?
- For us, the people who write skills: Are our descriptions clear enough to be picked, and do our instructions actually improve the answer?
The guardrails post answers the first question. This post answers the second.
Build with AI
commercetools for Builders
Describe what you need in plain language and let AI skills handle the commerce foundations.
Why skills need tests
An AI coding agent does two things with a skill. First, it picks one based on a short description of each skill’s purpose. Then it follows the skill’s instructions.
We ship skills for agents we don’t run, so both steps occur outside our visibility. Linters and link checks can tell us a skill is well-formed. They can’t tell us whether an agent picks it or follows it.
That leaves two quiet failures:
- The wrong skill gets picked. Rewording a description can steal requests from a neighboring skill, or stop a skill from being picked at all. The more skills exist, the more often this happens.
- The right skill gives a bad answer. The agent invents a custom connector when a certified one already exists. Or it picks a setting that can’t be undone, without asking first.
Both cost money. A needless custom connector is work someone pays for later. A silently chosen setting is a risk the customer never agreed to.
Code gets tested automatically on every change, so a small edit can’t quietly break something. We wanted the same for our skills, so we built two automated checks.
Check #1: Is the right skill picked?
The question: Are our skill descriptions clear enough that the right skill gets chosen?
How it works:
- The test agent sees only each skill’s header: Its name, its description and a few other fields. It gets no project files, no extra tools and no other skills.
- A script, not another AI model, holds the expected answers and marks each pick. The agent only ever sees the prompt, so the answer key can't leak into a pick.
Why we keep it so bare: With everything else removed, a wrong pick can only come from our own wording. That means every recurring failure has a clear fix.
What a test case can say:
- “This request should pick these skills.” Connecting a payment provider like Stripe requires two of our skills to work together, so one case expects both.
- “This request must not pick that skill.” Naming forbidden skills tests overlaps directly between neighboring skills.
What it costs: Very little. The check runs only when a skill's header changes, and a full run is far cheaper than Check #2. That's why it runs first.
Check #2: Is the answer what a reviewer would expect?
The question: Once the skill is picked, does it improve the answer?
How it works:
1. One agent answers each test case with the skill loaded.
2. A second agent marks every expectation as pass or fail.
3. A case passes when it meets 80% of its expectations.
Why we run misses again: Results wobble. The same case scored anywhere from 50% to 100% across identical runs, so a single run tells you very little. Now, a case that misses anything is rerun twice, and it fails only if two of its three runs fail.
Does the skill earn its keep? We compare each case against a baseline, the same test run without the skill. On our catalog migration skill, the average pass rate rose from 38% to 87%. One case scored 100% both with and without the skill, so the skill only added cost in that case. That’s useful to know, because it lets us ask whether a skill is worth running and get a real answer.
How to read the gain: It shows how closely answers follow the skill’s rules, not how good an answer is in general. To keep the test fair, the marking agent never sees the skill's text.
One bias to watch: If the marking agent is the same AI model as the agents it grades, it may favor answers written in its own style. Strict, specific marking criteria reduce this, and so does using a different model.
Both checks are advisory for now. They don’t block anyone’s work until they have a stable track record.
What our checks can’t tell you
Check #1 is a controlled test, so a failure tells us exactly what to fix. A pass is weaker evidence.
Real workspaces are crowded with project notes, company tools and other vendors’ skills. In that crowd, a coding tool may not show our skill descriptions to the agent at all. Better wording can’t fix that, because the tool makes that choice. Your own skills also compete with ours, and we don't control those either.
We don’t test a crowded workspace on purpose. Every workspace is cluttered differently, so when a test there fails, we can’t tell which part caused it, and we can’t fix what we can’t explain.
The takeaway: We make sure our descriptions are clear and distinct. What happens in your workspace is where your guardrails come in.
Where your guardrails pick up
Our checks and the five guardrails from ourbest-practices post cover opposite halves of the same problem.
The pattern is the same on both sides. Don’t ask the model to grade itself, and don’t trust a success message. Check the result with something independent.
Before you rely on a skill
What we do:
1. Test that the right skill gets picked whenever a skill’s description changes.
2. Test that the answer is better with the skill than without it.
3. Re-run misses before calling a failure, and tell you what each check leaves open.
What we suggest you do:
1. Map each task to the skill that should handle it before you start building.
2. Verify results through a path independent of the one that wrote them.
3. Keep a corrections file for each skill.
It takes a little longer to set up. But each step is cheaper to do first than to add after something has quietly gone wrong.
Explore the commercetools for Builders documentation — and start building in days. Included in our 60-day free trial.
Build with AI
commercetools for Builders
Describe what you need in plain language and let AI skills handle the commerce foundations.





![8 Sales Promotion Types to Boost Holiday Revenue [2026]](/_next/image?url=%2Fapi%2Fuploads%2Farticles%2F345e19e4d0b11fb82dc63f31180bccaa.webp&w=1920&q=75)






