Build the builder: scaling AI skill creation when everyone is a maker

One unintuitive way that software is changing is that everyone is a builder now. I love that idea. It means software engineers need fewer sessions to get the requirements from the business side right because the business users know best what they need from a workflow. It means that we can encode business logic in new unexpected ways, and, at Miro, we started leveraging this way of operating in the Claude ecosystem quite early.

However, as with most things AI, all that glitters is not gold. Being swamped with PoCs that you need to understand deeply to get them into a better state and enable security becomes a grinding and, above all, extremely cognitively demanding task. Engineers involved in trying to work with business teams are faced with overlapping skill content, skill sprawl, and having to manage a team with often varying skill levels in creating agents that do not adhere to the instructions and that create (in)accurate output.

But let’s start at the beginning. Conceptually agents in the Claude ecosystem can be defined as simply as Markdown syntax in a SKILL.md file. Some business users have become increasingly adept at defining their own workflows through that medium. The work of engineers in this new world lives at the tradeoff of putting some more upfront work into creating a safe way for skill authoring for non-technical people to contribute, versus sitting down with experts to understand the flow and build it out themselves. I have come to find that using the simplicity of contribution before moving a skill into something more maintainable like an agent SDK implementation can really help to separate the noise from the signal. That is what we will explore in this article.

Adopting a multiplier mindset: working on another altitude

I believe that what has become important in this age is to zoom out whenever there is a situation that scales with the number of users. Ultimately, what that means for you as an engineer is: Get ready to become a multiplier.

Think about what actually needs to be achieved. For example, think about how to automate as many repeated steps as possible to alleviate that pain an engineer feels when the number of PoCs exceeds their own ability to track quality implementation. A good indicator is the number of times you need to do a job. If that job increases linearly with the number of users, you are essentially going to be the bottleneck sooner or later unless the number of engineers also scales, which is rarely the case especially at this point in time.

For me, that thinking took the shape of creating a lightweight and simple process in the form of skill authoring workflows. It’s the way of marrying the best of both worlds — deep domain knowledge and technical best practices — and it lets you move fast to define PoCs for those agents.

Think of it as putting guardrails around business users’ implementation of the logic. Making sure that skills are not created and never changed again with respect to their quality definitions, but iteratively improved before committing to a production version. Tracking this progress in a pilot so that I can be sure that the effort of moving those workflows to production using a more production-ready framework like the Claude Agent SDK actually makes sense. That means that continuous effort is put into those workflows and that they have shown improvements to the process outcomes that can be specified by the business users concretely.

We are talking about a process that is, on the one hand, encoded in the way skills are created, so that the ways a skill can be misspecified are caught by the system at the time of creation. On the other hand, it is a lightweight tracking mechanism that helps show progress as skills move from alpha (#badversion), to beta (improved version ready for use by other teammates), to production (repeated usage has shown a concrete way this version is bringing value over the current way of working). That path also makes business users responsible for articulating why their version is better than a potentially pre-existing production version, as is sometimes the case in larger organizations. And it relieves the engineer of having to do that when the business logic came from someone with much more domain expertise.

Building for everyone might work for no one

In our team, after a first try using the much-promoted /skill-creator command, it became clear that there were a few common failure modes that were not handled well. A reminder that while foundation model companies are good at their work, they may not always understand your company’s business context; also, building for a general population is hard, so engineering with specificity to your own environment still matters (and, most likely, always will). As skills got longer, steps are skipped as state was not maintained, probably due to a lack of context management and state tracking embedded in this skill creation process. Since there was no notion of context engineering, input data could quickly become noisy defeating the purpose of the automation. Because context retrieval via MCP was just one step among many steps in the process, some context was simply non-existent, at best; at worst, completely missing context could lead to full hallucinations without any awareness of it, making debugging hard for engineers and errors invisible to business users.

This experience meant I recently spent some time thinking about how to enable skill authoring for non-technical teams with Claude Code. That is the new tension that engineers often seem to carry. It does not really matter whether it is from engineer to engineer or from business user to engineer. The pain point is the same. How do we work safely with systems that are the new mode of producing content? The crucial switch in the mindset is one that orients towards platform thinking, and that is an organizational shift to think about when engaging in any system that shifts the bottlenecks.

I, as an engineer, try to build the scaffolding that makes safe and secure skill creation the default for our team, encoding best practices while leaving the domain specifications in the hands of the business. What I learned is that, reasoning by analogy, software engineering standards can often be translated to be useful in this specific context. Skills can be seen as analogous to software repositories, where many people who never had an engineering background suddenly contribute. There are many technologies and practices developed in software engineering that are usually at least conceptually useful for business teams in this new world of AI agents.

Drawing on proven engineering practices

When everyone builds, organizations drown in disjoint workflows of poor quality. It is an unspoken fact of enabling whole teams with Claude with no guardrails. Enabling non-technical teams in a more structured way, has several important elements to it that often borrow from practices that software teams employ naturally.

• Scaffolding skill creation by encoding what Anthropic itself promoted as best practices in the skill authoring process itself, so that SMEs can focus on the business logic. It aims to make skill quality less dependent on the SME’s individual background knowledge and can do double duty by simultaneously encoding the team’s way of working.
• Skill refactoring as an agentic engineering practice encoded as workflows. This helps for use cases like managing skill sprawl, where the skill repository increasingly contains many functionalities for the same job to be done. Additionally, a mechanism to import skills that business users have created, which do not follow the best practices into the correct shape.
• Harness engineering, for example, by creating reusable security utilities that handle recurring problems like path validation. Guardrails that help make sure no company data is inadvertently leaked and increasingly leveraging the concept of code-as-harness so that a translation into a production system moving away from the Claude ecosystem if necessary could be simplified.

Framed one way, the first two of these are agentic engineering like authoring-time workflows that set the constraints of how a skill is allowed to exist. The third is mostly related to validation and checking, including a quality loop and guardrails that check and catch what slips through during skill execution.

Scaffolding: making the paved road easy

The first element that engineers can leverage is scaffolding. Essentially, another skill akin to a specialized coding agent that is used to make the skill creation process a guided process — encoding important elements of the team’s way of working and best practices in the skill rather than letting everyone in the team freestyle.

Critical defaults are encoded in the skill, such that not every team member makes individual decisions on common logic (e.g. where to store the resulting output, or what library to use to store the content). That freedom commonly leads to bringing in multiple, often-unnecessary dependencies.

Let’s say the team creates artifacts relating to various customers that should be stored in customer-specific folders. The skill creation process itself can encode a human-in-the-loop step that asks the user for the customer name, so that the storage capability can later upload the file into the correct customer folder instead of having to infer it from some content in the output. This minimizes the amount of error that the agent can introduce at each atomic step. Additionally, each skill will contain a certain file and folder structure to store references, outputs, and configurations, creating a structured representation of where things live, rather than burying this evidence in the skill’s prose. The common structure makes it easier to find. For example, scripts used to write files to output locations. Clarify on where to search for any configurations across various skills, making the engineers’ cleanup work more efficient and with time also makes work on reference drift easier for the SMEs.

Context management is an essential element in making sure skill output is accurate. On the one hand, skill scaffolding can encode a step that forks the context via “context: fork” in the skill frontmatter to provide an independent context window that does not inherit from the current conversation. On the other hand, a good skill authoring flow will make sure that subagents are called for context-independent steps. This can be done by implementing a mechanism to determine context-independent steps. Writing intermediate context to a file, making intermediate results observable. Effectively also offloading the working memory of the agent, reduces the probability of losing information through compression. Compression can happen on the instruction or the context itself, causing low quality results. A precondition for being able to debug the agents more easily is, at a minimum, accessible intermediate results.

Additionally, a planning algorithm, drawing on an experimental, independently built implementation of hierarchical planning and state tracking, helps make sure that each step in the skill is fully executed. Without it, large skills often suffer from skipped steps. The hierarchical planning approach externalizes the state into a file that exists independently of the conversation and thus can’t be compressed, as it does not compete with everything else in the context.

The beauty is that if you keep each of these steps modular (e.g. file writing, state tracking, etc.), then refactoring is much less tedious and error-prone, as well as more token-efficient, if something in the process changes. Change happens only in that particular skill file and translates to all calling skills, applying the software engineering principle of single responsibility. For planning, you can then easily swap out the planning algorithm, testing whether alternative versions serve your use case better, and leverage advancements.

The quality loop

Additionally, a quality loop is parametrized at the end of the skill creation, where either input-output examples could be provided to the skill creation workflow to define quality for output. Or the skill definition itself serves as the basis for quality criteria extraction. An extraction capability extracts quality contracts in the form of configurations that encode criteria with respect to dimensions like data freshness, signal quality, formatting, and groundedness.

Here, the idea is that an agent loop will, at execution time, use the extracted definitions to check for issues while the prerequisite information is still in working memory. The feedback loop can trigger, retrieving missing information, fixing signal, or formatting errors in the output. This is conceptually borrowed from “loop engineering,” where the goal in this case is defined by the quality specifications previously extracted.

As an example, a workflow where there are numeric expectations on the result tables generated went from missing rows and columns to being complete after the quality criteria were defined and applied to the final output by the quality loop capability. This also means that a well-specified loop reduces tokens spent significantly, as another outcome would have led to a rerun of the skill when a naive user detected the issue.

The quality definition can be arbitrarily complete or incomplete, and this is a major lever for getting the output to adhere to the desired standards. During skill authoring, the author will be asked to review and edit the quality criteria. Similar to the idea of evaluations being the new PRD in product development, where PMs are now creating the evals, here SMEs are in charge of defining the quality criteria as per their needs.

Refactoring: keep the ecosystem maintainable

Any engineer knows that skills can quickly become a nightmare. There is skill sprawl: you would have to read and keep track of all skills contributed by every user, keeping those in your head to find redundancies that are often not trivial because of subtle differences. Additionally, files that should rather be kept as references are often inline in the skill, polluting the context window with information that could instead be loaded into working memory on demand by a particular subagent, rather than by default through the skill body.

It is also a technical convention for skill definitions to stay under 500 lines. Otherwise, long-context rot becomes a more likely risk, so the goal of any engineer should be to enforce the line limit. I did this both via a pre-commit hook borrowing from software engineering best practices, and by supporting the SME in adhering to the standard, offering them proposals for how to do so during refactoring via the refactoring workflow. I believe this is one of the perfect agent workflows, as it involves complex decisions that are not linearly representable via a simple processing pipeline, as the workflows of many other agents could be.

This is where the concept of refactoring from engineering has inspired me to inform the last step in the skill authoring process. It encodes essentials of governing the skill library with respect to abstractions and deduplication. It goes through the current version of the authored skill, checking for the following issues and reducing the number of lines, while keeping the human in the loop in control of what refactoring steps would be executed:

• Inline content that could be offloaded to reference files is proposed and moved to a specific location, then referenced in the skill body.
• A capability-sweep skill detects existing “molecular skills,” which are small, single-purpose, reusable skills. These capabilities can replace an in-line step with a parametrized subagent call, reducing duplicated logic when a configured version could do the same. For example, think of when MCPs are queried for context throughout multiple skills doing various research tasks; a configured version of a context-retrieval capability is a concise way to both reduce line count and simplify context management, as well as make it easier to configure an improved query or prompt for context retrieval.

The important part here is that capability-sweep addresses the problem of skill sprawl in two ways. Firstly, if called in mode “extraction-candidates,” it looks through the registry of current skills and finds candidates to extract and manage. Secondly, any extracted capabilities will be proposed in the refactor as described above when called with mode `existing`.

In practice, both the inline-content extraction and the capability-sweep, in combination, were able to bring down the line count of each skill to within the required limit, extracting inline references and reusing extracted capabilities by calling them as a subagent flow instead. Of course, this depends on the initial skill size, but for skills ranging from around 600 to 780 lines, the mechanism brought it down to the required limit of 500 lines.

And capabilities extracted by the capability-sweep are reused; e.g. context retrieval is parametrized and reused 10 times in the repository. This helps me to maintain the repository by reducing the complexity and giving me and the business users control over the retrieval and extraction prompts through the configuration. It shows the power of automating such a step instead of leaving it manual for the engineer to resolve, which would require quite some investment.

Harness engineering: keep users inside safe boundaries

A harness, in the sense used here, is the code and infrastructure wrapped around the model, the part that decides what it’s allowed to do, checks its output, and determines what happens when it gets something wrong. Agents essentially wrap the model as a function and determine what is allowed or disallowed via the harness.

A lot can go wrong with agents. They find creative ways of circumventing guardrails, but similarly, we can’t expect business users to be aware of security and compliance issues that get introduced, which often depend on complex logic buried within the skill definition.

Some common patterns that make this work more difficult when working with business users who are in charge of skill definitions include:

• Introducing architectural components that are not necessary because they are covered by other already implemented components
• Being unaware of GDPR or security implications of processing data of internal systems
• Implementing in a variety of languages, which makes the implementation of security-relevant utilities dispersed and repetitive, blowing up the complexity of the code base

Some parts of this can again be solved by a CLAUDE.md file, for example, that gets loaded during the skill authoring process, enforcing quality standards on the code that gets implemented during skill creation. Additionally, enforcing standards helps fix recurring issues in the most concise way; for one, limiting the language reduces the complexity of implementing necessary path validation or safeguards for SQL injection to one rather than multiple languages. These can then be implemented in reusable and tested security utilities.

This goes for any of the scripts and reusable functionalities implemented in code. One of the single biggest benefits of using CI/CD when implementing with business users is that no unintended changes can be made to the reusable utils when business users contribute, barring changes to the pre-commit hooks themselves, which can be seen in version control systems immediately during a review.

Apart from the already-mentioned scaffolding-related harness components, such as planning and state tracking, I believe the best way of looking at the harness is by trying to implement as much as possible in deterministic code logic, as described in this recent paper. For example, wherever a skill does something verifiable, like computing the value of a guardrail, a reusable guardrail implemented in Python should return a boolean of whether it passed, and logic should be based on the return value of the script. If it is false, a human-in-the-loop step can be used to inspect and, if possible, rewrite the content in a way that lets the guardrail pass.

The concepts discussed in this article are excellent ways of getting the proof of concept for agents up and running. The code-as-harness concept additionally helps speed things up when needing to move into productionization efforts, when observability and stepwise execution become more crucial to debugging and improving the agents’ behavior.

Cutting through the noise: the omnipresent AI hype problem

The last issue I want to discuss is one of change. Some teams like ours may take a decision to work on the edge of what is technologically possible, introducing new tools and MCP servers on an experimental basis. There is a certain amount of movement from each profile necessary to become a participant in the new way of working.

Everyone is a builder now. It means that SMEs become acquainted with code and IDEs in a way that was not possible before. It requires continual learning while doing the job, which can be stressful for some. An engineer’s job is also expanding into how to, for example, address the fears associated with a terminal. From experience, this can only go so far by engineers making sure to make the entry barrier as low as possible. It is on everyone individually in this work environment to take steps toward becoming T-shaped, and that comes with some initial growing pain. I see those who take the plunge benefit a lot from it, and it also shapes their thinking about what AI can do for their daily jobs vs. what better stays in the hands of people as manual work.

For me this is where the magic lies, because this means that suddenly SMEs can start building a mental model and understanding of the limits of the technology hands-on, without the fog that gets created by the narrative of AI doomers and AI optimists.