Self-Improving Context Layer for AI Agents

In just a few short years, AI has transformed the discipline of data management. Building applications, dashboards, and extracting insights from data is no longer the hard part. While developer tools and autonomous agents advanced at breakneck speed, the day-to-day work of analytics has stayed stubbornly manual.

Why has business analytics been left behind? Part of the reason is that knowledge is siloed, inconsistent, and decentralized. Work on a single dashboard or analysis isn’t added to the sum of business knowledge, and it’s often not reusable. In a CDO Magazine survey, only 18% of organizations had scaled AI across multiple teams, and 61% reported silent failures in their data systems.

At Alation, we’ve approached this problem fundamentally differently. We observed that analysts, builders, and data consumers each bring a unique perspective on what data is important to running a business. Query logs and reports capture activity, but people hold the context. Builders and analysts know which tables matter and which dashboards are dead weight, while business consumers point the way toward the next critical question. These are all signals that can be captured, then transformed into discrete entities within Alation for composable reuse.

Often, the first signal comes from someone using an agent: an analyst correcting a metric, a finance leader explaining an exception or asking for a follow-up, or a user pointing out that the agent followed the wrong instructions. These interactions contain valuable knowledge. If it lives only in the conversation, the next user may have to explain the same thing again. While agent memory work is incredibly helpful for personalization for an individual, Alation specializes in extracting generalizable, corporate knowledge that can be reused for consistent answers across the business.

Alation’s usage-based feedback loops capture that work as proposed updates to governed objects in the catalog. The changes are reviewed, applied, and made available to authorized humans and agents that reference those objects. A correction becomes part of the organization’s shared understanding. Some examples:

• A query for spend value is too high – a user asks to exclude an R&D business unit from spend calculations.

A query for spend value is too high – a user asks to exclude an R&D business unit from spend calculations.

• The wrong column is used for analyzing shipping times; swapping columns gives the correct result

The wrong column is used for analyzing shipping times; swapping columns gives the correct result

• An agent proposes a new dimension to address a new line of business questions, and a user confirms it

An agent proposes a new dimension to address a new line of business questions, and a user confirms it

We tested this mechanism in two benchmarks: one focused on learning business metric definitions, and another on correcting agent configurations.

Two experiments: Teach the context, correct the agent

To demonstrate how this works in practice, we evaluated two distinct layers of agent execution: the context layer (what the agent knows) and the agent configuration (how the agent acts).

Two independent benchmarks test Alation’s ability to learn from chat traces. The first benchmark contains 50 cases that require knowledge of SQL metric definitions. In enterprise data, pointing an agent to the right table is just step one. To produce trustworthy answers, the agent needs the verified calculation, entity relationships, default aggregations, and business filters that give a metric real-world meaning.

Next, we simulated the human-in-the-loop experience using "teaching chats", conversations where an analyst corrects the agent's math (e.g., specifying which metrics define an enterprise phrase like net revenue).

The feedback loop operates in three steps:

• Capture: Alation extracts discrete business logic directly from the chat trace.

Capture: Alation extracts discrete business logic directly from the chat trace.

• Update: Proposed changes are written to the governed context (applied programmatically for this test, though governed by human stewards in production).

Update: Proposed changes are written to the governed context (applied programmatically for this test, though governed by human stewards in production).

• Re-evaluate: The benchmark runs again to verify if the agent resolved the failure.

Re-evaluate: The benchmark runs again to verify if the agent resolved the failure.

While evaluating an agent against feedback it just received might look like "training on the test set," that is precisely the point. The agent's model weights never change; instead, Alation captures tribal knowledge at the point of correction and turns it into compounding, reusable context.

The second benchmark contains eight cases involving agents with prompts and bad tool configurations. The problems lie in the agent configuration and are evidenced in its behavior. Changing the context available to the agent is only part of the problem. The agent itself could be inefficient, broken, or have conflicting instructions. Thus, the feedback loop must identify the responsible layer and direct the correction there. The main purpose here is to align agent behavior with evaluation cases and minimize both latency and errors.

Benchmark

Cases

Baseline accuracy

Accuracy after reviewed updates

SQL metric definitions, with analyst-style teaching chats

50%

84%

Misconfigured agents

100%

Together, the experiments examine both sides of an agent’s understanding: the business context it consumes and the configuration that determines how it behaves.

A trivial agent configuration update, automatically proposed based on chat traces.

A small data product update, from a metric correction from a chat trace.

The important mechanism is what happens to the work between runs. A clarification becomes a proposed change to a durable object. The steward or administrator responsible for the object or data product reviews it, and decides to keep, discard, or revise it. Further, evaluations measure how well the changes solve the issue, and whether they made any unforeseen regressions. In an earlier evaluation on a 20-question sales dataset, a SQL agent went from 60% (12 of 20) to 100% accuracy in two automated improvement iterations, with a human reviewing each change. This was a separate benchmark from the two described here.

Thus, the act of using the agent shows us how to improve it.

The unit of improvement is a governed object

Alation’s Intelligence Operating System, AIOS, brings data, context, agents, governance, and feedback together as a system. It acts as a self-improving context layer and a knowledge-sharing platform for the people and agents working across your business.

AIOS gives learning a place to live.

It provides the building blocks for data agents. A metric definition carries an analyst-approved calculation. Catalog object descriptions differentiate columns and tables, and provide insight on when to use them. A governed document supplies the policy behind an exception and stores standard operating procedures for reference. Data products package all these components together, along with relationships between them, to bring answers to a business’ most pressing questions. Together with an agent (off-the-shelf or purpose-built), they form the backbone of intelligent data apps which learn as they’re used. Each improvement is routed to an object with an appropriate scope, owner, and review process.

This changes the return on the work people already do, and pulls it all under one roof. An analyst who resolves an ambiguity contributes knowledge that other consumers can reference in related reports. A data steward’s review makes that knowledge suitable for shared use, and handles the metadata lifecycle. And all of this behavior is measured through evaluation cases, which ensure agent behavior is consistent.

The value extends beyond the agent that encountered the problem. Other authorized agents, applications, and people consuming the same approved context can benefit from the correction. They have a shared starting point for the next question. As more teams contribute, the organization develops a richer understanding of its business. That understanding remains discoverable and governed, with evidence of what changed and why.

Business knowledge is decentralized

Most business questions, while grounded in data, require knowledge that’s scattered across metadata, documents, and in people’s heads:

• A semantic model defines the calculation.

A semantic model defines the calculation.

• The warehouse contains revenue and cost entries.

The warehouse contains revenue and cost entries.

• A finance policy explains which costs belong in the management view.

A finance policy explains which costs belong in the management view.

• A policy document records the treatment of a newly acquired business and the quarterly reporting structure.

A policy document records the treatment of a newly acquired business and the quarterly reporting structure.

An agent needs all these sources to address a business question, along with the ad-hoc input and feedback provided by the user steering the task. The calculation alone may omit the exception; the policy alone cannot produce the number.

AIOS connects structured data context with governed, unstructured knowledge. Through Governed Collections, documents such as policies, operating procedures, and business glossaries become catalog objects that agents consume by reference. When a source document changes, the agent reads the current version.

Now consider finance clarifying how an acquisition should appear in a profitability analysis. That knowledge can be captured in the relevant governed context, linked to supporting documentation, and reviewed by its owner. Future consumers have access to both the rule and the explanation behind it.

Human review matters because a request for a one-time analysis and a change to an approved business definition have different scopes. The feedback loop prepares the change; the responsible owner determines where, and if, it applies. While agents are great at finding and suggesting issues, ultimately it's the people in a business who are best positioned to judge whether it’s correct. Most organizations draw the same line: in Grant Thornton’s 2026 AI Impact Survey of 950 senior business leaders, only 5% allow agents to execute high-stakes decisions without human review.⁷

Start with the knowledge you already have, then let it grow

Building this system does not require reconstructing your organization’s knowledge by hand. The best place to start is to identify a business process or function, then let Alation aggregate any existing sources or start from scratch:

• Connect your data sources, and Alation’s Metadata Extraction (MDE) establishes the technical inventory.

Connect your data sources, and Alation’s Metadata Extraction (MDE) establishes the technical inventory.

• Query Log Ingestion (QLI) adds evidence of how the data is used: the joins, filters, and usage patterns embedded in real queries.

Query Log Ingestion (QLI) adds evidence of how the data is used: the joins, filters, and usage patterns embedded in real queries.

• BI reports and saved queries contribute established metrics and analyses. [7, 8]

BI reports and saved queries contribute established metrics and analyses. [7, 8]

• Automatic semantic layer creation and mining turn those artifacts into a starting point for a business use case.

Automatic semantic layer creation and mining turn those artifacts into a starting point for a business use case.

• Existing queries reveal candidate relationships and calculations.

Existing queries reveal candidate relationships and calculations.

• Existing semantic models can be brought forward.

Existing semantic models can be brought forward.

• Guided creation helps owners review and refine that context into a governed data product.

Guided creation helps owners review and refine that context into a governed data product.

For a profitability use case, the team begins with a business goal: help leadership understand where margins are changing. Alation brings forward relevant assets and existing analytical knowledge. The owner confirms the intended definitions, scope, and supporting documents, and makes corrections along the way.

Curation Automation helps maintain the metadata standards that make this context reusable. Data Quality monitors the reliability of the underlying inputs. Those capabilities keep descriptions, ownership, and data health connected to the context agents consume.

• The result is a shared foundation that can be used immediately and improved through use. Evaluations then test whether that foundation supports the questions the business needs answered.

The result is a shared foundation that can be used immediately and improved through use. Evaluations then test whether that foundation supports the questions the business needs answered.

• Approved questions and expected outcomes establish a baseline.

Approved questions and expected outcomes establish a baseline.

• As gaps are found, the context is refined and tested again.

As gaps are found, the context is refined and tested again.

• The semantic layer has measurable acceptance criteria for agentic use cases.

The semantic layer has measurable acceptance criteria for agentic use cases.

Agent Studio puts that context to work. Teams can start with native agents and guided creation flows, or build custom agents and behaviors for their business processes. The same governed knowledge supports question answering, research, curation, and actions through connected tools.

Setup and learning form one lifecycle: bring existing knowledge forward, validate it for a use case, deploy it, and let production usage reveal what needs to improve.

Master knowledge centrally and use it where the business works

Shared intelligence needs to travel across the organization’s technology choices. Fundamentally, AIOS is open. Alation can govern context used by native agents, warehouse agents, copilots, and custom applications. Semantic Model Mastering brings existing models into a common governance process, enriches them, and synchronizes approved definitions back to supported warehouses, including Snowflake and Databricks.

Existing customers use Alation to share governed context and support agent-driven business workflows. The architecture should follow the outcome they want to achieve. Consider finance and customer analytics teams working in different warehouses. Their goal is to report consistently on the same business metrics while retaining their existing tools.

We recommend mastering their semantic context in Alation. The teams agree on definitions, validate representative questions, and distribute the approved models to their respective platforms. If an analyst later discovers an exception in the definition of an active customer, the correction returns to the shared object for review and evaluation. The approved update can then reach both environments. [12] For this goal, the context layer is the right integration point: consistency across platforms, with teams retaining their preferred experiences.

Now consider executives who want an ongoing profitability briefing and the ability to ask follow-up questions. The business goal includes the agent’s behavior, the answers it produces, and the way it responds to new requirements. Here, we recommend agent-level integration. A profitability agent built in Agent Studio draws on governed revenue, customer, and cost context. It can reach leaders through Copilot, Slack, Teams, or a custom application or dashboard. Intelligent Feeds provide recurring metrics and analysis, with questions feeding further exploration.

A finance leader clarifies that the management view should include support costs and exclude one-time restructuring charges. The finance owner confirms that definition, and the clarification becomes a reviewed update to the metric and a new evaluation case. When finance corrects a definition, the improvement belongs in shared context. When the agent is configured incorrectly, the improvement belongs in its definition. Evaluations verify the changes, and monitoring connects the next production interaction to the same loop.

Managing these together makes agent-level integration our preferred approach when the goal is an application that improves through use. Shared-context integration remains a strong fit when the primary goal is consistent meaning across established platforms.

Keep learning as the business changes

The loop continues after launch because both the data and the business keep moving.

Schema changes can trigger updated data product drafts for review. A revised policy can change the context supplied by a governed document. An analyst’s correction can update a metric. Repeated unanswered questions can reveal demand for a new source or data product.

Each signal creates an opportunity to improve the appropriate layer and add evidence to the evaluation suite. Existing cases check whether a change preserves the behavior people already rely on. New cases capture the business requirements discovered through use.

The two benchmarks make this concrete: teach the metric context when business meaning is missing; correct the agent configuration when its setup is responsible. In production, the same principle connects diagnosis, review, improvement, and reuse. More usage supplies more evidence. The governed feedback loop turns that evidence into shared knowledge.

That is why work compounds in Alation. The analyst’s explanation, the steward’s review, and the agent’s correction become part of a foundation the next person and the next agent can build on. As the business evolves, AIOS gives the organization a way to develop its intelligence together.

Sources & notes

Every external claim on this page is independently verifiable. The public sources are listed here.

• Only 18% of organizations have scaled AI across multiple teams; 61% report silent failures in their data systems. Report produced by CDO Magazine in partnership with Monte Carlo. — CDO Magazine, “The State of AI Reliability,” 14 April 2026 ↗ https://www.cdomagazine.tech/aiml/the-state-of-ai-reliability-why-trust-is-becoming-the-biggest-barrier-to-scaling-ai

Only 18% of organizations have scaled AI across multiple teams; 61% report silent failures in their data systems. Report produced by CDO Magazine in partnership with Monte Carlo. — CDO Magazine, “The State of AI Reliability,” 14 April 2026 ↗

• Alation Agents trace which context caused an error and propose a fix at the source. A steward approves it once and every agent reading that context improves. Agents are exposed over MCP and REST in tools such as Slack, Teams, and Copilot. — Alation, Agents ↗https://www.alation.com/aios/agents/

Alation Agents trace which context caused an error and propose a fix at the source. A steward approves it once and every agent reading that context improves. Agents are exposed over MCP and REST in tools such as Slack, Teams, and Copilot. — Alation, Agents ↗https://www.alation.com/aios/agents/

• Evaluation cases pair an input with an expected output and set a baseline score. They are re-run after each improvement, and evaluation sets should grow as users ask new questions. In one 20-question SQL evaluation, accuracy rose from 60% (12 of 20) to 100% in two automated iterations, with human review of each change. — Alation, “How to Build SQL Agents That Actually Work: Why Evaluations Matter,” 10 March 2026 ↗https://www.alation.com/blog/ai-agent-evaluations/

Evaluation cases pair an input with an expected output and set a baseline score. They are re-run after each improvement, and evaluation sets should grow as users ask new questions. In one 20-question SQL evaluation, accuracy rose from 60% (12 of 20) to 100% in two automated iterations, with human review of each change. — Alation, “How to Build SQL Agents That Actually Work: Why Evaluations Matter,” 10 March 2026 ↗https://www.alation.com/blog/ai-agent-evaluations/

• AIOS brings agents, context, data, governance, and feedback loops together as one system. It is open, so the feedback loop improves context for agents built in Copilot, Snowflake, Gemini, a homegrown framework, or Agent Studio. The post also describes customer deployments. — Alation, “A New Vision for Alation: Introducing AIOS,” 14 July 2026 ↗https://www.alation.com/blog/introducing-aios-alation-intelligence-operating-system/

AIOS brings agents, context, data, governance, and feedback loops together as one system. It is open, so the feedback loop improves context for agents built in Copilot, Snowflake, Gemini, a homegrown framework, or Agent Studio. The post also describes customer deployments. — Alation, “A New Vision for Alation: Introducing AIOS,” 14 July 2026 ↗https://www.alation.com/blog/introducing-aios-alation-intelligence-operating-system/

• Governed Collections make documents from Confluence, SharePoint, and S3 into governed catalog objects that agents consume by reference, reading the current version when a document changes. Intelligent Feeds deliver recurring governed metrics, visualizations, and chat. As of the 17 September 2026 release date, both are listed as early access. — Alation press release, 17 September 2026 ↗https://www.alation.com/news-and-press/alation-launches-aios-tm-expansion-with-six-products-that-accelerate-ai-transformation/

Governed Collections make documents from Confluence, SharePoint, and S3 into governed catalog objects that agents consume by reference, reading the current version when a document changes. Intelligent Feeds deliver recurring governed metrics, visualizations, and chat. As of the 17 September 2026 release date, both are listed as early access. — Alation press release, 17 September 2026 ↗https://www.alation.com/news-and-press/alation-launches-aios-tm-expansion-with-six-products-that-accelerate-ai-transformation/

• Agent Studio supports native agents with templates and a guided no-code flow, or custom agents, with tools and MCP/REST deployment. — Alation, Agent Studio ↗https://www.alation.com/product/agent-studio/

Agent Studio supports native agents with templates and a guided no-code flow, or custom agents, with tools and MCP/REST deployment. — Alation, Agent Studio ↗https://www.alation.com/product/agent-studio/

• Seventy-eight percent of surveyed executives lack strong confidence they could pass an independent AI governance audit within 90 days. Survey of 950 business leaders, fieldwork 23 February–18 March 2026. — Grant Thornton, 2026 AI Impact Survey Report ↗ https://www.grantthornton.com/services/advisory-services/artificial-intelligence/2026-ai-impact-survey

Seventy-eight percent of surveyed executives lack strong confidence they could pass an independent AI governance audit within 90 days. Survey of 950 business leaders, fieldwork 23 February–18 March 2026. — Grant Thornton, 2026 AI Impact Survey Report ↗

• Semantic Model Mastering ingests semantic models from platforms including Snowflake, Databricks, Power BI, Looker, and Cube, then governs them as data products. — Alation, Semantic Model Mastering ↗https://www.alation.com/product/semantic-model-mastering/

Semantic Model Mastering ingests semantic models from platforms including Snowflake, Databricks, Power BI, Looker, and Cube, then governs them as data products. — Alation, Semantic Model Mastering ↗https://www.alation.com/product/semantic-model-mastering/

• The Data Products Builder recommends assets, generates documentation, and surfaces trust signals so owners can review, refine, and approve before publishing. — Alation, Data Products Builder ↗https://www.alation.com/product/data-products-builder-agent/

The Data Products Builder recommends assets, generates documentation, and surfaces trust signals so owners can review, refine, and approve before publishing. — Alation, Data Products Builder ↗https://www.alation.com/product/data-products-builder-agent/

• Curation Automation lets teams declare metadata standards in plain language and has agents enforce them across the catalog, rerunning as data changes. — Alation, Curation Automation ↗https://www.alation.com/aios/curation-automation/