What to Do About Dark Data Before Your Next AI Pilot

The pressure to ship an AI pilot arrives before the picture of your data does. A team has a use case, a model, and a deadline, and the fastest path is to point the model at the largest pile of relevant-looking data and see what comes back.

That pile of data is usually dark. In Omdia’s latest research on enterprise data readiness, about half the organizations surveyed report that dark data makes up at least half of everything they store. Somewhere in it is the material the pilot needs, and somewhere in it is the material that could get the pilot shut down by legal, security, or finance.

The rule for telling them apart is short: Understand your data before you prepare it for AI. In practice, it’s a four-step readiness check that belongs in front of every pilot approval: discover, classify, contextualize, and—finally—vectorize.

What dark data actually is and why ‘feed it all to the model’ fails

Dark data is the information organizations collect, process, and store during normal business but rarely use for anything else. Most of it is unstructured data (old emails, recordings, documents, chat logs, and exports nobody deleted). A good share of it is redundant, obsolete, or trivial (a subcategory called “ROT”).

Some of it, though, is historical knowledge the business would pay to have back, and in the AI era, it’s training and retrieval material you already own.

Figure 1: More than half of organizations surveyed by Omdia reported that at least 51% of their stored data is “dark.”

The problem is that without proper context, the ROT and the usable records can look identical. That’s why 76% of the IT leaders Omdia surveyed call dark data a significant business risk they’re actively trying to manage, and 68% say data management is the hardest part of an enterprise-grade AI implementation.

And because sorting is hard, the common shortcut is to just skip it, hand the whole pile to the model, and let retrieval decide what’s relevant. That shortcut fails in four predictable ways:

• Compute cost rises with every irrelevant document in the corpus.
• Response time slows as the index grows.
• Security exposure multiplies with each sensitive file that gets embedded because a document full of personal or financial data becomes retrievable by anyone who can reach the assistant.
• The model produces confident, well-worded errors when it’s grounded in a superseded draft instead of the current record.

AI-ready data is the opposite of “all the data.” It’s the relevant, governed subset for a specific workload, with enough context attached to make it trustworthy. It’s a small fraction of what you store, and the only way to find it is to understand the whole before you touch any of it.

Step 1: Discover what you actually have

Discovery is inventory. Across the archive, the buckets, the SaaS exports, and the databases a use case touches, the question is the same: What is in there?

A good discovery pass tells you:

• What’s there, by count and format
• How much of it is duplicated, empty, or corrupted
• When each item was last opened
• Which locations hold authoritative records and which hold working copies

Discovery has to cover the full estate rather than the one location the pilot team named because the records in that location reference customers, products, and cases whose current versions live elsewhere.

The output is a map you can act on, and the real footprint is almost always smaller than the raw one. On any share that has grown unattended for a decade, the duplicate rate alone changes the conversation.

Step 2: Classify it, then sort by value and risk

Classification asks what kind of data each item is and what it carries. That means identifying personal and regulated information, flagging anything under residency or retention rules, distinguishing authoritative records from drafts and copies, and recording lineage so you know which version supersedes which. It has to run continuously because an inventory is stale the day after it’s written.

With discovery and classification done, you can place what you found on the grid Omdia uses for dark data and sort it by value to the business and risk.

Figure 2: Assessing enterprise data by risk and value can help organizations determine how to manage it.

Not all enterprise data should be treated the same. Assessing your data by its business value and risk can help you determine the next step:

• High value, low risk: Activate quickly. These are authoritative records with no sensitive content. This is what the pilot should see first.
• High value, high risk: Activate with strict governance. Think authoritative records containing personal, financial, or regulated data. Usable once access controls and lineage are in place.
• Low value, high risk: Secure or delete. This is sensitive content the use case doesn’t need—liability with no upside.
• Low value, low risk: Archive or delete to cut costs. These might be drafts, duplicates, or empty folders.

Now, a pile of data that arrived as one undifferentiated request has four bins, each with a decision attached.

Step 3: Add the context a model can’t infer

Classification tells you a document is an authoritative record containing personal data. It doesn’t tell you the record belongs to a customer with three active amendments, an open escalation, and a renewal next quarter. Those relationships are context, and a model can’t infer them from the document alone.

Adding context means mapping each data asset to its real-world business meaning and to the other assets it relates to. When someone asks about a customer’s obligations, retrieval returns the current agreement and the amendments that modify it, in order, instead of every document that mentions the customer’s name. Answers get more accurate and the context window gets smaller, which means a smaller token bill.

The output is a set of relationships retrieval can follow, so every answer comes from the current record and whatever modifies it.

Step 4: Vectorize only what passed, with its rules attached

Now embed. Split the documents that passed steps one through three into chunks, turn the chunks into embeddings, and index them so the model retrieves by meaning. Everything the pilot team wanted to do on day one happens here on a corpus that has already been inventoried, sorted, and connected.

The first three steps pay off immediately. Duplicates and drafts stay out of the index, so it’s smaller, cheaper to build, and faster to query, and the model can’t ground an answer in a superseded version. Classification tags become retrieval filters: A query from someone who isn’t cleared for a record returns a redacted or withheld answer, and lineage traces any answer back to its source chunk. Source system permissions don’t travel into a vector index on their own. The tags carry them.

Embed first and classify later, and none of this exists. Fixing it means discarding the index and rebuilding.

Step four produces a governed index the pilot reads through an API, and a pipeline that keeps running. New files pass steps one through three before they’re embedded, so the check keeps protecting the pilot after launch.

Run these 4 steps on the data where it lives

Each of these steps invites you to move the data first: into a discovery platform, a lakehouse, and a vector database with its own store. Every move makes a copy, and every copy needs its own inventory, classification, and access controls. A check meant to shrink your dark data grows it.

Instead, run them in place. Discover and classify data where it sits, across SaaS, cloud, on-prem, and legacy systems, without installing anything on the sources. Embed from the file and object stores you already run, with no staging copy, so data stays inside the environment that already governs it.

We built our data management tools on that principle. Everpure Data Intelligence covers the first three steps across the estate and supplies the knowledge graph behind step three. Everpure Data Stream runs step four on your existing NFS and S3 stores and applies access controls at the source before embedding.

The “Take Control with AI-Ready Data” white paper goes deeper on each step, from discovery through governed delivery, with the questions to ask of your own estate. Your next pilot’s best input is probably in the dark half of your data. Find out which part before you turn it on.