In my career before Amplitude, I ran campaigns as a marketing analyst, worked as a product analyst, and later became a director of data. In each role, I owned our implementation. I’ve learned a lot about the right and wrong ways to set up product analytics architecture.
Some of the best lessons came the hard way. Months after a launch, someone on my team would ask a question, and I'd find out the data couldn't answer it. Sometimes it was even worse: the data could answer it, but it was wrong.
There was good news in all those mistakes. I got into the habit of breaking down the flow of data to diagnose exactly where things went wrong and hypothesize about how I could have prevented those errors earlier in the process, before they became an urgent data validation exercise. This post is a chance for me to catalog those lessons so other teams don’t have to go through the same pains.
These steps matter more than ever because AI is a system that relies on data quality. What you get out depends on what you put in, and your analysis agents don’t know what they don't know. An agent with partial data will confidently hand you a wrong answer, the same way a personal chatbot will happily plan your day around a movie showing that doesn't exist.
At Amplitude, we believe product experiences should improve themselves. That means behavioral data needs to be collected in real time, supplemented with additional warehouse context, activated for users in live sessions, and measured to make better decisions over time. It’s a tall order, but modern technology can make it work.
In this blog, I’ll walk through a list of six data architecture decisions you need to make these better product loops possible.
The most common customer resistance to our SDK goes something like this: "We have an in-house pipeline, and our warehouse data is well governed. Why add another SDK?" Sean Kilcullen answered this question in Should I install the Amplitude SDK?, but the short answer is that SDK data is real-time and qualitatively different from warehouse data, so you have more flexibility to answer questions.
If I were starting over, I'd install the SDK on day one and turn on Session Replay and autocapture. The data in those replays is irreplaceable. I understand the usual objection: no one has the time to watch hours of recordings. But no one has to. An agent can review every session overnight and flag where users got stuck. It’s a shortcut to understanding your customers, and it’s available out of the box with the Amplitude SDK.
When starting data collection, review your user identity strategy. Unidentified users inflate retention and skew conversion tracking. So do anonymous users who turn out to be the same person. Identity lets the warehouse context attach to the right user as soon as they're identified.
Your team needs clarity about which customers can be identified and which traffic will be anonymous. It’s OK to have both. You just need to make a distinction so you don’t mistakenly compare the two.
These problems are much easier to solve at the beginning of your setup than to try repairing later.
What you get: an agent that finds problems without you having to guess where they'll be, and a retention number that holds up in front of leadership.
Collecting data in real time is only half the battle. The other part is turning that information into something useful for a user in the exact moment they need it. When your system notices that someone is struggling, it should be able to send that person a guide to point them in the right direction or even show them a new experience through Web Experimentation.
Both these processes run through our SDK, (either the unified SDK, the Guides and Surveys SDK, or the Experiment SDK). Warehouse data usually doesn’t move quickly enough to support real-time interactions like this. On top of the speed problems, warehouses often struggle to wrangle data from several different point solutions into one place and effectively turn it all into clear insights without significant engineering effort. Not to mention translating that insight into action in another system down the road and dealing with (re)-matching users.
What you get: a chance to make a real impact on a user by helping them while they're still struggling, instead of hearing about it the next day.
The biggest lesson I’ve learned from owning implementation is that accuracy has diminishing returns. Client-side data can be incomplete for a number of reasons, including ad blockers and spotty internet connectivity. That data will never survive a check designed to ensure comprehensive data collection, but it doesn’t have to.
Real-time behavioral data works like weather radar. It can show you something forming before it’s concrete enough to measure precisely. It won’t always match your source of truth exactly, but the goal is quickly finding directional learnings and acting on them.
This type of directionally accurate data is exactly what you need to be successful with AI. If you only track precise, discrete events, everything between them is invisible. The wrong turns, backtracking, and confusion fall into the gaps. With the full behavioral picture, an agent can sort every user who ran into the same problem into two groups: those who found a workaround and those who gave up. By comparing the groups, it’s easy to understand where and why friction is occurring.
What you get: agents that recognize patterns across the whole user experience, not just the events you thought to track.
It doesn’t take long working with data to learn that it’s never going to be perfect. The imperfections that do real damage are the ones you can't see, because the bad charts look just as right as the good ones. And the wrong insights feel just as true as the right ones.
How many times have you analyzed a marketing campaign with a goal of first customer purchase and found that that customer’s “first” purchase was actually years earlier, and the analytics tool hadn’t looked back that far? How many times has a funnel understated conversions because the analytics tool didn’t include conversions for users it didn’t see on the client?
People make mistakes when they’re working with bad information. They aren’t being careless. They’re reacting correctly to the data they see. The problem is that there are gaps that aren't visible in the chart. When the numbers don't match the warehouse, they blame the tool instead of the implementation because it’s the thing directly in front of them. Those types of invisible gaps corrode trust one "that doesn't look right" at a time.
The solution to this confusion is to look for the gaps first. Here are some helpful starting points:
- Document what the data means. Agree on shared definitions, what the data captures, and its known limitations.
- Validate it regularly. Automate checks against those expectations and flag changes that could affect decisions.
- Guide how teams use it. Explain which questions the data can answer and how to interpret differences between sources.
- Assign clear owners. Name who maintains definitions, resolves data issues, and keeps team guidance current.
AI raises the stakes for your data reliability, because a confident wrong answer does more damage than no answer. An LLM on top of a warehouse can read tables, but it usually doesn't know how the events got there. Without that context, it may misunderstand a question or invent logic to fill an analysis gap. SDK data carries extra context, so there’s less of a chance for AI to produce a wrong answer.
What you get: A well-documented, fully governed data workflow that reliably produces correct answers and increases trust across the entire organization.
Your slower-moving, source-of-truth data has an important role as a supplement to the behavioral data you’re tracking with the SDK.
The non-event data in your warehouse is extremely useful, including profile-level attributes like customer tier, geography, and lifetime value. This context changes slowly, so daily updates are sufficient, as long as the system is alerted when something changes.
Without that type of context, AI will act on behavioral data alone. If it sees a user struggling, it will jump in to help. That’s a good habit. But without information about whether that user is on a free trial or is your largest account, it may suggest using a feature the user doesn’t have access to. By combining behavioral data with warehouse data, your team gets the clearest possible picture of a customer, without sacrificing speed or trust.
What you get: recommendations and actions that survive contact with the real world take more than just the SDK's data.
Experimentation is a process that turns inexact directional insights into validated results.
Don't waste time lining up every number and report across your entire system. Reconcile the important conversion events and leave the rest as directionally correct. Then you can say with statistical significance that changing a flow moves more users to the next stage of their lifecycle.
Product analytics has always struggled with that last step. You could prove the beginning of the equation (a change increased clicks), and you knew users who clicked more were worth more, but connecting the dots from that change to increased value required a leap of faith.
With analysis, experiments, and rollouts in one platform, you solve that validation gap. Your team can go from a problem to a limited experiment to a validated answer without the guesswork. In the future, that connected system can produce the same results even faster by allowing AI agents to do some of the work.
What you get: impact you can measure and tie back to a decision, using numbers everyone agrees on.
You don't need to make all six decisions on day one
You shouldn't wait until you've worked through this whole checklist to make a change to your data architecture. A reasonable plan is to get the SDK running in the next 90 days and start learning from it. But the SDK is the first step, not the last. Don't stop there and judge the whole system from that one piece. Each step adds context to the others, so the value compounds.
The faster you move through this list, the better off you’ll be. Data you’re not collecting today can't be recovered later. However, messy data you collect today can be cleaned up later if you need it.
As far as your AI system goes, it doesn't know what it doesn't know. People reading charts have the same knowledge gaps. Most teams early in their analytics maturity don't know half the ways their data can mislead them.
While you get your self-improving product loop up and running, here are a few steps you can run manually to get pretty good results:
- Find a problem in Session Replay
- Check who it affects against your warehouse data
- Ship a fix to a slice of users as an experiment
- Measure the result on the conversion events you've reconciled
Amplitude’s product agent Wave is built to automate that manual workflow. It analyzes data signals together to find a product opportunity. With your approval, Wave takes action and measures the impact. The more of these steps you take, the more context you give Wave to work with and the better the eventual results will be.












