As many seasoned CROs know, not all test ideas are winners.

When starting A/B testing for the first time ever or restarting a program, ideating and producing wins can be pretty easy. There are likely many UX best-practices misses and a backlog of test ideas that were pent-up with no outlet to fuel an initial series of tests.

However, eventually low-hanging fruit ideas will run out, forcing CROs to base test ideas beyond UX best practices and known historical friction points on the site.

Website analytics can provide a treasure trove of data about how users interact and convert on a given site. However, depending on the analytics platform, how well it is set up, and who is interpreting the data, extracting viable test ideas can not only prove to be difficult, or even impossible. Where quantitative analytics data falls short, qualitative user studies can help brands looking to generate test ideas backed by the voice of customers instead of relying on intuition or guessing.

Here is a closer look at why user studies can be the key to unlocking your next massive testing win and how one brand and its CX consultancy, CXperts, used them to significantly increase conversion rates and average order value (AOV).

Benefits of user studies for ideation

When you base your testing roadmap purely on quantitative analytics, you are only seeing half the picture. By incorporating qualitative user research methods into your process, you can stop guessing and start solving real problems. The impact on your testing success rate is not just anecdotal; it is measurable. In our own analysis, we found that tests generated without a preceding user study had a 64% win rate. However, tests that were derived directly from a user study saw a massive jump to a 73% win rate.

Why does this happen? Here is why integrating customer feedback into your ideation phase is so effective:

Answers the “why” behind the “what”: Quantitative analytics (like bounce rates or exit pages) tell you where a problem is occurring, but qualitative user studies tell you why it is happening. These fuller insights eliminate guessing the root cause of user friction and throwing arbitrary solutions at the wall.
Significantly higher win rates: As the 73% stat shows, tests built on validated user feedback are far more likely to succeed. You are solving actual pain points that users have explicitly identified rather than testing based on internal “gut feelings.”
Eliminates internal bias: It shifts the conversation from “I think the user wants…” to “The user told us they want…” This helps settle internal debates and aligns testing roadmaps with true customer needs.
Uncovers psychological roadblocks: Analytics cannot measure emotions. User studies reveal psychological barriers – such as cognitive overload, decision fatigue, or a lack of trust – that heatmaps or click-tracking simply cannot capture.
Drives strategic overhauls, not just tweaks: Feedback from user studies often leads to foundational, high-impact tests (like rethinking a whole merchandising strategy or page layout) rather than superficial, low-impact changes (like simply changing CTA color or position).

The pitfall of the “Paradox of Choice”

To understand how powerful this can be in practice, let’s look at a recent A/B test conducted on Graco Baby (gracobaby.com, a Newell Brands company).

Graco Baby had a robust product listing page for their highly popular infant car seats. The page displayed the full assortment of products, allowing users to browse through numerous variations, models, and price points.

Intuition may say that more choice is a good thing as it provides more opportunities for the user to find a product. Analytics may tell you that users who go from the listing page to the detail page convert at a higher rate, which further supports the claim. However, intuition and analytics can mislead. More choices can overwhelm users, and it’s easy to mistake correlation with causation when looking at analytics.

CXperts’ engagement with Graco Baby included a user study that revealed surprising insights:

Through the user study, they discovered that shoppers were actually feeling overwhelmed by the sheer volume of choices. Infant car seats are high-consideration purchases; parents are deeply concerned with safety features, weight limits, and price. When confronted with a massive grid of slightly different car seats, cognitive overload sets in.

The study revealed a fascinating psychological insight: users actively preferred a limited, more simplified assortment of products. They wanted the brand to curate the experience, providing an educational journey that highlighted key styles and clearly differentiated them. This feedback perfectly aligned with the “Paradox of Choice”, a psychological principle suggesting that too much choice paralyzes consumers and delays decision-making.

Without direct customer feedback and relying solely on quantitative user behavior data, uncovering this “paradox of choice” barrier would have been nearly impossible. Intuition says “give them everything,” but the users explicitly asked for “less.”

Formulating a user-backed hypothesis and test design

Because this test idea was generated directly from user study findings, the hypothesis was highly targeted and strategically sound:

Hypothesis: Providing shoppers with a more targeted and educational product assortment will simplify their decision-making and enhance their product exploration experience.

To prove this, the team used Wingify to design an A/B test targeting infant car seat shoppers across all devices. Rather than sending users to the standard, full-assortment grid, the team built a completely new, educational landing page experience.

The test introduced a streamlined approach, reducing the primary view to just six key products. These were thoughtfully ordered based on price and feature sets to help parents easily understand the progression in value and safety features between models. (A link to view the full catalog was kept available for power shoppers who still wanted to browse everything).

By stripping away the excess and actively curating the user journey, the team wasn’t just optimizing a layout; they were fundamentally testing a user-backed merchandising strategy designed to reduce friction and build confidence.

Results: When qualitative insights pay off

The test ran its course, reaching statistical significance, and the results were a resounding validation of the initial user study.

By reducing cognitive load and presenting a logical progression of key products, shoppers found it significantly easier to compare features and make a confident decision. This streamlined, educational approach dramatically impacted the bottom line:

Infant car seat “Add to Carts” conversions jumped by 33%.
The conversion rate for infant car seat purchases doubled.
Average Order Value (AOV) increased by 7.3%.

By alleviating decision fatigue, Graco Baby didn’t just sell more items; they efficiently sold higher-value items because users could fully understand the value progression of the simplified lineup.

This staggering win highlights the ultimate ROI of qualitative research. When your test variations directly answer a documented user pain point, you stop throwing spaghetti at the wall. You implement strategic, high-impact changes that users are literally asking for.

Conclusion: Marrying the “what” with the “why”

Analytics and heatmaps will always be essential for identifying where the friction lies in your digital experience. But if you want to drastically improve the win rate of your A/B testing program, you must rely on user studies to uncover the why.

By listening to their customers, recognizing the paralyzing effect of choice overload, and using a robust platform to seamlessly execute and measure the solution, Graco Baby turned a qualitative insight into huge lifts in both conversion rate and AOV.

Stop guessing what your users want. Ask them, build a test around their answers, and watch your win rates soar.