The usability test went fine.
Users understood the interface. They completed the tasks. They rated the AI assistant highly on helpfulness and said they would use it again. The research team reported positive findings. The feature shipped.
Six weeks later, a subset of users in a specific demographic segment had been acting on AI recommendations that were subtly miscalibrated for their context. The recommendations looked right and sounded authoritative, and they passed every satisfaction metric the team had been tracking. Nobody reported a problem. Those users just quietly made worse decisions than they would have made without the AI.
The usability test did exactly what it was designed to do. It measured task completion and perceived helpfulness. For an AI product, those were the wrong places to stop, and most research teams haven’t yet built the methods to go further.
What makes AI products a different research subject
Traditional UX research was built for deterministic systems: software that gives the same output for the same input, every time, for every user. Research on those systems asks whether the product works as intended, whether it’s usable, whether users understand it, and whether it meets their needs.
AI products are probabilistic. Outputs vary by user and by condition. A model can be correct on average and still wrong for one person in one context. It may hold up well on inputs that resemble its training data, then degrade silently as those inputs drift. And it can express high confidence while being wrong, in ways users have no reliable way to detect.
These properties create four research challenges that traditional methods were not designed to handle.
The satisfaction-quality gap. Users report high satisfaction with AI outputs that are subtly wrong, because fluency and confidence read as quality signals even when they aren’t. Satisfaction surveys and usability tests capture perceived quality. With deterministic software, perceived and actual quality usually sit close enough together that the gap doesn’t matter much. With AI products, it can be wide and consequential.
The calibration problem. The central research question for an AI product is whether users rely on it appropriately, trusting it more when it’s reliable and less when it isn’t. Traditional methods measure satisfaction, usability, and self-reported trust. None of those tells you whether a user’s reliance matches the system’s actual reliability.
Silent degradation. Performance can slip gradually as user inputs drift from the training distribution, as behavior shifts, or as the model runs into edge cases it wasn’t optimized for. Traditional monitoring and research catch sudden failures. Gradual degradation produces no visible event to study, so users don’t report it, and satisfaction scores hold steady until things get bad. By the time the research team notices, it’s been a problem for a while.
Differential performance across user segments. AI systems often work better for users who resemble the training data and worse for users who don’t. Usability research usually aggregates across participants, which can mask those gaps. A study showing strong average usability may be hiding serious problems for a few segments, and those are the people the study most needed to find.
Where the methods break down
Usability testing and satisfaction surveys both run into the satisfaction-quality gap. A usability test can show whether people complete tasks and find the interface intuitive. What it can’t show is whether the AI outputs they’re acting on are reliable, whether their trust is calibrated, or whether their decisions are actually better with the AI’s help. Someone who completes every task while following miscalibrated advice looks like a usability success. Surveys compound the problem. Ratings of AI helpfulness are heavily shaped by fluency and confidence, which are stylistic properties of a response and say little about its accuracy or relevance to the decision. So a helpfulness survey measures how helpful the AI felt. That’s useful to know, and it’s a different number from how helpful the AI was.
A/B tests have a timing problem. They measure an intervention’s effect at one moment, on whoever happened to be in the sample. Whether the winning variant leads to better decisions over time, whether trust stays calibrated, and whether performance holds across segments are all outside what the test can see. Higher engagement or satisfaction for the AI variant says nothing about whether users are relying on it correctly.
Unmoderated remote research tends to bury segment-level gaps. Click patterns and session recordings get aggregated across users, so unless the study was built to surface differences, they disappear. If the people hitting the AI’s worst performance are a small share of the sample, their experience vanishes into the average.
Self-reported trust scales miss calibration entirely. Asking users how much they trust the AI captures a global attitude, one that doesn’t distinguish tasks where the AI is reliable from tasks where it isn’t. High reported trust could mean a user is appropriately trusting a reliable system or dangerously over-trusting an unreliable one. The scale can’t tell you which.
What the right research practice looks like instead
Most traditional methods still belong in AI product research. They need to be paired with methods aimed directly at the four challenges above.
Start with behavioral outcomes. Alongside satisfaction, measure what users did after receiving AI outputs, and whether those actions turned out better than they would have without the AI.
Calibration needs behavioral measures of trust, ones that show whether users lean on the system more for tasks where it’s reliable and less where it isn’t. The next piece in this series goes deep on this.
Silent degradation calls for longitudinal designs that track performance and calibration over time, paired with input distribution monitoring that flags when user behavior drifts in ways that could affect the model.
Disaggregated analysis should be the default. Every study of an AI product should be designed from the start to surface performance differences across relevant user segments.
The rest of this series works through each of these in practice.
This is the first in How to Do User Research on AI Products. Start with the series introduction here. Next: How to Study Trust Calibration Without a Survey.
Ground Truth is written by Shannon L. Coleman, PhD. Research and product strategy leader with a background in psychology and quantitative methods. Her work spans govtech, healthtech, fintech, and consumer product contexts, with particular depth in enterprise AI platforms. Portfolio: shannonlcoleman.com


