Organizations are making consequential decisions from evidence that was never designed to support them.
Here is what that looks like in practice. A research team spends three months on a study. The methodology is sound. The presentation is polished. The findings are interesting. Leadership nods, asks a few questions, and files it away. Six months later the same decision gets made based on gut instinct and competitive pressure. Nobody references the study. Nobody explains why.
Most people who have worked inside or alongside a research or insights function recognize this pattern immediately. What they disagree on is who is to blame. The researchers say leadership doesn’t value evidence. Leadership says the research wasn’t actionable. Both are partially right and neither diagnosis gets at the actual problem.
The actual problem is structural. Research functions are designed to produce findings. Decisions require something different: evidence calibrated to the specific choice at hand, with known uncertainty bounds, connected to the people who have to act on it. Most organizations have never explicitly designed for that. They have designed for rigor, for defensibility, for volume of output. And they have built insight functions that are very good at producing work that is methodologically sound and strategically inert.
This was always a problem. AI makes it a crisis.
The first series on Ground Truth, A Framework for Rigorous AI Product Practice, diagnosed a set of failure modes specific to consumer AI products: evaluation gaps, distribution shift, governance theater, and HITL systems that collapse under operational pressure. The through line across all four pieces was that AI product teams are optimizing for output impressiveness rather than decision quality.
Series two goes upstream. The measurement traps that undermine strategy in traditional research and insights functions are now being encoded into AI systems at scale. When the behavioral data informing model training was collected to answer the wrong questions, the model inherits those questions. When the evaluation framework was designed to be defensible rather than decision-relevant, the model gets optimized for the wrong outcomes. When qualitative signal gets systematically underweighted because it’s harder to present in a slide, that blind spot gets built into the product.
The organizations that figure this out early will build AI systems that actually improve decisions. The ones that don’t will build increasingly sophisticated systems for automating the same strategic errors they were making before AI.
The Series: Four Measurement Traps
Part 1: The Difference Between a Finding and a Decision
A finding tells you what is true. A decision requires you to know what to do next. These require different kinds of evidence, different research designs, and a fundamentally different relationship between the people producing insight and the people who have to act on it. Most organizations conflate them and wonder why their research doesn’t move anything.
Part 2: Why Most Insight Functions Produce Answers Nobody Asked For
Insight functions are structured around research production rather than decision support. They optimize for findings that are defensible rather than useful, for volume of output rather than quality of downstream decisions. In AI product contexts this means the data and research informing model training is systematically skewed toward what was easy to measure, not what was decision-relevant.
Part 3: How Research Gets Designed to Survive Scrutiny Rather Than Generate Insight
The incentive structure around research rewards defensibility over usefulness. Teams optimize for findings nobody can argue with rather than insights that force a decision. In AI contexts this produces evaluation frameworks that are methodologically sound and strategically useless, validated on the wrong questions at significant organizational cost.
Part 4: The Quant Default
When quantitative and qualitative evidence conflict, organizations default to quant because it provides accountability cover, not because it is more right. That default is a values decision masquerading as a methodological one. In AI product contexts it gets encoded directly into how models are trained, evaluated, and monitored, systematically underweighting the signal most likely to surface what the model is getting wrong.
This series is for research leads, insights strategists, people analytics leaders, and product teams who have noticed that their evidence infrastructure is not producing the quality of decisions it should. It is also for anyone building or evaluating AI products who wants to understand why the upstream measurement problems in their organization will eventually show up as downstream model failures.
The work that went into series one on AI product evaluation applies here too. The same rigor. The same willingness to name uncomfortable structural problems. The same bias toward evidence over the appearance of evidence.
The measurement traps have always been there. AI just made them impossible to ignore.
Start with Part 1: The Difference Between a Finding and a Decision
Part 2: Why Most Insight Functions Produce Answers Nobody Asked For
Part 3: How Research Gets Designed to Survive Scrutiny Rather Than Generate Insight
Part 4: The Quant Default
Previously: A Framework for Rigorous AI Product Practice — Series 1 intro
Ground Truth is written by Shannon Coleman, PhD. Product strategy leader and researcher with a background in psychology and quantitative methods. Her work spans govtech, healthtech, fintech, and consumer product contexts, with particular depth in enterprise AI platforms. Portfolio: shannonlcoleman.com


