Most AI product content is written by people who are excited about what AI can do.
This publication is about what happens after the excitement.
After the demo. After the launch. After the metrics look fine and the retro is congratulatory and six months later you are trying to explain why a subset of your users quietly stopped trusting the product and nobody saw it coming.
I started writing Ground Truth because I kept having the same conversation with product teams building on top of AI: smart people, real effort, genuine belief in what they were building, and almost no infrastructure for knowing whether it was actually working. Not working in the “the model runs” sense. Working in the “this improves the decisions our users are trying to make” sense.
Those are different things. The gap between them is where most AI products currently live, and it is where I spend most of my time.
My background is not a typical product path. I came up through research: marketing intelligence, child development research, a PhD in psychology and quantitative methods, years of applied research and strategy before moving into product leadership. I have spent a long time thinking about how you measure things that are hard to measure, how you design systems that are honest about their own uncertainty, and how organizations convince themselves they have evidence when what they have is a metric that is correlated with evidence in good conditions and disconnected from it in bad ones.
Consumer AI products are, structurally, a measurement problem. The teams that figure that out early will build something durable. The teams that don’t will build something impressive.
Ground Truth is for the teams that want to build something durable.
The First Series: A Framework for Rigorous AI Product Practice
The publication launches with a four-part series covering the full stack of what rigorous AI product practice actually requires, from evaluation through governance through systems design.
Part 1: A Practical AI Evaluation Framework for Consumer Products
Most AI product teams have output quality metrics and call it evaluation. Real evaluation operates across four layers: output quality, decision quality, trust infrastructure, and failure mode monitoring. Most teams only have one. This piece builds the full framework and explains why the missing layers are the ones that matter most when things go wrong.
Part 2: Why Most AI Product Metrics Fail Under Distribution Shift
Distribution shift is the mechanism by which your dashboard lies to you. Your A/B test result is still in the launch doc. Your aggregate satisfaction score is holding. A demographic subgroup you underrepresented in your training data has been receiving degraded outputs for months. This piece is about how that happens and what instrumentation can actually catch it.
Part 3: A Governance Checklist for Shipping LLM Features in Consumer Apps
Most AI governance checklists are compliance theater: documents that exist to give a team the feeling of rigor without requiring the organizational will to act on what the checklist surfaces. This piece builds a governance framework across five domains, pre-launch review gates, accountability ownership, user communication, legal exposure, and incident response, and names the organizational question underneath each one, because that is where governance failures actually live.
Part 4: How to Design Human-in-the-Loop AI Systems That Don’t Collapse
“Human-in-the-loop” has become a phrase that does a lot of reassurance work without requiring much design work. HITL systems collapse for specific, diagnosable reasons: automation bias, liability shield design, feedback loops that never close, alert fatigue, and unclear escalation ownership. This piece covers all five and ends with what a HITL system that actually holds up looks like in practice.
If you have been building AI products and feeling like the standard toolkit is not quite equipped for what you are actually dealing with, this series is for you.
If you are evaluating AI products and wondering why the metrics look fine but something feels off, this series is for you.
If you are hiring for or stepping into a Director of Product role on an AI product and want a clear framework for what rigorous practice looks like, this series is especially for you.
Ground Truth publishes on the premise that the next wave of durable AI product value will come from teams that can measure decision improvement and manage probabilistic failure modes. Everything here is oriented toward helping product teams get there.
Start with Part 1: A Practical AI Evaluation Framework for Consumer Products
Part 2: Why Most AI Product Metrics Fail Under Distribution Shift
Part 3: A Governance Checklist for Shipping LLM Features in Consumer Apps
Part 4: How to Design Human-in-the-Loop AI Systems That Don't Collapse
Ground Truth is written by Shannon Coleman, PhD. Product strategy leader and researcher with a background in psychology and quantitative methods. Her work spans govtech, healthtech, fintech, and consumer product contexts, with particular depth in enterprise AI platforms. Portfolio: shannonlcoleman.com


