Thank you for this, and you've named something I deliberately left underspecified in the piece. You're right that appropriate reliance is partly a values question, what counts as correct reliance depends on how much autonomy you want to preserve for the user versus how much you trust the model's judgment in a given context, and that's a product philosophy decision that most teams never make explicitly. It just gets baked in by default.
The conflation you're describing between model trustworthiness and user willingness to trust is one of the most consequential measurement errors I see in practice. They require completely different interventions and diagnosing the wrong one wastes a lot of effort.
On disaggregated cohort analysis, completely agreed! It's the 1st thing that gets cut when teams are moving fast and it's exactly where the failures that matter most tend to hide. Piece 2 in the series goes deeper on that specific mechanism if you're interested.
Thank you for this, and you've named something I deliberately left underspecified in the piece. You're right that appropriate reliance is partly a values question, what counts as correct reliance depends on how much autonomy you want to preserve for the user versus how much you trust the model's judgment in a given context, and that's a product philosophy decision that most teams never make explicitly. It just gets baked in by default.
The conflation you're describing between model trustworthiness and user willingness to trust is one of the most consequential measurement errors I see in practice. They require completely different interventions and diagnosing the wrong one wastes a lot of effort.
On disaggregated cohort analysis, completely agreed! It's the 1st thing that gets cut when teams are moving fast and it's exactly where the failures that matter most tend to hide. Piece 2 in the series goes deeper on that specific mechanism if you're interested.