A dashboard is a compression algorithm with a strong opinion. It takes something enormously detailed — everything that happened to everyone using your product — and reduces it to a number you can hold in your head. The reduction is the value. The reduction is also the problem, and the two cannot be separated.
The specific failure is not that metrics are wrong. It is that averages are stable while their contents churn. A retention number can sit flat for three quarters while the population underneath it is replaced entirely: the users you built for leaving at exactly the rate a different group arrives. The line says nothing happened. Everything happened.
A flat metric is not evidence of a stable system. It is evidence of a stable sum.
Three habits that help
- Look at the distribution once a month. Not the average — the shape. Bimodal distributions hiding behind a mean are the single most common way teams misunderstand their own product.
- Watch ten sessions a quarter. Recordings, calls, whatever is available. Ten is enough to break an assumption and few enough that you will actually do it.
- Ask what would have to be true. Before accepting a favorable number, name the alternative explanation that would produce the same number. Sometimes there isn't one. Often there are two.
None of this argues against measurement. Working without numbers means substituting the loudest anecdote for evidence, which is worse. It argues for treating every metric as a lossy summary whose losses you are responsible for knowing.