How do you verify the numbers in an AI-built dashboard are real?
Something else I’ve run into with vibe coding: the agent ships something that looks finished but isn’t actually reading anything.
One example: an agent built a summary report with a confident number nobody had measured. The dashboard looked done, but the stat was a hardcoded fallback from an early prompt. It took a full day to realize it was fake.
Nothing looked broken and the number was plausible, so nobody questioned it.
One cheap check I’ve seen: change the underlying data and reload. If the stat doesn’t move, it isn’t reading anything.
I’d love to hear how you handle this:
Have you hit this? How did you find the fake value?
Do you trace each metric back to its database query?
Do you wipe or inject deliberately broken data in staging before shipping?
Any tooling or review step that catches hardcoded fallbacks?
Practical setups over theory, please. Thanks!
Replies
The rule that helps me most is upstream of the dashboard: no fallbacks for cases that cannot happen. It is written into the instructions my coding agent reads before every task. Agents love a default value "just in case", and a default is exactly what turns a broken query into a plausible number. Without it, a missing value shows up as an empty cell or an error, which is ugly and honest. The second check is your reload trick, done on purpose: before I trust a new stat in my admin panel, I find one record I can count by hand, a single user or a single day, and compare. If the dashboard and my hand count disagree, the dashboard is wrong until proven otherwise.