Has anyone here actually tested restoring their database backup?

Last month I had a small scare on a side project. I fat-fingered a migration on a staging DB and thought, no worries, I have daily backups. Went to restore it and the dump file was 0 KB. Turns out my cron job had been failing silently for almost two months because a password changed and I never updated the script...

The dashboard said "backup enabled" the whole time, and that was the only thing I ever checked. Nothing ever errored or emailed me. If that had been prod, I would've been done...

I think a lot of us treat "backup configured" as "backup works", but those are two different things. A backup you've never restored is basically a guess. Nobody gets excited about this stuff because it's boring and nothing visibly breaks until the one day you desperately need it...

What I do now:

• A weekly script restores the latest backup into a throwaway DB and runs a couple of sanity queries (row counts, latest timestamp)

• If the backup is suspiciously small or the restore fails, it pings me on Discord

• I keep one copy in a different region/account, since the same account getting locked or deleted would take out my backups too

Still not sure if this is overkill for small projects...

Do you test your restores, or just trust the green checkmark???

And how often do you check, if at all???

Has a backup ever failed you when you actually needed it???