Ibrahim Hajjaj

I built a benchmark that runs nine WordPress backup plugins head to head on identical sites. One of the nine is mine, and it currently sits second. That is an obviously compromised position, so I have spent more time on what makes a vendor-run benchmark believable than on the measuring.

Three things I landed on, and I am not sure any of them are enough.

Publish the runs you threw away. 581 completed runs, 180 scored, 401 excluded, and every excluded one is browsable with its reason attached. Most are boring reruns rather than embarrassing failures, and saying so is part of the point: 360 superseded, 41 excluded for cause. The honest number includes the boring ones.

Say when a lead is not a lead. Mine sits five points above third place with plus or minus 113 error bars. That is a tie, and the page says tie rather than letting me quote the higher number.

I run a benchmark that puts WordPress backup plugins head to head on identical sites, and I am about to add more site profiles.

Right now there are eight, from a starter blog up to a WooCommerce store with a large product catalogue. They are fixed snapshots, so every plugin gets byte-identical input.