Test robots.txt rule changes work first before applying them