Scrawly is a free, open-source desktop SEO crawler for macOS, Windows and Linux. It crawls your site, renders JavaScript and runs 250 technical SEO, GEO and AI-search checks, then tells you in plain English what to fix. Unlike Screaming Frog or Sitebulb, there's no licence, no account and no page cap. It also tests whether GPTBot, ClaudeBot and PerplexityBot can reach you, audits llms.txt, applies reversible WordPress fixes and ships an MCP server. MIT licensed.

Hey Product Hunt 👋 I'm Ilias, a white label SEO consultant from Dhaka, Bangladesh. For the last few years I've done technical SEO audits for agencies, mostly with paid desktop crawlers. They're excellent tools. But two things kept bugging me: 1. Every freelancer and junior SEO I mentored hit the same wall: a licence fee or a free tier capped at a few hundred URLs, right when they needed to learn on real sites. 2. My clients started asking a new question: "Can ChatGPT and Perplexity actually see us?" None of my tools answered it properly. So I built the crawler I wanted, and I'm giving it away. What Scrawly does: • Crawls any site and renders JavaScript in real Chromium (desktop and Googlebot Smartphone) • Runs 250 checks across 23 areas: indexability, redirects, canonicals, titles by pixel width, structured data, Core Web Vitals, hreflang, security and more • Tests 10 crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended…) against robots.txt AND with a live fetch, so you catch the CDN or firewall rules that silently block AI bots • Audits llms.txt and scores how ready your site is for AI agents, from Level 0 to 5 • Pulls in Search Console, GA4 and PageSpeed data, exports white-label HTML and PDF reports • Fixes titles, meta descriptions, canonicals and redirects on WordPress, with a snapshot and an undo button for every change How the approach changed: I started with a simple "list the problems" crawler. Real audits taught me that a list of 900 warnings helps nobody. So Scrawly now ranks findings by how many pages they hit, never reports the same problem twice, and gives every finding a fix you can paste into your coding agent. It runs entirely on your computer. No account, no analytics, no page cap, MIT licensed. Install is one line in Terminal or PowerShell, and it keeps itself updated. I'd love your honest feedback, especially: → Which check is missing from your audit workflow? → What would make you switch from your current crawler? Thanks for checking it out 🙏

Testing Scrawly was a good experience, especially the white-label part, which impressed me. I plan to try aligning it with my SEO SOP. Thank you, Ilias, for making this tool open source.

I looked at Screaming Frog and Sitebulb before testing Scrawly. Those tools require paid licenses and put limits on free crawls, while Scrawly is open source with no page caps. The white-label feature and plain-English fixes make it much easier to fit into my agency SOP.

Testing Scrawly was a good experience, especially the white-label part, which impressed me. I plan to try aligning it with my SEO SOP. Thank you, Ilias, for making this tool open source.

Scrawly

Free open-source SEO crawler for Google and AI search

Free open-source SEO crawler for Google and AI search

What's great

vs Alternatives

What's great

vs Alternatives