Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
450 followers
Free Local AEO & SEO Spider and a Markdown content extractor
450 followers
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
How is that different from ScreamingFrog?
Fast local crawling is smart — privacy-first resonates. Do you see agency users pairing crawl data with their paid search optimization?
The Markdown extraction and multi-format export (JSON, CSV, Sitemap XML) make it especially useful for content audits and AEO workflows. Its real strength will depend on crawl depth accuracy, JavaScript rendering support, and how comprehensive its automated SEO checks are compared to established crawlers.
Finally, a crawler that doesn’t require allocating 16GB of RAM just to spin up! Screaming Frog is great for deep enterprise audits, but having a lightweight, Rust-based CLI tool that instantly dumps clean Markdown for LLM testing is exactly what the industry needs right now. This is bridging the gap between technical SEO and AI workflows perfectly. Great job shipping this!
Well done and congrats for the launch!
Crawler.sh
@weiss_arnaud Thank you Arnaud!
Here is my poor-mans open source alternative: https://github.com/pors/sitemapplus. It works for me, but PRs are always welcome
I've been looking to do this for my site and products! Exactly what I've been searching for - the AEO tools. Found one that was called "Profound". But will check out crawler for sure