Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
457 followers
Free Local AEO & SEO Spider and a Markdown content extractor
457 followers
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Love the local-first approach especially keeping everything fast and private
Curious — since it can crawl and extract entire sites, how are you handling safeguards around accidentally pulling sensitive or restricted data?
Feels like that could be important as usage scales
@mehmetkose Congrats on the launch! Doest it recommend actions & analysis report?
Great tool for SEO enthusiasts! How does Crawler.sh handle dynamic JavaScript-heavy websites during the crawl?
Triforce Todos
Fast, private, and outputs standard formats, that’s how tools should be. Congrats on the launch 🙌
Crawler.sh
@abod_rehman Thank you Abdul! It's written in Rust so, it's FAST. Just don't break anyone's website :)
Copperlane
Super cool - how does performance hold up on really large sites btw?
How is that different from ScreamingFrog?
Fast local crawling is smart — privacy-first resonates. Do you see agency users pairing crawl data with their paid search optimization?