Launched this week
Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
430 followers
Free Local AEO & SEO Spider and a Markdown content extractor
430 followers
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Crawler.sh
Hey Product Hunt! I built crawler.sh because I kept running into the same problem: every time I needed to audit a website's SEO or extract its content, I had to choose between bloated enterprise tools, slow cloud services, or stitching together a bunch of scripts.
crawler.sh is a single tool that does all three: crawling, SEO analysis, and content extraction, from the terminal or a native desktop app. It's built in Rust so it's fast, it runs locally so your data stays private, and it outputs standard formats so you can take the results anywhere.
@mehmetkose Congrats on your launch! You got me at bloated enterprise tools. After an SEO/AEO analysis, does this also offer suggestions to fix the detected issues?
Crawler.sh
@jerrybyday Thank you! and YES. You can see them on the UI or download them. I'm adding new features every day in our core modules to discover issues smartly
@mehmetkose Hey Mehmet
Thanks for that, it's just on time, the topic is hot
But what’s the exact moment or event in Crawler that makes me think: ‘This is worth paying for’?
like aha-moment
Loving the ease of this. Is there a team plan?
Crawler.sh
@jacklyn_i Thank you Jacklyn William I'm thrilled! I'm adding this on the roadmap! Will let you know when is implemented. Also the cloud functionality is on the way
@mehmetkose Awesome! I’d follow your account to stay connected on your journey as you continue to grow. Congrats on your launch again.
Local-first approach is underrated, most SEO tools are cloud-heavy and slow. The fact that this runs from terminal AND has a native desktop app is a nice touch for different types of users.
The Markdown extraction feature caught my eye specifically,been looking for something that can cleanly pull content without all the HTML noise. How does it handle JavaScript-heavy sites like React or Next.js? Does it wait for JS to render or purely static crawl?
Also curious, AEO (Answer Engine Optimization) is still pretty new territory. What kind of checks are you running for that specifically? Most tools haven't caught up there yet.
Congrats on the launch, terminal-based dev tools don't get enough love on PH!
Rumora
Man, markdown extraction is such a lifesaver when dealing with giant websites. Way easier than my old 'copy-paste marathon' method. 😂
Crawler.sh
@djordjevic_nikola Oh that's exactly what i would like to hear. Give me some more :)) Also check it out that you can copy your SEO issues as a prompt so your llm can go fix.
Congrats on the launch and the product!
I'm curious if you spotted a pattern regarding how people are using the product already - is it more focused on their own sites or competitor analysis?
Love the positioning, local, fast, no bloat is refreshing. How deep does the AEO analysis go? Is it just schema and structured data checks, or does it also evaluate content for AI-answer readiness?
I really like this local-first approach. Congrats!
Does this handle JavaScript-heavy sites (React, Next.js)? Or is it pure HTML crawling? I believe that's usually where terminal crawlers fall short vs the bloated enterprise tools, but I also might be mistaken.