Crawly - Never write another web scraper

I love this for its uniqueness ability to specify parameters. Hugely useful! Adding this to my running list of Scraping and Crawling Technologies! http://scrapinghub.com http://www.outwit.com/products/hub/ http://webroots.io http://kimonolabs.com http://grabby.io http://fullcontact.com http://emailhunter.co http://clearbit.com http://toofr.com http://import.io http://kimonolabs.com http://apifier.com (my number one favorite) http://elink.club http://www.eliteproxyswitcher.com/ - ;) http://www.uipath.com/ http://diffbot.com http://cloudscrape.com http://community.screen-scraper.... https://commoncrawl.org/ http://www.fminer.com/ https://scraperwiki.com/ http://nutch.apache.org/ http://www.ubotstudio.com/index7 http://mozenda.com http://fivefilters.org/

Hey ProductHunters! Crawly is a free tool I built that uses Diffbot's automatic article extraction api to turn web content into structured data. I've used it for creating a centralized database of all of our content, but you could also use it to do content audits / migrations or analyze your competitor's content. It's currently limited to 200 pages and only articles at the moment, but I plan to add support for scraping products in the future. Any other features you'd like to see added?

Awesome!!! Looking for a replacement for Kimono (https://www.kimonolabs.com/) since they got acquired by Palantir. Need something to power my Slack menu bot ;)

Looks great. Congrats on the launch! I can see some nice applications for this. One suggestion - I understand that it might take a while to scrape the data, but an instant email to say it will be X minutes, or just a notification after email input would be good, to manage expectations. I used it 10 mins ago, and am on the verge of tears that I still have no email... ;-)

I would love to be able to scrape sports stats easily!

Diffbot is such a useful service for manipulating published content - huge fan of this team.

How can I try this? I think there is some server issue Code 503.

The potential of this is really big and there are clearly needs across a lot of industries. Unfortunately, my tests with some ecommerce product pages had less than stellar results.

Awesome! Any plans for a command-line version in the future?