Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

pjscrape is a framework for anyone who's ever wanted a command-line tool for web scraping using Javascript and jQuery. Built to run with PhantomJS, it allows you to scrape pages in a fully rendered, Javascript-enabled context from the command line, no browser required.

ScraperWiki is all the tools you need for Screen Scraping, Data Mining & visualisation

SelectorGadget is an open source bookmarklet that makes CSS selector generation and discovery on complicated sites a breeze.