Metadata-Version: 2.0
Name: scrape
Version: 0.0.7
Summary: a webpage scraping tool
Home-page: https://github.com/huntrar/scrape
Author: Hunter Hammond
Author-email: huntrar@gmail.com
License: MIT
Keywords: scrape webpage website pdf text keyword crawl save page
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Requires-Dist: lxmlpdfkit

# scrape

## 
a web scraping tool

## Installation
* `pip install scrape`

## Usage
usage: scrape.py [-h] [-f [FILTER [FILTER ...]]] [-c [CRAWL [CRAWL ...]]]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[-ca]&nbsp;[-l&nbsp;LIMIT]&nbsp;[-t]&nbsp;[-vb]&nbsp;[-v]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[urls&nbsp;[urls&nbsp;...]]

a web scraping tool

positional arguments:
&nbsp;&nbsp;urls&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;urls&nbsp;to&nbsp;scrape

optional arguments:
&nbsp;&nbsp;-h,&nbsp;--help&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;show&nbsp;this&nbsp;help&nbsp;message&nbsp;and&nbsp;exit
&nbsp;&nbsp;-f&nbsp;[FILTER&nbsp;[FILTER&nbsp;...]],&nbsp;--filter&nbsp;[FILTER&nbsp;[FILTER&nbsp;...]]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;filter&nbsp;lines&nbsp;by&nbsp;keywords,&nbsp;text&nbsp;only
&nbsp;&nbsp;-c&nbsp;[CRAWL&nbsp;[CRAWL&nbsp;...]],&nbsp;--crawl&nbsp;[CRAWL&nbsp;[CRAWL&nbsp;...]]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;enter&nbsp;keywords&nbsp;to&nbsp;crawl&nbsp;links
&nbsp;&nbsp;-ca,&nbsp;--crawl-all&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;crawl&nbsp;all&nbsp;links
&nbsp;&nbsp;-l&nbsp;LIMIT,&nbsp;--limit&nbsp;LIMIT
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;crawl&nbsp;page&nbsp;limit
&nbsp;&nbsp;-t,&nbsp;--text&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;write&nbsp;to&nbsp;text&nbsp;instead&nbsp;of&nbsp;pdf
&nbsp;&nbsp;-vb,&nbsp;--verbose&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;show&nbsp;pdfkit&nbsp;errors
&nbsp;&nbsp;-v,&nbsp;--version&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;display&nbsp;current&nbsp;version

## Author
* Hunter Hammond (huntrar@gmail.com)

## Notes
* Unless specified using the --text flag, all webpages are saved as pdf files using pdfkit.

* The --filter flag may be used in conjunction with --text to only save lines matching one or more keywords provided

* Subsequent links may be followed by entering --crawl-all or --crawl. --crawl accepts a list of substrings to control which URL's to crawl.

* There is no limit to the number of pages to be crawled unless one is set using the --limit flag.



News
====

0.0.7
------

 - changed --keywords to --filter and positional arg url to urls

0.0.6
------

 - use --keywords flag for filtering text
 - can pass multiple links now
 - will not write empty files anymore

0.0.5
------

 - added --verbose argument for use with pdfkit
 - improved output file name processing

0.0.4
------

 - accepts 0 or 1 url's, allowing a call with just --version

0.0.3
------

 - Moved utils.py to scrape/

0.0.2
------

 - First entry




