Metadata-Version: 2.1
Name: simple-crawler
Version: 0.1
Summary: my simple crawler
Home-page: https://github.com/weaming/simple-crawler
Author: weaming
Author-email: garden.yuen@gmail.com
License: UNKNOWN
Project-URL: Source, https://github.com/weaming/simple-crawler
Project-URL: Bug Reports, https://github.com/weaming/simple-crawler
Keywords: crawler
Platform: UNKNOWN
Description-Content-Type: text/markdown
Requires-Dist: beautifulsoup4
Requires-Dist: requests

## Install

`pip3 install simple-crawler`

## Example

```
from simple_crawler import *


class MyCrawler(Crawler):
    name = 'output.txt'
    def custom_handler_page(self, page):
        print(page.url)
        tags = page.soup.select("#nr1")
        tag = tags and tags[0]
        with open(self.name, 'a') as f:
            f.write(tag.text)
        print(tag.text)

    def filter_url(self, url: URL) -> bool:
        return url.url.startswith("https://xxx.com/xxx")


c = MyCrawler("https://xxx.com/xxx")
c.start()
```


