Metadata-Version: 2.1
Name: silene
Version: 1.0.0
Summary: Silene is an open source web crawler framework built upon Pyppeteer.
Home-page: https://github.com/peterbencze/silene
Author: Peter Bencze
Author-email: benczepeter95@gmail.com
License: Apache License, Version 2.0
Project-URL: Bug Reports, https://github.com/peterbencze/silene/issues
Project-URL: Funding, https://www.paypal.com/paypalme/peterbencze
Project-URL: Source, https://github.com/peterbencze/silene
Description: # Silene
        
        Silene is an open source web crawler framework built upon [Pyppeteer](https://github.com/pyppeteer/pyppeteer).
        
        ## Requirements
        You must have at least [Python 3.7](https://www.python.org/downloads/) installed.
        
        ## Installation
        To install the latest release run `pip install silene`.
        
        ## Quickstart guide
        
        Each crawler must subclass the `Crawler` class and implement the abstract `configure` method. The `CrawlerConfiguration`
        specifies the initial requests to make and other properties of the crawler. Once a request is processed, the appropriate
        callback will be invoked. By default, in case of a successful request the
        `on_response_success` callback will be executed. This is where you can interact with the page content. You can also
        specify custom callbacks for your requests.
        
        Below you can find a very simple implementation.
        
        ### Example code snippet
        
        ```python
        from silene.crawl_request import CrawlRequest
        from silene.crawl_response import CrawlResponse
        from silene.crawler import Crawler
        from silene.crawler_configuration import CrawlerConfiguration
        
        
        class MyCrawler(Crawler):
            def configure(self) -> CrawlerConfiguration:
                return CrawlerConfiguration([CrawlRequest('https://example.com')])
        
            def on_response_success(self, response: CrawlResponse) -> None:
                # Do something with the response...
                pass
        ```
        
        ## Development instructions
        
        ### Prerequisite
        
        This project requires [Pipenv](https://docs.pipenv.org/) to be installed.
        
        ### Create environment
        
        Run `pipenv install --dev` to create a new virtual environment and install the necessary packages.
        
        ### Run tests
        
        Run `pytest` in the project root folder.
        
        ### Run tests with coverage
        
        Run `pytest --cov=silene` in the project root folder.
        
        ## License
        
        The source code of Silene is made available under
        the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
        
Keywords: crawler scraper spider framework pyppeteer data
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.7
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: dev
