Metadata-Version: 2.1
Name: scrapydweb
Version: 1.0.0
Summary: A full-featured web UI for Scrapyd cluster management, with Scrapy log analysis & visualization supported.
Home-page: https://github.com/my8100/scrapydweb
Author: my8100
Author-email: my8100@gmail.com
License: GNU General Public License v3.0
Description: English | [简体中文](https://github.com/my8100/scrapydweb/blob/master/README_CN.md)
        
        # ScrapydWeb: A full-featured web UI for Scrapyd cluster management, with Scrapy log analysis & visualization supported.
        
        [![PyPI - scrapydweb Version](https://img.shields.io/pypi/v/scrapydweb.svg)](https://pypi.org/project/scrapydweb/)
        [![Downloads - total](https://pepy.tech/badge/scrapydweb)](https://pepy.tech/project/scrapydweb)
        [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/scrapydweb.svg)](https://pypi.org/project/scrapydweb/)
        [![Coverage Status](https://coveralls.io/repos/github/my8100/scrapydweb/badge.svg?branch=master)](https://coveralls.io/github/my8100/scrapydweb?branch=master)
        [![GitHub license](https://img.shields.io/github/license/my8100/scrapydweb.svg)](https://github.com/my8100/scrapydweb/blob/master/LICENSE)
        [![Twitter](https://img.shields.io/twitter/url/https/github.com/my8100/scrapydweb.svg?style=social)](https://twitter.com/intent/tweet?text=@my8100_%20ScrapydWeb:%20A%20full-featured%20web%20UI%20for%20Scrapyd%20cluster%20management,%20with%20Scrapy%20log%20analysis%20%26%20visualization%20supported.%20%23python%20%23scrapy%20%23scrapyd%20%23webscraping%20%23scrapydweb%20&url=https%3A%2F%2Fgithub.com%2Fmy8100%2Fscrapydweb)
        
        
        ##
        ![overview](https://raw.githubusercontent.com/my8100/scrapydweb/master/screenshots/overview.png)
        
        ## Scrapyd ScrapydWeb
        ### Recommended Reading
        [How to efficiently manage your distributed web scraping projects](https://medium.com/@my8100/https-medium-com-my8100-how-to-efficiently-manage-your-distributed-web-scraping-projects-55ab13309820)
        
        
        ## Features
        <details>
        <summary>View contents</summary>
        
        - Scrapyd Cluster Management
          - All Scrapyd JSON API Supported
          - Group, filter and select any number of nodes
          - **Execute command on multinodes with just a few clicks**
        
        - Scrapy Log Analysis
          - Stats collection
          - **Progress visualization**
          - Logs categorization
        
        - Enhancements
          - HTML caching for the Log and Stats page
          - Basic auth for web UI
          - **Auto eggify your projects**
          - :e-mail: **Email notice**
          - Mobile UI
        
        </details>
        
        
        ## Preview
        - [Screenshots](https://github.com/my8100/files/tree/master/scrapydweb/README.md)
        
        - [Gif Demo](https://github.com/my8100/files/tree/master/scrapydweb/README_GIF.md)
        
        
        ## Getting Started
        <details>
        <summary>View contents</summary>
        
        ### Prerequisites
        **Make sure that [Scrapyd](https://github.com/scrapy/scrapyd) has been installed and started on all of your hosts.**
        
        Note that for remote access, you have to manually set 'bind_address = 0.0.0.0' in [the configuration file of Scrapyd](https://scrapyd.readthedocs.io/en/latest/config.html#example-configuration-file)
        and restart Scrapyd to make it visible externally.
        
        ### Install
        - Use pip:
        ```bash
        pip install scrapydweb
        ```
        
        - Use git:
        ```bash
        git clone https://github.com/my8100/scrapydweb.git
        cd scrapydweb
        python setup.py install
        ```
        
        ### Start
        1. Start ScrapydWeb via the `scrapydweb` command. (a config file would be generated for customizing settings on the first startup.)
        2. Visit http://127.0.0.1:5000 **(It's recommended to use Google Chrome for a better experience.)**
        
        ### Browser Support
        The latest version of Google Chrome, Firefox, and Safari.
        
        </details>
        
        
        ## Running the tests
        <details>
        <summary>View contents</summary>
        
        <br>
        
        ```bash
        $ git clone https://github.com/my8100/scrapydweb.git
        $ cd scrapydweb
        
        # To create isolated Python environments
        $ pip install virtualenv
        $ virtualenv venv/scrapydweb
        # Or specify your Python interpreter: $ virtualenv -p /usr/local/bin/python3.7 venv/scrapydweb
        $ source venv/scrapydweb/bin/activate
        
        # Install dependent libraries
        (scrapydweb) $ python setup.py install
        (scrapydweb) $ pip install pytest
        (scrapydweb) $ pip install coverage
        
        # Make sure Scrapyd has been installed and started, then update the custom_settings item in tests/conftest.py
        (scrapydweb) $ vi tests/conftest.py
        (scrapydweb) $ curl http://127.0.0.1:6800
        
        (scrapydweb) $ coverage run --source=scrapydweb -m pytest tests/test_a_factory.py -s -vv
        (scrapydweb) $ coverage run --source=scrapydweb -m pytest tests -s -vv
        (scrapydweb) $ coverage report
        # To create an HTML report, check out htmlcov/index.html
        (scrapydweb) $ coverage html
        ```
        
        </details>
        
        
        ## Built With
        <details>
        <summary>View contents</summary>
        
        <br>
        
        - Front End
          - [jQuery](https://github.com/jquery/jquery)
          - [Vue.js](https://github.com/vuejs/vue)
          - [Element](https://github.com/ElemeFE/element)
          - [ECharts](https://github.com/apache/incubator-echarts)
        - Back End
          - [Flask](https://github.com/pallets/flask)
          - [Flask-Compress](https://pypi.org/project/Flask-Compress/)
          - [Requests](https://github.com/requests/requests)
        
        </details>
        
        
        ## Changelog
        Detailed changes for each release are documented in the [HISTORY.md](./HISTORY.md).
        
        
        ## Author
        | [<img src="https://github.com/my8100.png" width="100px;"/>](https://github.com/my8100)<br/> [<sub>my8100</sub>](https://github.com/my8100) |
        | --- |
        
        
        ## Contributors
        | [<img src="https://github.com/simplety.png" width="100px;"/>](https://github.com/simplety)<br/> [<sub>Kaisla</sub>](https://github.com/simplety) |
        | --- |
        
        
        ## License
        This project is licensed under the GNU General Public License v3.0 - see the [LICENSE](./LICENSE) file for details.
        
Platform: UNKNOWN
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*
Description-Content-Type: text/markdown
