Metadata-Version: 2.1
Name: exoskeleton
Version: 0.9.1
Summary: A library to create a bot / spider / crawler.
Home-page: https://github.com/RuedigerVoigt/exoskeleton
Author: Rüdiger Voigt
Author-email: projects@ruediger-voigt.eu
License: UNKNOWN
Description: # Exoskeleton
        
        ![Python package](https://github.com/RuedigerVoigt/exoskeleton/workflows/Python%20package/badge.svg)
        ![Supported Python Versions](https://img.shields.io/pypi/pyversions/exoskeleton)
        ![Last commit](https://img.shields.io/github/last-commit/RuedigerVoigt/exoskeleton)
        
        For my dissertation I downloaded hundreds of thousands of documents and feed them into a machine learning pipeline. Using a high-speed-connection is helpful but carries the risk to run an involuntary denial-of-service attack on the servers that provide those documents. This creates a need for a crawler / scraper that avoids too high loads on the connection and instead runs permanently and fault tolerant to ultimately download all files.
        
        Exoskeleton is a python framework that aims to help you build a similar bot. Main functionalities are:
        * Managing a download queue within a MariaDB database.
        * Avoid processing the same URL more than once.
        * Working through that queue by either
            * downloading files to disk,
            * storing the page source code into a database table,
            * or making PDF-copies of webpages.
        * Managing already downloaded files:
            * Storing multiple versions of a specific file.
            * Assigning labels to downloads, so they can be found and grouped easily.
        * Sending progress reports to the admin.
        
        **[Exoskeleton has an extensive documentation.](https://github.com/RuedigerVoigt/exoskeleton/tree/master/documentation "Learn about using exoskeleton")**
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.6
Description-Content-Type: text/markdown
