Metadata-Version: 1.1
Name: scrapy-rotated-proxy
Version: 0.0.2
Summary: A middleware to change proxy rotated for Scrapy
Home-page: https://github.com/xiaowangwindow/scrapy-rotated-proxy.git
Author: Alex Wang
Author-email: xiaowangwindow@163.com
License: BSD
Description-Content-Type: UNKNOWN
Description: ======
        Scrapy-Rotated-Proxy
        ======
        
        .. image:: https://img.shields.io/pypi/v/scrapy-rotated-proxy.svg
           :target: https://pypi.python.org/pypi/scrapy-rotated-proxy
           :alt: PyPI Version
        
        .. image:: https://img.shields.io/travis/xiaowangwindow/scrapy-rotated-proxy/master.svg
           :target: http://travis-ci.org/xiaowangwindow/scrapy-rotated-proxy
           :alt: Build Status
        
        Overview
        ========
        
        Scrapy-Rotated-Proxy is a middleware to dynamically configure Request proxy for Scrapy.
        It can used when you have multi proxy ip, and need to attach rotated proxy to each Request.
        Scrapy-Rotated-Proxy support multi backend storage, you can provide proxy ip
        list through Spider Settings, File or MongoDB.
        
        Requirements
        ============
        
        * Python 2.7 or Python 3.3+
        * Works on Linux, Windows, Mac OSX, BSD
        
        Install
        =======
        
        The quick way::
        
            pip install scrapy-rotated-proxy
        
        OR copy this middleware to your scrapy project.
        
        Documentation
        =============
        
        In settings.py, for example::
        
            # -----------------------------------------------------------------------------
            # ROTATED PROXY SETTINGS (Spider Settings Backend)
            # -----------------------------------------------------------------------------
            ROTATED_PROXY_ENABLED = True
            PROXY_STORAGE = 'scrapy_rotated_proxy.extensions.file_storage.FileProxyStorage'
            # When set PROXY_FILE_PATH='', scrapy-rotated-proxy
            # will use proxy in Spider Settings default.
            PROXY_FILE_PATH = ''
            HTTP_PROXIES = [
                'http://proxy0:8888',
                'http://user:pass@proxy1:8888',
                'https://user:pass@proxy1:8888',
            ]
            HTTPS_PROXIES = [
                'http://proxy0:8888',
                'http://user:pass@proxy1:8888',
                'https://user:pass@proxy1:8888',
            ]
        
        
            # -----------------------------------------------------------------------------
            # ROTATED PROXY SETTINGS (Local File Backend)
            # -----------------------------------------------------------------------------
            ROTATED_PROXY_ENABLED = True
            PROXY_STORAGE = 'scrapy_rotated_proxy.extensions.file_storage.FileProxyStorage'
            PROXY_FILE_PATH = 'file_path/proxy.txt'
            # proxy file content, must conform to json format, otherwise will cause json
            # load error
            HTTP_PROXIES = [
                'http://proxy0:8888',
                'http://user:pass@proxy1:8888',
                'https://user:pass@proxy1:8888'
            ]
            HTTPS_PROXIES = [
                'http://proxy0:8888',
                'http://user:pass@proxy1:8888',
                'https://user:pass@proxy1:8888'
            ]
        
        
            # -----------------------------------------------------------------------------
            # ROTATED PROXY SETTINGS (MongoDB Backend)
            # -----------------------------------------------------------------------------
            # mongodb document required field: scheme, username, password, ip, port
            # document example: {'scheme': 'http', 'ip': '10.0.0.1', 'port': 8080,
            # 'username':'user', 'password':'password'}
            ROTATED_PROXY_ENABLED = True
            PROXY_STORAGE = 'scrapy_rotated_proxy.extensions.mongodb_storage.MongoDBProxyStorage'
            PROXY_MONGODB_STORAGE_URI = 'mongodb://10.255.0.0:27017'
            PROXY_MONGODB_STORAGE_DB = 'vps_management'
            PROXY_MONGODB_STORAGE_COLL = 'service'
        
        
        
Platform: UNKNOWN
Classifier: Framework :: Scrapy
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Plugins
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
