Metadata-Version: 2.1
Name: scrapy-scraperapi-middleware
Version: 0.2
Summary: Middleware to easily implement ScraperAPI in Scrapy projects
Home-page: UNKNOWN
Author: Patrick Klein
License: MIT
Project-URL: Source, https://github.com/patkle/scrapy-scraperapi-middleware
Keywords: scrapy middleware proxy ScraperAPI
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Utilities
Requires-Python: >=3
Description-Content-Type: text/markdown
Requires-Dist: importlib-metadata (~=1.0) ; python_version < "3.8"

# scrapy-scraperapi-middleware
This middleware lets you use [Scraper API](https://www.scraperapi.com/?fp_ref=patrick50) for every request you process with Scrapy.
You could instead set the `proxy` field of your request's meta attribute to `http://scraperapi.your=options:your_key@proxy-server.scraperapi.com:8001` and enable `HttpProxyMiddleware`.
This middleware's only purpose is to achieve this in a more convenient manner.

## Installation
```
$ pip install scrapy-scraperapi-middleware
```

## Settings
You need to specify your key for Scraper API in your settings.py or settings object. 

```Python
SCRAPER_API_KEY = 'your_key'
```

You also need to enable ScraperAPIMiddleware as well as Scrapy's HttpProxyMiddleware. 

```python
DOWNLOADER_MIDDLEWARES = {
    'your_project.middleware_file.ScraperAPIMiddleware': 350,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}
```

## Additional options 
Scraper API supports additional options as documented [here](https://www.scraperapi.com/documentation?fp_ref=patrick50#proxy-mode).
You could use these options by adding them to your settings as dictionary.

```python
SCRAPER_API_OPTIONS = {
    'render': 'true', 
    'country_code': 'us'
}
```

## Affiliate link
If this example is helpful to you and you do not yet have a subscription to Scraper API, consider using [my affiliate link](https://www.scraperapi.com/pricing?fp_ref=patrick50) if you plan on getting one. Be aware that all other links to ScraperAPI in this Readme are also affiliate links.


