Metadata-Version: 2.1
Name: robotspy
Version: 0.1.0
Summary: Robots Exclusion File protocol parser
Home-page: https://github.com/andreburgaud/robotspy
Author: Andre Burgaud
Author-email: andre.burgaud@gmail.com
License: MIT
Platform: UNKNOWN
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Description-Content-Type: text/markdown

# Robots Module

The `robots` module can be used as a substitute for [`urllib.robotparser`](https://docs.python.org/3/library/urllib.robotparser.html) available in the Python
standard library. The API is the same to allow for backward compatibility.

The main reason for this rewrite are the following:

1. This was initially intended to experiment with parsing `robots.txt` for a link checker project.
1. The implementation is attempting to follow the latest internet draft [Robots Exclusion Protocol](https://tools.ietf.org/html/draft-koster-rep-00).
1. It tries to be compliant with some rules not in the specs but commonly accepted like the ones in
[`urllib.robotparser`](https://docs.python.org/3/library/urllib.robotparser.html), examples: 
`Sitemap`, `Request-rate`, `Crawl-delay`...
1. It also includes the same tests as the [Google Robots.txt Parser](https://github.com/google/robotstxt),
except for some specific behavior specific to Google. 

## Installation

**Note**: Python 3.8.x is required

You preferably want to install the `robots` package after creating a Python virtual environment,
in a newly created directory, as follows:

```
$ mkdir project && cd project
$ python3 -m pip install robotspy 
```

## Usage

The `robots` package can be imported as a module and is also executable invokable with `python -m`.

### Execute the Package

After installing `robotspy`, you can validate the installation by running the following command:

```
$ python -m robots --help
usage: robots (<robots_path>|<robots_url>) <user_agent> <URI>

Shows whether the given user agent and URI combination is allowed or
disallowed by the given robots.txt file.

positional arguments:
  robotstxt      robots.txt file path or URL
  useragent      User agent name
  uri            Path or URI

optional arguments:
  -h, --help     show this help message and exit
  -v, --version  show program's version number and exit
```

A concrete example to check against http://www.pythontest.net/elsewhere/robots.txt if the user agent
`Nutch` can fetch the path `/brian/` would be done as follows:

```
$ python -m robots http://www.pythontest.net/elsewhere/robots.txt Nutch /brian/
user-agent 'Nutch' with URI '/brian/': ALLOWED
```

### Use the Module in a Project

Here is an example with the same data as above, using the `robots` package from the Python shell:

```
>>> import robots
>>> parser = robots.RobotFileParser('http://www.pythontest.net/elsewhere/robots.txt')
>>> parser.read()
>>> useragent = 'Nutch'
>>> path = '/brian/'
>>> result = parser.can_fetch(useragent, path)
>>> print(f"Can {useragent} fetch {path}? {result}")
Can Nutch fetch /brian/? True
>>>
```

## Development

The main development dependency is `pytest` for executing the tests. It is automatically
installed if you perform the following steps:

```
$ git clone https://github.com/andreburgaud/robotspy
$ cd robotspy
$ python -m venv .venv --prompt robotspy
$ . .venv/bin/activate
(robotspy) $ python -m pip install -r requirements.txt
(robotspy) $ python -m pip install -e .
(robotspy) $ make test
(robotspy) $ deactivate
$
```

Other dependencies are intended for deployment to the **Cheese Shop** (PyPI):

* wheel
* twine

## Release History

* 0.1.0: Initial release

## License

MIT License

