Metadata-Version: 2.1
Name: robotsparse
Version: 0.1
Summary: A python package that enhances speed and simplicity of parsing robots files.
Home-page: 
Author: xyzpw
Maintainer: xyzpw
License: MIT
Keywords: parsing,parser,robots,web-crawling,crawlers,crawling
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Intended Audience :: Developers
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests==2.*

# robotsparse
A python package that enhances speed and simplicity of parsing robots files.

## Usage
Basic usage, such as getting robots contents:
```python
import robotsparse

#NOTE: The `find_url` parameter will redirect the url to the default robots location.
robots = robotsparse.urlRobots("https://github.com/", find_url=True)
print(list(robots)) # output: ['user-agents']
```
The `user-agents` key will contain each user-agent found in the robots file contents along with information associated with them.<br>

Alternatively, we can assign the robots contents as an object, which allows faster accessability:
```python
import robotsparse

# This function returns a class.
robots = robotsparse.getRobots("https://duckduckgo.com/", find_url=True)
assert isinstance(robots, object)
print(robots.allow) # Prints allowed locations
print(robots.disallow) # Prints disallowed locations
print(robots.crawl_delay) # Prints found crawl-delays
print(robots.robots) # This output is equivalent to the above example
```

