Metadata-Version: 2.1
Name: pyTokenizer
Version: 1.1.2
Summary: A streaming tokenizer.
Home-page: https://github.com/Paebbels/pyTokenizer
Author: Patrick Lehmann
Author-email: Paebbels@gmail.com
License: UNKNOWN
Project-URL: Documentation, https://pyTokenizer.readthedocs.io/en/latest/
Project-URL: Source Code, https://github.com/Paebbels/pyTokenizer
Project-URL: Issue Tracker, https://github.com/Paebbels/pyTokenizer/issues
Keywords: Python3 Token Tokenizer Streaming
Platform: UNKNOWN
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Utilities
Requires-Python: >=3.5
Description-Content-Type: text/markdown

![PyPI - License](https://img.shields.io/pypi/l/pyTokenizer)
![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/Paebbels/pyTokenizer) 
![GitHub release (latest by date)](https://img.shields.io/github/v/release/Paebbels/pyTokenizer)
[![Documentation Status](https://readthedocs.org/projects/pytokenizer/badge/?version=latest)](https://pyTokenizer.readthedocs.io/en/latest/?badge=latest)  
[![PyPI](https://img.shields.io/pypi/v/pyTokenizer)](https://pypi.org/project/pyTokenizer/)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyTokenizer)
![PyPI - Wheel](https://img.shields.io/pypi/wheel/pyTokenizer)
![PyPI - Status](https://img.shields.io/pypi/status/pyTokenizer)

# pyTokenizer

A streaming tokenizer.


## Contributors:

* [Patrick Lehmann](https://github.com/Paebbels) (Maintainer)


## License

This library is licensed under [Apache License 2.0](LICENSE.md)

-------------------------

SPDX-License-Identifier: Apache-2.0


