Metadata-Version: 2.1
Name: count-tokens
Version: 0.1.0
Summary: count number of tokens in the text file using toktoken tokenizer from OpenAI
License: MIT
Author: Krystian Safjan
Author-email: ksafjan@gmail.com
Requires-Python: >=3.9,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: tiktoken (>=0.4.0,<0.5.0)
Description-Content-Type: text/markdown

# Count tokens

Simple tools that have one purpose - count tokens in a text file.

## Requirements

This package is using [tiktoken](https://github.com/openai/tiktoken) library for tokenization.

```shell

## Installation
For usage from comman line install the package in isolated environement with pipx:

```sh
$ pipx install count_tokens
```

or install it in your current environment with pip.


## Usage
Open terminal and run:

```shell
$ count-tokens document.txt
``` 

You should see something like this:

```shell
File: document.txt
Encoding: cl100k_base
Number of tokens: 67
```

if you want to see just the tokens count run:

```shell
$ count-tokens document.txt --quiet
```
and the output will be:

```shell
67
```

## Related Projects
- [tiktoken](https://github.com/openai/tiktoken) - tokenization library used by this package

## Credits

Thanks to the authors of the tiktoken library for open sourcing their work.

## License

[MIT](https://izikeros.mit-license.org/) © [Krystian Safjan](https://safjan.com).

