Metadata-Version: 2.1
Name: pfio
Version: 1.2.0
Summary: PFN IO library
Home-page: http://github.com/pfnet/pfio
Author: Tianqi Xu, Kota Uenishi
Author-email: tianqi@preferred.jp, kota@preferred.jp
License: UNKNOWN
Keywords: filesystem hdfs chainer development
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Build Tools
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: pyarrow (==3.0.0)
Requires-Dist: boto3
Provides-Extra: doc
Requires-Dist: sphinx ; extra == 'doc'
Requires-Dist: sphinx-rtd-theme ; extra == 'doc'
Provides-Extra: test
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: flake8 ; extra == 'test'
Requires-Dist: autopep8 ; extra == 'test'
Requires-Dist: parameterized ; extra == 'test'
Requires-Dist: isort ; extra == 'test'
Requires-Dist: moto ; extra == 'test'

## PFIO

PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports

- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.


## Dependency

- HDFS client and libhdfs for HDFS access
- CPython >= 3.6

## Installation and Document build

Installation

```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```

Documentation
```sh
$ cd pfio/docs
$ make html
$ open build/html/index.html
```

Test
```sh
$ cd pfio
$ pip install .[test]
$ pytest tests/
```

## How to use

Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.


