Metadata-Version: 2.1
Name: linux-aio
Version: 0.4.1
Summary: Linux aio ABI wrapper
Home-page: https://github.com/isac322/linux_aio
Author: Byeonghoon Yoo
Author-email: bh322yoo@gmail.com
License: LGPLv3+
Keywords: Linux AIO ABI API wrapper
Platform: Linux
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: Implementation :: Stackless
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System
Classifier: Topic :: System :: Filesystems
Classifier: Topic :: System :: Operating System
Classifier: Topic :: System :: Operating System Kernels
Classifier: Topic :: System :: Operating System Kernels :: Linux
Provides: linux_aio
Requires-Python: >=3.4
Description-Content-Type: text/markdown
Requires-Dist: linux-aio-bind

**\[english\]** | [\[한국어 (korean)\]](https://github.com/isac322/linux_aio/blob/master/README.kor.md)

# linux_aio: Python wrapper for [Linux Kernel AIO](http://lse.sourceforge.net/io/aio.html)

[![](https://img.shields.io/travis/com/isac322/linux_aio.svg?style=flat-square)](https://travis-ci.com/isac322/linux_aio)
[![](https://img.shields.io/pypi/v/linux_aio.svg?style=flat-square)](https://pypi.org/project/linux-aio/)
[![](https://img.shields.io/codecov/c/github/isac322/linux_aio.svg?style=flat-square)](https://codecov.io/gh/isac322/linux_aio)
[![](https://img.shields.io/pypi/implementation/linux_aio.svg?style=flat-square)](https://pypi.org/project/linux-aio/)
[![](https://img.shields.io/pypi/pyversions/linux_aio.svg?style=flat-square)](https://pypi.org/project/linux-aio/)
[![](https://img.shields.io/pypi/wheel/linux_aio.svg?style=flat-square)](https://pypi.org/project/linux-aio/)
[![](https://img.shields.io/pypi/l/linux_aio.svg?style=flat-square)](https://pypi.org/project/linux-aio/)
[![Say Thanks!](https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg?style=flat-square)](https://saythanks.io/to/isac322)

Python wrapper module that uses Linux Kernel AIO directly


## What is Linux Kernel AIO?

[Linux IO Models table](https://oxnz.github.io/2016/10/13/linux-aio/#io-models)

In summary, it allows non-blocking and asynchronous use of blocking IO operations such as [read(2)](http://man7.org/linux/man-pages/man2/read.2.html) and [write(2)](http://man7.org/linux/man-pages/man2/write.2.html).


### Related documents

- [Linux Asynchronous I/O](https://oxnz.github.io/2016/10/13/linux-aio/)
- [Linux Kernel AIO Design Notes](http://lse.sourceforge.net/io/aionotes.txt)
- [How to use the Linux AIO feature](https://github.com/littledan/linux-aio) (in C)


### **It is different from [POSIX AIO](http://man7.org/linux/man-pages/man7/aio.7.html)**

The POSIX AIO APIs have the `aio_` prefix, but the Linux Kernel AIO has the `io_` prefix.

There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally.
So, as you can see from the experiment below, it's much worse than using the blocking IO API.


## Implementation & Structure

### Package `linux_aio`

- Implemented based on [linux_aio_bind](https://pypi.org/project/linux-aio-bind) package which is low-level binding of Linux kernel AIO.
- Unlike [linux_aio_bind](https://pypi.org/project/linux-aio-bind), it can be used without knowledge of `ctypes`
- Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test).


## Example

Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test).


## Notes & Limits

- Obviously available only on Linux
- Because it is a wrapper, it brings the constraints of Linux.
	- It can not be used for files used as a kernel interface. (e.g. `cgroup`)
	- [Sometimes it works as Blocking.](https://stackoverflow.com/questions/34572559/asynchronous-io-io-submit-latency-in-ubuntu-linux)
		- There are some things that have been solved through development after posting.
	- Some features are being added because they are still under development.
	- There are also some features that are not supported when the Linux version is low
		- You need to check [Linux man pages (4.16)](http://man7.org/linux/man-pages/man2/io_submit.2.html) and its related API documentation


## Evaluation

[Experiment script](https://gist.github.com/isac322/8606f5c464fa390cb88b47354981cdab) (requires python 3.7)

### Setup

- Distribution: Ubuntu Server 16.04.5 LTS
- Linux: 4.19.0
- CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
- MEM: total 64GB
- Storage: SK hynix SC300B SATA 512GB
- Python: 3.7.2 ([Ubuntu ppa](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa))


- Attempts to read a total of `1000` times in` 1ms` intervals.
- The file size varies from `1KB` to` 100KB`, but it is small.
- Experiment with increasing the number of files read at the same time
- Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop.


### Comparison target

- [aiofiles](https://pypi.org/project/aiofiles/) - Uses Thread pool
- [aiofile](https://pypi.org/project/aiofile/) - Uses POSIX AIO
- [libaio](https://pypi.org/project/libaio/) - Uses [libaio](http://lse.sourceforge.net/io/aio.html)
- [python built-in open()](https://docs.python.org/3/library/functions.html#open)


**It is not a perfectly fair comparison.**

`aiofiles` and `aiofile` are libraries that support [asyncio](https://docs.python.org/3/library/asyncio.html). Since `open()` is blocking, there is a disadvantage that you can not do any other work while IO is going on. `libaio` and `linux_aio` are non-blocking, but must be polled.


### Results

**It may differ from environment to environment.**

#### Runtime

- Unit: second

| # of files 	|   1   	|   6   	|   12  	|   24  	|
|:---------:	|:-----:	|:-----:	|:-----:	|:-----:	|
|  aiofiles 	| 1.681 	| 3.318 	| 5.354 	| 9.768 	|
|  aiofile  	| 1.543 	| 1.958 	| 2.493 	| 3.737 	|
|   libaio  	| 1.311 	| 1.344 	| 1.362 	| 1.423 	|
|   open()  	| 1.252 	| 1.322 	| 1.375 	| 1.481 	|
| linux_aio 	| 1.305 	| 1.327 	| 1.353 	| 1.431 	|

#### Threads

| # of files 	|  1  	|  6  	|  12 	|  24 	|
|:---------:	|:---:	|:---:	|:---:	|:---:	|
|  aiofiles 	| 321 	| 321 	| 321 	| 321 	|
|  aiofile  	|   3 	|   8 	|  15 	|  26 	|
|   libaio  	|   1 	|   1 	|   1 	|   1 	|
|   open()  	|   1 	|   1 	|   1 	|   1 	|
| linux_aio 	|   1 	|   1 	|   1 	|   1 	|

#### Memory

- Physical memory (Virtual memory)

| # of files 	|       1       	|       6       	|       12      	|       24      	|
|:---------:	|:--------------	|:--------------	|:--------------	|:--------------	|
|  aiofiles 	| 21MB (22.6GB) 	| 21MB (22.6GB) 	| 21MB (22.6GB) 	| 21MB (22.6GB) 	|
|  aiofile  	|  17MB (258MB) 	|  17MB (654MB) 	| 17MB (1080MB) 	| 18MB (1949MB) 	|
|   libaio  	|   17MB (76MB) 	|   17MB (76MB) 	|   17MB (76MB) 	|   17MB (76MB) 	|
|   open()  	|   17MB (76MB) 	|   17MB (76MB) 	|   17MB (76MB) 	|   17MB (76MB) 	|
| linux_aio 	|   17MB (76MB) 	|   17MB (76MB) 	|   17MB (76MB) 	|   17MB (76MB) 	|

#### CPU Utilization

| # of files 	|   1   	|   6   	|   12   	|   24   	|
|:---------:	|------:	|------:	|-------:	|-------:	|
|  aiofiles 	| 42.8% 	| 85.0% 	| 102.2% 	| 113.2% 	|
|  aiofile  	| 31.4% 	| 52.4% 	|  67.0% 	|  84.0% 	|
|   libaio  	| 14.0% 	| 16.0% 	|  17.2% 	|  20.6% 	|
|   open()  	| 13.4% 	| 17.6% 	|  21.0% 	|  26.2% 	|
| linux_aio 	| 13.0% 	| 15.0% 	|  16.0% 	|  21.0% 	|

