Add performance benchmarks to the readme

[Archmonger]

trafficstars

Right now the performance difference between stdlib files, aiofiles, and aiofile is undocumented.

There should be at least two benchmarks:

1. Show the performance during single-threaded operations (single while loop where stdlib files is likely to be faster).
2. Show the performance improvement when using aiofile during high IO concurrency scenarios.

[mosquito]

The performance actually depends on what underlaying implementation of caio is currently used in the benchmark environment. The linux_aio-based implementation, of course, only for linux this have a better performance. The thread_aio and python_aio implementations should not show much improvement over aiofiles in the general case, but should be slightly better on POSIX-compatible OSes.

I have no ideas how to make this benchmark fair enough, if you have any share it with me, preferably the code.

[Archmonger]

For environment specific libraries like this one, I recommend re-running tests per environment. It would likely require a fairly large text matrix though.

For example, for ReactPy we run each part of our code through IOPS tests. The concept is the same, although we focus on network IO which means we need a live web browser.

Since 99% percent of users only care about read/write performance, your test result matrix might look something like this:

---

Read

This test repeatedly reads 5MB of binary data from a file, and measures how many reads occurred within 60 seconds (higher is better).

	1 Worker	500 Workers	1 Worker (Linux)	500 Workers (Linux)
aiofile	x	x	x	x
aiofiles	x	x	x	x
stdlib	x	x	x	x

Write

This test repeatedly writes 5MB of binary data to a file, and measures how many writes occurred within 60 seconds (higher is better).

	1 Worker	500 Workers	1 Worker (Linux)	500 Workers (Linux)
aiofile	x	x	x	x
aiofiles	x	x	x	x
stdlib	x	x	x	x

---

Since I probably don't need to show you how to create the test cases themselves, I'll give some top-level tips.

I recommend reading/writing binary data in order to minimize variables. Also, you should use asyncio.gather to spawn workers. For example...

import asyncio

async def read_binary_test():
  ...

async def main():
  tasks = [read_test() for _ in range(500)]
  await asyncio.gather(*tasks)

if __name__ == "__main__":
  asyncio.run(main())

The write test should make sure to store BINARY_DATA_5MB as a global to avoid reconstructing it every time, which would impact test results

I arbitrarily chose 5MB as a value that gives Python enough "downtime" to actually do something else while IO is being executed.

The test cases should be written to stop recording results after 60 seconds has elapsed.

[rooterkyberian]

This would be quite valuable. When starting new project I'm not even sure if I should bother with asyncio .

According to https://github.com/mosquito/aiofile/issues/18#issuecomment-497983435 performance before new implementation was atrocious (i.e. you will be better of having blocking stdlib write/reads most of the time).

https://github.com/mosquito/aiofile/issues/18#issuecomment-648102906 looks like a huge improvement have been achieved, but does not compare against stdlib, which is quite crucial.

As for which interface need the benchmark the most - even doing it for just for linux would be good enough, since most of the applications in which performance is critical, tend to be hosted on servers, which tend to be Linux-based.

I tried rerunning test from the aforementioned comment, but the results look bad. I hope this benchmark is somewhat botched ( I did not try to debug it, just changed it a little to produce table automatically): https://gist.github.com/rooterkyberian/a2c12fc6269c86bcf4e199149eb6b9ec .

Results I got:

Python version: 3.11.6
Platform: Linux-6.5.0-25-generic-x86_64-with-glibc2.35
aiofiles version: 23.2.1
aiofile version: 3.8.8
uvloop version: 0.19.0
aiofile default context <caio.linux_aio_asyncio.AsyncioContext object at 0x7e2eb3ac0f50>

iterations	sync	async executor 'dumb'	async executor w/ coroutines	async multiple executors	async aiofiles	async aiofile	aiofiles@uvloop	aiofile@uvloop
1000	0.006	0.01	0.007	0.01	0.748	0.303	0.455	0.293
10000	0.051	0.059	0.059	0.061	7.558	3.09	4.653	2.709
100000	0.481	0.49	0.666	0.518	74.853	190.855	47.254	185.326
1e+06	4.676	4.651	4.85	5.199	746.184

I left it running more than an hour after last result and still nothing, so it seems like either benchmarking script is broken or aiofile (caio backend) is.

[HaukurPall]

Thanks for this script. I ran it to test my fix in #93 and here are my results:

iterations	sync	async executor 'dumb'	async executor w/ coroutines	async multiple executors	async aiofiles	async aiofile	aiofiles@uvloop	aiofile@uvloop
1000	0.012	0.013	0.013	0.021	1.136	0.208	0.733	0.166
10000	0.11	0.107	0.07	0.074	9.942	1.122	3.372	0.755
100000	0.577	0.577	0.582	0.676	107.671	9.06	50.216	7.583

It seems to improve the performance quite a bit (and beat aiofiles!)

[Archmonger]

Perhaps I'm reading the results wrong, but it looks like the results suggest that using standard lib with asyncio.run_in_executor is more performant that both aiofiles and aiofile?

[Archmonger]

Okay I am seeing an issue with the testing methodology.

Fundamentally, all Python async file frameworks will perform slower for small reads/writes. This is because they add multiple additional layers of stack complexity compared to standard lib.

The only real benefit of aiofiles and aiofile is the ability to perform long-running IO independently of the Python interpreter. This allows the Python interpreter do other operations in the "IO downtime". So reading/writing four bytes (python integer) is far too small of an operation to actually gain any benefit. As seen above, a mixture of stack trace overhead and context-switching will compound run times logarithmically.

The tests would need to be modified to read/write something much larger, like a 5MB text file.

[HaukurPall]

I think you are mostly right @Archmonger.

I have been doing some reading on the subject of async file reading on Linux via Python and would like to add to your point. From what I've learnt; The most "accepted/performant/non-blocking" async file I/O subsystems in Linux are the Linux AIO and the (more recent) IO-uring. This package implements an interface for Linux AIO. Both of them are setup around a queue and can service multiple requests at the same time, f.ex. the client can batch multiple requests at once, for different files etc. Setting up such a capable system for a single file read, is an overkill. Furthermore, to add a nice async/await interface for Python (mapping callbacks to futures), considerable amount of Python code is required. Going from a simple open/iterate over file object implemented in C to a multilayered Python code will always be slower.

The performance tests in the link above is not comparing the async capabilities equally. The "async *executor*" tests make the full file read async, not each individual read operation like this package does. New tests should be added to run each open() and readline() on a worker thread to have equivalent async granularity.