High CPU usage when downloading

[pinnaculum]

I really like this project, great work :+1:

I've noticed that the CPU usage when downloading (even for a single download) can be pretty high. I've profiled it a bit with yappi, and some of the top methods for total CPU time spent seem to be:

• PeerManager._execute_peer_client
• Downloader._execute_block_requests
• Downloader._request_blocks
• PeerTCPClient.run

Adding asyncio.sleep() calls in those methods doesn't really affect performance.

One thing that lowered CPU usage is to add asyncio.sleep() calls after _handle_block() calls, since this method writes blocks to disk via a threadpool and is often called.

[borzunov]

I am happy to learn that you've found my project useful and made it a part of your new project :)

Thank you for the efforts to notice this issue and find its probable causes!

Unfortunately, I don't have time to experiment with such changes and do proper measurements right now, but I would be glad to accept a PR if you report that it improves the CPU usage and does not significantly affect the download speed :)

[pinnaculum]

Hey @borzunov,

Yeah it's been integrated in galacteek, and you can move torrents to IPFS. Initially i used it as a dependency but i ended up integrating the code in the repo. bit-torrent's code is under MIT license, i've put the LICENSE file in the galacteek.torrent python package and referenced your name and the software license in the README, is it ok or should i add something else ?

From the profiling i've done, the performance improvement is noticeable (especially with multiple torrents) but there are other functions/coroutines to optimize. This does not affect download speed. I've worked on a modified version of FileStructure that uses aiofile for file operations, this would be much more efficient at least on linux because it uses libaio. I'll make a PR when i can.

[borzunov]

is it ok or should i add something else ?

That's enough, thank you :)

this would be much more efficient at least on linux because it uses libaio. I'll make a PR when i can.

That would be great, thanks a lot!