tool-sync icon indicating copy to clipboard operation
tool-sync copied to clipboard

Adding async support for downloading assets

Open MitchellBerend opened this issue 2 years ago • 1 comments

As it stands now the tool downloads all tools sequentially. This can actually take a very long time if there are a lot of tools defined in the config, or if there is some type of ansible/salt-stack deployment going on that also might run sequentially. I suppose these downloads can actually be done asynchronously. There are however a number of things to consider.

  1. This would make it harder for people that are not already familiar with the code base, to jump in and contribute.
  2. This would add a dependency.
  3. The ureq library does not support async capibilities so this would need to be replaced (with reqwuest for instance).
  4. There is a rate limit of 60 calls per hour on the github api ^1.
  5. We would have to restructure the way tags are collected before the tool downloads the actual assets.

This should be weighed against the potential speed increase that can come from asynchronous downloads.

Im not sure if im missing something here so I would appreciate feedback (from all sources) on this.

MitchellBerend avatar Sep 06 '22 07:09 MitchellBerend

I was thinking about this as well. Indeed, introduction of concurrency would make the code more complex and require to change a lot of things.

I've created the following to help a bit with the speed (although, it's not clear how to do it at the moment):

  • #70

For now, it would be great to get more information about potential improvement. Say, write two shell scripts using curl to download assets sequentially and in parallel and compare the numbers. So we'll see whether it's actually worth it.

chshersh avatar Sep 07 '22 11:09 chshersh