aria2c this vastly increases the network transfer lower bounds using …

[jnorthrup]

…threaded transfers with https://aria2.github.io/

Template to be used in the PR description (remove this part when submitted the PR)

Description

iff a file xfer client named ariac2 is on the path and returns success for aria2c -v then a function in the script aliases wget to be the aria2c command for a resuming and multithreaded transfer in-place.

speaking for pacific regions and asia from experience, this is a normal enhancement to anything that touches nvidia network or openjdk.net or badly mirrored silicon valley sites.

Problem description

wget: 250K/Sec aria2c as switched: 5M/sec

Backend/s tested

Mark the backends affected by this PR.

• [x] PTX
• [x] OpenCL
• [ x ] SPIRV
• [x] Github blobs

OS tested

Mark the OS where this PR is tested.

• [*] Linux
• [x] OSx
• [x] Windows

Did you check on FPGAs?

If it is applicable, check your changes on FPGAs.

• [ ] Yes
• [*] No

How to test the new patch?

Provide instructions about how to test the new patch.

---

[jjfumero]

Optimizing the download of all packages is always good.

aria2c is not installed by default in most Linux. So this needs to be either a) Documented in the list of dependencies: - https://github.com/beehive-lab/TornadoVM/blob/master/assembly/src/docs/10_INSTALL_WITH_GRAALVM.md#installing-tornadovm-with-graalvm-2210 - https://github.com/beehive-lab/TornadoVM/blob/master/assembly/src/docs/12_INSTALL_WITH_JDK11_PLUS.md b) The auto-installer can download and compile the tool from the GitHub repo

Some questions (I am not familiarized with this tool): Is the aria2c tool multi-platform? Can it run on Windows and OSx?

[kotselidis]

Thanks for the contribution @jnorthrup we will test it internally on the various OSs we support and will update you. As Juan mentioned, we will need to update the dependencies page for that.

[jnorthrup]

Optimizing the download of all packages is always good.

aria2c is not installed by default in most Linux. So this needs to be either a) Documented in the list of dependencies: - https://github.com/beehive-lab/TornadoVM/blob/master/assembly/src/docs/10_INSTALL_WITH_GRAALVM.md#installing-tornadovm-with-graalvm-2210 - https://github.com/beehive-lab/TornadoVM/blob/master/assembly/src/docs/12_INSTALL_WITH_JDK11_PLUS.md b) The auto-installer can download and compile the tool from the GitHub repo

Some questions (I am not familiarized with this tool): Is the aria2c tool multi-platform? Can it run on Windows and OSx?

i did a project search for wget in the github source repo.

i didn't immediately turn up any such tools manifest items including wget, i would treat this identically.

The gravity of this tweak increases with network hops ultimately, so cloud and production builds can get away with almost no knowledge and will not likely possess this on an instance. tbh i have changed the wget semantics slightly to include "-c" transfer continuation from previous aborts and again, would not perceive this benefit on any cloud provider workflow but it makes a difference to edge development.

the availability is likely in OSX brew, and ubiquitous in linux distributions. the likelyhood of a posix mingw port is reasonable to undertake if not already available somewhere.

ultimately I don't know of any way this bash function will impact the existing system except if and only if aria2c is on the path so I think it is worth a mention as a coffee shop/conference wifi measure of improvement and much less a production feature.

[jnorthrup]

addendum

[jnorthrup]

woops 4-lines, didn't seem worth keeping an aging repo around for. I'm still network constrained and enthusiastic about mutlithreaded install downloads. re-opening

[jjfumero]

@jnorthrup , thanks for your interest. While we were tempted to merge it, we want to transition to a full installation/configuration implemented in Python in near future. Thus we will have the same scripts for Linux, OSx and Windows. It would be good if we have this functionality OS-agnostic. Do you have any recommendations to invoke something similar from Python and not have an OS-dependency?

[jnorthrup]

it has binaries on all three platforms, you plan to ditch wget completely?

On Tue, Oct 18, 2022 at 1:21 PM Juan Fumero @.***> wrote:

@jnorthrup https://github.com/jnorthrup , thanks for your interest. While we were tempted to merge it, we want to transition to a full installation/configuration implemented in Python in near future. Thus we will have the same scripts for Linux, OSx and Windows. It would be good if we have this functionality OS-agnostic. Do you have any recommendations to invoke something similar from Python and not have an OS-dependency?

— Reply to this email directly, view it on GitHub https://github.com/beehive-lab/TornadoVM/pull/192#issuecomment-1281829746, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAR6KQKQUIHNXTKXAI6SRLWDYXWJANCNFSM53FCZXOQ . You are receiving this because you were mentioned.Message ID: @.***>

[jnorthrup]

The intent of the patch doesn't impose on whatever naive defaults may exist. the test for the c++ multi-everything optimized downloader can fail and the python can proceed with whatever you have in mind on the low complexity budget. testing executables on the path is extremely cross-platform. the popular python script ytdl provides an aria2 option among others. I presume protocol specific parsing in python itself isn't a requirement based on the precedent of a purpose-built downloader tool.

[jjfumero]

Since we moved all installation scripts from Bash to Python, this PR is now obsolete. I will close this PR. Feel free to open new ones with new fixes/improvements using the latest version.