userland-fetch: improvements for downloading and hashing

[dbh201]

HASH_DIR support: search for an envvar HASH_DIR, a hashes/ directory in the directory of --file, or a hashes/ in the current directory. If found, look inside the files in that directory for checksums in the format outputted by sha256sum. Checksum files will be used if --hash is not provided.

partial downloads: any cancelled download leaves a .part file, which can be continued later if the server supports it or will be overwritten if not.

retrying: if there is a retryable network error, userland-fetch will try to continue the download once or twice before quitting.

security tests: fail on unsecured, unhashed download with no signature.

Removed deprecated splithost and splittype commands.

[dbh201]

Tested partial downloads, HASH_DIR, and unsecured downloading here.

[AndWac]

developer/cmocka has a problem with your userland-fetch: /export/home/andreas/oi-userland/tools/userland-fetch --file /Projekte/OpenIndiana/userland-sources/cmocka-cmocka-1.1.5.tar.gz --url https://gitlab.com/cmocka/cmocka/-/archive/cmocka-1.1.5/cmocka-cmocka-1.1.5.tar.gz --hash sha256:51eba78277d51f0299617bedffc388b2b4ea478f5cc9876cc2544dae79638cb0
INFO: not searching /Projekte/OpenIndiana/userland-sources/hashes for cmocka-cmocka-1.1.5.tar.gz (overridden with --hash) Source /Projekte/OpenIndiana/userland-sources/cmocka-cmocka-1.1.5.tar.gz... not found, skipping file copy Source https://gitlab.com/cmocka/cmocka/-/archive/cmocka-1.1.5/cmocka-cmocka-1.1.5.tar.gz... downloading... Traceback (most recent call last): File "/export/home/andreas/oi-userland/tools/userland-fetch", line 605, in main() File "/export/home/andreas/oi-userland/tools/userland-fetch", line 524, in main for name in download_from_paths(search_list, file_arg, url_arg, link_arg): File "/export/home/andreas/oi-userland/tools/userland-fetch", line 403, in download_from_paths name = download(url, file_arg, quiet,(scheme != 'ftp')) File "/export/home/andreas/oi-userland/tools/userland-fetch", line 208, in download length = int(i.headers['content-length']) TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' gmake: *** [/export/home/andreas/oi-userland/make-rules/prep-download.mk:82: /Projekte/OpenIndiana/userland-sources/cmocka-cmocka-1.1.5.tar.gz] Error 1

[AndWac]

Give me a hint when you are done and I can merge this.

[dbh201]

I still need to write docs for it, but all of the functionality seems to work.

[dbh201]

I'm going to prep the component dir overnight and see if anything extra fails.

[AndWac]

What is the state of this PR? Shall I merge #9269 first and you'll update yours then?

[dbh201]

Sorry about the delay. Yes, merge #9269 and I will rebase my branch.

[dbh201]

I'm going to see if I can finish testing in about a week (including making some changes to prep-*.mk). If you've taken a look and see anything you'd like changed or improved, let me know.

The last feature I'm putting in is to have userland-fetch grab hash or signature files automatically using -g and -G, respectively. After that we should be able to let the build system validate all archives without copy/pasting hashes.

[dbh201]

I'll need to do one more pass of the components dir to check for issues before this can be merged.

[dbh201]

This should be ready now. gmake prep downloaded, unpacked and patched most of the components, and I've fixed any errors related to userland-fetch. I also wrote a quick test script for the hashfile features, which are all working as intended.

None of the -gGnNc functionality is used by default, so it shouldn't disturb anything.