add support for --sources-url to fetch sources from additional URLs

[joeydumont]

This PR adds a --sources-url CLI option. It is a comma separated list of URLs that EasyBuild will fetch sources from. It replaces the hard-coded EASYBUILD_SOURCES_URL, but keeps it as a default value. The --sources-url-priority option is used to determine if these URLs are to be queried first, or last, i.e. as a fallback.

Unfortunately, this breaks 3 tests related to downloading the file, and I'm unsure of exactly why. It seems that build_option('sources_url') might return None when the function is run through the test harness, rather than getting populated with the default value. I've tried adding a init_config(args=["--sources-url=https://sources.easybuild.io"]), but that didn't work either.

If the feature is of interest, could you give me a hand in fixing the tests?

EDIT: After some discussion on Slack, I was able to fix the tests. I also renamed the options to --extra-source-urls and --extra-source-urls-priority. I changed the commit mesage to be imperative.

[joeydumont]

I see there are now merge conflicts for this. Let me know when you're ready for me to rebase, or if you prefer to handle that.

[jfgrimm]

I see there are now merge conflicts for this. Let me know when you're ready for me to rebase, or if you prefer to handle that.

go for it; if you have any issues/questions just ping me

[Flamefire]

Suggestion here: --sources-url supports multiple URLs, doesn't it? How are they separated?

For EasyConfigs we have --robot-paths which accepts a list of paths like /path1:/path2 and works nicely mit the default paths if you append or prepend a colon: :/path1:/path2 appends it to the default, /path1:/path2: prepends it while the original replaces it. How about replacing the priority option by something like this to have a uniform interface here?

[joeydumont]

As suggested, I made sure that this used the same add_flex logic that --robot-paths uses. However, I think URLs should be separated by | rather than a colon (:) or a comma (,), as both are valid in URLs. I added urllist to the list of options in ExtOption to allow for this separator to be used.