Fix: official repo to list Solidity's releases

[Neurone]

Added the official repo for Solidity's Compiler releases, before the Truffle custom one. Added a note for the last two deprecated repo.

[Neurone]

This PR fixes problems you can encounter if the truffle relay node is down for some reason. See this StackExchange question: https://ethereum.stackexchange.com/questions/125523/truffle-compile-list-not-showing-latest-versions-of-compiler

[gnidan]

Looks like our relay was pointing to the deprecated URLs. Fixing that now!

I think for this PR, though, we want to keep relay.trufflesuite.com as the primary source, since that will allow us to fix issues without requiring a Truffle release.

[gnidan]

Thanks for bringing this to our attention! Will discuss with the team to confirm the behavior we want moving forward.

[haltman-at]

Note: This is related to issue #4425.

[Neurone]

@gnidan I set that as the first repo because I found this note in the comment: this relay address exists so that we have a backup option in case more official distribution mechanisms fail, but I see your point :)

[cds-amal]

This precedence and motivation, should probably be documented

[eggplantzzz]

@cameel can you let us know which urls that Truffle should use when obtaining the solc compiler? Is https://binaries.soliditylang.org/bin/ correct or should we be using a different one?

[cameel]

My recommendation would be https://binaries.soliditylang.org/emscripten-wasm32/ with a fallback to https://binaries.soliditylang.org/emscripten-asmjs/.

Overall, this should be clear from Static Binaries section in the docs. Let me know if you think it still leaves something out.

[eggplantzzz]

Thanks @cameel! We'll make sure this gets updated in Truffle!

[eggplantzzz]

@Neurone are you interested in making these changes? First, should be our relay url followed by the two that @cameel recommends. If you don't want to do this work I'll be happy to take it from here, just let me know!

[Neurone]

@eggplantzzz sure, no problem, I will send an update to the PR soon.

[MicaiahReid]

Hi @Neurone, is this PR ready for review? I see that you've made some commits, but it isn't currently passing CI. Let us know if you are able to look into what's causing this to fail.

[davidmurdoch]

Should we keep https://solc-bin.ethereum.org/bin/ and https://ethereum.github.io/solc-bin/bin/ as backups for the backups?

[Neurone]

Hi, yes, it's ready.

I didn't change anything aside from the URLs in the arrays. The error is related to the content of the loaded list: the CI tries to find a nightly build in the list (0.4.16-nightly.2017.8.9+commit.81887bc7), but new URLs contain the release one only (0.4.16+commit.d7661dd9).

If you agree, I can change the test here https://github.com/trufflesuite/truffle/blob/3acd56290e1019ca5e3a739e3d3a597ff00a8fdc/packages/compile-solidity/test/test_supplier.js#L100 to use version 0.4.16+commit.d7661dd9

[cameel]

Should we keep https://solc-bin.ethereum.org/bin/ and https://ethereum.github.io/solc-bin/bin/ as backups for the backups?

It would not hurt. At least for https://ethereum.github.io. It no longer receives new versions but it's a completely different host so in case S3 goes down, you might still be able to get binaries for some older versions from there. https://solc-bin.ethereum.org on the other hand is pointing at the same S3 bucket as https://binaries.soliditylang.org and I don't expect that to change ever so that one's probably pointless. At most it might help if there's an issue with DNS or certificates so that one resolves and the other does not.

[cameel]

The error is related to the content of the loaded list: the CI tries to find a nightly build in the list (0.4.16-nightly.2017.8.9+commit.81887bc7),

Right. That's actually one difference between the new and old URLs - the nightlies are only available in bin/. Do you think we should add them to the new dirs too? I'd expect tools to be handing nightlies separately anyway.

[Neurone]

The error is related to the content of the loaded list: the CI tries to find a nightly build in the list (0.4.16-nightly.2017.8.9+commit.81887bc7),

Right. That's actually one difference between the new and old URLs - the nightlies are only available in bin/. Do you think we should add them to the new dirs too? I'd expect tools to be handing nightlies separately anyway.

I think it's the cleanest path to follow. But currently, both relays redirect to https://binaries.soliditylang.org, so if this is the way to go, we should convince the Solidity team to update their files.

[cameel]

Personally, I really don't like the fact that nightlies sit in the same dir as releases since there are tons of them. If you go to the github repo backing the S3 bucket, github won't even show you all the releases because it hits the limit of 1000 files.

But if you think that it's still better to have them in the new dirs, this is something we can definitely change. Could you submit an issue in the Solidity repo?

[Neurone]

Personally, I really don't like the fact that nightlies sit in the same dir as releases since there are tons of them. If you go to the github repo backing the S3 bucket, github won't even show you all the releases because it hits the limit of 1000 files.

But if you think that it's still better to have them in the new dirs, this is something we can definitely change. Could you submit an issue in the Solidity repo?

Sure, no problem. Another solution is to add a separate file filled only with release versions. This way, we have a clean list (list-releases.json) and a complete one (list.json). I'll prepare an issue for the Solidity repo soon.

[cameel]

For me it's only really an issue when browsing the dir manually so I don't think splitting the list would be a solution. When I do that I don't even look at the list :)

[Neurone]

I see :) I cannot list dir contents from the outside, so I didn't notice that behavior.

If you don't mind having duplicated files, another option is to use the main folders for release versions only and a specific subfolder for all versions, i.e. /emscripten-wasm32/ only releases, /emscripten-wasm32/full-list/ all releases.

[Neurone]

@cameel I cannot open issues on https://github.com/ethereum/solc-bin, so I opened it on https://github.com/ethereum/solidity

[cameel]

I see :) I cannot list dir contents from the outside, so I didn't notice that behavior.

See https://github.com/ethereum/solc-bin/tree/gh-pages/bin. It's the repo backing the S3 bucket. The fact that there are so many files in there makes it kinda hard to browse that dir.

I cannot open issues on https://github.com/ethereum/solc-bin, so I opened it on https://github.com/ethereum/solidity

That's fine. solc-bin does not have issues exactly because we want to have them all in one place - in the main repo, Maybe we should just put that in the README.

[kevinweaver]

@neurone it looks like this still isn't passing CI, we'll restart it to see if it's flaky. Otherwise, let us know if we can help resolve the issue.

[Neurone]

@kevinweaver we need to keep this PR pending until the Solidity team reaches an agreement about how to update the management of release and nightly builds (see here). When that thread is closed, I can update this PR accordingly.

[cds-amal]

Maintenance note: leaving this in waiting for more visibility

[eggplantzzz]

So it looks like axios throws an error and we retry the next url in the case that the server response is something other than 2xx. See handling errors. This is relevant because it seems like not all urls formats will have all versions of the Solidity compiler.

However a trickier thing is that in order to build the urls we need the version commit suffix to add to the base urls we have. We get those by querying the url to see what versions are available. We need to make sure, when this gets implemented, we have a source to query for all available versions of the Solidity compiler.

[eggplantzzz]

So additionally I found the following interesting things about these links:

• the solc-bin.ethereum link seems to have the complete list of compiler versions with the newest one being up-to-date as of the time of writing this (note that Truffle probably cannot handle versions of the compiler before a certain point - probably around 0.4.x for some x)
• the ethereum.github.io link only has versions of the compiler up to 0.7.1
• the wasm links have versions starting with 0.3.6
• the asmjs links only go up to version 0.6.1

[haltman-at]

So it looks like axios throws an error and we retry the next url in the case that the server response is something other than 2xx. See handling errors. This is relevant because it seems like not all urls formats will have all versions of the Solidity compiler.

Hm, so since it errors on a 404, that means that actually it's OK to use incomplete lists, right? Since if it doesn't find it on one, it'll just go to the next! So the case I was worried about (that it finds it's not on one list and then gives up) doesn't seem to actually be a concern, right?

[davidmurdoch]

@eggplantzzz is this a good summary: we're okay with the changes as proposed, but CI is failing, so those failures need to be investigated and resolved before we can merge this?