devcontainer for forest

[aatifsyed]

Oh, and let's reset the yarn.lock file.

Originally posted by @lemmih in https://github.com/ChainSafe/forest/pull/2821#pullrequestreview-1412302970

Some users can't run yarn install --immutable && yarn run prettier **/*.md --write without dirtying their lockfile.

Find a solution

[aatifsyed]

hit this again in a branch off 8f227579161c3eb9e3bde8ce84d96ab6751d9ccf

[aatifsyed]

hit a similar issue with cargo-spellcheck on #2857 - CI errors didn't repro locally (maybe different versions?)

[aatifsyed]

References: https://github.com/Jille/dockpin https://docs.earthly.dev/ https://nixos.org/ https://github.com/nektos/act

[aatifsyed]

Should also address rate limiting/other ci nonsense like

yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.npmjs.org/@zkochan/js-yaml/-/js-yaml-0.0.[6](https://github.com/ChainSafe/forest/actions/runs/4979290978/jobs/8910701002?pr=2846#step:3:7).tgz: Request failed \"404 Not Found\"".
info If you think this is a bug, please open a bug report with the information provided in "/home/runner/work/forest/forest/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

https://github.com/ChainSafe/forest/actions/runs/4979290978/jobs/8910701002?pr=2846

Or

Err:1 http://azure.archive.ubuntu.com/ubuntu jammy/universe amd64 clang amd64 1:14.0-55~exp2
  503  Service Unavailable [IP: 52.147.219.192 80]
Err:2 http://azure.archive.ubuntu.com/ubuntu jammy/main amd64 libjsoncpp25 amd64 1.9.5-3
  503  Service Unavailable [IP: 52.147.219.192 80]
Err:3 http://azure.archive.ubuntu.com/ubuntu jammy/main amd64 librhash0 amd64 1.4.2-1ubuntu1
  503  Service Unavailable [IP: 52.147.219.1[92](https://github.com/ChainSafe/forest/actions/runs/4979933861/jobs/8912177461?pr=2857#step:6:93) 80]
Err:4 http://azure.archive.ubuntu.com/ubuntu jammy/main amd64 dh-elpa-helper all 2.0.9ubuntu1
  503  Service Unavailable [IP: 52.147.219.192 80]
Err:5 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cmake-data all 3.22.1-1ubuntu1.22.04.1
  503  Service Unavailable [IP: 52.147.219.192 80]
Err:6 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cmake amd64 3.22.1-1ubuntu1.22.04.1
  503  Service Unavailable [IP: 52.147.219.192 80]
E: Failed to fetch [http://azure.archive.ubuntu.com/ubuntu/pool/universe/l/llvm-defaults/clang_14.0-55%7eexp2_amd64.deb](http://azure.archive.ubuntu.com/ubuntu/pool/universe/l/llvm-defaults/clang_14.0-55~exp2_amd64.deb)  503  Service Unavailable [IP: 52.147.219.192 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/libj/libjsoncpp/libjsoncpp25_1.9.5-3_amd64.deb  503  Service Unavailable [IP: 52.147.219.192 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/r/rhash/librhash0_1.4.2-1ubuntu1_amd64.deb  503  Service Unavailable [IP: 52.147.219.192 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/d/dh-elpa/dh-elpa-helper_2.0.9ubuntu1_all.deb  503  Service Unavailable [IP: 52.147.219.192 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/c/cmake/cmake-data_3.22.1-1ubuntu1.22.04.1_all.deb  503  Service Unavailable [IP: 52.147.219.192 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/c/cmake/cmake_3.22.1-1ubuntu1.22.04.1_amd64.deb  503  Service Unavailable [IP: 52.147.219.192 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

https://github.com/ChainSafe/forest/actions/runs/4979933861/jobs/8912177461?pr=2857

[LesnyRumcajs]

The simplest solution would be to create a Docker image with all the tools installed and host it on GHCR.

[aatifsyed]

If we were to do this, I'd be worried about a couple of things:

• multiarch can be a bit finicky
• non-reproducible builds (e.g ci uses forest-devcontainer:1.0.1) but when we git checkout 1.0.1 && docker build we get an error :/
  • What can we lock?
    • apt packages
    • npm packages
    • rust version?
• we'd essentially forgo github actions integration - part of the point of a PR like this would be to be able to run the same actions locally as in CI. Failure here looks like:
  • running actions/check-foo and docker run ... check foo have different outputs
• update process
  • taking new rustc
  • that yarn issue strikes, and starts causing our CI to fail. How painful will this be?
• cross repo responsibilities:
  • suppose I bump rustc version. I don't really want to have to jump to another repo if I can just change cargo.toml
  • we could keep a bunch of binary deps locked in forest using lockfiles and something like cargo bin

[LesnyRumcajs]

I don't really see an issue. We could lock the tooling image tag in the repo with a lockfile. Not sure about GH Actions integration, it's just running a docker command so it's doable, no? Multiarch may be painful but we were able to provide multiarch images for Forest so unless there's some strange binary we manually download, we should be good there as well.

I'm open to other suggestions.

[aatifsyed]

(I've fleshed out some comments above)

To be clear, I'm in favour of containerised tooling as well, hence wanted e.g a devcontainer. I just don't want to introduce unnecessary pain points, as this kind of step often does.

Key questions:

• What can we keep in and out of repo, while keeping us fast?
• What our our pain points so far? Notable examples for our poor poor rust devs: lerna, yarn
• What are other gotchas that we might fall prey to?

[aatifsyed]

https://github.com/ChainSafe/forest/actions/runs/4991539002/attempts/1 Hit again

[aatifsyed]

different version of clippy in CI to local: https://github.com/ChainSafe/forest/actions/runs/5154069457/jobs/9282067031?pr=2893

[LesnyRumcajs]

@aatifsyed what do we do with this?