build: upgrade all test infrastructure container images to [email protected]

[petermetz]

Description

The new version of the ubuntu-dind image just came out a couple of days ago which might have important bug-fixes or security fixes in it so we should upgrade or at least start the upgrade process by discovering if problems exist (e.g. can our tests till pass with the new base image version or not)

https://github.com/cruizba/ubuntu-dind/releases/tag/24.0.7

Acceptance Criteria

1. All images that use the ubuntu-dind base image are upgraded
2. All the images that were upgraded are tested with at least 2 different test cases that are using that image

[aleeusgr]

• [ ] tools/docker/corda-all-in-one/corda-v5/Dockerfile

have the image upgraded, the build must complete.

timebox (?)

[petermetz]

timebox (?)

@aleeusgr This is the definition I go by:

Instead of working on the task until it’s done, you proactively decide how much time you’ll spend on it and when (and even where).

Using it helps me a lot because on any given day I have about 30 hours worth of things I'd need to get done that day, but only 24 hours to finish them (minus sleep and the other mandatory activities to keep my body alive and my brain functioning) So what I usually do is assign time frames to tasks to avoid getting sucked into rabbit holes where I spend the entire day (or even week) debugging some exotic issue that turns out to be just a bug in the language runtime or the operating system itself. Might or might not work in your case, but regardless what I recommend is to take a look at the task at hand and spend a few hours max and if it's not coming along nicely then ask more questions/advice. :-)

[aleeusgr]

Thanks @petermetz

Yeah, makes sense.

I am looking for ways to transition to full time programming so it makes sense to keep my GitHub activity up to build connections and learn technologies.

It's a bit late so I will get to this task tomorrow, will post stand ups here and tag you if I have blockers I can't solve at two or three attempts on my own.

It's my first time interacting with Hyperledger so I expect progress to take some time.

[petermetz]

Thanks @petermetz

Yeah, makes sense.

I am looking for ways to transition to full time programming so it makes sense to keep my GitHub activity up to build connections and learn technologies.

It's a bit late so I will get to this task tomorrow, will post stand ups here and tag you if I have blockers I can't solve at two or three attempts on my own.

It's my first time interacting with Hyperledger so I expect progress to take some time.

@aleeusgr Nice! In that case, welcome to the discipline! As a teacher of mine once explained: Programming is hard because you start out with not seeing any progress and you just have to keep going. So hang in there and good luck!

Important: Don't forget about the daily pair programming calls because sometimes I have 50 to 100 messages to respond to on GitHub a day (and then I only have time to respond to 15 and the rest just days weeks or months later unfortunately)

[aleeusgr]

Details

Such a huge project, wow. begin with reading the contributing guide.

• [x] fork, clone, checkout at a new branch.
• [x] run tools/ci.sh get an error on line 117: docker not found. Fair enough, I must have disabled it in my config.
• [x] enable docker in configuration.nix, rebuild

07:27 $ docker --version
Docker version 24.0.5, build v24.0.5

• [x] run ci.sh: get an error: docker-compose not found. 🤔
• [x] research nixOS docs on how to enable docker-compose.

07:58 $ docker-compose --version
Docker Compose version 2.23.1

error:

./ci.sh: line 121: java: command not found

• [x] add zulu8 to home config

testing finished with

This looks like a moderately successful CI run to me, I think I can proceed with testing the specifics.

[petermetz]

Details Such a huge project, wow. begin with reading the contributing guide.

• [x] fork, clone, checkout at a new branch.
• [x] run tools/ci.sh get an error on line 117: docker not found. Fair enough, I must have disabled it in my config.
• [x] enable docker in configuration.nix, rebuild

07:27 $ docker --version
Docker version 24.0.5, build v24.0.5

• [x] run ci.sh: get an error: docker-compose not found. 🤔
• [x] research nixOS docs on how to enable docker-compose.

07:58 $ docker-compose --version
Docker Compose version 2.23.1

error:

./ci.sh: line 121: java: command not found

• [x] add zulu8 to home config

testing finished with

This looks like a moderately successful CI run to me, I think I can proceed with testing the specifics.

@aleeusgr You are definitely off to a good start! The CI has grown more complicated in the past year and the contributing.md document doesn't fully reflect that yet unfortunately. Long story short, you'll want to try and build the container image locally and test it out that way first. To do that, you can look at the relevant section of the README.md file of the sub-folder where the container image is located in (tools/docker/corda-all-in-one/README.md)

This is the section you want to be looking at:

DOCKER_BUILDKIT=1 docker build ./tools/docker/corda-all-in-one/corda-v5/ -f ./tools/docker/corda-all-in-one/corda-v5/Dockerfile -t caio5
docker run --privileged caio5

P.S.: Hello fellow NixOS user ;-)

[aleeusgr]

awesome! Nice to see nix fellows working on interesting projects! The declarative system administration is so convenient after getting through initial hurdle of learning the basics. There are so many shell scripts in Hyperledger, I wonder if they all could be improved with nix. Still docker is a popular tool, thanks for the opportunity to work with it.

I am able to see the bootstrapper running but I can't access it with the command in README.

Questions:

1. How do I test the bootstrapper?
2. Does this change makes sense?

details

run $DOCKER_BUILDKIT=1 docker build ./tools/docker/corda-all-in-one/corda-v5/ -f ./tools/docker/corda-all-in-one/corda-v5/Dockerfile -t caio5:

ERROR: failed to solve: process "/bin/sh -c apk add --no-cache openssh augeas" did not complete successfully: exit code: 2

docker: Error response from daemon: pull access denied for caio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

Line with the problem:

https://github.com/hyperledger/cacti/blob/f6c3541c4078b3eff54610c076b161e4657e946b/tools/docker/corda-all-in-one/Dockerfile#L34-L35

https://unix.stackexchange.com/questions/102201/what-is-exit-2-from-finished-background-job-status

remove --no-cache from line 35 and rerun $docker build:

$docker run --privileged caio5

test:

08:43 $ curl -u earthling:password --insecure -X POST "https://localhost:12112/api/v1/flowstarter/startflow" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{\"rpcStartFlowRequest\":{\"clientId\":\"launchpad-1\",\"flowName\":\"net.corda.solarsystem.flows.LaunchProbeFlow\",\"parameters\":{\"parametersInJson\":\"{\\\"message\\\": \\\"Hello Mars\\\", \\\"target\\\": \\\"C=GB, L=FOURTH, O=MARS, OU=PLANET\\\", \\\"planetaryOnly\\\":\\\"true\\\"}\"}}}"
curl: (7) Failed to connect to localhost port 12112 after 0 ms: Couldn't connect to server

try another port:

✘-7 ~/workshop/cacti [fix2853 L|✚ 1] 
08:44 $ curl -u earthling:password --insecure -X POST "https://localhost:22222/api/v1/flowstarter/startflow" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{\"rpcStartFlowRequest\":{\"clientId\":\"launchpad-1\",\"flowName\":\"net.corda.solarsystem.flows.LaunchProbeFlow\",\"parameters\":{\"parametersInJson\":\"{\\\"message\\\": \\\"Hello Mars\\\", \\\"target\\\": \\\"C=GB, L=FOURTH, O=MARS, OU=PLANET\\\", \\\"planetaryOnly\\\":\\\"true\\\"}\"}}}"
curl: (7) Failed to connect to localhost port 22222 after 0 ms: Couldn't connect to server

the unexpected change

[aleeusgr]

• [x] test ./tools/docker/corda-all-in-one/corda-v5/Dockerfile:1:FROM cruizba/ubuntu-dind-focal:24.0.7

success:

solar-system-bootstrapper  | Bootstrapper still listening for requests

Pushed the changes to the branch in my fork: https://github.com/aleeusgr/cacti/tree/fix2853

I will go through the PR checklist at my next session.

N.B. There are two images actually:

cruizba/ubuntu-dind:focal-24.0.7
cruizba/ubuntu-dind:jammy-24.0.7

I am using focal for now, I wonder what the difference is.

Details

1. How do I clear docker cache? docker system prune -a

2. where do I actually change the values to get the expected result?

07:34 $ grep -Rnw . -e 'ubuntu-dind'
./tools/docker/corda-all-in-one/corda-v5/Dockerfile:1:FROM cruizba/ubuntu-dind:20.10.9
./examples/supply-chain-app/Dockerfile:1:FROM cruizba/ubuntu-dind:19.03.11 as runner
./examples/cactus-example-cbdc-bridging-backend/Dockerfile:1:FROM cruizba/ubuntu-dind:19.03.11
./examples/carbon-accounting/Dockerfile:1:FROM cruizba/ubuntu-dind:19.03.11 as runner

[petermetz]

• [x] test ./tools/docker/corda-all-in-one/corda-v5/Dockerfile:1:FROM cruizba/ubuntu-dind-focal:24.0.7

success:

solar-system-bootstrapper  | Bootstrapper still listening for requests

Pushed the changes to the branch in my fork: https://github.com/aleeusgr/cacti/tree/fix2853

I will go through the PR checklist at my next session.

N.B. There are two images actually:

cruizba/ubuntu-dind:focal-24.0.7
cruizba/ubuntu-dind:jammy-24.0.7

I am using focal for now, I wonder what the difference is. Details

@aleeusgr Awesome, thank you for verifying! Until this other PR is ready (https://github.com/hyperledger/cacti/pull/2814) we don't have a way to verify the changes any further so for now I'll just recommend that you try and make the same image update in the rest of the Dockerfiles that are using the ubuntu-dind base image. Once those are ready as well, a PR can be opened. Alternatively if the other images turn out to be a larger piece of work we can break it up into multiple smaller PRs.

[aleeusgr]

Sorry for stalling: it's been a busy week and building docker images locally slows my laptop down considerably.

I just remembered I can start the build before going out, I'll do that on the next entry tomorrow.

[petermetz]

@aleeusgr No worries! Thank you for keeping us in the loop!

[aleeusgr]

built and run successfully:

./examples/supply-chain-app/Dockerfile:1:FROM cruizba/ubuntu-dind:focal-24.0.7 as runner

But the running container spams the info error:

INFO spawnerr: can't find command '/home/appuser/.nvm/versions/node/v16.8.0/bin/node'

README suggests visiting 0.0.0.0:3200 to test, both this and localhost produce an error: not found.

• [ ] investigate
• [ ] post log output as a file.

[aleeusgr]

N.B.

I notice considerable increase in build time for cruizba/ubuntu-dind:focal-24.0.7 as runner in tools/docker/corda-all-in-one/corda-v5/Dockerfile compared to the version that was used before.

[aleeusgr]

• [ ] test cbdc-bdridging-backend https://github.com/hyperledger/cacti/tree/main/examples/cactus-example-cbdc-bridging-backend

README doesn't give me any build specs. In both previous examples docker build commands were different.

• [x] try docker build
• [x] find docker build
• [ ] check run_cbdc_app.sh
• [ ] docker build --file ./examples/cactus-example-cbdc-bridging-backend/Dockerfile ./examples/cactus-example-cbdc-bridging-backend/
• [ ] add PR with improvements to docs
• [ ] look at Dockerfile L31
• [ ] check the logfile

Details

Building and Running the Container Image

sh docker build --file ./examples/cactus-example-cbdc-bridging-backend/Dockerfile ./examples/cactus-example-cbdc-bridging-backend/ --tag example-cbdc-backend

docker run example-cbdc-backend ... some more parameters

[aleeusgr]

• [ ] test carbon-accounting https://github.com/hyperledger/cacti/tree/main/examples/carbon-accounting

🔁🪫 😕

[TheJagpreet]

Hi @aleeusgr, any updates on this task?

[aleeusgr]

Hi, @jagpreetsinghsasan I did some research and documentation on what needs to be done but then got caught up with some things. The most valuable thing I did is that I found all the instances where the technology is used, so what's left is just to test each case separately. Also keep in mind that in the version I was testing the nomenclature changed and just changing the version doesn't work - there is a suffix you need to add, the thing got two options for the same version and I only tested one.

[petermetz]

@aleeusgr I recommend submitting a smaller initial PR with the image migrations that you've already nailed down and are working. Then we can divide and conquer the remaining images that are troublesome/not so straightforward to migrate and tackle those in their own pull requests that can be associated to separate issues that we create dedicated to the problematic images. In other words, we can break the work up to smaller, more manageable chunks for this issue. It doesn't have to be an all or nothing approach to the contribution.

[aleeusgr]

Sorry for delayed response.

It got stuck in the backlog: it's been a tough couple of years (I moved out of Russia to keep my opinion on the war and my life at the same time), I am regularly taking too many tasks and my backlog just sort of overflows and things get left behind. I have three deadlines right now: two are freelance tasks and the third is a job interview task. I hope to have more headspace once I get paid for the contracts, I noticed lately I am saving on food😕

[petermetz]

@aleeusgr Very sorry to hear about the troubles you are facing, I hope it gets better soon! No worries about the slow replies, I'll probably do some work on this myself in the meantime as well so just keep an eye out for other PRs to avoid duplicate effort.

[aleeusgr]

Great to hear @petermetz My sincere apologies for the inconvenience.