Quirks when building in GitHub Actions

[p5]

Will try and flesh this issue out over time, but writing down problems as I face them for now.

Problems

1. The ubuntu-22.04 runner uses Podman v3, which is very outdated
2. The ubuntu-24.04 runner has issues with mismatching overlay storage drivers
3. AWS credentials set through configure-aws-credentials action are not passed in

1 - Ubuntu 22.04 outdated Podman v3

The version of Podman shipped with Ubuntu 22.04 is extremely outdated. The only way around this is to install development builds of Podman from a third-party repository like OpenSUSE, which is suboptimal.
This will improve once Ubuntu 24.04 runners are GA (currently in beta), but will become outdated again in a few years.

2 - Ubuntu 24.04 runner mismatched storage drivers

When building using the beta Ubuntu 24.04 GitHub Actions runner, we face an issue where BIB complains about mismatching storage drivers. I did not check if the same was happening in the 22.04 runner.

Build log:

Trying to pull quay.io/centos-bootc/bootc-image-builder:latest...
Getting image source signatures
Copying blob sha256:6561cc7ebe8906d1df76fe352ee04b7fc5c8b2672fd77e4da8fca830da86
Copying blob sha256:bd040cb81b6cd538d233ceb9349b62466f2336d6f530ebe8d1bbbb781f238773
Copying blob sha256:ee3acbbc3fa48bbeb4950bf045e391f9894cc1012d9d6a9f107316b16e730fc2
Copying config sha256:15cc55bdacf1df591ca5e047a10c4ad5622fc65944307c64a55ec89b1734e8
Writing manifest to image destination
Generating manifest manifest-ami.json
Error: cannot build manifest: cannot get container size: failed inspect image: exit status 125, stderr:
Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch

2024/05/22 09:30:54 error: cannot build manifest: cannot get container size: failed inspect image: exit status 125, stderr:
Error: database graph driver "" does not match our graph driver "overlay": database configuration mismatch
Error: Process completed with exit code 1.

To resolve this, we needed to add the following bash commands before pulling the base image:

sudo rm -rf /var/lib/containers/storage
sudo mkdir -p /etc/containers
echo -e "[storage]\ndriver = \"overlay\"\nrunroot = \"/run/containers/storage\"\ngraphroot = \"/var/lib/containers/storage\"" | sudo tee /etc/containers/storage.conf

3 - AWS credentials not being read from environment variables

When trying to consume AWS credentials from environment variables, we received "no valid providers in chain" errors. From the below log, you can see the environment variables are set, and are being correctly defined in the podman run command.

Build log:

mkdir -p .osbuild/outputs
sudo podman run \
    --rm --privileged \
    --security-opt label=type:unconfined_t \
    -v $(pwd)/.osbuild/config.toml:/config.toml \
    -v $(pwd)/.osbuild/outputs:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    --env AWS_* \
    --env AWS_PROFILE=default \
    quay.io/centos-bootc/bootc-image-builder:latest \
        --type ami \
        --aws-ami-name bootc/centos-base \
        --aws-bucket bootc-ami-sandbox-euwest1 \
        --aws-region eu-west-1 \
        --local <redacted>.dkr.ecr.eu-west-1.amazonaws.com/bootc/centos-base:latest
shell: /usr/bin/bash -e {0}
env:
    IMAGE_REGISTRY: <redacted>.dkr.ecr.eu-west-1.amazonaws.com
    AWS_DEFAULT_REGION: eu-west-1
    AWS_REGION: eu-west-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***
Trying to pull quay.io/centos-bootc/bootc-image-builder:latest...
Getting image source signatures
Copying blob sha256:6561cc7ebe8906d301df76fe352ee04b7fc5c8b2672fd77e4da8fca31830da86
Copying blob sha256:bd040cb81b6cd538d233ceb9349b62466f2336d6f530ebe8d1bbbb781f238773
Copying blob sha256:ee3acbbc3fa48bbeb4950bf045e391f9894cc1012d9d6a9f107316b16e730fc2
Copying config sha256:15cc55bdacf1df591ca5e047a10c4ad5622fc65944307c64a55e32c89b1734e8
Writing manifest to image destination
Error: cannot handle AWS setup: retrieving AWS regions for 'eu-west-1' failed: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2024/05/22 09:02:40 error: cannot handle AWS setup: retrieving AWS regions for 'eu-west-1' failed: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

This was resolved by writing the variables to a .env file and pointing to that, rather than the individual variables.

Sample:

          mkdir -p .osbuild/outputs
          # Create envfile with credentials
          echo "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" > /tmp/aws.env
          echo "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" >> /tmp/aws.env
          echo "AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN" >> /tmp/aws.env
          # Run the image builder
          sudo podman run \
            --rm --privileged --pull=newer \
            --security-opt label=type:unconfined_t \
            -v $(pwd)/.osbuild/config.toml:/config.toml \
            -v $(pwd)/.osbuild/outputs:/output \
            -v /var/lib/containers/storage:/var/lib/containers/storage \
            --env-file /tmp/aws.env \
            quay.io/centos-bootc/bootc-image-builder:latest \
              --type ami --rootfs ext4 \
              --aws-ami-name ${{ steps.generate-image-name.outputs.image-name }} \
              --aws-bucket ${{ vars.AWS_AMI_BUCKET }} \
              --aws-region ${{ vars.AWS_REGION }} \
              --local ${{ steps.generate-image-name.outputs.image-name-full }}:latest

[github-actions[bot]]

This issue is stale because it had no activity for the past 365 days. Remove the "Stale" label or add a comment, otherwise this issue will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 365+30 days with no activity.