Builds not reproducible

[bjornbugge]

Environment:

• Jib version: 3.4.0
• Build tool: Gradle 8.4
• OS: MacOS

Description of the issue:

Some of the Docker image layers are given new sha256-hashes when invoking jib after a ./gradlew clean but otherwise do not make any changes. Locally it appears to be a layer that contains the output of a Copy task that changes checksum, but the file itself remains the same, with the same checksum. Even worse, on CI it's the "bottom" layer with the external libraries that change on every single build.

Expected behavior:

Invoking Jib twice on the same git rev and with the same parameters should yield the same image every time.

Probable cause

When inspecting the individual layers from the jibBuildTar task it seems that the culprits are the atime, mtime, ctime, and LIBARCHIVE.creationtime tar headers. They change between the builds dependning on whether input files have been deleted and recreated again. These headers should probably be cleared in ReproducibleLayerBuilder::setUserAndGroup where the username/group info is also cleared.

[chanseokoh]

Looks like this is a duplicate of #4131. The PR makes sense, and if it works, I think it would close both this and #4131.

Then the question is, does your PR #4142 work with the current apache-commons-compress version in this repo? That is, do those methods exist in the current library version? Otherwise, the PR needs to bump the version.

[bjornbugge]

So it is! :)

It works with the current version of apache-commons-compress yes. It's hard to write a unit test that provokes the bug with anything other than the mtime header, but I had to reset all four of them to get binary reproducibility.

[huhn-humane]

I've been investigating a similar issue and see this with files copied in via jib.extraDirectories.paths e.g.

jib {
    from {
        image = 'xxxxxx'
    }
    extraDirectories {
        paths {
            path {
                from = file(jibExtraDirectory)
            }
        }
    }
}

The birth and change times are not stable even though the file digests are the same and reproducible flags are set

tasks.withType(AbstractArchiveTask).tap {
    configureEach {

        preserveFileTimestamps = false
        reproducibleFileOrder = true
        archiveVersion = ""
    }
}

Stating the extra files

$ docker exec -it a07cbda7f2ef /bin/bash
root@a07cbda7f2ef:/# stat /opt/some-dir/foo.jar 
  File: /opt/some-dir/foo.jar
  Size: 17298075        Blocks: 33792      IO Block: 4096   regular file
Device: ach/172d        Inode: 3686231     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-20 20:21:44.209051010 +0000
Modify: 1970-01-01 00:00:01.000000000 +0000
Change: 2023-11-20 20:21:03.504850005 +0000
 Birth: 2023-11-20 20:21:03.291850005 +0000
root@a07cbda7f2ef:/# %
   ~ $ docker exec -it a86bf4a0c84c /bin/bash
stroot@a86bf4a0c84c:/# stat /opt/some-dir/foo.jar 
  File: /opt/some-dir/foo.jar
  Size: 17298075        Blocks: 33792      IO Block: 4096   regular file
Device: b6h/182d        Inode: 3686239     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-11-20 20:21:53.937355001 +0000
Modify: 1970-01-01 00:00:01.000000000 +0000
Change: 2023-11-20 20:21:33.602051005 +0000
 Birth: 2023-11-20 20:21:33.420051005 +0000

[twbecker]

I just lost several hours to this. It would be good to get it fixed since projects are upgrading commons-compress due to CVEs: https://github.com/davidmc24/gradle-avro-plugin/commit/7362bf7dc6cd66e07f02acc94fa0765890ca2d44

[glasser]

Does anyone have a workaround to this, until #4142 is fixed? I am finding it really hard to figure out how to pin this dependency to 1.21 in Gradle.

[huhn-humane]

@glasser i was able to override the commons version but have jib configured in a build convention / plugin so there's an added layer of indirection. I'm not sure how to do this as a standalone plugin dependency within a module, but pinning to 1.21 in my build convention's build.gradle seems to have worked with back-to-back builds. Removing the version pinning block results in the original issue of different digests for one of the layers.

HTH,

build convention's build.gradle:

    implementation libs.gradle.jib
    implementation ('org.apache.commons:commons-compress') {
        version { strictly '1.21' }
    }

I confirmed this with the jibDockerBuild task and using dive to inspect the digest layers for the two containers.

[bjornbugge]

@chanseokoh Are there any steps I need to take in order to get #4142 merged? As far as I can see it is awaiting review/approval by maintainers.

[chanseokoh]

Just nag the maintainers loudly.

[mpeddada1]

Thanks for your patience on this @bjornbugge! We've reviewed your PR. There are a few tests failing during to a checkstyle issue.

[izogfif]

@mpeddada1 nag() Please, take a look at #4204.

[mpeddada1]

Thanks for the follow up @izogfi ! Reviewed #4204.

[izogfif]

@mpeddada1 I updated #4204. Please take another look.