Outstanding build reproducibility issues

[emaste]

#2410 addressed many sources of non-reproducibility, and reproducible-builds.org's CI now reports 535 (99.4%) out of 538 FreeBSD files were reproducible in our test setup.

The three outstanding failures are:

• https://tests.reproducible-builds.org/freebsd/dbd/repo/FreeBSD:15:amd64/current/data.pkg.html
• https://tests.reproducible-builds.org/freebsd/dbd/repo/FreeBSD:15:amd64/current/packagesite.pkg.html
• https://tests.reproducible-builds.org/freebsd/dbd/repo/FreeBSD:15:amd64/current/FreeBSD-runtime-current.pkg.html

These are data structure ordering issues

[emaste]

(screenshots as the links will update to the latest results)

[bapt]

are you sure this was created with pkg 2.0.6

[emaste]

Reproducible CI folks confirmed it is

[emaste]

"deps" are processed like this:

        dbg(4, "Emitting deps");
        map = NULL;
        while (pkg_deps(pkg, &dep) == EPKG_OK) {
                submap = ucl_object_typed_new(UCL_OBJECT);
                MANIFEST_EXPORT_FIELD(submap, dep, origin, string);
                MANIFEST_EXPORT_FIELD(submap, dep, version, string);
                if (map == NULL) 
                        map = ucl_object_typed_new(UCL_OBJECT);
                ucl_object_insert_key(map, submap, dep->name, 0, false);
        }
        if (map)
                ucl_object_insert_key(top, map, "deps", 4, false);

Are we sure ucl will produce stable output (from the ucl_object_insert_keys)?

[emaste]

From IRC discussion @ifreund will take a look

[emaste]

In the most recent run, r-b.org's CI reports "538 (100.0%) out of 538 FreeBSD files were reproducible in our test setup !" https://tests.reproducible-builds.org/freebsd/freebsd.html

I don't believe pkg has changed from the last run, so there may be something intermittent here. We'll keep an eye on this.

[emaste]

Newest run a week later has two unreproducible packages, runtime and clang:

https://tests.reproducible-builds.org/freebsd/dbd/repo/FreeBSD:15:amd64/current/FreeBSD-runtime-current.pkg.html https://tests.reproducible-builds.org/freebsd/dbd/repo/FreeBSD:15:amd64/current/FreeBSD-clang-current.pkg.html

Directory ordering (/tmp and /dev) in one case, "deps" ordering in the other case -- so the same problem we've seen in the past.

[ifreund]

Screenshots of the above links for archival purposes:

[emaste]

In the most recent run 3 packages show these issues: FreeBSD-certctl-current FreeBSD-openssl vs FreeBSD-caroot ordering in "deps" FreeBSD-clang-current FreeBSD-libcompiler_rt-dev vs FreeBSD-lld ordering in "deps" FreeBSD-runtime-current /dev/ vs `tmp/ in the file list and "directories"

[emaste]

I checked at one point over the last few weeks (after the above update) and CI reported 100% reproducible at the time.

Checking now, in the most recent run one* package, FreeBSD-clang-current, was unreproducible. The difference was the same as the last case (FreeBSD-libcompiler_rt-dev vs FreeBSD-lld ordering).

• One package excluding data.pkg and packagesite.pkg, as they will always change if any other package does.

[emaste]

@bapt reports that the issue with dependencies is fixed as a side-effect of some refactoring (switching to vectors that are always sorted). The directory list issue still exists.

[emaste]

The outstanding issue in FreeBSD pkgbase should be solved as a side-effect of https://reviews.freebsd.org/D50636 (but we should still fix the pkg issue).