Using reproducible wheels into our securedrop-app-code Debian package on Focal

[kushaldas]

Description

Currently we are using Python dependencies installed from source during the Debian package build. As we are retiring the Xenial builds, we should move to using our own reproducible wheels in Focal packaging using the scripts from the securedrop-debian-packaing repository.

Works need to done

• [ ] Xenial package builds should be removed.
• [ ] Create a requirements.txt file for the app code (should be copy of securedrop/requirements/python3/securedrop-app-code-requirements.txt in the same directory
• [ ] Add all bootstapping wheels for Focal that is Python 3.8 in the securedrop-debian-packaging repository.
• [ ] Build all the wheels for securedrop-app-code and add them to securedrop-debian-packaging repository.
• [ ] Update the ansible role and the rules file to use our wheels to build the Debian package.

Note: For Focal we can use the dh-virtualenv package from https://launchpad.net/~jyrki-pulliainen/+archive/ubuntu/dh-virtualenv directly.

[eloquence]

During this sprint, @kushaldas will investigate this build workflow locally, to begin scoping work for the next sprint.

[kushaldas]

Few open questions/points:

• If we keep all the wheels in the securedrop-debian-packaging repository, we have to clone it during the package build process. This may slow down the whole package building effort. If we mount it on the build container from the local host system, then things will be much faster and the human will be in charge to keep the repo updated from git. Or else the package build will fail.
• If we put all the wheels for the securedrop-app-code in the same localwheels directory, then our CI will break as a few wheels are dependent on the Python version/ OS. Should we use a separate directory for Focal wheels?
• Should we add a separate job in CI for Focal based wheel building? I think yes.

Update:

We should mark the wheels are all related files (sha256sums and signatures) properly for each distro. For example, bootstrap-focal, bootstrap-buster and so on. Otherwise the code updates to continue supporting the current names (without distro specific directories and filenames) create too much extra maintenance burden. I can not understand my own updates after I almost managed to support the scenarios.

This will also enable us in future to add a new distro very easily. For example when Debian Bullseye comes out in Qubes.

[kushaldas]

One good point of having all wheel from a particular OS into separate directory is that we can also separate out the tests in CI machines running the same OS can handle only the wheels from that particular directory. I will update with my findings more later in the evening.

Also the bootstrapping requirements will be different for various OS/projects. For all the workstation projects, it is one set, and for the securedrop-app-code it is a slightly different set.

[kushaldas]

This branch https://github.com/freedomofpress/securedrop-debian-packaging/tree/wheels_on_focal needs to be rebased to latest securedrop-debian-packaging repo. And then rebuild the dependencies for app-code.

[eloquence]

^ That is the extent of our commitment for the current sprint; we're aiming to continue work on this leading up to the SecureDrop 2.1.0 release.

[legoktm]

This is the remaining issue preventing reproducibility of securedrop-app-code. The issue is really subtle, the only thing that varies is the "build ID". pip ends up building each wheel in a randomly named temporary directory, which is captured into the build ID and then becomes non-reproducible. In theory https://reproducible-builds.org/docs/build-path/ would help here, but that needs to be injected by setuptools or whatever is building the C extension, and I don't think we know the new path, so it's messy and doesn't work. Oh and pip deprecated and removed the setting that allowed using a fixed build path!

Of course, this is a solved problem by SDW. We use build to build the wheel in a fixed directory (see here). I would suggest we do the same for SD server, which Kushal already started working on.

FWIW, I played with a middle ground of using build to build the wheels on the fly and then install them: https://github.com/freedomofpress/securedrop/commit/6a491d795ba8d8dacf176e8ed439a190b06db0dc. It technically worked after some fixes I didn't push, but none of the bootstrapping was pinned, which kind of defeats the point. No point re-inventing the wheel either, when we've already invented reproducible wheels!

[legoktm]

Also, during package builds we're currently building all of the Python dependencies twice. Using pre-built wheels will cut that down to zero times.

[legoktm]

I filed #6546 to clean up how our requirements.txt files are organized before we embark down this.

[legoktm]

Current work in flight:

• https://github.com/freedomofpress/securedrop-debian-packaging/pull/382 - makes room to add sever wheels into the repo, reusing the same bootstrapping scripts
• https://github.com/freedomofpress/securedrop-debian-packaging/pull/383 is a draft of adding the server wheels, has its own TODO list.
• https://github.com/freedomofpress/securedrop/pull/6544 - not strictly a blocker, but I want this to land first so it's cleaner to add in reproducible wheels to the build process
• https://github.com/freedomofpress/securedrop/pull/6571 - needed because the current pytz version's source package is a zip, not tar.gz and our scripts expect tar.gz.