spesmilo
Unable to build docker image for building linux appimage. Old ubuntu packages no longer available (also wine windows exe)
I am unable to build the docker container used for building linux appimages. It appears to require packages which are no longer available in ubuntu's package repositories.
E: Version '1.0.2g-1ubuntu4.19' for 'libssl-dev' was not found
E: Version '1.0.2g-1ubuntu4.19' for 'libssl1.0.0' was not found
E: Version '1.0.2g-1ubuntu4.19' for 'openssl' was not found
The immediate cause is now fixed with https://github.com/spesmilo/electrum/pull/7472.
I'll leave this open however as clearly there is a larger underlying issue: we rely on version-pinned packages being available from Ubuntu repositories. As old version packages are not hosted by Ubuntu for long periods of time, this is a hurdle to e.g. historical reproducible builds (i.e. reproducing older builds).
It is mainly the AppImage and the Windows EXE builds that are affected.
(We have known about this shortcoming for a long time but turns out there hasn't been an explicit tracking issue for it.)
See e.g. https://github.com/spesmilo/electrum/pull/7299 https://github.com/spesmilo/electrum/pull/7091 https://github.com/spesmilo/electrum/pull/7030 https://github.com/spesmilo/electrum/commit/7afcfe7943584bf057d60c889485cc7d53f2af5c https://github.com/spesmilo/electrum/commit/c2d6a902dde63b117ff234764d2e7c60cd50c43c
I'll leave this open however as clearly there is a larger underlying issue: we rely on version-pinned packages being available from Ubuntu repositories. As old version packages are not hosted by Ubuntu for long periods of time, this is a hurdle to e.g. historical reproducible builds (i.e. reproducing older builds).
We could use Debian as a base for the build container instead, as Debian provides a 'wayback machine' for packages. This should reduce the maintenance burden (AND allow for reproducible builds of older versions), however we should of course monitor any security issues with the packages used in the build.
That sounds promising.
If we switched to a Debian base, I suppose we could pin a snapshot time in the Dockerfile and occasionally update that. Does that sound reasonable?
Or as a first step, just switch to Debian -- I guess that in itself might allow at least historical reproducibility as once a package is no longer provided, a user who wants to build could try to use a wayback machine snapshot at that point.
That sounds promising.
If we switched to a Debian base, I suppose we could pin a snapshot time in the Dockerfile and occasionally update that. Does that sound reasonable?
Yes, it's a one-liner in the sources.list file. Note however that you can't set arbitrary snapshot timestamps, you need to check the archived timestamps available.
Or as a first step, just switch to Debian -- I guess that in itself might allow at least historical reproducibility as once a package is no longer provided, a user who wants to build could try to use a wayback machine snapshot at that point.
We could freeze the snapshot timestamp as part of the release process, akin to freeze_packages.sh, and unpin in the commit right after?
One caveat: the docker base image could have issues with the snapshot repo, if the base image is (much) more recent and the snapshot triggers a dependency downgrade. We need to account for this situation, although I don't think it's likely to occur on the Debian stable series.
One caveat: the docker base image could have issues with the snapshot repo, if the base image is (much) more recent and the snapshot triggers a dependency downgrade
We pin the base image to a hash, should that not prevent the base image inadvertently becoming too new? We could just make sure not to update the base image hash pin without updating the snapshot timestamp.
https://github.com/spesmilo/electrum/blob/a59c8797dc0342d8fa14c812664564987a52837e/contrib/build-linux/appimage/Dockerfile#L4
We pin the base image to a hash, should that not prevent the base image inadvertently becoming too new? We could just make sure not to update the base image hash pin without updating the snapshot timestamp.
Yep, sounds good.
Thanks for the PR (https://github.com/spesmilo/electrum/pull/7926)! Now looking at it and thinking about this some more:
Re snapshot.debian.org, one question that just got me pondering is whether its trustworthiness is comparable to the main debian pkg archive.
For example, if a malicious package (for a non-niche / somewhat popular pkg we might use) got included into the main debian (ubuntu is probably comparable) archive, I imagine it would be discovered relatively fast. OTOH if a malicious pkg was inserted into the snapshot archive, only at a specific snapshot, I wonder how fast that would be discovered.
(it's not even just how fast malware would be discovered after-the-fact, but also getting malware in there in the first place might have considerably different difficulty)
In particular, consider the following attack scenario:
- we refresh the snapshot date in the Dockerfile
- multiple weeks go by
- an attacker (e.g. insider at
snapshot.debian.org) looks at the electrum repo, guesses that we will soon tag and release, and puts malware inside one of the deps only for that specific snapshot date - we build release binaries, verify they are reproducible between 2-3 people, do release
- attacker reverts change (removes malware from pkg in
snapshot.debian.org)
How realistic do you think this is? For example, as something less abstract, I wonder who else relies on these snapshots.
Although I agree that the snapshot archive gets less eyeballs than the main repo, the entire set of packages in a snapshot is guarded by a PGP signed summary file containing all the MD5 (hmm) checksums of the individual packages, as described in https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html
So I think we can trust the snapshot as much as we trust the PGP signing key for the main repo (and the strength of the MD5 algo, but that applies to the main repo as well).
That said, it might be necessary to check if the container currently does sufficient checks, or if we need to configure additional options to be really secure, ref https://wiki.debian.org/SecureApt
So I think we can trust the snapshot as much as we trust the PGP signing key for the main repo (and the strength of the MD5 algo, but that applies to the main repo as well).
Right, ok, good point. The signing mechanism and the set of trusted keys that is used to verify packages is the same regardless of the apt source used.