raspberrypi
[BUG]: 1.9.0 vendoring breaks distribution builds
What happened?
I package rpi-imager for Arch Linux and as I wanted to upgrade to the new version I noticed that it now vendors all dependencies and also tries to install them, leading to various conflicts with other system utilities (curl, xz, zstd etc.).
Maybe I have just missed it, but it seems like there is no way to disable this behaviour currently 😅
Also I noticed that the issue template was not yet adjusted for the new version (see PR #923 for this) 😊
The current build instructions look something like this (see the full PKGBUILD if relevant):
build() {
cmake -B build -S "${pkgname}-${pkgver}/src" \
-DCMAKE_BUILD_TYPE='None' \
-DCMAKE_INSTALL_PREFIX='/usr'
cmake --build build
}
package() {
DESTDIR="$pkgdir" cmake --install build
}
Version
1.9.0
What host operating system were you using?
Other Linux environment
Host OS Version
Rolling Releasee
Selected OS
Arch Linux
Which Raspberry Pi Device are you using?
not applicable
What kind of storage device are you using?
not applicable
OS Customisation
- [ ] Yes, I was using OS Customisation when the bug occurred.
Relevant log output
sudo pacman -U /home/custompkgs/rpi-imager-1.9.0-1-x86_64.pkg.tar.zst
[sudo] password for chris:
loading packages...
resolving dependencies...
looking for conflicting packages...
Package (1) Old Version New Version Net Change
rpi-imager 1.8.5-3 1.9.0-1 2.80 MiB
Total Installed Size: 4.22 MiB
Net Upgrade Size: 2.80 MiB
:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring [########################################################] 100%
(1/1) checking package integrity [########################################################] 100%
(1/1) loading package files [########################################################] 100%
(1/1) checking for file conflicts [########################################################] 100%
error: failed to commit transaction (conflicting files)
rpi-imager: /usr/bin/curl-config exists in filesystem (owned by curl)
rpi-imager: /usr/bin/lzmadec exists in filesystem (owned by xz)
rpi-imager: /usr/bin/lzmainfo exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xz exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzcmp exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzdec exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzdiff exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzegrep exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzfgrep exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzgrep exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzless exists in filesystem (owned by xz)
rpi-imager: /usr/bin/xzmore exists in filesystem (owned by xz)
rpi-imager: /usr/include/curl/curl.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/curlver.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/easy.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/header.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/mprintf.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/multi.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/options.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/stdcheaders.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/system.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/typecheck-gcc.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/urlapi.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/curl/websockets.h exists in filesystem (owned by curl)
rpi-imager: /usr/include/lzma.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/base.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/bcj.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/block.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/check.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/container.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/delta.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/filter.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/hardware.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/index.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/index_hash.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/lzma12.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/stream_flags.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/version.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/lzma/vli.h exists in filesystem (owned by xz)
rpi-imager: /usr/include/zconf.h exists in filesystem (owned by zlib)
rpi-imager: /usr/include/zdict.h exists in filesystem (owned by zstd)
rpi-imager: /usr/include/zlib.h exists in filesystem (owned by zlib)
rpi-imager: /usr/include/zstd.h exists in filesystem (owned by zstd)
rpi-imager: /usr/include/zstd_errors.h exists in filesystem (owned by zstd)
rpi-imager: /usr/lib/cmake/zstd/zstdConfig.cmake exists in filesystem (owned by zstd)
rpi-imager: /usr/lib/cmake/zstd/zstdConfigVersion.cmake exists in filesystem (owned by zstd)
rpi-imager: /usr/lib/cmake/zstd/zstdTargets-none.cmake exists in filesystem (owned by zstd)
rpi-imager: /usr/lib/cmake/zstd/zstdTargets.cmake exists in filesystem (owned by zstd)
rpi-imager: /usr/lib/libz.so exists in filesystem (owned by zlib)
rpi-imager: /usr/lib/libz.so.1 exists in filesystem (owned by zlib)
rpi-imager: /usr/lib/libz.so.1.3.1 exists in filesystem (owned by zlib)
rpi-imager: /usr/lib/pkgconfig/libcurl.pc exists in filesystem (owned by curl)
rpi-imager: /usr/lib/pkgconfig/liblzma.pc exists in filesystem (owned by xz)
rpi-imager: /usr/lib/pkgconfig/libzstd.pc exists in filesystem (owned by zstd)
rpi-imager: /usr/share/doc/xz/AUTHORS exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/COPYING exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/COPYING.0BSD exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/COPYING.GPLv2 exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/NEWS exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/README exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/THANKS exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/00_README.txt exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/01_compress_easy.c exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/02_decompress.c exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/03_compress_custom.c exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/04_compress_easy_mt.c exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/11_file_info.c exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/examples/Makefile exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/faq.txt exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/history.txt exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/lzma-file-format.txt exists in filesystem (owned by xz)
rpi-imager: /usr/share/doc/xz/xz-file-format.txt exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/lzmadec.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/lzmainfo.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xz.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzcmp.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzdec.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzdiff.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzegrep.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzfgrep.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzgrep.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzless.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man1/xzmore.1.gz exists in filesystem (owned by xz)
rpi-imager: /usr/share/man/man3/zlib.3.gz exists in filesystem (owned by zlib)
Errors occurred, no packages were upgraded.
Thanks for the report, @christian-heusel
Unfortunately, this is expected behaviour. The 1.8.x series releases brought out a range of exotic bugs where the differences were in some way related to individual platform builds (most typically, we'd see macOS builds working much better than the Windows or Linux builds).
Imager, unfortunately, does not see a wide range of support from external entities - and as such I've introduced vendoring to reduce the deployment complexity of rpi-imager. The upshot being that on any platform using our official builds, I can assert you're running the same code base - and so I don't have to consider why Linux Distribution X is having odd decompression behaviour, for example - because if you're running on a Linux host you should now be running an AppImage.
I'm prepared to review a PR that enhances the AppImage flow in a direction that better helps Arch users, however I will not entertain a move to remove AppImage support or circumvent the vendoring - both of which would increase the support burden while rpi-imager has a viable release.
As an aside - have you tried the AppImage?
wget ${appimage_path}
chmod +x ${appimage_file}
./${appimage_file}
@tdewey-rpi thanks for your fast reaction to this issue and all the work you put into rpi-imager! 😊
Unfortunately, this is expected behaviour. The 1.8.x series releases brought out a range of exotic bugs where the differences were in some way related to individual platform builds (most typically, we'd see macOS builds working much better than the Windows or Linux builds).
Ooof, that sounds like a few annoying issues to deal with! 🤔 For the future feel free to direct users to the Arch Linux Bugtracker for rpi-imager if you think that users are facing platform specific issues with our distribution or that something first needs some more downstream investigation 😄
I'm prepared to review a PR that enhances the AppImage flow in a direction that better helps Arch users, however I will not entertain a move to remove AppImage support or circumvent the vendoring - both of which would increase the support burden while rpi-imager has a viable release.
I have no problem with this project supporting distribution via appimages (so removal isn't a goal of mine), but I think the AppImage concept is mostly orthogonal to distribution supplied packages. I think for me it would already be good to have a few knobs in the CMakeLists.txt that would allow me to disable the vendoring so I could hook everything up with the system dependencies on our side of things. Another project I'm packaging has flags like -D audacity_use_vst3sdk=system for example, which is a pattern that I could imagine to be used here aswell, so each dependency would get their -Drpi_imager_use_curl={bundled | system} and packagers could utilize the latter option for their needs, while the first could be a default for the upstream AppImage building.
As an aside - have you tried the AppImage?
I tried it and seems to work well! 👍🏻
I played a bit with the current rpi-imager CMakeLists and can propose two options:
- Vendoring opt-out. Add an option, say
ENABLE_VENDORING, which when turned OFF brings the previous library linking behavior (mainly for the unix systems). Drafted how it would look like in code here - Component based install. I'm not a CMake expert, but found an interesting option, that would allow to selectively install only the files related to
rpi-imagerusingcmake --install <build> --component <name>. In this case, the app is still being built with vendored libraries (they are compiled-in statically, as far as I understand), but this would make the life of the linux distro maintainers easier during packaging, so that no need chase and strip off the unnecessary headers, libs, etc. Please, take a look at the draft
cc: @tdewey-rpi @christian-heusel
Thank you for the considered reply, @christian-heusel
Unfortunately the use of system dependencies is the core of the problem - with no agreement across distributions for which versions of each dependency you will package, and each distribution having a separate release cadence, distribution creators enforce a scenario where an application developer must be prepared to QA a huge number of potential library and supporting file combinations. This is a QA burden that I'm not prepared to commit to, and to suggest otherwise would be a disservice to users fairly raising issues on this issue tracker.
@alkersan Thank you for your investigation, however I fear it's misplaced - this isn't a matter of a technical hurdle, but one of release policy. Your point on component builds is particularly well received - and something I may consider to make the AppImage even smaller still.
All in all, distribution builds are broken by design, because they impose a QA burden on myself and other contributors to rpi-imager that is not supported by the distributions making the packaging decisions.
One piece of feedback I've received is that some distributions would be happy with an alternative packaging format - flatpaks, for example. I'd be happy to review a scheme for building a flatpak or snap distribution, and would consider including that infrastructure in this repository - neither of those schemes will introduce a disproportionately larger testing burden. The major condition of that inclusion would be essentially the same as submitting a patch to the Linux kernel - where if you introduce a PR for new infrastructure, there would be an expectation that you would maintain it.
Closing this one as 'Won't fix', as the discussion has gone stale and I haven't received a strong argument in favour of allowing unknown-dependency distribution builds.
As mentioned above, I'm happy to entertain other packaging formats that carry the same characteristics.
Unfortunately the use of system dependencies is the core of the problem - with no agreement across distributions for which versions of each dependency you will package, and each distribution having a separate release cadence, distribution creators enforce a scenario where an application developer must be prepared to QA a huge number of potential library and supporting file combinations. This is a QA burden that I'm not prepared to commit to, and to suggest otherwise would be a disservice to users fairly raising issues on this issue tracker.
It sounds like your fundamental concern is controlling the versions of dependencies for API and behavioral differences.
I don't really understand why you think the way you do though -- since when does allowing other people to build with system dependencies imply a requirement to support them in their endeavors? It's relatively common for software developers to do something like requiring the output of progname --bugreport as part of reporting an issue -- this would list the versions of all dependencies, who performed the build, what git commit or release tag etc. was used... and then simply close ALL issues that aren't trivially reproducible and where the reported info implies a distribution build, with the statement "works for me -- please seek support from your distro".
You can also just require minimum versions of dependencies corresponding to what you use yourself (or even exact versions?) and simply refuse to care about distros without the correct versions. It's not like you're supporting them for the past few weeks anyway, so what's the problem with permitting people who already have the right versions to use those versions?
(Commenting because a gentoo contributor was trying to work out these changes. Gentoo supports multiple versions of dependencies in the same repository so it's actually feasible to restrict to a specific version mandated by the tool, but only if system dependencies are actually supported. In general I'd still recommend minimum bounds without upper bounds though.)
One piece of feedback I've received is that some distributions would be happy with an alternative packaging format - flatpaks, for example. I'd be happy to review a scheme for building a
flatpakorsnapdistribution
To be clear, this mostly isn't something a Linux distribution can be happy or sad about, since "alternative packaging formats" exist outside of Linux distributions except to the extent that some (not all) distributions happen to provide an "officially supported packaging format" edition of the flatpak/snap programs themselves and after that it's really quite out of the distro's hands. Users of distributions may, perhaps, be happy about the alternative availability, especially users of "frozen in amber" distributions such as Debian/Ubuntu/RedHat.
(I guess the exception here would be Ubuntu silently replacing random Ubuntu packages with postinst scripts that execute the snap installer. Totally not a conflict of interest or anything, we swear!)
I noticed this issue when I plan to update the package to 1.9.0 in Fedora, https://src.fedoraproject.org/rpms/rpi-imager/pull-request/2. It is also recommended to build against system dependencies in Fedora linux. However, since you used the vendored dependencies for now, we can mark these dependencies as bundled, like Provides: bundled(<libname>) = <version>. https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling
2. Component based install. I'm not a CMake expert, but found an interesting option, that would allow to selectively install only the files related to
rpi-imagerusingcmake --install <build> --component <name>. In this case, the app is still being built with vendored libraries (they are compiled-in statically, as far as I understand), but this would make the life of the linux distro maintainers easier during packaging, so that no need chase and strip off the unnecessary headers, libs, etc. Please, take a look at the draft
@tdewey-rpi Can you consider adopting this related commit change to facilitate the installation? I tested that it worked.
Thanks for the feedback, @eli-schwartz
Unfortunately the use of system dependencies is the core of the problem - with no agreement across distributions for which versions of each dependency you will package, and each distribution having a separate release cadence, distribution creators enforce a scenario where an application developer must be prepared to QA a huge number of potential library and supporting file combinations. This is a QA burden that I'm not prepared to commit to, and to suggest otherwise would be a disservice to users fairly raising issues on this issue tracker.
It sounds like your fundamental concern is controlling the versions of dependencies for API and behavioral differences.
Yes - in particular, changing Qt versions underfoot is an extremely deleterious choice, and causes all sorts of problems - not least because the front-end of rpi-imager is evaluated at runtime, so a 'valid' build can still behave in unexpected ways. Qt also provides wrappers atop a range of our other dependencies - hence including a wide closure in releases post-v1.9.0.
I don't really understand why you think the way you do though -- since when does allowing other people to build with system dependencies imply a requirement to support them in their endeavors? It's relatively common for software developers to do something like requiring the output of
progname --bugreportas part of reporting an issue -- this would list the versions of all dependencies, who performed the build, what git commit or release tag etc. was used... and then simply close ALL issues that aren't trivially reproducible and where the reported info implies a distribution build, with the statement "works for me -- please seek support from your distro".
rpi-imager carries Raspberry Pi branding, and that carries an implied support expectation. The mechanism you describe still requires the maintainer of the project to manually evaluate the report and flag accordingly - a time sink not paid for by the building nor distributor. Finally, such a mechanism is reactive - it only comes into effect after the end user has had a poor experience. Building up-front with a supported closure reduces the cone of experience that end users can fall in to.
You can also just require minimum versions of dependencies corresponding to what you use yourself (or even exact versions?) and simply refuse to care about distros without the correct versions. It's not like you're supporting them for the past few weeks anyway, so what's the problem with permitting people who already have the right versions to use those versions?
>= allows a builder to use a later version of Qt, for example, which may include semantic behaviour breaks. I'm not prepared to commit to supporting an undefined problem space.
(Commenting because a gentoo contributor was trying to work out these changes. Gentoo supports multiple versions of dependencies in the same repository so it's actually feasible to restrict to a specific version mandated by the tool, but only if system dependencies are actually supported. In general I'd still recommend minimum bounds without upper bounds though.)
I'd be happy to review a PR from an interested party - but as above, any mechanism that widens the support cone is unlikely to be accepted.
@tdewey-rpi Can you consider adopting this related commit change to facilitate the installation? I tested that it worked.
I'm happy to review a PR, however I notice that commit looks like it might not form the correct filesystem for an AppImage - which is the bar that must be cleared for a Linux build.
@nethershaw yeah but also generally none of those resources, says that upstream projects should be "ashamed" or are "toxic" for not wanting to be packaged in Gentoo. It may be an unfortunate decision that prevents Gentoo users from installing the project, but as a general rule in FOSS we don't demand others write software for our usage.
Contrarily, the toxic behavior here is yours. It also makes it generally less likely that upstream project developers will be willing to work with distros at all.
Please show us your paid support contract entitling you to install rpi-imager, if you disagree with my assessment. (Note: I don't count buying an rpi as being a paid support contract for rpi-imager itself.)
P.S. adding a bunch of emoji thumbsup/heart reactions to your own comment is rather gauche.
You are hostile to source-based Linux distributions.
I would also like to note with some humor here that the original issue was opened by an Arch Linux packager, which is NOT a source based distribution.
@eli-schwartz Do not apologize for them.
It's not apologizing for them when I point out that you aren't owed anything, even the right to download and use software. If you don't like it, you're welcome to not use raspberry pi products, or whatever other remediation suits you. None of this gives you the right to act in a disreputable and antisocial manner towards other human beings.
As though this prevents the issue from affecting anyone else. You're being intentionally thoughtless, too.
Nowhere did I say this prevents the issue from affecting anyone else. Please reread what I said -- or perhaps read it for the first time, I can't be sure. :D
I replied to your claim that this is an act of hostility to "source-based Linux distributions" by pointing out that this has nothing to do with source-based anything. Whatever your thoughts are about rpi-imager's relationship with downstream redistributors, it quite clearly affects all redistributors whether they are source-based or not.
Your communication style is lacking and fails to effectively convey your intentions, that's all.
Anyways, @nethershaw you talk an awful lot about the seemingly lurid crimes that rpi-imager is engaging in but I can't help wondering, what is your skin in the game?
The original issue reporter is a linux distribution developer. I'm a linux distribution developer. A third linux distribution developer commented here previously.
None of us felt the needs of our linux distributions required us to be as rude as you are -- what's your stake in the linux distribution game that you feel compelled to make rude comments on behalf of "linux distributions"? You mentioned Gentoo but as far as I'm aware you don't represent the Gentoo project in any way (if you're applying to join Gentoo, I'd be happy to put a word in against you, in case that helps...)
@nethershaw please find somewhere else to "contribute". If you want to find someone to direct your drivel to, go find a mirror.
I was extremely disappointed this morning to see a flurry of frankly hostile comments on what is fundamentally a technical choice.
I note that the principal actor has apparently now deleted their comments - but they should be advised that full transcripts of each were sent to every observer of this thread. I would encourage them to seek help, as that manner of communication is simply not appropriate in any community.
I wish to thank other respondents for having cooler heads.
In terms of 'toxic choices', I note that source-based distributions are free to patch the build system and continue to use system dependencies. This isn't a choice I can directly support, for the reasons stated earlier in the thread, but is one that remains open because I didn't remove the build infrastructure, and continue to move this forward in the open. Indeed, the overall patch should be extremely straightforward.
I remain prepared to assist distros through this change - but again, I'm not prepared to directly widen the support cone, and actions to support your distro may be queued behind supported platform and new feature works. If a distro developer is particularly vexed by how to best bring Imager to their distro, please raise a Feature Request in this issue tracker.
Finally, I'm going to lock this thread, and apologise to the ~6 people who might've also got both barrels in their inbox this morning.