conan
conan copied to clipboard
conan-io
[question] upload all recipes, exclude special packages from upload
I've again revisited the idea of having a Yocto SDK as a conan package (see also https://github.com/conan-io/conan/issues/5059) + profile tool requirement.
There is the cross compilation profile: profile: cross
[settings]
arch=armv8
os=Linux
compiler=gcc
compiler.version=11
compiler.libcxx=libstdc++11
build_type=Release
[tool_requires]
yocto-sdk-toolchain/1.2.3@user/channel
There is also the application I want to build: application hierarchy
app/1.0.0@user/channel -> fmt/8.1.1@
In this environment, I can now issue a command to build the application and the SDK: build command
conan install app/1.0.0@user/channel -b missing -pr cross
After the build, a previously empty cache now contains everything that was required for the build: packages in cache
app/1.0.0@user/channel
fmt/8.1.1@
yocto-sdk-toolchain/1.2.3@user/channel
For simplicity, clients usually go for the following: upload command
conan upload "*" -c --all -r remote
There are a few issues with this now.
- The installer of the Yocto SDK is approximately 1GB of data, whereas the installed SDK is about 2GB. Users would be quicker to not use the Prebuilt package in this case. Furthermore, the server has to keep less data around.
- The installed SDK location does vary between users. Assuming I build it inside my personal cache, the installed SDK is optimized to operate from that exact location. This means that everyone but me cannot really use it.
What I've tried:
build_policy = "always"does prevent the upload of the unwanted package, but it also prevents every other package from uploading. Another sub-optimal thing here is that the SDK is installed way more often than intended.
As a developer of a SDK recipe, is there a way to enable the recipe upload, but prevent the prebuilt package upload, while still being able to upload everything else?
- [x] I've read the CONTRIBUTING guide.
Hi @dornbirndevelops
The installed SDK location does vary between users. Assuming I build it inside my personal cache, the installed SDK is optimized to operate from that exact location. This means that everyone but me cannot really use it.
Yes, that is true. In our first attempts I vaguely recall that we managed to make the SDK relocatable by some tweaking, but yes, it was not relocatable out of the box.
Quick question, if it was relocatable, would you consider packaging, uploading and download it, even if that would use extra storage and transfer, just for uniformity, or would you still prefer always installing it from an external place (and the Conan recipe used only for automating this install)?
It is true that there is no explicit mechanism to upload all binaries for packages except for a certain package recipe. The always can be inconvenient and suboptimal ,and the build_policy = "missing" doesn't block the uploads.
Conan 2.0 has an UploadAPI that would allow to build or trim what we call the "upload_bundle" which is json with the specification of what needs to be uploaded, so you could easily write your own upload command that does this, but you need 2.0, and maybe is a bit overkill.
I guess developers do not upload to servers, so this might only be an issue in CI servers? It might be easier to automated this in CI, maybe using the json output of the create or install and then applying the trimming there?
Hi @memsharded, thank you very much for the quick response.
Quick question, if it was relocatable, would you consider packaging, uploading and download it, even if that would use extra storage and transfer, just for uniformity, or would you still prefer always installing it from an external place (and the Conan recipe used only for automating this install)?
I see more cons than pros for a prebuilt package in the case of a SDK package.
- the intention was not a prebuilt package in the first place. IMHO a prebuilt package makes sense when:
- download size is smaller --> 1GB installer compared to 2GB installation is worse
- computational work (i.e. compiling stuff) is less --> download & run installer compared to unzip 2GB archive & reverse engineer the installers work
- relocatability is additional maintenance overhead with no future guarantee to work
- Artifactory has to keep not only the installer in a generic repository, but the installation as a huge conan package.
Looking at these arguments above, I would prefer that Conan just automates this install and not considers the installation as a candidate for uploads.
I guess developers do not upload to servers, so this might only be an issue in CI servers? It might be easier to automated this in CI, maybe using the json output of the create or install and then applying the trimming there?
Correct, CI server builds are the ones that provide prebuilt binaries. The problem I see with a dedicated CI solution is that in the context of application building, now the build needs to care about the filtering of maybe multiple different packages. This seems to be a bit contrary to the easy usability that the profile could offer here.
That being said, would it be maybe possible to have another option for the build_policy attribute like installer or something like that in 1.X?
It could behave like the missing build_policy with regards to build from source but just emit a warning (instead of always' exception) when uploading.
That being said, would it be maybe possible to have another option for the build_policy attribute like installer or something like that in 1.X?
Yes, I think it is a feature that is definitely worth discussing. It might not fit in the 1.X scope, as you know we are heavily focused on 2.0, but at least lets have the discussion.
Lets get some feedback from the Tribe, to see if we are missing some use cases.
I am drafting https://github.com/conan-io/conan/pull/12253 to discuss it, to be introduced in 1.54, as it would be necessary for the migration if we are removing build_policy = "always"