[bug] Inconsistent behaviour between package queries and package-id

[antoinebardoz]

Environment details

• Operating System+version: Ubuntu 22.04
• Compiler+version: N/A
• Conan version: 2.1.0
• Python version: 3.10.8

Steps to reproduce

1. Have a package that does not define any "settings".
2. Try to download this package, but filter by some setting, e.g: conan download package/1.0.0 -r myremote -p os=Linux
3. Conan will return ERROR: 404: Not Found. [Remote: myremote]

It is possible to remove settings which are not relevant for the package, as documented here: https://docs.conan.io/2/reference/conanfile/methods/package_id.html

A given package recipe can decide in its package_id() that the final binary is independent of some settings, for example if it is a header-only library, that uses input settings to build some tests, it might completely clear all configuration, so the resulting package_id is always the same irrespective of the inputs.

This is a very useful feature, since Conan is famously very strict about package-id. However, having queries filter by these deleted settings brings back the problem for me.

FYI: What I'm really trying to achieve is to download a pre-compiled package so I can access the binaries it contains, but there is no simple way to do this. You can use conan install, but then all of the package's dependencies are also downloaded, which I'm not interested in (they could also be very large). Or, you can use conan download, but then you have to use a completely different syntax to narrow down which package you want, which is confusing for my team and, I assume, other users as well.

What I would really love is to be able to use profiles with conan download, and have the semantics be the same as when you use conan install. Such as: conan download package/1.0.0 -r myremote -pr myprofile --settings build_type=Release

And if build_type is not a setting for package/1.0.0 it should be ignored.

In absence of that, I could live with settings being treated the same for package queries and package IDs.

Logs

No response

[memsharded]

Hi @antoinebardoz

You are right, the -p ... argument is only looking for "positive" matches. We had developed the possibility to pass profiles to these commands, with the goal of helping with the "Missing binaries" case, but it was lately replaced with the conan graph explain command.

The problem with the conan download package/1.0.0 --profile=myprofile approach is that it doesn't take into account dependencies at all, because they are not modeled in profiles. So if the package/1.0.0 has different binaries, lets say, because it linked with different versions of dependencies zlib/1.2.11, zlib/1.2.12, ..., zlib/1.3 and others, it could have many different binaries and it is not that evident which one would be the one you want to use. Because when you are downloading that package in particular is because you want to use it somehow?

In that regard, the recommendation in general is to use package-lists. When a dependency graph is computed, it can return a json with the graph, that can generate a package-list. That package-list is the mapping between a given profile and the different binaries you need.

In any case, I understand the use case of passing a profile to these commands (though maybe we only consider for conan list, then you pass the result to conan download, to avoid the proliferation of arguments in all commands), so I will present this to the team to reconsider. Thanks for the feedback!

[antoinebardoz]

The problem with the conan download package/1.0.0 --profile=myprofile approach is that it doesn't take into account dependencies at all

Well, in this case I would say "One man's trash is another man's treasure". For this application, I have no interest in the dependencies, I simply want to download a package so I can get its contents. If I wanted the dependencies, I could use conan install instead, for example with a custom deployer.

So if the package/1.0.0 has different binaries, lets say, because it linked with different versions of dependencies zlib/1.2.11, zlib/1.2.12, ..., zlib/1.3 and others, it could have many different binaries and it is not that evident which one would be the one you want to use.

I understand the problem, but this is a good example of how Conan can sometimes be needlessly strict. What if there is only a single package in my remote that matches my query? Why should Conan prevent me from downloading this package without me providing an entire dependency graph?

While I await your possible fix, I have implemented my own custom command. Its behaviour:

1. If there is a single package which matches a reference, like package/1.0.0, you don't even need to give a profile: conan mydownload package/1.0.0
2. If there is more than one package that matches, the command asks if I want to download them all, or cancel the download entirely. Then I can choose if I want to be more specific, and add a profile, option or setting requirement. As in: conan mydownload package/1.0.0 -pr myprofile --options myoption=True.

I wish Conan was a bit more lenient like this in general.

Because when you are downloading that package in particular is because you want to use it somehow?

This may be a strange and ill-considered use of Conan, but we use Conan to manage compiled binaries of our code. It seemed to us a logical use, since Conan has an immutable database of compiled packages, why not use Conan to download certain versions of our binaries? However, considering how hard it is to get this to work smoothly, maybe we're the only ones who thought this was a good idea?

[memsharded]

Well, in this case I would say "One man's trash is another man's treasure". For this application, I have no interest in the dependencies, I simply want to download a package so I can get its contents. If I wanted the dependencies, I could use conan install instead, for example with a custom deployer.

I understand the problem, but this is a good example of how Conan can sometimes be needlessly strict. What if there is only a single package in my remote that matches my query? Why should Conan prevent me from downloading this package without me providing an entire dependency graph?

Of course, this is why the conan download command exists, to not download the full dependencies, just the package that you want. I am not saying that it should download the dependencies. Lets say that you are downloading your application executable myapp/1.0, the binary for Linux. This application has been built and linked statically against several versions of zlib, so you have different binaries with different executables for myapp/1.0. When you conan download myapp/1.0 -p "os=Linux" it will download you 3 different binaries, executables, and it will not download any dependencies. You can execute any of them, and the 3 different executables will run with different speeds because they statically linked different versions of zlib.

If there is a single package which matches a reference, like package/1.0.0, you don't even need to give a profile: conan mydownload package/1.0.0

But this is already the default behavior of conan download isn't it? If there is only one binary, it will download only that one binary?

This may be a strange and ill-considered use of Conan, but we use Conan to manage compiled binaries of our code. It seemed to us a logical use, since Conan has an immutable database of compiled packages, why not use Conan to download certain versions of our binaries? However, considering how hard it is to get this to work smoothly, maybe we're the only ones who thought this was a good idea?

Of course not. This is the reason for example that the direct_deploy exist, you can do:

conan install  --requires=myapp/1.0 --deployer=direct_deploy --output-folder=my_output

And it will get you a local copy of the package, with several advantages:

• the Conan binary model will actually "skip" the downloading of unnecessary dependencies. If you are installing an executable that links statically against libraries, those libraries binaries will not be downloaded at all
• You can provide a --lockfile=mylockfile to guarantee that you are obtaining your myapp/1.0 that was linked against the specific set of dependencies, like the exact version of zlib/1.3 you want. This is a very strict requirement for many of our production users, they don' want to download a binary for Linux of their application, but they want to enforce that binary was linked against a very specific and locked set of dependencies.

In any case, I am not saying that there isn't value in providing this functionality, and we will try to add it soon. I was just suggesting that even if it is provided, there are some use cases that we have learnt from other users that are very important and wouldn't be satisfied by this feature.

[antoinebardoz]

Thank you for taking the time to help. It's quite difficult to be precise in what I'm trying to describe, especially because I'm trying not to waste your time with information you don't need, so there are a few misunderstandings.

I am not saying that it should download the dependencies.

Sorry, I was a bit unclear in my message. I understand why there might be multiple package revisions depending on which dependencies the package was built with. I just meant that in the case where only one package matches there would be no issue, and for us this is a common case.

In fact, now what I think about it, I'm not sure why it would conceptually be a problem that multiple packages could match, versus the current behaviour. The way it works now, conan download will already download multiple packages if it matches with more than one. Why is that different with profiles?

Of course not. This is the reason for example that the direct_deploy exist

conan install works well for the case where I'm trying to compile something new, and I want to get the dependencies in a folder. However, the use case I'm trying to describe is different.

Our final output is a pre-compiled image, which we will run on an embedded target. We create a conan package that has the image in it, this is our release. At this stage, the dependencies are already baked into the final package, since it is an image. Therefore it would, in many scenarios, be useful to download just this package, and ideally output it to a folder, instead of the conan cache.

It's this particular case, where we create a conan package of our final image, which will never be consumed by another Conan project, that I wonder if might be a use case you don't consider. And maybe we should instead use another mechanism?

My dream implementation of conan download would be:

1. Allows us to use profiles, settings and options to filter, rather than package queries (since they do not take into account that some settings may be disabled in some packages). Also, the syntax would be closer to other Conan commands, which would make things less confusing for inexperienced users.
2. Allow us to download the package into an output folder. Maybe this is a case where conan download could fail if more than one package was found? Ideally, it could print a list of the packages which matched, to help users figure out what they need to specify.

[memsharded]

Sure, sounds good.

Allows us to use profiles, settings and options to filter, rather than package queries (since they do not take into account that some settings may be disabled in some packages). Also, the syntax would be closer to other Conan commands, which would make things less confusing for inexperienced users.

yes, this is what we have in mind, and we agree it makes sense

Allow us to download the package into an output folder. Maybe this is a case where conan download could fail if more than one package was found? Ideally, it could print a list of the packages which matched, to help users figure out what they need to specify.

That might be a bit too much for the download command (too much responsibility), but maybe a custom command will be almost trivial for this custom logic (we are also adding the capability of custom commands calling other commands, which will simplify a lot this flow). In any case, lets go first for the --profile argument which is the core functionality, and we can discuss this later, this might be a feature request to something kind of deployers for download too.

[memsharded]

Closed by https://github.com/conan-io/conan/pull/15697, the --filter-profile, --filter-settings and --filter-options will be available in Conan 2.2 to conan list