conan icon indicating copy to clipboard operation
conan copied to clipboard

[bug] `conan list --graph-context=build-only` lists package that's also available in the host context

Open kambala-decapitator opened this issue 1 month ago • 8 comments

Describe the bug

TMU the build-only value should filter out packages that are also available in the host context (unlike the build value) according to this comment, but it's not so for onnx package.

Maybe related to using <host_version>? https://github.com/conan-io/conan-center-index/blob/888d9055518d7203c0986f3cbdb2675de83cadc8/recipes/onnx/all/conanfile.py#L102

> conan version
version: 2.23.0
conan_path: /Users/kambala/dev/conan/conan-venv/bin/conan
python
  version: 3.13.6
  sys_version: 3.13.6 (main, Aug  6 2025, 13:05:20) [Clang 17.0.0 (clang-1700.0.13.3)]
  sys_executable: /Users/kambala/dev/conan/conan-venv/bin/python3.13
  is_frozen: False
  architecture: x86_64
system
  version: Darwin Kernel Version 24.6.0: Wed Oct 15 21:12:21 PDT 2025; root:xnu-11417.140.69.703.14~1/RELEASE_X86_64
  platform: macOS-15.7.2-x86_64-i386-64bit-Mach-O
  system: Darwin
  release: 24.6.0
  cpu: i386

How to reproduce it

# assumes that onnx is already in the cache
conan graph info --requires onnx/1.16.2 -f json -b never -nr > graph.json
conan list --graph graph.json --graph-context=build-only -f json > pkglist.json

Expected: pkglist.json is empty Actual: it contains protobuf and zlib

kambala-decapitator avatar Dec 09 '25 12:12 kambala-decapitator

Hi @kambala-decapitator thanks a lot for taking the time to report this.

There's in fact some disagreement between the code and the expected usage of this feature, where the binary status (ie, if Conan will skip it, download it, or build it locally) influences how the packages are filtered when using the --graph-context={build-only,host-only} arg.

We will work on providing a solution, but for now you can unblock your workflow by either:

  • using --build=="*" so that all binaries are marked as binary: build as a workaround (But this will give you more binaries than the ones you would currently get if this was working correctly!)
  • Adding -c:a="tools.graph:skip_binaries=True", which will ask conan not to skip binaries, and work around this issue

Expected: pkglist.json is empty

Note that onnx has a tool_requirement on cmake, which would make it so the resulting pkglist contains cmake in the correct case

AbrilRBS avatar Dec 09 '25 14:12 AbrilRBS

Submitted https://github.com/conan-io/conan/pull/19368 with a reproduction case, will work on a fix there :)

AbrilRBS avatar Dec 09 '25 14:12 AbrilRBS

@AbrilRBS thank you for the quick response!

using --build=="*"

but that means that all the packages will be rebuilt? Doesn't sound like a good workaround :)

Adding -c:a="tools.graph:skip_binaries=True", which will ask conan not to skip binaries, and work around this issue

doesn't seem to change anything... I still get those 2 packages in the final JSON file.

> conan graph info --requires onnx/1.16.2 -f json -b never -nr -c:a="tools.graph:skip_binaries=True" > graph.json

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=17
os=Macos
[platform_tool_requires]
cmake/3.31.7
ninja/1.12.1
[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.files.unzip:filter=data
tools.graph:skip_binaries=True

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=17
os=Macos
[platform_tool_requires]
cmake/3.31.7
ninja/1.12.1
[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.files.unzip:filter=data
tools.graph:skip_binaries=True


======== Computing dependency graph ========
Graph root
    cli
Requirements
    onnx/1.16.2#b2afe58c5c3d85542d1aeef169586f42 - Cache
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88 - Cache
    zlib/1.3.1#b8bc2603263cf7eccbd6e17e66b0ed76 - Cache
Build requirements
    cmake/3.31.7 - Platform
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88 - Cache
    zlib/1.3.1#b8bc2603263cf7eccbd6e17e66b0ed76 - Cache
Resolved version ranges
    zlib/[>=1.2.11 <2]: zlib/1.3.1

======== Computing necessary packages ========
Requirements
    onnx/1.16.2#b2afe58c5c3d85542d1aeef169586f42:572234a1349144568cc81c5d460daad85c318cc8#0f0e9d0529b64101531f3ff50591bad2 - Cache
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88:fa7212abc4147d5e67a40a8822ba52748a07809b#fe290542b63cd17dfe6e3a70eb02e7f0 - Cache
    zlib/1.3.1#b8bc2603263cf7eccbd6e17e66b0ed76:a810d68ab5e5af5171e3e97c1419341bc8b7f209#d8e83ac57221509a71433b6adc1b492e - Cache
Build requirements
Skipped binaries
    cmake/3.31.7, protobuf/3.21.12, zlib/1.3.1

> conan list --graph graph.json --graph-context=build-only -f=json > pkglist.json
<no output>

Note that onnx has a tool_requirement on cmake, which would make it so the resulting pkglist contains cmake in the correct case

oh yes, you're right. I have CMake as platform_tool_requires :)

kambala-decapitator avatar Dec 09 '25 17:12 kambala-decapitator

doesn't seem to change anything... I still get those 2 packages in the final JSON file.

Typo! It should be False, ie `-c:a="tools.graph:skip_binaries=False"

but that means that all the packages will be rebuilt?

A conan graph info will not actually build the packages :)

AbrilRBS avatar Dec 09 '25 18:12 AbrilRBS

just verified that both workarounds work exactly as you had said. Thanks!

kambala-decapitator avatar Dec 10 '25 08:12 kambala-decapitator

@AbrilRBS I've tried --build="*" on a real project with a bigger graph and conan remove didn't delete any build context requirements at all. Is it expected? You can see it here: https://github.com/vcmi/vcmi-dependencies/actions/runs/20092722587/job/57643679450?pr=21#step:14:27

end of output (the JSON from conan list is then used in conan remove):

No binaries to remove for 'pkgconf/2.0.3#c7ba7fd1d7d22534ee070ae60ed79604'
No binaries to remove for 'meson/1.9.1#abbc783cd297bedce14581b4aec060b8'
No binaries to remove for 'yasm/1.3.0#c0bc84844465e1cd7850c820ad5b09b6'
No binaries to remove for 'pkgconf/2.5.1#93c2051284cba1279494a43a4fcfeae2'
No binaries to remove for 'b2/5.3.3#107c15377719889654eb9a162a673975'
No binaries to remove for 'nasm/2.16.01#31e26f2ee3c4346ecd347911bd126904'
No binaries to remove for 'strawberryperl/5.32.1.1#8d114504d172cfea8ea1662d09b6333e'
No binaries to remove for 'msys2/cci.latest#1996656c3c98e5765b25b60ff5cf77b4'
No binaries to remove for 'jom/1.1.4#070781f0f98cb84e95972172ba5333dd'
No binaries to remove for 'protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88'
No binaries to remove for 'zlib/1.3.1#cac0f6daea041b0ccf42934163defb20'
Remove summary:
Local Cache

kambala-decapitator avatar Dec 10 '25 12:12 kambala-decapitator

From https://github.com/vcmi/vcmi-dependencies/actions/runs/20092722587/workflow?pr=21#L230, I see that you're correctly calling the commands

Locally, I get:

~/c/m/r/l/all (pgi/libtorch/2.9.1|✔) 
$ conan graph info --requires onnx/1.16.2 -b='*' -f json > graph.json

======== Input profiles ========
Profile host:
[settings]
arch=armv8
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=17
os=Macos
[tool_requires]
*: ccache/4.12.1
[platform_tool_requires]
cmake/4.1.2
meson/1.9.1
ninja/1.13.1
[replace_tool_requires]
meson/*: meson/[*]
pkgconf/*: pkgconf/[*]
ninja/*: ninja/[*]
cmake/*: cmake/[*]
[conf]
tools.build:cxxflags=['-fcolor-diagnostics']
tools.cmake.cmaketoolchain:generator=Ninja

Profile build:
[settings]
arch=armv8
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=17
os=Macos
[tool_requires]
*: ccache/4.12.1
[platform_tool_requires]
cmake/4.1.2
meson/1.9.1
ninja/1.13.1
[replace_tool_requires]
meson/*: meson/[*]
pkgconf/*: pkgconf/[*]
ninja/*: ninja/[*]
cmake/*: cmake/[*]
[conf]
tools.build:cxxflags=['-fcolor-diagnostics']
tools.cmake.cmaketoolchain:generator=Ninja


======== Computing dependency graph ========
Graph root
    cli
Requirements
    onnx/1.16.2#b2afe58c5c3d85542d1aeef169586f42 - Cache
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88 - Cache
    zlib/1.3.1#cac0f6daea041b0ccf42934163defb20 - Cache
Build requirements
    ccache/4.12.1#9cd51178555f4babb32fc0b46aa9b765 - Cache
    cmake/4.1.2 - Platform
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88 - Cache
    zlib/1.3.1#cac0f6daea041b0ccf42934163defb20 - Cache
Replaced requires
    cmake/[>=3.18 <4]: cmake/[*]
Resolved version ranges
    zlib/[>=1.2.11 <2]: zlib/1.3.1

======== Computing necessary packages ========
ccache/4.12.1: Forced build from source
zlib/1.3.1: Forced build from source
protobuf/3.21.12: Forced build from source
onnx/1.16.2: Forced build from source
Requirements
    onnx/1.16.2#b2afe58c5c3d85542d1aeef169586f42:5e1bc6b93b3e68325f1abcbd6089c1762f838cd6 - Build
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88:aaa72c83801e95e6e3bed8abdf697c032f7b3ddd - Build
    zlib/1.3.1#cac0f6daea041b0ccf42934163defb20:ca639360bb681229d056d23829e74b0c62d596de - Build
Build requirements
    ccache/4.12.1#9cd51178555f4babb32fc0b46aa9b765:b0a3ef072d706ab8c266c04a37fb5bf38b7b5817 - Build
    cmake/4.1.2 - Platform
    protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88:aaa72c83801e95e6e3bed8abdf697c032f7b3ddd - Build
    zlib/1.3.1#cac0f6daea041b0ccf42934163defb20:ca639360bb681229d056d23829e74b0c62d596de - Build
~/c/m/r/l/all (pgi/libtorch/2.9.1|✔) 
$ conan list --graph graph.json --graph-context=build-only
Local Cache
  ccache
    ccache/4.12.1
      revisions
        9cd51178555f4babb32fc0b46aa9b765 (2025-10-31 11:45:14 UTC)
          packages
            b0a3ef072d706ab8c266c04a37fb5bf38b7b5817
              info
                settings
                  os: Macos
                  arch: universal

With the expected ccache entry (as in my case that's the only non-platform build-only requirement), so I'd expect that at least things like msys2 be present in your resulting package list. It does correctly identify the packages that are not being skipped as part of the proposed workaround, so it's interesting to see why the actual build-only ones do not show up. Is there a chance we can see the resulting packagelist.json?

AbrilRBS avatar Dec 10 '25 13:12 AbrilRBS

Is there a chance we can see the resulting packagelist.json?

here you are, and also the full graph:

graph.json pkglist.json

build URL: https://github.com/vcmi/vcmi-dependencies/actions/runs/20100794379/job/57670843048#step:14:419

P.S. The -c:a="tools.graph:skip_binaries=False" workaround has worked flawlessly!

kambala-decapitator avatar Dec 10 '25 14:12 kambala-decapitator