conan icon indicating copy to clipboard operation
conan copied to clipboard

Published 20 hours ago • verified publisher icon conan-io

How can I learn what's missing / wrong for this missing package (Possible defect with 'conan cache save|restore')

Open RochaStratovan opened this issue 1 year ago • 2 comments

What is your question?

How can I learn what's missing / wrong for this missing package

I created a tool_requires package that is os, compiler, build_type and arch agnostic. It is a "copy" of all the Windows .bat and Linux .sh scripts that are used in our build environment.

The basics for this tool_requires package is

class SvBuildToolsRecipe(ConanFile):
    name = "sv_build_tools"
    package_type = "application"

    repo_dir = "BuildTools"

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"
    python_requires = "stvn_recipes/[>=0.0]"


    def package_id(self):
        del self.info.settings.os
        del self.info.settings.compiler
        del self.info.settings.build_type
        del self.info.settings.arch

Documentation indicated that deleting the settings values basically make this a "clean" package_id.


I created this package on a Windows machine with eol set to lf.

I exported the package using conan cache save sv_build_tools.

I copied the tarball to a Linux Ubuntu 20 machine.

I imported the cache into the local cache using conan cache restore <tarball>, and observed that the package is now in my local cache.

However, if I try to use the package it fails with the message shown below

======== Computing dependency graph ========
Graph root
    conanfile.py (zlib/1.2.5): /home/stratarocha/Desktop/TEST-CONAN-PY-REQUIRES/ZLIB/conanfile.py
Build requirements
    sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f - Cache
Python requires
    stvn_recipes/1.0#8f709de1bcde37e0554f926191a4bdac - Cache
Resolved version ranges
    stvn_recipes/[>=0.0]: stvn_recipes/1.0
    sv_build_tools/[>=0.0]: sv_build_tools/1.0

======== Computing necessary packages ========
sv_build_tools/1.0: Compatible package ID 67a29b67316ef7c34b92640b41a667c370fb9dd2 equal to the default package ID: Skipping it.
Build requirements
    sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f:67a29b67316ef7c34b92640b41a667c370fb9dd2 - Missing

======== Installing packages ========
ERROR: Missing binary: sv_build_tools/1.0:67a29b67316ef7c34b92640b41a667c370fb9dd2

sv_build_tools/1.0: WARN: Can't find a 'sv_build_tools/1.0' package binary '67a29b67316ef7c34b92640b41a667c370fb9dd2' for the configuration:
[python_requires]
stvn_recipes/1.0.Z

ERROR: Missing prebuilt package for 'sv_build_tools/1.0'. You can try:
    - List all available packages using 'conan list sv_build_tools/1.0:* -r=remote'
    - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
    - Try to build locally from sources using the '--build=sv_build_tools/1.0' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

I read the specified web page but it doesn't help me.

The output from the conan graph explain follows


======== Computing dependency graph ========
Graph root
    conanfile.py (zlib/1.2.5): /home/stratarocha/Desktop/TEST-CONAN-PY-REQUIRES/ZLIB/conanfile.py
Build requirements
    sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f - Cache
Python requires
    stvn_recipes/1.0#8f709de1bcde37e0554f926191a4bdac - Cache
Resolved version ranges
    stvn_recipes/[>=0.0]: stvn_recipes/1.0
    sv_build_tools/[>=0.0]: sv_build_tools/1.0

======== Computing necessary packages ========
sv_build_tools/1.0: Compatible package ID 67a29b67316ef7c34b92640b41a667c370fb9dd2 equal to the default package ID: Skipping it.
Build requirements
    sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f:67a29b67316ef7c34b92640b41a667c370fb9dd2 - Missing

======== Retrieving and computing closest binaries ========
Missing binary: sv_build_tools/1.0
With conaninfo.txt (package_id):
[python_requires]
stvn_recipes/1.0.Z

Finding binaries in the cache
Finding binaries in remote conancenter

======== Closest binaries ========
sv_build_tools/1.0
  sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f%1707953883.7342465 (2024-02-14 23:38:03 UTC)

From the above output I see:

# It is looking for this
Build requirements
    sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f:67a29b67316ef7c34b92640b41a667c370fb9dd2 - Missing

# but it finds this
sv_build_tools/1.0
  sv_build_tools/1.0#e88720518e44ce08bd851c9e74297e7f%1707953883.7342465 (2024-02-14 23:38:03 UTC)

The first part of the hash match but one switches to colon (:) and the other to percent (%).

What can I check for this next?

I do not have the option of building this package locally on the Linux system at this time. I'd like to learn more about this lookup up mechanism and what's causing this confusion/breakage.

Have you read the CONTRIBUTING guide?

  • [X] I've read the CONTRIBUTING guide

RochaStratovan avatar Feb 15 '24 00:02 RochaStratovan

Additional Information.

I tried the following changes and it still fails:

  1. Test 1: Apply changes based on reading the header-only library example

    1. Remove the settings from the recipe
    2. remove the package_type,
    3. change cpp_info.{includedirs|libdirs|bindirs} to [] since I have none of these.

  2. Test 2: Add package_type as build-scripts keeping the other changes

In the readings I saw a syntax for the list command #:* to see additional information. When I do this it seems like conan cache isn't caching everything more below.

Step 1: View the cache on Windows machine

Output 
$ conan list "*#:*"
Local Cache
  stvn_recipes
    stvn_recipes/1.0
      revisions
        8f709de1bcde37e0554f926191a4bdac (2024-02-15 21:37:51 UTC)
          packages
> sv_build_tools
>   sv_build_tools/1.0
>     revisions
>       555aca21b7764c355471e8f99d4fa42c (2024-02-15 22:10:26 UTC)
>         packages
>           67a29b67316ef7c34b92640b41a667c370fb9dd2
>             info
>               python_requires
>                 stvn_recipes/1.0.Z

sv_build_tools is the package of interest.


Step 2: Create the cache

Output 
$ conan cache save "sv_build_tools"
Saving sv_build_tools/1.0: p/sv_bu0ac2091f0fe09
Local Cache
  sv_build_tools
    sv_build_tools/1.0
      revisions
        555aca21b7764c355471e8f99d4fa42c (2024-02-15 22:10:26 UTC)
          recipe_folder: p/sv_bu0ac2091f0fe09

Step 3: Export the cache to the Linux machine

Output 
$ conan cache restore /mnt/SHARED/conan_cache_save.tgz 
Restore: sv_build_tools/1.0 in p/sv_bu0ac2091f0fe09
Local Cache
  sv_build_tools
    sv_build_tools/1.0
      revisions
        555aca21b7764c355471e8f99d4fa42c (2024-02-15 22:24:48 UTC)
          recipe_folder: p/sv_bu0ac2091f0fe09

Step 4: List the cache on the Linux machine

Output 
$ conan list "*#:*"
Local Cache
  stvn_recipes
    stvn_recipes/1.0
      revisions
        8f709de1bcde37e0554f926191a4bdac (2024-02-14 23:51:37 UTC)
          packages
  sv_build_tools
    sv_build_tools/1.0
      revisions
        555aca21b7764c355471e8f99d4fa42c (2024-02-15 22:24:48 UTC)
          packages

Notice that the Linux output doesn't have anything listed under packages while the Windows output does.

Step 4: conan install for a package that needs sv_build_tools fails

Output 
$ conan install ZLIB --version 1.2.5

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu14
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux


======== Computing dependency graph ========
WARN: Unrecognized version range option "visible=False" in ">=0.0, visible=False"
Graph root
    conanfile.py (zlib/1.2.5): /home/stratarocha/Desktop/TEST-CONAN-PY-REQUIRES/ZLIB/conanfile.py
Build requirements
    sv_build_tools/1.0#555aca21b7764c355471e8f99d4fa42c - Cache
Python requires
    stvn_recipes/1.0#8f709de1bcde37e0554f926191a4bdac - Cache
Resolved version ranges
    stvn_recipes/[>=0.0, visible=False]: stvn_recipes/1.0
    sv_build_tools/[>=0.0]: sv_build_tools/1.0

======== Computing necessary packages ========
Build requirements
    sv_build_tools/1.0#555aca21b7764c355471e8f99d4fa42c:67a29b67316ef7c34b92640b41a667c370fb9dd2 - Missing

======== Installing packages ========
ERROR: Missing binary: sv_build_tools/1.0:67a29b67316ef7c34b92640b41a667c370fb9dd2

sv_build_tools/1.0: WARN: Can't find a 'sv_build_tools/1.0' package binary '67a29b67316ef7c34b92640b41a667c370fb9dd2' for the configuration:
[python_requires]
stvn_recipes/1.0.Z

ERROR: Missing prebuilt package for 'sv_build_tools/1.0'. You can try:
    - List all available packages using 'conan list sv_build_tools/1.0:* -r=remote'
    - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
    - Try to build locally from sources using the '--build=sv_build_tools/1.0' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

The piece it says it can't find is 67a29b67316ef7c34b92640b41a667c370fb9dd2 which appears in the Windows output but not the Linux output.

RochaStratovan avatar Feb 15 '24 22:02 RochaStratovan

Additional Information # 2

I was able to stand up an Artifactory server and upload these packages there. This worked, leading me to believe this is a defect with the conan cache tool.

RochaStratovan avatar Feb 15 '24 23:02 RochaStratovan

Hi @RochaStratovan

Sorry this fell through the cracks...

Documentation indicated that deleting the settings values basically make this a "clean" package_id.

Not completely. Dependencies and options also matter. In this case it doesn't have options, but it has dependencies to a python_requires. Is this python-requires really necessary for this recipe? It is possible that if the python_requires bumps its minor version, it will result in a new package_id, but this doesn't look the case from the logs. In any case, doing self.info.clear() is a better guarantee that the package_id will be independent of all inputs, settings, options, confs, dependencies.

I think the issue is related to:

$ conan cache save "sv_build_tools"

The save follow the same rules as list. If you don't save something like sv_build_tools/version:* it will only save the recipes, not the binaries. The output of $ conan cache save "sv_build_tools" is already not showing binaries at all.

Can you please try that and let us know? Thanks!

memsharded avatar Mar 13 '24 09:03 memsharded

@memsharded,

Thank you. I can confirm that If I use

conan cache save "sv_build_tools/*:*"

it will work.

I guess I don't fully undestand all the nuances of the conan list command. I'll ask about that in a separate issue.

TL;DR

This issue was cause by user error. I need to use command syntax that saves the binaries as well as the recipes.

RochaStratovan avatar Mar 14 '24 17:03 RochaStratovan