conan icon indicating copy to clipboard operation
conan copied to clipboard
verified publisher icon conan-io

[bug] Exception when using local recipes index

Open fschoenm opened this issue 1 year ago • 7 comments

Describe the bug

I'm trying to follow the steps in https://blog.conan.io/2024/04/23/Introducing-local-recipes-index-remote.html to use a local recipes index with our own recipes. However, I'm running into an exception and I'm not sure where it comes from or what/if I'm doing wrong.

The package I'm experimenting with is a very simple header-only lib that we have used before. I'm just trying to install it from the local recipes index as described in the blog article above. Unfortunately, the error message is not really helpful.

My folder structure looks like this:

recipes/dtoa/
recipes/dtoa/all/
recipes/dtoa/all/conanfile.py
recipes/dtoa/all/dtoa.patch
recipes/dtoa/config.yml

Remote setup:

$ conan remote list
recipes: /home/fschoenm/devel/conan-recipes [local-recipes-index, Enabled: True]

I can also search the local recipes index:

$ conan list -r recipes *
Found 1 pkg/version recipes matching * in recipes
recipes
  dtoa
    dtoa/1.0.4

However, the installation fails. The exception when running conan install looks like this:

$ conan install --requires dtoa/1.0.4 -vvv -r recipes

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
os.glibc=2.35
[platform_tool_requires]
cmake/3.29
[conf]
tools.cmake.cmake_layout:build_folder_vars=['settings.build_type', 'settings.arch']
tools.cmake.cmaketoolchain:generator=Ninja

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
os.glibc=2.35
[platform_tool_requires]
cmake/3.29
[conf]
tools.cmake.cmake_layout:build_folder_vars=['settings.build_type', 'settings.arch']
tools.cmake.cmaketoolchain:generator=Ninja


======== Computing dependency graph ========
dtoa/1.0.4: Not found in local cache, looking in remotes...
dtoa/1.0.4: Checking remote: recipes
Internal export for dtoa/1.0.4:

Graph root
    cli
Traceback (most recent call last):
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/proxy.py", line 39, in _get_recipe
    recipe_layout = self._cache.recipe_layout(reference)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/cache/cache.py", line 58, in recipe_layout
    return self._data_cache.get_recipe_layout(ref)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conan/internal/cache/cache.py", line 97, in get_recipe_layout
    ref_data = self._db.get_latest_recipe(ref)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conan/internal/cache/db/cache_database.py", line 77, in get_latest_recipe
    return self._recipes.get_latest_recipe(ref)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conan/internal/cache/db/recipes_table.py", line 122, in get_latest_recipe
    raise ConanReferenceDoesNotExistInDB(f"Recipe '{ref}' not found")
conans.errors.ConanReferenceDoesNotExistInDB: Recipe 'dtoa/1.0.4' not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/rest_client_local_recipe_index.py", line 145, in _export_recipe
    new_ref, _ = cmd_export(self._app, global_conf, conanfile_path,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/cmd/export.py", line 21, in cmd_export
    conanfile = loader.load_export(conanfile_path, name, version, user, channel, graph_lock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/loader.py", line 143, in load_export
    conanfile = self.load_named(conanfile_path, name, version, user, channel, graph_lock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/loader.py", line 102, in load_named
    conanfile, _ = self.load_basic_module(conanfile_path, graph_lock, remotes=remotes,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/loader.py", line 64, in load_basic_module
    self._pyreq_loader.load_py_requires(conanfile, self, graph_lock, remotes,
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/python_requires.py", line 81, in load_py_requires
    py_requires = self._resolve_py_requires(py_requires_refs, graph_lock, loader, remotes,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/python_requires.py", line 99, in _resolve_py_requires
    resolved_ref = self._resolve_ref(requirement, graph_lock, remotes, update)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/python_requires.py", line 118, in _resolve_ref
    self._range_resolver.resolve(requirement, "python_requires", remotes, update)
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/range_resolver.py", line 36, in resolve
    remote_resolved_ref = self._resolve_remote(search_ref, version_range, remotes, update)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/range_resolver.py", line 83, in _resolve_remote
    for remote in remotes:
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/graph_builder.py", line 344, in _create_new_node
    resolved = self._resolve_recipe(require.ref, graph_lock)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/graph_builder.py", line 242, in _resolve_recipe
    result = self._proxy.get_recipe(ref, self._remotes, self._update, self._check_update)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/proxy.py", line 25, in get_recipe
    resolved = self._get_recipe(ref, remotes, update, check_update)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/proxy.py", line 44, in _get_recipe
    layout, remote = self._download_recipe(reference, remotes, output, update, check_update)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/proxy.py", line 130, in _download_recipe
    remote, latest_rref = self._find_newest_recipe_in_remotes(ref, remotes, update, check_update)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/proxy.py", line 107, in _find_newest_recipe_in_remotes
    ref = self._remote_manager.get_latest_recipe_reference(reference, remote)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/remote_manager.py", line 223, in get_latest_recipe_reference
    return self._call_remote(remote, "get_latest_recipe_reference", ref)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/remote_manager.py", line 255, in _call_remote
    return local_folder_remote.call_method(method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/rest_client_local_recipe_index.py", line 71, in call_method
    return getattr(self, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/rest_client_local_recipe_index.py", line 122, in get_latest_recipe_reference
    ref = self._export_recipe(ref)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/rest_client_local_recipe_index.py", line 148, in _export_recipe
    raise ConanException(f"Error while exporting recipe from remote: {self._remote.name}\n"
conans.errors.ConanException: Error while exporting recipe from remote: recipes
'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conan/cli/cli.py", line 193, in run
    command.run(self._conan_api, args[0][1:])
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conan/cli/command.py", line 164, in run
    info = self._method(conan_api, parser, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conan/cli/commands/install.py", line 67, in install
    deps_graph.report_graph_error()
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/graph.py", line 395, in report_graph_error
    raise self.error
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/graph_builder.py", line 53, in load_graph
    new_node = self._expand_require(require, node, dep_graph, profile_host,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/graph_builder.py", line 97, in _expand_require
    new_node = self._create_new_node(node, require, graph, profile_host, profile_build,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fschoenm/.local/pipx/venvs/conan/lib/python3.12/site-packages/conans/client/graph/graph_builder.py", line 346, in _create_new_node
    raise GraphMissingError(node, require, str(e))
conans.client.graph.graph_error.GraphMissingError: Package 'dtoa/1.0.4' not resolved: Error while exporting recipe from remote: recipes
'NoneType' object is not iterable.

ERROR: Package 'dtoa/1.0.4' not resolved: Error while exporting recipe from remote: recipes
'NoneType' object is not iterable.

Environment

version: 2.3.2
conan_path: /home/fschoenm/.local/bin/conan
python
  version: 3.12.3
  sys_version: 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0]
  sys_executable: /home/fschoenm/.local/pipx/venvs/conan/bin/python
  is_frozen: False
  architecture: x86_64
system
  version: #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024
  platform: Linux-6.8.0-31-generic-x86_64-with-glibc2.39
  system: Linux
  release: 6.8.0-31-generic
  cpu: x86_64

How to reproduce it

No response

fschoenm avatar Jun 04 '24 09:06 fschoenm

Hi @fschoenm

Thanks for your report. A couple of quick questions:

  • Is the error deterministic? Does it happen always, or just with this particular recipe? For this recipe, it also happens always?
  • Could you please share with us the contents of dtoa, or at least a simplified version that would trigger this?

Many thanks!

memsharded avatar Jun 06 '24 13:06 memsharded

@memsharded Yes it always happens and also with the new 2.4.0 release. I haven't tried with another recipe yet but this one is extremely simple (see below).

I have the feeling it has to do with something else (in my ~/.conan2 directory?) because it complains about a conans.errors.ConanReferenceDoesNotExistInDB error and refers to conan functions working in the cache.

config.yml

versions:
  "1.0.4":
    folder: all

conanfile.py

import os

from conan import ConanFile
from conan.tools.files import copy
from conan.tools.scm import Git


class DtoaConan(ConanFile):
    name = "dtoa"
    version = "1.0.4"
    license = "MIT License"
    url = "http://10.30.21.21:7990/scm/trsb/dtoa"
    description = "C++ double-to-string conversion"

    python_requires = "py_license_helper/[>=1.0]@vsdk/stable"

    def source(self):
        scm = {
            "type": "git",
            "url": "http://10.30.21.21:7990/scm/trsb/dtoa.git",
            "revision": "v1.0.4",
        }

        git = Git(self)
        git.clone(url=scm["url"], target=self.source_folder)
        git.checkout(commit=scm["revision"])

        license_helper = self.python_requires["py_license_helper"].module
        license_helper.create_license_file(self.name, self.license, license_helper.read_license_file("license.txt"))

    def package(self):
        copy(
            self,
            "dtoa_milo.h",
            os.path.join(self.source_folder, "src/milo"),
            os.path.join(self.package_folder, "include", "milo"),
            keep_path=False,
        )
        copy(self, "*.lic", self.source_folder, os.path.join(self.package_folder, "licenses"), keep_path=False)

    def package_id(self):
        self.info.clear()

fschoenm avatar Jun 06 '24 13:06 fschoenm

Thanks for sharing, I suspect it might have something to do with the python_requires, as they don't exist in ConanCenter, it is possible that there is something there that hasn't been considered. I also realized now that the trace also reports about python_requires.py", line 118, in _resolve_ref, I had not seen that before.

I'll try to reproduce in a test.

memsharded avatar Jun 06 '24 14:06 memsharded

@memsharded You're right, if I remove the python_requires property (and related code), it works. However, running conan create manually on the recipe works also with the Python requirements.

fschoenm avatar Jun 06 '24 15:06 fschoenm

Ok, here it goes: https://github.com/conan-io/conan/pull/16420

It is a test, and a potential fix, as a draft. Indeed the issue was that local-recipes-index was designed for ConanCenter-like, and python-requires are not supported there.

There is a limitation, is that the python_requires must be available in the same repo, because the way it works it needs an internal evaluation within the repo, and that requires the python_requires to be evaluated in that closed context (no access to other remotes).

memsharded avatar Jun 06 '24 15:06 memsharded

Hi @fschoenm

Regarding the fix in https://github.com/conan-io/conan/pull/16420, can you please confirm if the python_requires lives in the same "local-recipes-index" repo than the recipes using it? Seems a difficult to change thing, but for python-requires in the same repo, https://github.com/conan-io/conan/pull/16420 might be a viable solution.

memsharded avatar Jun 10 '24 09:06 memsharded

Regarding the fix in #16420, can you please confirm if the python_requires lives in the same "local-recipes-index" repo than the recipes using it? Seems a difficult to change thing, but for python-requires in the same repo, #16420 might be a viable solution.

Hi @memsharded, yes, it's going to be part of the same repository (at least in this and most cases). I guess for specific other conan packages that use the same shared python-requires, we can keep living with the old method of running conan create directly on the conanfile.

fschoenm avatar Jun 10 '24 09:06 fschoenm