New ``local-recipes-index`` feature

[memsharded]

trafficstars

Changelog: Feature: Allow adding a folder with a clone of conan-center-index or same folder structure as a remote. Docs: https://github.com/conan-io/docs/pull/XXXX

How to use it:

$ git clone [email protected]:conan-io/conan-center-index
$ conan remote add c3ifork file:///path/to/cloned/conan-center-index
// use it as normal
$ conan search opencv*
$ conan graph info --requires=opencv/4.5.5 -r=c3ifork --format=html > file.html

[memsharded]

Right now running conan list ":" -r=local for the current head of cci fails because it finds Conan 1-only recipes and does not get past that. A try-catch in the select() method inner loop of subapi/list.py might be enough to skip those cases?

I have improved that by running a quick check of from conans in recipes, seems to be enough atm. I am also caching the result of the export to avoid re-exporting all the time.

[AbrilRBS]

To recap, some alternatives that might be nice to look into:

• conan remote add-local /home/barbarian/conan-center-index/
• conan remote add /home/barbarian/conan-center-index/ --local
• conan remote add /home/barbarian/conan-center-index/, @czoido suggests that if no "real" remote can have the protocol missing, there's no overlap between /absolute/paths and http(s)

I think the UX feels better this way, and less confusion about protocols (@czoido justo had the same issue about the number of / after the file: I experienced last week)

[czoido]

This is supercool, my only comments are about changing the UX a bit, I would vote for adding an argument or not adding anything and distinguish between them with the http:

[memsharded]

Lets add a explicit argument, just in case in the future we want to support the flow clone + use, and then we need a URL

[memsharded]

Added conan remote add /path/to/repo --type=local to allow future flexibility and have full command irrespective of url input arg.

Name --type and "local" can be discussed, of course.

[patmantru]

Request: rely on --type, don't decide based on file:// which would require a local clone of the repo, which isn't desirable in all use cases.

That would allow for something like: https://github.com/mycompany/conan-recipes --type=git file://localhost/conan-recipes --type=file

Please?

[memsharded]

Thanks for the feedback @patmantru This is something that we want to do, but it is still a bit early. Note that one of the main advantages of using the conan-center-index structure is that users using a fork can perfectly reproduce things by checking out any desired commit. Specifying just the URL of the git remote implies using the latest HEAD main branch, but that opens many other questions and feature requests:

• Automatically updating the fork to latest. If this is a pull, it can create conflicts.
• Being able to specify branches, tags, commits in the remote definition
• Automate the push of the repo for conan upload to that repo
• Cloning to a different location, to use different clones (what would be the location in the first place?)

So we think it is better to start with the lower level (folder in disk) until the core functionality, which is using packages from there is more stable, and after that and with the learnings from many more users, we will also know better different use cases about automating the clone and checkout from git remote.

[czoido]

After testing this a bit, two comments:

• I would prefer that the recipes are not cached. For my use case, mostly for testing fixes for conflicts in Conan Center, the ideal is that every change in a recipe is taken when I do a new conan install

• Also, I noticed that it seems that using this branch Conan does not to export the boost recipe correctly, because that recipe exports a yaml with dependencies that does not seem to be exported to the cache:

$ conan install --requires=boost/1.83.0 


======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=15
os=Macos

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=15
os=Macos


======== Computing dependency graph ========
boost/1.83.0: Not found in local cache, looking in remotes...
boost/1.83.0: Checking remote: c3ifork
Traceback (most recent call last):
  File "/Users/carlosz/.pyenv/versions/3.9.12/lib/python3.9/shutil.py", line 823, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.conan2/p/boost8fa1c3c0f8b8f/d/dependencies/dependencies-1.83.0.yml' -> '/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.conan2/p/boost8fa1c3c0f8b8f/e/dependencies/dependencies-1.83.0.yml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conan/cli/cli.py", line 272, in main
    cli.run(args)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conan/cli/cli.py", line 172, in run
    command.run(self._conan_api, self._commands[command_argument].parser, args[0][1:])
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conan/cli/command.py", line 134, in run
    info = self._method(conan_api, parser, *args)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conan/cli/commands/install.py", line 69, in install
    deps_graph = conan_api.graph.load_graph_requires(args.requires, args.tool_requires,
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conan/api/subapi/graph.py", line 126, in load_graph_requires
    deps_graph = self.load_graph(root_node, profile_host=profile_host,
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conan/api/subapi/graph.py", line 175, in load_graph
    deps_graph = builder.load_graph(root_node, profile_host, profile_build, lockfile)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/graph_builder.py", line 50, in load_graph
    new_node = self._expand_require(require, node, dep_graph, profile_host,
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/graph_builder.py", line 90, in _expand_require
    new_node = self._create_new_node(node, require, graph, profile_host, profile_build,
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/graph_builder.py", line 260, in _create_new_node
    resolved = self._resolve_recipe(require.ref, graph_lock)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/graph_builder.py", line 208, in _resolve_recipe
    result = self._proxy.get_recipe(ref, self._remotes, self._update, self._check_update)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/proxy.py", line 23, in get_recipe
    resolved = self._get_recipe(ref, remotes, update, check_update)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/proxy.py", line 41, in _get_recipe
    remote, new_ref = self._download_recipe(reference, remotes, output, update, check_update)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/proxy.py", line 133, in _download_recipe
    self._download(latest_rref, remote)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/graph/proxy.py", line 138, in _download
    self._remote_manager.get_recipe(ref, remote)
  File "/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.env/lib/python3.9/site-packages/conans/client/remote_manager.py", line 86, in get_recipe
    shutil.move(file_path, os.path.join(export_folder, file_name))
  File "/Users/carlosz/.pyenv/versions/3.9.12/lib/python3.9/shutil.py", line 843, in move
    copy_function(src, real_dst)
  File "/Users/carlosz/.pyenv/versions/3.9.12/lib/python3.9/shutil.py", line 444, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/carlosz/.pyenv/versions/3.9.12/lib/python3.9/shutil.py", line 266, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.conan2/p/boost8fa1c3c0f8b8f/e/dependencies/dependencies-1.83.0.yml'

ERROR: [Errno 2] No such file or directory: '/Users/carlosz/Documents/developer/conan/git-remotes-demo/conan-center-index/.conan-home/.conan2/p/boost8fa1c3c0f8b8f/e/dependencies/dependencies-1.83.0.yml'

[tsondergaard]

Instead of requiring a local clone in the filesystem, could this allow me to specify just a git URL? For example:

conan remote add tsondergaard-cci https://github.com/tsondergaard/conan-center-index.git

With this setup can I have recipes in git and binaries in Artifactory?

[memsharded]

Instead of requiring a local clone in the filesystem, could this allow me to specify just a git URL? For example:

We have considered, even implemented it, and reverted. A couple of reasons:

• Fetching always the "head" and using it is not that great. Users want stability, being able to use a specific tag, branch or commit in many many cases. And that required to invent a whole DSL to define it: conan remote add repo git-url --tag=mytag and the like, really dirty thing
• Another very important use case, probably the most important is also being able to locally edit the recipes, and have the changes immediately available. This require a "user space" clone that users have simple access. If conan remote adds git urls, then those clones need to be in a private Conan folder, Conan can't be doing clones in user space.
• The control and semantics of updating/pulling/merging is also lost, or Conan would also need to invent yet another DSL around remotes like conan remote myremote --pull

So it was just like creating a more limited, poorer layer over git repos and commands, while also not allowing users to modify sources, and it was dropped, not really worth it. Looks like convenient at first sight, but in practice it doesn't work great for majority of cases.

With this setup can I have recipes in git and binaries in Artifactory?

Yes, Conan 2 binary management allows to use recipes from one repo (including this new local-recipes-index one) and binaries from other repos, because using revisions everywhere to guarantee consistency it allows to do so and guarantee that the binaries from one repo belongs to the same source recipe (recipe revision). Note that when binaries are uploaded to Artifactory, the recipe is also uploaded there together with the binary, so once this happens, it is no longer needed to use both and users can also get recipes and binaries from server repo too. This would be a very typical setup, and we already have many users that are doing this approach, even without the local-recipes-index: they work with a fork of conan-center-index, build their binaries from there and upload to their servers, and their developers and CI are just "consumers" of their own recipes+binaries. This is the approach documented in: https://docs.conan.io/2/devops/using_conancenter.html

[tsondergaard]

Vcpkg suggests consuming projects add vcpkg as a git submodule. Would that pattern work with the mechanism you are building here? Let's say my company maintains a fork of conan-center-index at https://github.com/acme/conan-index.git that is extended also to contain our own libraries. Could consuming applications then add that as a git submodule and use that as the repository as the source registry and then artifactory would only be used for binary packages?

[memsharded]

Would that pattern work with the mechanism you are building here?

Yes, this would work with this new local-recipes-index repo too, it uses a very similar approach. Nevertheless, it is possible that this approach is not the best in all scenarios, depending on the needs.

that is extended also to contain our own libraries

The local-recipes-index is intended for third party libraries. For packaging your own libraries, the approach of putting the conanfile.py directly in the repo might work much better, specially if you need to maintain different versions of your packages, because in that way the recipe evolves in parallel to the code, and it enables simpler flows like git clone + conan install + cmake, with just one single repo, no submodules. Git submodules management is not that straightforward and has issues too.