A just builded revision is removed from the cash and the dependency cannont be resolved

[brosmar]

What is your question?

Conan version 1.61.0 Python 3.9.2 gcc (GCC) 11.4.0 Docker Linux 7bc5e2533399 3.10.0-1160.108.1.el7.x86_64 #1 SMP Thu Jan 25 16:17:31 UTC 2024 x86_64 GNU/Linux debian bullsey

When calling conan create . master/dev --profile linux_x86_64_gcc11_release I am getting:

TMDS_DatabaseScripts/0.4.5@master/dev: WARN: The package TMDS_DatabaseScripts/0.4.5@master/dev:9b4bcd014fea222959933b85507e3c5fa203cc33 doesn't belong to the installed recipe revision, removing folder and a little late this module is missing.

[2024-02-21T15:59:30.500Z] TMDS_DatabaseScripts/0.4.5@master/dev: WARN: Can't find a 'TMDS_DatabaseScripts/0.4.5@master/dev' package for the specified settings, options and dependencies:
[2024-02-21T15:59:30.500Z] - Settings: arch=x86_64, arch_build=x86_64, build_type=Release, compiler=gcc, compiler.cppstd=gnu11, compiler.version=11, os=Linux
[2024-02-21T15:59:30.500Z] - Options: fPIC=True, shared=False, DBGN_Generate:fPIC=True, DBGN_Generate:shared=False, SYVE_Version:fPIC=True, SYVE_Version:shared=False
[2024-02-21T15:59:30.500Z] - Dependencies: DBGN_Generate/0.4.5@master/dev, SYVE_Version/0.4.5@master/dev
[2024-02-21T15:59:30.500Z] - Requirements: DBGN_Generate/0.4.5, SYVE_Version/0.4.5
[2024-02-21T15:59:30.500Z] - Package ID: 9b4bcd014fea222959933b85507e3c5fa203cc33

Even if it is available in the remote repo. A curious thing is that i had to force conan to upload it. A normal "conan upload DBGN_Generate/0.4.5@master/dev --all' does not upload the recepie with the just build profile.

conan search DBGN_Generate/0.4.5@master/dev -r artifactory-dev-local

Existing packages for recipe DBGN_Generate/0.4.5@master/dev:

Existing recipe in remote 'artifactory-dev-local':

    Package_ID: 4fda3338ff9db87f8fc041ff024cc67b0420eeac
        [options]
            fPIC: True
            shared: False
        [settings]
            arch: x86_64
            build_type: Release
            compiler: gcc
            compiler.cppstd: gnu11
            compiler.version: 11
            os: Linux
        [requires]
            SYVE_Version/0.4.5@master/dev:800f8f565640878f5c8781b91ff373cc180819df
        Outdated from recipe: False

What can be the reason for this?

[memsharded]

Hi @brosmar

Thanks for your report.

I think this might be an issue of the Conan cache in 1.X being able to only host 1 revision at a time. So if for some reason the same dependency graph tries to force different revisions of the same package, the latest one might evict the previous one. This should only happen in some very edge cases, like explicitly requires = "pkg/version#revision1" using different revisions in different places, that are also "hidden", so they avoid the conflicts.

It would be useful to know if the conan create . master/dev --profile linux_x86_64_gcc11_release, that . what package is it? Is it the TMDS_DatabaseScripts or another one? Is the TMDS_DatabaseScripts a dependency?

Then the other part is referring to yet another package DBGN_Generate, but it is not fully clear the relationship, that is probably a different thing, we better focus on one thing first.

Please let me know if the above issue about revisions rings a bell or something.

[brosmar]

Thank you for your verry answer. You are right that the referring was false. Maybe I am an a false path. I will investigate further.

[memsharded]

I am closing this ticket as staled, but please re-open or create new tickets for any further question, thanks!

[brosmar]

Hello memshared,

I have the impression that Conan does not search all repositories for a suitable version and may stop searching after the first unsuccessful attempt. After making the following change in the remotes.json file, the previously not found TMDS_DatabaseScripts/0.4.5@master/dev module is successfully found. The new xentric-conan-dev repository is now a virtual repository that includes the https://center.conan.io repo.

Old remotes.json

{
 "remotes": [
  {
   "name": "conan-center",
   "url": "https://center.conan.io",
   "verify_ssl": true
  },
  {
   "name": "artifactory",
   "url": "https://artifactory.mydomain.ext/artifactory/api/conan/xentric-conan-dev",
   "verify_ssl": true
  }
 ]
}

new remotes.json

{
 "remotes": [
  {
   "name": "artifactory",
   "url": "https://artifactory.mydomain.ext/artifactory/api/conan/xentric-conan-dev",
   "verify_ssl": true
  }
 ]
}

[memsharded]

The new xentric-conan-dev repository is now a virtual repository that includes the https://center.conan.io/ repo.

A quick suggestion: Using a virtual repo that includes ConanCenter as a "remote" repo is not necessarily the recommended way for production usage of ConanCenter packages. We recommend instead to use a regular local repository and build the packages yourself from the conan-center-index repo, please check: https://docs.conan.io/2/devops/using_conancenter.html and https://docs.conan.io/2/devops/conancenter/hosting_binaries.html

I have the impression that Conan does not search all repositories for a suitable version and may stop searching after the first unsuccessful attempt. After making the following change in the remotes.json file, the previously not found TMDS_DatabaseScripts/0.4.5@master/dev module is successfully found.

This might happen in Conan 1 in the case of a previous remote resolution was stored for that specific package, or if a -r=myremote argument is explicitly used. Another different case is if the remote is unavailable, which is considered an error and raises. If the remote is not available, it can be removed or disabled, but Conan is not expected to silently continue to the next available remote.

I have verified it, and Conan iterates all remotes until the package is found. Maybe you can try to define a bit more complete repro case:

• Starting from a blank cache
• Show the remotes list with conan remote list
• conan install your package (provide the full command just in case)
• Show the full output of the command

[brosmar]

Thank you for the hint with the conan-center-index. I will concider to use it. But never touch a running system :-) . With the obove changes the pipeline is running fine.

Definitive the upload of the modules in our case is always done with conan upload -r artifactory-dev-local ... I thought that this is nessesary to address the target repositiory.

I have more than 130 Conan 1.x projects to maintain. A manual migration to 2.0 is a laborious undertaking. Therefore, I see currently no possibility of switching to 2.0. Is there an automatic approach available?

[memsharded]

Thank you for the hint with the conan-center-index. I will concider to use it. But never touch a running system :-) . With the obove changes the pipeline is running fine.

If the system is working for you, that is ok. I was advising that because we have seen a few reports of unhappy users reporting things breaking due to new revisions and changes in ConanCenter, as the community is continuously contributing and updating things, and sometimes it might break existing users. The approach I was suggesting makes user builds isolated of these changes and more robust.

In any case, I'd like to comment that ConanCenter will soon stop getting updates for Conan 1. That means that it will not break, it will not remove packages or close, but the repo will stop getting new versions, new fixes, new configurations, etc, it will be completely frozen. Only Conan 2 packages will get updates.

Definitive the upload of the modules in our case is always done with conan upload -r artifactory-dev-local ... I thought that this is nessesary to address the target repositiory.

If the upload happens in the same machine, with the same cache as the other operations it could be related to the behavior you are seeing. The upload operation "ties" the package in Conan 1 to the remote you uploaded it. So subsequent operations will use that stored remote to resolve. The conan remote command (https://docs.conan.io/1/reference/commands/misc/remote.html) has the conan remote list_ref, conan remote add_ref, etc to manage this remote association with packages. Conan 2 has fully removed this association, and resolution is always iterative in all remotes.

I have more than 130 Conan 1.x projects to maintain. A manual migration to 2.0 is a laborious undertaking. Therefore, I see currently no possibility of switching to 2.0. Is there an automatic approach available?

There is no automatic approach, the main effort is updating the conanfile.py recipes. The syntax has been introduced in Conan 1.X many years ago, so it was possible to gradually update recipes to make them ready for Conan 2, spreading the effort over time. This is a guide that can help to make the recipes ready. In any case, the recommendation is to do the update while in Conan 1.X, step by step, and then run a Conan 2 pipeline in parallel to start verifying readiness.

Even if 130 recipes is a relatively big number if you are the only maintainer, I'd like to comment on a few aspects to encourage to work on the Conan 2 update:

• There are risks associated to staying in Conan 1 that are outside of our control. It is not that we are removing our Conan 1 application from PyPI, but if some Conan python dependencies become vulnerable or have similar issues, it might not be possible to fix it, as it would be breaking (there are some Conan 1.X dependencies that have upper version limits, and changing that would break)
• The "freeze" of new updates for ConanCenter for Conan 1, to be done in the following months.
• Development has been fully shifted to Conan 2, which was released 1 year and a half ago. Releases in Conan 1.X are now much slower and very minimal, just some very minor maintenance.
• Conan 2 is way superior version than Conan 1. Different aspects like build system integrations, environment management, lockfiles, etc have greatly improved.
• Not only the Conan 2.0 version was better than Conan1, but Conan 2 has released in these 18 months more very desired, important and awaited features than in the 2-3 previous years of Conan 1 combined:
  • Backup sources feature: https://blog.conan.io/2023/10/03/backup-sources-feature.html
  • Vendoring packages: https://blog.conan.io/2024/07/09/Introducing-vendoring-packages.html
  • Local-recipes-index: https://blog.conan.io/2024/04/23/Introducing-local-recipes-index-remote.html
  • New and much better CLion and Visual Studio plugins
  • New [replace_requires] and [platform_requires] features, and new conan graph explain command: https://blog.conan.io/2024/02/20/Conan-2-graph-features.html
  • A fully transparent conan-cmake integration with CMake dependency providers
  • conan cache save/restore packages: https://blog.conan.io/2023/11/28/Conan-new-features-2-0-14.html
  • Metadata files: https://blog.conan.io/2023/10/24/Conan-launches-metadata-files.html
  • Package lists: https://blog.conan.io/2023/06/28/Conan-bulk-package-operations.html

I agree that if things work, changing things should be carefully considered, but lots of pains in Conan 1 have been solved in Conan 2, so depending on those pain levels, it might be worth checking things.