[bug] `conan search` broken

[stefansli]

Environment Details (include every applicable attribute)

• Operating System+version: Windows 20H2
• Conan version: 1.51.0
• Python version: PyInstaller
• Artifactory EnterpriseX: 7.41.7

Steps to reproduce (Include if Applicable)

conan search on a Artifactory repository gives unexpected results. For some reason using the artifactory remote for conancenter does work.

We updated to conan 1.51.0 yesterday but also upgraded Artifactory yesterday. See output below.

Logs (Executed commands with output) (Include/Attach if Applicable)

> conan search zlib -r artifactory-repository
Existing package recipes:

apr/1.7.0
autoconf/2.71
automake/1.16.3
automake/1.16.4
b2/4.5.0
b2/4.8.0
bison/3.7.1
bison/3.7.6
boost/1.72.0
boost/1.74.0
boost/1.76.0
brotli/1.0.9
bzip2/1.0.8
catch2/2.13.7
ccache/4.6
cmake/3.21.4
cppcheck/2.7.5
eigen/3.3.8
expat/2.4.1
ffmpeg/4.2.1
ffmpeg/4.3.2
flex/2.6.4
fmt/6.0.0
fmt/6.1.2
fmt/7.1.0
fmt/7.1.3
fmt/8.1.1
freetype/2.10.4
freetype/2.11.0
freetype/2.11.1
gmp/6.2.1
gnu-config/cci.20201022
gtest/1.10.0
gtest/1.11.0
gtest/1.12.1
hdf5/1.8.21
hdf5/1.12.0
hdf5/1.12.1
jansson/2.13.1
json-schema-validator/2.1.0
libiconv/1.16
libmp3lame/3.100
libpng/1.6.37
libpq/13.3
libsodium/1.0.18
libtool/2.4.6
libvpx/1.9.0
libvpx/1.10.0
libvpx/1.11.0
libwebp/1.2.0
libwebp/1.2.1
libwebp/1.2.2
libx264/20191217
libx265/3.4
lz4/1.9.3
m4/1.4.18
m4/1.4.19
matio/1.5.23
meson/0.59.2
minizip-ng/3.0.6
mpir/3.0.0
nasm/2.15.05
ninja/1.10.2
nlohmann_json/3.7.3
nlohmann_json/3.9.1
nlohmann_json/3.10.4
nlohmann_json/3.10.5
ogg/1.3.4
openh264/2.1.1
openjpeg/2.4.0
openssl/1.1.1l
openssl/3.0.1
opus/1.3.1
pcre/8.44
pcre/8.45
picojson/1.3.0
pkgconf/1.7.4
protobuf/3.11.4
protobuf/3.17.1
pybind11/2.5.0
pybind11/2.8.1
pybind11/2.9.1
rapidjson/cci.20200410
sml/1.1.4
spdlog/1.8.5
strawberryperl/5.30.0.1
swig/4.0.1
tinyxml/2.6.2
uriparser/0.9.6
vorbis/1.3.7
winflexbison/2.5.24
xerces-c/3.2.2
xerces-c/3.2.3
xz_utils/5.2.5
yaml-cpp/0.7.0
yasm/1.3.0
z3/4.8.8
zeromq/4.3.2
zlib/1.2.11
zlib/1.2.12
zstd/1.5.0
zstd/1.5.1
zstd/1.5.2

This works:

conan search zlib -r conan-io
Existing package recipes:

zlib/1.2.8
zlib/1.2.11
zlib/1.2.12

[czoido]

Hi @vectorsli,

Trying with the Artifactory 7.41.7 version I can't reproduce the issue. Maybe you can use the CONAN_TRACE_FILE to provide some traces of the API calls?

Thanks a lot.

[stefansli]

@czoido Thanks for looking into this. Here is the log output. I'm afraid this doesn't contain much.

{"_action": "COMMAND", "name": "search_recipes", "parameters": {"case_sensitive": false, "remote_name": "company-3rdparty-prod"}, "time": 1659682129.1066523}
{"_action": "REST_API_CALL", "duration": 0.40967702865600586, "headers": {"User-Agent": "Conan/1.51.0 (Windows 10; Python 3.6.8; AMD64)", "X-Client-Anonymous-Id": "**********", "X-Client-Id": ""}, "method": "GET", "time": 1659682129.52233, "url": "https://vistrpndart1.vi.company.int/artifactory/api/conan/company-3rdparty-conan-prod/v1/ping"}
{"_action": "REST_API_CALL", "duration": 0.29264354705810547, "headers": {"User-Agent": "Conan/1.51.0 (Windows 10; Python 3.6.8; AMD64)", "X-Client-Anonymous-Id": "**********", "X-Client-Id": ""}, "method": "GET", "time": 1659682129.815975, "url": "https://vistrpndart1.vi.company.int/artifactory/api/conan/company-3rdparty-conan-prod/v2/conans/search?q=zlib"}
{"_action": "COMMAND", "name": "get_remote_by_name", "parameters": {}, "time": 1659682129.829976}
{"_action": "EXCEPTION", "class": "NoRemoteAvailable", "message": "No remote 'all' defined in remotes", "time": 1659682129.8350115}

When accessing the API URL directly, I get the same output. However when I use the local repository instead of the virtual, then the search output is as expected. So this looks like an Artifactory bug. However we can't change to local repositories because of a different bug in Artifactory: https://www.jfrog.com/jira/browse/RTFACT-27302

Curl to the API with the local repository:

> curl https://vistrpndart1.vi.company.int/artifactory/api/conan/company-3rdparty-conan-prod-local/v2/conans/search?q=zlib
{
  "results" : [ "zlib/1.2.11@_/_", "zlib/1.2.12@_/_" ]
}

[stefansli]

@czoido We just found out, that every conan install in our infrastructure is broken because of this bug. So this is very urgent for us!

To paint the picture:

• We use version ranges
• We use different remotes for dev/staging/production
• We use Artifactory and virtual repositories

Expected behavior:

The package bapdb does not exist in remote prod and therefore conan should continue the search with other remotes.

Actual Behavior:

Conan decides the package exists in prod but not with a correct version.

Here is the trace file:

{"_action": "COMMAND", "name": "install", "parameters": {"build": null, "channel": null, "conf": null, "env": null, "generators": null, "install_folder": null, "lockfile": null, "lockfile_out": null, "manifests": null, "manifests_interactive": null, "name": null, "no_imports": false, "options": null, "output_folder": null, "path": ".", "profile_build": [null, null, null, null, null], "profile_names": ["v2.0.0/windows-x86_64-vs-16-md-release"], "remote_name": null, "require_overrides": null, "settings": null, "update": false, "user": null, "verify": null, "version": null}, "time": 1660046300.1019986}
{"_action": "REST_API_CALL", "duration": 0.2191002368927002, "headers": {"User-Agent": "Conan/1.49.0 (Windows 10; Python 3.6.8; AMD64)", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "[email protected]"}, "method": "GET", "time": 1660046300.4370995, "url": "https://vistrpndart1.vi.company.int/artifactory/api/conan/conan-prod/v1/ping"}
{"_action": "REST_API_CALL", "duration": 0.0516664981842041, "headers": {"User-Agent": "Conan/1.49.0 (Windows 10; Python 3.6.8; AMD64)", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "[email protected]"}, "method": "GET", "time": 1660046300.491775, "url": "https://vistrpndart1.vi.company.int/artifactory/api/conan/conan-prod/v2/conans/search?q=bapdb%2F%2A%40company%2Fstable&ignorecase=False"}
{"_action": "EXCEPTION", "class": "ConanException", "message": "Version range '~17.0' from requirement 'bapdb/[~17.0]@company/stable' required by 'conanfile.txt' could not be resolved in remote 'prod'", "time": 1660046300.4957514}

Here is the output of the API Call:

> curl.exe "https://vistrpndart1.vi.company.int/artifactory/api/conan/conan-prod/v2/conans/search?q=bapdb%2F%2A%40company%2Fstable&ignorecase=False"
{
  "results" : [ "product2Emu/2.46.0@company/tag", "some-application/1.6.0@team/testing", "some-application/1.8.0@team/testing", "some-libraries-trace/1.6.0@team/testing", "some-libraries-trace/1.8.0@team/testing", "some-libraries-libsome-wrapper/1.6.0@team/testing", "some-libraries-libsome-wrapper/1.8.0@team/testing", "product-animation-transmitter/0.1.0@team/testing", "product-product2-interface/6.4.0-alpha1@team/testing", "product-product2-interface/6.4.0-alpha2@team/testing", "product-cmake-toolchains/0.1.4@team/testing", "product-conan-python/0.1.5@team/testing", "product-conan-python/0.1.6@team/testing", "product-protocol-connector/2.0.0@team/testing", "product-protocol-connector/2.1.0@team/testing", "product-hdf5/0.1.0@team/testing", "product-osi-bridge/0.3.1@team/testing", "product-osi-bridge/0.4.2@team/testing", "product-osi-bridge/0.5.0@team/testing", "product-tdm/0.1.0@team/testing", "opensut/2.0.0.0@company/stable" ]
}

[stefansli]

Maybe this Artifactory bug is related?

[czoido]

Hi @vectorsli,

Yes, It looks like something related to Artifactory. We are trying to raise the awareness of the Artifactory team on it and as far as I know they are working on it. Did you already comment to Artifactory support that you are having that issue or open a Jira ticket for this? That would help too. We'll do as much as we can to keep track on this and make sure that things keep moving to solve this and I'm sure that the Artifactory team will solve this as soon as possible.

[iieklund]

We are also affected by this. All installations are basically broken. Tried with multiple conan client versions:

ERROR: Loop detected in context host: '<package_123>' requires '<package_123> which is an ancestor too

[stefansli]

We just discovered the underlying issue. Artifactory is not generating the search index during the update process.

The index gets updated every time a package is deployed. However we also found that this API call generates the index as a by product (we only tested this with packages that do not exist, such as abc/123@user/channel):

curl https://artifactory-host/artifactory/api/conan/local-repository/v1/conans/abc/1.2.3/user/channel/download_urls

[memsharded]

Is this problem finally solved in Artifactory side? Several fixes have been done in Artifactory IIRC, this should be working, please let us know. Thanks!

[stefansli]

I cannot say if this is fixed on Artifactories side since this only occured in very specefic circumstances. The repository needed to be created before the update to the specific artifactory version. In addition any package upload seemed to fix the issue. I'd say if this wasn't fixed yet, it wouldn't be a big problem anymore since the problem fades away and somewhat fixes itself if one simply uses the repo.

[memsharded]

yeah, thanks for the feedback, I agree, and since we haven't had any other reports, I think this can be closed.