[question]The conan upload operation intermittently fails when executed concurrently for three build types.

[AroshikaFernando]

What is your question?

Environment Details:

OS: RHEL 8.10 Conan: 2.16.0 Artifactory Instances:

Artifactory 1: 7.126.0 Artifactory 2: 7.111.9

Note: I am using two different versions and domains of JFrog Artifactory.

Scenario: I have three parallel GitLab pipeline jobs building packages for the same Conan recipe (same RREV) but for three different build types: debug, optimized, and optdebug. Each build uses a separate Conan cache, and the jobs set the relative build-type cache via CONAN_HOME. Folder Structure: /tmp/ | -- solution_debug | -- | -- project_a/src | -- | -- .conan_debug | -- | -- | -- .conan -> profiles/default -> build_type=debug | | -- solution_optimized | -- | -- project_a/src | -- | -- .conan_optimized | -- | -- | -- .conan -> profiles/default -> build_type=optimized | | -- solution_optdebug | -- | -- project_a/src | -- | -- .conan_optdebug | -- | -- | .conan -> profiles/default -> build_type=optdebug

When running three parallel GitLab jobs for these build types, two jobs succeed while one fails with the following error:

conan upload -r app-01110-conan-rel-federated platform_auth_messaging/5.14.0.t3@auth_1/tools_testing --format=json --out-file=platform_auth_messaging_output.json

platform_auth_messaging/5.14.0.t3@auth_1/tools_testing: Uploading recipe 'platform_auth_messaging/5.14.0.t3@auth_1/tools_testing#d123a679c924994762463fea68ac1e0d' (1.0KB)
ERROR: Permission denied for user: 'user1': 403: Artifact deletion error: Not enough permissions to delete/overwrite all artifacts under 'app-01110-conan-rel-federated/auth_1/platform_auth_messaging/5.14.0.t3/tools_testing/d123a679c924994762463fea68ac1e0d/export/conan_export.tgz' (user: 'user1' needs DELETE permission).. [Remote: app-01110-conan-rel-federated]

When I rerun the failed job, it succeeds. This behaviour occurs on both Artifactory instances mentioned above.

Permissions: Artifactory 1: user1 has delete/overwrite permission Artifactory 2: user1 does not have delete/overwrite permission

I have tried following conan upload retry also. But it didn't fix my issue. .conan2/global.conf core.upload:retry=3 core.upload:retry_wait=8

Could you please advise on the root cause or any recommended actions?

Thank you, Mareena Fernando

Have you read the CONTRIBUTING guide?

• [x] I've read the CONTRIBUTING guide

[memsharded]

Hi @AroshikaFernando

Thanks for the feedback. This race condition can happen when uploading concurrently the "recipe" part (the export package of Artifactory) of a Conan package from multiple places for the first time. See ticket https://github.com/conan-io/conan/issues/16703, and this is something that we would like to improve at some point, but it will take some time.

In the meantime, the best way to approach this is something like:

• before launching the parallell jobs to build the different binary flavors, do a non-parallel step that does just the conan export . + conan upload of that thing
• Then you can launch the parallel builds and uploads. As the recipe part is already there, there will be no race conditions.

A simpler, pragmatic but yet effective approach we have seen users following is implementing in the CI a sleep for some seconds, then a retry of the upload, as you have realized, the second run usually works, because the race condition is no longer happening.

I have tried following conan upload retry also. But it didn't fix my issue. .conan2/global.conf core.upload:retry=3 core.upload:retry_wait=8

This is an internal retry of the file upload/transfer. The CI retry of the full conan upload can fully skip the actual transfer, because it will realize that the recipe is already in the server because the other concurrent upload job finished uploading it.

Please let us know if this helps.

[AroshikaFernando]

Hi @memsharded

Thank you for your reply.

Currently I am retrying conan upload with sleep, if it fails.

before launching the parallel jobs to build the different binary flavours, do a non-parallel step that does just the conan export . + conan upload of that thing Then you can launch the parallel builds and uploads. As the recipe part is already there, there will be no race conditions.

I will try this and let you know.