conan-package-tools
conan-package-tools copied to clipboard
conan-io
How to access private conan repository when using docker?
Description of Problem, Request, or Question
I'm trying to build a package A which depeds on package B in our private conan repository.
Everything is fine if not using docker, but build with use_docker=True will end with:
conans.errors.NotFoundException: Unable to find 'package B' in remotes
I have figured out that's because out private repository needed login and the remote in docker container seems remain 'anonymous':
# Output of `self.run("conan user") in conanfile.py
Current user of remote 'private-repo' set to: 'None' (anonymous)
So what is the prefered way to login to private remote when using docker?
I've tried:
- using
docker_entry_script: not work since remotes is not added to conan yet - call
ConanMultiPackager.login: no effects in docker - call
conan user -r ...in theconanfile.py: worked but I doubt it's the good way to go
Environment Details
- Conan Package Tools Version: 0.29.3
- Operating System: macOS
- Operation System Version: 10.14.6
- Compiler+version: clang7
- Docker image: conanio/clang7
- Conan version: conan 1.19.2
- Python version: python 3.7.4
Steps to reproduce
Contents of build.py:
...
docker_builder = ConanMultiPackager(
username="local",
login_username="XXX",
password="********",
remotes="our-private-repo-url",
...
use_docker=True,
docker_image="conanio/clang7"
)
# docker_builder.login("myrepo")
docker_builder.add_common_builds()
docker_builder.run()
Hi @0xcccc !
There is no support Docker login for now. The authentication is related to Conan package server (bintray/artifactory/conan_server).
Docker login it's a new feature request. Why do you need private docker images, conanio images are not enough?
Hi @uilianries,
Docker login it's a new feature request. Why do you need private docker images, conanio images are not enough?
I think It's not a feature request about docker login and I'm using official images in docker hub actually.
It's more about "How could I get access to a private conan repo in docker conatiner, so conan could download the missing depedencies from there?"
hmm I see now what you mean. So it sounds like a bug. I think CPT should be able to login to any conan server when running a docker container.
I think this bug is still relevant. Any update to this after almost 2 years? #575 ( a very recent issue) might be related. It seems that this bug is not only still open, but also an issue which still causes pain.
@Parcley Indeed, I'll revisit this issue during this week. Thanks for pinging.
@uilianries do you have any updates on this? In our project, we are affected by this issue. If the solution will take some more time, we could introduce an (ugly) workaround on our side, but the preferred way is to use the actual solution :-)
Hello @Parcley and @kegt !
Have you tried the user and password as listed on the documentation?
https://github.com/conan-io/conan-package-tools#environment-configuration
The idea is passing the remote name with the login username and password.
For instance, you have an internal Artifactory running on your company and you want to integrate with CPT, so you need to configure your CI with the follow env vars (it could be parameters too):
export CONAN_REMOTES=http://192.168.0.21:8081/artifactory/api/conan/acme@False@ACME
Note here I'm using 192.168.0.12 as the Artifactory IP address, but it could be a domain name. Also I'm using the company name acme. The remotes structure also uses @False@ACME which means, do not check SSL certificate (False), use the custom name ACME for that remote. By default, CPT uses numeric index, like remote_1.
Now, we should set the credentials:
export CONAN_LOGIN_USERNAME_ACME=myusername
export CONAN_PASSWORD_ACME=kwdkqwpodqw0dk0kkdpaoskd
The login username here uses the suffix _ACME which must be the same as we used for the remote name.
The password is a token generate by Artifactory. If you don't known how to obtain your token, please read here
It should be enough to install any package from your private repository.
But if would like to upload to my private repository too? Here is:
export CONAN_UPLOAD=http://192.168.0.21:8081/artifactory/api/conan/acme@False@ACME
Use the same URL used for remote, actually, you don't need to declare REMOTES if you are using using UPLOAD. Also, there is no risk of conflict, CPT is smart enough to solve it.
For your case, you are using parameters, which is pretty similar:
from cpt.packager import ConanMultiPackager
if __name__ == "__main__":
builder = ConanMultiPackager(
login_username={"acme": "myusername"}.
password={"acme": "kwdkqwpodqw0dk0kkdpaoskd"},
remotes = [("http://192.168.0.21:8081/artifactory/api/conan/acme", False, "acme"))
builder.add_common_builds()
builder.run()
All these variable must work for Docker too, CPT will forward them to your container, it creates a mask with CPT_ env var, your private date (username/password) won't be showed on your logs.
Please, let me know if it's enough for your case or you are using a different approach.
Hi @uilianries
Thank you for your message and for taking the time.
Indeed I did not see the CONAN_LOGIN_USERNAME_XXX and CONAN_PASSWORD_XXX variables in https://github.com/conan-io/conan-package-tools#environment-configuration.
And what is best:
they work as described, adding a conan remote with
CONAN_REMOTES=http://192.168.0.21:8081/artifactory/api/conan/acme@False@ACME
and then setting credentials with CONAN_LOGIN_USERNAME_XXX and CONAN_PASSWORD_XXX works as expected. 👍
But here is the issue that still remains:
when adding the conan remotes implicitly by using a conan config (CONAN_CONFIG_URL) which also includes remotes, I can not set credentials for these remotes with CONAN_LOGIN_USERNAME_XXX and CONAN_PASSWORD_XXX. 👎
In this case I am NOT setting CONAN_REMOTES.
For XXX I was using the name for the conan remote as specified at the CONAN_CONFIG_URL.
This is the error message I am getting:
conans.errors.NoRemoteAvailable: No remote 'swp-conan' defined in remotes
[2021-10-14T07:49:07.063Z] Traceback (most recent call last):
[2021-10-14T07:49:07.063Z] File "/usr/local/lib/python3.7/site-packages/conans/client/cache/remote_registry.py", line 222, in __getitem__
[2021-10-14T07:49:07.063Z] return self._remotes[remote_name]
[2021-10-14T07:49:07.063Z] KeyError: '<MY_REMOTE_NAME>'
In my case, CONAN_UPLOAD is pointing to a different remote. For this remote the CONAN_LOGIN_USERNAME and CONAN_PASSWORD variables work fine.
@Parcley Sorry my delay.
It's not clear what's failing for you, I need details to understand your case. Is it failing when downloading a package or when uploading all built packages? Please, share your full log, but hide your sensitive data (username, address and password).
I tried to reproduce your error and so far is working for me.
The final error is because that package doesn't exist on my remote, which is true. But in terms of login, all worked as expected. I used my personal github repository with a remote file, which contains my personal artifactory account. That Artifactory doesn't allow anonymous request, only logged users can download packages, so it's a private repository. All login information, CONAN_LOGIN_USERNAME_UILIANR and CONAN_PASSWORD_UILIANR were passed by environment variable and worked into the Docker container. BTW, I'm using Linux as Docker host, but I'm don't know any Windows limitation about same case.
Details
$ python build.py
****************************************************************************************************
____ ____ _____ __ ____ ____ _ _____ _ __
/ ___| _ \_ _| / / / ___|___ _ __ __ _ _ __ | _ \ __ _ ___| | ____ _ __ _ ___ |_ _|__ ___ | |___ \ \
| | | |_) || | | | | | / _ \| '_ \ / _` | '_ \ | |_) / _` |/ __| |/ / _` |/ _` |/ _ \ | |/ _ \ / _ \| / __| | |
| |___| __/ | | | | | |__| (_) | | | | (_| | | | | | __/ (_| | (__| < (_| | (_| | __/ | | (_) | (_) | \__ \ | |
\____|_| |_| | | \____\___/|_| |_|\__,_|_| |_| |_| \__,_|\___|_|\_\__,_|\__, |\___| |_|\___/ \___/|_|___/ | |
\_\ |___/ /_/
Version: 0.36.0
[local_vars]
+---------------------------+------------------------------------------------------------+
| Configuration | value |
|---------------------------+------------------------------------------------------------|
| cwd | /home/uilian/Development/conan/conan-package-tools/package |
| skip_check_credentials | False |
| upload_only_when_stable | False |
| upload_only_when_tag | False |
| upload_only_recipe | False |
| upload_force | True |
| stable_branch_pattern | master$ main$ release.* stable.* |
| stable_channel | stable |
| conanfile | conanfile.py |
| use_docker | True |
| docker_conan_home | /home/conan |
| sudo_docker_command | sudo -E |
| sudo_pip_command | |
| pip_command | pip |
| docker_pip_command | pip |
| docker_shell | /bin/sh -c |
| docker_platform_param | |
| lcow_user_workaround | |
| exclude_vcvars_precommand | False |
| upload_dependencies | |
| update_dependencies | False |
| docker_32_images | False |
| force_selinux | False |
| conan_pip_package | conan==1.41.0 |
| vs10_x86_64_enabled | False |
| builds_in_current_page | [] |
| config_url | https://github.com/uilianries/bincrafters-config.git |
| skip_recipe_export | False |
+---------------------------+------------------------------------------------------------+
>> Running builds...
Page: 1/1
+-----+--------------------+--------+------------+--------------+-----------------+
| # | compiler.version | arch | compiler | build_type | foobar:shared |
|-----+--------------------+--------+------------+--------------+-----------------|
| 1 | 10 | x86_64 | gcc | Release | False |
| 2 | 10 | x86_64 | gcc | Debug | False |
| 3 | 10 | x86_64 | gcc | Release | True |
| 4 | 10 | x86_64 | gcc | Debug | True |
+-----+--------------------+--------+------------+--------------+-----------------+
>> Build: 1/4
[docker pull]
____________________________________________________________________________________________________
>> sudo -E docker pull conanio/gcc10-ubuntu16.04
____________________________________________________________________________________________________
Using default tag: latest
latest: Pulling from conanio/gcc10-ubuntu16.04
Digest: sha256:a413ab93f6b6215bd125e326534382306ef86cb29186951e399f9a5c9c128ccd
Status: Image is up to date for conanio/gcc10-ubuntu16.04:latest
docker.io/conanio/gcc10-ubuntu16.04:latest
[update conan]
____________________________________________________________________________________________________
>> sudo -E docker run -e CONAN_PASSWORD_uilianr="xxxxxxxx" -e CONAN_ARCHS="x86_64" -e CONAN_REVISIONS_ENABLED="1" -e CONAN_PASSWORD="xxxxxxxx" -e CONAN_GCC_VERSIONS="10" -e CONAN_PASSWORD_bincrafters="xxxxxxxx" -e CONAN_LOGIN_USERNAME_bincrafters="xxxxxxxx" -e CONAN_LOGIN_USERNAME_uilianr="xxxxxxxx" -e CONAN_DOCKER_IMAGE="conanio/gcc10-ubuntu16.04" -e CONAN_CONFIG_URL="https://github.com/uilianries/bincrafters-config.git" -e CONAN_LOGIN_USERNAME_BINCRAFTERS="xxxxxxxx" -e CONAN_PASSWORD_BINCRAFTERS="xxxxxxxx" -e CONAN_LOGIN_USERNAME_UILIANR="xxxxxxxx" -e CONAN_PASSWORD_UILIANR="xxxxxxxx" -e CONAN_REFERENCE="foobar/0.1" -e CPT_PROFILE="@@include(default)@@@@[settings]@@arch=x86_64@@build_type=Release@@compiler=gcc@@compiler.version=10@@[options]@@foobar:shared=False@@[env]@@@@[build_requires]@@@@" -e CPT_PROFILE_BUILD="@@include(default)@@@@[settings]@@@@[options]@@@@[env]@@@@[build_requires]@@@@" -e CONAN_TEMP_TEST_FOLDER="1" -e CPT_UPLOAD_RETRY="3" -e CPT_UPLOAD_FORCE="True" -e CPT_CONFIG_URL="https://github.com/uilianries/bincrafters-config.git" -e CPT_CONANFILE="conanfile.py" --name conan_runner conanio/gcc10-ubuntu16.04 /bin/sh -c " pip install conan_package_tools==0.36.0 --upgrade --no-cache && pip install conan==1.41.0 --no-cache"
____________________________________________________________________________________________________
Collecting conan_package_tools==0.36.0
Downloading conan_package_tools-0.36.0-py3-none-any.whl (85 kB)
Collecting six<=1.15.0,>=1.10.0
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting tabulate<0.9.0,>=0.8.0
Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)
...
You should consider upgrading via the '/opt/pyenv/versions/3.6.15/bin/python3.6 -m pip install --upgrade pip' command.
____________________________________________________________________________________________________
>> sudo -E docker commit conan_runner conanio/gcc10-ubuntu16.04
____________________________________________________________________________________________________
sha256:6fb2b322cd12b5e2292c8dfefba4b7a94385dc3fbe75b4babef62ecbcbccc973
____________________________________________________________________________________________________
>> sudo -E docker rm conan_runner
____________________________________________________________________________________________________
conan_runner
## .
## ## ## ==
## ## ## ## ===
/*********************\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
You are in Docker now! conanio/gcc10-ubuntu16.04
____________________________________________________________________________________________________
>> sudo -E docker run --rm -v "/home/uilian/Development/conan/conan-package-tools/package:/home/conan/project" -e CONAN_PASSWORD_uilianr="xxxxxxxx" -e CONAN_ARCHS="x86_64" -e CONAN_REVISIONS_ENABLED="1" -e CONAN_PASSWORD="xxxxxxxx" -e CONAN_GCC_VERSIONS="10" -e CONAN_PASSWORD_bincrafters="xxxxxxxx" -e CONAN_LOGIN_USERNAME_bincrafters="xxxxxxxx" -e CONAN_LOGIN_USERNAME_uilianr="xxxxxxxx" -e CONAN_DOCKER_IMAGE="conanio/gcc10-ubuntu16.04" -e CONAN_CONFIG_URL="https://github.com/uilianries/bincrafters-config.git" -e CONAN_LOGIN_USERNAME_BINCRAFTERS="xxxxxxxx" -e CONAN_PASSWORD_BINCRAFTERS="xxxxxxxx" -e CONAN_LOGIN_USERNAME_UILIANR="xxxxxxxx" -e CONAN_PASSWORD_UILIANR="xxxxxxxx" -e CONAN_REFERENCE="foobar/0.1" -e CPT_PROFILE="@@include(default)@@@@[settings]@@arch=x86_64@@build_type=Release@@compiler=gcc@@compiler.version=10@@[options]@@foobar:shared=False@@[env]@@@@[build_requires]@@@@" -e CPT_PROFILE_BUILD="@@include(default)@@@@[settings]@@@@[options]@@@@[env]@@@@[build_requires]@@@@" -e CONAN_TEMP_TEST_FOLDER="1" -e CPT_UPLOAD_RETRY="3" -e CPT_UPLOAD_FORCE="True" -e CPT_CONFIG_URL="https://github.com/uilianries/bincrafters-config.git" -e CPT_CONANFILE="conanfile.py" conanio/gcc10-ubuntu16.04 /bin/sh -c " cd project && run_create_in_docker "
____________________________________________________________________________________________________
>> Installing config from address https://github.com/uilianries/bincrafters-config.git
Trying to clone repo: https://github.com/uilianries/bincrafters-config.git
Repo cloned!
Defining remotes from remotes.txt
Skip LICENSE.txt
Skip README.md
Copying file android-clang8-armv7 to /home/conan/.conan/profiles
...
****************************************************************************************************
+---------------------+
| Profile |
|---------------------|
| include(default) |
| |
| [settings] |
| arch=x86_64 |
| build_type=Release |
| compiler=gcc |
| compiler.version=10 |
| [options] |
| foobar:shared=False |
| [env] |
| |
| [build_requires] |
+---------------------+
+------------------+
| Profile |
|------------------|
| include(default) |
| |
| [settings] |
| |
| [options] |
| |
| [env] |
| |
| [build_requires] |
+------------------+
[conan_create]
>> Calling 'conan create'
+--------------------+------------------------------------------------------+
| Configuration | value |
|--------------------+------------------------------------------------------|
| name | foobar |
| version | 0.1 |
| user | |
| channel | |
| build_modes | |
| profile_name | /tmp/tmponx3k1e_conan_package_tools_profiles/profile |
| profile_build_name | /tmp/tmpy59q2601conan_package_tools_profiles/profile |
+--------------------+------------------------------------------------------+
Exporting package recipe
foobar/0.1 exports_sources: Copied 1 '.txt' file: CMakeLists.txt
foobar/0.1 exports_sources: Copied 1 '.h' file: foobar.h
foobar/0.1 exports_sources: Copied 1 '.cpp' file: foobar.cpp
foobar/0.1: A new conanfile.py version was exported
foobar/0.1: Folder: /home/conan/.conan/data/foobar/0.1/_/_/export
foobar/0.1: Using the exported files summary hash as the recipe revision: a5387d56c1397643d03fdba68c4a8d52
foobar/0.1: Exported revision: a5387d56c1397643d03fdba68c4a8d52
Configuration (profile_host):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
foobar:shared=False
[build_requires]
[env]
Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
foo/0.1@uilian/stable: Not found in local cache, looking in remotes...
foo/0.1@uilian/stable: Trying with 'conan-center'...
foo/0.1@uilian/stable: Trying with 'bincrafters'...
foo/0.1@uilian/stable: Trying with 'uilianr'...
Please log in to "uilianr" to perform this action. Execute "conan user" command.
Remote 'uilianr' username: Got username 'xxxxxxxx' from environment
Please enter a password for "xxxxxxxx" account: Got password '******' from environment
ERROR: Failed requirement 'foo/0.1@uilian/stable' from 'foobar/0.1'
Traceback (most recent call last):
Hi @uilianries I tried to reproduce the issue, but I am not able to re-create it on my side. While I was trying to reproduce the issue, I found a similar one,but I don't know if it is really cpt related.
I see the errors when downloading a dependency from a password-protected remote (myPackage from myRemote).
Meanwhile, I have stopped using cpt's docker feature and I use cpt's native build instead, which works for me.
Instead I get a different error situation: myPackage is not found in any of my configured remotes (but it does exist in myRemote). When using the native (i.e. non-docker) build, the package can be successfully downloaded
ubuntu 20.04 python 3.7.5 conan-package-tools 0.37.0 conan 1.36.0
#!/usr/bin/env python3
import os
from cpt.packager import ConanMultiPackager
user = "XXXX"
password = "XXXX"
conan_config_uri = "https://XXXX/conan-config.git"
os.environ['CONAN_DOCKER_IMAGE'] = "XXXX/docker.io/conanio/gcc9:latest"
os.environ['CONAN_CONFIG_URL'] = conan_config_uri
os.environ['CONAN_GCC_VERSIONS'] = "9"
os.environ['CONAN_DOCKER_IMAGE_SKIP_UPDATE'] = "True"
os.environ['CONAN_USE_DOCKER'] = "1"
os.environ['CONAN_USERNAME'] = user
os.environ['CONAN_PASSWORD'] = password
os.environ['CONAN_ARCHS'] = "x86_64"
os.environ['CONAN_BUILD_TYPES'] = "Debug"
os.environ['CONAN_BUILD_POLICY'] = "missing"
# os.environ['CONAN_PASSWORD_my-remote'] = password
builder = ConanMultiPackager()
builder.add_common_builds()
# builder.login("my-remote")
builder.run()
# including the commented lines does not change the behaviour
here is the matching log: log.txt
@Parcley just to confirm, do you need a password to download your package? Are you running Artifactory as server, not Conan Center, right? I'll try to reproduce your case. Thank you for sharing more details.
yes, I need a password for "my-remote" to download "myPackage". All my remotes (also some, which do not require authentication) are hosted on Artifactory. thanks @uilianries for spending your precious time