Bug: clearml-agent cannnot clone repository with specific url format

[SIY1121]

Hi. I'm using self-hosted clearml-server and trying to setup clearml-agent on my gpu machine.

First, I have a git repository and pushed to self-hosted gitlab server.

$ git remote -v
origin  ssh://[email protected]:2222/path_to_repository.git (fetch)
origin  ssh://[email protected]:2222/path_to_repository.git (push)

I enqueued experiment task via task.execute_remotely() but agent pulling the task fails with the following error.

cloning: ssh://gitlab.example.com:2222/path_to_repository.git
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Repository cloning failed: Command '['clone', 'ssh://gitlab.example.com:2222/path_to_repository.git', '/home/user/.clearml/vcs-cache/path_to_repository.git.b5b221a037ad8ee546a3f4f154cfa29e_04767582566266212/path_to_repository.git', '--quiet', '--recursive']' returned non-zero exit status 128.
clearml_agent: ERROR: Failed cloning repository. 

I noticed that clearml-agent is trying to clone repository with the wrong url. The correct url is ssh://[email protected]:2222/path_to_repository.git but clearml-agent uses ssh://gitlab.example.com:2222/path_to_repository.git

A URL without a user name means that git will use the current user name, in this case the user running clearml-agent. manual: https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-clone.html#URLS

I also tried to enqueue task via Clearml Task CLI with the correct url and agent works correctly. clearml-task --repo ssh://[email protected]:2222/path_to_repository.git

version info:

clearml-server: 1.2.0
clearml (client library): 1.5.0
clearml-agent: 1.3.0

Thank you.

[DavidNativ]

Hi @SIY1121 can you please share the agent section of your clearml.conf (please blur the credentials :) ) ? this part :

     # Set GIT user/pass credentials (if user/pass are set, GIT protocol will be set to https)
     # leave blank for GIT SSH credentials (set force_git_ssh_protocol=true to force SSH protocol)
     git_user=""
     git_pass=""
     # Limit credentials to a single domain, for example: github.com,
     # all other domains will use public access (no user/pass). Default: always send user/pass for any VCS domain
     git_host=""

     # Force GIT protocol to use SSH regardless of the git url (Assumes GIT user/pass are blank)
     force_git_ssh_protocol: true
     # Force a specific SSH port when converting http to ssh links (the domain is kept the same)
     # force_git_ssh_port: 0
     # Force a specific SSH username when converting http to ssh links (the default username is 'git')
     # force_git_ssh_user: git

we also have released versions since the ones you use. Could you please update your packages ? Latest stable: clearml = 1.6.2 clearml-server = 1.5.0 clearml-agent = 1.3.0 (this one is up to date :) )

thanks !

[JDennisJ]

Hi @SIY1121 ,

The flow should be the same for internal (self-hosted) and external repos :)

Do you get this error also with the latest version of clearml and clearml-agent? Can you paste the clearml-agent logs (including clearml-agent's print at the beginning)?