dvc
dvc copied to clipboard
iterative
exp push: rejected https git push in Gitlab-CI
I get this error with dvc exp push origin :
DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
Note that everything is done inside a .gitlab-ci.yml file. And I don't dvc exp run, the artifact etc... are logged through Live and DVCLiveCallBack
I have an Azure Blob Storage, when pushed the data is stored on the Blob Storage. I can pull them from the CI script.
When I .log_artifact() outside of the Live's dir the artifact is pushed to the remote Azure blob, and the CI script is able to finish. But no models/model.dvc is pushed to the repo.
But when the artifact is logged inside the Live's dir ("dvclive") then I get this error : DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
I've tried to set up only hardlink and symlink, but I get the same. I've also tried to log the model through the studio with the files stored on the blob, but I get an "Unexpected error" with no ability to see the Traceback.
DVC version: 3.29.0 (pip)
-------------------------
Platform: Python 3.8.10 on Linux-5.4.0-1109-azure-x86_64-with-glibc2.29
Subprojects:
dvc_data = 2.20.0
dvc_objects = 1.2.0
dvc_render = 0.6.0
dvc_task = 0.3.0
scmrepo = 1.4.1
Supports:
azure (adlfs = 2023.10.0, knack = 0.11.0, azure-identity = 1.15.0),
http (aiohttp = 3.8.6, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.6, aiohttp-retry = 2.8.3)
Config:
Global: /root/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git
I get this error with dvc exp push origin : DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
This isn't the error, it's an informational message. Can you run the dvc exp push command with -v and post the full traceback here
thx a lot for the fast answer this is the full traceback :
vc exp push -v origin[0;m
2023-11-15 11:13:16,237 DEBUG: v3.29.0 (pip), CPython 3.8.10 on Linux-5.4.0-1109-azure-x86_64-with-glibc2.29
2023-11-15 11:13:16,237 DEBUG: command: /usr/local/bin/dvc exp push -v origin
2023-11-15 11:13:16,573 DEBUG: git push experiment ['refs/exps/e4/df55c85667dcb0392e199ac1f443e53829d824/beaky-luce:refs/exps/e4/df55c85667dcb0392e199ac1f443e53829d824/beaky-luce'] -> 'origin'
2023-11-15 11:14:28,739 ERROR: unexpected error - name 'urllib3' is not defined
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2238, in _http_request
resp = self.pool_manager.request(
File "/usr/local/lib/python3.8/dist-packages/urllib3/_request_methods.py", line 118, in request
return self.request_encode_body(
File "/usr/local/lib/python3.8/dist-packages/urllib3/_request_methods.py", line 217, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py", line 444, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/dist-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
File "/usr/local/lib/python3.8/dist-packages/dvc/cli/command.py", line 27, in do_run
return self.run()
File "/usr/local/lib/python3.8/dist-packages/dvc/commands/experiments/push.py", line 55, in run
result = self.repo.experiments.push(
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/__init__.py", line 381, in push
return push(self.repo, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/__init__.py", line 60, in wrapper
return f(repo, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/scm_context.py", line 152, in run
return method(repo, *args, **kw)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/push.py", line 120, in push
push_result = _push(repo, git_remote, exp_ref_set, force)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/push.py", line 162, in _push
results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/__init__.py", line 292, in _backend_func
result = func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/dulwich/__init__.py", line 630, in push_refspecs
result = client.send_pack(
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2074, in send_pack
resp, read = self._smart_request(
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2014, in _smart_request
resp, read = self._http_request(url, headers, data)
File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/dulwich/client.py", line 49, in _http_request
result = super()._http_request(
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2241, in _http_request
except urllib3.exceptions.HTTPError as e:
NameError: name 'urllib3' is not defined
2023-11-15 11:14:28,864 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
When I specify dvc exp push origin --run-cache I get this Traceback :
vc exp push --run-cache -v origin[0;m
2023-11-15 10:46:17,431 DEBUG: v3.29.0 (pip), CPython 3.8.10 on Linux-5.4.0-1109-azure-x86_64-with-glibc2.29
2023-11-15 10:46:17,432 DEBUG: command: /usr/local/bin/dvc exp push --run-cache -v origin
2023-11-15 10:46:17,846 DEBUG: git push experiment ['refs/exps/89/8490947bee9fd8b766a919b9ca17e2dc90969c/gauge-scud:refs/exps/89/8490947bee9fd8b766a919b9ca17e2dc90969c/gauge-scud'] -> 'origin'
2023-11-15 10:47:36,428 ERROR: unexpected error - unexpected http resp 502 for https://token:[MASKED][email protected]/BaptisteLoquette/demo_cml_pipeline.git/git-receive-pack
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
File "/usr/local/lib/python3.8/dist-packages/dvc/cli/command.py", line 27, in do_run
return self.run()
File "/usr/local/lib/python3.8/dist-packages/dvc/commands/experiments/push.py", line 55, in run
result = self.repo.experiments.push(
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/__init__.py", line 381, in push
return push(self.repo, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/__init__.py", line 60, in wrapper
return f(repo, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/scm_context.py", line 152, in run
return method(repo, *args, **kw)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/push.py", line 120, in push
push_result = _push(repo, git_remote, exp_ref_set, force)
File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/push.py", line 162, in _push
results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/__init__.py", line 292, in _backend_func
result = func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/dulwich/__init__.py", line 630, in push_refspecs
result = client.send_pack(
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2074, in send_pack
resp, read = self._smart_request(
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2014, in _smart_request
resp, read = self._http_request(url, headers, data)
File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/dulwich/client.py", line 49, in _http_request
result = super()._http_request(
File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2251, in _http_request
raise GitProtocolError(
dulwich.errors.GitProtocolError: unexpected http resp 502 for https://token:[MASKED][email protected]/My_Username/demo_cml_pipeline.git/git-receive-pack
These errors are not related to the artifact logging, that is working properly.
The problem is specifically with the dvc exp push, it looks like there is something wrong with the git credentials in your CI runner, DVC is unable to git push the experiment back to gitlab. If you are using CML in your workflow, you can run cml ci at the start of your workflow (before using dvc) and it may resolve the issue
https://cml.dev/doc/ref/ci
Okay I see, I do use cml and the script on the job that fails uses cml-ci as follow :
Are you suggesting to cml ci before dvc pull ? thx again
script:
- pip install -r requirements.txt
- dvc pull data
- cml ci
- python train.py
- dvc exp push -v origin
- dvc data status
- git status
- cat metrics.txt >> report.md
- cml comment create report.md
I get the same error while cml ci --fetch-depth=0 --user-email=mail --user-name=username, I don't get why. I have a personal access token but it shoudn't block anything
REPO_TOKEN personal acces token is defined with api, read_repository and write_repository scopes. And REPO_TOKEN is configured in CI variables, masked and non-protected. I don't understand.
Idk if it impacts anything but. Note that I didn't use SSH with git repo and wasn't using credential helpers. I just set up SSH, but still get same error.
@BaptisteLoquette Have you tried to do a regular git push in that CI script to see if it works?
Regular meaning no-SSH, with only login and password ? If so I did for last few days, I settled up SSH yesterday to see if it would resolve this.
When you run dvc exp push inside CI, you get an error because the Git push failed. What I mean is that you could try to replace dvc exp push by committing and pushing manually, like git add && git commit && git push and see if it works. That would tell you whether it's an issue with your git authentication or with how dvc is doing the push.
Oh yeah inside the CI. Yeah Ill try that. I do think it's purely with the git creds and not dvc, though. Do you know if having a token set up for me to git push (not on the CI) and not through password can block the dvc exp push ?
Okay so when manually done in the CI. git push works good. I think the error is about how I configure the studio
So I removed everything related to the studio to check. And it still yields the same error, I think I'll just git push manually
Makes sense, @BaptisteLoquette. @pmrowla It looks like this is an issue in scmrepo then.
Lowering the priority since we haven't been able to identify what's happening here and haven't heard additional reports
@dberenbaum, I am closing the issue as we don't have enough information to reproduce, and we do not hear any more reports. Feel free to reopen if you think it's important.
