PrefectHQ
[Bug] git_clone step fails on Prefect managed work pool (exit code 128, GitLab)
Bug summary
Hi Prefect team,
I'm encountering an issue where my deployment's git_clone step fails only on the prefect:managed work pool. The same configuration works perfectly on other (Google Cloud Run). This issue did not occur a few days ago — the same deployment and configuration used to work without any problem.
Details:
Git provider: GitLab (private repo)
Error:
RuntimeError: Failed to clone repository ... exit code 128
Prefect version: 3.4.25
Config used:
pull:
- prefect.deployments.steps.git_clone: repository: https://gitlab.com/repo-name branch: main credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-token-repo-name }}" directories: ["orchestration"]
Workaround: If we replace the configuration above with an inline URL that includes oauth2: before the Personal Access Token, the clone works successfully on the managed pool:
repository: "{{ prefect.blocks.secret.gitlab_url_with_access_token }}"
Here are the logs :
Unexpected exception encountered when trying to load flow Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps step_output = await run_step(step, upstream_outputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 137, in run_step result = await from_async.call_soon_in_new_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/compatibility/async_dispatch.py", line 94, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/pull.py", line 204, in git_clone run_coro_as_sync(_pull_git_repository_with_retries(storage)) File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 197, in run_coro_as_sync result = from_sync.call_in_new_thread(coroutine_wrapper) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/api.py", line 220, in call_in_new_thread return call.result() ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/_internal/retries.py", line 57, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/pull.py", line 47, in _pull_git_repository_with_retries await repo.pull_code() File "/usr/local/lib/python3.12/site-packages/prefect/runner/storage.py", line 415, in pull_code await self._clone_repo() File "/usr/local/lib/python3.12/site-packages/prefect/runner/storage.py", line 460, in _clone_repo raise RuntimeError( RuntimeError: Failed to clone repository 'https://gitlab.com/la-fourche/data/data-platform.git' with exit code 128.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/prefect/engine.py", line 112, in
Version info
We use the prefect managed work pool version: 3.4.25
Additional context
No response
It seems prefect pull via git remote-https origin https://<credenital-token>@gitlab.com/la-fourche/data/data-platform.git missing username in url even through username is not checked in gitlab. Without username, token is treated as username and password is empty. So failed gitlab authentication.
It seems GitLabCredentials.format_git_credentials failed to hit second if condition:
https://github.com/PrefectHQ/prefect/blob/7d86c932c8dde86f59c0acdea838f8ea0c7e3862/src/integrations/prefect-gitlab/prefect_gitlab/credentials.py#L67-L75
Workground: update gitlab credential token to: oauth2:<token>
I tried to set the url with oauth2 but does not work. Keep getting 128 error
What is your working URL ? I have been trying the following:
https://oauth2:<token>@gitlab.corp/group/project.git but does not work
Same issue for me, @comphilip adding oauth: in the block helped! prefect is at 3.6.5.
