PrefectHQ
Attempting to deploy with DeploymentImage results in "RuntimeError: This error is often thrown because Docker is not running."
First check
- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
Started with a new project on two different computers (both ARM Macs though). I have docker running and other build commands are working fine.
Unable to get prefect to build a docker image.
Related: https://discourse.prefect.io/t/runtimeerror-this-error-is-often-thrown-because-docker-is-not-running-please-ensure-docker-is-running/4069
Reproduction
deploy.py
from prefect import flow
from prefect.deployments import DeploymentImage
@flow(log_prints=True)
def my_flow(name: str = "world"):
print(f"Hello {name}! I'm a flow running on Cloud Run!")
if __name__ == "__main__":
my_flow.deploy(
name="my-deployment",
work_pool_name="my-cloud-run-pool",
cron="0 1 * * *",
image=DeploymentImage(
name="my-image:latest",
platform="linux/amd64",
)
)
$ python deploy.py
Error
Traceback (most recent call last):
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/requests/adapters.py", line 532, in send
conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/requests/adapters.py", line 400, in _get_connection
conn = self.poolmanager.connection_from_host(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host
return self.connection_from_context(request_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context
raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/requests/adapters.py", line 534, in send
raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/utilities/dockerutils.py", line 110, in docker_client
client = docker.DockerClient.from_env()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/client.py", line 96, in from_env
return cls(
^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/luke/Documents/Projects/Prefect/deploy.py", line 11, in <module>
my_flow.deploy(
^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 304, in coroutine_wrapper
return call()
^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
return self.result()
^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
return self.future.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
result = await coro
^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/flows.py", line 1084, in deploy
deployment_ids = await deploy(
^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/deployments/runner.py", line 980, in deploy
image.build()
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/deployments/runner.py", line 832, in build
build_image(**build_kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/utilities/dockerutils.py", line 164, in build_image
with docker_client() as client:
File "/opt/homebrew/Cellar/[email protected]/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/Users/luke/Documents/Projects/Prefect/.venv/lib/python3.12/site-packages/prefect/utilities/dockerutils.py", line 114, in docker_client
raise RuntimeError(
RuntimeError: This error is often thrown because Docker is not running. Please ensure Docker is running.
Versions
Version: 2.19.1
API version: 0.8.4
Python version: 3.12.1
Git commit: 17a1b1d8
Built: Thu, May 16, 2024 3:33 PM
OS/Arch: darwin/arm64
Profile: default
Server type: cloud
Additional context
No response
Did you by chance start using requests==2.32.0? I'm seeing the same thing when upgrading to that version. In my case, pinning to requests=2.31.0 resolved the issue for now.
See:
- https://github.com/psf/requests/issues/6707
Did you by chance start using
requests==2.32.0? I'm seeing the same thing when upgrading to that version. In my case, pinning torequests=0.31.0resolved the issue for now.See:
@ConstantinoSchillebeeckx this stems from a requirement of prefect - apprise
apprise has an unpinned/unbounded requirement for requests in their requirements.txt
Do you think this should be handled within the prefect requirements?
This seems to be an issue after requests was updated to 2.32.2. I also succeeded by installing any version below 2.32.2. In my case 2.31.0.
See also: https://github.com/docker/docker-py/issues/325
Hi there, I believe this can be closed by https://github.com/PrefectHQ/prefect/pull/13538. If not, feel free to comment and I can reopen it.
I encountered this issue as well and downgrading to requests==2.31.0 fixed the problem for me as well. However, can someone more knowledgable than me confirm that the prefect-docker python package doesn't accidentally inherit >2.31.0? If I take a look at my python dependency tree I can see the following:
prefect-docker==0.5.2
├── docker [required: >=6.1.1, installed: 6.1.3]
│ ├── packaging [required: >=14.0, installed: 24.0]
│ ├── requests [required: >=2.26.0, installed: 2.31.0] <------------ I manually installed 2.31.0 because 2.32 got automatically installed
To me it appears that it does indeed inherit whatever version pip decides to install (latest?) through the docker lib. It might be good to add a maximum allowed version of 2.31 to ensure that nobody else runs into this issue. Either way the issue is resolved for me now. Thanks for your help!
Even after locking the requests==2.31.0, I still get the error
Did you check your docker group for permission denied issue? I fix this error after locking requests==2.31.0 and setting docker group.
