OpenHands
OpenHands copied to clipboard
All-Hands-AI
[Bug]: docker_runtime.py Failed to start container. Cause http 500 on http+docker://localhost/v1.47/containers/
Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
Describe the bug and reproduction steps
Reproduce
Error happens on my local machine, Windows 10 using WSL.
- checkout the code of rev 1230b229b5a010b3c0ac9e3e909fec6547d81252 or later
- make build && make run-wsl
- access http://localhost:3001/
- open a new conversation "do a hello world pathon"
The container fails to start.
Origin of problem
I tried to reproduce this using various commits.
It starts happening exactly at #commit 1230b229b5a010b3c0ac9e3e909fec6547d81252 Replace use of requests with httpx (#7354) tofarr* Yesterday 14:37
The commit before it works fine.
Logs
14:02:53 - openhands:INFO: runtime_build.py:182 - Building image: ghcr.io/all-hands-ai/runtime:oh_v0.30.0_ts97n9542l0zp9u1_kuw9u5cc05eirpmk
14:02:53 - openhands:DEBUG: docker.py:254 - Checking, if image exists locally:
ghcr.io/all-hands-ai/runtime:oh_v0.30.0_ts97n9542l0zp9u1_kuw9u5cc05eirpmk
14:02:53 - openhands:DEBUG: docker.py:256 - Image found locally.
14:02:53 - openhands:DEBUG: runtime_build.py:211 - Reusing Image [ghcr.io/all-hands-ai/runtime:oh_v0.30.0_ts97n9542l0zp9u1_kuw9u5cc05eirpmk]
14:02:53 - openhands:INFO: docker_runtime.py:140 - [runtime 720332feed994a4ca38cb84222066538] Starting runtime with image: ghcr.io/all-hands-ai/runtime:oh_v0.30.0_ts97n9542l0zp9u1_kuw9u5cc05eirpmk
14:02:53 - openhands:DEBUG: docker_runtime.py:186 - [runtime 720332feed994a4ca38cb84222066538] Preparing to start container...
14:02:53 - openhands:DEBUG: docker_runtime.py:244 - [runtime 720332feed994a4ca38cb84222066538] Workspace Base: /home/ubuntu/projects/MyOpenHands/workspace
14:02:53 - openhands:DEBUG: docker_runtime.py:256 - Mount dir: /home/ubuntu/projects/MyOpenHands/workspace
14:02:53 - openhands:DEBUG: docker_runtime.py:262 - [runtime 720332feed994a4ca38cb84222066538] Sandbox workspace: /workspace
14:02:53 - openhands:ERROR: docker_runtime.py:303 - [runtime 720332feed994a4ca38cb84222066538] Error: Instance openhands-runtime-720332feed994a4ca38cb84222066538 FAILED to start container!
Traceback (most recent call last):
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 275, in _raise_for_status
response.raise_for_status()
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.47/containers/ab3b54aee98fd3f434bc785fc05407a2e3f5832193a052dc17c9134d390531d2/start
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/projects/MyOpenHands/openhands/runtime/impl/docker/docker_runtime.py", line 274, in _init_container
self.container = self.docker_client.containers.run(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/models/containers.py", line 883, in run
container.start()
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/models/containers.py", line 420, in start
return self.client.api.start(self.id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/api/container.py", line 1136, in start
self._raise_for_status(res)
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 277, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.47/containers/ab3b54aee98fd3f434bc785fc05407a2e3f5832193a052dc17c9134d390531d2/start: Internal Server Error ("Ports are not available: exposing port TCP 0.0.0.0:55063 -> 127.0.0.1:0: listen tcp 0.0.0.0:55063: bind: An attempt was made to access a socket in a way forbidden by its access permissions.")
File "/usr/lib/python3.12/threading.py", line 1032, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1012, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
work_item.run()
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/projects/MyOpenHands/openhands/utils/async_utils.py", line 17, in <lambda>
coro = loop.run_in_executor(None, lambda: fn(*args, **kwargs))
File "/home/ubuntu/projects/MyOpenHands/openhands/runtime/impl/docker/docker_runtime.py", line 303, in _init_container
self.log(
File "/home/ubuntu/projects/MyOpenHands/openhands/runtime/base.py", line 181, in log
getattr(logger, level)(message, stacklevel=2)
File "/usr/lib/python3.12/logging/__init__.py", line 1568, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.12/logging/__init__.py", line 1684, in _log
self.handle(record)
14:02:53 - openhands:ERROR: docker_runtime.py:307 - [runtime 720332feed994a4ca38cb84222066538] 500 Server Error for http+docker://localhost/v1.47/containers/ab3b54aee98fd3f434bc785fc05407a2e3f5832193a052dc17c9134d390531d2/start: Internal Server Error ("Ports are not available: exposing port TCP 0.0.0.0:55063 -> 127.0.0.1:0: listen tcp 0.0.0.0:55063: bind: An attempt was made to access a socket in a way forbidden by its access permissions.")
Traceback (most recent call last):
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 275, in _raise_for_status
response.raise_for_status()
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.47/containers/ab3b54aee98fd3f434bc785fc05407a2e3f5832193a052dc17c9134d390531d2/start
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/projects/MyOpenHands/openhands/runtime/impl/docker/docker_runtime.py", line 274, in _init_container
self.container = self.docker_client.containers.run(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/models/containers.py", line 883, in run
container.start()
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/models/containers.py", line 420, in start
return self.client.api.start(self.id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/api/container.py", line 1136, in start
self._raise_for_status(res)
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 277, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-BUkA0F11-py3.12/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.47/containers/ab3b54aee98fd3f434bc785fc05407a2e3f5832193a052dc17c9134d390531d2/start: Internal Server Error ("Ports are not available: exposing port TCP 0.0.0.0:55063 -> 127.0.0.1:0: listen tcp 0.0.0.0:55063: bind: An attempt was made to access a socket in a way forbidden by its access permissions.")
File "/usr/lib/python3.12/threading.py", line 1032, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1012, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
work_item.run()
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/projects/MyOpenHands/openhands/utils/async_utils.py", line 17, in <lambda>
coro = loop.run_in_executor(None, lambda: fn(*args, **kwargs))
File "/home/ubuntu/projects/MyOpenHands/openhands/runtime/impl/docker/docker_runtime.py", line 307, in _init_container
self.log('error', str(e))
File "/home/ubuntu/projects/MyOpenHands/openhands/runtime/base.py", line 181, in log
getattr(logger, level)(message, stacklevel=2)
File "/usr/lib/python3.12/logging/__init__.py", line 1568, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.12/logging/__init__.py", line 1684, in _log
self.handle(record)
OpenHands Installation
Development workflow
OpenHands Version
No response
Operating System
None
Logs, Errors, Screenshots, and Additional Context
No response
Unfortunately i just had it happen again. And that was after it was merged and I had updated my main.
21:54:37 - openhands:ERROR: session.py:162 - Error creating agent_session: [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/ubuntu/.cache/pypoetry/virtualenvs/openhands-ai-TKHEuTrS-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
.........
httpcore.ConnectError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/projects/MyOpenHandsStable/openhands/server/session/session.py", line 147, in initialize_agent
await self.agent_session.start(
File "/home/ubuntu/projects/MyOpenHandsStable/openhands/server/session/agent_session.py", line 117, in start
runtime_connected = await self._create_runtime(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/projects/MyOpenHandsStable/openhands/server/session/agent_session.py", line 316, in _create_runtime
await self.runtime.connect()
File "/home/ubuntu/projects/MyOpenHandsStable/openhands/runtime/impl/docker/docker_runtime.py", line 168, in connect
await call_sync_from_async(self._wait_until_alive)
......
File "/home/ubuntu/projects/MyOpenHandsStable/openhands/runtime/impl/docker/docker_runtime.py", line 376, in _wait_until_alive
self.check_if_alive()
I find the container. It was created but not running anymore. This is the message I get when I try to start it from docker desktop:
(HTTP code 500) server error - Ports are not available: exposing port TCP 0.0.0.0:50112 -> 127.0.0.1:0: listen tcp 0.0.0.0:50112: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
I think it happens because it tries to bind to a port on the host(windows), that while not used, is reserved.
It falls into my reserved ranges.
C:\Windows\System32>netsh interface ipv4 show excludedportrange protocol=tcp
Portausschlussbereiche für das Protokoll "tcp"
Startport Endport
---------- --------
49847 49946
50000 50059 *
50060 50159
* - Verwaltete Portausschlüsse.
The port was probably picked from APP_PORT_RANGE_1 = (50000, 54999) in docker_runtime.py
system.py:find_available_tcp_port is used to find a free one, and it does not check for reserved ranges. At least thats what chatgpt4o thinks. I will try to modify it and see if that prevents it.
import subprocess
import re
def get_excluded_ports():
output = subprocess.check_output(
["netsh", "interface", "ipv4", "show", "excludedportrange", "protocol=tcp"],
text=True
)
excluded = []
for match in re.finditer(r'(\d+)\s+(\d+)', output):
start, end = map(int, match.groups())
excluded.append((start, end))
return excluded
def is_port_excluded(port, excluded_ranges):
return any(start <= port <= end for start, end in excluded_ranges)
def is_port_usable(port):
excluded_ranges = get_excluded_ports()
if is_port_excluded(port, excluded_ranges):
return False
return check_port_available(port)
output = subprocess.check_output( ["netsh", "interface", "ipv4", "show", "excludedportrange", "protocol=tcp"], text=True
Does not work, because we can not assume netsh will be available.
I try doing sock.bind(('0.0.0.0', port)) instead of localhost, as this might work better for detecting reserved ports. I will try this locally for a while and see if the problem disappears.
Since the merge, this particular issue does not happen to me anymore! Thanks for the merge @xingyaoww
Issue persists: Internal Server Error ("ports are not available: exposing port TCP 0.0.0.0:50281 -> 127.0.0.1:0: listen tcp4 0.0.0.0:50281: bind: An attempt was made to access a socket in a way forbidden by its access permissions.")
