Add warm server functionality to local runtime
- [x] This change is worth documenting at https://docs.all-hands.dev/
- [x] Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below
End-user friendly description of the problem this fixes or functionality this introduces.
This PR adds a warm server functionality to the local runtime, which significantly improves the performance of the connect method. Instead of starting a new server each time, the system can now maintain a pool of pre-warmed servers that are ready for immediate use. This reduces the startup time and provides a smoother user experience.
Two new environment variables control this functionality:
-
INITIAL_NUM_WARM_SERVERS: Controls how many warm servers will be initialized during setup -
DESIRED_NUM_WARM_SERVERS: Controls the number of warm servers that should be maintained on an ongoing basis
Summarize what the PR does, explaining any non-trivial design decisions.
This PR implements a warm server pool for the local runtime to improve performance:
- Added a new global variable
_WARM_SERVERS- a list of ActionExecutionServerInfo objects waiting for use - Added environment variables to control the warm server functionality
- Modified the
connectmethod to use warm servers when available - Added methods to create warm servers in the background
- Modified the
execute_actionmethod to create new warm servers after an action if needed - Modified the
deletemethod to clean up warm servers when the runtime is deleted - Refactored the server creation logic into a common
_create_servermethod to reduce code duplication
The implementation ensures that:
- When a new server is needed, it first checks if a warm server is available
- After using a warm server, it creates a new one in the background if needed
- All resources are properly cleaned up when the runtime is deleted
Link of any specific issues this addresses:
N/A
To run this PR locally, use the following command:
docker run -it --rm -p 3000:3000 -v /var/run/docker.sock:/var/run/docker.sock --add-host host.docker.internal:host-gateway -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:16d6f35-nikolaik --name openhands-app-16d6f35 docker.all-hands.dev/all-hands-ai/openhands:16d6f35