skypilot icon indicating copy to clipboard operation
skypilot copied to clipboard

Published 20 hours ago • verified publisher icon skypilot-org

Implement parallel execution for DAG tasks

Open andylizf opened this issue 4 months ago • 14 comments

Closes #4055

This PR implements parallel execution for DAG tasks in the jobs controller, addressing issue #4055. The changes allow for efficient execution of complex DAGs with independent tasks running concurrently, significantly improving performance for workflows with parallel components.

Changes

  • Modified JobsController to identify and execute parallel task groups
  • Implemented thread-safe task execution and monitoring
  • Added concurrent resource management and cleanup

Tested (run the relevant ones):

  • [x] Code formatting: bash format.sh
  • [ ] Any manual or new tests for this PR (please specify below)
  • [ ] All smoke tests: pytest tests/test_smoke.py
  • [ ] Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • [ ] Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

andylizf avatar Oct 19 '24 01:10 andylizf