skypilot icon indicating copy to clipboard operation
skypilot copied to clipboard

Refactor optimizer: Ensure proper removal of sink/dummy nodes in DAG

Open andylizf opened this issue 4 months ago • 3 comments

Our optimizer currently removes sink and dummy nodes without checking if they're still depended on by other nodes. This violates DAG rules and forces us to use a workaround in the DAG.remove() method.

We should refactor the optimizer to properly handle node dependencies during removal. This will allow us to reinstate the original dependency checks in DAG.remove() and remove the workaround.

          This TODO exists because our optimizer's usage is not strictly following the DAG rules. It attempts to remove sink/dummy nodes while they're still being depended on by other nodes. 

          I believe it's better to keep this TODO for now. This check is actually beneficial for users, and we should address this issue when we refactor the optimizer.

Originally posted by @andylizf in https://github.com/skypilot-org/skypilot/pull/4067#discussion_r1804047058

andylizf avatar Oct 17 '24 20:10 andylizf