rules_py icon indicating copy to clipboard operation
rules_py copied to clipboard

[Idea]: Symlink tree minimization for py_venv_*

Open arrdem opened this issue 5 months ago • 1 comments

Brainfiring while working on #644.

Today the py_venv_* strategy is to create a link for every input file. This works nicely in general because it requires no real work planning and handles merging directory structures naturally.

Consider the inputs

@@foo_bar//site-packages/foo:__init__.py
@@foo_bar//site-packages/foo:bar.py
@@foo_baz//site-packages/foo:__init__.py
@@foo_baz//site-packages/foo:baz.py

In this case we have to generate a directory containing symlinks because we have files in the same destination directory which come from more than one source. But the more common case is stuff like

@@requests//site-packages/requests:__init__.py
@@clap//site-packages/clap:__init__.py

that is each external dependency provides a unique subtree and only rarely do they need to merge. In this second case we don't need to generate a link for each input file, we can instead create one link to the directory which contains only input files among which no merging is required.

This allows us to trade some compute (which is cheap) for I/O (which is not) and reduce the number of links we create in venv trees substantially.

arrdem avatar Sep 24 '25 17:09 arrdem