iree
iree copied to clipboard
[NFC] Create a python dir for common utilities in build_tools
Backgrounds
Currently under build_tools
we have executable python scripts in multiple directories (benchmarks
, docker
, ...). They contain some duplicate utility functions, which can be factored out into the common python modules.
The problem is that these scripts don't have a proper search path to find the common python modules, since they aren't packed as a python package and installed. Right now they can only import the modules in the same and child directories, so there is no way to share the python modules with the scripts in other directories.
Proposal
This PR adds a directory build_tools/python
for all common python modules. To solve the issue of search path, I use a hacky way to modify the search path at the beginning of each executable script. I hope there is a better way to do this 🙂
sys.path.append(
next(
# Construct the path: <iree>/build_tools/python
str(parent / "python")
for parent in Path(__file__).parents
# Search for the build_tools dir in the parents
if parent.name == "build_tools"))
We can still run those scripts anywhere without setting the PYTHONPATH, which is convenient for developers and CI.
Another good reason to do this is that we can freely put the executable scripts in sub-directories. Right now we have all benchmark scripts right under build_tools/benchmarks
because they need to import build_tools/benchmarks/common
. With this change, they can be put into sub-directories, for example: benchmarks/run_tools
, benchmarks/report_tools
, ...