iree icon indicating copy to clipboard operation
iree copied to clipboard

[NFC] Create a python dir for common utilities in build_tools

Open pzread opened this issue 2 years ago • 0 comments

Backgrounds Currently under build_tools we have executable python scripts in multiple directories (benchmarks, docker, ...). They contain some duplicate utility functions, which can be factored out into the common python modules.

The problem is that these scripts don't have a proper search path to find the common python modules, since they aren't packed as a python package and installed. Right now they can only import the modules in the same and child directories, so there is no way to share the python modules with the scripts in other directories.

Proposal This PR adds a directory build_tools/python for all common python modules. To solve the issue of search path, I use a hacky way to modify the search path at the beginning of each executable script. I hope there is a better way to do this 🙂

sys.path.append(
    next(
        # Construct the path: <iree>/build_tools/python
        str(parent / "python")
        for parent in Path(__file__).parents
        # Search for the build_tools dir in the parents
        if parent.name == "build_tools"))

We can still run those scripts anywhere without setting the PYTHONPATH, which is convenient for developers and CI.

Another good reason to do this is that we can freely put the executable scripts in sub-directories. Right now we have all benchmark scripts right under build_tools/benchmarks because they need to import build_tools/benchmarks/common. With this change, they can be put into sub-directories, for example: benchmarks/run_tools, benchmarks/report_tools, ...

pzread avatar Aug 11 '22 00:08 pzread