PyType does not support running in parallel?

[YingboFu]

Hi, thanks a lot for developing the amazing PyType!

I currently have thousands of Python files to run PyType on, which will take months. Thus, I'd like to run PyType on those files in parallel. However, I've tried both Python multiprocessing and GNU parallel, and neither of them works.

Suppose we have three python files <a.py, b.py, c.py> and I'd like to use three processors of my machine and each processor run PyType on one different file (i.e. one runs pytype a.py, one runs pytype b.py and one runs pytype c.py). I've tried the following Python script:

import multiprocessing
import subprocess
import os

def work(cmd):
    result = subprocess.run(cmd, capture_output=True, text=True)
    return f"{os.getpid()} - Command: {cmd}\nOutput: {result.stdout}\nError: {result.stderr}"


if __name__ == '__main__':
    files = ["a.py", "b.py", "c.py"]
    cmds = []
    for file in files:
        cmds.append(["pytype", file])
    count = multiprocessing.cpu_count()
    with multiprocessing.Pool(processes=count) as pool:
        results = pool.map(work, cmds)
    for result in results:
        print(result)

However, the results show that all processors run Pytype on the same file.

Then, I tried GNU parallel:

parallel -j 3 pytype {} ::: a.py b.py c.py

It gives the same results that all processors actually run Pytype on the same file.

Thus, is this because PyType internally does not allow running in parallel?

Thanks a lot for your help!

[h-joo]

Hi, may I ask how the three different input files look like? Are they co-dependent? (does one import another) Can you also elaborate on what you mean by "However, the results show that all processors run Pytype on the same file."