tflint icon indicating copy to clipboard operation
tflint copied to clipboard

Recursive inspection in parallel

Open wata727 opened this issue 1 year ago • 0 comments

Fixes https://github.com/terraform-linters/tflint/issues/1943

As explained in #1943, this PR parallelizes recursive inspection by multi-processes.

The process that runs tflint --recursive is called the coordinator, and it starts a worker process for each directory. The worker process executes with --chdir in each directory and outputs issues in the dedicated JSON format, just like a normal TFLint command. The coordinator aggregates JSON output from each process via stdout, and provides feedback to the user.

flowchart TD

    User -- "tflint --recursive" --> Coordinator
    Coordinator -- "tflint --act-as-worker --chdir=subdir1" --> Worker1
    Coordinator -- "tflint --act-as-worker --chdir=subdir2" --> Worker2
    Coordinator -- "tflint --act-as-worker --chdir=subdir3" --> Worker3

The default parallelism is the number of CPUs. This can be controlled with the --max-workers flag, but should not need to be changed in most cases. In my tested repository (over 200 root modules with calls to over 40 child modules), 4 cores cut the time in half.

Before

$ time tflint --recursive
real    5m10.744s
user    5m53.837s
sys     2m36.451s

After

$ time tflint --recursive
real    2m24.805s
user    6m50.139s
sys     2m33.531s

wata727 avatar Apr 13 '24 16:04 wata727