framework
framework copied to clipboard
feature request: multiprocessing the validation of a single file
Using the frictionless validate --parallel
flag at the command line or validate(parallel = True)
in the python code does not seem to trigger any parallel treatment.
Performance of the validation on a moderately large csv file (~30 mb) does not change with or without this option. In addition, monitoring cpu shows that only 1 core seems to be solicited.
In the case of a datapackage, the command does not even run (see #1644)
To reproduce
time frictionless validate --schema schema.json --parallel data.csv
vs time frictionless validate --schema schema.json data.csv