framework icon indicating copy to clipboard operation
framework copied to clipboard

feature request: multiprocessing the validation of a single file

Open pierrecamilleri opened this issue 2 months ago • 1 comments

Using the frictionless validate --parallel flag at the command line or validate(parallel = True) in the python code does not seem to trigger any parallel treatment.

Performance of the validation on a moderately large csv file (~30 mb) does not change with or without this option. In addition, monitoring cpu shows that only 1 core seems to be solicited.

In the case of a datapackage, the command does not even run (see #1644)

To reproduce

time frictionless validate --schema schema.json --parallel data.csv vs time frictionless validate --schema schema.json data.csv

pierrecamilleri avatar Dec 13 '24 15:12 pierrecamilleri