NVTabular icon indicating copy to clipboard operation
NVTabular copied to clipboard

[BUG] Warning of number of parquet files is wrong

Open bschifferer opened this issue 3 years ago • 0 comments

Describe the bug I convert a dataset of .csv files to .parquet with

dataset.to_parquet(output_path, out_files_per_proc=40)

and I get the warning

/usr/local/lib/python3.8/dist-packages/merlin/io/dataset.py:862: UserWarning: Only created 1 files did not have enough partitions to create 40 files.
  warnings.warn(

However, I can see 40 files in the output directory

bschifferer avatar Jul 18 '22 11:07 bschifferer