NVTabular
NVTabular copied to clipboard
[BUG] Warning of number of parquet files is wrong
Describe the bug I convert a dataset of .csv files to .parquet with
dataset.to_parquet(output_path, out_files_per_proc=40)
and I get the warning
/usr/local/lib/python3.8/dist-packages/merlin/io/dataset.py:862: UserWarning: Only created 1 files did not have enough partitions to create 40 files.
warnings.warn(
However, I can see 40 files in the output directory