ensure the existence of outputs after persist
In the prediction step, some outputs in parquet format are saved locally in a non-blocking way using dask's persist function. Afterwards these files are accessed, but there's no guarantee that they have been fully saved on disk. I added a simple fix to it, though there might be a better way.
Thanks for the contribution @senbaikang! I'm reviewing this, this is similar to the idea of segger.prediction.predict_multigpu. It looks fine but I have to test it thoroughly. will do soon!
Thank you for reviewing it! For the original code I encounter an error of inexistent output files for the prediction step, hence the small fix. Hope it helps and feel free to improve!