Daft
Daft copied to clipboard
Account for task crashes during writes on append mode
Describe the bug If a task crashes during a write on append mode, it will restart and write all the files again, leaving behind dirty files.
To Reproduce Steps to reproduce the behavior: Go to Daft/daft/table/table_io.py and manually insert a probabilistic crash, e.g.
if np.random.random() < failure_probability:
os._exit(0)
run a write task on ray runner.
To Fix We should implement a mechanism to clear dirty files from possible crashes during writes on append mode.