ecosystem
ecosystem copied to clipboard
Unable to delete files created in worker nodes.
When I do df..write.format("tfrecords").option('writeLocality','local').save(path)
or df..write.format("tfrecords").save(path)
the file is not getting created on the main Driver node. I dont see the folder as well. I am unable to overwrite as that mode is not working. I am unable to delete those files as they are in worker node. Is there a simple API that can delete the files created through this spark job