ecosystem icon indicating copy to clipboard operation
ecosystem copied to clipboard

Unable to delete files created in worker nodes.

Open nareshr8 opened this issue 5 years ago • 0 comments

When I do df..write.format("tfrecords").option('writeLocality','local').save(path) or df..write.format("tfrecords").save(path) the file is not getting created on the main Driver node. I dont see the folder as well. I am unable to overwrite as that mode is not working. I am unable to delete those files as they are in worker node. Is there a simple API that can delete the files created through this spark job

nareshr8 avatar Dec 18 '19 08:12 nareshr8