Mobius icon indicating copy to clipboard operation
Mobius copied to clipboard

Missing SparkFiles.get implementation?

Open assafi opened this issue 9 years ago • 3 comments

I may be missing something, but I couldn't find the implementation for SparkFiles.get(). This method is needed to retrieve files pushed via SparkContext.AddFile()

assafi avatar Oct 19 '16 23:10 assafi

@assafi Mobius does not have that API. It should be easy to add that if you really need it. Please share your scenario if you can. Also, consider contributing by adding SparkFiles API to Mobius.

skaarthik avatar Oct 24 '16 15:10 skaarthik

The scenario is quite simple actually. We have some .dll and other config/data files that needs to be on the mapping worker. It is possible to add them using sc.AddFile() but retrieving them idiomatically is done using SparkFiles.get()

assafi avatar Oct 24 '16 17:10 assafi

Looks like it should be straightforward to add this to Mobius

skaarthik avatar Nov 02 '16 06:11 skaarthik