beamexample icon indicating copy to clipboard operation
beamexample copied to clipboard
verified publisher icon eljefe6a

HDFS example

Open amitsela opened this issue 9 years ago • 5 comments

Now that we've established that there is no bug in the SparkRunner over HDFS as described here. I was wondering if you have plans to try this on GS ?

amitsela avatar Sep 23 '16 20:09 amitsela

I'm also wondering why is shading them all together an issue ?

amitsela avatar Sep 23 '16 20:09 amitsela

Mostly the really long class path. Someone from the Apache Beam team is working an a GS demo.

eljefe6a avatar Sep 23 '16 20:09 eljefe6a

So you're probably better of with local mode. I don't see TextIO issues resolved within the next 48 hours 😞

amitsela avatar Sep 23 '16 20:09 amitsela

I don't think WordCount will work with HDFS with any runner..

amitsela avatar Sep 23 '16 20:09 amitsela

I have enough to demo Spark+Beam. I'm just going to show it with local files.

eljefe6a avatar Sep 23 '16 20:09 eljefe6a