scalding Add a spark backend

Add a spark backend

Open johnynek opened this issue 7 years ago • 4 comments

To test the modularity, a basic spark backend based on RDDs should be implemented.

Should be easy to do to based on the cascading backend and the memory backend.

Oct 09 '17 03:10 johnynek

Can you point to the implementation of the inmem backend?

Jan 13 '18 10:01 EncodePanda

https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/memory_backend/MemoryBackend.scala

@rabbitonweb

sorry for the latency. Missed this somehow.

Jan 23 '18 20:01 johnynek

see #1782

That gets the memory backend usable (I think).

I'm going to work on a basic spark implementation soon (next couple of weeks).

Feb 03 '18 04:02 johnynek

this has begun. See #1832

Mar 03 '18 21:03 johnynek