sparkling
sparkling copied to clipboard
gorillalabs
Adding Spark Streaming support
Hi @chrisbetz Thanks for Sparkling, it's delicious. I just started working with it, and my employer Iris.tv is prepared to use it in production. I am going to put some effort into Spark Streaming support, and I wonder if you have anything to share about the status of it. I see your note in the code that it is incomplete/experimental. I want to make use of it in production, if possible, so I'm going to try to get it to a point where we'd be comfortable doing that. So, if you have a little time and you feel like explaining where things are (what works, what doesn't, what's left to be done), I would certainly appreciate it. If not, I will see what I can do on my own and send you PRs.
I played with this some today. My fork at https://github.com/jbtv/sparkling is working (locally) with Spark Streaming 1.3.1. I added support for direct Kafka streams, and implemented filter for DStreams.
When I get around to writing tests for the streaming ns, I'll PR you. Thanks @chrisbetz
Hi Blake,
that's awesome news. As you have seen, I've (almost) no support for Spark Streaming so far. I'm just too busy with the rest. However, extending sparkling to support Spark Streaming (and Dataframes and ML) would be really good.
So, I really appreciate your effort and I'm eagerly awaiting your PRs.
BTW: Would you mind making PRs against the "develop" branch? This is making things a little cleaner in my opinion. Should be just a click away, anyhow ;)
Cheers,
Chris
Am 01.06.2015 um 05:39 schrieb Blake Miller [email protected]:
I played with this some today. My fork at https://github.com/jbtv/sparkling is working (locally) with Spark Streaming 1.3.1. I added support for direct Kafka streams, and implemented filter for DStreams.
When I get around to writing tests for the streaming ns, I'll PR you. Thanks @chrisbetz
— Reply to this email directly or view it on GitHub.
Is there any update on bringing ML support to Sparkling? I'd love to switch Python for Clojure as my daily working environment!
Keep the good work!
Any follow-up on this? Would really enjoy a more Clojure-like interface for Spark Streaming.
