ruby-spark
ruby-spark copied to clipboard
ondra-m
Support for dataframes
I'm really interested in using spark and would love to be able to interact with it using Ruby. This gem looks like a great option. It doesn't look like it would natively support spark dataframes, right? Would there be any way to interact with dataframes using this gem? If not, what kind of effort would you expect would be required to build it in?
The project is still in its alpha stage, basically it is a only a proof of concept. We've tried using Spark API from Ruby and it works! :) Currently we support only a subset of Spark API functionality. We'd like to attract more developers and extend supported functions. Right now ruby-spark runs better on MRI than JRuby (this might change with JRuby 9.0.0.0 release).
ruby-spark interacts with JVM (Scala backend), almost anything that is possible in Python, we should be able to do in Ruby. Have a look here on some benchmarks: http://ondra-m.github.io/ruby-spark/
This is the kind of project I could be interested in contributing to. If you had to put a rough estimate on the number of developer hours you think it would take to build in Spark SQL support, what would it be?
SQL implementation will take a long time. Currently there are more important things to do (some RDD and Mllib methods are missing, beter Proc serialization, ...).
This is a really great gem and DataFrames will be the foundation to attract great projects for this gem. Could I help your development on the documentation?
