LMW-tree
LMW-tree copied to clipboard
Implement distributed version of streaming parallel EM-tree
Compress transmission of integer accumulators between machines vectors using https://github.com/lemire/FastPFOR.
Hadoop + HDFS (just get hadoop to hand over the bytes, or use HDFS directly).
ZeroMQ + GlusterFS.
Apache Spark might work well with python bindings for library, https://github.com/apache/spark.
HDFS + Erlang scheduler (gascheduler) + C++ code as a simple TCP server.