remap
remap copied to clipboard
gtoonstra
MapReduce platform in python
The map/reduce examples have clear boundaries between startup, reading data, processing data and writing it out to disk. The process lifetime doesn't extend beyond those boundaries, which always perpetuates the...
Add new examples for new use cases: - Distributed file compression - Distributed file decompression - Distributed grep example with insertion of lambda function - Aggregation of distributed grep example...
File progress is now calculated on a simple method that adds the number of bytes read from a line (len( x )) to a 'pos' attribute. It should use "tell()"...
The readers and writers now read from a file that has to be in the format specified, but large datasets are very inefficient when stored even on a cluster, because...
Pub/sub messaging is not 100% reliable. If throughput is not too high, then it can be assumed that all currently connected consumers will receive all produced messages. Some systems like...