bashreduce
bashreduce copied to clipboard
rcrowley
map/reduce in bash
NAME br - bashreduce, map/reduce in bash
SYNOPSIS
br [-h '
DESCRIPTION br implements a map/reduce framework that allows the map and reduce jobs to be constructed from standard Unix tools. It can operate on data piped into it or on files known to exist on the worker nodes.
INSTALLATION br can run out-of-the box but for convenience, put it somewhere on your PATH. The brutils come with a Makefile that will install these tools, probably to /usr/local/bin. If you leave these tools in place relative to br, they'll be picked up and used from there.
You'll need passwordless ssh access to each machine you wish to
use as a worker (even localhost, for now).
OPTIONS -h The list of hosts to be used, specified as a space- delimited list. Specifying the same host multiple times is allowed and is useful for taking explicit advantage of multiple cores.
-f Only pass filenames from the master to the workers,
under the assumption that they all have a mirror of
the data set. This is equivalent to prefixing your
map program with "xargs cat |".
-m The map program, which should expect initial data on
stdin and produce the intermediate data on stdout.
The stderr from this program will be logged [1].
-r The reduce program, which should expect intermediate
data on stdin and produce the final data on stdout.
The stderr from this program will be logged [1].
-M The merge program, which should expect each node's
final data on stdin and produce the unified final
data on stdout. The stderr from this program will
be logged [1].
-i A file or directory to serve as input. If it is a
file, this is equivalent to piping that file into
br. If it is a directory, it is equivalent to
piping the names of every file in that directory
into br.
-o The local file where output will be stored.
Defaults to stdout.
-c The column used by sort to produce the final output.
Defaults to 1.
-S This is directly the -S option from sort(1). It defaults
to 256M.
-t The temporary directory for storing br files. All of
these files will be prefixed with "br_" and, with the
exception of br_stderr, will be cleaned up upon
successful exit. Defaults to /tmp.
-? Shows a help message and details about the arguments.
FILES All of these files are in the tmp directory specified by the user, which defaults to /tmp.
br_job_*
The master uses named pipes to move data (or filenames)
to the worker(s). These pipes are stored here. This
directory is removed at the end of the job.
br_node_*
Each node uses named pipes to buffer data. These
are stored here and do not overlap, even if a
node is specified twice in the host list. These
directories are removed at the end of the job.
br_stderr
This file is cleared at the beginning of each
job. It records stderr from most commands run
by br. This is currently a point of contention
between jobs. This file is NOT removed at the
end of the job.
NOTES [1] Because you can specify a pipeline for any or all of the map, reduce and merge steps, br can only reliably log stderr from the last step in those pipelines.
AUTHORS Erik Frey [email protected] Richard Crowley [email protected]
SEE ALSO http://github.com/erikfrey/bashreduce http://github.com/rcrowley/bashreduce
LICENSE The goodwill of Erik Frey.
rcrowley