magpie

magpie copied to clipboard

once github picks a new default, change it to that

chu11

the basic tfadd is stupid, need to learn tensorflow distributed well enough to do (perhaps) a distributed add based on number of nodes. Also it should exit when the job...

chu11

add node mapping to magpie output

1

for example: node-0 -> mycluster18 node 1 -> mycluster43 node 2 -> mycluster48 so users can map anonymous "node-X" to the actual cluster name more easily

chu11

spark - option to configure queue size

1

spark.scheduler.listenerbus.eventqueue.size defaults to 10000

chu11

enable spark.worker.cleanup by default

1

i.e. export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=yes"

chu11

for example, when killing a script for running too long. Will help with grepping in large jobs outputs.

chu11

launch magpie w/ command line tool instead of submission script

1

For a variety of advanced scenarios, there have been requests to setup hadoop/spark/etc. with a command line tool. In addition, shutdown would be the user's responsibility via the command line...

chu11

What i need to do to get a working copy of magpie?

3

Hi, After some days trying to start using magpie i don't know what to do. I'm trying to use the basic terasort example but when i execute the job in...

grod-uy

RDMA Hadoop/Spark not working with Slurm submission scripts

3

I have configured rdma hadoop and spark by myself in an InfiniBand cluster and it works, but when I try to use the submission script magpie.sbatch-srun-spark-with-yarn-and-hdfs (just for testing hadoop...

casty8

can be confusing, user may think yes means "I can only run things one time".

chu11