magpie
magpie copied to clipboard
LLNL
Magpie contains a number of scripts for running Big Data software in HPC environments, including Hadoop and Spark. There is support for Lustre, Slurm, Moab, Torque. LSF, Flux, and more.
Some resource managers track exit codes, can be useful for debugging.
consider re-working MAGPIE_SHUTDOWN_TIME, instead each project/task/thing that needs to be shutdown is automatically given 'MAGPIE_SHUTDOWN_TIME_MINS'. So the shutdown begins with walltime - calculated-shutdown-time instead of some preset value.
may be useful for interesting scheduling options or for porting issues users have
In branch magpie20_erasureencodingterasort, will not work until later https://issues.apache.org/jira/browse/HDFS-7661 although it's possible to hack teragen/terasort to work around this. JIRA says may not be fixed for 3.0 release.
As seen in #202 there are some portability issues to non-GNU seds. ``` rm -f ../script-msub-slurm-srun/magpie.msub-slurm* cp magpie-magpie-customizations magpie-magpie-customizations-substitution if test "n" = "y"; then sed -i -e "/@MAGPIE_NO_LOCAL_DIR@/{r magpie-magpie-customizations-no-local-dir"...
Tachyon/Alluxio is now so much newer, I'm condensing all the older issues into one because each one individually is sort of meaningless now because tachyon support is so out of...
e.g. HDFS over Lustre for long term storage & HDFS on SSDs for caching short term. Right now can't work.
currently job scripts won't work with this because of configurations like ``` #SBATCH --ntasks-per-node=1 ``` In addition, options like ``` #SBATCH --no-kill ``` Present problems, as it makes sense for...
I'd like to see standalone configs so that when changes occur to the sh scripts we don't have to go in and remove machine specific variables. For instance, these variables:...
