find rsync parallel tar as reserved words within crush grammar

[omac777]

Rather than using outside find rsync parallel tar binaries being called from within crush, I would recommend integrating rustified versions of these within crush as first-class reserved words. The rustified versions should preserve all existing c-based rsync/parallel/tar cli capabilities.

export MAX_CPU_CORES=16

#create destination directory
mkdir -p /home/blahuser/testsourcedirectory/

#create sourcedirectory's directory tree structure within destination directory
rsync --archive --include='*/' --exclude='*' /home/blahuser/testdestinationdirectory/ /home/blahuser/testsourcedirectory/

# Make a file listing from the source directory 
# Pass it along to parallel which will run as many file copies as it can in parallel, 
# in this case 16 files simultaneously.
# It will invoke the next file copy as soon as one of the previous file copies is done
cd /home/blahuser/testdestinationdirectory/; find -L . -type f | parallel -j $MAX_CPU_CORES rsync --archive {} /home/blahuser/testsourcedirectory/{}

I perceive performance improvements from rustified versions for each mentioned tool:

• find : it could use the parallel directory walker and return file listings faster

• rsync : could make use of the parallel directory walker also while also making use of async file copying for better performance there too.

• parallel : could make use of flume/crossbeam channels and perform faster than gnu parallel.

• tar : could be made faster for the same above reasons as find, rsync and parallel.

Following all of this thought, I actually believe crush could become the next generation faster busybox. https://en.wikipedia.org/wiki/BusyBox "BusyBox is a software suite that provides several Unix utilities in a single executable file."

[liljencrantz]

I agree what these should be builtins. Patches are welcome!

[aaronjanse]

I bet https://github.com/sharkdp/fd could be useful :-)

Also see https://github.com/uutils/coreutils

[aaronjanse]

Oh, or https://github.com/redox-os/coreutils!