nathanhaigh
Don't works when source and destination contains space characters in their paths
Got issues when I tried using prsync using file paths with spaces in it like "/Volumes/SD Card"
Original command :
prsync -avhW --inplace --no-compress --progress --exclude='node_modules' /Volumes/SD-A\ 2023/Backup\ SD\ CARD\ Candice\ HDD\ Maxor\ 2TB\ Red\ Case/ /Volumes/Candice\ 2T/backup-fast
# Tested with this also
prsync -avhW --inplace --no-compress --progress --exclude='node_modules' \"/Volumes/SD-A 2023/Backup SD CARD Candice HDD Maxor 2TB Red Case\" \"/Volumes/Candice 2T/backup-fast\"
# And this
prsync -avhW --inplace --no-compress --progress --exclude='node_modules' \"/Volumes/SD-A 2023/Backup SD CARD Candice HDD Maxor 2TB Red Case\" \"/Volumes/Candice 2T/backup-fast\"
rsync --files-from=/var/folders/c8/g84x62xj5vjfd0vdh892yhb80000gn/T/tmp.6ozsPah7GM/chunk.6 -avhW --inplace --no-compress --progress --exclude=node_modules /Volumes/SD-A 2023/Backup SD CARD Candice HDD Maxor 2TB Red Case/ /Volumes/Candice 2T/backup-fast
# Return error code 1
# Rsync log the help command output
Ugh. My initial response is: don't use spaces in file paths as it's bad practice. However, I then have to remind myself that sometimes these things happen outside of our control. I'll try to take a look at making it handle whitespaces correctly.
Hey @nathanhaigh great tool you've created here mate. I am about to give it a test spin and then I saw this comment here. Unfortunately I am trying to copy one Plex library from one drive to another and there are a whole heap of white space characters littered all over the place lol... Kinda scared this script is going to fall over at it but going to give it a shot and see what happens anyway.
Thanks for the great work putting this together! I am going to go try it out now and see how it goes. I might dig a bit deeper and see if we can include the support for white spaces without over complicating the script too much!
Ah yes, failed because of my exclude (which had a Windows folder path for the (x86) folder
rsync -avxhHAWXS --numeric-ids --info=progress2 --exclude 'Program Files (x86)' /mnt/eHDD01/ /mnt/eHDD02/
Above is the rsync command I wanted to parallelise.
Output of prsync command error:
INFO: Using up to 5 processes for transfer ...
INFO: Determining file list for transfer ...
INFO: 12567 (4672767 MB) files to transfer.
INFO: Distributing files among chunks ...
")syntax error: operand expected (error token is "
DONE (2s)
INFO: Starting transfers ...
rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.2 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/
/usr/bin/bash: -c: line 1: syntax error near unexpected token `('
/usr/bin/bash: -c: line 1: `rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.2 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/'
local:0/1/100%/1.0s rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.4 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/
rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.1 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/
/usr/bin/bash: -c: line 1: syntax error near unexpected token `('
/usr/bin/bash: -c: line 1: `rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.4 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/'
local:1/2/100%/0.5s rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.0 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/
/usr/bin/bash: -c: line 1: syntax error near unexpected token `('
/usr/bin/bash: -c: line 1: `rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.1 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/'
local:1/3/100%/0.3s rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.3 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/
/usr/bin/bash: -c: line 1: syntax error near unexpected token `('
/usr/bin/bash: -c: line 1: `rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.0 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/'
local:1/4/100%/0.2s
Computers / CPU cores / Max jobs to run
1:local / 12 / 5
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:1/4/100%/0.2s /usr/bin/bash: -c: line 1: syntax error near unexpected token `('
/usr/bin/bash: -c: line 1: `rsync --files-from=/tmp/tmp.861qk8sjHY/chunk.3 -avxhHAWXS --numeric-ids --info=progress2 --exclude Program Files (x86) /mnt/eHDD01/ /mnt/eHDD02/'
local:0/5/100%/0.2s
I'll poke around a bit and see what I can figure out, thanks again for the effort putting the foundation together!
Did a tiny bit of trial and error, modified the script for my use case (not a universal fix). Transfer speed with parallels equals single thread for my use case, so I am going back to running rsync manually in one thread.
Thanks again! I am keeping this in my toolkit as I can see real value in parallelisation doing a sync across WAN or maybe even LAN.
I hit this edge case, in my case the target directory had a space, fixed it by replacing the argument with a symlink.
IMO a possible, more universal solution would be to pipe the file list using rsync ... --files-from=- ... together with parallel --pipe-part -a ${TMPDIR}/files.all, and then introduce the command line arguments using :::
This would upset the optimization you do with chunking / tac-cing. Maybe some manipulation of the files.all list could be implemented to ensure that small and large files are transferred simultaneously for the speed gain that I very much see with your script.
