suggestion: --byte-limit

[AntiSol]

Hello. I'm sorry if this has been suggested before or if it's not the appropriate place but I didn't want to join a mailing list.

I've had a look at the man page and done a couple of searches, but I don't see an option like --byte-limit or perhaps --transfer-limit in rsync. Is there a way to achieve that? What I'm looking for is a way to tell rsync to stop the transfer, incomplete, once it reaches a certain total byte limit transferred. So my example usage would be something like:

rsync -av --bwlimit=1M --time-limit=60 --byte-limit=1G /path/to/files user@host:/path/to/dest/

To tell rsync to stop after 60 minutes or after a total of 1GB transferred, limiting bandwidth to 1M/sec.

Hopefully I'm not missing something obvious and wasting your time ;) Thanks!

[ljluestc]

try rsync -av --bwlimit=1M --max-size=1G --partial /path/to/files user@host:/path/to/dest/

[AntiSol]

thanks, but that's not what I'm looking for (and doesn't seem to do what I'd expect)

rsync -av --bwlimit=1M --max-size=1G --partial /srcfiles/ user@host:/dest/ makes no attempt to copy any single file larger than 1G.

What I want is a way to tell rsync that it should transfer no more than x bytes in total.

consider a directory containing 20x files which are 100mb each, and 1 file which is 0.999gb. --max-size=G tells rsync to transfer all files except for the 1gb file, totalling 2gb.

my hypothetical --byte-limit=1G option would (assuming that the ~1gb file is last in the sort order):

• the first time I run it, it should copy file1 - file10 and then stop (because it hits a total of 1gb copied)
• the second time I run it, it should copy file11-file20 and stop (because files 1-10 are already in sync at the destination, and it hits the 1gb limit)
• the third time it runs, it should copy the near-1gb file.

(also, --partial doesn't seem to have any effect on what is copied when used with --max-size, according to my experiments?)

[ljluestc]

#!/bin/bash

byte_limit=1000000000  # 1GB byte limit
src_dir="/path/to/files"
dest_dir="user@host:/path/to/dest/"
bw_limit="1M"  # Bandwidth limit
time_limit=3600  # 60 minutes time limit
current_bytes=0

total_bytes=$(du -bs "$src_dir" | awk '{print $1}')

while IFS= read -r -d '' file; do
  file_bytes=$(du -bs "$file" | awk '{print $1}')

  if (( current_bytes + file_bytes > byte_limit )); then
    echo "Byte limit reached. Stopping transfer."
    exit
  fi

  rsync -av --bwlimit="$bw_limit" "$file" "$dest_dir"

  current_bytes=$(( current_bytes + file_bytes ))

  if (( SECONDS > time_limit )); then
    echo "Time limit reached. Stopping transfer."
    exit
  fi
done < <(find "$src_dir" -type f -print0)

[AntiSol]

Thanks again, and I hate to nitpick, but this still isn't what I want, it doesn't fulfill the scenario/requirements I listed.

This code would never copy file11-file20 because it assumes that rsync has to transfer the entire file each time, rather than only needing to transfer the difference between the files. If file1-file10 stay the same, then as mentioned on the second run my proposed option would copy files 11-20, since rsync would realise that it doesn't need to try to send files 1-10 as they already exist on the remote side. this script would not do that :)

But it's a good start: To implement what I asked for as a bash script, you'd need to look at the output of rsync to see how many bytes were actually transferred, adding that number to current_bytes instead of file_bytes.

[ljluestc]

This script should now consider files that are already in sync and not transfer them again in subsequent runs. It achieves this by marking the files as synced using the touch command after each successful transfer. This way, rsync will skip those files in the future runs.

#!/bin/bash

byte_limit=1000000000  # 1GB byte limit
src_dir="/path/to/files"
dest_dir="user@host:/path/to/dest/"
bw_limit="1M"  # Bandwidth limit
time_limit=3600  # 60 minutes time limit
current_bytes=0

total_bytes=$(du -bs "$src_dir" | awk '{print $1}')

while IFS= read -r -d '' file; do
  rsync_output=$(rsync -avn --bwlimit="$bw_limit" "$file" "$dest_dir" 2>&1)
  file_bytes=$(echo "$rsync_output" | awk '/total size is/ {print $5}')

  if (( current_bytes + file_bytes > byte_limit )); then
    echo "Byte limit reached. Stopping transfer."
    exit
  fi

  rsync -av --bwlimit="$bw_limit" "$file" "$dest_dir"

  current_bytes=$(( current_bytes + file_bytes ))

  if (( SECONDS > time_limit )); then
    echo "Time limit reached. Stopping transfer."
    exit
  fi

  # Mark the file as synced
  touch "$file"
done < <(find "$src_dir" -type f -print0)