Crash when setting --s3-copy-cutoff=0

[gramakri]

What is the problem you are having with rclone?

When setting --s3-copy-cutoff=0 there is a divide by 0 crash.

What is your rclone version (output from rclone version)

rclone v1.64.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-86-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Linux Ubuntu 22.04 64-bit

Which cloud storage system are you using? (e.g. Google Drive)

Vultr Object Storage

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

./rclone copyto -vvvv vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz.2 --s3-copy-cutoff=0

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

./rclone copyto -vvvv vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz.2 --s3-copy-cutoff=0
2023/10/26 07:41:57 DEBUG : rclone: Version "v1.64.2" starting with parameters ["./rclone" "copyto" "-vvvv" "vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz" "vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz.2" "--s3-copy-cutoff=0"]
2023/10/26 07:41:57 DEBUG : Creating backend with remote "vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz"
2023/10/26 07:41:57 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/10/26 07:41:57 DEBUG : vultr: detected overridden config - adding "{fjvxS}" suffix to name
2023/10/26 07:41:57 DEBUG : Resolving service "s3" region "us-east-1"
2023/10/26 07:41:57 DEBUG : fs cache: adding new entry for parent of "vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz", "vultr{fjvxS}:cloudron-backups2/snapshot"
2023/10/26 07:41:57 DEBUG : Creating backend with remote "vultr://cloudron-backups2/snapshot/"
2023/10/26 07:41:57 DEBUG : vultr: detected overridden config - adding "{fjvxS}" suffix to name
2023/10/26 07:41:57 DEBUG : Resolving service "s3" region "us-east-1"
2023/10/26 07:41:57 DEBUG : fs cache: renaming cache item "vultr://cloudron-backups2/snapshot/" to be canonical "vultr{fjvxS}:cloudron-backups2/snapshot"
2023/10/26 07:41:58 DEBUG : app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz: Need to transfer - File not found at Destination
panic: runtime error: integer divide by zero

goroutine 1 [running]:
github.com/rclone/rclone/backend/s3.(*Fs).copyMultipart(0xc0008f0000, {0x276b6e8?, 0x39854c0}, 0xc0009ed8c0, {0xc000de6a00, 0x11}, {0xc000de6a12, 0x3a}, {0xc000de6a50, 0x11}, ...)
	github.com/rclone/rclone/backend/s3/s3.go:4289 +0xa92
github.com/rclone/rclone/backend/s3.(*Fs).copy(0xc0008f0000, {0x276b6e8?, 0x39854c0}, 0xc0009ed8c0, {0xc000de6a00, 0x11}, {0xc000de6a12, 0x3a}, {0xc000de6a50, 0x11}, ...)
	github.com/rclone/rclone/backend/s3/s3.go:4217 +0x6cb
github.com/rclone/rclone/backend/s3.(*Fs).Copy(0xc0008f0000, {0x276b6e8, 0x39854c0}, {0x277f848?, 0xc000881170?}, {0x7ffd737dd76e, 0x31})
	github.com/rclone/rclone/backend/s3/s3.go:4360 +0x2ec
github.com/rclone/rclone/fs/operations.Copy({0x276b6e8, 0x39854c0}, {0x277f7d8, 0xc0008f0000}, {0x0?, 0x0?}, {0x7ffd737dd76e, 0x31}, {0x277f848, 0xc000881170})
	github.com/rclone/rclone/fs/operations/operations.go:396 +0x934
github.com/rclone/rclone/fs/operations.moveOrCopyFile({0x276b6e8, 0x39854c0}, {0x277f7d8, 0xc0008f0000}, {0x277f7d8?, 0xc0008f0000?}, {0x7ffd737dd76e, 0x31}, {0x7ffd737dd71b, 0x2f}, ...)
	github.com/rclone/rclone/fs/operations/operations.go:2058 +0xb03
github.com/rclone/rclone/fs/operations.CopyFile(...)
	github.com/rclone/rclone/fs/operations/operations.go:2078
github.com/rclone/rclone/cmd/copyto.glob..func1.1()
	github.com/rclone/rclone/cmd/copyto/copyto.go:60 +0x7b
github.com/rclone/rclone/cmd.Run(0x1, 0x1, 0x2?, 0xc000d2fc88)
	github.com/rclone/rclone/cmd/cmd.go:258 +0x130
github.com/rclone/rclone/cmd/copyto.glob..func1(0xc000a23900?, {0xc0009e94c0, 0x2, 0x4})
	github.com/rclone/rclone/cmd/copyto/copyto.go:56 +0xd2
github.com/spf13/cobra.(*Command).execute(0x391ed20, {0xc0009e9480, 0x4, 0x4})
	github.com/spf13/[email protected]/command.go:944 +0x863
github.com/spf13/cobra.(*Command).ExecuteC(0x3919400)
	github.com/spf13/[email protected]/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:992
github.com/rclone/rclone/cmd.Main()
	github.com/rclone/rclone/cmd/cmd.go:570 +0x71
main.main()
	github.com/rclone/rclone/rclone.go:14 +0xf

[gramakri]

Using a non-zero value like --s3-copy-cutoff=5Gi works fine.

[ncw]

Hmm, yes, rather confusingly rclone uses the --s3-copy-cutoff value as the chunk size for the copy. Perhaps these should be separated and have a separate --s3-copy-chunk-size value. In general you want to use a much larger value than --s3-chunk-size as the copying is done server size.

At the very least setting it to 0 should throw an error rather than crash :-)

[jbonzo]

Since this issue is still open I'd like to point out that the documentation for --s3-copy-cutoff still says a valid minimum value is 0.

It looks like this is no longer the case from the mentioned PR #7400