rclone
Union remote of google team drives does not work with --drive-server-side-across-configs flag
Output of rclone version
rclone v1.58.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.13.0-51-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none
Describe the issue
I and many others have noticed that if you wanted to sync a union of 2 team drives to either another google drive union mount or an individual google team drive while using the flag --drive-server-side-across-configs=true, rclone ignores the flag and downloads the files to the endpoint that calls the command and then uploads it to its destination.
Here is an example of my rclone config and the command:
rclone copy 'teamdrive-combo:TestFolder/large20GB.txt ' TeamDriveC:clones --fast-list --transfers=16 --checkers=32 --tpslimit-burst=16 -vP --ignore-errors --stats=30s --max-backlog=2000000 --ignore-case --size-only --no-update-modtime --drive-chunk-size=256M --drive-use-trash=false --use-mmap --drive-server-side-across-configs=true --drive-stop-on-upload-limit --drive-skip-gdocs --timeout=10s --tpslimit=4
rclone config:
[teamdrive-combo] type = union upstreams = TeamDriveA: TeamDriveB:
[TeamDriveA] type = drive client_id = xxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com client_secret = xxxxxxxxxxxxxxxxxxxxxxxx scope = drive token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","r>team_drive = xxxxxxxxxxxxxxxxxxxxxxxx root_folder_id =
[TeamDriveB] type = drive client_id = xxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com client_secret = xxxxxxxxxxxxxxxxxxxxxxxx scope = drive token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","r>team_drive = xxxxxxxxxxxxxxxxxxxxxxxx root_folder_id =
[TeamDriveC] type = drive client_id = xxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com client_secret = xxxxxxxxxxxxxxxxxxxxxxxx scope = drive token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","r>team_drive = xxxxxxxxxxxxxxxxxxxxxxxx root_folder_id =
I'm not sure this will fix it, but you have the server side configs on the union side but not on the remotes as the default is false.
Each Drive remote should have server side config on in it.
https://rclone.org/drive/#drive-server-side-across-configs
Can you enable it on each remote, test without the union and validate it's working and then test with the union. I feel it probably won't work on the union, but wanted to be sure it's working between remotes first.
I have tested prior and again just now that TeamDriveA -> TeamDriveB -> TeamDriveC works with the server side flag.
So can you share your updated rclone.conf that you have, the logs showing that it works and what happens with a debug log with the union?
I'm having the same problem. Drive to drive server side works fine, but drive to unionized drive does not. See this forum post.
You can also see a very recent commit made the server side flag global, even though backends still have to use it.
@halolordkiller3 can you try the latest beta with the --server-side-across-configs flag and see if that helps?
@halolordkiller3 can you try the latest beta with the
--server-side-across-configsflag and see if that helps?
I already did that. Nothing changed. Union probably needs a "passthrough mode" like chunker-same-settings and crypt-same-password
@halolordkiller3 can you try the latest beta with the
--server-side-across-configsflag and see if that helps?I already did that. Nothing changed. Union probably needs a "passthrough mode" like chunker-same-settings and crypt-same-password
Can you show a log with -vv of what happens here? I think this should work and I'd like to understand why not.
Can you show a log with -vv of what happens here? I think this should work and I'd like to understand why not.
Sure thing. I've attached both the test config and the logs of what i'm doing.
The main problem for me now is that after the upload the files show up after A LOT of time (almost 60 minutes, even though it seems to be shorter if the file is inside a folder).
Try this beta with the --server-side-across-configs flag
v1.60.0-beta.6439.317338358.fix-6287-server-side-across-configs on branch fix-6287-server-side-across-configs (uploaded in 15-30 mins)
I made the --server-side-across-configs flag allow cross backend server side copies which I think should fix your problem.
Hello, I tried but nothing changed. For context, I'm trying to copy a file (in root folder) from a drive to a union of 10 drives. server_side_across_configs is enabled both in config and in command line.
Hello, I tried but nothing changed.
Can you show the log with -vv please?
For context, I'm trying to copy a file (in root folder) from a drive to a union of 10 drives. server_side_across_configs is enabled both in config and in command line.
Hopefully the log will shed some light!
My guess is that the specific Copy function of the backend blocks the operation Which is here https://github.com/rclone/rclone/blob/bd787e8f457705de9072335fda879f575da4f3cf/backend/drive/drive.go#L2438 So I guess all the backends need to be changed to also allow the union backend Someone should check that all the remotes of a union are of the same type and are of the same type of the destination, though
Its the same line but in the union backend which is blocking the copy I think.
Try this which removes that line from the union and delegates the copying to the backend.
v1.60.0-beta.6441.4488bc0cd.fix-6287-server-side-across-configs on branch fix-6287-server-side-across-configs (uploaded in 15-30 mins)
Still the same, but now "Can't copy - not same remote type" does not appear anymore
Going at 500 kB/s instead of not being instant? Just take the last log I posted and remove the line i said
Hi, I am having this issue too, this message gets printed while transferring a file from a shared drive to my drive (see config below):
DEBUG : shared6/XXX/file.ext: Can't copy - not same remote type
Here is the complete debug log for the transfer:
user@host:~$ rclone -vv copy --server-side-across-configs --progress gdrive-all:"Shared Drives/shared6/XXX/file.ext" gdrive-all:"My Drive/test/"
2024/01/23 12:35:33 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "-vv" "copy" "--server-side-across-configs" "--progress" "gdrive-all:Shared Drives/shared6/XXX/file.ext" "gdrive-all:My Drive/test/"]
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-all:Shared Drives/shared6/XXX/file.ext"
2024/01/23 12:35:33 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared7:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared3:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared1:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared5:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared2:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared6:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive-shared4:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{WN5sg}" suffix to name
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{A31oQ}" suffix to name
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{X3kXp}" suffix to name
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{TjJbc}" suffix to name
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{l-Bn9}" suffix to name
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{02fYy}" suffix to name
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{02fYy}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared3:" to be canonical "gdrive{02fYy}:"
2024/01/23 12:35:33 DEBUG : Creating backend with remote "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:"
2024/01/23 12:35:33 DEBUG : gdrive: detected overridden config - adding "{jeEtu}" suffix to name
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{WN5sg}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared5:" to be canonical "gdrive{WN5sg}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{l-Bn9}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared4:" to be canonical "gdrive{l-Bn9}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{TjJbc}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared7:" to be canonical "gdrive{TjJbc}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{A31oQ}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared2:" to be canonical "gdrive{A31oQ}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{X3kXp}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared6:" to be canonical "gdrive{X3kXp}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:" to be canonical "gdrive{jeEtu}:"
2024/01/23 12:35:33 DEBUG : fs cache: renaming cache item "gdrive-shared1:" to be canonical "gdrive{jeEtu}:"
2024/01/23 12:35:33 DEBUG : Google drive root '': 'root_folder_id = XXXXXXXXXXXXXXXXXXX' - save this in the config to speed up startup
2024/01/23 12:35:34 DEBUG : fs cache: adding new entry for parent of "gdrive-all:Shared Drives/shared6/XXX/file.ext", "gdrive-all:Shared Drives/shared6/XXX"
2024/01/23 12:35:34 DEBUG : Creating backend with remote "gdrive-all:My Drive/test/"
2024/01/23 12:35:34 DEBUG : fs cache: renaming cache item "gdrive-all:My Drive/test/" to be canonical "gdrive-all:My Drive/test"
2024/01/23 12:35:35 DEBUG : file.ext: Need to transfer - File not found at Destination
2024/01/23 12:35:35 DEBUG : shared6/XXX/file.ext: Can't copy - not same remote type
2024/01/23 12:35:36 DEBUG : test/file.ext: Sending chunk 0 length 8388608
2024/01/23 12:35:37 DEBUG : test/file.ext: Sending chunk 8388608 length 8388608
2024/01/23 12:35:38 DEBUG : test/file.ext: Sending chunk 16777216 length 8388608
2024/01/23 12:35:38 DEBUG : test/file.ext: Sending chunk 25165824 length 8388608
2024/01/23 12:35:38 DEBUG : test/file.ext: Sending chunk 33554432 length 8388608
2024/01/23 12:35:39 DEBUG : test/file.ext: Sending chunk 41943040 length 8388608
2024/01/23 12:35:39 DEBUG : test/file.ext: Sending chunk 50331648 length 8388608
2024/01/23 12:35:39 DEBUG : test/file.ext: Sending chunk 58720256 length 8388608
2024/01/23 12:35:40 DEBUG : test/file.ext: Sending chunk 67108864 length 8388608
2024/01/23 12:35:40 DEBUG : test/file.ext: Sending chunk 75497472 length 8388608
2024/01/23 12:35:41 DEBUG : test/file.ext: Sending chunk 83886080 length 8388608
2024/01/23 12:35:41 DEBUG : test/file.ext: Sending chunk 92274688 length 8388608
2024/01/23 12:35:41 DEBUG : test/file.ext: Sending chunk 100663296 length 8388608
2024/01/23 12:35:42 DEBUG : test/file.ext: Sending chunk 109051904 length 8388608
2024/01/23 12:35:42 DEBUG : test/file.ext: Sending chunk 117440512 length 8388608
2024/01/23 12:35:43 DEBUG : test/file.ext: Sending chunk 125829120 length 8388608
2024/01/23 12:35:43 DEBUG : test/file.ext: Sending chunk 134217728 length 8044316
2024/01/23 12:35:44 DEBUG : file.ext: md5 = 9c2fa2ec844be571d1feb21d0393ba66 OK
2024/01/23 12:35:44 INFO : file.ext: Copied (new)
Transferred: 135.672 MiB / 135.672 MiB, 100%, 15.074 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 11.1s
2024/01/23 12:35:44 INFO :
Transferred: 135.672 MiB / 135.672 MiB, 100%, 15.074 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 11.1s
2024/01/23 12:35:44 DEBUG : 9 go routines active
Rclone config
I censored personal info, changed the shared drives names for privacy and updated the config with remotes name matching the new shared drives names, but the main structure of my real config file is the same, only the names change.
What I plan to use as main remote to interact with my Google Drive is gdrive-all: the configuration uses combine remotes and aliases to obtain a structure for gdrive-all which strictly mirrors the one in Google Drive web UI: it has two subdirectories, one for "My Drive" and one for "Shared Drives".
Here is a sketch of the "virtual directory structure" created (the remotes names are in brackets):
[gdrive-full]
├── My Drive [gdrive]
│ ├── ... files in my own drive
│ └── ...
└── Shared Drives [gdrive-shared]
├── shared1 [gdrive-shared1]
│ └── ... files in first shared drive
├── ...
├── shared7 [gdrive-shared7]
└── ... files in last shared drive
I also added server_side_across_configs everywhere to see if something would change but to no avail, the transfer is not server side as far as I can tell, because of the debug message reported above.
Here is my rclone.conf:
[gdrive]
type = drive
client_id = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.apps.googleusercontent.com
client_secret = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
scope = drive
token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"XXX"}
server_side_across_configs = true
[gdrive-shared1]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared2]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared3]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared4]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared5]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared6]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared7]
type = alias
remote = gdrive,team_drive=XXXXXXXXXXXXXXXXXXX,root_folder_id=:
server_side_across_configs = true
[gdrive-shared]
type = combine
upstreams = "shared1=gdrive-shared1:" "shared2=gdrive-shared2:" "shared3=gdrive-shared3:" "shared4=gdrive-shared4:" "shared5=gdrive-shared5:" "shared6=gdrive-shared6:" "shared7=gdrive-shared7:"
server_side_across_configs = true
[gdrive-all]
type = combine
upstreams = "Shared Drives=gdrive-shared:" "My Drive=gdrive:"
server_side_across_configs = true
