rclone
How rclone can read the realtime ak/sk/securytoken
In practical scenarios, we are often limited to using temporary access keys, secret keys, and security tokens to access S3, which typically expire after a few hours. When utilizing rclone mount, we don't seem to have a way to directly perceive or address the expiration of these authentication credentials. My question is, is there a method to dynamically update and maintain these credentials while they're in use, ensuring the mounted S3 directory remains accessible without any interruptions caused by their expiration? Your insights are eagerly awaited, and I sincerely hope to hear back from you soon.
agree that, i used env_auth=true, then config the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. But when I update these environment variables, it seems that the "mount" process is not loading the latest environment variables or credentials files.
You can update credentials via the API
Check out https://rclone.org/s3/#set for details.
You can update credentials via the API
Check out https://rclone.org/s3/#set for details.
This doesn't seem to work for mount points that have already been mounted.
Firstly, i mounted with the wrong ak/sk/session_token. You can see my mount points below:
[root@ab9b-f6b4 /]# rclone rc mount/listmounts
{
"mountPoints": [
{
"Fs": "obs-mount:pctest-obs-minpolicy/user3",
"MountPoint": "/data/obs-mount2",
"MountedOn": "2024-07-24T15:14:50.645714352+08:00"
}
]
}
When I try to view the files of the mount point with the "ls" command, it apparently gives an error:
# The ls command result:
[root@ab9b-f6b4 /]# ls /data/obs-mount2/
ls: reading directory '/data/obs-mount2/': Input/output error
# The rclone rcd log:
2024/07/24 15:59:25 DEBUG : : Dir.ReadDirAll error: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 00000190E3C08DBA401466F0638D9, host id: MOCKQjIxQTYxQzZDMDAwMDAxMzQwMzFCRTgwMDUyOTNBQUFBQUFBQ
2024/07/24 15:59:25 ERROR : IO error: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 00000190E3C08DBA401466F0638D9, host id: MOCKQjIxQTYxQzZDMDAwMDAxMzQwMzFCRTgwMDUyOTNBQUFBQUFBQ
2024/07/24 15:59:25 DEBUG : /: >ReadDirAll: item=-1, err=InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 00000190E3C08DBA401466F0638D9, host id: MOCKQjIxQTYxQzZDMDAwMDAxMzQwMzFCRTgwMDUyOTNBQUFBQUFBQ
In the meantime, I used the operations/list command as a control group for the experiment, it gives the same error:
[root@ab9b-f6b4 /]# rclone rc operations/list fs=obs-mount: remote=pctest-obs-minpolicy/user3
{
"error": "error in ListJSON: AccessDenied: Access Denied\n\tstatus code: 403, request id: , host id: ",
"input": {
"fs": "obs-mount:",
"remote": "pctest-obs-minpolicy/user3"
},
"path": "operations/list",
"status": 500
}
2024/07/24 15:56:31 Failed to rc: operation "operations/list" failed: error in ListJSON: AccessDenied: Access Denied
status code: 403, request id: , host id:
Next, I update the correct ak/sk/session_token with the "backend/command command=set" command, and you can see that the "operations/list" command works correctly, but the "ls" command still gives the error:
[root@ab9b-f6b4 /]# rclone rc backend/command command=set fs=obs-mount: -o access_key_id=correct_ak -o secret_access_key=correct_sk -o session_token=correct_st
{
"result": null
}
[root@ab9b-f6b4 /]# rclone rc operations/list fs=obs-mount: remote=pctest-obs-minpolicy/user3
{
"list": [
{
"IsDir": true,
"MimeType": "inode/directory",
"ModTime": "2000-01-01T00:00:00.000000000Z",
"Name": "project1",
"Path": "pctest-obs-minpolicy/user3/project1",
"Size": 0
}
]
}
[root@ab9b-f6b4 /]# rclone rc fscache/clear fs=obs-mount:pctest-obs-minpolicy/user3
[root@ab9b-f6b4 /]# ls /data/obs-mount2/
ls: reading directory '/data/obs-mount2/': Input/output error
rclone rc backend/command command=set fs=obs-mount: -o access_key_id=correct_ak -o secret_acces...
You have to use exactly the same fs that you used in the mount command. So something like
rclone rc backend/command command=set fs=obs-mount:pctest-obs-minpolicy/user3 ...
That should work hopefully.
Rclone instantiates a different backend for each fs argument which explains your results above.
rclone rc backend/command command=set fs=obs-mount: -o access_key_id=correct_ak -o secret_acces...
You have to use exactly the same
fsthat you used in the mount command. So something likerclone rc backend/command command=set fs=obs-mount:pctest-obs-minpolicy/user3 ...That should work hopefully.
Rclone instantiates a different backend for each
fsargument which explains your results above.
How do I see how many "backends" I currently have? and remove or modify these "backends"?
when i use rclone rc mount/mount fs=obs-mount:pctest-obs-minpolicy/user3 mountPoint=/data/obs-mount2/ vfsOpt='{"CacheMode": 2}' mountOpt='{"AllowOther": true}' -vv，the rc log is：
2024/07/24 18:35:52 DEBUG : rc: "mount/mount": with parameters map[fs:obs-mount:pctest-obs-minpolicy/user3 mountOpt:{"AllowOther": true} mountPoint:/data/obs-mount2/ vfsOpt:{"CacheMode": 2}]
2024/07/24 18:35:52 DEBUG : Creating backend with remote "obs-mount:pctest-obs-minpolicy/user3"
2024/07/24 18:35:52 DEBUG : Using anonymous credentials - did you mean to set env_auth=true?
2024/07/24 18:35:52 DEBUG : Resolving service "s3" region "xxx"
2024/07/24 18:35:52 INFO : S3 bucket pctest-obs-minpolicy path user3: poll-interval is not supported by this remote
2024/07/24 18:35:52 DEBUG : vfs cache: root is "/root/.cache/rclone"
2024/07/24 18:35:52 DEBUG : vfs cache: data root is "/root/.cache/rclone/vfs/obs-mount/pctest-obs-minpolicy/user3"
2024/07/24 18:35:52 DEBUG : vfs cache: metadata root is "/root/.cache/rclone/vfsMeta/obs-mount/pctest-obs-minpolicy/user3"
2024/07/24 18:35:52 DEBUG : Creating backend with remote "/root/.cache/rclone/vfs/obs-mount/pctest-obs-minpolicy/user3"
2024/07/24 18:35:52 DEBUG : Creating backend with remote "/root/.cache/rclone/vfsMeta/obs-mount/pctest-obs-minpolicy/user3"
2024/07/24 18:35:52 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2024/07/24 18:35:52 DEBUG : S3 bucket pctest-obs-minpolicy path user3: Mounting on "/data/obs-mount2/"
2024/07/24 18:35:52 DEBUG : Mount for S3 bucket pctest-obs-minpolicy path user3 created at /data/obs-mount2/ using mount
2024/07/24 18:35:52 DEBUG : rc: "mount/mount": reply map[]: <nil>
then i use rclone rc backend/command command=set fs=obs-mount:pctest-obs-minpolicy/user3 -o access_key_id xxxx, that's works well:
[root@ab9b-f6b4 /]# ls /data/obs-mount2/
project1
I was trying to confirm that the "set" command worked, so I re-set an incorrect ak/sk/session_token and found that the "ls" mount point still shows the files (as if the "set" command only works the first time):
[root@ab9b-f6b4 /]# ls /data/obs-mount2/
project1
[root@ab9b-f6b4 /]# rclone rc backend/command command=set fs=obs-mount:pctest-obs-minpolicy/user3 -o access_key_id=aa -o secret_access_key=bb -o session_token=cc
{
"result": null
}
[root@ab9b-f6b4 /]# ls /data/obs-mount2/
project1
# The rcd log is:
2024/07/24 18:49:38 DEBUG : rc: "backend/command": with parameters map[command:set fs:obs-mount:pctest-obs-minpolicy/user3 opt:map[access_key_id:aa secret_access_key:bb session_token:cc]]
2024/07/24 18:49:38 DEBUG : Resolving service "s3" region "xxx"
2024/07/24 18:49:38 NOTICE: S3 bucket pctest-obs-minpolicy path user3: Updated config values: secret_access_key, session_token, access_key_id
2024/07/24 18:49:38 DEBUG : rc: "backend/command": reply map[result:<nil>]: <nil>
rclone rc backend/command command=set fs=obs-mount: -o access_key_id=correct_ak -o secret_acces...
You have to use exactly the same
fsthat you used in the mount command. So something likerclone rc backend/command command=set fs=obs-mount:pctest-obs-minpolicy/user3 ...That should work hopefully.
Rclone instantiates a different backend for each
fsargument which explains your results above.
Another issue I found is that when I use the "mount" command, the backend log shows that multiple "backends" are created. So when I update via the "backend set" command, do I need to update multiple backends for it to take effect?
2024/07/25 18:45:13 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "mount2" "obs-mount:pctest-obs-minpolicy/user3" "/data/obs-mount2" "--uid=1000" "--gid=100" "--allow-other" "--vfs-cache-mode=full" "--daemon" "--cache-dir=/data/obs-mount2/../.cache/" "--vfs-cache-max-size=5G" "--log-file=/data/obs-mount2/../log/rclone.log" "-vv"]
2024/07/25 18:45:13 DEBUG : Creating backend with remote "obs-mount:pctest-obs-minpolicy/user3"
2024/07/25 18:45:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/07/25 18:45:13 DEBUG : Using anonymous credentials - did you mean to set env_auth=true?
2024/07/25 18:45:13 DEBUG : Resolving service "s3" region "xxx"
2024/07/25 18:45:13 DEBUG : rclone: Version "v1.67.0" starting with parameters ["/usr/bin/rclone" "mount2" "obs-mount:pctest-obs-minpolicy/user3" "/data/obs-mount2" "--uid=1000" "--gid=100" "--allow-other" "--vfs-cache-mode=full" "--daemon" "--cache-dir=/data/obs-mount2/../.cache/" "--vfs-cache-max-size=5G" "--log-file=/data/obs-mount2/../log/rclone.log" "-vv"]
2024/07/25 18:45:13 DEBUG : Creating backend with remote "obs-mount:pctest-obs-minpolicy/user3"
2024/07/25 18:45:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/07/25 18:45:13 DEBUG : Using anonymous credentials - did you mean to set env_auth=true?
2024/07/25 18:45:13 DEBUG : Resolving service "s3" region "xxx"
2024/07/25 18:45:13 INFO : S3 bucket pctest-obs-minpolicy path user3: poll-interval is not supported by this remote
2024/07/25 18:45:13 DEBUG : vfs cache: root is "/data/.cache"
2024/07/25 18:45:13 DEBUG : vfs cache: data root is "/data/.cache/vfs/obs-mount/pctest-obs-minpolicy/user3"
2024/07/25 18:45:13 DEBUG : vfs cache: metadata root is "/data/.cache/vfsMeta/obs-mount/pctest-obs-minpolicy/user3"
2024/07/25 18:45:13 DEBUG : Creating backend with remote "/data/.cache/vfs/obs-mount/pctest-obs-minpolicy/user3"
2024/07/25 18:45:13 DEBUG : Creating backend with remote "/data/.cache/vfsMeta/obs-mount/pctest-obs-minpolicy/user3"
2024/07/25 18:45:13 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2024/07/25 18:45:13 DEBUG : S3 bucket pctest-obs-minpolicy path user3: Mounting on "/data/obs-mount2"
I was trying to confirm that the "set" command worked, so I re-set an incorrect ak/sk/session_token and found that the "ls" mount point still shows the files (as if the "set" command only works the first time):
Remember rclone has a directory cache, so you need to wait for that to expire before the new creds will be needed
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
I suspect your testing has been confounded by that.
You can also use the rc command vfs/forget to flush the directory cache.
Another issue I found is that when I use the "mount" command, the backend log shows that multiple "backends" are created. So when I update via the "backend set" command, do I need to update multiple backends for it to take effect?
Just the one backend - the other ones are internal to manage the VFS cache. In your case this one
2024/07/25 18:45:13 DEBUG : Creating backend with remote "obs-mount:pctest-obs-minpolicy/user3"