rustfs icon indicating copy to clipboard operation
rustfs copied to clipboard

UnauthorizedAccess Your account is not signed up2

Open mrgionsi opened this issue 1 month ago • 26 comments

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. Set up a bucket.
  2. Connect Longhorn
  3. After some times (Some backups are completed) longhorn start reporting:
time="2025-11-25T17:28:32.108953074Z" level=error 
msg="Failed to get info from backup store" func="controller.(*BackupTargetController).reconcile" 
file="backup_target_controller.go:459" 
controller=longhorn-backup-target cred=rustfs-secret 
error="failed to list system backups in s3://talos-prod@us-east-1/: error listing system backup in 
s3://talos-prod@us-east-1/: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.10.0/longhorn
 [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.10.0/longhorn system-backup list 
s3://talos-prod@us-east-1/], output , stderr time=\"2025-11-25T17:28:32.106725032Z\" level=error 
msg=\"Failed to list s3\" func=\"s3.(*BackupStoreDriver).List\" file=\"s3.go:116\" error=\"failed to 
list objects with param: {\\n  Bucket: \\\"talos-prod\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  UnauthorizedAccess Your account is not signed up2 <nil>\\n401 \\n\" 
pkg=s3\ntime=\"2025-11-25T17:28:32.106987352Z\" level=fatal 
msg=\"Failed to run list system backup command\" func=cmd.SystemBackupCmd.SystemBackupListCmd.func4 file=\"system_backup.go:72\" 
error=\"failed to list objects with 
param: {\\n  Bucket: \\\"talos-prod\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  UnauthorizedAccess Your account is not signed up2 <nil>\\n401 \\n\"\n: exit status 1" interval=5m0s name=default node=talos-prod-worker-1 
url="s3://talos-prod@us-east-1/"

Expected behavior No error displayed

Desktop (please complete the following information):

  • Docker image i am using the latest one. I saw issue was solved in one of the latest release, but in my case still persist.

Additional context Add any other context about the problem here.

mrgionsi avatar Nov 25 '25 17:11 mrgionsi

What is your version number? Have you upgraded to the latest alpha.70?

loverustfs avatar Nov 26 '25 09:11 loverustfs

We ran the commands below, and everything worked as expected. We don't believe this is an S3 compatibility issue.

Could you please provide more logs from the RustFS side?

Also, could there be an error in this configuration?

#config AK & SK
root@playrustfs:/home/playrustfs# AWS_ACCESS_KEY_ID=rustfsadmin
root@playrustfs:/home/playrustfs# AWS_SECRET_ACCESS_KEY=rustfsadmin

# list bucket
root@playrustfs:/home/playrustfs# aws s3 ls s3://test/  \
--endpoint-url https://play.rustfs.com \
--region us-east-1

# result
2025-11-26 07:09:40      83298 1.jpg
2025-11-26 07:03:30      18691 68d4c4f0507117dede9333ecf300b767 (1).jpg
2025-11-26 04:06:53    2423296 Gif123.exe
2025-11-26 08:27:55         33 don't download.txt
2025-11-26 10:27:40         20 hello.txt
2025-11-26 07:07:00      21355 个人简历.docx

apiVersion: v1
kind: Secret
metadata:
  name: rustfs-secret
  namespace: longhorn-system
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "your_access_key"
  AWS_SECRET_ACCESS_KEY: "your_secret_key"
  AWS_ENDPOINTS: "http://<RustFS-IP>:<PORT>"  # Important: Do not include the bucket name. The protocol (http/https) is required.

loverustfs avatar Nov 26 '25 11:11 loverustfs

Many thanks for replying.

Yes, I used below tag:

    image: rustfs/rustfs:latest
    image: rustfs/rustfs:1.0.0-alpha.70

I created the same secret, using RUSTFS API FQND.

Actually longhorn starts the backup and transferring files, I have them in the bucket but after some times longhorn starts complaining UnauthorizedAccess Your account is not signed up2

Image

mrgionsi avatar Nov 26 '25 11:11 mrgionsi

No problem. this is a really strange problem. We will find the cause and solve this problem.

loverustfs avatar Nov 26 '25 11:11 loverustfs

Thanks a lot. I tried to enable DEBUG but no logs showed in container. If it can help, please guide me how to enable logs and I will share it.

mrgionsi avatar Nov 26 '25 11:11 mrgionsi

Thanks a lot. I tried to enable DEBUG but no logs showed in container. If it can help, please guide me how to enable logs and I will share it.

-e RUSTFS_LOG_LEVEL=info can modify log level.

docker run -d \
  --name rustfs_container \
  -p 9000:9000 \
  -p 9001:9001 \
  -v /mnt/rustfs/data:/data \
  -e RUSTFS_ACCESS_KEY=rustfsadmin \
  -e RUSTFS_SECRET_KEY=rustfsadmin \
  -e RUSTFS_CONSOLE_ENABLE=true \
  -e RUSTFS_SERVER_DOMAINS=example.com \
  -e RUSTFS_LOG_LEVEL=info
  rustfs/rustfs:latest \
    

loverustfs avatar Nov 26 '25 11:11 loverustfs

Thanks a lot. I tried to enable DEBUG but no logs showed in container. If it can help, please guide me how to enable logs and I will share it.

-e RUSTFS_LOG_LEVEL=info can modify log level.

docker run -d \
  --name rustfs_container \
  -p 9000:9000 \
  -p 9001:9001 \
  -v /mnt/rustfs/data:/data \
  -e RUSTFS_ACCESS_KEY=rustfsadmin \
  -e RUSTFS_SECRET_KEY=rustfsadmin \
  -e RUSTFS_CONSOLE_ENABLE=true \
  -e RUSTFS_SERVER_DOMAINS=example.com \
  -e RUSTFS_LOG_LEVEL=info
  rustfs/rustfs:latest \
    

Done: - RUSTFS_LOG_LEVEL=debug

But on logs nothing change:

Initializing data directories: /data
Initializing log directory: /logs
Starting: /usr/bin/rustfs  /data
   API: http://172.28.0.2:9000  http://127.0.0.1:9000
Console WebUI available at: http://172.28.0.2:9001/rustfs/console/index.html
Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html

mrgionsi avatar Nov 26 '25 12:11 mrgionsi

Thanks a lot. I tried to enable DEBUG but no logs showed in container. If it can help, please guide me how to enable logs and I will share it.

-e RUSTFS_LOG_LEVEL=info can modify log level.

docker run -d \
  --name rustfs_container \
  -p 9000:9000 \
  -p 9001:9001 \
  -v /mnt/rustfs/data:/data \
  -e RUSTFS_ACCESS_KEY=rustfsadmin \
  -e RUSTFS_SECRET_KEY=rustfsadmin \
  -e RUSTFS_CONSOLE_ENABLE=true \
  -e RUSTFS_SERVER_DOMAINS=example.com \
  -e RUSTFS_LOG_LEVEL=info
  rustfs/rustfs:latest \
    

Done: - RUSTFS_LOG_LEVEL=debug

But on logs nothing change:

Initializing data directories: /data
Initializing log directory: /logs
Starting: /usr/bin/rustfs  /data
   API: http://172.28.0.2:9000  http://127.0.0.1:9000
Console WebUI available at: http://172.28.0.2:9001/rustfs/console/index.html
Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html

@houseme , Has this parameter changed?

loverustfs avatar Nov 26 '25 13:11 loverustfs

RUST_LOG="rustfs=info,ecstore=info,s3s=debug,iam=info" ?

loverustfs avatar Nov 26 '25 13:11 loverustfs

Thank you for your response. You mentioned that there is a file named rustfs_r2025-11-26_xx-xx-xx.log in the logs directory, which is very helpful. Please share the relevant log content with us so that we can further analyze the issue. Additionally, please confirm if there are any other log files or configuration information that might help us diagnose the problem. Looking forward to your reply!

houseme avatar Nov 26 '25 13:11 houseme

Hello,

I changed parameters but nothing changed. Also no file in logs folder:

services:
  rustfs:
    image: rustfs/rustfs:1.0.0-alpha.70
    container_name: rustfs-server
    environment:
      # === Core RustFS configuration ===
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      #- RUSTFS_SERVER_DOMAINS=
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_ACCESS_KEY=
      - RUSTFS_SECRET_KEY=.
      - RUSTFS_LOG_LEVEL=debug
      - RUSTFS_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"


      # === Telemetry configuration ===
      - RUSTFS_OBS_EXPORTER=prometheus
      - RUSTFS_OBS_PROMETHEUS_ADDRESS=0.0.0.0:9464  # Prometheus scrape endpoint
      - RUSTFS_OBS_TRACING_ENABLED=false             # Disable Jaeger/Tempo tracing
      # =================================

    ports:
.......

mrgionsi avatar Nov 26 '25 13:11 mrgionsi

Thank you for your feedback. According to your description, after setting the RUSTFS_OBS_LOG_DIRECTORY parameter, logs will be output to the specified log directory and files similar to rustfs_r2025-11-26_xx-xx-xx.log will be created. Please try setting this parameter and check if the corresponding log files are generated in the log directory. If the files are generated, please share the relevant log content with us for further analysis of the issue. We look forward to your reply!

services:
  rustfs:
    image: rustfs/rustfs:1.0.0-alpha.70
    container_name: rustfs-server
    environment:
      # === Core RustFS configuration ===
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      #- RUSTFS_SERVER_DOMAINS=
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_ACCESS_KEY=
      - RUSTFS_SECRET_KEY=.
      - RUSTFS_LOG_LEVEL=debug
      - RUSTFS_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUSTFS_OBS_LOG_DIRECTORY="/logs"


      # === Telemetry configuration ===
      - RUSTFS_OBS_EXPORTER=prometheus
      - RUSTFS_OBS_PROMETHEUS_ADDRESS=0.0.0.0:9464  # Prometheus scrape endpoint
      - RUSTFS_OBS_TRACING_ENABLED=false             # Disable Jaeger/Tempo tracing
      # =================================

    ports:
.......


The three parameters 'RUSTFS_OBS_EXPORTER','RUSTFS_OBS_PROMETHEUS_ADDRESS' and 'RUSTFS_OBS_TRACING_ENABLED' do not exist in the project.

houseme avatar Nov 26 '25 15:11 houseme

Thank you for your feedback. According to your description, after setting the RUSTFS_OBS_LOG_DIRECTORY parameter, logs will be output to the specified log directory and files similar to rustfs_r2025-11-26_xx-xx-xx.log will be created. Please try setting this parameter and check if the corresponding log files are generated in the log directory. If the files are generated, please share the relevant log content with us for further analysis of the issue. We look forward to your reply!

services: rustfs: image: rustfs/rustfs:1.0.0-alpha.70 container_name: rustfs-server environment: # === Core RustFS configuration === - RUSTFS_VOLUMES=/data - RUSTFS_ADDRESS=0.0.0.0:9000 #- RUSTFS_SERVER_DOMAINS= - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 - RUSTFS_CONSOLE_ENABLE=true - RUSTFS_ACCESS_KEY= - RUSTFS_SECRET_KEY=. - RUSTFS_LOG_LEVEL=debug - RUSTFS_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info" - RUSTFS_OBS_LOG_DIRECTORY="/logs"

  # === Telemetry configuration ===
  - RUSTFS_OBS_EXPORTER=prometheus
  - RUSTFS_OBS_PROMETHEUS_ADDRESS=0.0.0.0:9464  # Prometheus scrape endpoint
  - RUSTFS_OBS_TRACING_ENABLED=false             # Disable Jaeger/Tempo tracing
  # =================================

ports:

.......

The three parameters 'RUSTFS_OBS_EXPORTER','RUSTFS_OBS_PROMETHEUS_ADDRESS' and 'RUSTFS_OBS_TRACING_ENABLED' do not exist in the project.

Thanks a lot. I don't see any logs in the folder after settings env variables:

services:
  rustfs:
    image: rustfs/rustfs:1.0.0-alpha.70
    container_name: rustfs-server
    environment:
      # === Core RustFS configuration ===
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      #- RUSTFS_SERVER_DOMAINS=rustfs.gionsihome.duckdns.org
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_ACCESS_KEY=
      - RUSTFS_SECRET_KEY=.
      - RUSTFS_LOG_LEVEL=debug
      - RUSTFS_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUST_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUSTFS_OBS_LOG_DIRECTORY=/app/logs
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # Web console
      - "9464:9464"   # Prometheus metrics
    volumes:
      - /mnt/s3_storage/data:/data
      - /mnt/s3_storage/logs:/app/logs

I set both RUST_LOG and RUSTFS_LOG but nothing changes. Am I missing something?

mrgionsi avatar Nov 26 '25 16:11 mrgionsi

services:
  rustfs:
    image: rustfs/rustfs:1.0.0-alpha.70
    container_name: rustfs-server
    environment:
      # === Core RustFS configuration ===
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      #- RUSTFS_SERVER_DOMAINS=rustfs.gionsihome.duckdns.org
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_OBS_LOGGER_LEVEL=debug
      - RUST_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUSTFS_OBS_LOG_DIRECTORY=/app/logs
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # Web console
    volumes:
      - ./data:/data
      - ./logs:/app/logs

Thank you for your feedback. Please try changing RUSTFS_LOG_LEVEL=debug to RUSTFS_OBS_LOGGER_LEVEL=debug, and then check again if the corresponding log file is generated in the log directory. If the file is generated, please share the relevant log content so we can further analyze the issue. Looking forward to your reply!

Image

houseme avatar Nov 27 '25 03:11 houseme

I tried, but no files are generated, neither in the docker console any log appears.

Could you please share your config? Or if you want we can have a call to go through it.

mrgionsi avatar Nov 27 '25 18:11 mrgionsi

I tried, but no files are generated, neither in the docker console any log appears.

Could you please share your config? Or if you want we can have a call to go through it.

We'll look into the cause and get back to you.

loverustfs avatar Nov 27 '25 20:11 loverustfs

I tried, but no files are generated, neither in the docker console any log appears.

Could you please share your config? Or if you want we can have a call to go through it.

services:
  rustfs:
    image: rustfs/rustfs:1.0.0-alpha.70
    container_name: rustfs-server
    environment:
      # === Core RustFS configuration ===
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      #- RUSTFS_SERVER_DOMAINS=rustfs.gionsihome.duckdns.org
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_OBS_LOGGER_LEVEL=debug
      - RUST_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUSTFS_OBS_LOG_DIRECTORY=/app/logs
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # Web console
    volumes:
      - ./data:/data
      - ./logs:/app/logs

Thank you for providing the complete configuration information. According to your description, you have already tried setting the RUSTFS_LOG_LEVEL and RUSTFS_LOG parameters, but still no log files are being generated. To further troubleshoot the issue, I suggest you try the following steps:

  1. Check log directory permissions: Ensure the Docker container has permission to write to the specified log directory. You can add the appropriate permission settings in the Dockerfile, or use the --volume option when running the container to mount the log directory and set permissions.

  2. Manually execute commands: Manually execute the relevant commands inside the Docker container to see if any error messages are output. For example, you can try manually executing the list system backup command to check for any error messages.

  3. View Docker container logs: Use the docker logs command to view the logs of the Docker container to see if any error messages are output.

If you still encounter issues after trying the above steps, please provide more log information and configuration details so that we can further analyze the problem. Additionally, if you are willing, we can arrange a phone conference to discuss this issue in detail. Looking forward to your reply!

houseme avatar Nov 28 '25 02:11 houseme

rustfs_r2025-11-28_10-11-30.log

Here we go.

In the logs there should be also a initial successful connection and after that, unauthorized

mrgionsi avatar Nov 28 '25 10:11 mrgionsi

rustfs_r2025-11-28_10-11-30.log

Here we go.

In the logs there should be also a initial successful connection and after that, unauthorized

@weisd

houseme avatar Nov 29 '25 08:11 houseme

Hey @mrgionsi ,

We seem to be reproducing the problem, it's possible that AK and SK have disappeared.

How were the ak and sk values ​​you assigned to longhorn generated? How long are AK/SK valid?

loverustfs avatar Dec 03 '25 06:12 loverustfs

Hello @loverustfs,

Here is the access key created, they still exists:

Image

mrgionsi avatar Dec 03 '25 08:12 mrgionsi

There's a hidden folder named .rustfs.sys in the rustfs data directory. Do your ak and sk files still exist in this folder?

For example:

Image

loverustfs avatar Dec 03 '25 08:12 loverustfs

/data/.rustfs.sys/config/iam/sts # ls -larth
total 12K
drwxr-xr-x 3 root   root   4.0K Dec  3 08:16 V8Y9WXOQHE3BJBP0KTSY
drwxr-xr-x 7 rustfs rustfs 4.0K Dec  3 08:16 ..
drwxr-xr-x 3 root   root   4.0K Dec  3 08:16 .
/data/.rustfs.sys/config/iam/sts # ^C

/data/.rustfs.sys/config/iam/sts # ^C

/data/.rustfs.sys/config/iam/sts # ^C

/data/.rustfs.sys/config/iam/sts # cd V8Y9WXOQHE3BJBP0KTSY/identity.json/
/data/.rustfs.sys/config/iam/sts/V8Y9WXOQHE3BJBP0KTSY/identity.json # ls
xl.meta

It seems yes.

mrgionsi avatar Dec 03 '25 08:12 mrgionsi

/data/.rustfs.sys/config/iam/sts # ls -larth total 12K drwxr-xr-x 3 root root 4.0K Dec 3 08:16 V8Y9WXOQHE3BJBP0KTSY drwxr-xr-x 7 rustfs rustfs 4.0K Dec 3 08:16 .. drwxr-xr-x 3 root root 4.0K Dec 3 08:16 . /data/.rustfs.sys/config/iam/sts # ^C

/data/.rustfs.sys/config/iam/sts # ^C

/data/.rustfs.sys/config/iam/sts # ^C

/data/.rustfs.sys/config/iam/sts # cd V8Y9WXOQHE3BJBP0KTSY/identity.json/ /data/.rustfs.sys/config/iam/sts/V8Y9WXOQHE3BJBP0KTSY/identity.json # ls xl.meta It seems yes.

Ok, got it.

loverustfs avatar Dec 03 '25 08:12 loverustfs

@mrgionsi Did you find the below logs in longhorn-engine pod?

time="2025-11-25T17:28:32.108953074Z" level=error 
msg="Failed to get info from backup store" func="controller.(*BackupTargetController).reconcile" 
file="backup_target_controller.go:459" 
controller=longhorn-backup-target cred=rustfs-secret 
error="failed to list system backups in s3://talos-prod@us-east-1/: error listing system backup in 
s3://talos-prod@us-east-1/: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.10.0/longhorn
 [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.10.0/longhorn system-backup list 
s3://talos-prod@us-east-1/], output , stderr time=\"2025-11-25T17:28:32.106725032Z\" level=error 
msg=\"Failed to list s3\" func=\"s3.(*BackupStoreDriver).List\" file=\"s3.go:116\" error=\"failed to 
list objects with param: {\\n  Bucket: \\\"talos-prod\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  UnauthorizedAccess Your account is not signed up2 <nil>\\n401 \\n\" 
pkg=s3\ntime=\"2025-11-25T17:28:32.106987352Z\" level=fatal 
msg=\"Failed to run list system backup command\" func=cmd.SystemBackupCmd.SystemBackupListCmd.func4 file=\"system_backup.go:72\" 
error=\"failed to list objects with 
param: {\\n  Bucket: \\\"talos-prod\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  UnauthorizedAccess Your account is not signed up2 <nil>\\n401 \\n\"\n: exit status 1" interval=5m0s name=default node=talos-prod-worker-1 
url="s3://talos-prod@us-east-1/"

I setup the environment to reproduce the issue.

  • my longhorn env
kubectl -n longhorn-system get pods
NAME                                                READY   STATUS    RESTARTS   AGE
csi-attacher-b5547c89c-75p7b                        1/1     Running   0          126m
csi-attacher-b5547c89c-b92kf                        1/1     Running   0          126m
csi-attacher-b5547c89c-z8pj2                        1/1     Running   0          126m
csi-provisioner-6d9fbbd668-gjvrd                    1/1     Running   0          126m
csi-provisioner-6d9fbbd668-kzbjf                    1/1     Running   0          126m
csi-provisioner-6d9fbbd668-nhnc5                    1/1     Running   0          126m
csi-resizer-588c678f58-2s966                        1/1     Running   0          126m
csi-resizer-588c678f58-62qbh                        1/1     Running   0          126m
csi-resizer-588c678f58-9jgtn                        1/1     Running   0          126m
csi-snapshotter-589f996dc-7dlcd                     1/1     Running   0          126m
csi-snapshotter-589f996dc-7r2wv                     1/1     Running   0          126m
csi-snapshotter-589f996dc-ftxc6                     1/1     Running   0          126m
engine-image-ei-3154f3aa-zs2mj                      1/1     Running   0          126m
instance-manager-e0cc4cd1e3a99d3805bf212ce1f0d4a1   1/1     Running   0          126m
longhorn-csi-plugin-qvlp6                           3/3     Running   0          126m
longhorn-driver-deployer-5bb579d858-8fbtp           1/1     Running   0          3h43m
longhorn-manager-mtxn8                              2/2     Running   0          127m
longhorn-ui-66dc4bd597-9llrs                        1/1     Running   0          3h43m
longhorn-ui-66dc4bd597-pksf2                        1/1     Running   0          3h43m
  • longhorn ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: longhorn-system
  annotations:
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # prevent the controller from redirecting (308) to HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required '
    # custom max body size for file uploading like backing image uploading
    nginx.ingress.kubernetes.io/proxy-body-size: 10000m
spec:
  ingressClassName: nginx
  rules:
  - host: your.rustfs.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80
  • rustfs container
services:
  rustfs:
    image: rustfs/rustfs:latest
    container_name: rustfs
    hostname: rustfs
    environment:
      # Use service names and correct disk indexing (1..4 to match mounted paths)
      - RUSTFS_VOLUMES=/data/rustfs{1...4}
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
        #- RUSTFS_EXTERNAL_ADDRESS={{ ansible_default_ipv4.address }}:9000  # Same as internal since no port mapping
      - RUSTFS_EXTERNAL_ADDRESS=0.0.0.0:9000  # Same as internal since no port mapping
      - RUSTFS_ACCESS_KEY=rustfsadmin
      - RUSTFS_SECRET_KEY=rustfsadmin
      - RUSTFS_CMD=rustfs
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -f http://localhost:9000/health && curl -f http://localhost:9001/health"
        ]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
    ports:
      - "9000:9000"  # API endpoint
      - "9001:9001"  # Console
    volumes:
      - rustfs-data1:/data/rustfs1
      - rustfs-data2:/data/rustfs2
      - rustfs-data3:/data/rustfs3
      - rustfs-data4:/data/rustfs4

volumes:
  rustfs-data1:
  rustfs-data2:
  rustfs-data3:
  rustfs-data4:
  • rustfs ak & sk
Image
  • longhorn backup target
Image
  • triggered the volume backup
Image
  • data in rustfs longhorn bucket
Image

I do not find the UnauthorizedAccess Your account is not signed up2 log in engine-image-ei-3154f3aa-zs2mj as well as rustfs container.

majinghe avatar Dec 03 '25 13:12 majinghe

Did you find the below logs in longhorn-engine pod?

Yes I am.

I am exposing Rustfs via traefik:

services:
  rustfs:
    user: "0:0"
    image: rustfs/rustfs:1.0.0-alpha.70
    container_name: rustfs-server
    environment:
      # === Core RustFS configuration ===
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      #- RUSTFS_SERVER_DOMAINS=domain
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_ACCESS_KEY=<username>
      - RUSTFS_SECRET_KEY=accesskey
      - RUSTFS_OBS_LOGGER_LEVEL=debug
      - RUST_LOG=debug
      - RUSTFS_OBS_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUST_LOG="rustfs=debug,ecstore=info,s3s=debug,iam=info"
      - RUSTFS_OBS_LOG_DIRECTORY=/app/logs
      - LOG_LEVEL=DEBUG
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # Web console
      - "9464:9464"   # Prometheus metrics
    volumes:
      - /path/data:/data
      - /path/logs:/app/logs
    networks:
      - rustfs-network
      - traefik_default
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_default"
      # S3 API endpoint via Traefik
      - "traefik.http.routers.rustfs-gui.rule=Host(`fqdn`)"
      - "traefik.http.routers.rustfs-gui.entrypoints=websecure"
      - "traefik.http.routers.rustfs-gui.tls=true"
      - "traefik.http.routers.rustfs-gui.service=rustfs-gui"
      - "traefik.http.routers.rustfs-gui.middlewares=local-ipwhitelist@file"
      - "traefik.http.services.rustfs-gui.loadbalancer.server.scheme=http"
      - "traefik.http.services.rustfs-gui.loadbalancer.server.port=9001"
      - "traefik.http.routers.rustfs-gui.tls.certresolver=letsencrypt"

      # Web console via Traefik
      - "traefik.http.routers.rustfs-api.rule=Host(`another-fqdn)"
      - "traefik.http.routers.rustfs-api.entrypoints=websecure"
      - "traefik.http.routers.rustfs-api.tls=true"
      - "traefik.http.services.rustfs-api.loadbalancer.server.port=9000"
      - "traefik.http.routers.rustfs-api.service=rustfs-api"
      - "traefik.http.routers.rustfs-api.middlewares=local-ipwhitelist@file"      
      - "traefik.http.services.rustfs-api.loadbalancer.server.scheme=http"
      - "traefik.http.routers.rustfs-api.tls.certresolver=letsencrypt"


I tried to connect longhorn with rustfs using http://: and it looks working now.

Edit: I confirm it works 😄 I would prefer to use it via Traefik with https and a valid certificate. Is that something am I missing?

mrgionsi avatar Dec 05 '25 19:12 mrgionsi

Hey @mrgionsi ,

This bug has been fixed in the latest release. I'm closing this issue for now.

https://github.com/rustfs/rustfs/releases/tag/1.0.0-alpha.73

If the problem persists, please leave a comment and we'll reopen it.

loverustfs avatar Dec 11 '25 03:12 loverustfs