rustfs icon indicating copy to clipboard operation
rustfs copied to clipboard
Published 9 hours ago • verified publisher icon rustfs

MNMD doesn't work with TLS enabled - nodes can't connect to each other

Open sample opened this issue 1 week ago • 19 comments

Describe the bug When running a rustfs cluster with TLS enabled (using https:// URLs in RUSTFS_VOLUMES), nodes cannot connect to each other. The cluster fails to form. Without TLS (using http:// URLs) and no RUSTFS_TLS_PATH, everything works correctly.

To Reproduce Set up a 4-node rustfs cluster in Docker Compose Generate TLS certificates and place them in a shared directory Configure environment variables: RUSTFS_VOLUMES: "https://node{1...4}:9000/data/" RUSTFS_TLS_PATH: "/opt/tls" Start the cluster with docker compose up See TLS handshake errors in logs, cluster doesn't form

Expected behavior Nodes should connect to each other over HTTPS and form a working cluster, similar to how it works with HTTP URLs.

Screenshots Logs show these errors: WARN TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead DEBUG TLS handshake failure details: peer_addr="172.30.0.13:47112", error_details="tls handshake eof" WARN get_format_erasure_in_quorum fi: [Some(FormatV3 {...}), None, None, None]

Desktop (please complete the following information):

  • OS: MacOS (Docker Decktop with containers based on rustfs image)
  • Browser safari
  • rustfs Version: 1.0.0-alpha.73

Additional context TLS directory structure: /opt/tls/nodeX/rustfs_cert.pem and /opt/tls/nodeX/rustfs_key.pem

The HTTPS server itself starts correctly (can connect from browser) Switching to http:// URLs and disable RUSTFS_TLS_PATH makes the cluster work immediately The error "tls handshake eof" suggests the client might be trying plain HTTP to an HTTPS endpoint

Is TLS for inter-node communication supported? Am I missing some configuration?

sample avatar Dec 19 '25 11:12 sample

Hello! Thank you for providing a detailed description of the issue and the log information. Based on your description, when TLS is enabled, the nodes in the rustfs cluster cannot connect to each other, whereas they work normally when using HTTP. This is indeed a bug worth paying attention to.

Regarding your question, your certificate should be a domain-specific certificate for each node. Specifically, each node (node{1..4}) should have a corresponding domain-specific certificate, such as node1.example.com, node2.example.com, etc. This ensures that each node can correctly verify the identity of the other during TLS handshake.

Additionally, based on your log information, the following issues exist:

  • TLS handshake failed, indicating that the client should connect to the HTTP port.
  • The specific error of TLS handshake failure is EOF (end of file), which may mean that the client is trying to use HTTP to connect to an HTTPS port.

To further troubleshoot the issue, we recommend:

  1. Confirm that the certificate for each node is properly configured and that the domain name is correct.
  2. Check the rustfs configuration file for TLS settings, ensuring all relevant configuration items are correct.
  3. Try using different TLS versions or certificate libraries to test and see if the issue can be reproduced.

If you have any further questions or need help during the troubleshooting process, please feel free to let me know. We will do our best to assist you in resolving the issue. Thank you for your patience and cooperation!

houseme avatar Dec 19 '25 13:12 houseme

We implement a strict HTTPS protocol.

The domain name must match the certificate.

HTTPS + IP address is an invalid protocol.

loverustfs avatar Dec 19 '25 13:12 loverustfs

We currently have no priority support plans for HTTPS + IP.

loverustfs avatar Dec 19 '25 13:12 loverustfs

Thank you for the guidance! I made the following changes:

  1. Change URLs from IPs to hostnames: RUSTFS_VOLUMES: "https://node{1...4}:9000/data/"
  2. Place CA certificate to the container so it can trust the self-signed certificates, Checked dns resolving and https connectivity in container
docker compose exec -it  node1 curl -v https://node2:9000
* Host node2:9000 was resolved.
* IPv6: (none)
* IPv4: 172.30.0.12
*   Trying 172.30.0.12:9000...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=node2
*  start date: Dec 19 09:35:27 2025 GMT
*  expire date: Mar 23 09:35:27 2028 GMT
*  subjectAltName: host "node2" matched cert's "node2"
*  issuer: C=DE; O=RustFS; CN=rustfs-internal-ca
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to node2 (172.30.0.12) port 9000
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://node2:9000/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: node2:9000]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.14.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: node2:9000
> User-Agent: curl/8.14.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Request completely sent off
< HTTP/2 500
< content-type: application/xml
< vary: origin, access-control-request-method, access-control-request-headers
< access-control-allow-origin: *
< access-control-expose-headers: *
< x-request-id: 67b71411-8227-4bb0-b8e3-6970466143f2
< date: Fri, 19 Dec 2025 14:03:25 GMT
< content-length: 106
<
* Connection #0 to host node2 left intact
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>Not init</Message></Error>
  1. Checked my certificates already have the correct SANs for each node:
docker compose exec -it  node1 cat  /opt/tls/rustfs_cert.pem | openssl x509 -noout  -subject -ext subjectAltName -issuer
subject=CN=node1
X509v3 Subject Alternative Name:
    DNS:node1, IP Address:172.30.0.11
issuer=C=DE, O=RustFS, CN=rustfs-internal-ca
docker compose exec -it  node2 cat  /opt/tls/rustfs_cert.pem | openssl x509 -noout  -subject -ext subjectAltName -issuer          
subject=CN=node2
X509v3 Subject Alternative Name:
    DNS:node2, IP Address:172.30.0.12
issuer=C=DE, O=RustFS, CN=rustfs-internal-ca
docker compose exec -it  node3 cat  /opt/tls/rustfs_cert.pem | openssl x509 -noout  -subject -ext subjectAltName -issuer
subject=CN=node3
X509v3 Subject Alternative Name:
    DNS:node3, IP Address:172.30.0.13
issuer=C=DE, O=RustFS, CN=rustfs-internal-ca
docker compose exec -it  node4 cat  /opt/tls/rustfs_cert.pem | openssl x509 -noout  -subject -ext subjectAltName -issuer
subject=CN=node4
X509v3 Subject Alternative Name:
    DNS:node4, IP Address:172.30.0.14
issuer=C=DE, O=RustFS, CN=rustfs-internal-ca

However, I'm still getting the same error (example log from node1):

node1  | {"timestamp":"2025-12-19T14:07:49.017148466Z","level":"DEBUG","fields":{"message":"opt: Opt { volumes: [\"https://node{1...4}:9000/data\"], address: \"0.0.0.0:9000\", server_domains: [], access_key: \"rustfsadmin\", secret_key: \"rustfsadmin\", console_enable: true, console_address: \"0.0.0.0:9001\", obs_endpoint: \"\", tls_path: Some(\"/opt/tls\"), license: None, region: None, kms_enable: false, kms_backend: \"local\", kms_key_dir: None, kms_vault_address: None, kms_vault_token: None, kms_default_key_id: None, buffer_profile_disable: false, buffer_profile: \"GeneralPurpose\" }"},"target":"rustfs","filename":"rustfs/src/main.rs","line_number":141,"span":{"name":"run"},"spans":[{"name":"run"}],"threadName":"main","threadId":"ThreadId(1)"}
...
node1  | RustFS API: https://172.30.0.11:9000  https://127.0.0.1:9000
node1  | RustFS Start Time: 2025-12-19 14:07:49
...

node1  | {"timestamp":"2025-12-19T14:07:49.040400925Z","level":"DEBUG","fields":{"message":"TLS handshake start"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":589,"threadName":"rustfs-worker","threadId":"ThreadId(25)"}
node1  | {"timestamp":"2025-12-19T14:07:49.040897258Z","level":"WARN","fields":{"message":"TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead","peer_addr":"172.30.0.12:56194"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":608,"threadName":"rustfs-worker","threadId":"ThreadId(25)"}
node1  | {"timestamp":"2025-12-19T14:07:49.041051258Z","level":"DEBUG","fields":{"message":"TLS handshake failure details","peer_addr":"172.30.0.12:56194","error_type":"std::io::error::Error","error_details":"tls handshake eof"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":630,"threadName":"rustfs-worker","threadId":"ThreadId(25)"}```

sample avatar Dec 19 '25 14:12 sample

https://node2:9000 shouldn't match the TLS certificate, right?

If it's a certificate issued by *.rustfs.com,

your hosts file should be set to node2.rustfs.com.

loverustfs avatar Dec 19 '25 14:12 loverustfs

Just to clarify - I'm using hostnames (node1, node2, etc.) rather than FQDNs. The curl test shows that TLS verification works correctly with these names:

* subjectAltName: host "node2" matched cert's "node2"
* SSL certificate verify ok.

But maybe rustfs handles this differently internally? Should I try using FQDNs like node1.test, node2.test instead? I can regenerate the certificates with those names if that's required.

sample avatar Dec 19 '25 14:12 sample

Ok, looking forward to your good news.

loverustfs avatar Dec 19 '25 14:12 loverustfs

I tried using FQDNs (node1.test, node2.test, etc.) but unfortunately it didn't help. The same error persists. Configuration: RUSTFS_VOLUMES: "https://node{1...4}.test:9000/data"

DNS resolution works correctly:

{"timestamp":"2025-12-19T14:44:35.307579084Z","level":"INFO","fields":{"message":"Cache miss for domain node1.test, querying system resolver."},"target":"rustfs_utils::net","filename":"crates/utils/src/net.rs","line_number":189,"span":{"name":"run"},"spans":[{"name":"run"}],"threadName":"main","threadId":"ThreadId(1)"}
{"timestamp":"2025-12-19T14:44:35.307754709Z","level":"INFO","fields":{"message":"System query for domain node1.test: {172.30.0.11}"},"target":"rustfs_utils::net","filename":"crates/utils/src/net.rs","line_number":204,"span":{"name":"run"},"spans":[{"name":"run"}],"threadName":"main","threadId":"ThreadId(1)"}
{"timestamp":"2025-12-19T14:44:35.30776925Z","level":"INFO","fields":{"message":"Create pool endpoints host 'node1.test' resolved to ips {172.30.0.11} for endpoint 'https://node1.test:9000/data'","host":"node1.test","endpoint":"https://node1.test:9000/data","from":"get_host_ip"},"target":"rustfs::ecstore::endpoints","filename":"crates/ecstore/src/endpoints.rs","line_number":260,"span":{"name":"run"},"spans":[{"name":"run"}],"threadName":"main","threadId":"ThreadId(1)"}

Certificates have correct SANs:

docker compose exec -it  node1 cat  /opt/tls/rustfs_cert.pem | openssl x509 -noout  -subject -ext subjectAltName -issuer
subject=CN=node1.test
X509v3 Subject Alternative Name:
    DNS:node1.test, DNS:node1, IP Address:172.30.0.11
issuer=C=DE, O=RustFS, CN=rustfs-internal-ca

But TLS handshake still fails:

TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead
TLS handshake failure details: peer_addr="172.30.0.13:38488", error_details="tls handshake eof"

The curl test from inside the container still works perfectly: docker compose exec node1 curl -v https://node2.test:9000 SSL certificate verify ok, connection successful

So external TLS connections work, but internal node-to-node communication fails with the same "tls handshake eof" error.

sample avatar Dec 19 '25 14:12 sample

Okay, we'll test it again. If it's confirmed, we'll fix the bug.

loverustfs avatar Dec 19 '25 15:12 loverustfs

Hello @sample ,

Steps

We have conducted tests based on your report and observed the following results:

  • The service starts up normally when configured with https://node{1...4}.xxx.com:9000/data/rustfs{0...3}.

  • However, requests must be sent using the corresponding domain name (e.g., https://node*.xxx.com:9001).

  • Accessing via the original direct IP address (e.g., http://12.12.13.14) is no longer possible in this configuration.

Question

  1. Could you please confirm if these results match what you are experiencing?
  2. Additionally, could you clarify your expected behavior?
  3. Are you expecting to support both domain and IP access simultaneously?

Certificate configuration screenshot

Image Image

loverustfs avatar Dec 20 '25 15:12 loverustfs

I think there might be a misunderstanding.

My issue is not about client access to the cluster. External client access via HTTPS works. The issue is about inter-node (node-to-node) communication within the cluster. The cluster fails to form when TLS is enabled.

When I configure: RUSTFS_VOLUMES: "https://node{1...4}.test:9000/data"

I have only WARN TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead error_details: "tls handshake eof"

And the cluster cannot form: get_format_erasure_in_quorum fi: [None, None, None, None]

full log on start one of the node:

docker compose logs -f node1
node1  | Initializing data directories:
node1  | OBS log directory not configured and logs outputs to stdout
node1  | !!!WARNING: Using default RUSTFS_ACCESS_KEY or RUSTFS_SECRET_KEY. Override them in production!
node1  | Starting: /usr/bin/rustfs
node1  | {"timestamp":"2025-12-20T18:14:56.257712926Z","level":"WARN","fields":{"message":"WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable."},"target":"rustfs::main::run","filename":"rustfs/src/main.rs","line_number":182,"threadName":"main","threadId":"ThreadId(1)"}
node1  | RustFS API: https://172.30.0.11:9000  https://127.0.0.1:9000
node1  | RustFS Start Time: 2025-12-20 18:14:56
node1  | {"timestamp":"2025-12-20T18:14:56.264039926Z","level":"WARN","fields":{"message":"Detected default credentials 'rustfsadmin:rustfsadmin', we recommend that you change these values with 'RUSTFS_ACCESS_KEY' and 'RUSTFS_SECRET_KEY' environment variables"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":185,"threadName":"main","threadId":"ThreadId(1)"}
node1  | Console WebUI Start Time: 2025-12-20 18:14:56
node1  | Console WebUI available at: https://172.30.0.11:9001/rustfs/console/index.html
node1  | Console WebUI (localhost): https://127.0.0.1:9001/rustfs/console/index.html
node1  | {"timestamp":"2025-12-20T18:14:56.28313426Z","level":"WARN","fields":{"message":"TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead","peer_addr":"172.30.0.13:52806"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":581,"threadName":"rustfs-worker","threadId":"ThreadId(22)"}
node1  | {"timestamp":"2025-12-20T18:14:56.283245385Z","level":"WARN","fields":{"message":"TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead","peer_addr":"172.30.0.14:49860"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":581,"threadName":"rustfs-worker","threadId":"ThreadId(24)"}
node1  | {"timestamp":"2025-12-20T18:14:56.283131718Z","level":"WARN","fields":{"message":"get_format_erasure_in_quorum fi: [None, None, None, None]"},"target":"rustfs_ecstore::store_init","filename":"crates/ecstore/src/store_init.rs","line_number":174,"threadName":"main","threadId":"ThreadId(1)"}
node1  | {"timestamp":"2025-12-20T18:14:56.285009385Z","level":"WARN","fields":{"message":"TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead","peer_addr":"172.30.0.12:42046"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":581,"threadName":"rustfs-worker","threadId":"ThreadId(22)"}
node1  | {"timestamp":"2025-12-20T18:14:58.289207344Z","level":"WARN","fields":{"message":"TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead","peer_addr":"172.30.0.12:43570"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":581,"threadName":"rustfs-worker","threadId":"ThreadId(22)"}
node1  | {"timestamp":"2025-12-20T18:14:58.289707386Z","level":"WARN","fields":{"message":"TLS handshake failed. If this client needs HTTP, it should connect to the HTTP port instead","peer_addr":"172.30.0.14:43738"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":581,"threadName":"rustfs-worker","threadId":"ThreadId(22)"}
node1  | {"timestamp":"2025-12-20T18:14:58.290960677Z","level":"WARN","fields":{"message":"get_format_erasure_in_quorum fi: [None, None, None, None]"},"target":"rustfs_ecstore::store_init","filename":"crates/ecstore/src/store_init.rs","line_number":174,"threadName":"main","threadId":"ThreadId(1)"}
...

and cluster doesn't work, response from port 9000 <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>Not init</Message></Error>

when i changed https:// scheme to http:// in RUSTFS_VOLUMES and remove RUSTFS_TLS_PATH env, the cluster starts working.

sample avatar Dec 20 '25 18:12 sample

Okay. We'll fix this bug.

loverustfs avatar Dec 21 '25 01:12 loverustfs

Hey @sample ,

This bug has been fixed, please wait for next version release.

loverustfs avatar Dec 21 '25 11:12 loverustfs