[BUG]: Upgrade to version CSI 2.10.0 , we had an TLS illegal parameter error

[adarsh-dell]

Bug Description

User currently using csi2.9.1( powerflex - vxflexos )  and when upgrade the version to csi2.10.0 , we had an tls illegal parameter error and the same has been attached here

Host OS : Ubuntu 20.04.6 LTS  

Kernel : 5.4.0-172-generic,

User shared Additional info if this helps:

how to update cipher in vxflexos config ?

Below is the CIPHERS Configured on PowerFlex Gateway Server.

The ciphers used by the GW's embedded Tomcat webserver are listed in /opt/emc/scaleio/gateway/conf/server.xml line 116 (the ciphers attribute of the SSL Connector definition)

ciphers="xxxxxxxxxxx,yyyyyyyyyyyyyy,xxxxxxxxxxxxxxxxxxxxxxxxxx,zzzzzzzzzzzzzzzzzzzzzz"

Logs

NA

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

Update the cipher suites on the csi-powerflex cluster and upgrade the driver.

Expected Behavior

Driver should be in running state.

CSM Driver(s)

CSI-powerflex v2.10.1

Installation Type

No response

Container Storage Modules Enabled

No response

Container Orchestrator

k8

Operating System

Ubuntu 20.04.6

[csmbot]

@adarsh-dell: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at [email protected].

[gallacher]

Unrelated to CSM so closing this issue.

[gallacher]

@adarsh-dell - I'm re-opening this issue to get a better understanding if this failure is dependent on what is configured for ciphers in the GW configuration. This seems to be happening on the network connection between the driver and the GW server. Please provide additional information on if this error only happens in cases where the GW server is configured a certain way. Thank you.

[adarsh-dell]

Investigation and Case Study on TLS Handshake Failure

Overview

Below is the observations and case study conducted during the investigation of a TLS handshake failure between an HTTP client (driver) and the PFX gateway server. The aim is to understand the root cause of the issue and explore potential solutions.

TLS Configuration in HTTP Client

The HTTP client in this scenario can set the TLS configuration during client creation, either by specifying a set of cipher suites or by leaving this field unset to use the default cipher suites. The crypto/tls package in Go provides the CipherSuites() function, which returns only secure cipher suites.

Example: Setting TLS Configuration in HTTP Client

package main

import (
    "crypto/tls"
    "net/http"
)

func main() {
    tlsConfig := &tls.Config{
        CipherSuites: []uint16{
            tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
            tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
        },
    }

    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: tlsConfig,
        },
    }

    resp, err := client.Get("https://example.com")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
}

TLS Configuration on Server Side

The PFX gateway server allows users to configure cipher suites by editing the server.xml file. Although the server comes with predefined secure suites, users can manually update this configuration to meet specific security requirements.

Cipher Suite Concept

For a TLS handshake to succeed, there must be at least one mutually supported cipher suite between the client and server. If such a cipher suite exists, the handshake should proceed successfully, and the mutually supported suite will be used for further secure communication.

Observations and Findings

• Despite having a matching cipher suite configured on both the client and server sides, the TLS handshake was failing.
• The same TLS configuration and cipher suite worked successfully with a dummy Go server, indicating that the issue was not with the client or driver side.
• Using the curl utility to explicitly specify the cipher suites also resulted in handshake failures, further suggesting an issue on the server side.

Conclusion

The investigation revealed that the problem likely lies with the PFX gateway server. Even though the client and server had mutually supported cipher suites, the handshake was failing due to server-side issues. And the issue is seen with few suites only and is working fine with other suites. We have tested the upcoming i.e. CSI v2.11.0 driver with 3 versions of array after setting only secured suite and it was working fine and filed a ticket against PFX with our findings.

Recommendations

1. Server Configuration Review: Ensure the PFX gateway server configuration is correctly set and there are no discrepancies in the server.xml file.
2. Testing with Different Servers: Test the client with different servers to confirm the server-side issue.
3. Debugging Server Logs: Review server logs to identify any errors or misconfigurations that could cause the handshake failure.
4. Server Update: Consider updating the server software to the latest version to fix any known issues related to TLS handshakes.

Additional Resources

• Go TLS CipherSuites Documentation
• TLS Handshake Process

References

• Go HTTP Client Configuration
• OpenSSL Cipher Suites

Thanks, Adarsh

@gallacher I Hope this much of information will be fine.