Latency increases on enabling authentication for high number of connections

[varungup90]

We are using Cassandra version 3.0.14 and use gocql driver. Number of open connections are 15k (which I understand is anti-pattern).

Read = 8k qps Write = 3k qps

Latencies for read/write are below 100ms.

After we enabled the password based authentication, latency spiked in seconds. After I reduced number of connections below 4k, latencies dropped in ms. Here, I am trying to understand the reason, why did latency spike up for same number of connections when auth is enabled.

Cassandra config settings. Role and Permission cache period = 5 seconds. NTP request = 4096

From my understanding, connections are authenticated once, and then re-used for following queries, from that theory latency should spike initially then reset back to original level.

[Zariel]

Not sure what would cause that, I would suggest profiling both cases on the Go side and then if they look comparable allocation / cpu usage time then check Cassandra. I would be interested to see the CPU profiles of both cases.

The code to do the authentication involves more RTT and more calls but I wouldn't expect it to cause such a huge drop, as you say the authentication happens only on connection side on the driver then once the connection session is established it does no extra work on the driver.

One thought is that it could cause a pathological case in Cassandra somewhere and cause a slowdown.