management-api-for-apache-cassandra
management-api-for-apache-cassandra copied to clipboard
k8ssandra
K8SSAND-1486 ⁃ Getting error `Cannot create Driver CQLSession as the driver socket has not been created` even though Cassandra has started
Hi, thanks for this API, I am trying to test it , I have a kind cluster with a single node Cassandra , installed using the Bitnami Helm chart.
I have your API configured as a sidecar.
The container for the API is running but it's printing this error in its log:
Cannot create Driver CQLSession as the driver socket has not been created. This should resolve once Cassandra has started and created the socket at /tmp/cassandra.sock
Cassandra is up and running, this is a nodetool status:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN xx.xx.xx.xx 556.09 KiB 256 100.0% 1937b32f-ea42-4d8d-82b1-3f76477c76dc rack1
I have searched I couldn't find anything useful.
Any suggestion of what I may be doing wrong? Thanks
┆Issue is synchronized with this Jira Task by Unito ┆friendlyId: K8SSAND-1486 ┆priority: Medium
We are facing similar issue. with k8ssandra cluster. Following are the configs. cassandra: enabled: true resources: requests: cpu: 2 memory: 2Gi limits: cpu: 6 memory: 6Gi tolerations: [ ] cassandraLibDirVolume: storageClass: gp3 size: 5Gi heap: size: 2Gi datacenters: - name: us-east-1 size: 1 racks: - name: us-east-1a affinityLabels: topology.kubernetes.io/zone: us-east-1a
Error Log: INFO [nioEventLoopGroup-3-14] 2022-11-04 14:46:07,478 UnixSocketCQLAccess.java:88 - Cannot create Driver CQLSession as the driver socket has not been created. This should resolve once Cassandra has started and created the socket at /tmp/cassandra.sock │
INFO [nioEventLoopGroup-2-1] 2022-11-04 14:46:07,478 Cli.java:617 - address=/ url=/api/v0/probes/readiness status=500 Internal Server Error
I too facing same issue when cluster gets restarted or we do scale down and scaleup for any upgrades/downtime Is there any workaround to fix this ?
What version are you on?
On Fri, Dec 2, 2022 at 4:20 PM rakhya @.***> wrote:
We have the same problem as well.
— Reply to this email directly, view it on GitHub https://github.com/k8ssandra/management-api-for-apache-cassandra/issues/189#issuecomment-1335854732, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJBOOQYT5HWBLZZV6MZ3TWLJRZJANCNFSM5U2PWOQQ . You are receiving this because you are subscribed to this thread.Message ID: <k8ssandra/management-api-for-apache-cassandra/issues/189/1335854732@ github.com>
--
- John
We are using k8ssandra/cass-operator version v1.13.1 which is latest And cassandra version 4.0.1 Management Api Image : k8ssandra/cass-management-api:4.0.1
We are still facing the same issue when we increase the resource or make any changes to the cluster or when there is node flip.We are using below operator versions.
- Cass-operator - 1.10.4
- K8ssandra operator - 1.0.1
- K8ssandra - 1.5.1
On Fri, Dec 2, 2022 at 6:09 PM sbbagal13 @.***> wrote:
We are using k8ssandra/cass-operator version v1.13.1 which is latest And cassandra version 4.0.1 Management Api Image : k8ssandra/cass-management-api:4.0.1
— Reply to this email directly, view it on GitHub https://github.com/k8ssandra/management-api-for-apache-cassandra/issues/189#issuecomment-1335947904, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVVNIUUOPKXTDVSR4RKGLQ3WLJ6TNANCNFSM5U2PWOQQ . You are receiving this because you commented.Message ID: <k8ssandra/management-api-for-apache-cassandra/issues/189/1335947904@ github.com>
Note that the message itself is not an error or indicative of any error. It's just an informal message telling that the Cassandra hasn't started yet, but the error why it hasn't started is not this.
Can someone suggest version which does not have this issue please? We are facing the issue as well however we are in early days of development so just need the cluster up.
Thank you
As @burmanm said, this message indicates that the Cassandra process hasn't completed startup. However, it is not indicative of any specific issue. It is normal to see this message a few times when Cassandra is first starting up, but it should clear once the Management API agent has been created towards the very end of the Cassandra startup process.
If this message does not clear, it means that something has prevented Cassandra from fully starting. This could be for a number of reasons, and the only way to tell is to look at the Cassandra startup logs. If you are running this inside of kubernetes k8ssandra or k8ssandra-operator, you may be able to get some of the startup logs from the system-server-logger container in the pod. You can also try to exec/bash into the pod while it's starting and try to inspect the logs at /var/log/cassandra. In that directory, there should be a stderr.log file that might have more clues as to why Cassandra isn't successfully starting. There might also be something in debug.log as well.
@emerkle826 - thank you. It seems looking into system-server-logger logs Cassandra is not starting. "textPayload: "tail: cannot open '/var/log/cassandra/system.log' for reading: No such file or directory". Any option enabling verbose logging much appreciated.
I've seen this behaviour many times before. As far as I understand system-manager should receive a request from k8ssandra-operator to start cassandra within a container and it does not happen for some reason. Is there a way to understand the reason?
As far as I understand system-manager should receive a request from k8ssandra-operator to start cassandra within a container and it does not happen for some reason. Is there a way to understand the reason?
The request can be seen in the cassandra container (search for /start REST call). If you don't see it happening, then the cass-operator logs often tell why it's looping over something else and not starting the resources.
However, if /start fails due to Kubernetes, then you need to search for the reason with kubectl describe / events / kubelet logs etc.
@burmanm can you provide the list of permissions I need toward the namespace where k8ssandra operator lives? My cluster is in a separate namespace.
These details and minimum CPU and memory should be well documented, if not pre-defined in the manifests. Shouldn't the logs on the K8ssandraCluster also provide enough information, or at least point us in a plausible direction; creating GitHub issues should be the last resort where we make changes for future users to avoid the same issue.
Hi, Pretty new at this, but got the same issue. but looks like a naming issue, don't know what oss-mgmt.lock is used for...
INFO [main] 2023-02-08 10:47:16,288 IPCController.java:139 - Started Server
Started service on file:///tmp/oss-mgmt.sock
INFO [nioEventLoopGroup-3-1] 2023-02-08 10:47:23,144 UnixSocketCQLAccess.java:88 - Cannot create Driver CQLSession as he driver socket has not been created. This should resolve once Cassandra has started and created the socket at /tmp/cassandra.sock
I'm using k8ssandra operator version 0.39.3, with spec using Cassandra version 4.0.5
@emerkle826 - thank you. It seems looking into system-server-logger logs Cassandra is not starting. "textPayload: "tail: cannot open '/var/log/cassandra/system.log' for reading: No such file or directory". Any option enabling verbose logging much appreciated.
Hi,
I got my issue resolved. Nothing to do with k8ssandra. We are using on Google GKE. We enabled ecrypting on the Persistent Disk and need to set it "volumeBindingMode: "WaitForConsumer". It has been set to "immediate" Hope it helps