Any support of Clickhouse-keeper?

[TheXs]

Hi, there. Is there any plan to support configuring param "clickhouse-keeper: other than "zookeeper" in deployment yaml? Just realized both clickhouse and clickhouse-operator only support hard-coded configs (server-id) for that feature. Which is reasonably inconvinient on cluster scale up(in this way you can change shardCount or replicaCount to achieve). Regards!

[alex-zaitsev]

We are investigating the best way to support Keeper. The best practices are yet to be developed.

[pieceofchalk]

What's the progress? There is a desire to try with the built-in clickhouse-keeper? )) Any tests/examples of it?

[spoofedpacket]

Same. I'm currently evaluating clickhouse keeper, but as a seperate deployment to clickhouse itself using the example: https://github.com/Altinity/clickhouse-operator/blob/master/deploy/clickhouse-keeper/clickhouse-keeper-3-nodes.yaml It works but isn't exactly production ready for my use case.

It would be amazing to have it rolled into the operator. Happy to help with testing.

[jasine]

any progress?

[davidspek]

I'm testing out an automated setup in with clickhouse keeper. You can see it here.

[lazywei]

@spoofedpacket when using the clickhouse-keeper-3-nodes.yaml do you specify the service clickhouse-keeper.namespace.svc inside the zookeeper setting in the clickhouse server, or do you explicitly specify each of the pod via the headless service like clickhouse-keeper-0.clickhouse-keeper-headless.namespace.svc + clickhouse-keeper-1.clickhouse-keeper-headless.namespace.svc + clickhouse-keeper-2.clickhouse-keeper-headless.namespace.svc?

[prashant-shahi]

@lazywei I suppose it's the latter one using headless service. The first one will make sense when only one instance of keeper is being used.

[lazywei]

@prashant-shahi thanks, that's what i figured. For people who's trying to figure this out: for some reason I'm unable to run the keeper healthily with the clickhouse/clickhouse-keeper image, so I just switch to clickhouse/clickhouse-server image but run the clickhouse-keeper command instead. With very similar config as above, it works.

[spoofedpacket]

@lazywei I suppose it's the latter one using headless service. The first one will make sense when only one instance of keeper is being used.

Yep, that's what I'm doing

[davidspek]

Just thought I'd report back here as well that the setup I describe in https://github.com/Altinity/clickhouse-operator/issues/959#issuecomment-1455956595 is what we are currently using in production to configure ClickHouse-Keeper within the ClickHouse installation CR and it's working well. It should be possible to integrate this into the operator without needing a dedicated CRD with some automated smarts and/or restriction around scaling but I don't think I will have the time to do this any time soon.

[stasDomb]

I'm sorry. Could you please help me? I'm trying to use the example: clickhouse-keeper-3-nodes.yaml And I got the keeper deployment but it doesn't connect to ClickHouse. I think, I have bad configuration. Here is the part of my cluster config:

metadata:
  name: "data-warehouse"
spec:
  configuration:
    zookeeper:
      nodes:
        - host: clickhouse-keeper-0
           port: 2181
        - host: clickhouse-keeper-1
           port: 2181
        - host: clickhouse-keeper-2
          port: 2181
    clusters:
      - name: "staging"
        layout:
          shardsCount: 1
          replicasCount: 3

I think I need to put different host for the keeper, but I don't understand. Thanks and sorry for

[stasDomb]

The problem was fixed when I just put service name as a host:

      nodes:
        - host: clickhouse-keeper
           port: 2181

[sourabhaggrawal]

I am also struggling with the same issue,

I have installed clickhouse-operator using helm here without making any change in values files. I have also installed cluster using this with 1 shard and 2 replicas.

Have also configured separate keeper config clickhouse-keeper-3-nodes.yaml and in my clickhouse cluster I have tried nodes:

        - host: clickhouse-keeper-0
           port: 2181
        - host: clickhouse-keeper-1
           port: 2181
        - host: clickhouse-keeper-2
          port: 2181

as well as this

  nodes:
        - host: clickhouse-keeper
           port: 2181

but at the time of creating a table that I want to replicate I get this error,

Received exception from server (version 22.3.19): Code: 999. DB::Exception: Received from localhost:9000. DB::Exception: All connection tries failed while connecting to ZooKeeper. nodes: 100.68.182.175:2181 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale (Connection loss): while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 22.3.19.6 (official build)), 100.68.182.175:2181 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale (Connection loss): while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 22.3.19.6 (official build)), 100.68.182.175:2181 Code: 209. DB::NetException: Timeout exceeded while reading from socket (100.68.182.175:2181, 10000 ms): while receiving handshake from ZooKeeper. (SOCKET_TIMEOUT) (version 22.3.19.6 (official build)), 100.68.182.175:2181

How do I tell clickhouse-cluster to use keeper url instead of localhost ?

[sourabhaggrawal]

I am also struggling with the same issue,

I have installed clickhouse-operator using helm here without making any change in values files. I have also installed cluster using this with 1 shard and 2 replicas.

Have also configured separate keeper config clickhouse-keeper-3-nodes.yaml and in my clickhouse cluster I have tried nodes:

        - host: clickhouse-keeper-0
           port: 2181
        - host: clickhouse-keeper-1
           port: 2181
        - host: clickhouse-keeper-2
          port: 2181

as well as this

  nodes:
        - host: clickhouse-keeper
           port: 2181

but at the time of creating a table that I want to replicate I get this error,

Received exception from server (version 22.3.19): Code: 999. DB::Exception: Received from localhost:9000. DB::Exception: All connection tries failed while connecting to ZooKeeper. nodes: 100.68.182.175:2181 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale (Connection loss): while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 22.3.19.6 (official build)), 100.68.182.175:2181 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale (Connection loss): while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 22.3.19.6 (official build)), 100.68.182.175:2181 Code: 209. DB::NetException: Timeout exceeded while reading from socket (100.68.182.175:2181, 10000 ms): while receiving handshake from ZooKeeper. (SOCKET_TIMEOUT) (version 22.3.19.6 (official build)), 100.68.182.175:2181

How do I tell clickhouse-cluster to use keeper url instead of localhost ?

My keeper nodes were having issues, this issue got sorted after fixing that.

[madhur-df]

@sourabhaggrawal What was the issue with keeper nodes and how did you fix it?

[sourabhaggrawal]

Hi @madhur-df Its been long time now but I vividly remember it was because my keeper was not running healthy. So I guess once that got sorted my issue got sorted.