Problem with failed node

[yudinkr]

Hello!

We have 3 worker nodes and a coordinator. Distributed tables with partitioning. One of the nodes has failed (hardware problem). The database has been corrupted and I want to add a new clean database to the cluster. So I'm trying to remove or disable the failed node by selecting citus_disable_node or citus_remove_node, but I get an error like "ERROR: cannot remove or disable the node node_1:5432 because because it contains the only shard placement for shard 113661 (this is one of my parent table) DETAIL: One of the table(s) that prevents the operation complete successfully is tablename HINT: To proceed, either drop the tables or use undistribute_table() function to convert them to local tables".

Is there another way to remove a node (even if it involves deleting its data from the cluster) without converting it to local tables?

I have no hope of recovering data from a failed node, I want to restore all nodes in cluster(even if the data from the failed node is lost)

Perhaps this is in the documentation, but I did not find a procedure for action when a node fails.

I ask for help and will be glad to any advice.

UPD. Citus version 12.0-1

Also, it is not possible to execute citus_drain_node before disabling/deleting the node, because the node’s database is damaged. I need to completely “unhook” it from the coordinator without deleting the data of other nodes.

[JelteF]

Ooooff, that sounds like a bad situation. Generally the way to do this is to create a new node from a backup from the one that is now broken. And then run citus_update_node(...) with the hostname+port.

If you don't have backups of the node at all, then it's sadly a situation that is not really supported by Citus. The only way out at that point would be using some manual hackery. One option I can think of is:

1. create a new node, run citus_update_node to change to that node.
2. manually create the shards that are expected to be on that node
3. and then probably: set metadatasynced to false for that node in pg_dist_node. To force a metadata sync.

[yudinkr]

Ooooff, that sounds like a bad situation. Generally the way to do this is to create a new node from a backup from the one that is now broken. And then run citus_update_node(...) with the hostname+port.

If you don't have backups of the node at all, then it's sadly a situation that is not really supported by Citus. The only way out at that point would be using some manual hackery. One option I can think of is:

1. create a new node, run citus_update_node to change to that node.

2. manually create the shards that are expected to be on that node

3. and then probably: set metadatasynced to false for that node in pg_dist_node. To force a metadata sync.

Thanks for the answer!

How can I correctly disconnect the damaged node from the coordinator so that I can try to make a backup?

[JelteF]

There's not really a built in way to remove a node if you want to allow for data loss. The only way is to replace it with another one using citus_update_node().

Also can you explain a bit more what is damaged about the node, because it sounds like it's still working good enough to take a back up?

[yudinkr]

Also can you explain a bit more what is damaged about the node, because it sounds like it's still working good enough to take a back up?

No. My mistake. Its totally corrupted.

So, in my understanding, if a node fails irrevocably, the only way is to backup the remaining nodes and rebuild the cluster (restore from backup)?

In other words, if we lose one node, we lose the cluster. Correct?

[JelteF]

In other words, if we lose one node, we lose the cluster. Correct?

If you don't have backups of your nodes, yes. Otherwise you can create a new node that you restore from the backup of the node that failed. And then use citus_update_node to swap the failed node with the restored copy of it.

[yudinkr]

Got it. Thank you very much for clarify!

[yudinkr]

2. manually create the shards that are expected to be on that node

One more question please.

How can I do it? I didn't find a way to manually create shards in the documentation. Maybe some function from the coordinator?

[JelteF]

How can I do it? I didn't find a way to manually create shards in the documentation. Maybe some function from the coordinator?

I really meant really manually, i.e. create the backing shard-tables on the new node yourself. e.g. running CREATE TABLE my_distributed_table_102022