Using Statefulset and PVC causing problems when or if Pod restarts

[ayush268]

Currently I am using the setup provided in this repo for statefulset with a few exceptions -

• NOT using configure-node.sh
• I am using "terminationGracePeriodSeconds" to be 10 instead of 0
• In couchbase service, I have not added "clusterIP: None" option
• NOT using "couchbase-ui service".

The issue I am facing is that when or if Pod - "couchbase-0" restarts, its ip will change due to flannel and because of that I am not able to connect to the UI by port-forwarding and the error log says

[ns_server:error,2017-07-20T19:23:34.313Z,nonode@nohost:dist_manager<0.142.0>:dist_manager:init:178]Configured address '10.2.17.3' seems to be invalid. Will refuse to start for safety reasons.

Since we are using persistant volumes the ip must be stored in them and if I go in the pod and manually change the "/opt/couchbase/var/lib/couchbase/ip" file, it kind of resets the settings I had done to the couchbase server and I have to set it up all again. Also it should detect the two servers i.e. "couchbase-0 and couchbase-1" by itself but it does not do that, I have to manually add the servers and rebalance them.

I have also opened all the ports required - 8091,8092,11207,11209-11211, 11214,11215,18091,18092,21100-21299 but still the problem persists. Could you tell me what could I do to fix this issue?

[arun-gupta]

@Laurafautley or @msanilkumar2020 can provide more guidance on this.

[anilkumar29]

Hi @ayush268 thank you for posting your question. I was going to publish blog announcing availability of developer preview of Couchbase running on Kubernetes using statefulsets. Here is the GitHub repository for developer preview - https://github.com/couchbase-partners/couchbase-kubernetes-openshift Feel free to email me directly at [email protected]. Thanks!

[anilkumar29]

Here is the link to blog - https://blog.couchbase.com/couchbase-openshift-enterprise-kubernetes-developer-preview-available/

[mrjumpy]

same here. any update?

[msavlani]

Hi @mrjumpy @ayush268 @arun-gupta @anilkumar29 , Did we solve this problem ? I am facing same issue where couchbase is writing its IP address in PV and when it restarts (Due to node unavailability, etc), it get new IP. And now couchbase does not come due to different IP address in PV and new container.

Please let me know if you resolved this problem.

[mrjumpy]

I just change to use VMs instead of POD I think K8s not suitable for couchbase now

[ghost]

Did anyone manage to work around this issue?

[anilkumar29]

Hi @msavlani, @mrjumpy, @oliver-whiteman, I wanted to share with you what we have been working for last few months. We have built native integration with Kubernetes using Couchbase Operator. I announced BETA release last week and for details on what’s in this release visit our blog post and to get started quickly follow steps under Prerequisites and Setup to deploy Couchbase Operator on Kubernetes platform.

[ghost]

Wow, completely missed that @anilkumar29 Is there a minimum supported version of couchbase server? We are currently on 4.5 so would we need to investigate upgrades?

[adarshaj]

@anilkumar29 - where can i find the source code for this operator? Also, I couldn't see a way to customize the storage being used, is it under plans?

[adarshaj]

bump. If not source code, where can I file issues with couchbase operator?

[Laurafautley]

Hi you file a question for help on forums.couchbase.com or if you'd like to email me directly I can put you directly in contact with he developer. I can be contacted at laura@ couchbase.com

On 10 Mar 2018 14:43, "Adarsh J" [email protected] wrote:

bump. If not source code, where can I file issues with couchbase operator?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arun-gupta/couchbase-kubernetes/issues/7#issuecomment-372059615, or mute the thread https://github.com/notifications/unsubscribe-auth/ALVdddHReAcY9gMXLqJhXeSL7s452Shaks5tdCzTgaJpZM4OfznK .

[misko566]

I have similar problem. After pod restart it has new IP and Couchbase service won't start because It has persistent volumes data. Can I fix this by some script or overwriting some values with startup script?

[matsunanaro]

I think the issue here is the use of IP address as the node name. When a k8s pod is restarted, it would be automatically assigned a new IP which is different to the one saved in Couchbase. This would make Couchbase fail to find its master and nodes after reboot.

I fixed the issue by using the full hostname instead, e.g

from IP=hostname -I to IP=hostname -f

It would give something like couchbase-N.couchbase.default.svc.cluster.local where N is an index starting from 0 Our cluster seems now to be working after reboot as well.