snappydata icon indicating copy to clipboard operation
snappydata copied to clipboard

Published 20 hours ago • verified publisher icon TIBCOSoftware

Kubernetes support

Open elgalu opened this issue 6 years ago • 4 comments

Any plans to provide Kubernetes support? i.e. to provide all required manifests including auto scaling support.

Perhaps in alignment with Spark 2.3 Kubernetes efforts.

elgalu avatar Apr 04 '18 11:04 elgalu

Hi @elgalu this is a big part of our engineering effort right now. Unfortunately we don't have much that is public-facing currently because we are still merging Spark 2.3, but expect announcements in the very near future. For now you can check out: https://github.com/SnappyDataInc/spark-on-k8s. Also you can ask questions about it to our engineering on our public slack chat here: http://snappydata-slackin.herokuapp.com/

piercelamb avatar Apr 04 '18 16:04 piercelamb

So glad to read you are working on it!

elgalu avatar Apr 05 '18 01:04 elgalu

@elgalu Leo, thanks. You mentioned "auto scaling" ... do you have specific features you would like to see? The above mentioned spark-on-k8s project is all Spark. Here is the core YAML to start a Snappy cluster. It deploys Service objects for external access and uses persistent volumes for logs/config/data. We will clean all this up soon.

jramnara avatar Apr 05 '18 03:04 jramnara

Hi @jramnara , yes maybe I mixed things while mentioning auto scaling in relation to K8s.

Now reading back the cluster_architecture I realize the Lead Node, for instance, doesn't scale horizontally based on load but one has to tune it vertically by increasing CPU cores, RAM and so on?

elgalu avatar Apr 05 '18 04:04 elgalu