citus
citus copied to clipboard
citusdata
Kubernetes support
@rimusz Thanks for opening this issue!
I've had this on my list for a while to look at, although I haven't gotten to a fully blown setup yet. Could you describe your use case in more detail?
Happy to update this issue once I've done some more testing :-)
@lfittl I'm just looking an easy way to bootstrap citus cluster on kubernetes for now
@lfittl I would love to have an easy way to bootstrap citus cluster on Kubernetes as e.g. youtube vitess does for mysql http://vitess.io/getting-started/
@rimusz 💯, I'm a fan of the work YouTube did with Vitess.
I haven't been able to find time for this just yet (we've been busy with the launch of Citus Cloud), but its on my list still - although someone else from the team might find time sooner.
Out of curiosity, are you running on Google Compute Engine, or is this for another K8S-based environment?
@lfittl I'm a big fan of GKE, no needs to setup/maintain CoreOS based Kube clusters anymore :) This is why I tweeted back about Citus Cloud on Google Cloud as well.
This would indeed be awesome. And I think with the release of Kubernetes 1.3, and the inclusion of "Pet Sets", this would fit perfectly with the goal of Citus.
Big +1 here as well. Has anybody started working on this yet? If so I'd like to help out!
UPDATE: After looking around a bit I found this: https://github.com/jberkus/atomicdb/tree/master/citus_petset - haven't tested it myself yet.
@xarg -- As you noted, @jberkus has been looking at Kubernetes integration with Citus.
Josh had a talk at PGConf SV where he demoed a Citus cluster deployment with Kubernetes: http://www.pgconfsv.com/ccp-containerized-clustered-postgres-0
Once the talk is online, I'll also post the link to it here.
Also happy to send an introduction to @jberkus and also be a part of this conversation. We're paying close attention to Kubernetes.
First cut at this is up on my project: https://github.com/jberkus/atomicdb/tree/master/citus_petset
Not really production-quality yet, working on it. Contributions welcome.
@jberkus Glad for your great work! Question: "Due to an outstanding bug with PetSet and PVs on AWS;" Could you link to that issue so I can follow it?
that actually has to do with KubeAdm more than anything.
I'll be building a Helm chart this week.
Hello Team,
Do you have any plans to have citus Integrated with Kubernetes?.
I heard, that citus has plans to integrate with K8s in their pipeline.
Thanks and Regards, Arjun.M
Looking to run this in Kube as well. I would love to help out with an OSS solution if I can. Writing an helm chart is probably the easiest way, but an operator is probably the best. @lfittl
Working example for "local-ssd-storage" aka affinity on special node node01.vm.domain.com where available.
apiVersion: v1
kind: PersistentVolume
metadata:
name: database
spec:
capacity:
storage: 100Gi
# volumeMode field requires BlockVolume Alpha feature gate to be enabled.
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-ssd-storage
local:
path: /mnt/disks/ssd1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node01.vm.domain.com
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: citus-master-pvc
labels:
app: citus-master
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: local-ssd-storage
local:
path: /mnt/local-storage/ssd/disk1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node01.vm.domain.com
apiVersion: v1
kind: Service
metadata:
name: citus-master
labels:
app: citus-master
spec:
selector:
app: citus-master
ports:
- port: 5432
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: citus-master
spec:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node01.vm.domain.com
selector:
matchLabels:
app: citus-master
replicas: 1
template:
metadata:
labels:
app: citus-master
spec:
containers:
- name: citus
image: citusdata/citus:8.0.0
ports:
- containerPort: 5432
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: citus-secrets
key: password
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: citus-secrets
key: password
volumeMounts:
- name: storage
mountPath: /var/lib/postgresql/data
livenessProbe:
exec:
command:
- ./pg_healthcheck
initialDelaySeconds: 60
volumes:
- name: storage
persistentVolumeClaim:
claimName: citus-master-pvc`
apiVersion: v1
kind: Service
metadata:
name: citus-workers
labels:
app: citus-workers
spec:
selector:
app: citus-workers
clusterIP: None
ports:
- port: 5432
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: citus-worker
spec:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node01.vm.domain.com
selector:
matchLabels:
app: citus-workers
serviceName: citus-workers
replicas: 3
template:
metadata:
labels:
app: citus-workers
spec:
containers:
- name: citus-worker
image: citusdata/citus:8.0.0
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- if [ ${POD_IP} ]; then psql --host=citus-master --username=postgres --command="SELECT * from master_add_node('${HOSTNAME}.citus-workers', 5432);" ; fi
ports:
- containerPort: 5432
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: citus-secrets
key: password
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: citus-secrets
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: wstorage
mountPath: /var/lib/postgresql/data
livenessProbe:
exec:
command:
- ./pg_healthcheck
initialDelaySeconds: 60
volumeClaimTemplates:
- metadata:
name: wstorage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Gi
storageClassName: local-ssd-storage
local:
path: /mnt/local-storage/ssd/disk1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node01.vm.domain.com
apiVersion: v1
kind: Secret
metadata:
name: citus-secrets
type: Opaque
data:
password: XXXXXXXXXXXXXXXXXXXXXXXXXXX
Thanks for those kubernetes files of a citus deployment! I was wondering if anybody got it to work to run multiple independent citus instances in the same k8s cluster. I tried changing ports but that didn't got well. I miss something to assign workers to a specific master, don't know where to look
@jenswachtel You might also want to give Azure Arc enabled PostgreSQL Hyperscale a try:
https://docs.microsoft.com/en-us/azure/azure-arc/data/create-postgresql-hyperscale-server-group
This is the officially supported version of Citus on Kubernetes (with a slightly more complex product name 😄), thats currently in Public Preview. Supports Citus running on Kubernetes anywhere (i.e. not just on Azure).
Running multiple Citus instances is a built-in capability that you can easily use from either the "azdata" CLI, or directly through Kubernetes resource definitions.
@lfittl wow, thank you so much! It works perfectly!
I can highly recommend Azure ARC with Citus. I tested it on AKS, but looks like you can do this with GCP etc.
Pay attention to the smaller details e.g. don't provision cluster with different availability zones, it works pretty neatly also with existing clusters!
Step 1) Activate AKS ARC on Cluster & Connect Described here https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/connect-cluster#install-cli-extensions
Step 2)
Install ARC Data Services on Cluster
I can recommend going the way via Azure Portal
Add > "Azure Arc data Controller"
Fill out blanks and download juniper notebook which you then will have to run inside
Azure Data Studio
(minor addition, add this line to Step 5 in the scripts
os.environ["ACCEPT_EULA"] = "yes"
Step 3) Create Citus Group Cluster as mentioned by @lfittl https://docs.microsoft.com/en-us/azure/azure-arc/data/create-postgresql-hyperscale-server-group
I am so amazed, I can finally provision citus clusters in a matter of minutes!
@jenswachtel does it require having microsoft accounts and or install stuff on azure? or is it entirely controlled by say GCP?
Please be honest and close this issue as this won't be done officially. MS won't provide us with a helm chart to install this on kubernetes. They try to nudge customers to Azure. Azure arc is a paid service.
passing by, just to say I got a working POC on kubernetes https://github.com/docteurklein/citus-test
@docteurklein Yes, thanks for the start. The docker images are over a year old. MS really tries to raise the bar for entry. At least you can build your own fairly current docker images from this branch
@GeroL The hub images look pretty recent to me https://hub.docker.com/r/citusdata/citus/tags