directpv
directpv copied to clipboard
minio
Deploy direct-csi as helm chart
Hi, since GitOps is the preffered way for developing and deploying applications to k8s, our organization advocates packaging everything as a helm chart and deploying it via umbrella charts. Also we are air-gapped, so we need to customize registry paths and etc. We would like to use your plugin in our single nodes k8s clusters but we are entitled to the kubectl-direct-csi binary for manually deploying the plugin. Can you please provide a clean and simple helm chart with values.yaml and manifests so we can have a GitOps solution for your plugin?
@ShayBenyo11 direct-csi cli already allows you to customize registry path. We have numerous air-gapped installations currently.
You can generate the spec using
kubectl direct-csi install --dry-run
This will give you the equivalent of a generated helm chart that you can deploy directly.
We also would love to install it via helm-chart since we install our core-apps for the kubernetes cluster using terraform. We 'chart'ez local-path by rancher too and would love to have the same way with direct-csi
@EugenMayer @ShayBenyo11 We will add docs shortly with details for generating a helm chart. Expect something by today/tomorrow-ish
Thank you for the effort! We created the helm chart in the meantime (using generate). Generally it works, The biggest issue is the array based structure. Considering that helm cannot deep merge arrays one can neither use --set nor -f. Luckily we use terraform and use a values.y'all template there to have all the parametrization there. The main issue here is the image name which needs to be maintained manually, otherwise we won't ever do an actual update.
@wlan0 it's internal, but there is really not a lot to see in general. we used the krew plugin to generate it and then installed it.
We could not test it due to the drive management, see below (but it seemed properly installed).
Sorry fur the wrong info with the yaml file, that was for minio-operator not for direct-csi, please excuse my confusion here. So the values.yaml via terraform part is actually for minio-operator.
We did not pursue further with direct-csi because it's management and concept is far more effort then comparing local-path by rancher, which we went back to. Especially the drive management and the need of to disks is what turned it away for us at least for now. (due to 2 disk requirement and the tunability to maintain disk via helm chart AFAICS). I understand why drives can make sense, also a 2 disk setup, but usually i want to avoid 2 disk setups in our iaas/openstack environment.
I would at least suggest that the krew plugin to generate the chart lets you define the --drives so one can generate a helm chart which does it all, not just the bare installation (IMHO), but also manages the drives (statically)
I suppose that the second disk is considered ephemeral, right? (sorry, wrong question here)
I also need to a helm chart for automatic cluster deployment. Is this still in progress?
it'd be also nice to be able to format the drives from code, creating the DirectCSIDrives with a chart.
I also need to a helm chart for automatic cluster deployment. Is this still in progress?
We provide a better approach using our kubectl-plugin. You can automate cluster deployment using the kubectl directpv install command. In addition to deploying directPV, it also handles upgrades and downgrades of CRD versions.
it'd be also nice to be able to format the drives from code, creating the DirectCSIDrives with a chart.
If you know which drives you want to format, you can simply run kubectl directpv format from a shell script.
I'm not sure I follow how helm chart will automate formatting. Helm will help in parametrizing yamls, but it requires the drive CRDs to begin with. It would be much easier to simply run the plugin command IMO. That being said, do let me know if I'm missing an important usecase of yours - I'll be happy to solve the issue if there is no better way.
We provide a better approach using our kubectl-plugin
With a Helm chart we don't need to "kubectl ..." things, it give us a possibility to use GitOps for example, to automate deployment and make it more "agnostic".
Imagine 10 kubernetes clusters and you switching between contexts to do a kubectl directpv install...it's hard to manage.
Krew works pretty well for local tests but for production environments no.
We provide a better approach using our kubectl-plugin
With a Helm chart we don't need to "kubectl ..." things, it give us a possibility to use GitOps for example, to automate deployment and make it more "agnostic". Imagine 10 kubernetes clusters and you switching between contexts to do a
kubectl directpv install...it's hard to mange. Krew works pretty well for local tests but for production environments no.
Exactly this. The krew plugin is nice, but if every project would start to provide krew plugins rather than the canonical way to use the k8s API to get things done, it would easily become a mess very soon.
The ability to use DirectPV via simple manifests is something crucial in the age of GitOps...
I'm not sure I follow how helm chart will automate formatting. Helm will help in parametrizing yamls, but it requires the drive CRDs to begin with. It would be much easier to simply run the plugin command IMO. That being said, do let me know if I'm missing an important usecase of yours - I'll be happy to solve the issue if there is no better way.
Let me join the endorsement of this very important enhancement - once available, giving also the ability to define the desired bootstrapping drives format (maybe via some node selector and tolerations (might be needed in Rancher mixed (Windows and Linux) clusters) directive?) will eliminate the need to deal with the plugin and allow for a coherent installation method. I for once, would be very glad to give this feature a test run once ready (assuming the designated helm chart would be published on a helm repository)...
Question - does this driver support Windows nodes?
Hi, There is another problem with dry-run, it requires KUBECONFIG and therefore it is hard to use it for the creation of HELM chart.
In next release ./kubectl-directpv install --dry-run will work without kubeconfig.
In next release
./kubectl-directpv install --dry-runwill work without kubeconfig.
How does this fix deploying direct-csi via gitops and/or helm?
In next release
./kubectl-directpv install --dry-runwill work without kubeconfig.How does this fix deploying direct-csi via gitops and/or helm?
We don't support helm chart. As --dry-run flag outputs installation yaml, users would need to deal with their own installation methods using the yaml.
@balamurugana I'm sorry, but installing CRDs is not the problem here. CRDs can be installed without the krew plugin, but anything else beyond having the CRDs is impossible with it. We don't need Helm chart, but something to actually use this - otherwise pretty good - software in real-life scenarios.
@balamurugana There was already a release since November, but the feature does not seem to be implemented yet. Is it still on the roadmap?
$ kubectl directpv install --dry-run
Error: Could not connect to kubernetes. KUBECONFIG=
$ kubectl krew info directpv
NAME: directpv
INDEX: default
URI: https://github.com/minio/directpv/releases/download/v3.2.2/kubectl-directpv_darwin_arm64.zip
SHA256: 77a8f52917ecf1ac5beceb6bcfc69f38150f68fc66598760a9cbdd3312c23cf2
VERSION: v3.2.2
HOMEPAGE: https://github.com/minio/directpv
DESCRIPTION:
Deploys and manages the lifecycle of DirectPV CSI
driver - A CSI for dynamically provisioning direct persistent volumes.
We are working on a new release. Stay tuned!
If you are testing DirectPV, you could try our snapshot development version which has the feature.
- Download https://github.com/balamurugana/directpv/releases/download/v3.1.0-40-gd2dec861-6-gfbabf7d7-4-g5bfcce74-2-gfec61f8f-1-g99ebc570/kubectl-directpv
- Install DirectPV snapshot by
$ kubectl-directpv install --registry docker.io --org baladevel