Provide helm-based installation

[YiannisGkoufas]

It might be easier to provide a helm chart than custom scripts with envsubst

[christian-pinto]

This should also include a better build system. We are currently rebuilding everything every time a make is executed, while binaries and container images should only be re-built if need-be.

[davidyuyuan]

I think that this would benefit users significantly. Please consider to support Helm 3 only. It is much better than Helm 2.

[YiannisGkoufas]

~~Since Kustomize is utilized in the operator-sdk 1.0 we need to update the deployment and manifests building process.~~

Switching back to Helm based installation, we hit some of kustomize's limitations with our templates

[masterkain]

I have trouble installing the current helm chart, I get this (using flux2)

LAST SEEN   TYPE     REASON   OBJECT                 MESSAGE
2m7s        Normal   info     helmrelease/datashim   HelmChart 'flux-system/datashim-datashim' is not ready
119s        Normal   info     helmrelease/datashim   Helm install has started
119s        Normal   error    helmrelease/datashim   Helm install failed: error while running post render on files: may not add resource with an already registered id: ~G_v1_ServiceAccount|datashim|csi-attacher...
119s        Normal   error    helmrelease/datashim   Helm uninstall failed: uninstall: Release not loaded: datashim: release: not found...
119s        Normal   error    helmrelease/datashim   reconciliation failed: Helm uninstall failed: uninstall: Release not loaded: datashim: release: not found
37s         Normal   error    helmrelease/datashim   reconciliation failed: previous release attempt remediation failed

[masterkain]

I ended up using the yaml file directly

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf.yaml

[jalberto]

When installing current helm by helm install datashim ./chart -ndatashim --create-namespace there is one pod, csi-controller-h3-0, refusing to run, here is the log:

I0913 15:45:56.790213       1 main.go:77] Version: v1.0.1-0-g0660c6c6
I0913 15:45:56.790445       1 main.go:80] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0913 15:45:56.790468       1 connection.go:69] Connecting to /csi/csi.sock
I0913 15:45:56.791257       1 connection.go:96] Still trying, connection is CONNECTING
I0913 15:45:56.791613       1 connection.go:93] Connected
I0913 15:45:56.791635       1 main.go:88] Calling CSI driver to discover driver name.
I0913 15:45:56.791658       1 connection.go:137] GRPC call: /csi.v1.Identity/GetPluginInfo
I0913 15:45:56.791666       1 connection.go:138] GRPC request: {}
I0913 15:45:56.794550       1 connection.go:140] GRPC response: {"name":"csi-h3","vendor_version":"v1.1.0"}
I0913 15:45:56.798864       1 connection.go:141] GRPC error: <nil>
I0913 15:45:56.798873       1 main.go:96] CSI driver name: "csi-h3"
I0913 15:45:56.798878       1 main.go:100] Loading kubeconfig.
I0913 15:45:56.799281       1 k8s_register.go:54] AttachRequired: true
I0913 15:45:56.799288       1 k8s_register.go:55] PodInfoOnMountVersion: "v1"
I0913 15:45:56.799293       1 k8s_register.go:58] Registering csidrivers
E0913 15:45:56.817897       1 k8s_register.go:71] the server could not find the requested resource

[jalberto]

Same error by using recommended method kubectl apply -f https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf.yaml

[jalberto]

@masterkain did you take any extra step to make it work?

[masterkain]

@masterkain did you take any extra step to make it work?

it started installing with this https://github.com/datashim-io/datashim/issues/7#issuecomment-917619393 so no extra steps involved, however I had to temporary leave this solution because of crashes, will try again in future

[jalberto]

Thanks, but for me not even that method works

[olevski]

Hi this is a really cool project. I am working on using it over at https://github.com/SwissDataScienceCenter/renku and it would be really nice if there was a helm chart repo because we use Helm extensively over at Renku. As far as I can tell there isn't one though.

I am happy to write some github workflows to set one up. If someone with proper permissions sets up github pages and lets me know when should charts be built/published I am happy to submit a PR for a github workflow that publishes the helm charts.

To not stall my work over on Renku I made a simple repo that periodically checks this repo for new charts and publishes them over at https://github.com/SwissDataScienceCenter/datashim-charts. Hopefully this is ok. I am more than happy to shut this down when the helm charts are published officially.

[jalberto]

@olevski there is a helm chart in the repo, but it doesn't work out of the box

[olevski]

@olevski there is a helm chart in the repo, but it doesn't work out of the box

Hmm. I have not noticed @jalberto. Maybe it is because I only need the s3 functionality. I see your earlier comments are about the h3 controller. I did not really test that but I am pretty sure I installed it ok.

Perhaps the installation problems you are having are tied to your k8s cluster setup? I have so far tested mostly with kind.

[olevski]

Is there any interest in publishing the images and helm charts?

What I have here can be very easily adapted to the original repo here.

I am happy to open a PR and do this. All I need is someone to add the docker registry username and password as secrets to this repo so the images can be pushed. The helm charts can be hosted with github pages.

[srikumar003]

@olevski What you have done is quite useful. can you make a PR please ? The images for datashim are hosted on quay.io. I'll attempt to host the helm charts in the project github pages

[5cat]

Hey @srikumar003 , do you have any updates on this? I cant find a URL/OCI for a repository that has datashim as a helm chart. And I had issues with using helm+git plugin to install it directly from the git repo.