Question: Multiple instances of same app in same cluster

[jonaskello]

To me it looks like the layout in this repo assumes that each instance of the app is in the same namespace but in separate clusters. We have some cases where we have multiple instances of the same app in the same cluster. Usually it is the development cluster that has some extra instances that is more stable than the CI/CD build. In some cases we have the staging version in the same cluster as dev.

My question is how this is best accomplished using flux2 multi-tenancy. For example could I somehow create multiple namespaces for the same tenant repository (one ns per app instance)? Or is there some other way to handle this that is better suited for flux2 (not using separate ns)?

I know there are HNS which seems suited for this but it also seems a bit experimental and adds another extension to manage.

[lsolovey]

Hi, I have the same question. Any update on this? Thanks!

[jonaskello]

In my case I ended up using a single namespace for the tenant and two kustomiztion.yaml files with different nameprefix setting for the two deployments of the same app. Both kustomiztion.yaml for the deployed instances reference a base kustomiztion.yaml where all the manfiests for the app lives. The nameprefix setting will give unique names to the objects created for each deployment.

[lsolovey]

In my case I ended up using a single namespace for the tenant and two kustomiztion.yaml files with different nameprefix setting for the two deployments of the same app. Both kustomiztion.yaml for the deployed instances reference a base kustomiztion.yaml where all the manfiests for the app lives. The nameprefix setting will give unique names to the objects created for each deployment.

Thank you @jonaskello, this makes sense.

@stefanprodan - just wondering if there is any other flux-recommended way to install the same app into multiple namespaces?

[olljanat]

FYI. I have been looking for to create multi-tenant setup with hierarchical namespaces noticed similar limitation on it that multiple copies of application on same cluster would be problematic especially in-case where application is split to multiple namespaces. I have been written about it to https://github.com/kubernetes-sigs/hierarchical-namespaces/issues/79 which might be useful on here too.

[Bujail]

In my case I ended up using a single namespace for the tenant and two kustomiztion.yaml files with different nameprefix setting for the two deployments of the same app. Both kustomiztion.yaml for the deployed instances reference a base kustomiztion.yaml where all the manfiests for the app lives. The nameprefix setting will give unique names to the objects created for each deployment.

Hi @jonaskello Did you do this in flux2 or flux v1? I'm unable to achieve this in flux2. Can you please share your kustomization file?

[jonaskello]

@Bujail I'm using flux2. The prefixing is not a flux specific feature, you just use nameprefix in regular kustomization.yaml like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: myapp
namespace: myns
nameprefix: my-prefix-
resources:

[Bujail]

I fixed my issue by using following kustomize.toolkit file:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: external-dns-one
  namespace: flux-system
spec:
  interval: 5m0s
  path: "./external-dns/"
  prune: false
  sourceRef:
    kind: GitRepository
    name: external-dns-one
  validation: none
  targetNamespace: kube-system
  patches:
    - patch: |
        - op: add
          path: /spec/template/spec/containers/0/args/0
          value: --source=my-gateway
      target:
        group: apps
        version: v1
        kind: Deployment
        annotationSelector: "patchsel=external-dns"
  postBuild:
    substitute:
      name: "external-dns-one"
      replicas: "1"
      zone: "private"
      owner_id: "nginx-one-"
      ingress_class: "nginx-one"
      cpu_limit: "100m"
      mem_limit: "200Mi"
      cpu_req: "20m"
      mem_req: "100Mi"
      node_group: "agent"