app-group deploy - kapp config

[RafPe]

Hi,

Documentation refers that kapp config can be specified explicitly with -f param however the part of cli kapp app-group deploy does not support this param.

NOTE: kapp is only affected by a Config (whether wrapped in a ConfigMap or not) when supplied as a direct input (i.e. as a -f argument).

How should I provide the config file for kapp rebase rules when deploying group apps ?

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this bug fixed as soon as possible" 👎 "There are more important bugs to focus on right now"

We are also happy to receive and review Pull Requests if you want to help fix this issue.

[gcheadle-vmware]

Hey @RafPe,

You are correct that you cannot use the -f flag with the kapp app-group command. To include your rebase rules when deploying, you can place the kapp config in its own yaml file, and place that file in the directory which that config applies. Thus you would need to provide a kapp config for each directory (each dir is a kapp app in the kapp app-group command case) that needs one.

This seems like a good signal for us to update some of the documentation around the kapp-app command. Was the lack of documentation one of the issues here?

Please let us know if we can help any further. cc: @DennisDenuto

[RafPe]

You are correct that you cannot use the -f flag with the kapp app-group command. To include your rebase rules when deploying, you can place the kapp config in its own yaml file, and place that file in the directory which that config applies. Thus you would need to provide a kapp config for each directory (each dir is a kapp app in the kapp app-group command case) that needs one.

Well this for me its a bit of cumbersome approach.I would like to avoid duplications. I recognize where we can have scenarios where separate rules per app may have a way of be. From my side I would suggest if possible:

• adding a switch which would apply provided rules file across iteration of applying into group
• adding switch to specify config map which holds the rules for evaluation

This seems like a good signal for us to update some of the documentation around the kapp-app command. Was the lack of documentation one of the issues here? Yes - documentation regarding the rules config file itself and the way it is applied could be imho improved

[cari-lynn]

@RafPe, could you elaborate on what you mean by

adding switch to specify config map which holds the rules for evaluation

This issue came up in our march 29th community meeting, this is what we talked about: @pivotaljohn @joaopapereira

kapp app-group command has come up in a ytt issue in the past, in that issue it was brought up that kapp app-group could be done by a script iterating over directories to deploy and was mentioned that perhaps it isn't a pattern we want to follow across the Carvel tools and invest resources in. This is still an open question.

One way to approach this is to give kapp app-group a kapp Config resource that applies to all of the apps in subdirectories by placing the file containing the kapp Config at the root directory. That way would be consistent with the directory structure as input app-group has currently.

app-group command was added to be friendly for use with GitOps, so should we encourage keeping any configuration (like kapp Config) needed to deploy an app in the same place where configuration for that app lives? Would it be confusing to not have configuration in a app's directory?

[RafPe]

@RafPe, could you elaborate on what you mean by

adding switch to specify config map which holds the rules for evaluation

With this one I was just tinkering if something like kapp app-group --global-config-cm some-namespace/config-map-name could be implemented. This of course is one of many approaches that could be taken.

kapp app-group command has come up in a ytt issue in the past, in that issue it was brought up that kapp app-group could be done by a script iterating over directories to deploy and was mentioned that perhaps it isn't a pattern we want to follow across the Carvel tools and invest resources in. This is still an open question.

I think this something we would like to address. We could try to get some feedback from community - however in scenarios where I came across this it makes sense that I do not replicate the kapp config file across the app folders but just have one which by default applies to all resources. If I were to resource to scripting - then it defeats the purpose of using kapp from my point of view

One way to approach this is to give kapp app-group a kapp Config resource that applies to all of the apps in subdirectories by placing the file containing the kapp Config at the root directory. That way would be consistent with the directory structure as input app-group has currently.

That is something which would work in my opinion.

app-group command was added to be friendly for use with GitOps, so should we encourage keeping any configuration (like kapp Config) needed to deploy an app in the same place where configuration for that app lives? Would it be confusing to not have configuration in a app's directory?

I see your point here - hence my suggestion to have specific config in the target cluster of kubernetes in configMap.

[cari-lynn]

@RafPe would you mind sharing what your kapp Config is and what you have in your configuration so we can better understand your use case? I'm curious how you have your kapp Config wrapped in a ConfigMap.

I think we want to move forward with this issue of allowing a kapp Config that can be applied to all apps in an kapp app-group command. Currently kapp will ignore resources in the root directory of the app-group command, I think we want to keep this behavior and only use a kapp Config if it is present in the root directory of the command.

Acceptance Critera

Config applies to all apps

Given I have a Config file I want to apply to all my apps When I deploy using kapp app-group deploy -g app-name Then I see my Config has applied to all apps

Given this directory structure:

.
|_ app1/deployment.yml
|_ kapp-config.yml

and a kapp-config.yml

apiVersion: kapp.k14s.io/v1alpha1
kind: Config

additionalLabels:
  department: marketing
  cost-center: mar201

Then when I deploy using kapp app-group deploy -g app-name and inspect my deployment with kubectl get deployment nginx-deployment -o yaml I should see it has the labels

department: marketing
cost-center: mar201

---

⚠️ Updated Acceptance criteria at bottom of this thread ⚠️

~~Does not include root level resources other than Config~~

Given this directory structure:

.
|_ app1/deployment.yml
|_ other-resource-thats-not-kapp-config.yml

where app1/deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

and other-resource-thats-not-kapp-config.yml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: root-deployment-dont-use-me
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

Then when I deploy using kapp app-group deploy -g app-name and inspect my deployment with kubectl get deployments I should see only a single deployment called app1-deployment.

[RafPe]

@RafPe would you mind sharing what your kapp Config is and what you have in your configuration so we can better understand your use case? I'm curious how you have your kapp Config wrapped in a ConfigMap.

So wrapping into ConfigMap was just an idea - its not something I did :) I have it as just flat file

Structure which Im working with looks following

├── apps
│   ├── cadvisor
│   ├── cert-manager
│   ├── custom-metrics
│   ├── grafana-cloud
│   ├── haproxy
│   └── kube-state-metrics
└── datasources
    ├── acc
    ├── prd
    └── tst

What happens is that I'm working with 3 env - and what follows - 3 kubernetes clusters. Each of the app is templated using gomplate and custom data sources per env.

My motivation here is to have maybe one ( would be perfect ) or per env kapp config file which I could apply per deployment.

I think we want to move forward with this issue of allowing a kapp Config that can be applied to all apps in an kapp app-group command. Currently kapp will ignore resources in the root directory of the app-group command, I think we want to keep this behavior and only use a kapp Config if it is present in the root directory of the command.

I agree that this is a good idea. I personally like the switch of config - it makes it very explicit then.

Acceptance Critera

Config applies to all apps

Given I have a Config file I want to apply to all my apps When I deploy using kapp app-group deploy -g app-name Then I see my Config has applied to all apps

Given this directory structure:

.
|_ app1/deployment.yml
|_ kapp-config.yml

and a kapp-config.yml

apiVersion: kapp.k14s.io/v1alpha1
kind: Config

additionalLabels:
  department: marketing
  cost-center: mar201

Then when I deploy using kapp app-group deploy -g app-name and inspect my deployment with kubectl get deployment nginx-deployment -o yaml I should see it has the labels

department: marketing
cost-center: mar201

Looks exactly as what we would like to achieve :)

Does not include root level resources other than Config

Given this directory structure:

.
|_ app1/deployment.yml
|_ other-resource-thats-not-kapp-config.yml

where app1/deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

and other-resource-thats-not-kapp-config.yml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: root-deployment-dont-use-me
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

Then when I deploy using kapp app-group deploy -g app-name and inspect my deployment with kubectl get deployments I should see only a single deployment called app1-deployment.

This would probably not be applicable if we make the config to be explicit to take effect via command line argument

[cppforlife]

I personally like the switch of config - it makes it very explicit then.

i would recommend this as well. in fact i would take this further and suggest just adding -f flag that can take any configuration (including kapp's Config). these -fs would be propagated to each kapp deploy executed.

[cari-lynn]

Updated the acceptance criteria to reflect these changes

Acceptance Critera

Scenario 1: Config applies to all apps

Given I have a Config file I want to apply to all my apps When I deploy using kapp app-group deploy -g app-name Then I see my Config has applied to all apps

Given this directory structure:

.
|_ app1/deployment.yml
|_ kapp-config.yml

where app1/deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

and a kapp-config.yml

apiVersion: kapp.k14s.io/v1alpha1
kind: Config

additionalLabels:
  department: marketing
  cost-center: mar201

Then when I deploy using kapp app-group deploy -g app-name OR kapp app-group deploy -g app-name -f kapp-config.yml and inspect my deployment with kubectl get deployment app1-deployment -o yaml I should see it has the labels

department: marketing
cost-center: mar201

---

Scenario 2: Other resources apply to all apps

Given this directory structure:

.
|_ app1/deployment.yml
|_ other-resource-thats-not-kapp-config.yml

where app1/deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

and other-resource-thats-not-kapp-config.yml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: root-deployment
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

Then when I deploy using kapp app-group deploy -g app-name OR with kapp app-group deploy -g app-name -f other-resource-thats-not-kapp-config.yml and inspect my deployment with kubectl get deployments I should see two deployments called app1-deployment and root-deployment.

[aaronshurley]

Thanks for solidifying the acceptance criteria. We're going to move this to the Unprioritized Backlog for now. This signals that the maintainers are not planning to prioritize this soon but would accept contributions.