Expose full Pod spec in Schedule

[ccremer]

Summary

As K8up user
I want to customize the pods spawned by K8up with specs specific to my environment
So that I can run backups in my special environment

Context

We have many feature requests open that ask for customization of the spawned K8up pods. Examples are

• image pull secrets (e.g. for proxy registry with auth)
• Security context
• Env vars
• Labels and annotations
• Volumes, secrets, configmaps
• Resources (CPU, memory)
• Init containers

Reasons why users ask for these is that Schedule and Backup spec are not exposing arbitrary Pod specs. Rather, we exposed selected specs on request.

This story is about opening up the specification in a Schedule so that users can customize the spawned job, similar to how it's done in PrebackupPod.

The difficulty of this story lies in the sophisticated merge behaviour of K8up-required values and the customized pod spec. Especially Arrays are a problem with any deep-merge strategy. We probably have to define some rules and framework where customization can happen and where not.

Out of Scope

No response

Further links

• PreBackupPodSpec
• #581
• #580

Acceptance Criteria

Given a `k8up.io/v2/Schedule` spec
When I edit the field `.spec.template.spec.containers[0].image`
Then K8up will update the resulting CronJob with my customized image

Given a `k8up.io/v2/Schedule` spec
When I edit the field `.spec.template.spec.containers[0].{args,volumeMounts,env}`
Then K8up operator will append the array items to the mandatory items at the end.

Implementation Ideas

• mergo library can merge arrays by appending: https://pkg.go.dev/github.com/imdario/mergo?utm_source=godoc#WithAppendSlice

[megian]

Keycloak is able to run in a full restricted mode, but k8up does still create pods without a security context:

Warning: existing pods in namespace "test" violate the new PodSecurity enforce level "restricted:latest"
Warning: backup-backup-backup-djm28-jjwxn (and 9 other pods): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile

[ccremer]

@megian In https://github.com/k8up-io/k8up/pull/499 (K8up v2.0.0) we introduced the possibility to set security context. Is this not sufficient to resolve the mentioned warnings?

[megian]

@ccremer is there a possibility to set allowPrivilegeEscalation=false in the securityContect of the container? If I saw that correctly, #499 just allows to configure the podSecurityContext.

[ccremer]

Hm, no, not that field... container security context cannot yet be modified and PodSecurityContext doesn't have the field allowPrivilegeEscalation.

How important is this? Is it worth its own story and make container security context customizable in v2 still?

[megian]

I think it's very important to have this on the roadmap. I think it's less urgent. I think should be available 3-6 months from now.

[megian]

As we have passed the 3-6 months. Would it be possible to set the PodSecurityContext. Is there any reason allowPrivilegeEscalation=false can't be the default?

[Kidswiss]

@megian it's possible to the the PodSecurityContext see here: https://k8up.io/k8up/2.7/references/api-reference.html#k8s-api-github-com-k8up-io-k8up-v2-api-v1-backupschedule

[megian]

Sorry I was wrong. allowPrivilegeEscalation applies to pod.spec.containers.securityContext not pod.spec.securityContext. To allow running K8up in the Pod Security Standards restricted mode it would be required to configure the container security context itself or apply all the values required by the restricted mode.

[Kidswiss]

It's now possible to specify the whole podSpec in schedules and jobs: https://docs.k8up.io/k8up/2.10/how-tos/schedules.html#_customize_pod_spec