[BUG]: Custom annotations are not set in `garden-buildkit` Pods, but only in Deployment

[xdvpser]

Bug

https://docs.garden.io/reference/providers/kubernetes#providers-.clusterbuildkit.annotations

Providing custom annotations in providers[].clusterBuildkit.annotations section does not set annotations for pod template, but only for deployment.

Current Behavior

apiVersion: apps/v1
kind: Deployment
metadata:
  name: garden-buildkit
  ...
  annotations:
    deployment.kubernetes.io/revision: '1'
    custom.annotation: garden-buildkit
spec:
  ...
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
        container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
        #??? <<<--- No custom annotation!

Expected behavior

apiVersion: apps/v1
kind: Deployment
metadata:
  name: garden-buildkit
  ...
  annotations:
    deployment.kubernetes.io/revision: '1'
    custom.annotation/example: garden-buildkit
spec:
  ...
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
        container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
        custom.annotation/example: garden-buildkit    ## <<<--- Should be also set!!!

My project spec

kind: Project
name: root
...
providers:
  - name: kubernetes
    ...
    buildMode: cluster-buildkit
    clusterBuildkit:
      rootless: true
      annotations:
        custom.annotation/example: garden-buildkit
...

Additional info

I've tried to start with fresh environment, but it didn't help. Here is what I did:

1. removed all garden deployed resources with k8s namespace
2. removed .garden/ directory
3. deployed resources again

Your environment

• OS: Ubuntu 22.04
• How I'm running Kubernetes: Self-managed on EC2 instances using Kops

Garden version

Having this bug in 0.12.48 and 0.12.52 versions. Didn't test others.

[edvald]

Thanks @xdvpser! This seems to simply be an oversight, and easy to address. We'll triage this.