Label pod with workflow's labels

[hadim]

Currently, the pods of a given workflow are labelled with workflows.argoproj.io/completed and workflows.argoproj.io/workflow. Could we also add the labels of the workflow to that list, something like:

workflows.argoproj.io/workflow/label/project = "xxxx"
workflows.argoproj.io/workflow/label/tenant = "xxxx"

[alexec]

I think you can label your pods as follows. Look at examples/pod-metadata.yaml

[hadim]

Thanks. I am aware of this and should have mentioned it actually in my first message. I am looking for an automated way of doing it.

But maybe it's a too specific feature for Argo? (feel free to close if you think it is)

As a workaround, I'll use examples/pod-metadata.yaml.

[alexec]

Would you like to submit a PR to add this feature?

[hadim]

I'll consider it for sure if the examples/pod-metadata.yaml workaround becomes too cumbersome.

[arghya88]

@alexec I am interested to contribute this.Can you assign this to me?

[hadim]

I started to look at various entry points in the code to integrate this. I found https://github.com/argoproj/argo/blob/91bce2574fab15f4fab4bc4df9e50563aa748838/workflow/controller/workflowpod.go#L131 to be a potentially good candidate to add the labels of the workflow spec on-the-fly during pod creation. Maybe @alexec would like to see this elsewhere?

Another question is do we want to force by default? or is it best to add an option in the configmap about it?

[hadim]

@arghya88 still interested working on that?

[terrytangyuan]

@hadim You can now define podMetadata at workflow level and then those metadata will also appear in pods. See https://github.com/argoproj/argo-workflows/blob/master/examples/pod-metadata-wf-field.yaml for an example. Is this what you need? Or are you looking for inheriting existing labels from workflow to pods?

[terrytangyuan]

If you need the latter, what is your use case? The workflow is the owner of your pods so you can also obtain the labels from the referenced workflow.

[hadim]

I was looking for a way to automatically set those labels from the labels of the workflow. But yeah I'll use podMetadata if that's not something you think is worse adding.

[terrytangyuan]

The labels workflows.argoproj.io/completed and workflows.argoproj.io/workflow are there for other purposes. I am not sure if it's necessary to inherit other labels/metadata as well since they can be obtained easily by owner reference. @alexec WDYT?

[ludydoo]

You can use a MutatingWebhookConfiguration to add labels to pods that have a Workflow owner reference

// main.go 
package main

import (
	"context"
	"flag"
	"fmt"
	"github.com/google/martian/log"
	"github.com/sirupsen/logrus"
	kwhhttp "github.com/slok/kubewebhook/v2/pkg/http"
	kwhlogrus "github.com/slok/kubewebhook/v2/pkg/log/logrus"
	kwhmodel "github.com/slok/kubewebhook/v2/pkg/model"
	kwhmutating "github.com/slok/kubewebhook/v2/pkg/webhook/mutating"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/apis/meta/v1"
	"net/http"
	"os"
	"strings"
	"time"
)

type config struct {
	certFile string
	keyFile  string
}

func initFlags() *config {
	cfg := &config{}

	fl := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
	fl.StringVar(&cfg.certFile, "tls-cert-file", "", "TLS certificate file")
	fl.StringVar(&cfg.keyFile, "tls-key-file", "", "TLS key file")

	if err := fl.Parse(os.Args[1:]); err != nil {
		err := fmt.Errorf("could not parse flags: %w", err)
		log.Errorf(err.Error())
		panic(err)
	}

	log.Infof("Using tls cert file at path: %s", cfg.certFile)
	log.Infof("Using tls key file at path: %s", cfg.keyFile)

	return cfg
}

func checkArgoWorkflowLabelNotPresent(pod *corev1.Pod) (*kwhmutating.MutatorResult, error) {
	if pod.Labels == nil {
		return &kwhmutating.MutatorResult{}, nil
	}
	if _, ok := pod.Labels["argo"]; ok {
		return &kwhmutating.MutatorResult{}, fmt.Errorf("forbidden: pod cannot have label argo=workflow as it is not owned by an Argo Workflow")
	}
	return &kwhmutating.MutatorResult{}, nil
}

func run() error {

	logrusLogEntry := logrus.NewEntry(logrus.New())
	logrusLogEntry.Logger.SetLevel(logrus.DebugLevel)
	logger := kwhlogrus.NewLogrus(logrusLogEntry)

	logger.Infof("running...")
	cfg := initFlags()

	logger.Infof("creating mutator func")
	var mf kwhmutating.MutatorFunc = func(ctx context.Context, review *kwhmodel.AdmissionReview, object v1.Object) (*kwhmutating.MutatorResult, error) {
		pod, ok := object.(*corev1.Pod)
		if !ok {
			return &kwhmutating.MutatorResult{}, nil
		}
		if pod.OwnerReferences == nil || len(pod.OwnerReferences) == 0 {
			return checkArgoWorkflowLabelNotPresent(pod)
		}
		var found = false
		for _, reference := range pod.OwnerReferences {
			if reference.Kind == "Workflow" && strings.Contains(reference.APIVersion, "argoproj.io/") {
				found = true
				break
			}
		}
		if !found {
			return checkArgoWorkflowLabelNotPresent(pod)
		}
		if pod.Labels == nil {
			pod.Labels = make(map[string]string)
		}
		currentValue, ok := pod.Labels["argo"]
		if ok && currentValue != "workflow" {
			return &kwhmutating.MutatorResult{}, fmt.Errorf("invalid label value argo='%s'", currentValue)
		}
		pod.Labels["argo"] = "workflow"
		return &kwhmutating.MutatorResult{MutatedObject: pod}, nil
	}

	logger.Infof("creating webhook config")
	mcfg := kwhmutating.WebhookConfig{
		ID:      "argo-workflow-labeler",
		Mutator: mf,
		Logger:  logger,
	}

	logger.Infof("creating webhook")
	wh, err := kwhmutating.NewWebhook(mcfg)
	if err != nil {
		return fmt.Errorf("error creating webhook: %w", err)
	}

	logger.Infof("creating http handler")
	whHandler, err := kwhhttp.HandlerFor(kwhhttp.HandlerConfig{Webhook: wh, Logger: logger})
	if err != nil {
		return fmt.Errorf("error creating webhook handler: %w", err)
	}


	// Serve.
	logger.Infof("Listening on :8080")
	err = http.ListenAndServeTLS(":8080", cfg.certFile, cfg.keyFile, whHandler)
	if err != nil {
		return fmt.Errorf("error serving webhook: %w", err)
	}

	return nil

}

func main() {

	log.Infof("sleeping a bit...")
	time.Sleep(5 * time.Second)

	log.Infof("starting app")
	err := run()
	if err != nil {
		log.Errorf("error running app: %v", err)
		time.Sleep(time.Second)
		os.Exit(1)
	}
}

# Dockerfile
FROM golang:1.16 as builder
WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download
COPY main.go main.go
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o argo-workflow-labeler main.go
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /workspace/argo-workflow-labeler /argo-workflow-labeler
USER nonroot:nonroot
CMD ["/argo-workflow-labeler"]

# Makefile
VERSION ?= latest
IMAGE ?= my-container-registry/argo-workflow-labeler

# Build the docker image
pack:
	docker build . -t ${IMAGE}:${VERSION}

# Push the docker image
push: pack
	docker push ${IMAGE}:${VERSION}

# Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argo-workflow-labeler-tls
  namespace: argo
spec:
  secretName: argo-workflow-labeler-tls
  dnsNames:
    - argo-workflow-labeler.argo.svc
    - argo-workflow-labeler.argo.svc.cluster.local
  commonName: argo-workflow-labeler.argo.svc
  issuerRef:
    name: selfsigned
---
# Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: argo
spec:
  selfSigned: {}
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: argo-workflow-labeler
  namespace: argo
spec:
  ports:
    - port: 443
      targetPort: 8080
      protocol: TCP
      name: webhook
  selector:
    app: argo-workflow-labeler
---
# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-workflow-labeler
  namespace: argo
---
# MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: argo-workflow-labeler
  labels:
    app: argo-workflow-labeler
    kind: mutator
  annotations:
    cert-manager.io/inject-ca-from: argo/argo-workflow-labeler-tls
webhooks:
  - name: argo-workflow-labeler.example.com
    admissionReviewVersions:
      - v1
    sideEffects: NoneOnDryRun
    clientConfig:
      service:
        port: 443
        name: argo-workflow-labeler
        namespace: argo
        path: "/mutate"
    objectSelector:
      matchExpressions:
        - key: app
          operator: NotIn
          values: [ argo-workflow-labeler ]
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [ "" ]
        apiVersions: [ "v1" ]
        resources: [ "pods" ]
---
# Deployment
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: argo-workflow-labeler
  namespace: argo
spec:
  host: argo-workflow-labeler.argo.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE
  exportTo:
    - "*"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-workflow-labeler
  namespace: argo
spec:
  selector:
    matchLabels:
      app: argo-workflow-labeler
  template:
    metadata:
      labels:
        app: argo-workflow-labeler
        control-plane: argo-workflows
      annotations:
        traffic.sidecar.istio.io/excludeInboundPorts: "8080"
    spec:
      serviceAccountName: argo-workflow-labeler
      containers:
        - name: argo-workflow-labeler
          image: digitalbackbonecr.azurecr.io/argo-workflow-labeler:latest
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          ports:
            - containerPort: 8080
              name: webhook
          args:
            - -tls-cert-file=/etc/webhook-certs/tls.crt
            - -tls-key-file=/etc/webhook-certs/tls.key
          securityContext:
            privileged: false
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1002
            runAsGroup: 1002
          volumeMounts:
            - mountPath: /etc/webhook-certs
              name: webhook-tls
      volumes:
        - name: webhook-tls
          secret:
            secretName: argo-workflow-labeler-tls

[stefansedich]

We ran into this issue and wanted a way for labels on templates to automagically get propogated from the template to the worklowMetadata/podMetadata.

Thought about it or a while and considered a PR to argo-workflows to add this support however I was able use Kyverno to create a mutating policy that would load the related template for a workflow and propogate all labels for us.

Ideally going forward our teams will set podMetadata/workflowMetadata but at least with this solution we can ensure the tags set only on templates today get set on the workflow pods themselves by default.

[FernandoArteaga]

using controller.workflowDefaults.podMetadata you can set labels or annotations to the Workflow Pod.

controller:
  workflowDefaults:
      podMetadata:
        labels:
          foo: "bar"

[kkrboyaa]

using controller.workflowDefaults.podMetadata you can set labels or annotations to the Workflow Pod.

controller:
  workflowDefaults:
      podMetadata:
        labels:
          foo: "bar"

This is doing exactly what I needed. After adding this now I have labels on all pods. If you want to have them both on the Workflow and on the Pods use a combination of podMetadata and metadata. Add this to your values.yaml:

controller:
  workflowDefaults:
    metadata:
      labels:
        foo: "bar"
    spec:
      podMetadata:
        labels:
          foo: "bar"