Hide secrets in logs

[jjacobelli]

Summary

Currently, the secrets are printed in the logs without any filtering. It would be nice if the secrets could be redacted in the logs while being printed

Use Cases

This would prevent unintentional leaks of a secret in the log by running something like env in the workflow

NOTE: This does not prevent intentionally leaks of the secrets. Someone can still run something like env | base64 to extract the secrets from the logs, as the filtering will not be able to filter out the secret output.

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[alexec]

Could you please provide more info? Which secrets? Which logs?

[jjacobelli]

Could you please provide more info? Which secrets? Which logs?

Yes, sorry. I'm talking about the logs from the pods available using the UI or the CLI. I would like to be able to hide the Kubernetes secrets used in a Workflow. For example, currently if we use this example https://argoproj.github.io/argo-workflows/walk-through/secrets/, the output of $MYSECRETPASSWORD will be printed without any filtering in the logs. I would like to have this filtered out and the logs output would be something like secret from env: <redacted>. This only prevents unintentional credential leaks, but it's better than nothing IMHO

[alexec]

The redaction would be done on this line:

https://github.com/argoproj/argo-workflows/blob/92b3ef27af7a7e6b930045e95072a47c8745b1d3/util/logs/pods-logger.go#L60

Add a new environment variable ARGO_LOGS_REGSUB the operator can configure. This is a regsub that is applied to data to redact it.

Would you like to submit a PR?

[Daklon]

If no one is working on this, I could try to implement it. Is okay for you @Ethyling, @alexec ?

[jjacobelli]

If no one is working on this, I could try to implement it. Is okay for you @Ethyling, @alexec ?

Hey, yes, go ahead. I don't have much time to work on this currently

[deepto98]

Can I pick this up, if its still open?

[terrytangyuan]

@deepto98 Assigned to you.

[deepto98]

@terrytangyuan @alexec I'm working on this, have been trying to run the project locally using the VS Code remote container, following the docs here, but I keep getting this error when running make start:

deepto ➜ ~/go/src/github.com/argoproj/argo-workflows (master ✗) $ make start
GIT_COMMIT=18be9593e76bdeb456b5de5ea047a6aa8d201d74 GIT_BRANCH=master GIT_TAG=untagged GIT_TREE_STATE=dirty RELEASE_TAG=false DEV_BRANCH=false VERSION=latest
KUBECTX=k3d-k3s-default DOCKER_DESKTOP=false K3D=true DOCKER_PUSH=false
RUN_MODE=local PROFILE=minimal AUTH_MODE=hybrid SECURE=false STATIC_FILES=false ALWAYS_OFFLOAD_NODE_STATUS=false UPPERIO_DB_DEBUG=0 LOG_LEVEL=debug NAMESPACED=true
kubectl get ns argo || kubectl create ns argo
The connection to the server 0.0.0.0:38977 was refused - did you specify the right host or port?
The connection to the server 0.0.0.0:38977 was refused - did you specify the right host or port?
make: *** [Makefile:415: install] Error 1

Can you please help with this?

[terrytangyuan]

@tczhao Would you want to help take a look at the error above regarding dev container?

[tczhao]

@deepto98 the error looks like you have problem connect to k8s you can try

# to verify k8s connection
kubectl version
# to verify docker connection
docker images
# these should point you to the fix of your issue

It would be good if you can also provide which platform you are running it on (e.g. apple m1)

[terrytangyuan]

@deepto98 Any update?

[or-shachar]

This issue is still open and important IMO.

From digging in - I see that #9859 did not merge and was closed because it tried to read secrets directly from the server (see this comment).

I wonder if we can tackle this using the emissary worker / some wrapper script

Consider this (very initial):

#!/usr/bin/env python3
import os
import sys

def mask_env_vars(input_string):
    # Store the current environment variables in a dictionary
    env_vars = os.environ

    # Replace known strings with asterisks of the same length as the original string
    for key, value in env_vars.items():
        input_string = input_string.replace(value, "*" * len(value))
    return input_string

for line in sys.stdin:
  print(mask_env_vars(line))

And from there research how to:

• make sure that this script works in every system and is applied ot both stdout and stdin without too much of overhead (must)
• mask only subset of env variables, given by user input / pod metadata (if possible) etc...

WDYT?

[binnie268]

We also are using kubernetes secrets and am looking for a feature like this too in argo worklfows.

[mitendraanand]

We also are using secrets accessed via env var and am looking for a feature like this too in argo worklfows. Please prioritise.

[da-head0]

can you assign me this issue? @terrytangyuan

[terrytangyuan]

can you assign me this issue? @terrytangyuan

If you have a PR, feel free to submit. I don't want to block others who are interested in contributing when the assignee stopped working on a feature without any updates.