[BUG] Secrets mounted as environment variables disappear after fast registration

[matheusMoreno]

Describe the bug

Secrets that are mounted as environment variables are unavailable to tasks that require them if a fast registration process occurs. Secrets mounted as files do not appear to be affected by this.

Expected behavior

Any type of secret should be readily available for a tasks that requests it, even after a fast registration process. It doesn't make sense to rebuild a image every time just because the tasks require secrets as environment variables.

Additional context to reproduce

Consider the following minimal project:

.
├── Dockerfile             # The usual Dockerfile for Flyte tasks
├── app
│   ├── __init__.py
│   └── workflows.py
└── requirements.txt    # Only one requirement: flytekit>=0.30.3

The file app/workflows.py is defined as:

from flytekit import task, workflow, Secret, current_context

env_secret = Secret(group='test-group', key='test-env', mount_requirement=Secret.MountType.ENV_VAR)
file_secret = Secret(group='test-group', key='test-file', mount_requirement=Secret.MountType.FILE)

@task(secret_requests=[env_secret, file_secret])
def hello_world() -> str:
    secrets = current_context().secrets
    secret_from_file = secrets.get('test-group', 'test-file')
    secret_from_env = secrets.get('test-group', 'test-env')
    return f"Hello world, these are my secrets: {secret_from_env} / {secret_from_file}"

@workflow
def run_hello_world() -> str:
    response = hello_world()
    return response

Start the sandbox, build the image, create the secrets, and register the tasks:

flytectl sandbox start --source .
flytectl sandbox exec -- docker build . --tag "secret-test:v1"
kubectl create secret generic test-group \
    --from-literal=test-env=TESTING_ENV \
    --from-literal=test-file=TESTING_FILE \
    --namespace flyteexamples-development
pyflyte --pkgs app package --image "secret-test:v1" --force
flytectl register files --project flyteexamples --domain development --archive flyte-package.tgz --version v1

If we go to the console and run the workflow, everything works as expected:

    o0:
    Hello world, these are my secrets: TESTING_ENV / TESTING_FILE

Now, let's change the hello_world task by simply adding a print() statement:

@task(secret_requests=[env_secret, file_secret])
def hello_world() -> str:
    print("Get ready: this task will fail!")    # Added line
    secrets = current_context().secrets
    secret_from_file = secrets.get('test-group', 'test-file')
    secret_from_env = secrets.get('test-group', 'test-env')
    return f"Hello world, these are my secrets: {secret_from_file} / {secret_from_env}"

If we do a fast registration of the package and register it again, i.e.,

pyflyte --pkgs app package --image "secret-test:v1" --force --fast
flytectl register files --project flyteexamples --domain development --archive flyte-package.tgz --version v2

the task now fails:

Traceback (most recent call last):

      File "/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 203, in user_entry_point
        return wrapped(*args, **kwargs)
      File "/root/app/workflows.py", line 12, in hello_world
        secret_from_env = secrets.get('test-group', 'test-env')
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/context_manager.py", line 380, in get
        raise ValueError(

Message:

    Unable to find secret for key test-env in group test-group in Env Var:_FSEC_TEST-GROUP_TEST-ENV and FilePath: /etc/flyte/secrets/test-group/test-env

User error.

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[matheusMoreno]

I just found another bug while trying the workaround of mounting secrets as files. This bug is actually on flytekit, but since it's not possible to open issues there, I'll post it here.

Secrets are being mounted the same way as they are defined, but flytekit tries to retrieve them using unnecessary preprocessing. For instance, if I create a Secret named secret-group with a key MY_SECRET_KEY, the secret is mounted at /etc/flyte/secrets/secret-group/MY_SECRET_KEY. But, on flytekit, the secret is retrieved by doing a key.lower(), and the key cannot be retrieved, even when is mounted:

https://github.com/flyteorg/flytekit/blob/2ca5607d6b313fa9d6aeb88a807febcefcea7cdb/flytekit/core/context_manager.py#L310-L315

I know the key is mounted because I used print(os.listdir(<path-to-secret>)) to verify that it is. My workaround for now is reading the key directly from the file, and not using the context manager.

[kumare3]

@matheusMoreno thank you for the bug. This is weird that it happens only on fast register. Cc @wild-endeavor @pingsutw

[matheusMoreno]

Hi! Any updates on this? I could help, but I really don't know where to start. Should I look at FlytePropeller? Which component is responsible for mounting secrets?

[wild-endeavor]

already DM'ed but can't repro the main issue. will keep this open for now though, want to investigate the flytekit casing issue as well.

[wild-endeavor]

Tested this again (using K8s secrets). Still could not reproduce the issue. But I did run into an old issue - the env var mount will not work if there are "-" in the group name. Shouldn't be different between fast register and non fast register though.

We should fix the - issue some time too.

[hamersaw]

closing as not reproducible, if this is incorrect please re-open with a minimal repro using latest flyte (1.10+).