Secrets references on stack component attributes
Describe changes
This PR implements a generic way to reference secrets for (most) stack component attributes.
Instead of passing in the attribute value directly, stack component attributes now allow a reference to a secret in the following syntax: {{secret_name.secret_key}}. The secret_name is the name of a ZenML secret, and the secret_key selects one of the secrets key-value pairs.
Resolving the secret values happens only when someone directly accesses the attribute (by calling value = component.attribute). There are many reasons why we decided to have the resolving as late as possible:
- Currently the secrets manager is part of the stack, so just creating a stack component in isolation doesn't give us information about how to resolve the secrets.
- Not every user that instantiates stack components (e.g. by calling
Repository().active_stack) will have read access to the secrets manager, and therefore resolving will fail for them. - Resolving secrets can be quite slow.
Another idea worth considering would be to even prevent secret resolving all together if someone wants to access these values in an environment without secrets manager access.
Limitations:
- It only works when the component is part of the active stack: A stack component can be part of many stacks, and currently a secrets manager is associated with a stack. This means we're not able to identify the 'correct' secrets manager that the user wanted to resolve the secrets in a general way.
- It only works when the active stack contains a secrets manager
- It only works for string attributes: Handling other data types would require us to dive deep into pydantics internals to handle this in a general way.
- Secret references are not allowed for attribute that are being validated using a pydantic
@validator. Again this would require code modification inside pydantic which we decided is not worth it at the moment, as we use@validatorsonly in very few cases which are not particularly security-relevant.
How to use it
The following commands show how to reference a secret in your stack component (and a convenient way to register them):
zenml alerter register my_alerter --flavor=slack --slack_token={{slack_secret.token}}
zenml stack update --alerter my_alerter
# The following command registers values for all secrets specified in a stack
# (Make sure the stack has a secrets manager at this point)
zenml stack register-secrets
Secret validation when validating a stack
During stack validation, we now additionally validate the existence of secrets that are referenced in the attributes of all components of the stack. This validation comes in two forms:
- When registering or updating a stack, the secrets aren't required and we only log a warning if they don't exist.
- When running a pipeline, the secrets are required and an exception is raised if they don't exist.
The validation above might fail on local machines where the user does not have read access on the secret manager.
To handle this case, we introduce an environment variable ZENML_SECRET_VALIDATION_LEVEL which can be used to control how ZenML validates secrets:
- No validation is happening.
- We check just the existence of secrets with the given name. This might be useful if users have permissions to list secrets, but not read the secret values.
- We check the existence of secrets and that a value for the required key exists.
Secret Fields
This PR also adds a utility function SecretField which is a drop-in replacement for a pydantic.Field and specifies that this field contains sensitive information. This is currently only used to warn users that they're specifying sensitive information in plain-text. Once we get to a point that the secrets manager is outside of the stack and required, we should probably raise an exception instead to enforce security best practices.
Pre-requisites
Please ensure you have done the following:
- [x] I have read the CONTRIBUTING.md document.
- [x] If my change requires a change to docs, I have updated the documentation accordingly.
- [ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
- [x] I have added tests to cover my changes.
Types of changes
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Other (add details above)
@AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
@AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
@AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
@AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
@AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
@stefannica, @wjayesh, @AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
I just wanted to say, even before I start reviewing the code changes, that I absolutely love how you designed this, based only on the PR description. I particularly appreciate how much thought you put in the validation part.
@stefannica, @wjayesh, @AlexejPenner 1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.
Nice ❤️