Cannot set sensitive VDK configuration like passwords

[antoniivanov]

What is the feature request? What problem does it solve? Currently, VDK doesn't provide a way to set sensitive configuration like passwords, such as trino_password. This can be problematic, especially for secure connections where sensitive information needs to be stored securely.

Although users can set configuration variables like db_default_type through environment variables and config files, sensitive data like passwords cannot be set in this manner.

This gap creates a security vulnerability that needs to be addressed. We need to find a way to allow users to securely set sensitive configuration like passwords in VDK.

Suggested solution Users can set their own sensitive values using vdk properties and vdk secrets but those are not recognized for configured required by VDK or VDK Plugins.

This is the approach taken for vdk-oracle plugin in https://github.com/vmware/versatile-data-kit/pull/2961

That works well enough that I think we should incorporate it for everything . Likely the change should be done here https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/vdk_config.py To have a config provider that reads form properties and secrets. secrets can have higher priority.

[sabadzhiev]

Triaged.

We will keep it, as being an feature that we would like to address as part of: #2420