Securing Sensitive Pipeline Configuration Data

[cmanning09]

Is your feature request related to a problem? Please describe. Currently, Data Prepper pipeline configurations are loaded from a static plain text file. Sensitive configuration data (ie. usernames and passwords) can be stored in pipeline configurations.

Describe the solution you'd like I would like to support to load sensitive data from secure locations. Some examples are:

• Docker secrets
• A key manager
• others

Ideally, this feature would be supported through plugins allowing the community to build their own as well.

Describe alternatives you've considered (Optional) A clear and concise description of any alternative solutions or features you've considered.

Additional context This came out of a discussion from: #947

[mike-geiger]

This is a pretty big issue for my team. One option would be if we could reference a file from the config files. Then we can mount secrets into the container. eg:

keyStorePassword: !file /path/to/secret

[dlvenable]

@mike-geiger , Thank you for commenting on this issue. It helps to know what users are needing to prioritize issues.

The solution we use should allow users to use existing fields. The syntax here would work with that.

An alternative would be to support loading properties from a secret file and then access those properties, similar to what was discussed in #947.

[elvince]

Hi,

@dlvenable Have you been able to make some progress on this?

We are very interested in such a system to secure the sensitive data out of the config file.

Thanks,