Make it work with docker/k8s secrets

[janober]

Hello,

this library is really great and saved me a lot of time and headache. One issue I however still have is if I want to use it with docker/k8s secrets. If they get used the "secret-data" is made available to the container via q special file. For that reason do many containers have now additional sets of environment variables. The regular one which contains the "secret-data" directly and another one which simply supplies the path to the secret-file.

For examples the postgres container has the environment variables:

• POSTGRES_PASSWORD
• POSTGRES_PASSWORD_FILE

It would be great if convict would support something similar. That simply some kind of key could be added to the configuration which says something like "supportsFile" and if it is set to "true" it will read in the data from the env "_FILE" variable (in case it is present) and saves it in the not "_FILE" version.

But guess it would not be so easy or "pretty" as it would mean that "convict" either would have to become async or it would have to use blocking calls to still work like before (which is maybe not the worst considering that it would just have to happen once on startup).

blue skies

Jan

[janober]

Ok did now write a simple wrapper for my project to do that. It can be found here: https://github.com/n8n-io/n8n/commit/13c5f99a7e68786b517b84b4bfd15fe5d10fe041#diff-dd741574890cbb6247da67e8296021edR64

Not sure what you prefer. To integrate that functionality somehow in convict, that you create an own module which wrapps convict or you simply do not care about that functionality at all. Then I can just create a wrapper module.

[brettneese]

@janober you can also leverage the config.load() method: https://github.com/mozilla/node-convict#configloadobject. I'll send you a more full fledged example when I get a chance to write one up.

[janober]

Hi Brett! Thanks a lot for your answer! Am however not sure how that would help me exactly with the problem I am facing. Guess the example later will make it clear. Have a great day!

[zaverden]

@brettneese I'm not sure if config.load() can be efficiently applied in this case since it requires to duplicate info about config structure.

@janober I have built loadFromEnvFiles(config) function. And here is an example of usage.

Several points:

• it checks *_FILE for ALL config fields with env key
• if depends on private field config._env - may not work in future versions if internal structure will be changed
• if *_FILE exists it will override command line argument value (for build-in cases command line arguments has priority

[A-312]

I watch @zaverden example, I think convict should be try to load file if schema.file is defined. We should add file loader in the schema parser like we doing for : env, arg.

    name: {
      doc: 'Database name',
      format: String,
      default: 'users',
      file: '/mycustompath/togetmy/name'
    }

Or allow custom getter :

const schema = {
  db: {
    name: {
      doc: 'Database name',
      format: String,
      default: 'users',
      getter: 'file',
      file: '/mycustompath/togetmy/name'
    }
  }
}

// convict.addGetter(name, key, priority, getter)
convict.addGetter('file', 'file', 70, (filepath) => { return readFile(filepath) })

An we should allow to set custom priority level : https://github.com/mozilla/node-convict/blob/master/lib/convict.js#L600-L601

Priority level	Getter
0	Default value
20	File (config.loadFile())
40	Environment variables (only used when env property is set in schema; can be overridden using the env option of the convict function)
60	Command line arguments (only used when arg property is set in schema; can be overridden using the args option of the convict function)
80	Set and load calls (config.set() and config.load())

[brettneese]

@zaverden Why? Just create an object with your schema and pass it to the function in load(). IE:

let conf = convict(configSchema).load(provider(configSchema));

@janober Here you go. I can't imagine it would be too difficult to hack the code @zaverden provided into the "provider" function in my example and as a bonus, this could be distributed as its own npm module. I'm working on one for the AWS Param store but if I get a chance I might consider packaging that up.

@A-312 Not sure that would help in this case, as the issue is that in this case is that the filename itself is an environmental variable. Although I suppose you could do:

const schema = {
 db: {
   name: {
     doc: 'Database name',
     format: String,
     default: 'users',
     getter: 'file',
     file: process.env.POSTGRES_PASSWORD_FILE
   }
 }
}

I'm also not sure we should add more code to convict core if similar functionality could be provided via functions injected into load() as it's already pretty gnarly.

I do however love the idea of custom getters as the issue with injecting "providers" via load() is that the values are static. With custom getters we could achieve a decent, pluggable external provider infrastructure that's also dynamic. For instance, I could rotate DB secrets every 5 minutes without a care in the world as every time convict would load the freshest secret.

[zaverden]

@brettneese thank you for interesting idea with loader. I will try to make something useful from it, but at the moment I see one big downside:

If provider gets whole schema object then I have to implement deep-walk logic (which is not a big problem) and make it consistent with convict logic (more specific: how to check that I don't need to go deeper). It would be great if convict could expose this logic, but it will require some refactoring in convict code.

Anyway, your approach works fine for specific project when I know schema structure

[brettneese]

@zaverden No problem! :)

I'm actually working through that problem right now. It probably doesn't make sense to expose that via Convict itself, but it might make sense to pull that part of the code out and into it's own little npm module for people building custom providers like this to use.

[brettneese]

Hey @zaverden! I'm still working on documentation, testing, and releasing this as its own NPM module, but I thought you might want to have a peak here: https://github.com/brettneese/convict-provider-awsSsm/blob/use-native-convict/lib/convictUtils.js

In essence, I pulled a few pieces out of Convict itself to help with parsing deep objects the Convict way, and created a helper library which you pass a schema, a property (on the Convict config schema itself) to be passed to a getter function, also passed, something like this:

const utils = require("./convictUtils");

let schema = {
  foo: {
    bar: {
      baz: {
        default: "fred",
        format: "String",
        file: "/foo/bar/baz"
      }
    },
    baz: {
      default: "fred",
      format: "String",
      file: "/foo/baz"
    }
  }
};

let getValue = function(path) {
  return readFile(path);
};

return utils.getValues(schema, "file", getValue);

and it'll properly loop over your Convict schema, including deeply nested objects, and apply the getter function to each key (passing it the value from the "property," ie, a file path.)

The cool thing about this is that it'll actually allow us to hopefully easily share logic between providers should we get custom getters, getValues essentially just applies the same custom getter to the entire schema (at boot-time, instead of dynamically, but still.)

[A-312]

I think node-convict 6, answer to this issue