Render / Inspect validate compose content

[WTFKr0]

Hi, we are facing an issue chen using docker-app Today we use compose file with env vars in, like :

version: "3.6"
services:
  web:
  image: myimage;tag
  replicas: ${WEB_REPLICAS}

And we deploy with env vars, like : WEB_REPLICAS=3 docker stack deploy ... With that, our deployment tool can easily grab variables on the compose file and look for values in our key/value store for the target envrionment (prod/saging/other)

We want to reproduce same thing with docker-app, but the inspect/render command need a valid docker-compose file, and if we push a docker-app with replicas: ${WEB_REPLICAS}, it's not valid

Our deployment tool is unable to list keys that are needed for deployement in that case. I think this works well in v0.3, but now in v0.6 a validate step is done in render/inspect command

How can we bypass this validate ?

Thanx

[silvin-lubecki]

Hello @WTFKr0 , thank you for filling this issue.

docker-app variable substitution is mainly based on settings. So when it validates the application package, it checks if every variable can be substituted using at least the default values in the settings.yml file. (Note that this will be renamed to parameters.yml in a near future when #438 will be merged). But you can still give another settings file at runtime for render or inspect commands using the -f flag

$ docker-app inspect myapp -f dev-settings.yml

You can also override only one setting using --set flag:

$ docker-app inspect myapp --set my.setting=80

With this I propose 2 solutions:

1. Use the --set flag instead of an env var:

$ docker-app inspect yourapp --set WEB_REPLICAS=$WEB_REPLICAS

I think this solution can be straightforward for you, without modifying too much things for you.

2. Generate a prod settings from your key/value store. It's just a YAML key/value file:

$ cat prod-settings.yml
WEB_REPLICAS: 3

It may need more work from you but it feels more the docker-app way to do it 😄

Does this answer your question?

[WTFKr0]

Thanx for reply, yeah i know about all that, my main problem is my deployment tool need to know the list of variables that need to be set. Here it seems to be mandatory to set a default value to a var. In my use case, I would like to have variables without default value, and be able to do a inspect to list them. This was possible in older version.

For context, we have like ~30 target environments, and a 500+ key/values per envrionment. So wh have to inject in docker-app only vars related to our app, not the 500+ :smile:

[silvin-lubecki]

Ok thank you for explaining all the context ! But reading that, maybe all what you need is a way to use programmatically the result of a docker-app inspect ?

Something like this?

$ docker-app inspect myapp --quiet --section=parameters
my.param1
my.service.param2
other.service2.param
...

--quiet would only show the names, and --section=parameters would only list parameters section? What do you think about that? @garethr I think we already talked about that, do you agree with this?

[silvin-lubecki]

Or maybe just docker-app inspect myapp --format=json | jq ... ?

[WTFKr0]

Hum I think you are missing one of my needs, here is a simplified compose i use today :

version: "3.6"
services:

  web:
    image: nginx:alpine
    deploy:
      replicas: ${REPLICAS_OF_MY_APP}

Then our CI/CD system detect those vars, check values in our key/value store for the target env, then deploy with : REPLICAS_OF_MY_APP=5 docker stack deploy -c docker-compose.yml mystack

With docker-app i can : Create a docker app like that :

version: 0.0.1
name: template
description: 
maintainers:
  - name: Me
    email: [email protected]
targets:
  swarm: true
  kubernetes: true

---

version: "3.6"
services:

  web:
    image: nginx:alpine
    deploy:
      replicas: $${VAR_REPLICAS}

---

I double the $$ so that it should be rendered with one I can push this docker app on our registry without problem

BUT

I can't use it When i tried to render/inspect this app, the compose validation tell me that replicas should be an int All i can do is fork that app, then deploy my compose like before... but it's a bit tricky

[silvin-lubecki]

Maybe I'm missing some important points, but let's say we add a feature to let you list all the variables, for example:

$ docker-app inspect myapp --format=json | jq --raw-output '.Parameters|keys[]'
VAR_REPLICAS

But to be able to do that, you need to at least add some default values in the settings.yml file:

version: 0.0.1
name: template
description: 
maintainers:
  - name: Me
    email: [email protected]
targets:
  swarm: true
  kubernetes: true

---

version: "3.6"
services:

  web:
    image: nginx:alpine
    deploy:
      replicas: ${VAR_REPLICAS}

---
VAR_REPLICAS:1

Then can your CI/CD system use this list to query your key/value store and then create a new settings file?

You will then have something like this:

$ docker-app inspect myapp --format=json | jq --raw-output '.Parameters|keys[]' | my-ci-cd-system-kvstore > prod-settings.yml
$ docker-app render myapp -f prod-settings.yml | docker stack deploy -c - mystack

Does that make sense?

[WTFKr0]

Well i can already do that, but i want to avoid the "default" value, as i prefer the deployment fail if not set. I don't want deployment use a default value. Furthermore, all my vars are set 2 times in the file, in the compose.yml and in the parameters.yml

I'm testing a solution without docker app too cause i think i don't need the settings thing The unique feature of docker-app that i use is finally the ability to store a compose in the registry i think...

[simonferquel]

I think you can declare a parameter with VAR_REPLICAS: null or something similar and it should do what you want.