Parameter overrides as a first class section in samconfig.toml

[benkehoe]

The parameter_overrides field in [default.deploy.parameters] consists of space-separated Key=Value pairs. While toml supports multiline strings, it causes an error because it does not process newlines as whitespace. Fixing that would be an improvement, but what I'd really prefer is that these key-value pairs get their own proper section. Say my samconfig.toml would look like:

version=0.1
[default.deploy.parameters]
stack_name = "my-app-stack"
s3_bucket = "my-source-bucket"

[default.deploy.parameters.template_parameters]
TemplateInput1 = "Value1"
TemplateInput2 = "Value2"

equivalent to

version=0.1
[default.deploy.parameters]
stack_name = "my-app-stack"
s3_bucket = "my-source-bucket"
parameter_overrides = "TemplateInput1=Value1 TemplateInput2=Value2"

One of the things this would make easier is updating the values in this file programmatically.

[diegogurpegui]

I like this idea.

It could also solve the current problem that we cannot have some default parameters to override in the toml file, and override some others from the command line. If you use "parameter-overrides" in the command line, it completely discards the ones in the toml file because the entire argument "paramter-overrides" gets replaced (that's what I saw). This is useful when you have some common environment parameters in the toml file, but you left secrets and other critical values for the command line to pass on runtime.

[benkehoe]

That is an excellent point!

[gannaramu]

I Tried this in the samconfig.toml file and it seems to work for me is this relevant to the issue you raised @benkehoe ?

parameter_overrides=["TemplateInput1=Value1", TemplateInput2=Value2"]

[benkehoe]

@gannaramu This does appear to work, and allows the key-value pairs to placed on separate lines.

parameter_overrides=[
  "TemplateInput1=Value1",
  "TemplateInput2=Value2"
]

However, I'd like to keep this issue open, as I think a proper section would be a better user experience, and fits with @diegogurpegui's idea that there should be a command line option to keep the parameters defined in the config file but selectively override/add to them.

[yiuc]

That is good idea. The parameter_overrides really hard to manage on IaC view

[omenking]

Even with three parameters it's too long.

version=0.1
[default.deploy.parameters]
parameter_overrides = "EvaluatorRuntime=ruby2.7 EvaluatorMemorySize=128 EvaluatorTimeout=10"

I like @benkehoe solution because as a developer its how I thought it would have worked

[elbayaaa]

@benkehoe Do you think a separate section is still a better experience? regarding @diegogurpegui 's idea, I believe the current experience can be improved to handle it. right?

[benkehoe]

@elbayaaa I do think a separate section is a better experience, because it's clear and concise. Additionally, @diegogurpegui's idea can't be implemented for the existing configuration method, because it would break backwards compatibility, but this new separate section could have different semantics.

[paulie4]

Also, if we want to use a multi-line string for the value, it would be a lot clearer if the key was separate from the value.

FYI, I'm thinking of using the samconfig.toml for the "Content" of an AWS::AppConfig::HostedConfigurationVersion, since that seems like it would be a lot cleaner than having the large JSON string in the template.yaml.

[rhbecker]

I found this issue looking for multiline options to improve the readability when differencing versions of the file, and I was excited to see the array option mentioned by @gannaramu. But then I ran a guided deploy and found that the cli reset my multiline array back to a single line. So, unless there are other options already available to allow and preserve multiline, then I'm +1 on this enhancement idea.

[mountHouli]

I agree that SAM needs better support for ingesting parameters.

For example, I think it should support a parameter file. In addition, SAM CLI doesn't have an easy, clean, usable way to ingest multi-line parameters. The lack of "easiness" and "cleanliness" is because doing sam deploy --parameter-overrides $(cat my-key-value-file.env) (which would be easy and clean) does not support multi-line parameters (nor does using a .json config file and jq rather than cat).

Practical example: I have a SAM stack that has an AWS::SecretsManager::secret, and so I need to pass a private RSA key in PEM format (i.e. multiline) into it (because that key has to go into my AWS::Serverless::Function). Need to pass a PEM (i.e. multiline) file into a SAM stack is a very common use case. It's frustrating that SAM CLI doesn't support this.

However, I am not a proponent of solving the problem via the samconfig.toml file.

Instead, I propose that --parameters-overrides accept a file URL prameter like --parameter-overrides file://a-key-value-file-or-a-json-file.env.or.json as proposed in https://github.com/aws/aws-sam-cli/issues/2054

...

Reason 1

In my opinion, SAM/CloudFormation stack configuration (in samconfig.toml) is not the same as application config, and I have reasons I don't want to manage them together.

I personally do want to commit to my repo a-key-value-and-or-json-file-that-is-an-example-of-the-env-vars-my-application-needs-with-real-values-that-are-not-secret-and-dummy-values-for-secrets-so-that-developers-can-use-this-file-and-sub-their-real-secrte-env-values.env.or.json.

However, I don't want to commit samconfig.toml to my repo.

Thus, I want these sets of configuration separate.

Thus, I am not a proponent of the change proposed above.

Reason 2

Along the same lines...

I am a proponent of the solution (and rationale) that was posted in thread https://github.com/aws/aws-sam-cli/issues/2054 on 4/5/2021 by @whereisaaron

Final Thought

If SAM CLI wants to solve the problem both via --parameter-overrides file://a-key-value-file-or-a-json-file.env.or.json (per https://github.com/aws/aws-sam-cli/issues/2054), AND via the samconflig.toml file as proposed above, I currently see no problem with that.

[benkehoe]

@mountHouli would you expect the parameter overrides file to have sections that correspond to the --config-env value? Or are you expecting that command might look like ENV=PROD sam deploy --config-env $ENV --parameter-overrides file://params-$ENV.json?

[mountHouli]

Hi @benkehoe!

Great question! I guess I hadn't thought about it.

Considering it now...

If the parameter overrides file was JSON, then (at first thought I'm having) something like this (as you hint at) would be great:

my-one-and-only-params-file.json

{
  "default": {
    "param1": "value 1",
    "param2": "value 2",
  },
  "development": {
    "param1": "some other value 1",
    "param2": "some other value 2",
  }
}

However, as I write it, I realize there are problems with this.

A. It wouldn't work with a file in plain old key/value format (i.e... my-key-value-parameters-file.env

# no way to specify environments here.
param1="value 1"
param2="value 2"

B. Even with the JSON part working like that, you'd still have to specify the --config-env option whenever you need to use something other than the default SAM env.

Thus, I think your suggestion is great! (And so it would look like pairing your suggested command with files like this...)

params-default.json
params-development.json
# and/or
params-default.env
params-development.env

What do you think?

[39otrebla]

We indeed ended up writing our own scripts that work exactly the same, more than a year ago. We are happy with the way how they do work from a configuration perspective, but it's cumbersome to replicate them on multiple projects so it'd be GREAT to have this built-in in SAM CLI.

We currently have a bunch of scripts in package.json:

{
  "sam:deploy:dev": "DEPLOYMENT_STAGE=dev ./scripts/deploy.sh"
  "sam:deploy:qa": "DEPLOYMENT_STAGE=qa ./scripts/deploy.sh"
  "sam:deploy:prod": "DEPLOYMENT_STAGE=staging ./scripts/deploy.sh"
}

The script deploy.sh searches the filesystem for a params.$DEPLOYMENT_STAGE.json file, parses it, and converts all parameters into the format accepted by SAM CLI via command-line (i.e. ParameterKey=....,ParameterValue=....).

So +1 for @mountHouli suggestion, I'm all in with it.

[turiya-fps]

parameter_overrides=["TemplateInput1=Value1", TemplateInput2=Value2"]

This is a definite improvement in readability, but for a parameter value that is an array it's still cumbersome due to the need for escape characters:

parameter_overrides = [
    "ALBSubnets=\"subnet-07f85fc46d44c163e, subnet-0d230a9e996276381\" ",
    "VpcId=vpc-06be20efc1ab7704c"
]

[Amir-Ahmad]

Here's another way of doing it that I've been using:

parameter_overrides = """\
    Project=Myproject \
    App=Myapp \
"""

[jsunico]

I got here with the same sentiment as diegogurpegui's comment. The workaround I found is to use --guided. It will allow me to retain or override parameters that are defined in the toml but at the same time ask me about parameters that are identified at runtime. It's still more tedious than having it in one command though so I also agree with this improvement/proposal.

[alessandrobologna]

I'd also add the the same issue is there with tags, especially the fact that specifying tags in the command line will discard the tags in the config. So, if i want to have the common, static tags in the config and dynamic ones in the command line, I can't.

[nidhikul23]

Hi @benkehoe! I found this comment trail while finding option to pass json file to --parameter-overrides in samconfig.toml file. Is this functionality already implemented? I am looking to retrieve parameters from seperate env specific file while consolidating everything (tag, parameters and other arguments to SAM CLI) in samconfig.toml file so that I don't have to write complex SAM CLI command.

Thanks. Nidhi

[specious]

For anyone who might find this useful, I'm currently working on an open source app that has a bash script which uses grep to get the saved values in the stack parameter overrides in samconfig.toml that works with the format they are currently stored in.

If this feature gets implemented, that code will need to be adjusted, but I'm certainly not complaining.

[dbaghdanov]

This is a definite improvement in readability, but for a parameter value that is an array it's still cumbersome due to the need for escape characters:

parameter_overrides = [
    "ALBSubnets=\"subnet-07f85fc46d44c163e, subnet-0d230a9e996276381\" ",
    "VpcId=vpc-06be20efc1ab7704c"
]

@novik-71 I worked around this with:

parameter_overrides = [
  "ALBSubnets=subnet-07f85fc46d44c163e,subnet-0d230a9e996276381",
  "VpcId=vpc-06be20efc1ab7704c"
]

[sriram-mv]

I would ask folks to also leave feedback at https://github.com/aws/aws-sam-cli/discussions/4591

[benkehoe]

SAM now supports YAML for the samconfig file. I wonder if YAML merge keys could help work around this issue. Note that I do not think config-file-language-level solutions are the answer here, SAM needs first-class support for combining parameter values coming from multiple independent sources

[AlexBurkey]

SAM needs first-class support for combining parameter values coming from multiple independent sources

This is a feature I definitely need. I'm trying to use some relatively static parameter overrides that are living in a samconfig.toml file but also a parameter that will change on each deploy (image tag) but it doesn't need to be stored in the samconfig file. Currently --parameter-overrides conflicts with using parameter_overrides in the file.