parameter-overrides on command line block samconfig.toml values

[mousedownmike]

Description:

I have most of my parameter_overrides defined in my samconfig.toml file for my local commands. However, I want to pass a single value via the command line --parameter-overrides value. The documentation states:

For the parameter_overrides entry, both the parameter values that you provide on the command line and entries in the configuration file take precedence over corresponding objects declared in the Parameters section of the template file.

This would indicate to me that you can provide both and they'll be merged but this doesn't appear to be happening.

Steps to reproduce:

samconfig.toml

version = 0.1
[default.local_invoke.parameters]
parameter_overrides = "myFuncBucket=\"dev-my-func\"" 

Make the following call:

sam local invoke MyFunc --parameter-overrides 'ParameterKey=myFuncQueue,ParameterValue=QueueValue'

Observed result:

The value of myFuncQueue is passed to the function but the value of myFuncBucket is not.

Expected result:

Both myFuncQueue and myFuncBucket should be available to the function.

or...

The documentation should be updated to reflect that passing a command line parameter-overrides will prevent the use of the parameter_overrides.

I can make that documentation update if that is the case. However, I'm hoping that this is a bug.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

$ cat /etc/issue
Ubuntu 20.04.1 LTS \n \l
$ sam --version
SAM CLI, version 1.7.0
$ go version
go version go1.15.4 linux/amd64

[qingchm]

Hey @mousedownmike I followed your steps and was able to get the same results! However, by design we have command line arguments overriding the toml configs, can you specify which use case you need to have parameter override in both toml file and command line? Sorry for the inconvenience, but unless you have a use case that strongly feels the need for those two parts to be merged, we will update the docs to clarify on this instead of merging the toml input and command line input! Hope this helps!

[mousedownmike]

@qingchm, I don't think my use case would be a strong driver for implementing this functionality but here are the basic bullet points.

• My samconfig.toml contains parameter_override values that cover 90% of my local test cases.
• To avoid excessive resource consumption when testing one function, I want to change one parameter.
• As we've seen overriding that one parameter on the command line isn't possible.

I probably don't understand TOML enough but I thought I could create some type of inheritance in my configuration but that doesn't seem possible.

I ended up creating a separate TOML environment for this one case. Unfortunately all the other overrides have to be duplicated and the TOML syntax for them isn't great... but it IS just a copy paste/update so it's definitely survivable.

[wchengru]

Hello @mousedownmike, If you provide both parameter-overrides in command line, and parameter_overrides in toml config. The one in the command line take precedence over the one in toml config. This rule applies to all parameters as shown in the doc. In your case, 'ParameterKey=myFuncQueue,ParameterValue=QueueValue' from command line will replace the whole string from toml. If you want to use command line to change parameter_override, you need to put all of the key-value pairs from toml, and change the one you need. Also, I do agree that merging toml with command line for parameter_override is a good feature request.

[mousedownmike]

Hi @wchengru , I don't believe the documentation clearly reflects the fact that a parameter-overrides command line argument completely replaces all of the values in the samconfig.toml parameter_overrides value. You linked to the .cn docs so maybe they're different but what I reference in my initial ticket doesn't seem that clear:

For the parameter_overrides entry, both the parameter values that you provide on the command line and entries in the configuration file take precedence over corresponding objects declared in the Parameters section of the template file.

Something like the following might be more clear:

For the parameter_overrides entry, the parameter values that you provide on the command line or the entries in the configuration file take precedence over corresponding objects declared in the Parameters section of the template file. If parameter-overrides are provided on the command line, none of the values in the the configuration file will be used.

[qingchm]

@mousedownmike hey I agree that our documents on this can be improved! But one thing I don't understand about your use case is that why does that 90% of test cases needs to use parameter-override instead of just specifying the parameters in the template? Doesn't setting a parameter default like this and having the rest 10% to use parameter override in toml file work for you? Glad to hear back from you! Parameters: InstanceTypeParameter: Type: String Default: t2.micro

[wchengru]

@mousedownmike Sorry about the wrong link, the .cn link is identical to the .com one and your document citation is correct. We can definitely improve the documentation with more details. Thanks for your suggestions!

[mousedownmike]

@qingchm, for the specific thing I was trying to achieve, the default template value results in about 250,000 records being processed. This is normal but it takes time. 10% of the time, I want to use a smaller sample set so I just want to change the one value to point to a test data set. I was hoping to do that without duplicating the other parameters that aren't changing.

[qingchm]

@mousedownmike I see! will take your case to start a discussion with the team, I'll let you know if we think it's a good move to merge in command line parameters into toml base!

[softprops]

My use case is only a subset of values require dynamic resolution which would require cli provided values but a majority are static values in which case the samconfig.toml file offers a great convenience as it really shortens my command line to deploy.

+1 for merging values parameter overrides specifically and let conflicting keys be resolved by preferring the command line overrides

[SaurabhGarg1]

Also ran into this, the documentation really needs correction here. My usecase is -

All default parameters are in config.toml and then pass a another parameter which will vary according the environment of deployment e.g. Env=dev or Env=prod

in that case, it would be really good if both params are merged

[SaurabhGarg1]

btw, this need of different parameters on cli and config.toml can be solved by using --config-env or --config-file

more info here : https://github.com/aws/aws-sam-cli/pull/2176 https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

I think this issue can be closed.

[mousedownmike]

Please don't close this issue. The referenced issue (#2176) does not resolve the underlying problem of not being able to consistently override individual paramters.

[kyler-hyuna]

I just came across this. It would be great to have samcofig.toml parameter_overrides merged with the cli values.

In my samconfig.toml I used parameter_overrides to specify staging/prod environments:

version = 0.1
[staging]
[staging.deploy]
[staging.deploy.parameters]
stack_name = "boom-staging"
parameter_overrides = "Environment=\"staging\""

[prod]
[prod.deploy]
[prod.deploy.parameters]
stack_name = "boom-prod"
parameter_overrides = "Environment=\"prod\""

This makes it super easy to do a deploy script by just passing --config-env in ci. However I also pass in values to my stack that are generated at deploy time, so I pass these in via --parameter-overrides.

For example:

sam deploy --config-env staging --parameter-overrides LastUpdated=$(date '+%Y-%m-%d')

Since --parameter-overrides via the cli overrides the samconfig.toml, the settings defined there are basically useless and I have to type them over again.

---

So what should happen should be a merge, with the command-line taking preference.

[softprops]

I thought of a work around in my case but I don’t like it.

In my case I have various config envs that represent env configuration parameter overrides that are static and a single parameter that needs to be dynamic as it’s value is produced in a deployment pipeline.

I can leverage parameter store as a transport between the command line and my template by populating the ssm parameter before calling sam deploy and reading it through my templates ability to read ssm parameters.

Why I don’t like this.

• I’m using a network service as a transport passing variable for a command line program!
• Ssm costs money so I’m literally paying the price for a work around for not have a better alternative with Sam cli

[softprops]

The idea of dyanamic value resolution got me thinking. This goes a bit into the territory Serverless framework went with config variables but what would be the level of interest in the ability to dynamically resolve data within a samconfig.toml file in addition to defining static values

straw man syntax borrowing from Serverless framework for env and cli resolvers

${variableSource, default}

[prod.deploy.parameters]
parameter_overrides=[
 "Static=value",
 "EnvDynamic=${env:VARIABLE_NAME, defaultValue}"
]

sam deploy --config-env prod # EnvDynamic resolves to 'defaultValue'

VARIABLE_NAME=override sam deploy --config-env prod # EnvDynamic resolves to 'override'

pros:

• provides a solution to the problem outlined in this issue without the drawbacks of merging parameters as is which might create ambiguity in which samconfig fields are mergeable and which aren't and why
• safe and clear fallback to static values
• extensible, if env resolvers end out not being the best way to resolve dynamic values you can reuse the syntax for others

cons:

• toml file is more that a toml file. it's a toml file template
• inventing syntax. you need to know more than toml to understand what goes on in these files albeit not a completely new idea for cloudformation
• adds complexity to sam cli

[softprops]

Opting for now to go low tech

sed "s/{{RELEASE}}/$(RELEASE)/g" samconfig.template.toml > samconfig.toml 

[dominusmi]

I'm having the same issue, where I have a config-env for each environment, but part of the parameters are generated at runtime. The expected behaviour is merge and I was quite surprised to see it overwrites the config-env entirely. I think this is a really relevant feature and should be strongly considered.

[pappnase99]

Just ran into this issue. From the documentation I also assumed specific parameters I want to override on the command line are to be merged with parameters from the config file that don't need to be changed. Some effort to resolve this issue would be greatly appreciated.

[solarmosaic-kflorence]

I also ran into this issue, assuming that the command line arguments would override the values already defined in the configuration. The documentation is not clear that it will override everything. I also need to pass some values in at runtime, the rest are static.

[solarmosaic-kflorence]

Using cat samconfig.template.toml | envsubst | tee samconfig.toml as a workaround for now. The samconfig.toml file is now listed in .gitignore.

[solarmosaic-kflorence]

Another good workaround is here https://github.com/aws/aws-sam-cli/issues/2054#issuecomment-762550286 -- except that this will not support falling back to default values.

[eugene-bright]

I'd like to see sam overriding parameters from the configuration file only on parameter name clash Please increase the priority of the ticket @mousedownmike could you rework the issue title the way it would describe desired behavior?

[shearn89]

Another use case from me: being able to inject or override a single parameter from some other data source, be it a secret store, some dynamic account info, a build tag, etc., without having to re-specify all parameters.

This merging behaviour could also apply to tags: for example, my customer has a big list of mandatory tags to apply to the environment. From a support perspective, if I could have all my deployments inject a build-ref tag that had the Git short ref or tag, I'd be able to more easily tell which version of a stack is deployed and debug issues.

For parameters: injection of a secret without hard-coding it into the repo would be nice (and yes, I'm aware of the implication of using it as a parameter, caveats around NoEcho, etc. etc. I'd just like the option...)

[rovellipaolo]

One more use case here: most of my parameters are static, depending on the environment (which I pass via --config-env arg), but couple of them requires dynamic resolution (e.g., the current version of a object in an S3 bucket). It would be really handy to be able to pass in the --parameter-overrides arg only those couple of dynamic parameters.

[domvo]

One more usecase: we have a few functions that run on both dev and prod. Most of the configuration options are public/open and static. I want to pass some secrets via --parameter-overrides and nothing else. This declutters our CI/CD files.

[tomjowitt]

Our use case is that we have specific runtime parameters that need to be overridden in the CI system such as the build tag version. There is currently no way to add these to our config without using sed.

[cortexcompiler]

I don't understand why this has not had more focus, perhaps it is too hard to do. I think that the parameter overrides passed in through the CLI individually overriding the samconfig.toml values (which have lower precedence) makes the most sense intuitively and matches similar configuration schemes developers are familiar with. In addition to that the documentation still makes it sound like that is how it would work. I have a similar use case where most of the parameters can be overridden in the samconfig.toml, but one is determined dynamically at deploy time (in this case the ECR registry ID). This cannot be hard coded per environment in the config file as it is possible the account for the environment could change.

[mew1033]

I just ran into this again. I would really love to have the ability to merge cli parameters with samconfig parameters.

[ramakrishnanh]

I ran into this today. My use-case is to pass the role ARNs as parameter overrides to be used for the Serverless functions from the parent CloudFormation which deploys the whole pipeline stack. Would love to have the command line parameter overrides merged with the ones defined on samconfig file.

[YangusKhan]

I ran into this problem today. For my use case, I have a set of nested CloudFormation stacks that all take mostly the same static parameter values, except for one parameter which is intended to be a unique string prefix that gets added to things like Lambda and Step Function names. Rather than duplicating the config environment block in toml and changing this prefix string, it would be much more elegant to simply pass in a different prefix parameter via SAM CLI --parameter-overrides.