Enhancement: ability to easily specify alternate endpoints for services

[thetumper]

I'm using sceptre for deploying my cloudformation on AWS. Meanwhile, I use localstack for offline development. I'd like to use sceptre to deploy the same templates to localstack services, but have not found an obvious way to tell sceptre to use the local endpoints. It seems to be driven entirely by the AWS region, which is still "us-east-1" when running offline on localstack.

I'm guessing it may be possible to use alternate endpoints by specifying the endpoint_url when creating the boto client:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

However, this will get a bit verbose/complicated. It would be much simpler if sceptre could take an argument either specifying an endpoint configuration file, or directly specifying service endpoints, and then internally take care of using those alternate endpoints, so the caller does not have to handle creation of boto session and clients for each service. For example, an argument like this:

--endpoints=s3::http://localhost:4572,cloudformation::http://localhost:4581

[zaro0508]

Sceptre can refer to an AWS profile to execute it's commands therefore I think the root issue is that aws cli doesn't support a --endpoint-url config https://github.com/aws/aws-cli/issues/1270

That issue does have a reference to https://github.com/wbingli/awscli-plugin-endpoint which may help.

[thetumper]

Thanks @zaro0508 -- that looks very promising. I created a local profile, and used "aws configure" to set custom endpoints. E.g.:

aws configure --profile local set s3.endpoint_url http://localstack:4572

Then, in my cloudformation template config file, I added:

profile: local

I'm running this within a Docker container, and volume mounting my aws config directory, but seeing the following now:

myapp-function | [2020-02-18 21:51:35] - No cloudformation client found, creating one... myapp-function | [2020-02-18 21:51:35] - Getting Boto3 session myapp-function | [2020-02-18 21:51:35] - No Boto3 session found, creating one... myapp-function | [2020-02-18 21:51:35] - Using cli credentials... myapp-function | "The config profile (local) could not be found"

Not sure why it wouldn't be finding the profile. Any ideas on how to triage that?

[thetumper]

Figured out the mount issue -- on Mac, the HOME env variable is already set, and use/changing in docker-compose seems to not have the desired effect on the container mount path. Changing to hard-code the mount path yielded success in mounting the .aws directory.

Next issue: sceptre seems to pick up the config, but is not using the custom service endpoints. Realized, as others have noted on other issues, that the above-mentioned plugin only works with CLI, not with SDK or boto3. Sceptre uses the latter. It's a gap. Sceptre can either wait for AWS to add support via .aws/config, or do something itself to provide support for setting service endpoints in config, or via params to its own commands.

[ngfgrant]

I think it would be good to support this if boto3 makes it easy. The changes could be non breaking and but we need to make sure we get the endpoints set in the right place.

From the description it sounds like boto3 doesn’t make this easy/possible. I’d be strongly against trying to implement something that diverges from boto3.

The issues you are having with the route zaro suggested sound a lot like Docker and docker network issues.

[thetumper]

Yes, my issues with the above-mentioned plugin were partly due to how to get a mount set up in my Docker container, to make the .aws config available. But, that is not the core thing.

Seems that although the AWS CLI supports an --endpoint-url parameter, it is not configurable in the .aws config, nor would sceptre use it.

Ultimately, we just need a way to tell sceptre a local endpoint to use. As simple as a command line arg works for me. TerraForm offers it:

https://www.terraform.io/docs/providers/aws/guides/custom-service-endpoints.html#getting-started-with-custom-endpoints

Guessing it would be easy enough to add support, because boto3 also has the capability; see endpoint_url on the session:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

[zengqiangfang]

Hi, @zaro0508, I got the same issue like you. I want to sceptre using localstack to create cloudformation. But I don't know how to do the configuration. Could do u plz share your experience. Thx.

[jfalkenstein]

@zengqiangfang The place to start looking would be in the ConnectionManager class. This is the abstraction Sceptre uses to directly make boto3 calls.

[thetumper]

Looks like this is finally coming, from the AWS side. https://github.com/boto/boto3/issues/2099

[thetumper]

Was hoping that the recent changes (https://github.com/Sceptre/sceptre/pull/1300) may help with ability to pick up a local profile having local endpoints. Created with, e.g.:

aws configure --profile local set dynamodb.endpoint_url http://localstack:4566

[profile local]
sqs =
    endpoint_url = http://localstack:4566
sns =
    endpoint_url = http://localstack:4566
ssm =
    endpoint_url = http://localstack:4566
dynamodb =
    endpoint_url = http://localstack:4566

And, setting before invoking sceptre:

export AWS_PROFILE=local

But, still seeing:

"An error occurred (InvalidClientTokenId) when calling the DescribeStacks operation: The security token included in the request is invalid."

The reason being that the actual AWS us-east-1 region endpoint is being used, instead of the one from the local profile.

Shouldn't this be expected to work? Or does Sceptre just not pick up the endpoint values from AWS_PROFILE? If not, why not?

[quinnjr]

@thetumper aren't you missing the cloudformation endpoint in your config?

[thetumper]

@thetumper aren't you missing the cloudformation endpoint in your config?

Well, yes, in that most recent example. But the point is, sceptre doesn't use the endpoints in that profile (any of them).

[quinnjr]

@thetumper aren't you missing the cloudformation endpoint in your config?

Well, yes, in that most recent example. But the point is, sceptre doesn't use the endpoints in that profile (any of them).

I wouldn't expect it to. Sceptre is for Cloudformation.

[thetumper]

@thetumper aren't you missing the cloudformation endpoint in your config?

Well, yes, in that most recent example. But the point is, sceptre doesn't use the endpoints in that profile (any of them).

I wouldn't expect it to. Sceptre is for Cloudformation.

Yes, but Sceptre isn't picking it up from there for cloudformation either. As mentioned above, there's a noted inconsistency across various AWS SDKs and CLI, regarding use of config and env vars, or only by setting the endpoint as a param upon creating a client. That's detailed here:

https://github.com/boto/boto3/issues/2099

It seems like something Sceptre could provide its own capability for. Meanwhile, the solution design from AWS remains pending. This would not have to conflict with what they have pending; just establish a "priority" order so that when the default behavior of boto adds the new functionality from AWS, Sceptre users will understand where the Sceptre capability falls in pecking order.

[zaro0508]

The AWS CLI supports an endpoint_url parameter in ~/.aws/config file now. https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/

That should allow users to set an alternative endpoint in the AWS profile and allow users to pass the AWS_PROFILE to Sceptre.