aws-sdk-go-v2
aws-sdk-go-v2 copied to clipboard
aws
503 error (api error Unavailable: The service is unavailable) when trying to create VpcEndpoint on new VpcEndpointService
Describe the bug
We are using the aws go sdk v2 (ec2) package to create a VpcEndoint. I we are working on a new VpcEndpointService, We encounter an a 503 error stating that the VpcEndpointService isn't ready yet. We tried checking the serviceState of the VpcEndpointService but we get 'ServiceStateAvailable' promptly after the creation. With that said - we are still receiving the 503 errors when trying to create the VpcEndpoint.
Is there any way to check that the service is ready before creating the VpcEndpoint?
Expected Behavior
Once creating the VpcEndpointService, I don't expect to receive 503 service unavailable messages when trying to create a VpcEndpoint. If the VpcEndpointService isn't ready - I would like to have some kind of (real) indication when it is indeed available. As stated above, the serviceState isn't reliable.
Current Behavior
At the moment, we are receiving 503 errors (service unavailable) after trying to create a VpcEndpoint on a newly created VpcEndpointService.
Reproduction Steps
Create a new VpcEndpointService
endpointServiceInput := &ec2.CreateVpcEndpointServiceConfigurationInput{
NetworkLoadBalancerArns: []string{*awsProviderLoadBalancerArn},
AcceptanceRequired: aws.Bool(false),
TagSpecifications: []ec2types.TagSpecification{Tags},
}
AwsEc2Client.CreateVpcEndpointServiceConfiguration(ctx, endpointServiceInput)
Create a new VpcEndpoint:
endpointResourceInput := &ec2.CreateVpcEndpointInput{
VpcId: edgeClusterVpcID,
ServiceName: endpointServiceName,
VpcEndpointType: ec2types.VpcEndpointTypeInterface,
SubnetIds: []string{*subnetID},
SecurityGroupIds: []string{*securityGroupID},
TagSpecifications: []ec2types.TagSpecification{Tags},
}
endpointResourceOutput, err := awsEc2Client.CreateVpcEndpoint(ctx, endpointResourceInput)
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2 v1.18.1 github.com/aws/aws-sdk-go-v2/service/ec2 v1.61.0
Compiler and Version used
go1.20.5
Operating System and version
darwin/amd64 - but runs in a kubernetes on docker image
Hi @Tyzanol ,
If I understand correctly the issue is that CreateVpcEndpoint fails even when the result of CreateVpcEndpointServiceConfiguration contains a serviceState = Available, but in fact the resource is not ready?
This sounds like a service-side limitation and not a particular issue with the SDK.
In services like EC2 you might encounter propagation delays, so even though the endpoint service itself might be ready, other transient resources that are needed to provision the service are not so it gives you a false positive of serviceState being available.
Can you please enable the request and response debug logs and share the logs here?
Here is how to enable logging:
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody))
I want to make sure there's nothing obviously wrong with the request, but following this I'll probably ask you to cut an internal ticket via the AWS console. That way you'll get assigned a Technical Account Manager that can drive this case internally on your behalf and investigate this with the service team.
Thanks, Ran~
This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi @RanVaknin , I have reproduced the issue and I'll like to share all the details so that you can continue the investigation. AccountID: <REDACTED> EndpointServiceID: <REDACTED> ServiceName: <REDACTED>
EndpointDetails: VpcID: <REDACTED> SubnetIds: <REDACTED> SecurityGroupIds: <REDACTED> All these resources where available before trying to create the endpoint resource.
RequestID: dc734f1c-d080-45c1-8b95-fe3f8c705b41
Let me know if you need any more details. Thanks, Daena
Hi @Tyzanol ,
I have redacted the resource information you provided since this is sensitive information that should only be shared with the service team internally. As I mentioned, this will likely get investigated by the service team itself, and not the SDK team. The SDK is a client, so we don't have access to logs / service side resources.
I asked to enable the logger in order to inspect the request and response logs so I can make sure the SDK doesn't send / receives corrupted data, or fails to serialize / deserializes it. Thus ruling out SDK specific behavior.
To me, the logical thing would have been to use serviceState as an indicator for when the resource is ready, but from the sounds of it, its not the case. This might be a limitation of the service itself (or a potential bug) and the process of investigating this will be for you to create a support ticket in your AWS console, and ask to be routed to the service team.
Once you provide the logs I'll have a bit more clarity about the problem at hand. (please make sure to redact any sensitive info from your logs like Credentials and Account information.
Thanks, Ran~
Hi @RanVaknin, Here are the logs of the Endpoint Service creation and the error we get when trying to create an Endpoint resource right away:
SDK 2023/08/30 14:19:00 DEBUG Request POST / HTTP/1.1 Host: ec2.eu-central-1.amazonaws.com User-Agent: aws-sdk-go-v2/1.18.1 os/macos lang/go/1.20.4 md/GOOS/darwin md/GOARCH/amd64 api/ec2/1.61.0 Content-Length: 128 Amz-Sdk-Invocation-Id: 6a45568a-3115-40a7-a709-a83e2ee3597e Amz-Sdk-Request: attempt=1; max=3 Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ3ZXGRO4SGHC52U4/20230830/eu-central-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=4f6690a398c57b09e3cafd018f46a8e2560f592c3b95577e92beede66aaea66e Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20230830T111900Z Accept-Encoding: gzip
Action=DescribeVpcEndpointServiceConfigurations&Filter.1.Name=tag%3AName&Filter.1.Value.1=ugw-michal-rl-aws-c&Version=2016-11-15 SDK 2023/08/30 14:19:00 DEBUG Response HTTP/1.1 200 OK Transfer-Encoding: chunked Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:18:59 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: accept-encoding X-Amzn-Requestid: 63657c0a-abbc-45a9-806b-3c292dbb03e2
7e1
SDK 2023/08/30 14:19:01 DEBUG Request POST / HTTP/1.1 Host: ec2.eu-central-1.amazonaws.com User-Agent: aws-sdk-go-v2/1.18.1 os/macos lang/go/1.20.4 md/GOOS/darwin md/GOARCH/amd64 api/ec2/1.61.0 Content-Length: 172 Amz-Sdk-Invocation-Id: 6aa9929c-fc6e-4da9-87ef-780da8fb5521 Amz-Sdk-Request: attempt=1; max=3 Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ3ZXGRO4SGHC52U4/20230830/eu-central-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=1c14ea61d50b6f3e3035ba890872e3c01652bda096836dde6c90048a8424ed16 Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20230830T111901Z Accept-Encoding: gzip
Action=ModifyVpcEndpointServicePermissions&AddAllowedPrincipals.1=arn%3Aaws%3Aiam%3A%3A678183734201%3Arole%2FAWS-IAM&ServiceId=vpce-svc-006ccfab24f741253&Version=2016-11-15 SDK 2023/08/30 14:19:01 DEBUG Response HTTP/1.1 200 OK Transfer-Encoding: chunked Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:00 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: accept-encoding X-Amzn-Requestid: ec906309-48c5-4aee-aabb-6a6d0b6c4355
26d
{"level":"info","ts":"2023-08-30T14:19:01.398+0300","caller":"ratelimit/awsSetup.go:392","msg":"vpc endpoint service ID: com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253"} SDK 2023/08/30 14:19:02 DEBUG Request POST / HTTP/1.1 Host: ec2.eu-central-1.amazonaws.com User-Agent: aws-sdk-go-v2/1.18.1 os/macos lang/go/1.20.4 md/GOOS/darwin md/GOARCH/amd64 api/ec2/1.61.0 Content-Length: 108 Amz-Sdk-Invocation-Id: b7265412-f862-431a-8898-a0333a6c57f7 Amz-Sdk-Request: attempt=1; max=3 Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ3ZXGRO4SGHC52U4/20230830/eu-central-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=34579f8d7ee40550a74560cb9b2c527cde73b5869df448c6853a11b9bd335098 Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20230830T111902Z Accept-Encoding: gzip
Action=DescribeVpcs&Filter.1.Name=tag%3AName&Filter.1.Value.1=shoot--ugw-michal--rl-aws-e&Version=2016-11-15 SDK 2023/08/30 14:19:02 DEBUG Response HTTP/1.1 200 OK Content-Length: 1341 Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:01 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Amzn-Requestid: 067cbf8f-8869-4656-b8f6-a85bf0a0c3d1
Action=DescribeSubnets&Filter.1.Name=tag%3AName&Filter.1.Value.1=shoot--ugw-michal--rl-aws-e-nodes-z0&Version=2016-11-15 SDK 2023/08/30 14:19:02 DEBUG Response HTTP/1.1 200 OK Content-Length: 1871 Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:02 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Amzn-Requestid: 249e1d71-6dcb-4854-8d90-1181a8d01de4
Action=DescribeSecurityGroups&Filter.1.Name=tag%3AName&Filter.1.Value.1=ugw-michal-rl-aws-c-rl-aws-e&Version=2016-11-15 SDK 2023/08/30 14:19:02 DEBUG Response HTTP/1.1 200 OK Content-Length: 1876 Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:02 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Amzn-Requestid: 21787987-e00d-4f9c-86c8-f73ebf21dd47
Action=DescribeVpcEndpoints&Filter.1.Name=tag%3AName&Filter.1.Value.1=ugw-michal-rl-aws-c-rl-aws-e&Version=2016-11-15 SDK 2023/08/30 14:19:02 DEBUG Response HTTP/1.1 200 OK Transfer-Encoding: chunked Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:02 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: accept-encoding X-Amzn-Requestid: 4a369b9e-2927-4d65-9eb6-84cb6e87911a
ec
{"level":"info","ts":"2023-08-30T14:19:02.825+0300","caller":"ratelimit/awsSetup.go:332","msg":"start creating Endpoint resource"} SDK 2023/08/30 14:19:02 DEBUG Request POST / HTTP/1.1 Host: ec2.eu-central-1.amazonaws.com User-Agent: aws-sdk-go-v2/1.18.1 os/macos lang/go/1.20.4 md/GOOS/darwin md/GOARCH/amd64 api/ec2/1.61.0 Content-Length: 461 Amz-Sdk-Invocation-Id: a65e7773-65e6-40c3-9a9e-92c071f9b467 Amz-Sdk-Request: attempt=1; max=3 Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ3ZXGRO4SGHC52U4/20230830/eu-central-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=471ff393099f951335dc9113b36e0b78dc5a55769967d0b4b2d12317a39cd63e Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20230830T111902Z Accept-Encoding: gzip
Action=CreateVpcEndpoint&SecurityGroupId.1=sg-0820367be7c04a8bb&ServiceName=com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253&SubnetId.1=subnet-0ece697f9d5c58eb6&TagSpecification.1.ResourceType=vpc-endpoint&TagSpecification.1.Tag.1.Key=Name&TagSpecification.1.Tag.1.Value=ugw-michal-rl-aws-c-rl-aws-e&TagSpecification.1.Tag.2.Key=Landscape&TagSpecification.1.Tag.2.Value=rl-aws-c&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-0cca0eb40f93494a2 SDK 2023/08/30 14:19:03 DEBUG Response HTTP/1.1 503 Service Unavailable Connection: close Transfer-Encoding: chunked Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:02 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: accept-encoding X-Amzn-Requestid: 23e09378-9ae0-4236-9bf2-db8ff946eb5e
f7
<Response><Errors><Error><Code>Unavailable</Code><Message>The service is unavailable. Please try again shortly.</Message></Error></Errors><RequestID>23e09378-9ae0-4236-9bf2-db8ff946eb5e</RequestID></Response> 0
SDK 2023/08/30 14:19:04 DEBUG Request POST / HTTP/1.1 Host: ec2.eu-central-1.amazonaws.com User-Agent: aws-sdk-go-v2/1.18.1 os/macos lang/go/1.20.4 md/GOOS/darwin md/GOARCH/amd64 api/ec2/1.61.0 Content-Length: 461 Amz-Sdk-Invocation-Id: a65e7773-65e6-40c3-9a9e-92c071f9b467 Amz-Sdk-Request: attempt=2; max=3 Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ3ZXGRO4SGHC52U4/20230830/eu-central-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=962a1505fad6fd549c8a9b40510e96e94e9d7e2eda1a022463eb2e1219b7f428 Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20230830T111904Z Accept-Encoding: gzip
Action=CreateVpcEndpoint&SecurityGroupId.1=sg-0820367be7c04a8bb&ServiceName=com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253&SubnetId.1=subnet-0ece697f9d5c58eb6&TagSpecification.1.ResourceType=vpc-endpoint&TagSpecification.1.Tag.1.Key=Name&TagSpecification.1.Tag.1.Value=ugw-michal-rl-aws-c-rl-aws-e&TagSpecification.1.Tag.2.Key=Landscape&TagSpecification.1.Tag.2.Value=rl-aws-c&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-0cca0eb40f93494a2 SDK 2023/08/30 14:19:05 DEBUG Response HTTP/1.1 503 Service Unavailable Connection: close Transfer-Encoding: chunked Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:04 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: accept-encoding X-Amzn-Requestid: 24632107-60ab-40b4-990c-1f3ee3266550
f7
<Response><Errors><Error><Code>Unavailable</Code><Message>The service is unavailable. Please try again shortly.</Message></Error></Errors><RequestID>24632107-60ab-40b4-990c-1f3ee3266550</RequestID></Response> 0
SDK 2023/08/30 14:19:08 DEBUG Request POST / HTTP/1.1 Host: ec2.eu-central-1.amazonaws.com User-Agent: aws-sdk-go-v2/1.18.1 os/macos lang/go/1.20.4 md/GOOS/darwin md/GOARCH/amd64 api/ec2/1.61.0 Content-Length: 461 Amz-Sdk-Invocation-Id: a65e7773-65e6-40c3-9a9e-92c071f9b467 Amz-Sdk-Request: attempt=3; max=3 Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ3ZXGRO4SGHC52U4/20230830/eu-central-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=3c50cae9a90ea4780c158c99e7f9a33edaac5a6c65e5c1fe7a5994fd269c49c5 Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20230830T111908Z Accept-Encoding: gzip
Action=CreateVpcEndpoint&SecurityGroupId.1=sg-0820367be7c04a8bb&ServiceName=com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253&SubnetId.1=subnet-0ece697f9d5c58eb6&TagSpecification.1.ResourceType=vpc-endpoint&TagSpecification.1.Tag.1.Key=Name&TagSpecification.1.Tag.1.Value=ugw-michal-rl-aws-c-rl-aws-e&TagSpecification.1.Tag.2.Key=Landscape&TagSpecification.1.Tag.2.Value=rl-aws-c&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-0cca0eb40f93494a2 SDK 2023/08/30 14:19:09 DEBUG Response HTTP/1.1 503 Service Unavailable Connection: close Transfer-Encoding: chunked Cache-Control: no-cache, no-store Content-Type: text/xml;charset=UTF-8 Date: Wed, 30 Aug 2023 11:19:08 GMT Server: AmazonEC2 Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: accept-encoding X-Amzn-Requestid: fe76542d-1655-4eff-871d-38c705fb4ee4
f7
<Response><Errors><Error><Code>Unavailable</Code><Message>The service is unavailable. Please try again shortly.</Message></Error></Errors><RequestID>fe76542d-1655-4eff-871d-38c705fb4ee4</RequestID></Response> 0
Hi @rayakoren,
I'm not sure why are you running into this. I'm able to create the necessary resources and the VPC endpoint:
package main
import (
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/ec2"
"log"
)
func main() {
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody))
if err != nil {
log.Fatalf("unable to load SDK config, %v", err)
}
client := ec2.NewFromConfig(cfg)
vpcResp, err := client.CreateVpc(context.TODO(), &ec2.CreateVpcInput{
CidrBlock: aws.String("11.0.0.0/16"),
})
if err != nil {
log.Fatalf("failed to create VPC, %v", err)
}
_, err = client.CreateSubnet(context.TODO(), &ec2.CreateSubnetInput{
CidrBlock: aws.String("11.0.1.0/24"),
VpcId: vpcResp.Vpc.VpcId,
})
if err != nil {
log.Fatalf("failed to create subnet, %v", err)
}
sgResp, err := client.CreateSecurityGroup(context.TODO(), &ec2.CreateSecurityGroupInput{
GroupName: aws.String("MySecurityGroup2"),
Description: aws.String("My security group2"),
VpcId: vpcResp.Vpc.VpcId,
})
if err != nil {
log.Fatalf("failed to create security group, %v", err)
}
_, err = client.CreateVpcEndpoint(context.TODO(), &ec2.CreateVpcEndpointInput{
VpcId: vpcResp.Vpc.VpcId,
ServiceName: aws.String("com.amazonaws.us-east-1.s3"),
VpcEndpointType: "Interface",
SecurityGroupIds: []string{*sgResp.GroupId},
})
if err != nil {
log.Fatalf("failed to create VPC endpoint, %v", err)
}
}
My logs:
SDK 2023/10/03 09:28:37 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 59
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162837Z
Accept-Encoding: gzip
Action=CreateVpc&CidrBlock=11.0.0.0%2F16&Version=2016-11-15
SDK 2023/10/03 09:28:37 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 914
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:37 GMT
Server: AmazonEC2
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Amzn-Requestid: [REDACTED]
<?xml version="1.0" encoding="UTF-8"?>
<CreateVpcResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<vpc>
<vpcId>vpc-03d742b7c97aea389</vpcId>
<ownerId>[REDACTED]</ownerId>
<state>pending</state>
<cidrBlock>11.0.0.0/16</cidrBlock>
<cidrBlockAssociationSet>
<item>
<cidrBlock>11.0.0.0/16</cidrBlock>
<associationId>vpc-cidr-assoc-[REDACTED]</associationId>
<cidrBlockState>
<state>associated</state>
</cidrBlockState>
</item>
</cidrBlockAssociationSet>
<ipv6CidrBlockAssociationSet/>
<dhcpOptionsId>dopt-02e990d0c6df32328</dhcpOptionsId>
<instanceTenancy>default</instanceTenancy>
<isDefault>false</isDefault>
</vpc>
</CreateVpcResponse>
SDK 2023/10/03 09:28:37 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 90
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162837Z
Accept-Encoding: gzip
Action=CreateSubnet&CidrBlock=11.0.1.0%2F24&Version=2016-11-15&VpcId=vpc-03d742b7c97aea389
SDK 2023/10/03 09:28:38 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 1302
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:37 GMT
Server: AmazonEC2
Strict-Transportation-Security: max-age=31536000; includeSubDomains
X-Amzn-Requestid: [REDACTED]
<?xml version="1.0" encoding="UTF-8"?>
<CreateSubnetResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<subnet>
<subnetId>subnet-0ed487786a0539fea</subnetId>
<subnetArn>arn:aws:ec2:us-east-1:[REDACTED]:subnet/subnet-0ed487786a0539fea</subnetArn>
<state>available</state>
<ownerId>[REDACTED]</ownerId>
<vpcId>vpc-03d742b7c97aea389</vpcId>
<cidrBlock>11.0.1.0/24</cidrBlock>
<ipv6CidrBlockAssociationSet/>
<availableIpAddressCount>251</availableIpAddressCount>
<availabilityZone>us-east-1a</availabilityZone>
<availabilityZoneId>use1-az4</availabilityZoneId>
<defaultForAz>false</defaultForAz>
<mapPublicIpOnLaunch>false</mapPublicIpOnLaunch>
<assignIpv6AddressOnCreation>false</assignIpv6AddressOnCreation>
<privateDnsNameOptionsOnLaunch>
<hostnameType>ip-name</hostnameType>
<enableResourceNameDnsARecord>false</enableResourceNameDnsARecord>
<enableResourceNameDnsAAAARecord>false</enableResourceNameDnsAAAARecord>
</privateDnsNameOptionsOnLaunch>
<ipv6Native>false</ipv6Native>
<enableDns64>false</enableDns64>
</subnet>
</CreateSubnetResponse>
SDK 2023/10/03 09:28:38 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 136
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request,
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162838Z
Accept-Encoding: gzip
Action=CreateSecurityGroup&GroupDescription=My+security+group2&GroupName=MySecurityGroup2&Version=2016-11-15&VpcId=vpc-03d742b7c97aea389
SDK 2023/10/03 09:28:38 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 283
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:38 GMT
Server: AmazonEC2
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Amzn-Requestid: [REDACTED]
<?xml version="1.0" encoding="UTF-8"?>
<CreateSecurityGroupResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<return>true</return>
<groupId>sg-070e9304b4b5052a8</groupId>
</CreateSecurityGroupResponse>
SDK 2023/10/03 09:28:38 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 175
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162838Z
Accept-Encoding: gzip
Action=CreateVpcEndpoint&SecurityGroupId.1=sg-070e9304b4b5052a8&ServiceName=com.amazonaws.us-east-1.s3&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-03d742b7c97aea389
SDK 2023/10/03 09:28:38 DEBUG Response
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:38 GMT
Server: AmazonEC2
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: accept-encoding
X-Amzn-Requestid: [REDACTED]
<CreateVpcEndpointResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<vpcEndpoint>
<creationTimestamp>2023-10-03T16:28:38.795Z</creationTimestamp>
<dnsEntrySet/>
<dnsOptions>
<dnsRecordIpType>ipv4</dnsRecordIpType>
</dnsOptions>
<groupSet>
<item>
<groupId>sg-070e9304b4b5052a8</groupId>
<groupName>MySecurityGroup2</groupName>
</item>
</groupSet>
<ipAddressType>ipv4</ipAddressType>
<networkInterfaceIdSet/>
<ownerId>[REDACTED]</ownerId>
<privateDnsEnabled>false</privateDnsEnabled>
<requesterManaged>false</requesterManaged>
<routeTableIdSet/>
<serviceName>com.amazonaws.us-east-1.s3</serviceName>
<state>pending</state>
<subnetIdSet/>
<vpcEndpointId>vpce-02f257d4a7f27fe01</vpcEndpointId>
<vpcEndpointType>Interface</vpcEndpointType>
<vpcId>vpc-03d742b7c97aea389</vpcId>
</vpcEndpoint>
</CreateVpcEndpointResponse>
I'm not using describeSubnets at all. Im not even sure that is the reason why the resource is failing. A 503 was not given with any information about why the creation is failing.
You can check out my code, it is all done in a synchronous fashion, so once subnet is created, the vpc creation code will get called.
Let me know if this helps. Thanks, Ran~
This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.
@RanVaknin where are you creating a vpcEndpointService? We came across the issue when creating a vpcEndpoint that is bound to a new vpcEndpointService. What is your VpcEndpoint bound to? I see you are using a hardcoded service name - was this created previously?
Hi there,
Sorry for the long wait.
I see you are using a hardcoded service name - was this created previously?
It has been a while since I last attempted to reproduce this, but I believe I did create this ahead of time.
In my current reproduction I called createVpcEndpoint after I have awaited the state of the endpoint service and see that the state was indeed Available even though it wasn't - resulting in the same 503.
This is a service API issue, and not an issue with the SDK. The VPC service team did not define a waiter to await the status of the endpoint service, but even after writing my own waiter, its evident that the VPC server returns a false positive.
I have created an internal ticket with the VPC service in order to fix this problem (ticket id #P147369190).
Will let you know once we hear back.
Thanks, Ran~