localstack icon indicating copy to clipboard operation
localstack copied to clipboard
verified publisher icon localstack

bug: API Gateway does not persist correctly when you restart the localstack docker container

Open bdowson opened this issue 1 year ago • 1 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Current Behavior

I have a working api gateway created with localstack. When I restart the container and try to query the same url, I get this message: {"message": "The API id '0e0cf92f' does not correspond to a deployed API Gateway API"}.

Details:

First I create my API and confirm it works:

$ awslocal apigatewayv2 get-apis
{
    "Items": [
        {
            "ApiEndpoint": "http://0e0cf92f.execute-api.localhost.localstack.cloud:4566",
            "ApiId": "0e0cf92f",
            "ApiKeySelectionExpression": "$request.header.x-api-key",
            "CorsConfiguration": {
                "AllowHeaders": [
                    "*"
                ],
                "AllowMethods": [
                    "*"
                ],
                "AllowOrigins": [
                    "*"
                ],
                "ExposeHeaders": [
                    "*"
                ]
            },
            "CreatedDate": "2024-10-16T05:24:49.452000+00:00",
            "DisableExecuteApiEndpoint": false,
            "Name": "XpedigoAPI_v2",
            "ProtocolType": "HTTP",
            "RouteSelectionExpression": "$request.method $request.path",
            "Tags": {},
            "Version": "2024-09-25 01:18:37UTC"
        }
    ]
}

$ awslocal apigatewayv2 get-stages --api-id=0e0cf92f
{
    "Items": [
        {
            "CreatedDate": "2024-10-16T05:24:49.524619+00:00",
            "DefaultRouteSettings": {
                "DetailedMetricsEnabled": false
            },
            "DeploymentId": "4d3d207f",
            "LastUpdatedDate": "2024-10-16T05:24:49.524619+00:00",
            "RouteSettings": {},
            "StageName": "local",
            "StageVariables": {
                "baseurl": "alb-localstack-bdowson.ngrok.io",
                "env": "local"
            },
            "Tags": {}
        }
    ]
}

$ awslocal apigatewayv2 get-deployments --api-id=0e0cf92f
{
    "Items": [
        {
            "AutoDeployed": false,
            "CreatedDate": "2024-10-16T05:24:49.529068+00:00",
            "DeploymentId": "4d3d207f",
            "DeploymentStatus": "DEPLOYED"
        }
    ]
}

Confirm it works:

$ curl -v https://0e0cf92f.execute-api.localhost.localstack.cloud:4566/local/accounts/health
* Trying 127.0.0.1:4566...
* TCP_NODELAY set
* Connected to 0e0cf92f.execute-api.localhost.localstack.cloud (127.0.0.1) port 4566 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost.localstack.cloud
*  start date: Sep  6 00:00:00 2024 GMT
*  expire date: Dec  5 23:59:59 2024 GMT
*  subjectAltName: host "0e0cf92f.execute-api.localhost.localstack.cloud" matched cert's "*.execute-api.localhost.localstack.cloud"
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5b8d78082650)
> GET /local/accounts/health HTTP/2
> Host: 0e0cf92f.execute-api.localhost.localstack.cloud:4566
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< server: TwistedWeb/24.3.0
< date: Wed, 16 Oct 2024 05:25:16 GMT
< content-type: text/html; charset=UTF-8
< cache-control: private, must-revalidate
< expires: -1
< pragma: no-cache
< x-powered-by: PHP/8.1.9RC1
< content-length: 2
< apigw-requestid: 5f9a3aa7
< 
* Connection #0 to host 0e0cf92f.execute-api.localhost.localstack.cloud left intact
OK

Now I stop localstack, and restart it with docker-compose up. The api gateway no longer works correctly:

$ curl -v https://0e0cf92f.execute-api.localhost.localstack.cloud:4566/local/accounts/health
*   Trying 127.0.0.1:4566...
* TCP_NODELAY set
* Connected to 0e0cf92f.execute-api.localhost.localstack.cloud (127.0.0.1) port 4566 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost.localstack.cloud
*  start date: Sep  6 00:00:00 2024 GMT
*  expire date: Dec  5 23:59:59 2024 GMT
*  subjectAltName: host "0e0cf92f.execute-api.localhost.localstack.cloud" matched cert's "*.execute-api.localhost.localstack.cloud"
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x6550ac6c5650)
> GET /local/accounts/health HTTP/2
> Host: 0e0cf92f.execute-api.localhost.localstack.cloud:4566
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 404 
< server: TwistedWeb/24.3.0
< date: Wed, 16 Oct 2024 05:29:09 GMT
< content-type: application/json
< content-length: 86
< 
* Connection #0 to host 0e0cf92f.execute-api.localhost.localstack.cloud left intact
{"message": "The API id '0e0cf92f' does not correspond to a deployed API Gateway API"}

But the configurations are all the same as before:

$ awslocal apigatewayv2 get-apis
{
    "Items": [
        {
            "ApiEndpoint": "http://0e0cf92f.execute-api.localhost.localstack.cloud:4566",
            "ApiId": "0e0cf92f",
            "ApiKeySelectionExpression": "$request.header.x-api-key",
            "CorsConfiguration": {
                "AllowHeaders": [
                    "*"
                ],
                "AllowMethods": [
                    "*"
                ],
                "AllowOrigins": [
                    "*"
                ],
                "ExposeHeaders": [
                    "*"
                ]
            },
            "CreatedDate": "2024-10-16T05:24:49.452000+00:00",
            "DisableExecuteApiEndpoint": false,
            "Name": "XpedigoAPI_v2",
            "ProtocolType": "HTTP",
            "RouteSelectionExpression": "$request.method $request.path",
            "Tags": {},
            "Version": "2024-09-25 01:18:37UTC"
        }
    ]
}

$ awslocal apigatewayv2 get-deployments --api-id=0e0cf92f
{
    "Items": [
        {
            "AutoDeployed": false,
            "CreatedDate": "2024-10-16T05:24:49.529068+00:00",
            "DeploymentId": "4d3d207f",
            "DeploymentStatus": "DEPLOYED"
        }
    ]
}

$ awslocal apigatewayv2 get-deployments --api-id=0e0cf92f
{
    "Items": [
        {
            "CreatedDate": "2024-10-16T05:24:49.524619+00:00",
            "DefaultRouteSettings": {
                "DetailedMetricsEnabled": false
            },
            "DeploymentId": "4d3d207f",
            "LastUpdatedDate": "2024-10-16T05:24:49.524619+00:00",
            "RouteSettings": {},
            "StageName": "local",
            "StageVariables": {
                "baseurl": "alb-localstack-bdowson.ngrok.io",
                "env": "local"
            },
            "Tags": {}
        }
    ]
}

Expected Behavior

API gateway should work correctly even after a localstack container restart.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

docker-compose.yml:

localstack:
    container_name: localstack
    image: localstack/localstack-pro:latest
    ports:
      - 4566:4566
      - 4510-4559:4510-4559
    environment:
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEBUG=1
      - PERSISTENCE=1
      - SNAPSHOT_LOAD_STRATEGY=ON_STARTUP
      - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY}
      - PROVIDER_OVERRIDE_APIGATEWAY=next_gen
    networks:
      app_network:
        ipv4_address: 10.0.2.20
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/localstack-data:/var/lib/localstack"
  1. docker-compose up localstack
  2. Import API Gateway with awslocal apigatewayv2 import-api --body file://t.json
  3. Create stage with awslocal apigatewayv2 create-stage --api-id 54ae753d --stage-name local --auto-deploy
  4. Confirm it works with curl -v https://[gateway url]/local/whatever
  5. Stop localstack
  6. Run docker-compose up localstack again
  7. Try and curl the api again and you will get an error

Environment

- OS: Ubuntu 20.04.5 LTS
- LocalStack: 
  LocalStack version: 3.8.2.dev33
  LocalStack Docker image sha: localstack/localstack-pro@sha256:b533e1bcfbe8f5462483725276a0e7f8fbd9ded32b1be2dac5ec9cee5e822023
  LocalStack build date: 2024-10-15
  LocalStack build git hash: 318e1adc

Anything else?

After this error appears, even if I delete the API and recreate it I still get the message {"message": "The API id 'xxxx' does not correspond to a deployed API Gateway API"}. The only way for me to resolve it is to delete my local locastack snapshot folder and rebuild everything.

bdowson avatar Oct 16 '24 05:10 bdowson

Hello @bdowson and thank you so much for your report!

I was able to determine the root cause of the issue, and will submit a fix shortly. I'll keep you updated once the fix is available.

bentsku avatar Oct 16 '24 20:10 bentsku

Hello @bdowson! The fix for the persistence issue has been merged, could you please give it a try and see if that resolves the issue? You can run docker pull localstack/localstack-pro:latest. Thanks!

bentsku avatar Oct 21 '24 16:10 bentsku

Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

localstack-bot avatar Nov 04 '24 17:11 localstack-bot