500 error from serverless on deployment

[whummer]

trafficstars

Copied from: https://github.com/localstack/localstack/issues/855 Koalk

---

Can't seem to create tickets on the new project, feel free to copy this into it once issues are enabled on it. I am trying to deploy a local version of 2 lambdas plus a stepfunction, all quite basic stuff. I get a 500 error from the serverless side of the deployment and quite the amount of logs in the localstack side. I'll put both here in case they are of any use. Localstack logs Serverless logs serverless.yml

Is suppose to be a basic "lambda one loads new data somewhere and lambda two does some processing on it" with the step function as a glue between them and failover.

┆Issue is synchronized with this Jira Task by Unito

[mpfeil]

I am getting the following 500 error when trying to deploy to localstack:

...
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 400 0.054s 0 retries] describeStacks({ StackName: 'myService-local' })
Serverless: Creating Stack...
Serverless: [AWS cloudformation 500 61.999s 3 retries] createStack({ StackName: 'myService-local',
  OnFailure: 'DELETE',
  Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
  Parameters: [ [length]: 0 ],
  TemplateBody:
   '{"AWSTemplateFormatVersion":"2010-09-09","Description":"The AWS CloudFormation template for this Serverless application","Resources":{"ServerlessDeploymentBucket":{"Type":"AWS::S3::Bucket","Properties":{"BucketEncryption":{"ServerSideEncryptionConfiguration":[{"ServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}}}},"Outputs":{"ServerlessDeploymentBucketName":{"Value":{"Ref":"ServerlessDeploymentBucket"}}}}',
  Tags: [ { Key: 'STAGE', Value: 'local' }, [length]: 1 ] })
Serverless: Recoverable error occurred (500), sleeping for 5 seconds. Try 1 of 4
Serverless: [AWS cloudformation 500 64.294s 3 retries] createStack({ StackName: 'myService-local',
  OnFailure: 'DELETE',
  Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
  Parameters: [ [length]: 0 ],
  TemplateBody:
   '{"AWSTemplateFormatVersion":"2010-09-09","Description":"The AWS CloudFormation template for this Serverless application","Resources":{"ServerlessDeploymentBucket":{"Type":"AWS::S3::Bucket","Properties":{"BucketEncryption":{"ServerSideEncryptionConfiguration":[{"ServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}}}},"Outputs":{"ServerlessDeploymentBucketName":{"Value":{"Ref":"ServerlessDeploymentBucket"}}}}',
  Tags: [ { Key: 'STAGE', Value: 'local' }, [length]: 1 ] })
Serverless: Recoverable error occurred (500), sleeping for 5 seconds. Try 2 of 4
...

My serverless.yml looks like this:

service: myService

provider:
  name: aws
  runtime: nodejs6.10
  memory: 128
  region: us-east-1

plugins:
  - serverless-localstack

custom:
  localstack:
    host: http://localhost

functions:
  hello:
    handler: handler.hello

I have started localstack with docker-compose:

version: '2.1'

services:
  localstack:
    image: localstack/localstack
    ports:
      - "4567-4593:4567-4593"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=serverless,cloudformation,iam
      - DEBUG=1
      - LAMBDA_EXECUTOR=docker
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

@whummer do you have any idea and can help me?

[whummer]

@mpfeil This should be fixed in https://github.com/localstack/localstack/pull/1237 - can you try pulling the latest version of the localstack/localstack Docker image?

@Koalk - can you also give it a try with the latest version and confirm whether the issue you reported still persists?

[mpfeil]

@whummer thanks for your feedback. I have pulled the latest version and the original error is gone. But now it runs into a socker hang up error.

Here is the serverless log:

Serverless: Using serverless-localstack
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 400 0.085s 0 retries] describeStacks({ StackName: 'myService-dev' }) Serverless: Creating Stack...
Serverless: [AWS cloudformation 200 0.367s 0 retries] createStack({ StackName: 'myService-dev',
  OnFailure: 'DELETE',
  Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
  Parameters: [ [length]: 0 ],
  TemplateBody:
   '{"AWSTemplateFormatVersion":"2010-09-09","Description":"The AWS CloudFormation template for this Serverless application","Resources":{"ServerlessDeploymentBucket":{"Type":"AWS::S3::Bucket","Properties":{"BucketEncryption":{"ServerSideEncryptionConfiguration":[{"ServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}}}},"Outputs":{"ServerlessDeploymentBucketName":{"Value":{"Ref":"ServerlessDeploymentBucket"}}}}',
  Tags: [ { Key: 'STAGE', Value: 'dev' }, [length]: 1 ] })
Serverless: Checking Stack create progress...
Serverless: [AWS cloudformation 200 0.023s 0 retries] describeStackEvents({ StackName:
   'arn:aws:cloudformation:us-east-1:123456789:stack/myService-dev/0df5ed0e-f3ca-43af-9217-43dc37817320' })
..
Serverless: Stack create finished...
Serverless: [AWS cloudformation 200 0.016s 0 retries] describeStackResource({ StackName: 'myService-dev',
  LogicalResourceId: 'ServerlessDeploymentBucket' })
Serverless: [AWS s3 200 0.032s 0 retries] listObjectsV2({ Bucket:
   'myService-dev-ServerlessDeploymentBucket-7A-Z9889A-ZA-Z3797',
  Prefix: 'serverless/myService/dev' })
Serverless: [AWS sts undefined 0.317s 3 retries] getCallerIdentity({})
Serverless: Recoverable error occurred (socket hang up), sleeping for 5 seconds. Try 1 of 4
Serverless: [AWS sts undefined 0.328s 3 retries] getCallerIdentity({})
Serverless: Recoverable error occurred (socket hang up), sleeping for 5 seconds. Try 2 of 4
Serverless: [AWS sts undefined 0.436s 3 retries] getCallerIdentity({})
Serverless: Recoverable error occurred (socket hang up), sleeping for 5 seconds. Try 3 of 4
Serverless: [AWS sts undefined 0.324s 3 retries] getCallerIdentity({})
Serverless: Recoverable error occurred (socket hang up), sleeping for 5 seconds. Try 4 of 4
Serverless: [AWS sts undefined 0.435s 3 retries] getCallerIdentity({})

  Serverless Error ---------------------------------------

  socket hang up

Update: It is working if I provide the services one by one inside the docker-compose.yml:

environment:
	- SERVICES=s3,lambda,cloudformation,sts,iam,cloudwatch

If I use a combination of serverless,cloudformation,iam,sts,cloudwatch the services are not getting ready. Also turning on DEBUG=1 in combination with sts in the services a lot of empty log messages appear within the docker logs.

[Koalk]

I won't be able to test it out for a while @whummer As soon as I get access to a computer I'll check it out. Thank you.

[whummer]

@mpfeil forgot to mention - you'll also need to add sts to your list of SERVICES (as you already pointed out in your comment). At some point we may include that as part of the serverless composite service, but for now it has to be specified manually.

Glad it's working when you specify the services directly. I'll look into the other issue with empty log messages etc shortly.

[nealeu]

Hi,

Is the issue here simply that you don't haven't specified ?

  localstack:   
     stages: 
       - local

so the plugin isn't active?

I've just spent some time scratching my head and stepping through the debugger, so I've created #22 to alert the user.

[delucca]

I'm having the same problem here. It appears that localstack latest image is buggy,

[whummer]

@mpfeil @Koalk @odelucca @nealeu This should be fixed in https://github.com/localstack/localstack/pull/1504 . Can you please give it a try with the latest version of the Docker image, as well as version ^0.4.19 of serverless-localstack?

[Can-Sahin]

Also getting error

Serverless: [AWS cloudformation 500 68.735s 3 retries] createStack({ StackName: 'XXX-local',
  OnFailure: 'ROLLBACK',
  Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
  Parameters: [ [length]: 0 ],
  TemplateURL:
   'http://localhost:4572/xxx/serverless/xxx/local/1568292545505-2019-09-12T12:49:05.505Z/compiled-cloudformation-template.json',
  Tags: [ { Key: 'STAGE', Value: 'local' }, [length]: 1 ] })
Serverless: Recoverable error occurred (500), sleeping for 5 seconds. Try 1 of 4

I am using the latest commit. I have no idea why. Cannot see any logs

[whummer]

@Can-Sahin can you please check and post the logs from the LocalStack Docker container:

docker logs localstack_main

Thanks

[CodeNinja1395]

Hi, I`m still facing the same issue

Error [NetworkingError]: socket hang up at connResetException (internal/errors.js:604:14) at Socket.socketOnEnd (_http_client.js:460:23) at Socket.emit (events.js:323:22) at Socket.EventEmitter.emit (domain.js:482:12) at endReadableNT (_stream_readable.js:1204:12) at processTicksAndRejections (internal/process/task_queues.js:84:21) { message: 'socket hang up', code: 'NetworkingError', region: 'us-east-1', hostname: 'localhost', retryable: true, time: 2020-04-15T14:10:13.935Z }

my docker config is:

version: '3.7' services: localstack: image: localstack/localstack:latest ports: - "4567-4597:4567-4597" - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}" environment: - SERVICES=lambda,apigateway,dynamodb,cloudformation,sts,iam,cloudwatch - DEBUG=${DEBUG- } - DATA_DIR=${DATA_DIR- } - PORT_WEB_UI=${PORT_WEB_UI- } - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- } - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- } - DOCKER_HOST=unix:///var/run/docker.sock volumes: - "${TMPDIR:-/tmp/localstack}:/tmp/localstack" - "/var/run/docker.sock:/var/run/docker.sock"

[CodeNinja1395]

I fixed this issue with downgrading to version 0.10.5 and explicitly adding those services to docker-composefile: SERVICES=lambda,s3,apigateway,dynamodb,cloudformation,sts,iam,cloudwatch

[steffyP]

closing old issue with no activity