serverless-dynamodb-local ECONNREFUSED for sls dynamodb start

ECONNREFUSED for sls dynamodb start

Open SamCB opened this issue 5 years ago • 27 comments

Actual Behaviour

Trying to start the database with sls dynamodb start it hangs for a while then gives error:

connect ECONNREFUSED 127.0.0.1:8000

I was able to get it running earlier, but after restarting the computer it stopped working.

Restarting the computer, changing the port and clearing reinstalling node_modules doesn't change anything.

There is nothing running on that port, and there are no java processes already running to suggest the database is already alive. http://localhost:8000/shell does not run and the serverless functions timeout trying to connect to the database.

Expected Behaviour

Run sls dynamodb start and it runs.

Steps to reproduce it

// package.json
{
  // ...
  "devDependencies": {
    // ...
    "dynamodb-local": "^0.0.24",
    // ...
    "serverless-dynamodb-local": "^0.2.35",
    "serverless-offline": "^3.31.3",
    // ...
  }
}

# serverless.yml
custom:
  # ...
  dynamodb:
    stages:
      - dev
    start:
      migrate: true
      inMemory: true
      noStart: true

plugins:
  - serverless-step-functions
  - serverless-plugin-typescript
  - serverless-dynamodb-local
  - serverless-step-functions-offline
  - serverless-offline

# ...

resources:
  Resources:
    ExampleTable:
      Type: AWS::DynamoDB::Table
      Properties:
        AttributeDefinitions:
          - AttributeName: foo
            AttributeType: S
        KeySchema:
          - AttributeName: foo
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMODB_TABLE}

LogCat for the issue

$ SLS_DEBUG=* yarn sls dynamodb start
yarn run v1.12.3
$ serverless dynamodb start
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command invoke
Serverless: Load command invoke:stepf
Serverless: Load command dynamodb
Serverless: Load command dynamodb:migrate
Serverless: Load command dynamodb:seed
Serverless: Load command dynamodb:start
Serverless: Load command dynamodb:noStart
Serverless: Load command dynamodb:remove
Serverless: Load command dynamodb:install
Serverless: Load command step-functions-offline
Serverless: Load command offline
Serverless: Load command offline:start
Serverless: Invoke dynamodb:start
Serverless: DynamoDB - Error -

  Networking Error ---------------------------------------

    connect ECONNREFUSED 127.0.0.1:8000

        For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

NetworkingError: connect ECONNREFUSED 127.0.0.1:8000
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1121:14)
From previous event:
    at ServerlessDynamodbLocal.createTable (/Users/foo/bar/node_modules/serverless-dynamodb-local/index.js:290:16)
    at BbPromise.each (/Users/foo/bar/node_modules/serverless-dynamodb-local/index.js:169:55)
From previous event:
    at ServerlessDynamodbLocal.migrateHandler (/Users/foo/bar/node_modules/serverless-dynamodb-local/index.js:169:26)
    at BbPromise.resolve.then (/Users/foo/bar/node_modules/serverless-dynamodb-local/index.js:219:45)
From previous event:
    at ServerlessDynamodbLocal.startHandler (/Users/foo/bar/node_modules/serverless-dynamodb-local/index.js:219:10)
    at BbPromise.reduce (/Users/foo/bar/node_modules/serverless/lib/classes/PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (/Users/foo/bar/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.run (/Users/foo/bar/node_modules/serverless/lib/classes/PluginManager.js:422:17)
    at variables.populateService.then.then (/Users/foo/bar/node_modules/serverless/lib/Serverless.js:157:33)
    at processImmediate (timers.js:632:19)
    at process.topLevelDomainCallback (domain.js:120:23)
From previous event:
    at Serverless.run (/Users/foo/bar/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/Users/foo/bar/node_modules/serverless/bin/serverless:44:28)

  Get Support --------------------------------------------
    Docs:          docs.serverless.com
    Bugs:          github.com/serverless/serverless/issues
    Issues:        forum.serverless.com

  Your Environment Information -----------------------------
    OS:                     darwin
    Node Version:           11.3.0
    Serverless Version:     1.34.1

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Dec 16 '18 22:12 SamCB

Ok. Figured out the fix. Had the .dynamodb directory lying around.

rm -r .dynamodb

And I was able to start up the database. Though is this expected behavior? Why did it cause this particular error and how do I stop it in the future without manually deleting things?

Dec 17 '18 22:12 SamCB

Eh, unfortunately not a fix. Error cropped up again and this time there was no .dynamodb directory.

Dec 18 '18 02:12 SamCB

Having the same problem here.

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.10.0
     Serverless Version:     1.35.1

Dec 20 '18 20:12 kjf

The problem should be the line noStart: true

From the documentation: # Uncomment only if you already have a DynamoDB running locally # noStart: true

Jan 03 '19 13:01 nnyegaard

Having the same problem. I don't have noStart: true either. The output I'm getting is:

ais-service-codestore | Dynamodb is already installed on path!
ais-service-codestore | Dynamodb Local Started, Visit: http://localhost:8000/shell
ais-service-codestore | Serverless: ERROR: NetworkingError: connect ECONNREFUSED 127.0.0.1:8000

Jan 15 '19 16:01 EwanValentine

Did you figure out what the issue was in the end @SamCB ?

Jan 15 '19 16:01 EwanValentine

My workaround was to use AWS’s DynamoDB local docker container, point the config to that, and use this tool only to handle the database migrations.

I actually prefer it as a solution because it’s now really easy to clean up and restart the database.

Jan 15 '19 20:01 SamCB

@nnyegaard I was under the impression that noStart: true simply stopped trying to spin up the database whenever you ran sls offline and was for cases that the user tried manually starting the database with sls dynamodb start.

Is that not the case?

Jan 15 '19 20:01 SamCB

@SamCB Good to know, as I've just gone down that route myself! Thanks :)

Jan 15 '19 22:01 EwanValentine

If you're running in a Windows 10 environment, then there is an issue with the dynamodb start command using child_process to spawn the java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb.

The solution I found was to modify node_modules/dynamodb-localhost/dynamodb/start.js and include the flag shell: true into the options at line 38.

Jan 21 '19 13:01 triqi

I have the same issue in the original report above.

OS:                     darwin
Node Version:                    8.11.4
Serverless Version:            1.38.0
serverless-dynamodb-local   0.2.36

Feb 22 '19 22:02 mycargus

Currently also have this issue... any assistance would be great.

OS: darwin Node Version: 10.15.2 Serverless Version: 1.38.0 serverless-dynamodb-local 0.2.35

Mar 08 '19 16:03 mteleskycmp

@triqi is correct - after many frustrating hours of trying to make things work, just adding shell: true to the options passed into spawn() did the job for me - at least as a temporary fix.

The problem is, you'd never know it, since the serverless dynamodb start exits the command line process after completing the start process. However, if you run the command twice, you should get the error java.net.BindException: Address already in use: bind telling you that a process is already running on that port.

In your terminal run netstat -ano | find "8000" (replace 8000 with your port # if different) to find which process ID the spawned java process is running on (last column). Go to Task Manager > Details (if on Win10) and order by PID to find the java process with that ID. Once you're done running your code locally, just end the process and then restart again using the serverless dynamodb start as needed.

Regarding the original connect ECONNREFUSED 127.0.0.1:8000 error, if using netstat and the Task Manager, you can see a node and java process running but for some reason they stop and the error is thrown (not sure which happens first). There is no node process if using the shell: true option.

Also, I haven't investigated this, but before I added shell: true, my database would only start (java and node processes spawned) if I passed in a --port argument to serverless dynamodb start that was different than the port I set in serverless.yml. Of course, it would error out with the ECONNREFUSED shortly after actually starting.

I troubleshooted these with versions 0.2.30, 0.2.35 and 0.2.36 of serverless-dynamodb-local. The latter version required installing the db with the sls dynamodb install --localPath ./bin command due to a different bug in that version.

Serverless versions used were 1.39.xx (don't remember exactly which) and 1.40.0.

Node version was 8.xx and 10.15.3

I gotta say, this was a great time to start using dynamodb 🤣

Kudos to @triqi for the workaround! I hope it works for others, not just Win10 users.

Mar 29 '19 15:03 DGempler

Question is, how do you stop it once using the workaround? Running Powershell on Win 10 CTRL + C doesn't work because it's not running in the active shell.

Closing the my shell window seemed to do the trick but that seems inconvenient.

Apr 01 '19 11:04 sethmoore

@sethmoore close the java process in Task Manager. If you have more than one running, use the netstat command I mention above to find the process ID of the correct one to terminate.

Not ideal either way, but at least I can run dynamodb locally this way.

Apr 01 '19 13:04 DGempler

I ended up just greping for the process that was running dynamo and killing it before starting again.

Then eventually moved the dynamodb into a docker container so its easy to kill now along withe everything else :D

Apr 01 '19 14:04 mteleskycmp

In my case was a bad indentation of noStart: true, which has to be at the same level as start

May 06 '19 02:05 diosney

I'm on MacOS. This command kill the java process that is using port 8000: kill -9 $(lsof -i TCP:8000| grep LISTEN | awk '{print $2}')

Credit: https://medium.com/@valgaze/utility-post-whats-running-on-port-8000-and-how-to-stop-it-2ed771fbb422

May 15 '19 21:05 jpblancoder

Any news on this? I have similar issues whereby ^C doesn't terminate the dynamodb-local process, although is said otherwise in the documentation. It only stops the serverless offline process.

However, once I do

 kill $(lsof -t -i :8000)

to kill all process runing on that port, and then run

sls offline start

for the next time around, the terminal hangs in this stage:

....
Dynamodb Local Started, Visit: http://localhost:8000/shell
....

and then it finally displays the error

Networking Error ---------------------------------------

  connect ECONNREFUSED 127.0.0.1:8000

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

NetworkingError: connect ECONNREFUSED 127.0.0.1:8000

before terminating the prompt.

However even after the prompt is terminated, i.e.

error Command failed with exit code 1.

http://localhost:8000/shell/ works, which is very very weird and sad.

Restarting the whole process would yield the exact same behavior :( And yes, even after running the command sls dynamodb remove, the exact same behavior is still happening.

Any help would be appreciated :)

Jun 20 '19 04:06 jasonlimantoro

Here's a one-liner I was able to put together for killing the process in Windows (Git Bash):

tskill $(netstat -aon | grep :8000 | grep LISTENING | head -1 | awk '{print $5}')

Nov 17 '19 02:11 DGempler

This appears to happen whenever you define the port that serverless-offline is running on sls offline start --port 3008 fails whilst sls offline start works fine

Feb 17 '20 11:02 SamWSoftware

Finally fix this issue (working with typescript and webpack from this code base)

Install: sls dynamodb install
Reset: sls dynamodb remove
Run locally: serverless offline start --stage test

# serverless.yml
plugins:
  - serverless-webpack
  - serverless-dynamodb-local
  - serverless-offline

custom:
  dynamodb:
    start:
      inMemory: true
      migrate: true
    stages:
      - test

// package.json
{
 "dependencies": {
    "aws-lambda": "^0.1.2",
    "aws-sdk": "^2.625.0",
    "node": "^12.3.1",
    "uuid": "^3.3.2"
  },
  "devDependencies": {
    "@types/aws-lambda": "8.10.1",
    "@types/node": "^8.0.57",
    "@types/uuid": "^3.4.4",
    "serverless-dynamodb-local": "^0.2.39",
    "serverless-offline": "^4.10.6",
    "serverless-webpack": "^5.1.1",
    "source-map-support": "^0.5.6",
    "ts-loader": "^4.2.0",
    "typescript": "^2.9.2",
    "webpack": "^4.5.0"
  }
}

Feb 25 '20 14:02 Jeremy38100

This happens to me when I run these commands inside a container. It doesn't happen on my local machine. I haven't found out why :/

Mar 05 '20 19:03 ahonarmand-xealth

In my case was a bad indentation of noStart: true, which has to be at the same level as start

Unbelievable solution. This fixed it for me @diosney . Thank you. Hopefully it will help someone else

Mar 12 '20 05:03 alleypa

This happens to me when I run these commands inside a container. It doesn't happen on my local machine. I haven't found out why :/

@ahonarmand-xealth I've got same issue. It's about container, actually JVM. JVM use C runtime internally such as glibc, or musl. DynamoDB offline is running on JVM with glibc, not musl. If you use an container based alpine, it uses musl, so dynamoDB is not working. You can use an docker image with glibc and JVM for dynamodb offline.

If you want to use dynamodb offline with alpine linux, check the links below.

https://github.com/jetsbee/dockerInDocker
https://blog.gilliard.lol/2018/11/05/alpine-jdk11-images.html

The way in second link isn't working, because some JVM modules for running dynamodb-offline are not included. but you can get some related info in this link.

May 07 '20 10:05 jetsbee

@jetsbee I really didn't want to use the openjdk as base, and I was taking an approach similar to point 2. After hitting this error, started searching around and found this issue. Reluctantly I switched the image to the jdk one, and yes, it solved the ECONNREFUSED. I really thought I was covering the glibc with my original approach (basically very similar to what is shown, with the sgerrand alpine pkg. Any idea which modules we could potentially be missing to make this work with an alpine base? For now I'll switch it, but it would be cool to make it work. Thanks for the info.

May 24 '20 22:05 peterfiorito

I run into this problem when I start tests which want to connect to the DynamoDB.

Setup: GitLab CI Custom docker image: Ubuntu with Node and Open JDK 11

The error does NOT occur during startup (all tables seem to be created successfully), but then connecting fails in the subsequent jest tests.

NetworkingError: connect ECONNREFUSED 127.0.0.1:8000Error: connect ECONNREFUSED 127.0.0.1:8000

Needless to say, on dev machines running Linux or Mac OS, this is all running fine.

Oct 27 '20 16:10 mindhaq

serverless-dynamodb-local serverless-dynamodb-local copied to clipboard

ECONNREFUSED for sls dynamodb start

serverless-dynamodb-local
serverless-dynamodb-local copied to clipboard