rethinkdbdash Fail to create a new connection for the connection pool.

Just trying out the library for the first time, with a simple

> r = require('rethinkdbdash')({db: 'test'})

I am getting the following:

Creating a pool connected to localhost:28015
{ [Function: _r]
  row: { [Function: term] _query: [ 13 ], _r: [Circular] },
  monday: { [Function: term] _query: [ 107 ], _r: [Circular] },
  tuesday: { [Function: term] _query: [ 108 ], _r: [Circular] },
  wednesday: { [Function: term] _query: [ 109 ], _r: [Circular] },
  thursday: { [Function: term] _query: [ 110 ], _r: [Circular] },
  friday: { [Function: term] _query: [ 111 ], _r: [Circular] },
  saturday: { [Function: term] _query: [ 112 ], _r: [Circular] },
  sunday: { [Function: term] _query: [ 113 ], _r: [Circular] },
  january: { [Function: term] _query: [ 114 ], _r: [Circular] },
  february: { [Function: term] _query: [ 115 ], _r: [Circular] },
  march: { [Function: term] _query: [ 116 ], _r: [Circular] },
  april: { [Function: term] _query: [ 117 ], _r: [Circular] },
  may: { [Function: term] _query: [ 118 ], _r: [Circular] },
  june: { [Function: term] _query: [ 119 ], _r: [Circular] },
  july: { [Function: term] _query: [ 120 ], _r: [Circular] },
  august: { [Function: term] _query: [ 121 ], _r: [Circular] },
  september: { [Function: term] _query: [ 122 ], _r: [Circular] },
  october: { [Function: term] _query: [ 123 ], _r: [Circular] },
  november: { [Function: term] _query: [ 124 ], _r: [Circular] },
  december: { [Function: term] _query: [ 125 ], _r: [Circular] },
  minval: { [Function: term] _query: [ 180 ], _r: [Circular] },
  maxval: { [Function: term] _query: [ 181 ], _r: [Circular] },
  nextVarId: 1,
  _Term: [Function: Term],
  _poolMaster: 
   PoolMaster {
     _r: [Circular],
     _line: Dequeue { start: 0, end: 0, buffer: [Object] },
     _pools: { unknownPools: [Object] },
     _healthyPools: [ [Object] ],
     _healthy: true,
     _init: false,
     _index: 0,
     _indexUnknown: 0,
     _discovery: false,
     _options: { db: 'test', buffer: 50, max: 1000 },
     _log: [Function],
     _draining: false,
     _numConnections: 0,
     _numAvailableConnections: 0,
     _hasPrintWarningLocalhost: false,
     _feed: null,
     _consecutiveFails: -1,
     _timeoutError: 1000,
     _maxExponent: 6,
     _seed: 0,
     _servers: [ [Object] ] },
  _options: {} }
> Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Exiting slow growth mode

RethinkDB is running, as I am able to access it via both the dashboard and the rethinkdb package, and the port is default.

I am able to use the resultant r object to query the DB, too.

> r.tableList().run().then( console.dir )
Promise {
  _bitField: 0,
  _fulfillmentHandler0: undefined,
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
> [ 'consumers' ]

Environment Info

Machine: Intel 64bit OS: Ubuntu 17.04 (Zesty) Node: v7.10.0 RethinkDB: built from git (next branch) rethinkdbdash: 2.3.29

May 31 '17 12:05 Panoplos

Did you start rethinkdb after starting your nodejs script?

Jun 01 '17 05:06 neumino

No, it is running as a daemon on the system.

Jun 01 '17 05:06 Panoplos

Any insight into this? This is obviously a show-stopper for us.

Jun 02 '17 01:06 Panoplos

I tested a slightly different configuration, where I set pool: false, then attempted the following:

import Rethinkdbdash from 'rethinkdbdash'
const r = Rethinkdbdash({pool: false})
r.connect()
  .then(debug.log)
  .error(debug.error)

Running as follows:

⟩ env DEBUG=test node --require 'babel-register' test.js

Results in:

  test
---------------------------------------
ReqlDriverError
Failed to connect to localhost:28015 in less than 20s.
ReqlDriverError
    at Timeout._onTimeout (/home/panoplos/test/node_modules/rethinkdbdash/lib/connection.js:103:12)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)
---------------------------------------

Note that I have also tested the following:

import r from 'rethinkdb'
r.connect( {host: 'localhost', port: 28015}, (err, conn) => {
  if (err) {
    debug.error(err)
  } else {
    debug.log("Connected successfully!")
  }
})

With the following result:

⟩ env DEBUG=test node --require 'babel-register' test.js 
  test   Connected successfully!

Jun 02 '17 02:06 Panoplos

More testing:

I have tested this against next, v2.4.x and v2.3.x, and it works only on v2.3.x of rethinkdb, so it seems the driver is not working with v2.4+ versions.

Jun 02 '17 04:06 Panoplos

I don't see a 2.4.x release for RethinkDB - Does it work with the official driver?

Jun 02 '17 04:06 neumino

v2.4.x is a branch of the git repo. It is not an official release, but it is stable, as is next. And, yes, the official driver works against these.

Jun 02 '17 04:06 Panoplos

Hum, rethinkdb doesn't build on my workstation (next or v2.4.x). That being said, it's probably an issue with rethinkdb though.

I don't see any change in the official driver for 2.4, so it's supposed to use the same protocol (and I don't see any mention of a different API in the changelog)

Jun 05 '17 00:06 neumino

What system are you attempting to build on? I had a small issue with building it on Ubuntu, but it was just a matter of compiling with the correct version of gcc.

Jun 06 '17 04:06 Panoplos

I get same error with rethinkdb docker image package verion - 2.3.5, Initially it connects and then throws the error

{
    servers: [
        { host: '192.168.99.100', port: 32780 },
        { host: '192.168.99.100', port: 32779 }
    ],
    pool: true,
    buffer: 300,
    max: 3000
}


Creating a pool connected to 192.168.99.100:32780
Creating a pool connected to 192.168.99.100:32780
(node:24627) [DEP0016] DeprecationWarning: 'root' is deprecated, use 'global'
{"kind":"notice","notice":"hello seneca cmj0k3p0xwii/1502666893090/24627/3.3.0/-","level":"info","when":1502666895646}
Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to 192.168.99.100:32780 in less than 20s","message":"Failed to connect to 192.168.99.100:32780 in less than 20s.","isOperational":true}
Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to 192.168.99.100:32780 in less than 20s","message":"Failed to connect to 192.168.99.100:32780 in less than 20s.","isOperational":true}

Aug 13 '17 23:08 BhaskaranR

@Panoplos were you able to fix this?

Aug 14 '17 11:08 BhaskaranR

No. I had to downgrade my RethinkDB installation.

Aug 14 '17 13:08 Panoplos

@BhaskaranR - you are running the stable version, you probably have a different problem. Please open a new bug instead

Aug 14 '17 13:08 neumino

This really needs to be fixed.. I hate the default rdb module.. I love this for the ease of use and would love to continue to use it. But this is preventing me from doing so... This doesn't seem to be a rethinkdb server issue.. This is only happening since updating rdb.

Dec 06 '17 09:12 dustinrouillard

Yes this an issue here as well.

Dec 08 '17 22:12 adamierymenko

With next.

Dec 08 '17 22:12 adamierymenko

It looks as if _compareDigest is not being called with newer versions which is never calling clearTimeout to clear the connection timeout timer.

Dec 08 '17 23:12 adamierymenko

i'm also getting this same issue i have installed rethinkdb 2.3.5 (GCC 5.4.0) rethinkdbdash 2.3.31,

Dec 13 '17 07:12 Zalasanjay

Everything was running well and suddenly this started happening for me as well. rethinkdb 2.3.6~0xenial (GCC 5.3.1)

(restarted my docker container, not rethinkdb, but the application and it was okay).

Feb 14 '18 08:02 heathweaver

I am also facing the same problem, Is it resolved after restarting the docker container?

Jun 16 '18 06:06 SatyaVaraprasad1978

Same problem over here. I have 2 containers, one works while the other gets connection error mentioned above and it gets stuck...

I even attempted to try healthy checks as mentioned in the documentation, but those aren't working either. Example of attempts:

r.getPoolMaster().on('healthy', function(healthy) {
  if (healthy === true) {
    console.log('We can run queries.');
  }
  else {
    console.log('No queries can be run.');
  }
});

Another attempt:

//attempt to get connections length and available connections, always seems to be 0
r.getPoolMaster().getLength()

What works is to restart container with a timeout... but that's a bad practice on a big project in production... Any ideas on this? (I downgraded from the recent August update, but it seems I' still having the same issue)

Aug 06 '18 21:08 gamebak

Still having this issue too, it makes rethink nearly unusable. I still haven't been able to find a solution. This issue really needs more attention.

Aug 08 '18 01:08 sudomoose

My current solution is to have my own rate limit, because my current architecture allows it. I am reading from the message queue at an interval of time and I kept a close eye when I had timeouts, when that happen I reduced the number of messages read.

I hope this gives someone at least a solution on how to deal with this sort of thing. Another solution that I had in mind was to drain the pool periodically, but you will need to add extra checks in your code, in case if a pool is being drained you will need to not do any query and to wait for that to finish, by default you'll get an error if it's not treated.

Aug 08 '18 11:08 gamebak

My current solution is to have my own rate limit, because my current architecture allows it. I am reading from the message queue at an interval of time and I kept a close eye when I had timeouts, when that happen I reduced the number of messages read.

I hope this gives someone at least a solution on how to deal with this sort of thing. Another solution that I had in mind was to drain the pool periodically, but you will need to add extra checks in your code, in case if a pool is being drained you will need to not do any query and to wait for that to finish, by default you'll get an error if it's not treated.

good solution, but a hell code situation. better dive into the origin of the bug. lots of headaches here

Nov 01 '18 13:11 ssribeiro

I was wondering if anyone has figured out what is going on that is causing this bug. I dug around and it does seem like _compareDigest isn't being called. After looking even more only one data event is being sent per a connection, which seems to be causing the time out, as the self.state in connection.js is only reaches 1 then stops, where to execute _compareDigest it needs to be equal to 2.

Mar 20 '19 22:03 MattMacGregor

rethinkdbdash rethinkdbdash copied to clipboard

Fail to create a new connection for the connection pool.

Environment Info

rethinkdbdash
rethinkdbdash copied to clipboard