Unhandled exception in query orchestrator

[gizmodus]

Describe the bug I just had an unhandled exception which brought the whole server down. The error originates in the query-orchestrator and seems to have something to do with Redis.

Here is the full log:

events.js:288 throw er; // Unhandled 'error' event ^ AbortError: Ready check failed: Connection forcefully ended and command aborted. It might have been processed. at RedisClient.flush_and_error (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\index.js:362:23) at RedisClient.end (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\lib\extendedApi.js:52:14) at Object.RedisPool.destroy (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\orchestrator\RedisPool.js:12:64) at Pool._destroy (D:\Data\git\sublimd-application\build\node_modules\generic-pool\lib\Pool.js:137:42) at Pool._evict (D:\Data\git\sublimd-application\build\node_modules\generic-pool\lib\Pool.js:382:14) at Timeout._onTimeout (D:\Data\git\sublimd-application\build\node_modules\generic-pool\lib\Pool.js:392:14) at listOnTimeout (internal/timers.js:549:17) at processTimers (internal/timers.js:492:7) Emitted 'error' event on RedisClient instance at: at RedisClient.on_info_cmd (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\index.js:496:14) at D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\index.js:535:14 at Object.callbackOrEmit [as callback_or_emit] (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\lib\utils.js:89:9) at Command.callback (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\lib\individualCommands.js:157:15) at RedisClient.flush_and_error (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\index.js:374:29) at RedisClient.end (D:\Data\git\sublimd-application\build\node_modules@cubejs-backend\query-orchestrator\node_modules\redis\lib\extendedApi.js:52:14) [... lines matching original stack trace ...] at processTimers (internal/timers.js:492:7) { code: 'NR_CLOSED', command: 'INFO'

To Reproduce Unfortunately I could not reproduce the error. It probably happened when I woke my computer from sleep mode.

Expected behavior There should be no unhandled exceptions that can kill the server.

Version: 0.18.7

Additional context I have had similar experiences with the Redis client. Not sure if this helps, but in our project, we could solve the problem by explicitly handling errors of the Redis client with the following code:

this.redisClient.on('error', (error) => {
    sharedLogger.error(error.message);
});

[paveltiunov]

@gizmodus Hey Reto! Thanks for posting this one! This is interesting. Never seen this before. Yeah. I think it makes sense to handle these error messages and reconnect then.

[gizmodus]

I found a way to reproduce it: If you stop the Redis server while the application server is running, the above exception is immediately thrown and kills the whole server.

[gizmodus]

I can also reproduce the problem when I perform a load test that puts Redis under stress. The following error is thrown by Redis in the query orchestrator and leads to a server crash:

Ready check failed: Connection forcefully ended and command aborted. It might have been processed.

I was able to solve the problem by adding the following code at this position https://github.com/cube-js/cube.js/blob/c92f6848fabc6f5d41a46c6128456ae3fc9c6ad2/packages/cubejs-query-orchestrator/orchestrator/RedisFactory.js#L20

client.on('error', (error) => {
    // Log error
});

[rpaik]

👋 a quick reminder that we will be replacing Redis with Cube Store as announced in this blog post.

[igorlukanin]

For the record, Redis support is deprecated and will be removed from Cube in the future. It is not recommended to use Cube with Redis anymore.

Docs: https://cube.dev/docs/product/deployment#redis

Some details: https://cube.dev/blog/how-you-win-by-using-cube-store-part-1