Strange errors in CloudWatch logs for TestCafe Dashboard deployments

[pavsenin]

Describe the bug There are plenty of strange errors that we see in CloudWatch logs for our stage (x0f8ha) and prod (hefkwa) deployments. Now we are filtering them with the following filter:

  @message not like 'Client was closed and is not queryable'
  @message not like 'remaining connection slots are reserved'
  @message not like 'No vacant time to process the query'
  @message not like 'UnknownEndpoint: Inaccessible host'
  @message not like 'timeout expired'
  @message not like 'top-level event handler execution'
  @message not like 'Runtime exited with error: exit status 128'
  @message not like 'Failed to post handler success'
  @message not like 'Connection terminated unexpectedly'
  @message not like 'Unknown application error occurred'
  @message not like 'disposing resolve entries'
  @message not like 'Adapter has been already disposed'

To Reproduce To reproduce the behavior execute the following query for the deployments mentioned above:

fields @message as m
| filter m like /(?i)error/
  and m not like 'WARN' and m not like 'INFO'
  and m not like 'UnauthorizedError: jwt'
  and m not like '/api/commands'

Expected behavior No error logs.

Desktop (please complete the following information):

• ReSolve Version [e.g. 0.31.10]

[MrCheater]

• Warning: 'No vacant time to process the query' Reason: https://github.com/reimagined/resolve/blob/356fc6700cd7f63a367fda26940c4b56fc3714ef/packages/runtime/adapters/eventstore-adapters/eventstore-postgresql/src/check-request-timeout.ts#L12

• Warning: 'UnknownEndpoint: Inaccessible host' Additional context: Search Results: "aws-sdk Inaccessible host" "AWS S3 Inaccessible host" https://github.com/aws/aws-sdk-js/issues/2618 https://stackoverflow.com/questions/55705043/aws-s3-inaccessible-host https://forums.aws.amazon.com/thread.jspa?threadID=165751

• Warnings: 'timeout expired', 'Client was closed and is not queryable', 'remaining connection slots are reserved' Status: Postgres Problem. EventStore/ReadModel adapter.

• Warning: 'Runtime exited with error: exit status 128' Additional context: https://answers.netlify.com/t/async-serverless-function-trouble/25049

• Warning: 'LAMBDA_RUNTIME Failed to post handler success response' Status: Unknown Error Additional context: The maximum payload size for the request body or the response body of a Serverless Function is 5MB. If a Serverless Function receives a payload in excess of the limit it will return an error - 413: FUNCTION_PAYLOAD_TOO_LARGE. This is a hard limit and cannot be increased.

• Warning: 'Connection terminated unexpectedly' Status: Unknown Error

• Warnings: 'Unknown application error occurred', 'top-level event handler execution' Status: Unknown Fatal Error. See resolve.monitoring.group({ Part: 'Internal' })

• Warnings: 'disposing resolve entries', 'Adapter has been already disposed' Additional Context: https://github.com/reimagined/resolve/blob/c23cd2743eb2b9469b5791327516c79fbfbf798f/packages/runtime/runtime/src/common/dispose-resolve.js#L26 Solution: Add catch: https://github.com/reimagined/resolve/blob/c23cd2743eb2b9469b5791327516c79fbfbf798f/packages/runtime/adapters/readmodel-adapters/readmodel-postgresql/src/build.ts#L141-L149

[pavsenin]

Hello guys,

The list of new errors (not filtered with the current whitelist) for the last week:

Deployment: x0f8ha Error: 2021-09-25T23:26:54.499Z resolve:runtime:cloud-entry error RequestTimeoutError: Query read timeout

Deployment: x0f8ha Error: 2021-09-25T23:26:54.552Z c31f218d-b64c-4895-8593-e43b0988c4d1 ERROR Lambda fatal error: RequestTimeoutError: Query read timeout at executeStatement (/var/task/node_modules/@resolve-js/eventstore-postgresql/lib/execute-statement.js:44:15) at processTicksAndRejections (internal/process/task_queues.js:97:5) { code: 503, name: 'ConnectionError' }

Deployment: x0f8ha Error: 2021-09-25T23:26:54.499Z resolve:runtime:cloud-entry error Query read timeout

Deployment: x0f8ha Error: 2021-09-29T21:45:01.846Z resolve:runtime:dispose-resolve Error: connect ETIMEDOUT 172.31.45.46:5432

Deployment: hefkwa Error: 2021-10-01T11:27:06.504Z 8c51150c-022e-47af-9974-6be195f52e38 ERROR Lambda fatal error: Error: connect ETIMEDOUT 172.31.10.92:5432 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16) at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:126:14) { errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect', address: '172.31.10.92', port: 5432 }

Deployment: x0f8ha Error: RequestId: 02a5526a-836e-485e-b2c1-1898073efa0a Error: Runtime exited with error: signal: segmentation fault (core dumped) Runtime.ExitError

Deployment: x0f8ha Error: 2021-10-03T14:21:15.691Z 54526c5a-3ae2-4584-89d5-6a5ac72b113c ERROR Lambda fatal error: Error: read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:209:20) at TCP.callbackTrampoline (internal/async_hooks.js:126:14) { code: 503, name: 'ConnectionError' }

Deployment: x0f8ha Error: 2021-10-03T14:21:13.806Z 548428e6-45cd-4bf0-b8e6-e5c07d9f729d ERROR Lambda fatal error: Error: read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:209:20) at TCP.callbackTrampoline (internal/async_hooks.js:126:14) { errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }

[pavsenin]

Hello guys,

The list of new errors (not filtered with the current whitelist) for the last week:

Error: TimeoutError: socket hang up Error: 2021-10-20T15:15:48.596Z resolve:runtime:dispose-resolve Error: Read model adapter pool is null Error: severity: 'ERROR', Error: at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8) Error: at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27) Error: 2021-10-16T12:20:28.414Z 61ec8be6-0460-4fe3-9f8a-4be9a00376bb ERROR Lambda fatal error: Error: connect ETIMEDOUT 172.31.45.46:5432 Error: 2021-10-16T12:20:28.384Z resolve:runtime:cloud-entry error Error: connect ETIMEDOUT 172.31.45.46:5432 Error: 2021-10-15T14:01:01.203Z resolve:runtime:cloud-entry error Error: getaddrinfo ENOTFOUND resolve-prod-system.cluster-cby46by5hywj.eu-central-1.rds.amazonaws.com Error: 2021-10-15T14:01:01.226Z 398b6ba4-2566-41ce-a275-6cd603dd3d86 ERROR Lambda fatal error: Error: getaddrinfo ENOTFOUND resolve-prod-system.cluster-cby46by5hywj.eu-central-1.rds.amazonaws.com

[pavsenin]

Hello guys,

Warnings: 'disposing resolve entries' Ignore already disposed error on loadEvents in readmodel build #2047 Merged

Warning: 'Connection terminated unexpectedly' Status: Fixed Failed to load events in eventstore adapter if connection is terminated while events loading #1901

Still many such errors in 0.32.0

[pavsenin]

Hello guys,

A few new errors:

Deployment: y1oy3x Error: at generateCommandError (/var/task/node_modules/@resolve-js/core/lib/aggregate/get-aggregates-interop-builder.js:37:10)

Deployment: aaw7mo Error: at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)

Deployment: aaw7mo Error: at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)

Deployment: aaw7mo Error: 2021-12-03T19:17:32.054Z 2ac0a063-b934-4b8f-bfc1-b5b02a79f225 ERROR Lambda fatal error: error: terminating connection because backend initialization completed past serverless scale point at Parser.parseErrorMessage (/var/task/node_modules/pg-protocol/src/parser.ts:369:69) at Parser.handlePacket (/var/task/node_modules/pg-protocol/src/parser.ts:188:21)

Deployment: aaw7mo Error: 2021-12-06T08:09:33.550Z 94cd40be-77c6-4f9b-8714-7a8256845dce ERROR Invoke Error {"errorType":"StatementTimeoutException","errorMessage":null,"code":"StatementTimeoutException","message":null,"time":"2021-12-06T08:09:33.541Z","requestId":"f03e90f8-3be8-4814-a54e-ccebcdf47dc2","statusCode":400,"retryable":false,"retryDelay":41.535869554201675,"stack":["StatementTimeoutException: "," at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)","

Deployment: aaw7mo Error: 2021-12-08T13:38:28.330Z 69920495-d893-4a8b-a7a3-11a17508abe2 ERROR Lambda fatal error: error: could not resize shared memory segment "/PostgreSQL.869018437" to 1585152 bytes: Cannot allocate memory at Parser.parseErrorMessage (/var/task/node_modules/pg-protocol/src/parser.ts:369:69) at Parser.handlePacket (/var/task/node_modules/pg-protocol/src/parser.ts:188:21)

Deployment: aaw7mo Error: 2021-12-08T13:38:28.301Z resolve:runtime:cloud-entry error error: could not resize shared memory segment "/PostgreSQL.869018437" to 1585152 bytes: Cannot allocate memory

Deployment: aaw7mo Error: 2021-12-08T14:12:56.031Z 971af871-48ba-4ca9-8645-07151c275b5c ERROR Lambda fatal error: error: Too many connection errors: incorrect credentials or database is not available at Parser.parseErrorMessage (/var/task/node_modules/pg-protocol/src/parser.ts:369:69) at Parser.handlePacket (/var/task/node_modules/pg-protocol/src/parser.ts:188:21)

Deployment: aaw7mo Error: 2021-12-09T08:50:44.014Z 86a22235-4eda-46ff-989f-bf5c108ae3f0 ERROR Error: Variable "Status code" should be one of following types: function Number() { [native code] }. Received option: TooManyRequestsException at validateOptionShape (/var/task/node_modules/@resolve-js/runtime/lib/cloud/wrap-api-handler.js:198:13)

Deployment: aaw7mo Error: 2021-12-09T10:33:01.527Z 702afec8-4fd2-4354-a30f-657e50948b27 ERROR Lambda fatal error: Error: connect ETIMEDOUT 172.31.10.92:5432 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:130:17) { errno: -110, code: 'ETIMEDOUT', syscall: 'connect', address: '172.31.10.92', port: 5432 }