near-lake-framework-js icon indicating copy to clipboard operation
near-lake-framework-js copied to clipboard

Block Streaming fails from time to time

Open nabeel99 opened this issue 1 year ago • 0 comments

So after running the near indexer for a extended period of time, while going thru the logs i realized there are time periods where indexer errors out , for e.g the errors below were observed :

    at throwDefaultError (/Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at deserializeAws_restXmlListObjectsV2CommandError (/Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5146:51)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '1E6VR7P5BS1JNHGZ',
    extendedRequestId: 'dsWWeYKE3OJf/IjghH1scNhZMrwI9cddsCxPSl75SuowSWMmxlS2XayaQjQbpUKLTW8DD1n3nDZFz9Mc/O1iQg==',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'RequestTimeTooSkewed',
  RequestTime: '20230410T222612Z',
  ServerTime: '2023-04-10T22:44:13Z',
  MaxAllowedSkewMilliseconds: '900000',
  RequestId: '1E6VR7P5BS1JNHGZ',
  HostId: 'dsWWeYKE3OJf/IjghH1scNhZMrwI9cddsCxPSl75SuowSWMmxlS2XayaQjQbpUKLTW8DD1n3nDZFz9Mc/O1iQg=='
}
Failed to fetch 123532466/shard_3.json. Retrying in 200ms RequestTimeTooSkewed: The difference between the request time and the current time is too large.
    at throwDefaultError (/Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at deserializeAws_restXmlGetObjectCommandError (/Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4356:51)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:58:20
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: 'MT47XMQXYH8VGKKS',
    extendedRequestId: 'e3u9V3Sy0wUnq3ZRk1n94b9W1nXk/aH8LQo+sjfpFXpmXv59dy/mlKShJaZ+pRcHQHPgjsS3DQs=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'RequestTimeTooSkewed',
  RequestTime: '20230410T222614Z',
  ServerTime: '2023-04-10T22:44:14Z',
  MaxAllowedSkewMilliseconds: '900000',
  RequestId: 'MT47XMQXYH8VGKKS',
  HostId: 'e3u9V3Sy0wUnq3ZRk1n94b9W1nXk/aH8LQo+sjfpFXpmXv59dy/mlKShJaZ+pRcHQHPgjsS3DQs='
}
Failed to fetch 123532482/shard_3.json. Retrying in 200ms RequestTimeTooSkewed: The difference between the request time and the current time is too large.
    at throwDefaultError (/Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at deserializeAws_restXmlGetObjectCommandError (/Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4356:51)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:58:20
    at async /Users/aaran/Desktop/understanding-aramid-finance-soldier-app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: 'MT46DPRSX1NH26R4',
    extendedRequestId: 'vHF/oZYPgOk0CoegjkrR0iTaBUPMPV7lduI0Pvp/spYL9cvhGa5pcP4ICGyzvNGuMmyTbuYQbFM=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'RequestTimeTooSkewed',
  RequestTime: '20230410T222614Z',
  ServerTime: '2023-04-10T22:44:14Z',
  MaxAllowedSkewMilliseconds: '900000',
  RequestId: 'MT46DPRSX1NH26R4',
  HostId: 'vHF/oZYPgOk0CoegjkrR0iTaBUPMPV7lduI0Pvp/spYL9cvhGa5pcP4ICGyzvNGuMmyTbuYQbFM='
}
Failed to list blocks. Retrying. Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
  errno: -54,
  code: 'ECONNRESET',
  syscall: 'read',
  name: 'TimeoutError',
  '$metadata': { attempts: 4, totalRetryDelay: 425 }
}

Just wanted to know if its a known issue and is there any workaround.

nabeel99 avatar Apr 10 '23 22:04 nabeel99