[BUG] opensearch-dashboard 2.2 crashing with Unhandled Promise rejection detected

[ervikrant06]

Describe the bug

A clear and concise description of what the bug is.

While trying to load the dashboards or visualizations. Following query was ending with 503 error.

https://kibana-0.test.com/api/saved_objects/_find?fields=title&per_page=10000&type=index-pattern

intermittenly crash of opensearch-dashboards is noticed in logs

Sep 26 12:39:18 kibana-0 opensearch-dashboards[13185]: {"type":"response","@timestamp":"2022-09-26T12:39:18Z","tags":[],"pid":13185,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Medium.woff2","method":"get","headers":{"host":"kibana-0.test.com","connection":"keep-alive","sec-ch-ua":"\"Google Chrome\";v=\"105\", \"Not)A;Brand\";v=\"8\", \"Chromium\";v=\"105\"","origin":"https://kibana-0.test.com","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Linux\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetchdest":"font","referer":"https://kibana-0.test.com/app/dashboards","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"xx.xx.87.183","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","referer":"https://kibana-0.test.com/app/dashboards"},"res":{"statusCode":200,"responseTime":242,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Medium.woff2 200 242ms - 9.0
B"}
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: {"type":"log","@timestamp":"2022-09-26T12:39:38Z","tags":["error","opensearch","data"],"pid":13185,"message":"[TimeoutError]: Request timed out"}
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: Unhandled Promise rejection detected:
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: TimeoutError: Request timed out
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: at ClientRequest.onTimeout (/usr/share/opensearch-dashboards/node_modules/@opensearch-project/opensearch/lib/Connection.js:120:16)
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: at ClientRequest.emit (events.js:400:28)
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: at TLSSocket.emitRequestTimeout (_http_client.js:790:9)
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: at Object.onceWrapper (events.js:519:28)
Sep 26 12:39:38 kibana-0 opensearch-dashboards[13185]: at TLSSocket.emit (events.js:412:35)
Sep 26 12:39:38 kibana-0 systemd[1]: opensearch-dashboards.service: main process exited, code=exited, status=1/FAILURE
Sep 26 12:39:38 kibana-0 systemd[1]: Unit opensearch-dashboards.service entered failed state.
Sep 26 12:39:38 kibana-0 systemd[1]: opensearch-dashboards.service failed.
Sep 26 12:39:38 kibana-0 systemd[1]: opensearch-dashboards.service holdoff time over, scheduling restart.
Sep 26 12:39:38 kibana-0 systemd[1]: Stopped "OpenSearch Dashboards".
Sep 26 12:39:38 kibana-0 systemd[1]: Started "OpenSearch Dashboards".

After 2-3 hours automatically opensearch-dashboards started working.

Expected behavior A clear and concise description of what you expected to happen.

OpenSearch Version 2.2.0

Dashboards Version 2.2.0

Plugins

Please list all plugins currently enabled.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information): 3.10.0-1160.6.1.el7.x86_64

Additional context

Add any other context about the problem here.

[ervikrant06]

From the opensearch dashboards logs

Sep 26 15:20:52 kibana-0 opensearch-dashboards[13284]: {"type":"log","@timestamp":"2022-09-26T15:20:52Z","tags":["error","opensearch","data"],"pid":13284,"message":"[TimeoutError]: Request timed out"}
Sep 26 15:20:52 kibana-0 opensearch-dashboards[13284]: {"type":"response","@timestamp":"2022-09-26T15:20:22Z","tags":[],"pid":13284,"method":"get","statusCode":503,"req":{"url":"/api/saved_objects/_find?fields=title&per_page=10000&type=index-pattern","method":"get","headers":{"host":"kibana-0.test.com","connection":"keep-alive","sec-ch-ua":"\"Google Chrome\";v=\"105\", \"Not)A;Brand\";v=\"8\",\"Chromium\";v=\"105\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","osd-version":"2.2.0","sec-ch-ua-platform":"\"Linux\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://kibana-0.test.com/app/visualize","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"pt"},"remoteAddress":"xx.xx.87.183","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","referer":"https://kibana-0.test.com/app/visualize"},"res":{"statusCode":503,"responseTime":30011,"contentLength":9},"message":"GET /api/saved_objects/_find?fields=title&per_page=10000&type=index-pattern 503 30011ms - 9.0B"}

[joshuarrrr]

In general, OpenSearch Dashboards requires OpenSearch to be available, and make requests. From the dashboards side, we'd be interested in better handling and communicating the errors from the cluster. Can you provide some example screenshots of the UI?

Triage: We need further research to identify where the request error handling failed.

[joshuarrrr]

This may be part of the larger epic/proposal here: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1625