security-dashboards-plugin icon indicating copy to clipboard operation
security-dashboards-plugin copied to clipboard

Published 20 hours ago • verified publisher icon opensearch-project

[BUG] Dashboard fails with error 502 Bad Gateway when using OIDC-based authn with Keycloak

Open mcaliandro opened this issue 1 year ago • 5 comments

What is the bug? The Dashboard application crashes with error 502 Bad Gateway when an user authenticated & authorized by Keycloak tries to access Opensearch.

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Go to Dashboard URL, i.e., https://opensearch.example.com/
  2. Click on 'Log in with single sign-on'
  3. Wait a couple of seconds
  4. Error 502 Bad Gateway is shown in a plain HTML page

What is the expected behavior? An authenticated & authorized user should see the application's homepage, and navigate through the Dashboards' menus and sections.

What is your host/environment?

  • Kubernetes v1.25
  • Opensearch K8s Operator v2.4.0
  • Opensearch v2.7.0
  • Opensearch Dashboards v2.7.0

Do you have any screenshots? screenshot

Do you have any additional context? Logs from Dashboard pod

{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["info","plugins","securityDashboards"],"pid":453,"message":"Error: unexpected end of file\n    at Zlib.zlibOnError [as onerror] (zlib.js:187:17)\n    at processChunkSync (zlib.js:451:12)\n    at zlibBufferSync (zlib.js:175:12)\n    at Object.inflateSync (zlib.js:786:14)\n    at inflateValue (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/utils/compression.ts:25:43)\n    at getExtraAuthStorageValue (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/session/cookie_splitter.ts:48:15)\n    at OpenIdAuthentication.getExtraAuthStorageValue (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:293:20)\n    at OpenIdAuthentication.buildAuthHeaderFromCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:308:44)\n    at MultipleAuthentication.buildAuthHeaderFromCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/multiple/multi_auth.ts:181:50)\n    at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:163:42\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:112:22)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32) {\n  errno: -5,\n  code: 'Z_BUF_ERROR'\n}"}
{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["debug","opensearch","opendistro_security","query"],"pid":453,"message":"401\nGET /_plugins/_security/authinfo\n"}
{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["error","plugins","securityDashboards"],"pid":453,"message":"Failed to resolve user tenant: Error: Failed authentication: Authentication Exception"}
{"type":"response","@timestamp":"2023-08-23T15:22:20Z","tags":[],"pid":453,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"31e8b2144848a84d81c45b3228a9854f","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>1","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","dnt":"1","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"},"res":{"statusCode":302,"responseTime":951,"contentLength":9},"message":"GET / 302 951ms - 9.0B"}
{"type":"response","@timestamp":"2023-08-23T15:22:21Z","tags":[],"pid":453,"method":"get","statusCode":200,"req":{"url":"/app/login","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"a04ce13246ed28ce94e457fd15825111","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","dnt":"1","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"},"res":{"statusCode":200,"responseTime":96,"contentLength":9},"message":"GET /app/login 200 96ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/restapiinfo","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"1303f354be8246a84e57cb2c9da8c5ea","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":7,"contentLength":9},"message":"GET /api/v1/restapiinfo 401 7ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"74ba3c6c69e4635edef572e43286a0dc","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":11,"contentLength":9},"message":"GET /api/v1/configuration/account 401 11ms - 9.0B"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Regular.woff2","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"02474602e978893c4c1cb9ba4757279f","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"identity","dnt":"1","referer":"https://<opensearch-dashboard-url>/app/login?","sec-fetch-dest":"font","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":200,"responseTime":10,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Regular.woff2 200 10ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/auth/dashboardsinfo","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"be35376a0d887498f9d5489655d82bd9","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":4,"contentLength":9},"message":"GET /api/v1/auth/dashboardsinfo 401 4ms - 9.0B"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"697ccde49f00404b213e4e6f83b6fd5c","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"668","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","origin":"https://<opensearch-dashboard-url>","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":200,"responseTime":20,"contentLength":9},"message":"POST /api/core/capabilities 200 20ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/auth/type","method":"get","headers":{"host":"<opensearch-dashboard-url>":"3295e240b72e8b42054827c32e6aa077","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":5,"contentLength":9},"message":"GET /api/v1/auth/type 401 5ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/multitenancy/tenant","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"4233165a1ad68dc1e142f7e52ec49fb5","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":4,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 401 4ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"352d954928c1c715138065349775b8c5","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":6,"contentLength":9},"message":"GET /api/v1/configuration/account 401 6ms - 9.0B"}

mcaliandro avatar Sep 26 '23 14:09 mcaliandro

[Triage] This seems like a good issue. Marking triaged since it has a clear issue to be investigated and fixed.

stephen-crawford avatar Oct 30 '23 19:10 stephen-crawford

We are also facing the similar issue . Please let us know your analysis

kannanvr avatar Jan 02 '24 07:01 kannanvr

If you're using ingress-nginx as your ingress controller, this issue can be caused by the ingress you use to access Dashboards. You might want to look into your ingress controller logs for details or just add the following annotation to the ingress and see what happens: nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"

ldk300 avatar Jan 09 '24 18:01 ldk300

Adding nginx.ingress.kubernetes.io/proxy-buffer-size: "256k" to ingress annotations fixed the same issue, thank you @ldk300

maxemontio avatar Jan 26 '24 15:01 maxemontio

Is this issue solved? I am also having similar problems while deploying KPI dashboard using SSL certificate.

ajila-123 avatar Mar 08 '24 07:03 ajila-123