Login two separate Kibana instances, user in first Kibana instance is auto kicked out with AUTHENTICATION_ERROR

[justincr-elastic]

Kibana version: 8.2.2 Elasticsearch version: 8.2.2 Server OS version: Windows 10 Pro 19043.1706 Browser version: Chrome Version 101.0.4951.67 (Official Build) (64-bit) Browser OS version: Windows 10 Pro 19043.1706 Original install method (e.g. download page, yum, from source, etc.): Downloaded elasticsearch 8.2.2 and kibana 8.2.2 zip files for Windows Describe the bug:

Steps to reproduce:

1. Create cluster1 = 1 Kibana (c1k1 localhost:5611) + 2 Elasticsearch (c1e1 localhost:9311 and c1e2 localhost:9312)
2. Create cluster2 = 1 Kibana (c2k1 localhost:5621) + 2 Elasticsearch (c2e1 localhost:9321 and c1e2 localhost:9322)
3. Login to cluster1 Kibana (c1k1) as elastic/elastic1 in a Chrome tab.
4. Login to cluster2 Kibana (c2k1) as elastic/elastic2 in a separate Chrome tab.
5. Observe that after 1-3 seconds, the user in Kibana1 is automatically redirected to the login page with an AUTHENTICATION_ERROR, but there are no logs in both Kibanas.

Expected behavior: I would expect multiple Kibana nodes to run on the same system, if they are using different directories, certificates, elastic/kibana/kibana_system passwords, and ports. Screenshots (if relevant): Errors in browser console (if relevant): I don't know if these are relevant. These are what I found in Chrome console after being redirected to the login page.

login:286 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.

bootstrap-anonymous.js:42 ^ A single error about an inline script not firing due to content security policy is expected!
login:1 This site does not have a valid SSL certificate! Without SSL, your site's and visitors' data is vulnerable to theft and tampering. Get a valid SSL certificate before  releasing your website to the public.

Provide logs and/or server output (if relevant): No errors in Kibana logs of both servers. Logging was left at the default log level. Any additional context: I verified each cluster is isolated.

Cluster1 Kibana (c1k1 localhost:5611) console shows localhost:9311 and localhost:9312

GET /_cluster/state
{
  "cluster_name" : "cluster1",
  "cluster_uuid" : "Dn8nyrRqToKkKgevcadqhw",
  "version" : 156,
  "state_uuid" : "WemfWsQ-RMGwPEOVEMkgdA",
  "master_node" : "Nhi6xRkbThulxS2c8Gd9mQ",
  "nodes" : {
    "XBLvF10DTXW3zOTeHlH73Q" : {
      "name" : "cluster1-tran-elasticsearch1",
      "ephemeral_id" : "LT1cfQTZSnG186JtEIEIKA",
      "transport_address" : "127.0.0.1:9311",
    },
    "Nhi6xRkbThulxS2c8Gd9mQ" : {
      "name" : "cluster1-tran-elasticsearch2",
      "ephemeral_id" : "3gIZMOCxTZKvJRHy_8CO9A",
      "transport_address" : "127.0.0.1:9312",
    }
  },

Cluster2 Kibana (c2k1 localhost:5621) console shows localhost:9321 and localhost:9322

GET /_cluster/state
{
  "cluster_name" : "cluster2",
  "cluster_uuid" : "O9yPMHTERhuwNXhThaFj7Q",
  "version" : 147,
  "state_uuid" : "oWLBsWljTge-_tJT7pQiSA",
  "master_node" : "WJD1ccz3Tt673gs8saKkDA",
  "nodes" : {
    "_mGs-6vWTxSffPv3ZvAv5Q" : {
      "name" : "cluster2-tran-elasticsearch1",
      "ephemeral_id" : "sQr0StTDRcmhlccUzJoCyw",
      "transport_address" : "127.0.0.1:9321",
    },
    "WJD1ccz3Tt673gs8saKkDA" : {
      "name" : "cluster2-tran-elasticsearch2",
      "ephemeral_id" : "3VsuX92WSHCmUC0B--R9dg",
      "transport_address" : "127.0.0.1:9322",
    }
  },

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[jportner]

I think I know the problem, TIL that cookies are not isolated for different ports on the same host.

This is unfortunately a browser problem, not a Kibana problem. But you can work around it by setting a basePath for one of your Kibana instances. As long as they have different base paths, the cookies will be isolated properly.

Can you try it and confirm? If so, I think we can add some documentation for this scenario and then close out this issue.

Note, you might have to set a different basePath for both Kibana instances (if you leave one without a basePath, I'm not 100% sure if the cookies for the "root" on one will still conflict with the cookie for the "child" on the other, if that makes sense). It's worth testing.

[justincr-elastic]

What if multiple Kibana instances are behind a web proxy? Are there any implicates for ESS/ECE service proxy routing to different Kibana instances? In other words, should Kibana instances apply scope to their cookies by default?

Use cases of a proxy:

1. Chrome tab 1 => proxy:1001 => Kibana 1 => Connect to Elasticsearch nodes in cluster 1.
2. Chrome tab 2 => proxy:1002 => Kibana 2 => Connect to Elasticsearch nodes in cluster 2.

I assume sticky sessions in the proxy takes care of proxy:1001 connecting to the same Kibana 1 each time, if there are multiple Kibana instances for cluster 1. However, if that proxy can also forward proxy:1002 to Kibanas in a different cluster, then that is a similar situation that I configured.

[justincr-elastic]

Are mitigations for this issue documented in Kibana anywhere? Is this part of known issues in any release notes?

From the Kibana doc page https://www.elastic.co/guide/en/kibana/master/settings.html, looking at the description for server.basePath does not imply that I have that issue, because I am not using a proxy.

server.basePath Enables you to specify a path to mount Kibana at if you are running behind a proxy. Use the server.rewriteBasePath setting to tell Kibana if it should remove the basePath from requests it receives, and to prevent a deprecation warning at startup. This setting cannot end in a slash (/).

There are other workarounds mentioned in that Stack Overflow link (ex: append Kibana port to cookie name).

https://stackoverflow.com/questions/1612177/are-http-cookies-port-specific#:~:text=Similarly%2C%20cookies%20for%20a%20given,content%20retrieved%20via%20different%20ports.&text=Cookies%20do%20not%20provide%20isolation%20by%20port.

Does it make sense for Kibana instances to support isolating their cookies automatically, instead of manually configuring unique server.basePaths. Furthermore, I assume using server.basePath probably only works if you are using a single proxy port to access two groups of Kibana instances, and server.basePath gets appending to https://address:port/. That doesn't necessary help customers who want to use two different ports on the same proxy, so if they don't want to append something to their URLs.

[justincr-elastic]

If I understand this correctly, there are 3 variables in the URL that impact cookie isolation:

1. address
2. port
3. request base path

Use cases:

1. If address is same, different request base paths will isolate the cookies, no matter if port is same or different.

• https://proxyaddress:1001/basePath1
• https://proxyaddress:1001/basePath2
• https://proxyaddress:1002/basePath3
• https://proxyaddress:1002/basePath4

2. If address and base path is same, different ports does not isolate the cookies.

• https://proxyaddress:1001/sameBasePath
• https://proxyaddress:1002/sameBasePath

I found this issue with two localhost ports, but it would be reproducible with a proxy address too.

[jportner]

What if multiple Kibana instances are behind a web proxy? Are there any implicates for ESS/ECE service proxy routing to different Kibana instances? ... I assume sticky sessions in the proxy takes care of proxy:1001 connecting to the same Kibana 1 each time, if there are multiple Kibana instances for cluster 1. However, if that proxy can also forward proxy:1002 to Kibanas in a different cluster, then that is a similar situation that I configured.

I think you're correct, the proxy situation would have the same outcome.

Are mitigations for this issue documented in Kibana anywhere? Is this part of known issues in any release notes?

No, as I mentioned in my comment above, "Can you try it and confirm? If so, I think we can add some documentation for this scenario and then close out this issue."

If I understand this correctly, there are 3 variables in the URL that impact cookie isolation:

1. address
2. port
3. request base path

Sort of. According to the SO comment that I mentioned above, browsers determine "origin" solely on hostname/address, and they do not factor the port in. Kibana's cookie is named "sid" by default, but you can change it. So I'd say the three aspects of cookie isolation are:

1. The Kibana server hostname (or, that of a reverse proxy that is used to access Kibana)
2. The configured basePath value (which is applied to the cookie's "Path" attribute)
3. The configured xpack.security.cookieName value (which is applied to the cookie's Name)

I'll stress that this would happen with any web application, not just Kibana. It's because of how browsers behave.

Does it make sense for Kibana instances to support isolating their cookies automatically, instead of manually configuring unique server.basePaths.

It's an interesting idea to automatically change the cookie name, but I'm not sure we want to attempt to do that. (1) Some customers have other tooling in place for scans/etc and would not appreciate a breaking change that makes the cookie name dynamic, (2) I'm not sure how Kibana itself would know that the cookie name needs to be different, it's very common to have reverse proxies in place to access web servers and Kibana may always be listening on port 5601 locally for example.

I don't really think we need to make a change in Kibana to try to automatically accommodate this edge case, I think a docs change is enough.

[jportner]

It's an interesting idea to automatically change the cookie name, but I'm not sure we want to attempt to do that. ...

I'll add, (3) If you have multiple Kibanas in a HA configuration, they would all have to somehow be aware of this dynamic cookie name. Multiple Kibanas are not aware of each other today.

So I'll reiterate that this is an unfortunate edge case that can be solved with a configuration change, and we should just clearly document that.

Edit: this is probably a good page to document it: https://www.elastic.co/guide/en/kibana/current/production.html

[rahuldimri]

@jportner is there any thing pending on this. I mean updating the document ?. If you want I can look into this.

[jportner]

Hi @rahuldimri, thanks for asking!

Let me rephrase what I was suggesting above to be a bit less confusing:

You can set up two separate Kibana instances on the same host listening on different ports -- however, if you do so, this will result in authentication errors because their session cookies will conflict with each other. In this scenario, to ensure that these session cookies are properly isolated and avoid authentication errors, you can do one of the following:

• In kibana.yml, configure different server.basePath values for each Kibana instance
• In kibana.yml, configure different xpack.security.cookieName values for each Kibana instance
• Use two different hostnames to access each Kibana instance

It would probably be good to add this information in a new section in the "Use Kibana in a production environment" page in the docs. That file is located at docs/user/production-considerations/production.asciidoc.

I'm no longer working on Kibana, but if you'd like to submit a PR to update the docs it would be appreciated, and you can request a review from @azasypkin!

[legrego]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.