The solr admin UI is successfully logged in with token received from IDP and is able to access security, list collections etc. but the core creation fails with invalid PKI header. The Solr is deployed on GKE with istio proxy.
{
"textPayload": "
| MESSAGE: | Could not validate PKI header. |
|---|
",
"insertId": "xabmc3lmfdshg1akrcw",
"resource": {
"type": "k8s_container",
"labels": {
"location": "us",
"pod_name": "podname",
"cluster_name": "cluster_name",
"container_name": "solrcloud-node",
"namespace_name": "solr",
"project_id": "xxx"
}
},
"authentication": {
"class": "solr.MultiAuthPlugin",
"schemes": [{
"scheme": "bearer",
"blockUnknown":false,
"class":"solr.JWTAuthPlugin",
"adminUiScope": "api://ttt/admin",
"principalClaim":"unique_name",
"iss":"https://sts.windows.net/abc/",
"aud":"api://xyz",
"wellKnownUrl":"https://login.microsoftonline.com/abc/v2.0/.well-known/openid-configuration",
"redirectUris": "https://localhost:8983/solr/",
"clientId":"xyz",
"authorizationFlow":"code_pkce",
"trustedCertsFile":"/path/to/certificate",
"jwkCacheDur":"60",
},{
"scheme": "basic",
"blockUnknown": false,
"class": "solr.BasicAuthPlugin",
"realm":"Solr Basic Auth",
"credentials": {
"solr":"bfjbf",
},
"forwardCredentials": false
}]
},
I am using solr version 9.6.0 and solr operator 0.8.1 . Below are the logs:
2024-09-06 07:25:56.992 INFO (qtp1212191909-53-solr-solrcloud-0.solr-solrcloud-headless.solr-343) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-343] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/health params={} status=0 QTime=0
2024-09-06 07:25:57.503 INFO (OverseerThreadFactory-19-thread-1) [c:testing s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection testing
2024-09-06 07:25:57.710 INFO (OverseerStateUpdate-72062713179013124-solr-solrcloud-0.solr-solrcloud-headless.solr:8983_solr-n_0000000002) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
"core":"testing_shard1_replica_n1",
"node_name":"solr-solrcloud-0.solr-solrcloud-headless.solr:8983_solr",
"base_url":"http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr",
"collection":"testing",
"shard":"shard1",
"state":"down",
"type":"NRT",
"operation":"ADDREPLICA",
"waitForFinalState":"false"}
2024-09-06 07:25:57.837 INFO (zkCallback-13-thread-4) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testing/state.json zxid: -1] for collection [testing] has occurred - updating... (live nodes size: [1])
2024-09-06 07:25:58.139 ERROR (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from: http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr => org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447)
org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447) ~[?:?]
at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:808) ~[?:?]
at org.noggit.JSONParser.next(JSONParser.java:1013) ~[?:?]
at org.noggit.JSONParser.nextEvent(JSONParser.java:1059) ~[?:?]
at org.noggit.ObjectBuilder.(ObjectBuilder.java:85) ~[?:?]
at org.apache.solr.common.util.Utils.lambda$static$1(Utils.java:331) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:283) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:264) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:260) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.fetchPublicKeyFromRemote(PKIAuthenticationPlugin.java:367) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.getOrFetchPublicKey(PKIAuthenticationPlugin.java:227) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeaderV2(PKIAuthenticationPlugin.java:235) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:166) ~[?:?]
at org.apache.solr.security.AuthenticationPlugin.authenticate(AuthenticationPlugin.java:91) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:366) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:240) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) ~[?:?]
at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) ~[?:?]
at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) ~[?:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.20.jar:10.0.20]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-09-06 07:25:58.139 WARN (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Key is null when attempting to validate signature; skipping...
2024-09-06 07:25:58.139 WARN (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Failed to verify signature, trying after refreshing the key
2024-09-06 07:25:58.166 ERROR (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from: http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr => org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447)
org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447) ~[?:?]
at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:808) ~[?:?]
at org.noggit.JSONParser.next(JSONParser.java:1013) ~[?:?]
at org.noggit.JSONParser.nextEvent(JSONParser.java:1059) ~[?:?]
at org.noggit.ObjectBuilder.(ObjectBuilder.java:85) ~[?:?]
at org.apache.solr.common.util.Utils.lambda$static$1(Utils.java:331) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:283) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:264) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:260) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.fetchPublicKeyFromRemote(PKIAuthenticationPlugin.java:367) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeaderV2(PKIAuthenticationPlugin.java:244) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:166) ~[?:?]
at org.apache.solr.security.AuthenticationPlugin.authenticate(AuthenticationPlugin.java:91) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:366) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:240) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) ~[?:?]
at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) ~[?:?]
at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) ~[?:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.20.jar:10.0.20]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-09-06 07:25:58.166 WARN (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Key is null when attempting to validate signature; skipping...
2024-09-06 07:25:58.167 ERROR (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Could not validate PKI header.
2024-09-06 07:25:58.188 ERROR (OverseerThreadFactory-19-thread-1) [c:testing s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr => org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr/admin/cores: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html.
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr/admin/cores: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html.
Error 401 Could not validate PKI header.
HTTP ERROR 401 Could not validate PKI header.
| URI: | /solr/admin/cores |
|---|
| STATUS: | 401 |
|---|
| MESSAGE: | Could not validate PKI header. |
|---|
| SERVLET: | default |
|---|
Knowledge sharing, because documentation does not disclose this
Solr version: 9.5
source code
When enabling authentication you need to allow /admin/info/key public access, otherwise Solr will not be able to get keys from incoming request node during RequestForwarding process. Why Public Keys are not saved in zookeeper?
Also looks like solr does not pass JWT Principal with its full claims and it's looses scopes/roles(looks like a bug).
You need to add all your principals into Basic authentication access matrix so RequestForwarding will work.
Configuration MultiAuthRuleBasedAuthorizationPlugin that works for me
{
"authorization": {
"class": "solr.MultiAuthRuleBasedAuthorizationPlugin",
"permissions": [{
"collection": null,
"name": "inner-node-comm",
"path": "/admin/info/key",
"role": null
}, {
"collection": null,
"method": ["HEAD", "GET"],
"name": "k8s-probe-0",
"path": "/admin/info/system",
"role": null
}, {
"collection": null,
"method": ["HEAD", "GET"],
"name": "k8s-probe-1",
"path": "/admin/info/health",
"role": null
}, {
"collection": null,
"method": ["HEAD", "GET"],
"name": "k8s-metrics",
"path": "/admin/metrics",
"role": null
}, {
"collection": null,
"method": ["HEAD", "GET"],
"name": "k8s-zk",
"path": "/admin/zookeeper/status",
"role": null
}, {
"collection": "*",
"method": ["HEAD", "GET"],
"name": "k8s-ping",
"path": "/admin/ping",
"role": null
}, {
"collection": null,
"method": ["HEAD", "GET"],
"name": "k8s-collection",
"params": {
"action": ["LIST", "CLUSTERSTATUS"]
},
"path": "/admin/collections",
"role": null
}, {
"name": "health",
"role": null
}, {
"name": "metrics-read",
"role": null
}, {
"name": "security-read",
"role": ["admin", "k8s"]
}, {
"name": "security-edit",
"role": ["admin", "k8s"]
}, {
"name": "schema-edit",
"role": ["admin", "k8s"]
}, {
"name": "schema-read",
"role": ["admin", "k8s"]
}, {
"name": "config-read",
"role": ["admin", "k8s"]
}, {
"name": "config-edit",
"role": ["admin", "k8s"]
}, {
"name": "core-admin-edit",
"role": null
}, {
"name": "core-admin-read",
"role": null
}, {
"name": "collection-admin-read",
"role": ["admin", "k8s"]
}, {
"name": "collection-admin-edit",
"role": ["admin", "k8s"]
}, {
"name": "update",
"role": ["admin", "k8s"]
}, {
"name": "read",
"role": ["admin", "k8s"]
}, {
"name": "zk-read",
"role": ["admin", "k8s"]
}, {
"name": "all",
"role": ["admin", "k8s"]
}
],
"schemes": [{
"class": "solr.ExternalRoleRuleBasedAuthorizationPlugin",
"scheme": "Bearer"
}, {
"class": "solr.RuleBasedAuthorizationPlugin",
"scheme": "Basic",
"user-role": {
"admin": ["admin", "k8s"],
"k8s-oper": ["k8s"],
"{{JWT.(clientId|name|email|...)}}": ["admin"]
}
}
]
}
}
MultiAuthPlugin configuration
{
"authentication": {
"class": "solr.MultiAuthPlugin",
"schemes": [{
"adminUiScope": "{{AdminScope}}",
"blockUnknown": true,
"class": "solr.JWTAuthPlugin",
"issuers": [{
"clientId": "{{UIClientId}}",
"name": "admin-ui",
"wellKnownUrl": "https://{{IDP.Url}}/.well-known/openid-configuration"
}, {
"aud": "{{ApiAudience}}",
"name": "api",
"wellKnownUrl": "https://{{IDP.Url}}/.well-known/openid-configuration"
}
],
"principalClaim": "client_id",
"redirectUris": "https://localhost:8983/solr/",
"rolesClaim": "scope",
"scheme": "Bearer",
"scope": "{{Scopes that will act as roles}}"
}, {
"blockUnknown": false,
"class": "solr.BasicAuthPlugin",
"credentials": {
"admin": "...",
"k8s-oper": "..."
},
"forwardCredentials": false,
"realm": "Solr",
"scheme": "Basic"
}
]
},
}
P.s. "blockUnknown": false, should be true, but Prometheus Exporter does NOT send basic authentication configured with basicauth variable.
Anyone looking for SolrPrometheusExporter 9.6 or older and having problems with Basic Auth and "blockUnknown": true upgrade exporter to 9.7 and change SolrPrometheusExporter CRD:
customKubeOptions:
podOptions:
envVars:
- name: CREDENTIALS
valueFrom:
secretKeyRef:
key: {{keyName with combined `username:password` format }}
name: {{secret name}}
image:
repository: library/solr
tag: '9.7'
Command source
Solr Exporter source
Yeah, something smells "off" here. Our dispatch code has a pretty explicit exception for /admin/info/key 🤔
In any case, it sounds like some folks are able to get JWT+basic multi-auth working without issue in Kubernetes. So I suspect we can close this out?
Why Public Keys are not saved in zookeeper?
Others might easily see good reasons not to do this that I'm missing. Perhaps theres some security implication?
But it's an interesting idea. It kindof makes sense conceptually to have nodes put their public-key as the "content" of their live-node entry. And it'd save a lot of "fetch the public key" requests to boot. Maybe worth raising on the dev@ list to see what people think...
apache
