kyuubi [Bug] get errors when i use "curl -v -X 'POST' --negotiate -u : 'https://hostname:10099/api/v1/sessions' to start a session

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug

i used curl to post rest request for creating a sessions , but both http and https didn't work.

curl -v debug info show as follows:

# curl -v -X 'POST' --negotiate -u : 'https://myhostname:10099/api/v1/sessions'

About to connect() to myhostname port 10099 (#0)
Trying 10.30.17.237...
Connected to myhostname (10.30.17.237) port 10099 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
NSS error -12263 (SSL_ERROR_RX_RECORD_TOO_LONG)
SSL received a record that exceeded the maximum permissible length.
Closing connection 0 curl: (35) SSL received a record that exceeded the maximum permissible length.

# curl -v -X 'POST' --negotiate -u : 'http://myhostname:10099/api/v1/sessions'

About to connect() to myhostname port 10099 (#0)
Trying 10.30.17.237...
Connected to myhostname (10.30.17.237) port 10099 (#0)

POST /api/v1/sessions HTTP/1.1 User-Agent: curl/7.29.0 Host: myhostname:10099 Accept: /

< HTTP/1.1 500 Internal Server Error < Date: Wed, 18 Oct 2023 01:49:37 GMT < Content-Type: application/json < Content-Length: 16 < Server: Jetty(9.4.52.v20230823) <

Connection #0 to host myhostnameleft intact {"message":null}

It seems like that jetty server is not configed with ssl on, but i don't know how to fix it.

Thanks very much for your helps!

Affects Version(s)

master/1.7.0

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

kyuubi.authentication=KERBEROS
kyuubi.kinit.principal=hive/[email protected]
kyuubi.kinit.keytab=/etc/security/keytabs/hive.keytab

kyuubi.backend.server.event.loggers=JSON
kyuubi.backend.server.event.json.log.path=/var/log/kyuubi/event

kyuubi.ha.addresses=nm-bigdata-030017237.ctc.local:2181,nm-bigdata-030017238.ctc.local:2181,nm-bigdata-030017239.ctc.local:2181
kyuubi.ha.namespace=kyuubi_root


kyuubi.frontend.rest.bind.host=10.30.17.237
kyuubi.frontend.bind.host=10.30.17.237
kyuubi.frontend.protocols=REST,THRIFT_BINARY

kyuubi.spnego.keytab=/etc/security/keytabs/HTTP.keytab
kyuubi.spnego.principal=HTTP/[email protected]


kyuubi.server.administrators=anonymous

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

[x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
[ ] No. I cannot submit a PR at this time.

Oct 18 '23 07:10 YetiCuzMountain

Hello @YetiCuzMountain, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.

Oct 18 '23 08:10 github-actions[bot]

When I traced the related code of the kyuubi-admin script, I found the configuration “kyuubi.ctl.rest.base.url” & “ kyuubi.ctl.rest.auth.schema”

it seems that the scipt kyuubi-ctl is used to add a http request header of 'Autehtication : Negotiate : ${token}'.

And i get error as follows :

Exception in thread "main" org.apache.kyuubi.client.exception.KyuubiRestException: Failed to generate spnego auth header for 10.10.37.137
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:50)
        at org.apache.kyuubi.client.KyuubiRestClient.getAuthHeader(KyuubiRestClient.java:137)
        at org.apache.kyuubi.client.SessionRestApi.listSessions(SessionRestApi.java:41)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.$anonfun$doRun$1(ListSessionCommand.scala:36)
        at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:34)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:29)
        at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47)
        at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46)
        at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79)
        at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87)
        at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala)
Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateToken(SpnegoAuthHeaderGenerator.java:80)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:46)
        ... 11 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.doGenerateToken(SpnegoAuthHeaderGenerator.java:97)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.lambda$generateToken$0(SpnegoAuthHeaderGenerator.java:80)
        ... 15 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:466)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
        ... 19 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 25 more

Oct 19 '23 02:10 YetiCuzMountain

When I traced the related code of the kyuubi-admin script, I found the configuration “kyuubi.ctl.rest.base.url” & “ kyuubi.ctl.rest.auth.schema”

it seems that the scipt kyuubi-ctl is used to add a http request header of 'Autehtication : Negotiate : ${token}'.

And i get error as follows :

Exception in thread "main" org.apache.kyuubi.client.exception.KyuubiRestException: Failed to generate spnego auth header for 10.10.37.137
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:50)
        at org.apache.kyuubi.client.KyuubiRestClient.getAuthHeader(KyuubiRestClient.java:137)
        at org.apache.kyuubi.client.SessionRestApi.listSessions(SessionRestApi.java:41)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.$anonfun$doRun$1(ListSessionCommand.scala:36)
        at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:34)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:29)
        at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47)
        at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46)
        at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79)
        at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87)
        at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala)
Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateToken(SpnegoAuthHeaderGenerator.java:80)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:46)
        ... 11 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.doGenerateToken(SpnegoAuthHeaderGenerator.java:97)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.lambda$generateToken$0(SpnegoAuthHeaderGenerator.java:80)
        ... 15 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:466)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
        ... 19 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 25 more

Perhaps i have to add --spnegoHost during using the script kyuubi-ctl so as to get a correct kerberos principal.

Oct 19 '23 03:10 YetiCuzMountain

I used the command

bin/kyuubi-ctl list session -b --hostUrl http://myhostname:10099/api --version v1 --authSchema spnego  --spnegoHost  myhostname

to list session, and the same error occured even though the autication http header was attached to the http request.

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) ~[?:1.8.0_251]
        at sun.security.ssl.InputRecord.read(InputRecord.java:527) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400) ~[?:1.8.0_251]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140) ~[httpclient-4.5.14.jar:4.5.14]
        at **org.apache.kyuubi.client.RestClient.doRequest**(RestClient.java:181) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.client.RestClient.get(RestClient.java:80) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.client.RestClient.get(RestClient.java:74) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
        at org.apache.kyuubi.client.RetryableRestClient.invoke(RetryableRestClient.java:82) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at com.sun.proxy.$Proxy22.get(Unknown Source) ~[?:?]
        at org.apache.kyuubi.client.SessionRestApi.listSessions(SessionRestApi.java:41) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.$anonfun$doRun$1(ListSessionCommand.scala:36) ~[kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:34) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:29) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]

Oct 19 '23 04:10 YetiCuzMountain

After my exploration, I guess, some kyuubi fe services , such as KyuubiTHttpFrontendService and TBinaryFrontedService, support SSL, but others does not seem to support SSL, e.g KyuubiRestFrontendService.

The existing configuration are all about KyuubiTHttpFrontendService and KyuubiTHttpFrontendService , such as 'kyuubi.frontend.ssl.keystore.path' & 'kyuubi.frontend.thrift.http.ssl.keystore.password'.

Oct 19 '23 04:10 YetiCuzMountain

Perhaps i have to add --spnegoHost during using the script kyuubi-ctl so as to get a correct kerberos principal.

The server principal name, a.k.a. SPN, is definitely required for Kerberos/SPNego authentication mechanism.

Oct 19 '23 05:10 pan3793

Usually, we deploy Kyuubi with plain HTTP behind the Nginx which uses HTTPs, do you plan to expose Kyuubi directly to the end users with HTTPs protocols?

Oct 19 '23 05:10 pan3793

Usually, we deploy Kyuubi with plain HTTP behind the Nginx which uses HTTPs, do you plan to expose Kyuubi directly to the end users with HTTPs protocols?

yes, no nginx , directly depolyed the 10099

Oct 19 '23 05:10 YetiCuzMountain

Thanks for your instruction. i use this command : " # bin/kyuubi-ctl list session -b --hostUrl http://myhostname:10099--version v1 --authSchema spnego --spnegoHost myhostname" and get correct response.

kyuubi-ctl --hostUrl , no "/api"

Oct 19 '23 06:10 YetiCuzMountain

It seems that it will not be automatically added -H 'Authentication : Negotiate : token ' when using the 'curl' command , unless use kyuubi-ctl script.

Oct 19 '23 06:10 YetiCuzMountain

It seems that it will not be automatically added -H 'Authentication : Negotiate : token ' when using the 'curl' command

it's unexpected behavior, but I have no idea what happened inside curl

Oct 19 '23 06:10 pan3793

I fixed the 'kyuubi-ctl' script for rest submission in condition of SSL + SPNEGO.

What should I do next？

Oct 26 '23 02:10 YetiCuzMountain

How did you fix that? is it a usage issue or requires code changes?

Oct 26 '23 02:10 pan3793

How did you fix that? is it a usage issue or requires code changes?

Perhaps it requires code changes, in which i add some keystore Configs during initializing a JettyServer object.

Oct 27 '23 01:10 YetiCuzMountain

I have fixed this problem in #5548. Thanks greatly for your instructions!

Oct 27 '23 08:10 YetiCuzMountain

when use curl like curl -X 'GET' --negotiate -u : "http://XXX:10099/api/v1/sessions" ，you must confirm curl feature support GSS-API/SSPI Or SPNEGO Older versions of curl that support GSS-Negotiate cannot work properly.

Apr 28 '24 07:04 jiaoqingbo