trino-python-client [Trino-python-client] query failures on v454 Trino with Ingress-HAP as proxy in the middle

[Trino-python-client] query failures on v454 Trino with Ingress-HAP as proxy in the middle

Open vinay-kl opened this issue 5 months ago • 1 comments

Queries submitted via Trino-python-client started failing when we migrated to Trino v454 version with no changes on client-side or server side, but the same was working as expected on our different version clusters v474 & v448

/Users/300070018/PycharmProjects/trino-python-client-main/venv/bin/python /Users/300070018/PycharmProjects/trino-python-client-main/local-tests/local-test.py 
Traceback (most recent call last):
  File "/Users/300070018/PycharmProjects/trino-python-client-main/local-tests/local-test.py", line 12, in <module>
    cur.execute("<query>")
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/dbapi.py", line 589, in execute
    self._iterator = iter(self._query.execute())
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/client.py", line 805, in execute
    status = self._request.process(response)
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/client.py", line 606, in process
    self.raise_response_error(http_response)
  File "/Users/300070018/PycharmProjects/trino-python-client-main/trino/client.py", line 597, in raise_response_error
    raise exceptions.HttpError(
trino.exceptions.HttpError: error 400: b'
<html>

<head>
	<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1" />
	<title>Error 400 Invalid Authority</title>
</head>

<body>
	<h2>HTTP ERROR 400 Invalid Authority</h2>
	<table>
		<tr>
			<th>URI:</th>
			<td>https:/v1/statement</td>
		</tr>
		<tr>
			<th>STATUS:</th>
			<td>400</td>
		</tr>
		<tr>
			<th>MESSAGE:</th>
			<td>Invalid Authority</td>
		</tr>
	</table>
	<h3>Caused by:</h3>
	<pre>org.eclipse.jetty.http.BadMessageException: 400: Invalid Authority
	at org.eclipse.jetty.http.HttpCompliance.assertAllowed(HttpCompliance.java:421)
	at org.eclipse.jetty.http.HttpCompliance.checkHttpCompliance(HttpCompliance.java:407)
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.onRequest(HttpStreamOverHTTP2.java:97)
	at org.eclipse.jetty.http2.server.internal.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:145)
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:102)
	at org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1119)
	at org.eclipse.jetty.http2.server.internal.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120)
	at org.eclipse.jetty.http2.HTTP2Connection.onHeaders(HTTP2Connection.java:239)
	at org.eclipse.jetty.http2.parser.BodyParser.notifyHeaders(BodyParser.java:115)
	at org.eclipse.jetty.http2.parser.HeadersBodyParser.onHeaders(HeadersBodyParser.java:269)
	at org.eclipse.jetty.http2.parser.HeadersBodyParser.onHeaders(HeadersBodyParser.java:264)
	at org.eclipse.jetty.http2.parser.HeadersBodyParser.parse(HeadersBodyParser.java:208)
	at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
	at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
	at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
	at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:342)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:210)
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:442)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
	at org.eclipse.jetty.util.thread.MonitoredQueuedThreadPool$1.run(MonitoredQueuedThreadPool.java:73)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:1570)
	Suppressed: org.eclipse.jetty.util.StaticException: Unconsumed request content
</pre>

</body>

</html>'

Our N/W stack is client -> Ingress/HAP-> backend/server

Upon further investigation slack-thread: https://trinodb.slack.com/archives/CGB0QHWSW/p1724955298573889

We found out the following

trino-python-client makes use of requests lib which sends HTTP/1.1 only
Ingress/HAP (HAProxy Ingress Controller v1.8.12) automatically upgrades to HTTP/2.0
due to recent change at server wherein jetty version was bumped/upgraded, requires mandatory :authority header which isn't passed along when Ingress/HAP upgrades, which is resulting in 400 Invalid Authority error https://github.com/haproxy/haproxy/issues/2592 https://github.com/jetty/jetty.project/issues/9436

These are the following steps which can be performed to make it work

within the requests lib of adapters.py class file, under send function of class HTTPAdapter(BaseAdapter):, changing the line to url = request.url makes it work as it internally specifies the url with absolute one instead of relative path

DEBUG:urllib3.connectionpool:https://host:443 "POST https://host:443/v1/statement HTTP/1.1"

Or make an explicit entry in Ingress Config to add http-header

haproxy.org/backend-config-snippet: |
      http-request set-uri https://%[req.hdr(host)]%[pathq]

Or add the following in Ingress YAML, which enforces HTTP/1.1 protocol by disabling default alpn h2,http1.1 negotiation

haproxy.org/server-proto: h1

Sep 03 '24 10:09 vinay-kl

trino-python-client trino-python-client copied to clipboard

[Trino-python-client] query failures on v454 Trino with Ingress-HAP as proxy in the middle

trino-python-client
trino-python-client copied to clipboard