quarkus
quarkus copied to clipboard
quarkusio
Quarkus version change from 2.10.1.Final to 2.11.1.Final has breaking changes with Percona SqlProxy
Describe the bug
Quarkus with hibernate reactive and mysql database configured with a version to 2.11.1 version of quarkus allowed data retrieval from Percona SqlProxy. It worked very well in fact. As of 2.11.1 I'm not able to retrieve data.
Expected behavior
Executing queries writen using hibernate reactive with panache should continue to function as they did in version 2.10.1
Actual behavior
a) on the client Failed to execute statement [select terminalen0_.TERMINAL_ID as terminal1_22_, terminalen0_.COMPANY_ID as company_2_22_, terminalen0_.NAME as name3_22_, terminalen0_.UUID as uuid4_22_ from TERMINAL terminalen0_ where TERMINAL_ID=?]: could not execute query: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable: Pending requests failed to be sent due to connection has been closed.
b) on SqlProxy itself
2022-08-19 15:20:29 MySQL_Session.cpp:3364:handler___status_NONE_or_default(): [ERROR] Unexpected packet from client 127.0.0.1 . Session_status: 21 , client_status: 0 Disconnecting it 2022-08-19 15:20:29 MySQL_Session.cpp:149:kill_query_thread(): [WARNING] KILL CONNECTION 1254813 on 10.2.5.21:3306
How to Reproduce?
- configure a mysql cluster with a Percona SqlProxy
- configure a quarkus test to retrieve data via reactive hibernate panache
- use verstion 2.10.1 of quarkus to verify the data is returned.
- upgrade to version 2.11.1 and observer the test will error as above.
Output of uname -a or ver
Linux locus-dev01 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Output of java -version
openjdk version "11.0.16" 2022-07-19 OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
GraalVM version (if different from Java)
No response
Quarkus version or git rev
2.11.1
Build tool (ie. output of mvnw --version or gradlew --version)
No response
Additional information
Only breaks when going through the percona SqlProxy
![quarkus-bot[bot] avatar](https://avatars.githubusercontent.com/in/90234?v=4 "Published by a pub.dev verified publisher"){kind=small}
Can you provide a test project reproducing the error? Also, as I have never used Percona SqlProxy, maybe a Docker compose file can help in setting up the environment to reproduce this issue
I would be extremely difficult for me to provide a test project that would have 3 mysql servers in a cluster and another 3 percona sql proxies. The setup of a proxy itself is no walk in the park.
What about taking the what did we change between versions route? I have feeling it may just be a configuration default that we may be able to test reverting.
We use the Reactive Client drivers from Vert.x - the most likely "change" was a version upgrade in these drivers.
Do you think you could test by overriding the vert.x version for client? Unfortunately the coupling Quarkus/Vert.x is quite high so it might not be possible to get a functioning application with a different driver, but provided you're only changing some micro versions it might work well enough to narrow this problem down.
cc/ @tsegismont
On 22 Aug 2022, at 17:08, Sanne Grinovero @.***> wrote:
We use the Reactive Client drivers from Vert.x - the most likely "change" was a version upgrade in these drivers.
Do you think you could test by overriding the vert.x version for client? Unfortunately the coupling Quarkus/Vert.x is quite high so it might not be possible to get a functioning application with a different driver, but provided you're only changing some micro versions it might work well enough to narrow this problem down.
cc/ @tsegismont https://github.com/tsegismont — Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1222572054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVBAOWGWXMDJIMNMSMTEFDV2OQXDANCNFSM57BD4YQA. You are receiving this because you authored the thread.
I’ll see if I can do that. What’s the easiest way to do an override the vert.x version? Just in the build file itself?
In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-mysql-client</artifactId>
<version>something-something</version>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-bom</artifactId>
<version>${quarkus.platform.version}</version>
<scope>import</scope>
<type>pom</type>
</dependency>
Sorry I took me some time because we are on Gradle.
implementation("io.vertx:vertx-mysql-client:4.3.0"){ isForce = true } Seems to be the culprit
On 22 Aug 2022, at 17:19, Sanne Grinovero @.***> wrote:
In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence:
<dependencyManagement> <dependencies> <dependency> <groupId>io.vertx</groupId> <artifactId>vertx-mysql-client</artifactId> <version>something-something</version> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-bom</artifactId> <version>${quarkus.platform.version}</version> <scope>import</scope> <type>pom</type> </dependency>— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1222588319, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVBAOQEXM3GKCGHPVWN5PDV2OSBNANCNFSM57BD4YQA. You are receiving this because you authored the thread.
This might be related to pipelining (https://github.com/eclipse-vertx/vertx-sql-client/pull/1168).
I am trying to reproduce atm
I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:
- with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received
- with Quarkus 2.10.2, the client waits for response ok packet before sending close packet
Great that you found a reason, haven’t got clue obviously how it can be changed/fixed if it’s broken or sql proxy just can’t cope - Lol
On 23 Aug 2022, at 17:35, Thomas Segismont @.***> wrote:
I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:
with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received with Quarkus 2.10.2, the client waits for response ok packet before sending close packet — Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224319209, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA. You are receiving this because you authored the thread.
I'm trying to figure out a workaround for now. I'll get in touch with MySQL proxy community.
Le mar. 23 août 2022 à 20:01, harpermj66 @.***> a écrit :
Great that you found a reason, haven’t got clue obviously how it can be changed/fixed if it’s broken or sql proxy just can’t cope - Lol
On 23 Aug 2022, at 17:35, Thomas Segismont @.***> wrote:
I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:
with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received with Quarkus 2.10.2, the client waits for response ok packet before sending close packet — Reply to this email directly, view it on GitHub < https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224319209>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA . You are receiving this because you authored the thread.
— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224489631, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALOLNWHBKGBKVXBX2OKZGLV2UGYXANCNFSM57BD4YQA . You are receiving this because you were mentioned.Message ID: @.***>
I reached out to Percona because we have support with them initially. They did say we were their first client to mention reactive hibernate so they were keen to set up a meeting. I can ask my contact at percona to link up with you if necessary.
On 24 Aug 2022, at 09:32, Thomas Segismont @.***> wrote:
I'm trying to figure out a workaround for now. I'll get in touch with MySQL proxy community.
Le mar. 23 août 2022 à 20:01, harpermj66 @.***> a écrit :
Great that you found a reason, haven’t got clue obviously how it can be changed/fixed if it’s broken or sql proxy just can’t cope - Lol
On 23 Aug 2022, at 17:35, Thomas Segismont @.***> wrote:
I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:
with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received with Quarkus 2.10.2, the client waits for response ok packet before sending close packet — Reply to this email directly, view it on GitHub < https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224319209>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA . You are receiving this because you authored the thread.
— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224489631, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALOLNWHBKGBKVXBX2OKZGLV2UGYXANCNFSM57BD4YQA . You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1225374029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVBAOUGDCSQFACH4AV34K3V2XM2FANCNFSM57BD4YQA. You are receiving this because you authored the thread.
Sure. My address is [github id] at [redhat] dot com.
Le mer. 24 août 2022 à 10:39, harpermj66 @.***> a écrit :
I reached out to Percona because we have support with them initially. They did say we were their first client to mention reactive hibernate so they were keen to set up a meeting. I can ask my contact at percona to link up with you if necessary.
On 24 Aug 2022, at 09:32, Thomas Segismont @.***> wrote:
I'm trying to figure out a workaround for now. I'll get in touch with MySQL proxy community.
Le mar. 23 août 2022 à 20:01, harpermj66 @.***> a écrit :
Great that you found a reason, haven’t got clue obviously how it can be changed/fixed if it’s broken or sql proxy just can’t cope - Lol
On 23 Aug 2022, at 17:35, Thomas Segismont @.***> wrote:
I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:
with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received with Quarkus 2.10.2, the client waits for response ok packet before sending close packet — Reply to this email directly, view it on GitHub <
https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224319209
,
or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA
. You are receiving this because you authored the thread.
— Reply to this email directly, view it on GitHub < https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1224489631 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AALOLNWHBKGBKVXBX2OKZGLV2UGYXANCNFSM57BD4YQA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub < https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1225374029>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKVBAOUGDCSQFACH4AV34K3V2XM2FANCNFSM57BD4YQA . You are receiving this because you authored the thread.
— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1225381309, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALOLNTU2ZCCDONFBUHZLULV2XNUHANCNFSM57BD4YQA . You are receiving this because you were mentioned.Message ID: @.***>
I believe we can keep it open until the fixed version is integrated in Quarkus.
Great, this saved me. Thank you. This error gave me no stacktrace even in TRACE level. After trying hours with onFailure just after the query execution, found this issue here just before creating a new one. For anybody trying to use proxysql with the latest 2.12.final version Just downgrade the vertx mysql client as mentioned above. Below version is working.
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-mysql-client</artifactId>
<version>4.2.7</version>
</dependency>
Fixed upstream in https://github.com/eclipse-vertx/vertx-sql-client/pull/1236 (Vert.x 4.3.4)
Thanks for the update @tsegismont.
Just out of curiosity, when is Vert.x 4.3.4 planned?
Great, thank you!
On 28 Sep 2022, at 06:05, Georgios Andrianakis @.***> wrote:
Thanks for the update @tsegismont https://github.com/tsegismont.
Just out of curiosity, when is Vert.x 4.3.4 planned?
— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1260392364, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVBAOQM3VBYAHQGBKN4UVTWAPG2FANCNFSM57BD4YQA. You are receiving this because you authored the thread.
@geoand very soon, beginning of next week maybe, right @vietj?
@harpermj66 you're welcome. Any news from your contacts at Percona? Are they still willing to talk about this?
Embarressingly not willing to engage for reasons which I don't really go along with. Is there something they should be doing for future proofing?
No worries, it is fine to not support pipelining. I can think of reasons why ProxySQL doesn't (shared connections).
4.2.7 is a version that works with Quarks 2.11.1 and is the one bundled in 2.10.1 The next one I could find was 4.3.0 which didn’t work with 2.11.1
On 22 Aug 2022, at 17:19, Sanne Grinovero @.***> wrote:
In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence:
<dependencyManagement> <dependencies> <dependency> <groupId>io.vertx</groupId> <artifactId>vertx-mysql-client</artifactId> <version>something-something</version> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-bom</artifactId> <version>${quarkus.platform.version}</version> <scope>import</scope> <type>pom</type> </dependency>— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/27386#issuecomment-1222588319, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVBAOQEXM3GKCGHPVWN5PDV2OSBNANCNFSM57BD4YQA. You are receiving this because you authored the thread.
I don't know where that comment came from 4 hours ago :-) I've already been using 2.13.1.Final and had no issues but we'll do a bit more testing.
Hi everyone!
We are developing a Quarkus based app (with reactive Panache stack), and we cannot make it work with version 2.13.3.Final (or with any version equals or greater than 2.11.0) with a database that sits behind a SQL proxy. It works perfectly when we connect with a DB without a proxy in the middle, or when we use up to 2.10.4 version. We can override vertx mysql driver to version 4.2.7, but from 2.13.0 that override produces runtime errors.
The error we get when we execute transactions against a DB behind a SQL proxy is:
(vert.x-eventloop-thread-0) Unexpected error: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
[Exception 0] io.vertx.core.impl.NoStackTraceThrowable: Fail to read any response from the server, the underlying connection might get lost unexpectedly.
[Exception 1] io.vertx.core.impl.NoStackTraceThrowable: Connection is not active now, current status: CLOSED
Is there any workaround to this problem?
Thanks!
@diegoherrojo thanks for reporting.
Can you share a simple reproducer? We have a test upstream which verifies connectivity through ProxySQL, so I'm confused.
Can you verify with mvn dependency:tree if your project uses vertx-mysql-client 4.3.4 ?
Thanks for your quick response @tsegismont!
Yes, I verified that we are using vertx-mysql-client 4.3.4.
Let me create a simple test to share here.
Hi everyone!
@tsegismont I created a small app reproducing the error with Quarkus 2.13.2.Final.
In order to run it, please unzip test.zip and then execute run.sh. Docker compose, maven and jdk 17 are needed for the test.
The test spins up a MySQL and a ProxySQL and then executes 2 times a POST to the app API (which creates some entities in DB):
- Once connecting directly to MySQL (successful).
- Second time connecting through ProxySQL (fails).
Thanks!