clickhouse-java icon indicating copy to clipboard operation
clickhouse-java copied to clipboard
verified publisher icon ClickHouse

HTTP compress enabled and magic wrong

Open maskshell opened this issue 2 years ago • 3 comments

Describe the bug

When enabled http compress(<enable_http_compression>1</enable_http_compression>) in server user settings on ClickHouse 23.8, JDBC will raise an error like this:

Error log

eclipse.buildId=unknown java.version=17.0.4.1 java.vendor=Eclipse Adoptium BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en Framework arguments: -keyring /Users/x/.eclipse_keyring Command-line arguments: -os macosx -ws cocoa -arch x86_64 -keyring /Users/canger/.eclipse_keyring

org.jkiss.dbeaver.model Error Thu Sep 14 18:45:41 CST 2023 Magic is not correct - expect [-126] but got [-63], server ClickHouseNode [uri=http://x.x.x.x:8123/system, > options={socket_timeout=600000,use_server_time_zone=false,use_time_zone=false}]@-2029288357

java.sql.BatchUpdateException: Magic is not correct - expect [-126] but got [-63], server ClickHouseNode [uri=http://x.x.x.x:8123/system, > options={socket_timeout=600000,use_server_time_zone=false,use_time_zone=false}]@-2029288357 at com.clickhouse.jdbc.SqlExceptionUtils.batchUpdateError(SqlExceptionUtils.java:109) at com.clickhouse.jdbc.internal.SqlBasedPreparedStatement.executeAny(SqlBasedPreparedStatement.java:208) at com.clickhouse.jdbc.internal.SqlBasedPreparedStatement.execute(SqlBasedPreparedStatement.java:419) at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCPreparedStatementImpl.execute(JDBCPreparedStatementImpl.java:261) at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCPreparedStatementImpl.executeStatement(JDBCPreparedStatementImpl.java:205) at org.jkiss.dbeaver.model.impl.jdbc.cache.JDBCObjectCache.loadObjects(JDBCObjectCache.java:109) at org.jkiss.dbeaver.model.impl.jdbc.cache.JDBCObjectCache.getAllObjects(JDBCObjectCache.java:73) at org.jkiss.dbeaver.ext.clickhouse.model.ClickhouseDataSource.(ClickhouseDataSource.java:72) at com.dbeaver.db.clickhouse.model.ClickhouseProDatasource.(ClickhouseProDatasource.java:40) at com.dbeaver.db.clickhouse.ClickhouseProDataSourceProvider.openDataSource(ClickhouseProDataSourceProvider.java:36) at org.jkiss.dbeaver.registry.DataSourceDescriptor.openDataSource(DataSourceDescriptor.java:1206) at org.jkiss.dbeaver.registry.DataSourceDescriptor.connect(DataSourceDescriptor.java:1071) at org.jkiss.dbeaver.runtime.jobs.ConnectJob.run(ConnectJob.java:77) at org.jkiss.dbeaver.runtime.jobs.ConnectJob.runSync(ConnectJob.java:104) at org.jkiss.dbeaver.ui.actions.datasource.DataSourceHandler.connectToDataSource(DataSourceHandler.java:115) at org.jkiss.dbeaver.ui.actions.datasource.UIServiceConnectionsImpl.initConnection(UIServiceConnectionsImpl.java:87) at org.jkiss.dbeaver.model.navigator.DBNDataSource.initializeNode(DBNDataSource.java:158) at org.jkiss.dbeaver.model.navigator.DBNDatabaseNode.getChildren(DBNDatabaseNode.java:225) at org.jkiss.dbeaver.model.navigator.DBNDatabaseNode.getChildren(DBNDatabaseNode.java:1) at org.jkiss.dbeaver.model.navigator.DBNUtils.getNodeChildrenFiltered(DBNUtils.java:80) at org.jkiss.dbeaver.ui.navigator.database.load.TreeLoadService.evaluate(TreeLoadService.java:49) at org.jkiss.dbeaver.ui.navigator.database.load.TreeLoadService.evaluate(TreeLoadService.java:1) at org.jkiss.dbeaver.ui.LoadingJob.run(LoadingJob.java:88) at org.jkiss.dbeaver.ui.LoadingJob.run(LoadingJob.java:72) at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

ClickHouse server

  • ClickHouse Server version: 23.8
  • ClickHouse Server non-default settings, if any: User setting: <enable_http_compression>1</enable_http_compression>
  • JDBC version: 0.4.6, 0.3.2-patch11

Others

When switch the property compress_algorithm's value from default LZ4 to ZSTD, it OK to connect and operating. (Only 0.4.6, 0.3.2 not add the ZSTD support yet)

maskshell avatar Sep 14 '23 10:09 maskshell

It is same from Java code.

maskshell avatar Sep 14 '23 11:09 maskshell

Thanks for the report. LZ4 is currently hard-coded to use CityHash for checksum as shown below. Before the issue has been fixed, can you set custom_http_params to enable_http_compression=0?

https://github.com/ClickHouse/clickhouse-java/blob/e059567fa661aba66d740b1e0595c6ae30fdc895/clickhouse-data/src/main/java/com/clickhouse/data/stream/Lz4InputStream.java#L85-L89

zhicwu avatar Sep 21 '23 00:09 zhicwu

Hi @zhicwu ! Any news on this?

I'm trying to use jdbc driver. - I spent several hours debugging before I realized what was happening. And then several hours of sorting through options in order to find a combination that will work.

Do you understand that your client does not work at all if the server is configured with <enable_http_compression>1</enable_http_compression>? The problems:

  • "lz4" : "Magic is not correct..." exception
  • "deflate": java.sql.SQLException: java.io.EOFException: Unexpected end of ZLIB input stream
  • "gzip": java.sql.SQLException: Input is not in the .gz format

Only these two options only worked, thanks to @maskshell for the tip about zstd (set jdbc driver property):

  • setProperty("compress_algorithm", "zstd") // with luben's zstd-jni
  • setProperty("compress_algorithm", "none") // this works! But this for some reason not - custom_http_params to enable_http_compression=0

Gdimitron avatar Mar 19 '24 00:03 Gdimitron