clickhouse-java
clickhouse-java copied to clipboard
Problem when send batch with compression
Hello, when i send compression like in the example using diferents compressions(gzip, brotli)
ClickHouseStatement sth = connection.createStatement();
sth
.write()
.sql("INSERT INTO default.my_table (a,b,c)")
.data(new MyCustomInputStream(), ClickHouseFormat.JSONEachRow)
.dataCompression(ClickHouseCompression.brotli)
.addDbParam(ClickHouseQueryParam.MAX_PARALLEL_REPLICAS, 2)
.send();
i get the next error
inflateReset failed: data error. (ZLIB INFLATE FAILED)
I tried to change the compression level of zlib but nothing
I don't quite understand why it uses zlib if I mark as gzip compression. But apart from that I can't get it to insert any element into the database when I indicate the type of compression
Your data needs to be compressed before sending to server. If your input stream is just about plain text, can you comment data compression part and see if it will work?
inflateReset failed: data error. (ZLIB INFLATE FAILED)
very old CH ? select version()
maybe the problem is your CH version?,please check it
My version of clickhouse is v21.9.2.17-stable, the last version i think. i am trying with compress the stream using
ClickHouseConnection con = new ClickHouseConnectionImpl(clickhouseConfig.getUrl()); st = con.createStatement(); wri = st.write() .addDbParam(ClickHouseQueryParam.ENABLE_HTTP_COMPRESSION, enableHttpCompression)
.dataCompression(ClickHouseCompression.valueOf(compression))
.table(nameTable)
.format(ClickHouseFormat.JSONEachRow);
wri.data(new GzipCompressingEntity(new InputStreamEntity(stream, -1)).getContent()).send();
But it dont work.
if I try to send the stream without compression, it works as expected.
I have also tried putting in the url jdbc: clickhouse: // localhost: 8123 / {name.db}? Decompress=1 but it doesn't insert anything into the database either.
I have also tried to compress using this syntax
GZIPOutputStream zos = null; ByteArrayOutputStream rstBao = new ByteArrayOutputStream(); try { zos = new GZIPOutputStream(rstBao); zos.write(stream.toString().getBytes(StandardCharsets.UTF_8)); zos.close(); } catch (IOException e) { e.printStackTrace(); } InputStream is = new ByteArrayInputStream(rstBao.toByteArray());
wri.data (is) .send ();
But dont work. i get the exception : DB::ParsingException: Cannot parse input: expected '{' before: 'java.io.ByteArrayInputStream@333159da': While executing JSONEachRowRowInputFormat: (at row 1)
the configuration that I am using is
url: "jdbc:clickhouse://localhost:8123/test" nameTable: "con" httpCompression: "gzip" enableHttpCompression: "true"