clickhouse-java icon indicating copy to clipboard operation
clickhouse-java copied to clipboard

Problem when send batch with compression

Open Lucgarg opened this issue 4 years ago • 4 comments

Hello, when i send compression like in the example using diferents compressions(gzip, brotli)

ClickHouseStatement sth = connection.createStatement(); sth .write() .sql("INSERT INTO default.my_table (a,b,c)") .data(new MyCustomInputStream(), ClickHouseFormat.JSONEachRow) .dataCompression(ClickHouseCompression.brotli)
.addDbParam(ClickHouseQueryParam.MAX_PARALLEL_REPLICAS, 2) .send();

i get the next error

inflateReset failed: data error. (ZLIB INFLATE FAILED)

I tried to change the compression level of zlib but nothing

I don't quite understand why it uses zlib if I mark as gzip compression. But apart from that I can't get it to insert any element into the database when I indicate the type of compression

Lucgarg avatar Sep 14 '21 11:09 Lucgarg

Your data needs to be compressed before sending to server. If your input stream is just about plain text, can you comment data compression part and see if it will work?

zhicwu avatar Sep 14 '21 11:09 zhicwu

inflateReset failed: data error. (ZLIB INFLATE FAILED)

very old CH ? select version()

den-crane avatar Sep 14 '21 12:09 den-crane

maybe the problem is your CH version?,please check it

juliito19 avatar Sep 14 '21 13:09 juliito19

My version of clickhouse is v21.9.2.17-stable, the last version i think. i am trying with compress the stream using

ClickHouseConnection con = new ClickHouseConnectionImpl(clickhouseConfig.getUrl()); st = con.createStatement(); wri = st.write() .addDbParam(ClickHouseQueryParam.ENABLE_HTTP_COMPRESSION, enableHttpCompression)

        .dataCompression(ClickHouseCompression.valueOf(compression))
        .table(nameTable)
        .format(ClickHouseFormat.JSONEachRow);

wri.data(new GzipCompressingEntity(new InputStreamEntity(stream, -1)).getContent()).send();

But it dont work.

if I try to send the stream without compression, it works as expected.

I have also tried putting in the url jdbc: clickhouse: // localhost: 8123 / {name.db}? Decompress=1 but it doesn't insert anything into the database either.

I have also tried to compress using this syntax

GZIPOutputStream zos = null; ByteArrayOutputStream rstBao = new ByteArrayOutputStream(); try { zos = new GZIPOutputStream(rstBao); zos.write(stream.toString().getBytes(StandardCharsets.UTF_8)); zos.close(); } catch (IOException e) { e.printStackTrace(); } InputStream is = new ByteArrayInputStream(rstBao.toByteArray());

            wri.data (is) .send ();

But dont work. i get the exception : DB::ParsingException: Cannot parse input: expected '{' before: 'java.io.ByteArrayInputStream@333159da': While executing JSONEachRowRowInputFormat: (at row 1)

the configuration that I am using is

url: "jdbc:clickhouse://localhost:8123/test" nameTable: "con" httpCompression: "gzip" enableHttpCompression: "true"

Lucgarg avatar Sep 14 '21 14:09 Lucgarg