How can rows_before_limit be extracted from an HTTP response?

[estaine]

Hello, I've got a problem after upgrading from 0.3.1-patch to the latest version.

After I get a response by running statement.unwrap(ClickHouseRequest.class).query(query).executeAndWait() , I can't extract rows_before_limit from it.

As far as I can see from the codebase, ClickHouseResponseSummary.Statistics within the ClickHouseResponse is set to a non-default value only in classes used by the gRPC client, while for HTTP it's always set to null in constructor and never changed downstreams.

I clearly see that the value is returned within the HTTP response set within com.clickhouse.client.http.ClickHouseHttpConnection#postData, so I thought of returning a raw JSON and parsing it on my end, however I don't see a way for that either.

So, my questions are:

1. Why is rows_before_limit not set for a non-gRPC case?
2. If everything works as designed and isn't going to be adjusted, how do I extract that value from the response?

Thanks in advance

[zhicwu]

Hi @estaine, please see my answers below inline.

1. Why is rows_before_limit not set for a non-gRPC case?

Currently the ClickHouseResponseSummary is extracted from X-ClickHouse-Summary from http response. Are you talking about JSON data format? It requires to extract rows_before_limit or rows_before_limit_at_least from JSON response.

# curl -v 'http://localhost:8123?query=select+1'
*   Trying 127.0.0.1:8123...
* Connected to localhost (127.0.0.1) port 8123 (#0)
> GET /?query=select+1 HTTP/1.1
> Host: localhost:8123
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 12 May 2023 08:12:19 GMT
< Connection: Keep-Alive
< Content-Type: text/tab-separated-values; charset=UTF-8
< X-ClickHouse-Server-Display-Name: 6d633a9984f4
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: 20331724-4811-42e3-9bae-d647d9d2d149
< X-ClickHouse-Format: TabSeparated
< X-ClickHouse-Timezone: America/Toronto
< Keep-Alive: timeout=3
< X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"0","result_bytes":"0"}
< 
1
* Connection #0 to host localhost left intact

2. If everything works as designed and isn't going to be adjusted, how do I extract that value from the response?

The JDBC driver is far from mature, that's why the version is just 0.4.6 :p I think we can add JsonDataProcessor to extract rows_before_limit and put it into ClickHouseResponseSummary.

[estaine]

Hi @zhicwu! Thanks for a quick answer! Yeah, I see that the ClickHouseResponseSummary is extracted exclusively from that header (which does not contain anything related to ClickHouseResponseSummary.Statistics) I'd say my question is not about a format itself (though it would be very useful to be able to parse metadata from JSON). My question is how did everything work in 0.3.1-patch? :) In that version I was able to execute a query without specifying a format explicitly and get a total number of rows:

ClickHouseStatement chs = (ClickHouseStatement) jdbcTemplate.getDataSource().getConnection().createStatement();
ClickHouseResponse chr = chs.executeQueryClickhouseResponse(query);
int total = chr.getRows_before_limit_at_least();

[zhicwu]

Thanks @estaine. Apparently we "missed" a few functions after the refactoring, my fault :p

In v0.3.1, the driver issues the given query using JSONCompact format, and then deserialize response using Jackson. The closest workaround I have at this point is:

try (Statement stmt = conn.createStatement();
  ClickHouseResponse resp = stmt.unwrap(ClickHouseRequest.class).format(ClickHouseFormat.JSONCompact).query(query).executeAndWait()) {
  // gson
  MyPojo mine = new GsonBuilder().setPrettyPrinting().create().fromJson(new InputStreamReader(resp.getInputStream()), MyPojo.class);
  // jackson, slower
  // MyPojo mine = new ObjectMapper().readValue(resp.getInputStream(), MyPojo.class);
}

Will raise a separate PR to add JsonDataProcessor along with additional properties in ClickHouseResponseSummary to improve usability.

[estaine]

Thanks a lot, @zhicwu!

[kliakos]

Hi, can someone confirm whether this is possible with v2 client or not?

[chernser]

Good day, @kliakos! It is possible if you parse the content of a response. rows_before_limit is present only for specific formats and server for their purposes. Client-v2 doesn't parse text base formats. We may be will add it in the future.