indices().stats failes if index is bigger than 2GB

[Lumilie]

Java API client version

7.17.3

Java version

11

Elasticsearch Version

7.17.3

Problem description

Current behavior If you want to get the stats of an index which has a size bigger than the max value of integer, it will cause an exception: IndicesStatsResponse indexResponse = client.indices().stats(new IndicesStatsRequest.Builder().index(indexName).build());

Exception:

co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.StoreStats: jakarta.json.stream.JsonParsingException: Jackson exception: Numeric value (2859969493) out of range of int (-2147483648 - 2147483647)
 at [Source: (org.apache.http.nio.entity.ContentInputStream); line: 1, column: 175] (JSON path: _all.primaries.store.size_in_bytes) (line no=1, column no=175, offset=-1)
	at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
	at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:206)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:328)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:294)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.stats(ElasticsearchIndicesClient.java:2061)

Expected behavior: It should be able to get the statistics of a index bigger than 2 GB/int-max-value

I guess the issue is this here: https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_types/StoreStats.java#L55

[paseip]

Is there any known workaround for this?

EDIT: I used the RestClient instead of the ElasticsearchClient for this specific request. The RestClient is needed to create a ElasticsearchClient, so it is already present in the code. I then used the jakarta.json.JsonReader to read the value: Request req = new Request("GET", "/_cluster/stats"); Response res = restClient.performRequest(req); final String responseBody = EntityUtils.toString(res.getEntity()); JsonReader reader = Json.createReader(new StringReader(responseBody)); final JsonObject json = reader.readObject(); Long storeSizeBytes = json.getJsonObject("indices").getJsonObject("store").getJsonNumber("size_in_bytes").longValue();

Maybe this helps other people facing this issue.

[john-meeder]

We have the same issue in our project as described by @Lumilie Java API client version 7.17.3

Java version 11

Elasticsearch Version 7.17.3

As a workaround, I used the RestClient as suggested by @paseip for this case. The following method returns the size_in_bytes by each index.

    public Map<String, Long> getIndicesSize(List<String> indices) throws IOException {

        Map<String, Long> indexSizes = new HashMap<>();
        String cs = String.join(",", indices);

        Request req = new Request("GET", "/" + cs+ "/_stats/store");
        Response res = restClient.performRequest(req);
        final String responseBody = EntityUtils.toString(res.getEntity());
        JsonReader reader = Json.createReader(new StringReader(responseBody));
        final JsonObject json = reader.readObject();
        JsonObject storeStatsByIndex = json.getJsonObject("indices");

        storeStatsByIndex.entrySet().stream().forEach(entry -> {
                indexSizes.put(entry.getKey(), entry.getValue()
                                                    .asJsonObject()
                                                    .getJsonObject("primaries")
                                                    .getJsonObject("store")
                                                    .getJsonNumber("size_in_bytes")
                                                    .longValue());

        });

        return indexSizes;
    }

[divadpoc]

I'm using client and server w/ version 8.6.0, when calling ElasticsearchClient.cluster().stats() and get

Error deserializing co.elastic.clients.elasticsearch._types.StoreStats: jakarta.json.stream.JsonParsingException: Jackson exception: Numeric value (6579560065) out of range of int (-2147483648 - 2147483647)
at [Source: (org.apache.http.nio.entity.ContentInputStream); line: 1, column: 511] (JSON path: indices.store.size_in_bytes) (line no=1, column no=511, offset=-1), cause: Jackson exception: Numeric value (6579560065) out of range of int (-2147483648 - 2147483647)

furthermore, when calling ElasticsearchClient..nodes().info() I get

Error deserializing co.elastic.clients.elasticsearch.nodes.info.NodeInfo: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NodeInfoSettings.cluster' (JSON path: nodes.u7y_VK_PSUSlGedT80bcvA.settings) (line no=1, column no=1600, offset=-1), cause: Missing required property 'NodeInfoSettings.cluster'
co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch.nodes.info.NodeInfo: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NodeInfoSettings.cluster' (JSON path: nodes.u7y_VK_PSUSlGedT80bcvA.settings) (line no=1, column no=1600, offset=-1)

[l-trotta]

fixed with #759 , the issue will be solved in the next release.