auto_date_histogram aggregation with ip_ranges sub-aggregation fails with illegal_state_exception

[123joshuawu]

Elasticsearch Version

7.16.2

Installed Plugins

No response

Java Version

openjdk 11.0.13

OS Version

Ubuntu 18.04.1

Problem Description

Using auto_date_histogram with a ip_range sub-aggregation with <n> ranges fails with an illegal_state_exception with reason Expected [<n>] buckets, but got [0].

Steps to Reproduce

Create index with date field and ip field and put in test data

PUT /testing_bug
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "ip": {
        "type": "ip"
      }
    }
  }
}
POST /testing_bug/_doc
{
  "@timestamp": "2022-08-01T07:00:00.000Z",
  "ip": "192.168.1.1"
}
POST /testing_bug/_doc
{
  "@timestamp": "2022-08-02T07:00:00.000Z",
  "ip": "192.168.1.2"
}

Do auto_date_histogram aggregation with ip_ranges sub-aggregation

GET /testing_bug/_search
{
  "aggs": {
    "testing": {
      "auto_date_histogram": {
        "field": "@timestamp",
        "buckets": 10
      },
      "aggs": {
        "ip_ranges": {
          "ip_range": {
            "field": "ip",
            "ranges": [
              {
                "from": "10.0.0.5",
                "to": "10.0.0.10"
              }
            ]
          }
        }
      }
    }
  }
}

Error response:

{
  "error" : {
    "root_cause" : [ ],
    "type" : "search_phase_execution_exception",
    "reason" : "",
    "phase" : "fetch",
    "grouped" : true,
    "failed_shards" : [ ],
    "caused_by" : {
      "type" : "illegal_state_exception",
      "reason" : "Expected [1] buckets, but got [0]"
    }
  },
  "status" : 500
}

Logs (if relevant)

[2022-09-17T01:00:53,509][WARN ][r.suppressed             ] [ip-*********] path: /********/_search, params: {pretty=true, index=*******}
org.elasticsearch.action.search.SearchPhaseExecutionException:
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:103) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.2.jar:7.16.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Expected [5] buckets, but got [0]
        at org.elasticsearch.search.aggregations.bucket.range.InternalBinaryRange.reduce(InternalBinaryRange.java:242) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:253) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:267) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.reduceBucket(InternalAutoDateHistogram.java:395) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.mergeConsecutiveBuckets(InternalAutoDateHistogram.java:562) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.maybeMergeConsecutiveBuckets(InternalAutoDateHistogram.java:547) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.reduce(InternalAutoDateHistogram.java:517) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:253) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:188) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:538) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:507) ~[elasticsearch

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[nik9000]

Confirmed on main. Here's the curl version in case anyone wants it:

curl -uelastic:password -HContent-Type:application/json -XDELETE localhost:9200/testing_bug
curl -uelastic:password -HContent-Type:application/json -XPUT localhost:9200/testing_bug -d'{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "ip": {
        "type": "ip"
      }
    }
  }
}'
curl -uelastic:password -HContent-Type:application/json -XPOST 'localhost:9200/testing_bug/_doc?pretty&refresh' -d'{
  "@timestamp": "2022-08-01T07:00:00.000Z",
  "ip": "192.168.1.1"
}'
curl -uelastic:password -HContent-Type:application/json -XPOST 'localhost:9200/testing_bug/_doc?pretty&refresh' -d'{
  "@timestamp": "2022-08-02T07:00:00.000Z",
  "ip": "192.168.1.2"
}'

curl -uelastic:password -HContent-Type:application/json -XGET 'localhost:9200/testing_bug/_search?pretty&error_trace' -d'{
  "aggs": {
    "testing": {
      "auto_date_histogram": {
        "field": "@timestamp",
        "buckets": 10
      },
      "aggs": {
        "ip_ranges": {
          "ip_range": {
            "field": "ip",
            "ranges": [
              {
                "from": "10.0.0.5",
                "to": "10.0.0.10"
              }
            ]
          }
        }
      }
    }
  }
}'

And the error:

java.lang.IllegalStateException: Expected [1] buckets, but got [0]
	at [email protected]/org.elasticsearch.search.aggregations.bucket.range.InternalBinaryRange.reduce(InternalBinaryRange.java:260)
	at [email protected]/org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:159)
	at [email protected]/org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.reduceBucket(InternalAutoDateHistogram.java:418)
	at [email protected]/org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.mergeConsecutiveBuckets(InternalAutoDateHistogram.java:599)
	at [email protected]/org.elasticsearch.search.aggregations.bucket.histogram.InternalAutoDateHistogram.maybeMergeConsecutiveBuckets(InternalAutoDateHistogram.java:580)

[nik9000]

I've opened a PR that should fix it!

[123joshuawu]

Great thank you @nik9000! what version of elastic will the fix be available?

[nik9000]

I've got it targeted at 8.6.0 and 8.5.1 but it depends on how reviews go.

[nik9000]

I've got it targeted at 8.6.0 and 8.5.1 but it depends on how reviews go.

Retargeted for 8.5.0.