[BUG] strict_date_optional_time date format does not handle negative epoch millis with leading zeros correctly

[jed326]

Describe the bug

When trying to index negative epoch millis timestamps with leading zeros the strict_date_optional_time will incorrectly try to parse these as Years.

These negative epoch millis timestamps are able to be parsed just fine by the epoch_millis format, however when the default format strict_date_optional_time||epoch_millis is used the parsers are evaluated left to right and strict_date_optional_time throws an exception in this case. This is not how the || operator is supposed to behave for formats as if one parser is not able to parse the value then it should try with the others.

Related component

Indexing

To Reproduce

Reproduced this on a local OpenSearch cluster in a few different scenarios:

1. The default format strict_date_optional_time||epoch_millis cannot parse timestamp -9392400000. The "verbose" epoch millis for this timestamp should be -0009392400000, however OpenSearch also does not allow ingesting numerics with leading zeros.

dengjay@88665a3f24d4 OpenSearch % curl -XPUT "http://localhost:9200/testindex" -H 'Content-Type: application/json' -d'            
{
  "mappings" : {
    "properties" :  {
      "release_date" : {
        "type" : "date",
        "format" : "strict_date_optional_time||epoch_millis"
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"testindex"}%                                                                                       dengjay@88665a3f24d4 OpenSearch % curl -X PUT "localhost:9200/testindex/_doc/14?pretty" -H 'Content-Type: application/json' -d'   
{ "release_date": -9392400000 }
'               
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [release_date] of type [date] in document with id '14'. Preview of field's value: '-9392400000'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [release_date] of type [date] in document with id '14'. Preview of field's value: '-9392400000'",
    "caused_by" : {
      "type" : "date_time_exception",
      "reason" : "Invalid value for Year (valid values -999999999 - 999999999): -9392400000"
    }
  },
  "status" : 400
}

2. epoch_millis is able to parse timestamp -9392400000 just fine

dengjay@88665a3f24d4 OpenSearch % curl -XPUT "http://localhost:9200/testindex2" -H 'Content-Type: application/json' -d'            
{                               
  "mappings" : {
    "properties" :  {
      "release_date" : {
        "type" : "date",
        "format" : "epoch_millis" 
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"testindex2"}%                                                                                      dengjay@88665a3f24d4 OpenSearch % curl -X PUT "localhost:9200/testindex2/_doc/14?pretty" -H 'Content-Type: application/json' -d'   
{ "release_date": -9392400000 }
'               
{
  "_index" : "testindex2",
  "_id" : "14",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

3. Parsers seem to parse left to right, so using mapping epoch_millis||strict_date_optional_time works just fine as well:

dengjay@88665a3f24d4 OpenSearch % curl -XPUT "http://localhost:9200/testindex33" -H 'Content-Type: application/json' -d'            
{                               
  "mappings" : {
    "properties" :  {
      "release_date" : {
        "type" : "date",
        "format" : "epoch_millis||strict_date_optional_time"
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"testindex33"}%                                                                                     dengjay@88665a3f24d4 OpenSearch % curl -X PUT "localhost:9200/testindex33/_doc/14?pretty" -H 'Content-Type: application/json' -d'  
{ "release_date": -9392400000 }
'
{
  "_index" : "testindex33",
  "_id" : "14",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

4. Whenever one parser is not able to parse a value, other parsers should be used. This is how it works for positive epoch timestamps. Timestamp 999999999000 is not able to be parsed by strict_date_optional_time, however unlike with negative epoch millis it is able to be parsed by strict_date_optional_time||epoch_millis

dengjay@88665a3f24d4 OpenSearch % curl -XPUT "http://localhost:9200/testindex4" -H 'Content-Type: application/json' -d'            
{                               
  "mappings" : {
    "properties" :  {
      "release_date" : {
        "type" : "date",
        "format" : "strict_date_optional_time" 
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"testindex4"}%                                                                                      dengjay@88665a3f24d4 OpenSearch % curl -X PUT "localhost:9200/testindex4/_doc/14?pretty" -H 'Content-Type: application/json' -d'
{ "release_date": 999999999000 }
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [release_date] of type [date] in document with id '14'. Preview of field's value: '999999999000'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [release_date] of type [date] in document with id '14'. Preview of field's value: '999999999000'",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "failed to parse date field [999999999000] with format [strict_date_optional_time]",
      "caused_by" : {
        "type" : "date_time_parse_exception",
        "reason" : "Text '999999999000' could not be parsed at index 0"
      }
    }
  },
  "status" : 400
}

dengjay@88665a3f24d4 OpenSearch % curl -XPUT "http://localhost:9200/testindex5" -H 'Content-Type: application/json' -d'            
{                               
  "mappings" : {
    "properties" :  {
      "release_date" : {
        "type" : "date",
        "format" : "strict_date_optional_time||epoch_millis"
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"testindex5"}%                                                                                      dengjay@88665a3f24d4 OpenSearch % curl -X PUT "localhost:9200/testindex5/_doc/14?pretty" -H 'Content-Type: application/json' -d'  
{ "release_date": 999999999000 }
'               
{
  "_index" : "testindex5",
  "_id" : "14",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Will try to add a UT/IT demonstrating these scenarios in the near future.

Expected behavior

Date time parsers should all be evaluated on a given timestamp for negative epoch times. Ingesting timestamp -9392400000 should not fail.

Additional Details

Discovered on OpenSearch 2.5, reproducible on main (3.x) as well.

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Reprocued on OS 2.5 and 3.
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[andrross]

[Triage - attendees 1 2 3 4 5 6] @jed326 Thanks for filing and providing all the details on the reproduction. Looking forward to seeing a fix here.

[srikanthpadakanti]

Hello @jed326 @andrross Please find the below observations.

curl -X PUT "localhost:9200/my-index-000009?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "strict_date_optional_time||epoch_millis"
      }
    }
  }
}
'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "my-index-000009"
}

curl -X PUT "localhost:9200/my-index-000009/_doc/example?refresh&pretty" -H 'Content-Type: application/json' -d'
{ "date": -292275055 }
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [date] of type [date] in document with id 'example'. Preview of field's value: '-292275055'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [date] of type [date] in document with id 'example'. Preview of field's value: '-292275055'",
    "caused_by" : {
      "type" : "arithmetic_exception",
      "reason" : "long overflow"
    }
  },
  "status" : 400
}

curl -X PUT "localhost:9200/my-index-000009/_doc/example?refresh&pretty" -H 'Content-Type: application/json' -d'
{ "date": -292275054 }
'
{
  "_index" : "my-index-000009",
  "_id" : "example",
  "_version" : 1,
  "result" : "created",
  "forced_refresh" : true,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Note: I didn't verified if the inserted value unfolds to right date in the dashboard/UI