Convert org.apache.kafka.connect.data.Date to epoch millis before sending to Elasticsearch

[aleksanderlech]

Problem

Lets consider the following kafka schema:

{
      "name": "dateOfBirth",
      "type": [ "null", {"type": "int", "logicalType": "date"}],
      "default": null
}

And the target index field:

  "dateOfBirth": {
      "type": "date"
    }

Currently the date type (org.apache.kafka.connect.data.Date) is being converted to int (epoch days) and ends up in Elasticsearch being interpreted as epoch millis turning it into some datetime long in the past.

Elasticsearch does not suport epoch days type for date fields.

Solution

Assuming the field is properly indexed as date we should pass some of default supported formats like epoch_millis.

Does this solution apply anywhere else?

• [ ] yes
• [x] no

If yes, where?

Test Strategy

Testing done:

• [ ] Unit tests
• [x] Integration tests
• [x] System tests
• [x ] Manual tests

Release Plan

Bugfix, it might be have some incompatibles with existing user workarounds though.