data-prepper
data-prepper copied to clipboard
[BUG] DynamoDB source export converts Numbers ending in 0 to scientific notation
Describe the bug
Given an item in with a Number type ending in 0, such as 1702062202420
, the DynamoDB source will convert it to scientific notation for export items.
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 702062202420 }
To Reproduce Steps to reproduce the behavior:
- Create a pipeline with a dynamodb source with export and an opensearch
- Once the export is complete, the document will be sent to the OpenSearch sink as the following with the
my_number_ending_in_0
key converted to scientific notation
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 1.70206220242E+12 }
Expected behavior The Numbers ending in 0 should not be manipulated and the above example should result in
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 702062202420 }
Additional context
The conversion only happens for export values when converting from the ion line here (https://github.com/opensearch-project/data-prepper/blob/91ff22d6da2b14d8a27ade89ee516341181c8bd6/data-prepper-plugins/dynamodb-source/src/main/java/org/opensearch/dataprepper/plugins/source/dynamodb/converter/ExportRecordConverter.java#L82), but Data Prepper JacksonEvent
also converts to scientific notation when converting to json string (https://github.com/opensearch-project/data-prepper/blob/91ff22d6da2b14d8a27ade89ee516341181c8bd6/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L621). I initially had created a custom deserializer that iterated over and converted all decimals of this format to not use scientific notation, however this may not be the best approach