data-prepper
data-prepper copied to clipboard
[BUG] Index template_file and template_content truncates index pattern incorrectly
Describe the bug
Index patterns are being created with a truncated index pattern from the index template index patterns passed in the config
This bug occurs when there is no ism_policy_file
passed to the source. This is here in the code (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexManagerFactory.java#L103) where the decision to use NoIsmPolicyManagement
class is made. This class has the following method which does not add a pattern back (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/NoIsmPolicyManagement.java#L43), which is called here (after removing the pattern suffix) (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L230). With an ism policy file, this method will be called instead, which adds back the pattern (https://github.com/opensearch-project/data-prepper/blob/463fa4590d5f9d7f0a397430de0dbd634384dd03/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagement.java#L145)
To Reproduce Steps to reproduce the behavior:
- Create a pipeline with the following opensearch sink configuration
sink:
- opensearch:
index: "test-index-1"
template_type: "index-template"
template_content: |
{
"index_patterns": [
"test-*"
],
"template": {
"aliases": {
"my_test_logs": {}
},
"settings": {
"number_of_shards": 5,
"number_of_replicas": 2,
"refresh_interval": -1
},
"mappings": {
"properties": {
"timestamp": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
},
"value": {
"type": "double"
}
}
}
}
}
- Run data prepper and then go to OpenSearch Dashboards to look at the created index template
GET _index_template/test-index-template
You will see that the index_pattern in the template is only test
and not test-*
Expected behavior To not truncate the index pattern
Additional context
Maybe same here.