[BUG] Failed to rollover index

[ElfoLiNk]

Describe the bug Rollover index is not working:

{
    "cause": "no permissions for [indices:admin/rollover] and associated roles [dataprepper, own_index]",
    "message": "Failed to rollover index [index=otel-v1-apm-span-000001]"
}

To Reproduce

Install dataprepper and use a specific user with the following role

# DataPrepper Role
dataprepper:
reserved: true
cluster_permissions:
  - cluster_all
  - indices:admin/template/get
  - indices:admin/template/put
index_permissions:
  - index_patterns:
      - 'otel-v1*'
      - '.opendistro-ism-config'
      - 'events-*'
      - 'metrics-*'
    allowed_actions:
      - 'indices_all'
  - index_patterns:
      - '*'
    allowed_actions:
      - 'manage_aliases'

Expected behavior

Rolling policy should work

Environment (please complete the following information):

• OS: 2.11.0
• Version 2.6.0

[asifsmohammed]

@ElfoLiNk This looks like an OpenSearch issue and not related to Data Prepper. Can you provide further details?

[ElfoLiNk]

Hi @asifsmohammed i configured data prepper user and role following https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/opensearch_security.md the policy are created automatically by data prepper no?

I found this on OS project https://github.com/opensearch-project/security/issues/1861

[JannikBrand]

This is probably not related to an OpenSearch issue but to the permission setup of Data Prepper. I am not using the OpenSearch admin user but my own defined user with the permissions listed here. I could fix it by adding the "indices:admin/rollover" permission to the OpenSearch role:

data_prepper_role:
      reserved: true
      description: "role description"
      cluster_permissions:
        - "cluster_all"
        - "indices:admin/index_template/get"
        - "indices:admin/index_template/put"
      index_permissions:
        - index_patterns:
            - "otel-v1-apm*"
            - ".opendistro-ism-config"
          allowed_actions:
            - "indices_all"
        - index_patterns:
            - "*"
          allowed_actions:
            - "manage_aliases"
            - "indices:admin/rollover"

Background: An ISM policy seems to be linked to the user who created it. This makes sense, since otherwise a user which does not have e.g. permissions to delete indices could create an ISM policy which would delete indices. Previously (probably due to an older OpenSearch version), the rollover actions worked without having this permissions. I suspect that there was a change in some OpenSearch version update that linked the user to the ISM policy. I tested the mitigation with version 1.3.15.

When I checked out the .opendistro-ism-config index e.g. like this...

GET .opendistro-ism-config/_search
{
  "query": {
    "match_all": {}
  },
  "size": <select number to show all hits>
}

...there is an entry like the following:

{
  "_index" : ".opendistro-ism-config",
  "_type" : "_doc",
  "_id" : "OYi_mi3vQG6sLArwXLVBoA",
  "_score" : 1.0,
  "_routing" : "OYi_mi3vQG6sLArwXLVBoA",
  "_source" : {
    "managed_index" : {
      "name" : "otel-v1-apm-span-000001",
      "enabled" : false,
      "index" : "otel-v1-apm-span-000001",
      "index_uuid" : "OYi_mi3vQG6sLArwXLVBoA",
      "schedule" : {
        "interval" : {
          "start_time" : 1706018691311,
          "period" : 5,
          "unit" : "Minutes"
        }
      },
      "last_updated_time" : 1706780244142,
      "enabled_time" : null,
      "policy_id" : "raw-span-policy",
      "policy_seq_no" : -2,
      "policy_primary_term" : 0,
      "policy" : {
        "policy_id" : "raw-span-policy",
        "description" : "Managing raw spans for trace analytics",
        "last_updated_time" : 1706018690526,
        "schema_version" : 13,
        "error_notification" : null,
        "default_state" : "current_write_index",
        "states" : [
          {
            "name" : "current_write_index",
            "actions" : [
              {
                "retry" : {
                  "count" : 3,
                  "backoff" : "exponential",
                  "delay" : "1m"
                },
                "rollover" : {
                  "min_size" : "50gb",
                  "min_index_age" : "24h"
                }
              }
            ],
            "transitions" : [ ]
          }
        ],
        "ism_template" : [
          {
            "index_patterns" : [
              "otel-v1-apm-span-*"
            ],
            "priority" : 0,
            "last_updated_time" : 1706018690526
          }
        ],
        "user" : {
          "name" : "data_prepper_user",
          "backend_roles" : [ ],
          "roles" : [
            "own_index",
            "data_prepper_role"
          ],
          "custom_attribute_names" : [ ],
          "user_requested_tenant" : null
        }
      },
      "change_policy" : null,
      "jitter" : 0.6
    }
  }
} 

As you can see there is the otel-v1-apm-span-000001 index which is linked to the raw-span-policy which has a user field and a link to your data_prepper_role.