Index Patterns Refresh field list automatically update for new fields

[teebu]

Is your feature request related to a problem? Please describe.

I was using Dashboards, and was trying to figure out why new dynamic fields were not showing up in the selection. It turns out this problem was solved in Kibana https://github.com/elastic/kibana/pull/82223 a while ago, but I still had to manually hit the refresh fields in the Index Patterns in this version.

Describe the solution you'd like

Just like in Kibana, I believe you don't have to do anything, it just works. They solved it by removing field caching

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Other discussions on such topic: https://discuss.elastic.co/t/refresh-field-list-through-api-call/198764 https://github.com/elastic/kibana/issues/6498 https://github.com/elastic/kibana/pull/82223

[JacobBrandt]

As another user pointed out after Kibana made this change...it broke some peoples use cases. In some instances people used this old behavior to purposely hide fields from their users. It's NOT a field security reason they do this either. Just want to make sure that this point is highlighted.

Removing field caching is a good feature but it should at the same time come with the ability to hide fields from users. Kibana never implemented a way to accommodate the use case that was possible with the old behavior.

[florianakos]

Any news on when this feature could land in OSD?

[joshuarrrr]

@ashwin-pc to research further

[ashwin-pc]

The main problem here is with Dashboards caching the index pattern field list. And to solve this multiple approaches have been proposed:

1. Refresh list API
2. Remove caching

Looking into this a lil more, this should be fairly simple to implement if we make caching optional. It should satisfy both requirements of maintaining the existing behavior while also giving users the flexibility to override this behavior and always update the field list.

This can also be a setting that one can set in the advanced setting's pane.

@ahopp do we need to prioritize this?

[KannappanSomu]

+1 ,

any news on this feature ?

[aigerim-suleimenova]

Hi, please assign this issue to me!

[joshuarrrr]

@Aigerim-ai Let's get at least one of your other open PRs merged first, before assigning. In the meantime, there are a number of other CCI PRs (https://github.com/opensearch-project/OpenSearch-Dashboards/pulls?q=is%3Aopen+is%3Apr+label%3ACCI) that you can help peer review.

[djablonski-moia]

Any progress here?

[ashishvaishno]

Are there any API's available to do the refresh programatically? I tried to send a PUT request on the saved objects (_dashboards/api/saved_objects/index-pattern/index-pattern-id) with the below parameters , but that doesnt work 👎 payload = {"attributes":{"title": "index-*", "refresh_fields": 'true'}} headers = {"Content-Type": "application/json", "osd-xsrf": "true", "security_tenant": "global" }

[yb-yu]

How can I remove field cache manually by API?

[cameronattard]

Would be great for this to get some attention. It feels like a really bad user experience to have to manually refresh the cache very time a new field is added, particularly in use cases where fields are dynamic in nature like log analytics.

[vshunkov]

We are managing index static template deployment in automated manner, but still have to do manual operations. Looking forward for the proper implementation.

[jphilaire]

Are there any API's available to do the refresh programatically? I tried to send a PUT request on the saved objects (_dashboards/api/saved_objects/index-pattern/index-pattern-id) with the below parameters , but that doesnt work 👎 payload = {"attributes":{"title": "index-*", "refresh_fields": 'true'}} headers = {"Content-Type": "application/json", "osd-xsrf": "true", "security_tenant": "global" }

It could be nice to have an API like that. So people could automate or not the refresh of index patterns. From my point of view, I would like to refresh all index patterns of each tenant one time a day.

[yb-yu]

I'm currently running a script to refresh the index pattern every 10 minutes.

This script mimics the same steps I'd take manually in the OpenSearch Dashboard to refresh the fields.

It fetches the current index pattern and retrieves all its fields using the GET index pattern API.

Instead of using Kibana API's refresh parameter, the OpenSearch Dashboard refresh process includes all existing fields from the index pattern, and the script follows that approach.

I’m not sure why it’s implemented this way, but I found that by performing the same steps, the index pattern gets refreshed...

Ideally, it would be great if it could be done with just one API call to refresh the fields, but for now, this script works as a workaround:

from urllib.parse import urljoin
import json

import boto3
import requests
from requests_aws4auth import AWS4Auth


DOMAIN = "https://your.opensearch.domain"
INDEX_PATTERN = "your-index-pattern-*"
TENANT = "your-tenant"
REGION = "us-east-1"

FIND_SAVED_OBJECTS_URI = urljoin(DOMAIN, "/_dashboards/api/saved_objects/_find")
INDEX_PATTERN_URI = urljoin(DOMAIN, "/_dashboards/api/saved_objects/index-pattern")
INDEX_PATTERN_FIELDS_URI = urljoin(DOMAIN, "/_dashboards/api/index_patterns/_fields_for_wildcard")

cred = boto3.Session().get_credentials()
auth = AWS4Auth(cred.access_key, cred.secret_key, REGION, "es", session_token=cred.token)
headers = {"Content-Type": "application/json", "osd-xsrf": "true", "securitytenant": TENANT}

# Step 1: Find the index pattern and get id & version
params = {"type": "index-pattern", "fields": "id", "search": INDEX_PATTERN}
index_pattern_info = requests.get(FIND_SAVED_OBJECTS_URI, auth=auth, headers=headers, params=params).json()

index_id = index_pattern_info["saved_objects"][0]["id"]
index_pattern_version = index_pattern_info["saved_objects"][0]["version"]

# Step 2: Get the index pattern fields
params = {"pattern": INDEX_PATTERN, "meta_fields": ["_source", "_id", "_type", "_index", "_score"]}
index_pattern_fields = requests.get(INDEX_PATTERN_FIELDS_URI, auth=auth, headers=headers, params=params).json()["fields"]

# Step 3: Update the index pattern
put_index_pattern_info = {
    "attributes": {
        "title": INDEX_PATTERN,
        "fields": json.dumps(index_pattern_fields),
        "version": index_pattern_version,
    }
}
put_index_pattern = requests.put(
    INDEX_PATTERN_URI + "/" + index_id, auth=auth, headers=headers, json=put_index_pattern_info
)

[jphilaire]

I'm currently running a script to refresh the index pattern every 10 minutes.

This script mimics the same steps I'd take manually in the OpenSearch Dashboard to refresh the fields.

It fetches the current index pattern and retrieves all its fields using the GET index pattern API.

Instead of using Kibana API's refresh parameter, the OpenSearch Dashboard refresh process includes all existing fields from the index pattern, and the script follows that approach.

I’m not sure why it’s implemented this way, but I found that by performing the same steps, the index pattern gets refreshed...

Ideally, it would be great if it could be done with just one API call to refresh the fields, but for now, this script works as a workaround:

from urllib.parse import urljoin
import json

import boto3
import requests
from requests_aws4auth import AWS4Auth
.
.
.

Well done ! It works fine ! Thank you @yb-yu for sharing !

[niklasweimann]

Any progress on this issue?

[Its-Ankush]

I'm currently running a script to refresh the index pattern every 10 minutes.

This script mimics the same steps I'd take manually in the OpenSearch Dashboard to refresh the fields.

It fetches the current index pattern and retrieves all its fields using the GET index pattern API.

Instead of using Kibana API's refresh parameter, the OpenSearch Dashboard refresh process includes all existing fields from the index pattern, and the script follows that approach.

I’m not sure why it’s implemented this way, but I found that by performing the same steps, the index pattern gets refreshed...

Ideally, it would be great if it could be done with just one API call to refresh the fields, but for now, this script works as a workaround:

from urllib.parse import urljoin
import json

import boto3
import requests
from requests_aws4auth import AWS4Auth
......

I believe it doesn't work now.

# Step 3: Update the index pattern
put_index_pattern_info = {
    "attributes": {
        "title": INDEX_PATTERN,
        "fields": json.dumps(index_pattern_fields),
        "version": index_pattern_version,
    }
}

I took a look at the original network request from chrome and it seems that it refreshing the index pattern needs a payload which looks like this -

{
    "attributes": {
        "title": "index_pattern_name",
        "timeFieldName": "date_field",
        "fields": "[{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false}......]"
    },
    "version": "version_from_step_1"
}

Fields from Step 2 look like this

{
  'name': 'additional_field',
  'type': 'string',
  'esTypes': ['text'],
  'searchable': True,
  'aggregatable': False,
  'readFromDocValues': False
}

While it actually looks for

{
    "count": 0,
    "name": "additional_field",
    "type": "string",
    "esTypes": ["text"],
    "scripted": false,
    "searchable": true,
    "aggregatable": false,
    "readFromDocValues": false
}

Im getting blank screen when i click on my index pattern. Also this may not work for clusters with SAML/cognito enabled [I still need to check and confirm that]

@yb-yu is it working for you ?

Edit - this is what I see on Console

[yb-yu]

@Its-Ankush Hi,

The code is running smoothly on my end.

The actual response from step 2 looks like following in my case.

[
  {'name': '_id', 'type': 'string', 'esTypes': ['_id'], 'searchable': True, 'aggregatable': True, 'readFromDocValues': False}, 
  {'name': '_index', 'type': 'string', 'esTypes': ['_index'], 'searchable': True, 'aggregatable': True, 'readFromDocValues': False}, 
  {'name': '_score', 'type': 'number', 'searchable': False, 'aggregatable': False, 'readFromDocValues': False},
  ... # other custom fields
]

I have been using this code on aws managed opensearch since version 2.11 (now it's 2.15), and SAML is also enabled through the Microsoft Enterprise Application.