opensearch-project
data is not stored in the format of [<metric_name>: <value>] in OpenSearch.
Versions: opensearch & opensearch dashboard 2.11.1 data-prepper 2.6.1 opentelemetry-collector-contrib 0.92.0
When utilizing DataPrepper's metric pipeline to transmit data collected through OpenTelemetry to OpenSearch, the data is not stored in the format of <metric_name> : <value> in OpenSearch.
OTLE collector to collect kubelet data. If you check in the opensearch dashboard, the metric name is stored in the name field.
{
name: <metric_name>
value: <value>
}
It seems to save as
I want to easily use the visualization features you provide.
{
<metric_name>: <value>
}
, is it possible to change it to something like this?
[name field]
[data-prepper config]
The issue here is the limitations to 1000 fields per index. If your suggestion is followed, this limit will be reached pretty fast. Additionally, this creates a lot of sparse fields, which might impact performance badly. Also note, that the current format follows the OpenTelemetry metrics data model.
@mothercow-dev , The point that @KarstenSchnitter made is good and explains the reason why we do it this way.
Because of that, I don't think we would support this in the otel_metrics source and processor. However, you could use a processor to make the transformation. I would not suggest that because of the issues @KarstenSchnitter made, but it would be possible.
Is there a particular reason you want this format?
@KarstenSchnitter , Thank you
@dlvenable , While creating a Visualize for opensearchd, I was not seeing the metric name of the kubeletstatreceiver collected in Available fields, so I asked a question.
As I understand it, when I create a Visualize, I can create it through the values in Available fields.
Based on your description, can I use the processor in data-prepper to convert it to what I want?
<metric_name> : <value>
I am currently working on a dashboard for the kubeletstatreceiver metrics in OpenSearch with data ingested via DataPrepper. You can use saved searches to filter by the name field or use the filters aggregation, that supports arbitrary DQL expressions. If you use TSVB visualizations, you can provide a global DQL expression for the visualization. This allows to visualize the value of the data points quite nicely.
@dlvenable do you think, it would make sense to write an OpenSearch blog post about visualizing metrics data ingested that way? @mothercow-dev would that be helpful to you?
HI,
@KarstenSchnitter It will be a great help in configuring visualizations in opensearch dashboards.