[Index Management] [Serverless] Display index and data stream size

[alisonelizabeth]

trafficstars

We need to surface the index and data stream size in Index Management in serverless.

We should add this as a column in the list view and details flyout for both the "Indices" and "Data streams" tabs.

The Elasticsearch team recently implemented a _metering/stats API that should return this information.

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)

[yuliacech]

After testing the _metering/stats API and confirming with the ES team, I think we could add the size (and docs count) to the Indices and Datastreams tabs in Index Management as following:

• The call to the ES API from the Kibana server using the new asSecondaryAuthUser added in https://github.com/elastic/kibana/pull/184901. After the api is added to the client types, we don't need to use transport.

await client.asSecondaryAuthUser.transport.request({
      method: 'GET',
      path: `/_metering/stats`,
    });

• The API response should look like

{
  _total: { num_docs: 16, size_in_bytes: 93208 },
  indices: [ array of indices stats ],
  datastreams: [ array of data streams stats ]
}
// index stats 
{
  name: '.ds-.kibana-event-log-ds-2024.08.07-000003',
  datastream: '.kibana-event-log-ds',
  num_docs: 19,
  size_in_bytes: 111792
}

// datastream stats
{ 
  name: '.kibana-event-log-ds', 
  num_docs: 23,
  size_in_bytes: 234534
}

• The API could also be used for individual indices/datastreams GET /_metering/stats/{index_pattern}
• Since the ES client also sends the actual user data to the ES, the response of the API only returns matched indices/datastreams that the user have permissions for.
• If no indices/datastreams are found, the API returns 404.
• According to the ES team, the sizes and docs counts are not recalculated on every request, so it should not affect the performance of the deployment.

[joemcelroy]

For the upcoming onboarding work with search index detail page, we need to add doc + size information to the index GET API route. We (search solution team) plan to add this.

I discussed with @yuliacech as we plan to:

• introduce a new config variable isSizeAndDocCountEnabled which will only be enabled for ES3 elasticsearch project
• return both size and doc count when this boolean is enabled to fetch_indices API

https://github.com/elastic/kibana/blob/main/x-pack/plugins/index_management/server/lib/fetch_indices.ts#L46

Edit: PR Opened https://github.com/elastic/kibana/pull/191631/files#

[alisonelizabeth]

Updated issue to include designs

[alisonelizabeth]

introduce a new config variable isSizeAndDocCountEnabled which will only be enabled for ES3 elasticsearch project

@joemcelroy Is it necessary to only enable this for ES3? Once this issue is implemented, it will be leveraged for all project types.

[joemcelroy]

introduce a new config variable isSizeAndDocCountEnabled which will only be enabled for ES3 elasticsearch project

@joemcelroy Is it necessary to only enable this for ES3? Once this issue is implemented, it will be leveraged for all project types.

So my thinking was that I would only add it to ES3 elasticsearch projects and let Appex roll out the changes across the other project types.

I've changed that stance now and my PR will now enable it for all ES3 project types. This is due to challenges with the common FTR tests that are used across all solutions. Having only one project type enabled will cause larger changes within the API FTRs to be solution independent. Its simpler to enable the change across all and update the test.