amoro icon indicating copy to clipboard operation
amoro copied to clipboard
verified publisher icon apache

[Feature][AMS]: Add overview page in AMS dashboard

Open zhoujinsong opened this issue 3 years ago • 4 comments

Description

We need an overview page in AMS dashboard to disply the summary information and metrics of Arctic.

Use case/motivation

Add an overview page in AMS dashboard to disply the summary information and metrics of Arctic.

Describe the solution

  • Collect summary information and metrics in AMS.
  • Add an overview page in AMS dashboard and disply the information with chart and table list etc.

Related issues

No response

Are you willing to submit a PR?

  • [x] Yes I am willing to submit a PR!

Code of Conduct

zhoujinsong avatar Sep 23 '22 02:09 zhoujinsong

visual design for overview page: image including:

  • overview statistics：table/catalog count, space size, optimizing resources
  • optimizing brief：resource usage curve, optimizing table list
  • table space brief: space size curve, table size list

majin1102 avatar Oct 18 '22 06:10 majin1102

Hi @zhoujinsong @majin1102 , have you considered stats for Iceberg table health? For example, estimate the time cost of a simple query, if it is greater than a reasonable time, then the table is probably an unhealthy table. Then Arctic can directly tell the users which tables are not suitable for reading, instead of giving us feedback when the query is too slow.

xxubai avatar Feb 06 '23 08:02 xxubai

@XBaith , I agree that it will be very useful if we can tell users if iceberg tables are healthy. As far as I can see, the unhealthy conditions of the table may be：

  • Too many small files
  • Too many delete data

And yes, Arctic is trying to help tables avoid the problems above.

zhoujinsong avatar Feb 09 '23 08:02 zhoujinsong

@zhoujinsong Correct. But for the native iceberg format, "eq-delete rate" is also an important indicator. We can calculate eq-delete rate by total-eq-delete-records / total-records. On the iceberg JMM benchmark, query performance may be unacceptable when eq-delete is greater than 0.6 (estimated value)

xxubai avatar Feb 10 '23 06:02 xxubai

I've updated the visual design of the overview page
VisualDesign

including:

overview statistics：table/catalog count, space size, optimizing resources
Table Format statistics
Optimizing Status statistics
Unhealth Tables
Latest Operations

MarigWeizhi avatar Jul 12 '24 06:07 MarigWeizhi

Can we add some charts to display the trend of overall data(data size, file count)?

tcodehuber avatar Jul 12 '24 07:07 tcodehuber