paimon icon indicating copy to clipboard operation
paimon copied to clipboard
verified publisher icon apache

[Improment] The issue of indexing large volumes of data tables.

Open wg1026688210 opened this issue 2 years ago • 1 comments

Search before asking

  • [X] I searched in the issues and found nothing similar.

Motivation

Now we have seen significant improvements in query performance through index creation. However, we have encountered several issues during the indexing process.

  1. Some tasks before sorting and writing to Paimon are getting stuck waiting for data for a long time.
  2. Due to data skewness, some indexing tasks take a long time to execute.
  3. The flink batch Job of building index will global failover after taskManager oom
  4. FilesTable can not query the paritions which has built index

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

  • [ ] I'm willing to submit a PR!

wg1026688210 avatar Nov 03 '23 04:11 wg1026688210

issue 1,2 has been fixed related #3081 #2749

issue 3 need a remote suffle service for task failover when using flink

wg1026688210 avatar Apr 08 '24 08:04 wg1026688210