planner, stats: reduce the persist memory usage of stats after startup

[winoros]

What problem does this PR solve?

Issue Number: close https://github.com/pingcap/tidb/issues/46867

Problem Summary:

We've introduced lite init before. Only load the metadata(NDV, NULL count etc) of the column/index's statistics. But the Column/Index's meta object still consumes too much memory.

What is changed and how it works?

So, in this pull, we use a tiny map to represent the existence of statistics of Column/Index. And load the stats only when it's used.

The changes in the pull are all around this idea.

And there's still some remaining problems to be addressed later, like that Pseudo table still costs much memory than we expected.

Check List

Tests

• [x] Unit test
• [ ] Integration test
• [ ] Manual test (add detailed scripts or steps below)
• [ ] No need to test

  • [ ] I checked and no code files have been changed.

Side effects

• [ ] Performance regression: Consumes more CPU
• [ ] Performance regression: Consumes more Memory
• [ ] Breaking backward compatibility

Documentation

• [ ] Affects user behaviors
• [ ] Contains syntax changes
• [ ] Contains variable changes
• [ ] Contains experimental features
• [ ] Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[codecov[bot]]

Codecov Report

Merging #47496 (83df8cf) into master (a1cad29) will increase coverage by 2.0678%. Report is 1 commits behind head on master. The diff coverage is 83.5793%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #47496        +/-   ##
================================================
+ Coverage   70.7470%   72.8148%   +2.0678%     
================================================
  Files          1476       1476                
  Lines        437518     437798       +280     
================================================
+ Hits         309531     318782      +9251     
+ Misses       108612      98991      -9621     
- Partials      19375      20025       +650     

Flag	Coverage Δ
integration	48.8848% <68.2070%> (?)
unit	70.5042% <79.8893%> (-0.0647%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	53.9957% <ø> (ø)
parser	∅ <ø> (∅)
br	51.4669% <ø> (+5.5384%)	:arrow_up:

[hawkingrei]

/retest

[winoros]

/retest

[ti-chi-bot[bot]]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[winoros]

/retest

[time-and-fate]

Should tryAutoAnalyzeTable() also be updated? Looks like it should use HasAnalyzed to check if a new index hasn't been analyzed yet.

And also the newly introduced checkIndexesNeedAnalyze().

[ti-chi-bot[bot]]

[LGTM Timeline notifier]

Timeline:

• 2023-10-10 09:15:34.810015589 +0000 UTC m=+1130132.397125720: :ballot_box_with_check: agreed by hawkingrei.
• 2024-03-12 08:13:05.74996119 +0000 UTC m=+759612.772207580: :ballot_box_with_check: agreed by time-and-fate.

[ti-chi-bot[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, tangenta, time-and-fate

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [hawkingrei,tangenta,time-and-fate]
• ~~pkg/parser/OWNERS~~ [tangenta]
• ~~pkg/table/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment