databend icon indicating copy to clipboard operation
databend copied to clipboard

refactor: speed up ColumnBuilder::repeat(Scalar::Null).

Open youngsofun opened this issue 1 year ago โ€ข 2 comments

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Problem Description

The original implementation calls push_default n times, which is slow because each call is recursive and includes pattern matching.

Impact

The repeat method is used when flattening Value::Scalar to Value::Column. This approach significantly affects or even dominates overall performance when there are many null columns.

related issue: https://github.com/datafuselabs/databend/issues/15908 https://github.com/datafuselabs/databend/pull/15880 will be faster after this

Improvement

This change introduces an ColumnBuilder::repeat_null that create the whole builder at once;

Tests

  • [ ] Unit Test
  • [ ] Logic Test
  • [ ] Benchmark Test
  • [x] No Test - Explain why

Type of change

  • [ ] Bug Fix (non-breaking change which fixes an issue)
  • [ ] New Feature (non-breaking change which adds functionality)
  • [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • [ ] Documentation Update
  • [x] Refactoring
  • [ ] Performance Improvement
  • [ ] Other (please describe):

This change isโ€‚Reviewable

youngsofun avatar Jul 01 '24 07:07 youngsofun

PR: https://github.com/longhorn/longhorn-manager/pull/2824

ChanYiLin avatar May 27 '24 09:05 ChanYiLin

cc @ChanYiLin

derekbit avatar Sep 04 '24 14:09 derekbit

Hi @derekbit The PR is ready for review here: https://github.com/longhorn/longhorn-manager/pull/2824

ChanYiLin avatar Sep 05 '24 06:09 ChanYiLin

Oh I already put it in previous comment

ChanYiLin avatar Sep 05 '24 06:09 ChanYiLin

Pre Ready-For-Testing Checklist

  • [ ] Where is the reproduce steps/test steps documented? The reproduce steps/test steps are at:
    1. Create a BackingImage and a volume/PV/PVC using it
    2. Create a system-backup
    3. kubectl get lhbbi, lhb -n longhorn-system will see the backup of the Volume and the Backingimage
    4. Delete the Volume and the BackingImage
    5. Restore the system-backup
    6. The BackingImage and the Volume will be restored. The BackingImage source type will be restored and the checksum will be verified.

PRs:

  • https://github.com/longhorn/longhorn-manager/pull/2824

longhorn-io-github-bot avatar Sep 05 '24 06:09 longhorn-io-github-bot

@ChanYiLin This is a new feature. I suggest we don't need to backport it to v1.6.x and just add a note in v1.6.x doc.

cc @innobead

derekbit avatar Sep 09 '24 13:09 derekbit

Agreed with @derekbit , even though we regarded this as an improvement initially.

innobead avatar Sep 09 '24 14:09 innobead

we are not going to backport this feature to v1.6.x

The limitation is already in the doc https://longhorn.io/docs/1.6.2/advanced-resources/system-backup-restore/backup-longhorn-system/ Warning: Longhorn does not backup BackingImages. We will improve this part in the future. See Restore Longhorn System - Prerequisite for restoring volumes created with the backing image.

ChanYiLin avatar Sep 10 '24 08:09 ChanYiLin