kepler.gl icon indicating copy to clipboard operation
kepler.gl copied to clipboard
verified publisher icon keplergl

[Bug] DuckDB - Zoom in and out very jittery and delayed

Open realAudi opened this issue 11 months ago • 4 comments

Describe the bug The new 3.1.0 DuckDB preview release is very slow, nearly unusable.

To Reproduce Steps to reproduce the behavior:

  1. Load any reasonable size dataset (i.e. 10K points from CSV)
  2. Zoom in and out
  3. Response is very jittery and delayed, often several seconds, same applies to other functions in the config menu

Expected behavior Smooth zooming and other operations

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • mac OS 15.1 (24B83
  • Browser Chrome
  • Version latest

Additional context I do very much appreciate though the fix for the tooltip interaction. Previous release would not show the tooltip once you had at any time configured a trip layer. So I am hoping you can make this new release just as performant

Thanks for this wonderful product!

realAudi avatar Jan 30 '25 15:01 realAudi

@igorDykhta

If I were to guess I think this is at first caused by the loading of the dataset into the DB, then perhaps the continuous jittery is just the overhead of the DB. I have seen this on other apps my team built in the past that put a lot of data in a browser DB.

Would it be possible maybe to have a setting to turn on or off the use of the DB or is it so integrated that this cannot be done?

audi

realAudi avatar Feb 05 '25 10:02 realAudi

@realAudi Hi Audi.

  1. You can use the kepler.gl version without DB plugin here kepler.gl/demo.

  2. If the jitter remains on kepler.gl/demo then most likely this is due to Arrow files loading issue when a file is loaded in small batches and each batch creates a separate sublayer for rendering. I'm looking into this.

igorDykhta avatar Feb 05 '25 11:02 igorDykhta

@igorDykhta

on the normal kepler, yes same dataset and config works well. It is only when I tried the same on the duckDB preview that I saw the jitter

However I did see in the preview some other fixes that would be nice to have. I have been having problems with the data view modal popup when I use a trace panel

realAudi avatar Feb 05 '25 12:02 realAudi

Update for the future:

  • The issue is most likely related to the fact that DuckDB in the browser outputs Arrow tables in relatively small batches. Currently, kepler.gl renders each batch as a separate sublayer.

  • There is currently no option to specify the batch size for Arrow tables generated by DuckDB in the browser.

  • Apache Arrow does not have built-in functionality to merge batches without additional code, and is probably too memory consuming.

igorDykhta avatar May 18 '25 14:05 igorDykhta