lonboard icon indicating copy to clipboard operation
lonboard copied to clipboard

Add option to not serialize data to Parquet

Open kylebarron opened this issue 1 year ago • 3 comments

Serializing to Parquet can actually be quite a bit of overhead, see https://github.com/xarray-contrib/xdggs/pull/67#discussion_r1786659267. We should have a switch that lets you turn it off when you know that your Python environment is local and data isn't traveling over a network to be visualized.

kylebarron avatar Oct 03 '24 18:10 kylebarron

Now that Arrow JS has an imminent release with support for IPC compression, we should add support for that.

We may want to add a light config system so that a user can do lonboard.config.send_parquet = False or something like that. Or lonboard.config.local = True as a meta-config which turns off any settings expected to help remote data. Maybe look to pandas for how they implement options.

kylebarron avatar Oct 01 '25 21:10 kylebarron

Turn on local mode by default on Windows and Mac, while turn it off by default on Linux...? It's annoying and surprising to have differences in functionality based on platform though.

kylebarron avatar Oct 02 '25 22:10 kylebarron

Moving to 0.14 because at least now we parallelize the Parquet serialization across threads (ref https://github.com/developmentseed/lonboard/pull/902)

kylebarron avatar Oct 28 '25 00:10 kylebarron