marimo icon indicating copy to clipboard operation
marimo copied to clipboard

Dataframe viewer's column chart are aggregated on the backend

Open mscolnick opened this issue 11 months ago • 0 comments

Description

Currently we send up to 20k rows of data to the frontend to aggregate columns summaries. This aggregation happens in vega-lite and can cause memory issues and slowdowns.

Suggested solution

We would like to:

  1. move this aggregation to the backend. either using vega-fusion (if installed) or narwhals.Expr.hist (if/when implemented)
  2. lazily make these requests based on content visibility on the frontend

Alternative

No response

Additional context

This could help: https://github.com/narwhals-dev/narwhals/issues/1561

mscolnick avatar Dec 10 '24 20:12 mscolnick