Datagrid (was: Large tables render slowly in React.js)

[dustingetz]

• http://tank.hyperfiddle.net/:mbrainz/

Chrome debugger hits the "Paused before OOM" threshold, but it chugs through. It takes a while.

Hyperfiddle and Datomic and Transit seem to be doing just fine.

[khardenstine]

Heh I would not say our browser and ui are pulling their weight:

2 seconds for us to compute and render a duplicate key error (navigating from view to data mode, #678)

[dustingetz]

I'm gonna have to fix this in November, likely, for client work. I can also look into doing enterprise grids from userland renderers, the issue being that they will lose reagent value renderers. I think it is no problem to portal them back in against a reaction, but it is probably better to just respect the grid's native API, disallowing recursion, etc. Most grids will surely have a first-class api to apply React.js fns.

[dustingetz]

• react-tablulator - has virtual scroll
  • https://codesandbox.io/s/oxmj02v696
  • http://tabulator.info/examples/4.0

[dustingetz]

I am okay with the server refusing to respond to queries with >100 results, in combination with adding a nice way to search (mostly ux right?) – may be very high bang for buck perf hack.

[dustingetz]

https://techblog.commercetools.com/advanced-data-tables-in-react-dbe33f8345ab

• https://nadbm.github.io/react-datasheet/
• https://www.ag-grid.com/example.php#/
• http://adazzle.github.io/react-data-grid/ http://adazzle.github.io/react-data-grid/docs/ReactDataGrid API looks workable
• https://www.grapecity.com/en/wijmo-flexgrid https://www.grapecity.com/en/blogs/wijmo-flexgrid-best-react-data-grid
• https://www.telerik.com/kendo-react-ui/components/grid/advanced-features/hierarchy/
• https://examples.sencha.com/ExtReact/6.6.0/kitchensink/#/

Bad cljs projects

• https://github.com/Kah0ona/re-datagrid

[dustingetz]

An option here is to let a fiddle receive the datomic result value as javascript datastructures (e.g. directly deserialized from transit-js instead of transit-cljs and then converted) - i dont know if this helps

[khardenstine]

FWIW field inference would be tougher with js datastructures

[dustingetz]

Requirements for datagrid

• Update performance
• Reactjs cell renderers that are efficient
• Cell-level buttons that don't disrupt layout (on hover or something)
• Must be able to embed datagrids inside a cell (:many :many) – multidimensional
• Optional: Protocols that can work with cljs datastructures

[dustingetz]

Ways to vizualize recursive pulls with uniform shape (linked lists):

1. Flatten tree like this:

; homogenous tree
(d/pull [{:org/child 10}
         :group/type])

https://www.ag-grid.com/javascript-grid-tree-data/#example-org-hierarchy

; homogenous tree with custom renderer
(d/pull [:db/id
         {:org/direct-report 10}
         :first-name :last-name :job/title :job/area])

http://adazzle.github.io/react-data-grid/docs/examples/tree-view

[dustingetz]

2. master/detail expandos:

; master/detail tree - subtree has different shape
(d/pull [:category/id
         :category/name
         :category/description
         {:category/products
          [:product/id
           :product/name
           :product/unit-price]}])

https://www.telerik.com/kendo-react-ui/components/grid/advanced-features/hierarchy/

[dustingetz]

React-virtualized looks promising, and it would work with our existing tables! It uses array index addressing, so you pass it a count and a cellRenderer(index) which means it can be driven by clojurescript data structures.

https://github.com/bvaughn/react-virtualized/blob/master/docs/Grid.md

[dustingetz]

This is a tree-view, note the inner headers are the same as the outer which is what pull recursion means. So this can fully flatten, and the identity column can indicate nesting with indentation.

[dustingetz]

(defn ui [ks items]
  [:> js/ReactVirtualized.AutoSizer
   (fn [m]
     (reagent.core/as-element
       (into
         [:> js/ReactVirtualized.Table
          {:height 800
           :className "inspector"
           :width (aget m "width")
           :headerHeight 70
           :rowHeight 30
           :rowCount (count items)
           :rowClassName (fn [m]
                           (when (odd? (aget m "index"))
                             "table-odd"))
           :rowGetter (fn [m]
                        (get items (aget m "index")))}]
         (->> ks
              (map (fn [k]
                     [:> js/ReactVirtualized.Column
                      {:key (name k)
                       :label (name k)
                       :dataKey (name k)
                       :cellDataGetter (fn [m]
                                         (let [row (aget m "rowData")
                                               k (keyword (aget m "dataKey"))]
                                           (get row k)))
                       :width 200}]))))))])

[dustingetz]

I got this far with react-virtualized. It is not faster on this simple table.

In this profile, the first is the hyperfiddle.ui/table (xray), and the second is react-virtualized grid. The thrashing where the majority of the time is spent is rendering each individual table-field, on the order of 4ms to 70ms each, average near 15ms.

[dustingetz]

It does help with large results – on the schema query, if you really shrink it down, the initial render is 7.7s down from 16s, and then scrolls a couple seconds.

[dustingetz]

Reagent reaction loop internals is calling context/focus ~7x instead of once (it costs 1ms each time, per cell)

context/focus should be only called once. What's going on with the anon function? Is that a Reagent closure? Here is disassembly:

/**
 * Works in a form or table context. Draws label and/or value.
 */
hyperfiddle.ui.field = (function hyperfiddle$ui$field(var_args){
var args__4534__auto__ = [];
var len__4531__auto___69443 = arguments.length;
var i__4532__auto___69444 = (0);
while(true){
if((i__4532__auto___69444 < len__4531__auto___69443)){
args__4534__auto__.push((arguments[i__4532__auto___69444]));

var G__69445 = (i__4532__auto___69444 + (1));
i__4532__auto___69444 = G__69445;
continue;
} else {
}
break;
}

var argseq__4535__auto__ = ((((2) < args__4534__auto__.length))?(new cljs.core.IndexedSeq(args__4534__auto__.slice((2)),(0),null)):null);
return hyperfiddle.ui.field.cljs$core$IFn$_invoke$arity$variadic((arguments[(0)]),(arguments[(1)]),argseq__4535__auto__);
});
goog.exportSymbol('hyperfiddle.ui.field', hyperfiddle.ui.field);

hyperfiddle.ui.field.cljs$core$IFn$_invoke$arity$variadic = (function (relative_path,ctx,p__69438){
var vec__69439 = p__69438;
var _QMARK_f = cljs.core.nth.call(null,vec__69439,(0),null);
var props = cljs.core.nth.call(null,vec__69439,(1),null);
if(cljs.core.truth_(ctx)){
} else {
throw (new Error("Assert failed: ctx"));
}

var ctx__$1 = hypercrud.browser.context.focus.call(null,ctx,relative_path);
var Body = (function (){var or__3949__auto__ = _QMARK_f;
if(cljs.core.truth_(or__3949__auto__)){
return or__3949__auto__;
} else {
return hyperfiddle.ui.hyper_control;
}
})();
var Head = (function (){var or__3949__auto__ = new cljs.core.Keyword(null,"label-fn","label-fn",-860923263).cljs$core$IFn$_invoke$arity$1(props);
if(cljs.core.truth_(or__3949__auto__)){
return or__3949__auto__;
} else {
return hyperfiddle.ui.hyper_label;
}
})();
var props__$1 = cljs.core.dissoc.call(null,props,new cljs.core.Keyword(null,"label-fn","label-fn",-860923263));
var props__$2 = cljs.core.update.call(null,props__$1,new cljs.core.Keyword(null,"class","class",-2030961996),contrib.css.css,hyperfiddle.ui.semantic_css.call(null,ctx__$1));
var G__69442 = new cljs.core.Keyword("hyperfiddle.ui","layout","hyperfiddle.ui/layout",361246945).cljs$core$IFn$_invoke$arity$1(ctx__$1);
var G__69442__$1 = (((G__69442 instanceof cljs.core.Keyword))?G__69442.fqn:null);
switch (G__69442__$1) {
case "hyperfiddle.ui.layout/table":
return cljs.core.with_meta(new cljs.core.PersistentVector(null, 5, 5, cljs.core.PersistentVector.EMPTY_NODE, [hyperfiddle.ui.table_field,ctx__$1,Body,Head,props__$2], null),new cljs.core.PersistentArrayMap(null, 1, [new cljs.core.Keyword(null,"key","key",-1516042587),[cljs.core.str.cljs$core$IFn$_invoke$arity$1(relative_path)].join('')], null));

break;
default:
return cljs.core.with_meta(new cljs.core.PersistentVector(null, 5, 5, cljs.core.PersistentVector.EMPTY_NODE, [hyperfiddle.ui.form_field,ctx__$1,Body,Head,props__$2], null),new cljs.core.PersistentArrayMap(null, 1, [new cljs.core.Keyword(null,"key","key",-1516042587),[cljs.core.str.cljs$core$IFn$_invoke$arity$1(relative_path)].join('')], null));

}
});

[dustingetz]

Basic react-virtualized hyperfiddle grid

(defn row [ctx k]
  ^{:key (pr-str k)}
  [:> js/ReactVirtualized.Column
   {:label (reagent.core/as-element [hyperfiddle.ui/field [k] ctx])
    :dataKey (hypercrud.transit/encode k)
    :cellDataGetter 
    (fn [m]
      (let [k (hypercrud.transit/decode (aget m "dataKey"))
            ctx (aget m "rowData")
            #_#_ctx (hypercrud.browser.context/attribute ctx k)]
        ;(hypercrud.browser.context/data ctx)
        (reagent.core/as-element 
         [hyperfiddle.ui/field [k] ctx])))
    :width 60}])

(defn cell [ctx jm]
  (let [columnIndex (goog.object/get jm "columnIndex")
        rowIndex (goog.object/get jm "rowIndex")
        react-key (goog.object/get jm "key")
        style (goog.object/get jm "style")
        cols (hypercrud.browser.context/children ctx)
        row (get @(:hypercrud.browser/result ctx) rowIndex)
        row-key (hypercrud.browser.context/row-key ctx row)
        ctx (hypercrud.browser.context/row ctx row-key)
        col-key (get (vec cols) columnIndex)
        #_#_ctx (hypercrud.browser.context/attribute ctx col-key)]
    (reagent.core/as-element
     [:div {:key react-key :style style}
      #_[hyperfiddle.ui/field [col-key] ctx]
      (pr-str [c r])
      ])))

(defn rv-grid [ctx props]
  (let [ctx (assoc ctx 
                   ;:hypercrud.browser/head-sentinel true
                   :hyperfiddle.ui/layout :hyperfiddle.ui.layout/table)
        items (hypercrud.browser.context/data ctx)]
    [:> js/ReactVirtualized.AutoSizer
     (fn [m]
       (reagent.core/as-element
        [:> js/ReactVirtualized.Grid
         {:cellRenderer (partial cell ctx)
          :columnCount (count (hypercrud.browser.context/children ctx))
          :columnWidth 60
          :height 300
          :className "inspector hyperfiddle unp"
          :width (aget m "width")
          :rowHeight 30
          :rowCount (count items)}]))]))

[dustingetz]

https://docs.google.com/document/d/1jeQfyaIsv_RRYEL00Wn6BiPn_KAfVAbPWwHPHpwITHg/edit#

[dustingetz]

TIL that forms should never be the default. Collapse all forms to tables

Never draw this:

Always draw this (even if one row)

Nesting extends horizontally, and the pull expression defines the columns in a way that flattens up to the spanning columns

Proof in Reltron (note the sparse primary key column Germany). Also see how the nesting is indicated in the header and the table is rendered joined (sparse)

Note that tables transpose into forms

If for UX reasons we want to transpose, that can be a custom renderer (that way the scrolling sees more fields, rather than scrolling sees more records)

From the Database perspective, records can be paginated, fields can not.

Implementing this will instantly turn recursive pulls into lists (linked lists as tables) which is correct, this is like tail recursion where the recursion should flatten out to iteration.

[dustingetz]

Here is a datagrid layout that doesn't require popovers

It is stateful, you click on "..." to expand a layer deeper and note the merged-horiz cells. card/many buttons (hf/new & tx) get their own row at entity level. :card/one buttons go inline in the cell (maybe only visible when hovering or focused)

Critically it is tabular/cartesian, so there is not recursion, and it fits into a sheets datagrid ui. There are no forms (vertical layout). However it is possible to do – just transpose the grid for a form (the labels pyramid out to the left).

[dustingetz]

This design "solves" perf because high degrees of nesting can never happen, the UI just elides it until you click.

[dustingetz]

https://github.com/tannerlinsley/react-table/releases/tag/v7.0.0