Best approaches for dealing with sequences of items - items-view, keyed collections, etc.

[aaronc]

[ul]

I'm near to complete draft of my version of items-view where I trying to answer this question partially. Please, give me some hints on debugging & benchmarking my solution — e.g. how to trace VDOM & DOM operations performed by freactive.dom engine.

[aaronc]

So, in terms of debugging and benchmarking, what I usually do is put println's where I need them and wrap portions that need to be benchmarked with the time macro. I also use Chrome's developer tools including the profiler and debugger there... Other than that, I don't really have any default trace statements builtin but maybe that will be an option in the future.

[aaronc]

I'd suggest we take the space here to describe approaches we're thinking of even if they're not fully implemented.

One idea I have is a bind-keys macro with the form of:

(bind-keys tag-kw attr-map? [key-expr keys-expr] & body) ;; keys-expr is implicitly wrapped in a `rx`

Ex:

(bind-keys :div [key (keys @my-map)]
  (let [item-cursor (cursor my-map [key])
        local-state (atom {:a 0})]
    [:div (rx (:value @item-cursor))
          (rx (:a @local-state))]))

This would be the equivalent of:

(rx
  [:div
    (for [key (keys @my-map)]
      (let [item-cursor (cursor my-map [key])]
        [:div (rx (str (:value @item-cursor)))]))])

Except that it only binds each key once and only once - so even if there's reordering, local state is preserved and new nodes don't need to be created. I think this approach is maybe half-way between what React does and what I was proposing for my items-view. There could still be a more full-fledged items-view but this might be a good ad-hoc way to do what React does with keys in a way that is actually slightly more efficient and I think more "Clojure-like".

To bind to a vector with this, you could write something like:

(bind-keys :div [i (range (count @my-vector))]
  (let [item-cursor (cursor my-vector [i])]
    [:div (rx (str (:value @item-cursor)))]))

What do you think?

[ul]

But when number of elements in map or vector changes, whole block will be rerendered? I like core idea of freactive to deliver changes exactly to the point of interest, and I really want it to be implemented for lists of items. So, now I really like idea of items-view if it will do the job even if it will not look like simple wrapper. I'm stuck trying to guarantee more-or-less consistent state snapshot for sequences w/o doing much of computations. I found O(NP) implementation of sequence diff and I use proxy atom to hold state snapshot and mapping of proxy indices to source indices to not rerender items that were not changed even if their indices have been changed. But I don't understand how to make more-or-less efficient update of this mapping while keeping consistency of elements, proxy and source.

[aaronc]

Hi Ruslan,

So, what I am suggesting with bind-keys would be efficient - it would only do one binding per key. So there is no need to do a complex sequence diff - we just find out which keys/indices are added and removed - it is the user's responsibility to create a cursor from a given key. There is no need to worry about not re-rendering things that weren't changed because we simply won't create new elements and keep a map of the old elements. Even if the order changes it's just a matter of calling insertChild, appendChild to reorder DOM elements. Local bindings are designed to stick to actual DOM elements so even if they're moved the bindings will still work. When we have a full-fledged items-view/observable-collection framework this can all be very efficient...

On Mon, Dec 1, 2014 at 1:01 AM, Ruslan Prokopchuk [email protected] wrote:

But when number of elements in map or vector changes, whole block will be rerendered? I like core idea of freactive to deliver changes exactly to the point of interest, and I really want it to be implemented for lists of items. So, now I really like idea of items-view if it will do the job even if it will not look like simple wrapper. I'm stuck trying to guarantee more-or-less consistent state snapshot for sequences w/o doing much of computations. I found O(ND) implementation of sequence diff https://github.com/brentonashworth/clj-diff and I use proxy atom to hold state snapshot and mapping of proxy indices to source indices to not rerender items that were not changed even if their indices have been changed. But I don't understand how to make more-or-less efficient update of this mapping while keeping consistency of elements, proxy and source.

— Reply to this email directly or view it on GitHub https://github.com/aaronc/freactive/issues/23#issuecomment-65024073.

[luxbock]

In the README it says:

IObservableCollection could eventually be extended to support a database-backed collection and then we have something like Meteor in Clojurescript..!

I've been playing around with DataScript and Freactive, with this kind of approach. There's two different things I came up with:

• The first one is a type that treats a DataScript entity as though it was an atom, so we can use swap! and reset! on it and also create cursors from it.
• The second approach creates a cursor into the DataScript connection, and you'd probably want to use it with the db-cursor helper.

I don't really know how IObservableCollection works since it doesn't appear to be implemented yet, so I just wanted to ask if you think I'm on the right track here or if you would use a different approach altogether, and how you think all of this might play along with the implementation of item-view / bind-keys?

[aaronc]

So I think either approach would work. The cursor approach would also give you something "atom-like". Observable collections would have special collection cursors which might be closer to your first approach.

The idea with the observable collection is that it would let you manage a whole "collection" of entities so there would also be notifications when entities are added and removed... and individual entity cursors would be notified about changes to that entity only. My idea for observable collections is that basically they would basically be observable collections of observable cursors if that makes sense...

I'm actually planning on posting a gist about this soon for discussion - I'll post the link here when I have it.

[aaronc]

Okay, here's a gist I posted of my ideas for observable collections (another one coming on the actual items-view soon): https://gist.github.com/aaronc/0654151190b9145dd473

[aaronc]

Just posted this gist about the actual items-view for discussion: https://gist.github.com/aaronc/5d497aa61e27ce924178

[aaronc]

Arghh... PLEASE Ignore my previous comments about discussing this in the gist - gist does not support comment notifications - PLEASE discuss here instead.

[ul]

Hmmm... I was sure that I have posted my current items-view implementation here 2 weeks ago... But now cannot find it. Anyway, https://gist.github.com/ul/552e1ba67718e3a01a45

[ul]

I actually like your items-view spec, but do not understand why we need observable collection to be exposed to the user? As a developer using freactive and its items-view plugin as a library I will be happy to use reactive atoms/cursors everywhere and pass to items-view usual reactive cursor (which is awesome as it is, because it also could be binded to database or whatever).

Also, please, describe how this kind of plugin could be implemented in freactive, it is fantastic (I mean using plugins as custom namespaced elements):

[:freactive/items-view
  {:items items
   :container [:ul]
   :template (fn [item] [:li @item])}]

[aaronc]

Well, the benefit of having observable collections exposed to the user is that it removes the need to do diffing when the collection is updated because you can track changes locally. i.e. when someone calls (update! coll :a inc), the collection knows exactly that the element at key :a changed and knows to notify its cursor and only its cursor; and to update the sort order based on this change.

Something like what you are proposing - using a diffing library - would be useful in the case where people want to use plain old cursors and atoms, but I think would be significantly heavier in terms of code size and algorithmic complexity. Still, it's probably useful to have an implementation of observable collection based on diffing like what you've written. I can see a good use case say for someone who has existing business logic already based on regular atoms and cursors.

In general, I think we should have items-view depend on a generic observable collection interface - then someone can use my observable-map & observable-vector, or a diff-based wrapper, or datascript, etc.

By the way, for observable collections, I think now that ITransactableCollection is more fundamental than what I described for IObservableCollection - I plan to update my gist to reflect this. The reason for this has to do with the items-view needing to receive the transaction log to do efficient sorting. Does that make sense?

[aaronc]

Regarding custom namespaced plugins, it's pretty simple. When freactive gets a tag it does something like:

(let [tag-ns (namespace tag)
       tag-name (name tag)]
  (if tag-ns
    ....
    ....))

So we can have strings or functions be registered as node and attribute namespace handlers - something like: (register-node-prefix! prefix fn-or-string). Strings will be interpreted as an xml namespaces. In the function case, we'll allow handlers such as (fn [node node-state attr-name attr-value]) for attributes and (fn [tag-name tail]) for nodes.

[aaronc]

Also, because freactive.dom/bind-attr* is pretty generic it can be the basis for binding attributes to custom "nodes" such as the items-view for instance - thus bindable sorting, filtering, etc.

[ul]

Sounds very sweet (both observables and registering handlers for element namespaces), can't wait to see implementation! Will think how I can contribute to it.

[aaronc]

Okay so the most important thing right now is deciding on the right IObservableCollection API - this is the common thing that will be shared between any potential observable collection implementations and any items view implementation...

This is what I'm thinking of for the collection watch mechanism (updated from what I posted last night):

(defprotocol IObservableCollection
  "Defines the minimum protocol required for an observable collection
  to be bound to an items-view."
  (add-collection-watch [coll key f]
    "Where f is a functional taking 3 arguments:
      the key, the collection, a sequence of changes in the form of:
        [[key1 new-value1]
         [key2 new-value2]
         [key3] ;; missing new value indicates the element was removed
         ]")
  (remove-collection-watch [coll key]))

Any comments?

The advantages to this approach as I see it are:

• A change set is basically a set of key value pairs - similar to doing seq on a map and everything could be an argument to assoc or dissoc
• Change sets can be quickly assoced and dissoced to a sorted view of items for efficient reordering
• Changes can be delivered easily to the correct cursor and only that cursor (if a map of cursors is maintained)
• It is easy to derive this type of list from observed assoc and dissoc like operations

A possible disadvantage is:

• Doesn't indicate whether something was an insertion or update, but then again assoc never specifies whether it's an insertion or update so I'm not sure it matters

[twashing]

Hopefully this is the current thread for discussion on contributions to an items-view component? There seem to be 2 at the moment.

1. Clean up items-view, observable collections and document #12, with freactive-observable-collections.md and freactive-items-view.md
2. Best approaches for dealing with sequences of items - items-view, keyed collections, etc. #23

I'm building a collections-based UI at the moment. So I'd be keen on contributing any code that I can. I'll just need to get my head wrapped around the Observable concept. I see the ObservableCollection and it's usage.

A) I have a use case where there's a list of items. And the system will need to drilldown on one particular item. Here, i) the list and ii) the drilldown detail are both views that are composed into a larger main view. I would need the render of ii) to depend on actions in i). Ie, if a user updates or adds an item, the single view ii) updates based on the selection of i). The current approach seems to assume that cursors for each item in i), will point to separate views. Is this correct?

B) I'll also need to chain drilldown views. Can we cascade cursors? So can there be a iii) third view , with a cursor that depends on cursor ii)?

C) Freactive's cursors seem to be similar to Om's Reference Cursors. I'm not sure if the items-view needs to consider some of the other use cases brought up in that library.

• Request Channel: How do we get data from an ancestor component without having to pass data to everyone in between the root of the application and where the component actually lives?
• The Publish & Notification Channels: Component B and Component C do not share a common parent Component A yet they need to communicate to each other.

[aaronc]

Hey Tim, so I'm not sure I understand the scenario you're describing in A) and B) but what I can say is that freactive observable collections would be able to wrap cursors and nested cursors. So we could have a scenario something like this:

(def state (atom {:a {0 1 2 3} :b {4 5 6 7}}))
(def state-observable (observable-map state))
(def a (observable-map (get-cursor state-observable :a)))

So a is sort of a nested observable-map. Does this address some of what you're asking? If not, maybe you can clarify a little bit your use cases.

Regarding the overall design, I want to reiterate that the observable maps & vectors and the items-view will be completely independent components only linked via an IObservableCollection protocol - so if something doesn't work the way you need it to it should be possible to swap in custom cursor, observable collection, and items view implementations (as long as IObservableCollection is reasonably generic).

For C), as I understand it every atom, cursor and rx in freactive would behave somewhat like a reference cursor in om except that the observe step happens automatically (by this reactive data binding mechanism). For channels, we would just use core.async channels as in om.

[twashing]

Ok, this is great. It definitely addresses points B) and C). Let me ask A) in a different way. Let's say I have a list of items, generated by freactive's list-view component. When I click on one of the items, what mechanism renders a "detail view", and binds a cursor to any DOM controls? Is that "detail view" rerendered (and mounted) on every click? Or is that detail view just changing based on a reactive atom, or some other thing?

Also are you building items-view out, now? Can I help in any way? Thanks.

[aaronc]

Your sub-item will be generated by the :template parameter which takes a (fn [cursor-to-item]) which should return a view for that item (see: https://gist.github.com/aaronc/5d497aa61e27ce924178). You could bind an :on-click handler to whatever view your template generates. Maybe :template isn't the right name for this? Maybe it should be :view-fn or something? Not sure...

I hope to build the items-view soon - I was actually hoping to do it last week over the holiday but didn't get time. I'll let you know when I get started if I need something. Right now, the most critical piece is defining the IObservableCollection protocol. See: https://github.com/aaronc/freactive/issues/23#issuecomment-67572743

[twashing]

Ok, that makes sense. Maybe this is an implementation detail. But the only other thing I'm wondering about, is where that :template (or :view) will get mounted onto the DOM tree. The current implementation is a function that generates the view structure. It's up to the calling code to insert that generated view into the DOM tree.

I'm wondering how that will happen in this updated approach. If it's not yet decided, i) maybe a path to a DOM location? Or just providing a parent DOM id?

[aaronc]

So, my current design is to have the items-view function simply return a DOM node for the entire items view. freactive allows DOM nodes to be passed directly in as child nodes in virtual DOM (hiccup) vectors or arguments to mount!, append-child!, etc. so that is how the items view itself can be mounted. Now, the items-view will manage mounting whatever is returned by the :template function for each item in the DOM inside the element specified by :container (and after any :header or :footer elements). It will also manage moving the elements returned by :template within the :container based on sorting, filtering and ranges applied to the items view. This moving of DOM nodes will be done in such away respects data binding (i.e. :template can return an rx and the items view will just move whatever node is currently bound). Does this answer your question?

[twashing]

So, my current design is to have the items-view function simply return a DOM node for the entire items view. freactive allows DOM nodes to be passed directly in as child nodes in virtual DOM (hiccup) vectors or arguments to mount!, append-child!, etc. so that is how the items view itself can be mounted.

Ok cool.

Now, the items-view will manage mounting whatever is returned by the :template function for each item in the DOM inside the element specified by :container (and after any :header or :footer elements).

So the _:container_ will be an ID to a mounted DOM node, into which items-view will mount the _:template_? The reason I ask is that my use case will have an existing DOM node in which I'll need to mount that :template view.

It will also manage moving the elements returned by :template within the :container based on sorting, filtering and ranges applied to the items view. This moving of DOM nodes will be done in such away respects data binding (i.e. :template can return an rx and the items view will just move whatever node is currently bound). Does this answer your question?

Ok awesome.

[aaronc]

Well :container will be something like [:div] although I guess you could theoretically pass in a DOM element that you retrieved by ID. :container will be passed to build-dom-element (https://github.com/aaronc/freactive/blob/develop/src-cljs/freactive/dom.cljs#L884-L910) which takes either virtual DOM and builds an element or a real DOM node and just passes it through. Just curious - why do you need to have the items displayed in an existing DOM node? Why not just wrap in a div or span?

[twashing]

My situation is that I have a list view and a details view which live side-by-side on the same web page. When a list item is clicked, I need the already mounted detail view DOM to be replaced with the contents of that _:template_.

Does that make sense? I'd want to be sure that I can specify a location (the _:container_?) to a mounted DOM node, into which items-view will mount the :template.

[aaronc]

Okay, so I think these are two separate things. You don't need an items view at all for the details view if I understand what you are trying to do correctly. An items view is only for list view type things and the :template function is called for each item in the vector/map of things. What you can have is some local state for the selected item (an atom/cursor/whatever) that changes when you select an item in the list view. The details view can be bound to that selected item. Does that make sense?

[twashing]

Okay, so I think these are two separate things. You don't need an items view at all for the details view if I understand what you are trying to do correctly. An items view is only for list view type things and the :template function is called for each item in the vector/map of things.

Correct.. kind of :) Isn't _:template_ what would provide that detail view rendering (tell me, if I'm wrong)? The core of my question is where the :template result (or its :container), gets mounted.

What you can have is some local state for the selected item (an atom/cursor/whatever) that changes when you select an item in the list view. The details view can be bound to that selected item. Does that make sense?

This is what I first attempted. Of course, changes in that detail view, need to get propagated back to the original item in the list view (using a cursor).

I thought the only way to seed a cursor into a detail view, would be to re-render that view, with a new cursor to the selected item. If DOM nodes are listening to a cursor, simply resetting that cursor will update the DOM nodes?

[aaronc]

:template provides the view for each item in a sequence (list view), not for the detail view.

Take this for example:

[:ul
 (for [i [0 1 2 3]]
  [:li i]]

This code would generate the exact same DOM except that it is reactive - i.e. the text of each :li will update if the element at that index is changed and if elements are appended to the list they'll be added.

(def my-data (observable-vector (atom [0 1 2 3])))
(items-view
 {:container [:ul]
  :template (fn [item-cursor] [:li item-cursor])
  :items my-data})

You can definitely share cursors between views and yes changes to the cursor in any location should propagate to both views. You could have a selected item atom or cursor which points to the key or to the selected cursor itself that the detail view is bound to. Everything bound to state should respond reactively to changes in that state.

[twashing]

A ha, so that was my misunderstanding. That makes sense now. I'll try it out and let you know how I do.

Thanks for the feedback :)