Proposal: New method to reorder child nodes

[josepharhar]

Summary

The goal of this method is to allow for the reordering of child nodes without the need to remove them and re-append them. Currently, reordering child nodes (by re-appending) causes several undesired side effects:

• Reloading of iframes
• Restarting of animations
• Loss of selection which cannot be restored in certain input types

This is a problem for several frameworks, including React, Preact, and Mithril.

Adding a new API vs changing existing APIs

Instead of adding a new API, we could change existing uses of DOM APIs to avoid reparenting. For example:

<div id=parent>
  <div id=firstchild></div>
  <div id=secondchild></div>
</div>
<script>
  parent.appendChild(firstchild);
</script>

In this case, you can see that the script wants to move firstchild past secondchild and doesn't necessarily want the browser to remove firstchild from the DOM and reparent it, so we could try to change appendChild to keep the parent throughout the DOM modification and avoid the loss of state.

However, I think a new API would be a better solution:

• Changing appendChild and other DOM modification methods would get more complicated due to difficulties defining and standardizing the existing behavior.
• If there are any sites out there which rely on the full reparenting logic, they could become broken.

API shape

I have some ideas for what this method could look like:

• Provide a full list with all children in desired order. parentNode.reorderChildren([childOne, childTwo, childThree])
• Move one child one spot forward or one spot backwards. childNode.movePastNextSibling() childNode.moveBeforePreviousSibling()
• Move child before or after another child. childNode.moveBefore(otherChildNode) childNode.moveAfter(otherChildNode)

I don't want to bikeshed too much on this until it really sounds like we will add a new method, but if any API shape seems particularly good please speak up so I can start prototyping.

Relationship to https://github.com/whatwg/dom/issues/808

@annevk is https://github.com/whatwg/dom/issues/808 blocking us from having a new way to reorder child nodes? That issue seems more concerned about insertions and mutation events which don't really seem to apply to reordering. Couldn't we just have new spec steps with no special functionality for iframes and scripts and no mutation events?

I made this issue based on feedback in this discussion: https://github.com/whatwg/html/issues/5484

[annevk]

I think without fully understanding (and specifying) #808 it's hard to reason about how moving-within-a-parent should work. I.e., what side effects we need and do not need.

I think the most logical API that follows existing conventions would be parent.moveChildBefore(Node child, Node? referenceChild). However, if sole use case is reordering multiple children parent.moveChildren(Node...) probably makes more sense. Either way this also requires designing new mutation records as this is a new mutation primitive. And we should probably forbid firing (legacy) mutation events.

I would also like to see some more rationale as to why an arbitrary move (with both parents sharing a common root) is less feasible. What are the particular implementation challenges that we would only expose a more restricted move?

[dead-claudia]

I'm not convinced of spec complexity of modifying existing operations being that much higher (though it would technically be somewhat higher): https://github.com/whatwg/dom/issues/880#issuecomment-671033686

[bathos]

If there are any sites out there which rely on the full reparenting logic, they could become broken.

The disconnectedCallback and connectedCallback reactions fire on custom elements when nodes are shuffled like this. It seems very unlikely to me that this would not break sites (including ours).

[josepharhar]

I would also like to see some more rationale as to why an arbitrary move (with both parents sharing a common root) is less feasible. What are the particular implementation challenges that we would only expose a more restricted move?

This is due to @rniwa's comments in https://github.com/whatwg/html/issues/5484 against reparenting iframes without reloading them.

[josepharhar]

I think without fully understanding (and specifying) #808 it's hard to reason about how moving-within-a-parent should work. I.e., what side effects we need and do not need.

Moving within a parent should:

• Not fire mutation events
• Not run script inside of <script> elements, since that is supposed to be run on insertion
• Not fire loading/unloading events for iframes, since those are supposed to happen when the iframe is appended or removed.

Are there other types of side effects I'm not aware of?

[dead-claudia]

@josepharhar Why shouldn't it fire a mutation event of a new type? If a new method is added specifically for moving elements around, they can't possibly fire that mutation without invoking that previously-unknown method anyways, so in theory, a page would have to have its contents updated to experience breakage.

Also, I would expect it to invoke a new hook within custom elements to notify it of nodes being reordered. (This is very useful info for them, BTW, as libraries like A-Frame don't use shadow roots to render their data but otherwise need to know layout order at all times.)

As for the rest, I agree.

[josepharhar]

@josepharhar Why shouldn't it fire a mutation event of a new type? If a new method is added specifically for moving elements around, they can't possibly fire that mutation without invoking that previously-unknown method anyways, so in theory, a page would have to have its contents updated to experience breakage.

Yeah I presume that we would add new mutation signaling stuff for MutationObserver and whatever the equivalent is for custom elements (I'm not super familiar with either yet). I don't think that we should make new mutation events since they seem to be deprecated and people are interested in removing them from the web - I don't have historical context on this though. Right now I'm just trying to work through the relationship between this new proposed method and #808, which is interested in mutation events. To clarify - when I say "mutation events," I am referring to this: https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Mutation_events

[rniwa]

I would really like to better understand what the use cases are. Could someone provide a concrete scenario (not just framework X does Y so they need it) in which this capability is desirable?

[josepharhar]

I would really like to better understand what the use cases are. Could someone provide a concrete scenario (not just framework X does Y so they need it) in which this capability is desirable?

@sebmarkbage @isiahmeadows @marvinhagemeister @developit could any/all of you elaborate on scenarios where state is lost when reordering child nodes? Links to issues would be greatly appreciated, as well as live examples if possible.

@isiahmeadows already made some live examples with four different frameworks listed in the description of #880. It does sound like appendChild has the desired behavior in that particular case though...?

[dead-claudia]

@josepharhar The transitions issue is that of this:

• Partial removal + reinsertion leaves unexpected transition state
• Full removal + reinsertion resets everything

Though now that I'm taking a closer look, I'm not sure movement semantics would help with transitions, as it reproduces even with simple add/remove. So my issue is independent of this.

Edit: Further confirmed that the transitions issue is independent of this.

[annevk]

This is due to @rniwa's comments in whatwg/html#5484 against reparenting iframes without reloading them.

I don't think that discussion was really settled though. Those arguments would apply to moving within a parent as well as you have to contend with subtrees and multiple elements there too.

[josepharhar]

I don't think that discussion was really settled though. Those arguments would apply to moving within a parent as well as you have to contend with subtrees and multiple elements there too.

@rniwa said that "Since a node can't have multiple parents, it needs to be disconnected at some point in some internal engine state. That's precisely what caused the problem." When reordering children instead of reparenting them across the DOM, we will never have to change the pointer to the parent node, only the node's sibling pointers and the parent's first/last node pointers. @rniwa this does address your concern, right?

[rniwa]

I don't think that discussion was really settled though. Those arguments would apply to moving within a parent as well as you have to contend with subtrees and multiple elements there too.

@rniwa said that "Since a node can't have multiple parents, it needs to be disconnected at some point in some internal engine state. That's precisely what caused the problem." When reordering children instead of reparenting them across the DOM, we will never have to change the pointer to the parent node, only the node's sibling pointers and the parent's first/last node pointers. @rniwa this does address your concern, right?

Well, it does address the previous concern but re-ordering nodes isn't something we've ever done so we'd likely encounter new list of issues with it. I'd still like to learn more about what specific use cases would require this new capability, and why careful manipulations of nodes in the user land won't suffice.

[annevk]

@rniwa what problem would be caused by disconnecting an <iframe> from its parent if we can also guarantee that no script would run until it is attached again?

[rniwa]

@rniwa what problem would be caused by disconnecting an <iframe> from its parent if we can also guarantee that no script would run until it is attached again?

Well, that's a big "if". Something like that is not possible to implement in WebKit today.

[techsin]

if DOM had pass by reference equivalent this would not be an issue. Need pointers in DOM.

[rniwa]

@rniwa what problem would be caused by disconnecting an <iframe> from its parent if we can also guarantee that no script would run until it is attached again?

Well, that's a big "if". Something like that is not possible to implement in WebKit today.

Now that I'm thinking about this problem more, one serious challenge is correctly invalidating and updating the computed style of each node when this happens due to things like sibling selector, :first-of-type, :first-child, etc... that could put display: none on object and embed elements since right now, making those elements display: none would unload the "plugin". There might be other subtle challenges that I haven't thought of like what happens to things like focus, selection, etc... since selection end points may clip if the reorder were to happen.

[mindplay-dk]

I'm particularly interested in parentNode.reorderChildren, which could probably greatly simplify the reconciliation algorithms in some UI libraries, where reordering (in as few operations as possible) is often the only really "tricky" part. Reconciliation generally adds a lot of complexity - there are many different implementations with many pros and cons, and the code tends to be fairly difficult to understand.

Some concerns/reservations about this feature though:

1. Adding this feature would most likely be a breaking change in terms of MutationObserver, which would likely require a new type of mutation? Existing code that currently captures all mutations would essentially be incompatible with any newer code that uses the feature - and so, most likely, must be regarded as a breaking change?

2. It probably can't be polyfilled? If you implement a polyfill that uses existing DOM mutations to affect the same change, MutationObserver would broadcast these as individual changes.

3. What happens if you pass nodes that aren't already child-nodes of the given parent? Throwing errors of course is one option. Alternatively, this could work like appendChild and remove the node from somewhere else, if already present in the document.

4. What happens if you omit an existing child-node from the list? Again, throwing errors is an option. More likely, what someone meant to do though, was remove the missing nodes.

(3) and (4) makes me think a method like setChildren might be more meaningful - this would clear out nodes no longer in the given list of node, add or move nodes already present in the document, essentially forcing the list of children into a particular state, in a single operation, which is what most UI libraries are trying to do. This seems more consistent with e.g. appendChild and is probably more generally useful? For example, this would make it easy (and fast) to implement sortable table rows.

[techsin]

regarding parentNode.reorderChildren, there is handy hackish way that this can be sort of achieved without much reordering, by using css order attribute with flexbox.

[mindplay-dk]

regarding parentNode.reorderChildren, there is handy hackish way that this can be sort of achieved without much reordering, by using css order attribute with flexbox.

That won't work for anything like a UI library, where someone would expect to be able to use the order attribute for their own purposes - responsive design, etc. Also, CSS order does not affect tab-order, screen readers, etc. and therefore really has entirely different, totally unrelated uses. Let's stay on topic.

[wlib]

I came across this issue while searching for existing solutions for part of a UI library I'm writing. In essence, the DOM needs a way to move a child node among its sibling nodes without going through the state reset issues caused by having to first take the node out.

This is possible for special cases where we can just move well-behaved sibling nodes .before() and .after() nodes that are trickier to deal with, like document.activeElement, iframes, custom elements with connectedCallback(), &c. For the general case, we start needing to find out how to minimize the nodes that we move - which means we have to find the longest common subsequence between the current and desired childNodes. That translates to massive diff, patch, and state normalizing algorithms - all just to try to minimize the number of times we need to (attempt to correctly) reset reordered element state.

Fundamentally, we could solve this by adding in a primitive operation that moves a node without removing it from its parent first. It wouldn't actually "break" anything except for in a hypothetical pathological case with a third party MutationObserver making bad assumptions, since this operation is equivalent to moving sibling nodes instead. For convenience, we can have analogs for before, prepend, append, after, replaceChildren, and replaceWith that simply don't start by removing their own child nodes.

Just consider how much more efficient it would be to add a .replaceChildren() analog that only had to internally switch its childNodes. Reconciliation (both very time and space-complex) would be massively simplified, leading to browsers decreasing resource consumption by massive amounts for certain classes of UI's. UI libraries get to shrink and browsers have to deal with less code across the board. It would be an absolute shame to block this on the grounds of maybe breaking a backwards third party's MutationObserver.

I'm sure that we will have to also make something along the lines of MutationRecord.movedNodes too. In any case, this issue is much more important than it seems to be getting attention for.

[techsin]

We basically need a swap function that doesn't remove elements from DOM.

I'm not intimately familiar with DOM innerworkings but if DOM is seen as a Tree then it's not clear why swapping is so hard. In a tree structure, if I wanted to swap I'd assign one child to a temporary variable and then temp to node being swapped.

t  = children[1]
children[1] = children[2]
children[2] = t

But maybe overwriting children[1] causes element to be taken out of DOM and wiped clean, DOM is recalculated/rerendered, even though it's added back in next step at children[2]'s position. So I see two solutions:

1. Either have hidden carry over child, that is never visible, so when swapping 1st element it can be assigned to this ever present placeholder child element and therefore remains part of the DOM preventing it getting cut of from DOM tree and wiped clean.

2. Or have batch render/compute. Don't trigger render on element being removed, allow internal swap function to do step 1,2,3 and then recompute/rendering logic.. Preventing children[1] content from being wiped right when it's over written as it's still referred by DOM.

Thoughts?

[rniwa]

Please go read the past discussion in this issue as well as https://github.com/whatwg/html/issues/5484 before making any suggestions or asking why something is hard. A lot of use cases have been already mentioned, and many questions have already been raised as well as corresponding implementation challenges. It's really counterproductive to keep repeating the same discussion every 2-3 years.

[wlib]

Please go read the past discussion in this issue as well as whatwg/html#5484 before making any suggestions ... It's really counterproductive to keep repeating the same discussion every 2-3 years.

I have gone down the rabbit-hole around this issue and I think that an atomic DOM child node move operation would be extremely valuable. Completely ignoring any convenience functions on top - and even going as far as excepting certain tricky cases in order to get this ball rolling - it would still be worth pursuing.

Given that we know this type of operation is highly sought after in libraries like React (to name one), and we know that it would greatly improve web performance in many resource-intensive cases, how do we move forward?

[josepharhar]

an atomic DOM child node move operation would be extremely valuable

Are you suggesting that the reordering idea proposed here isn't good enough, and that we need to be able to move things all around the document? Just clarifying.

how do we move forward?

@rniwa has pointed out several difficulties with adding this capability to the DOM, and there's clearly a lot more complications and corner cases to consider with full tree moving than there is with the reordering proposed here.

I failed to gain traction in this proposal because I didn't get the feedback I wanted from React or Preact in this comment: https://github.com/whatwg/dom/issues/891#issuecomment-690853328 And the Mithril maintainer said that this isn't actually related to their problem here: https://github.com/whatwg/dom/issues/891#issuecomment-690868930

[wlib]

Sorry for the miscommunication, I meant atomic move among siblings, the primitive operation enabling the proposal. The animation problem that the Mithril maintainer mentioned would be orthogonal to this - if I understood it right - because it was due to operations on siblings. Regardless of any complications due to iframes or any other exceptions we can make, this proposal is extremely worthwhile in order to keep input state and prevent unnecessary (dis)connectedCallbacks. Even if you didn't get an enthusiastic comment from a react/preact member here, it's almost guaranteed that you would if this had their attention. This is extremely underrated and you should push for it as much as you can.

[shannonmoeller]

counterproductive to keep repeating the same discussion every 2-3 years

Except to show that this is still very much a current, common, desired feature that shouldn't be cancelled or overlooked due to an apparent lack of participation and interest.

[Bobris]

I actually don't know any UI library/framework which would not benefit from efficient replaceChildren. Efficient and state preserving reordering/adding/removing of nodes is basically their most important and also most difficult to implement function.

[WebReflection]

DOM diffing is what everyone uses these days to make anything on the Web, because it's behind every single library, utility, framework, out there, because lists, baskets after buying, calendars, anything dynamic, needs to update their view without trashing their content all over all the time ... on my side, I've also created and investigated all possible algorithms, including:

• Levenshtein distance based DOM diffing
• E.Meyers's O(ND) Based Diffing Algorithm
• and ultimately, udomdiff, passing through many others, including list-diff, snabdom, stage0, spect, and heckel, but I am sure there are more, including petit-dom attempts, and literally every other utility out there

I believe if we had a primitive for this case, we'll be innovating on other areas, instead of keep solving the same thing we needed to solve since AJAX existed 10+ years ago, but I am sure all these attempts used in the real-world aren't a good-enough evidence everyone would benefit from native diffing, right?

counterproductive to keep repeating the same discussion every 2-3 years

it's also very counterproductive to keep ignoring developers voice for all these years, imho.

[mindplay-dk]

DOM diffing is what everyone uses these days to make anything on the Web, because it's behind every single library, utility, framework, out there

For the record, this is not an exaggeration - even frameworks that do precise updates (Svelte, Solid, Sinuous, etc.) need to perform reordering operations one level deep, when dealing with lists.

It's also worth noting that this feature would support both types of frameworks equally well: frameworks that do precise updates can reorder a single range of child nodes - while frameworks that use a virtual DOM (or some other means of recursive updates) can call this function recursively. In the latter case, having a native reorder method would eliminate most of the complexity either way.

Is it too soon to start thinking about a prototype/proposal/polyfill?

Actually, can this functionality be polyfilled?

It probably can't be precisely polyfilled? If we're talking about details like triggering reflows and repaints etc.?

@WebReflection you have some experience in this area, I think? Any idea how much these details matter? I mean, they matter in terms of perf, which is one reason I think this proposal is relevant - but in terms of things like reflows, repaints, focus management, input and iframe states and so on, a native implementation probably could/should do certain things "better" than what we can do with existing DOM APIs in a polyfill, right? 🤔