Proposal: a DocumentFragment whose nodes do not get removed once inserted

[WebReflection]

edit I wouldn't mind Node.DOCUMENT_PERSISTENT_FRAGMENT_NODE name/kind neither

TL;DR

The document fragment is a great primitive to wrap together numerous nodes and append these directly as batch, however a fragment loses all its children as soon as appended, making it's one-off usage limited in those cases where a list of nodes, at a certain position, is meant.

This proposal would like to explore the possibility of a live document fragment that:

• does not lose its childNodes once appended
• it's still transparent / unaddressable from CSS
• it's also transparent for any other node so that it's still possible, as example, to append many <TD> or <TR> through such fragment, and keep a reference for future updates

Why

Both virtual DOM based libraries, such React, as well as direct DOM based one, such as hyperHTML, or lit-html, have been implementing their own version of a persistent fragment in a way or another.

If there was a primitive to directly reference more than a node, through a fragment with a well known position on the DOM, I am pretty sure all libraries would eventually move to adopt such primitive, so that a tag function could handle both <p>1</p> and <p>1</p><p>2</p> without needing to re-invent a similar wheel every single time, and making portability between libraries and frameworks easier than ever: it's just a DOM node!

Example

// either persistent or live
const pf = document.createLiveFragment();

// append zero, one, or any amount of nodes
pf.appendChild(document.createElement('TD'));
pf.appendChild(document.createElement('TD'));

// make it live
const lastTr = document.querySelector('#data tr:last-child');
lastTr.appendChild(pf);

// references are still there
pf.childNodes.length; // 2

// the node is invisible though
lastTr.lastChild === lastTr; // false
lastTr.lastChild === lastTr.childNodes[1]; // true

There should be no way to interfere with CSS and/or selectors, the fragment is either referenced somewhere else or it won't exist for the DOM.

How

The way hyper/lit-html are doing this is by abusing comment nodes as boundaries of these virtual fragments. The itchy part of these libraries is mostly represented by these virtual fragments, 'cause it's obvious if the primitive proposed here would exists, these libraries would've used it instead (happy to be corrected, but at least I would never create my own virtual fragment if I could use something else).

The way this could be implemented, is by weakly referencing nodes to such fragment only if this is held in memory.

// example implementation of the live fragment
const references = new WeakMap;
class LiveFragment extends DocumentFragment {
  #childNodes = [];
  #appendChild = node => {
    this.#removeChild(node);
    this.#childNodes.push(node);
  };
  #removeChild = node => {
    const i = this.#childNodes.indexOf(node);
    if (-1 < i)
      this.#childNodes.splice(i, 1);
  };
  appendChild(node) {
    this.#appendChild(node);
    return super.appendChild(reference.call(this, node));
  }
  append(...nodes) {
    nodes.forEach(this.#appendChild);
    return super.append(...nodes.map(reference, this));
  }
  removeChild(node) {
    this.#removeChild(node);
    references.delete(node);
    return super.removeChild(node);
  }
  // the value of this LiveFragment when moved around
  valueOf() {
    this.append(...this.#childNodes);
    return this;
  }
}
function reference(node) {
  references.set(node, this);
  return node;
}

// amend on the appendChild standard
const {appendChild} = Node.prototype;
Node.prototype.appendChild = function (node) {
  appendChild.call(this, asFragment(node));
};

const {append} = Element.prototype;
Element.prototype.append = function (...nodes) {
  append.apply(this, nodes.map(asFragment));
};

function asFragment(node) {
  return node instanceof LiveFragment ?
    // it could be just node.valueOf() for everything
    // explicit here for explanation sake
    node.valueOf() :
    node;
}

Possible F.A.Q. Answers

• if a node is manually appended somewhere it's fine. But as soon as the LiveFragment owner will append such fragment again, that node would be moved back (nodes ownership by creator)
• if the LiveFragment has no references, then nothing changes from a fragment
• a live fragment always exposes its childNodes as immutable, as it is for regular fragments
• everything is the same, except that live fragments creators, owner of their live fragment content, can use this primitive instead of polluting the DOM with comments
• it is always possible for DOM engines to know if a node belong to a fragment, as long as this is still referenced somewhere. If unnecessary, due forced re-append on valueOf(), the references part can be ignored

---

Thanks in advance for eventually considering this, happy to answer any possible question.

[WebReflection]

FWIW this is apparently a very demanded feature from the Web Developers community

Previously on a similar proposal: https://discourse.wicg.io/t/proposal-live-fragments/2507

[WebReflection]

As simplified approach, and for demo sake, I'll leave a playground that works out of the box in Chrome Canary (but not yet in Code Pen, however I've informed them about it) https://codepen.io/WebReflection/pen/moRQRV?editors=0010

Now I'll wait for any outcome 👋

[WebReflection]

@annevk just FYI I think there's a typo in the title: onse => once

also, if I might ask, what does the label "needs implementer interest" mean?

What can I do to move this forward? Should it be me the implementer?

Thanks.

[annevk]

Do https://whatwg.org/working-mode#changes and https://whatwg.org/faq#adding-new-features help?

I suspect we'll need something like this to build templating on top of.

cc @rniwa @justinfagnani @wycats

[developit]

This is interesting. Other names I've seen used to casually refer to something along these lines are "Persistent Fragment" (given that it doesn't empty when appended) and just "Range Fragment".

Example: https://discourse.wicg.io/t/proposal-fragments/2312

[justinfagnani]

Edit: I didn't see that in this version of the idea the fragment doesn't live in the tree. I think that makes it functionally equivalent to NodePart from TemplateInstantiation, or a wrapper around StaticRange, and addresses most of the issue below, which were based on other variations of the idea I'm familiar with.

@annevk I think the Template Instantiation Part interface is essentially this, without being a Node living in the childNodes list. display: contents also covers some similar use cases.

One concern with a new node type is that much existing tree traversal code will not know how to handle it, so a live fragment and its children will likely be skipped. Depending on where such live fragments are intended to be used this may or may not be a problem. There is a lot of code out there that assumes that only Elements can contain other Elements.

Another concern is that right now I believe that ever Node subclass reachable by tree-walking a document is serializable to HTML (including <template> and it's unique innerHTML implementation). The fact that DocumentFragment cannot be placed into a childNodes list preserves this. I'm not sure how much this matters, but this would be the first node type in the tree (but not tree-of-trees) that can't survive a serialize/parse round-trip.

We'd also have to consider how other APIs work. Do live fragments show up on event.path? Can children of a live fragment be slotted into the fragment's parent's ShadowRoot? etc...

I'm not sure if the issues are insurmountable, but I've been working on Template Instantiation with the theory that the least disruption will be caused with a new non-Node interface that lives outside the tree, like Range. Then all existing tree processing code will work as-is.

[rniwa]

Yeah, we've definitely considered this approach before making the template instantiation proposal but fundamentally, all we need is tracking where the inserted contents need to be. I don't think there is any reason to create a new node type and keep it in the DOM if we can avoid it.

[WebReflection]

Nothing is kept in the DOM. It's a fragment that acts like a fragment. I'm working on a better playground that shows the idea fully polyfilled so please wait for it to be online and evaluable before closing this, thanks.

[WebReflection]

So, I've uploaded the previously mentioned polyfill, which should have 100% code coverage. https://github.com/WebReflection/document-persistent-fragment

There is a live test page too.

the what

The idea is to have a 1:1 DocumentFragment alter-ego that doesn't lose its nodes.

The fragment is exposed only to the owner, so that there is no way to retrieve it from third parts, unless passed around, and there's nothing live on the DOM, if not, eventually its child nodes.

The proposal exposes to the owner common nodes methods based on its content.

As example, dpf.previousSibling would return the dpf.firstChild.previousSibling if there is a firstChild. If the dpf.isConnected is false, the result is null.

The DPF (in short) has only one extra method, compared to DocumentFragment, which is dpf.remove(). Such method doesn't need much explanation: it removes all nodes from the DOM.

All operations performed through the DPF are reflected live on the document, and while this might be just a stretch goal, it is super easy and nice to simply update an owned reference and see everything changing live.

Nodes ownership

It is possible to grab random nodes and destroy these, or change these, affecting indirectly the content owned by the DPF instance, but it's always been possible to be obtrusive on the DOM and destroy third parts libraries so I think this shouldn't be concern.

However, I could implement the WeakMap that relate each node to a specific DPF instance in a way that it's not possible to move a node owned by a DPF into another DPF, or even perform any action on the DOM through elements that are not the DPF owner.

This, however, would introduce an ownership concept that is too different from what we've used so far, but I believe this proposal is for all libraries that need such persistent fragment, and that owns their own nodes, libraries that are currently somehow already breaking things if 3rd parts obtrusive libraries destroy, or manipulate, DOM nodes in the wild.

As summary

The fact, beside some Safari glitch I'm sure I can solve, this whole proposal can be already polyfilled, and the fact browsers have a way to optimize it and make it blazing fast, should be considered as a plus, 'cause instantly adoptable by the community, so that we can have quick feedbacks of how much this is welcomed or needed out there.

Please don't hesitate to file issues there or ask me more here before discarding this proposal.

Thank You.

[WebReflection]

@rniwa FYI I've filed the bug that makes current polyfill ~~not usable~~ * in WebKit/Safari

edit I've fixed the current polyfill with a workaround after a feature detection, so this can work on Safari/WebKit too 👋

[rniwa]

Nothing is kept in the DOM. It's a fragment that acts like a fragment.

Then what you created is indistinguishable from NodeTemplatePart we proposed. It's just a matter of syntax / naming differences.

[thysultan]

@rniwa NodeTemplatePart seems to be tightly linked to shadow dom and template instantiation; how would you use NodeTemplatePart to replicate the proposed?

[rniwa]

@rniwa NodeTemplatePart seems to be tightly linked to shadow dom and template instantiation

It's nothing to do with Shadow DOM.

how would you use NodeTemplatePart to replicate the proposed?

In the latest iterations of the proposal @justinfagnani at Google and we're working on, NoteTemplatePart is a thing that could be used without any template although we probably need to rename it to something else.

[WebReflection]

@rniwa I've no idea what is this NodeTemplatePart and I cannot find anything online, however, if you would read at least the test you'll see this proposal is literally nothing new, it's fully based on current standard, backward compatible, and polyfillable.

It's a document-fragment at all effects, it's indeed inheriting the same constructor, but it works transparently and only if the owner/creator keeps a reference around.

If this is exactly what this NodeTemplatePart does, can you please point me at it's specification or API?

Thanks.

[WebReflection]

@justinfagnani

Edit: I didn't see that in this version of the idea the fragment doesn't live in the tree.

would you mind amending/canceling that comment since nothing in there is relevant to this proposal, so that people don't get distracted by concerns that are not part of this proposal?

links to the solutions previously discussed would be more than welcome too.

Thanks.

[annevk]

The context you're missing is https://github.com/w3c/webcomponents/blob/gh-pages/proposals/Template-Instantiation.md and some F2F discussion (probably somewhere in minutes linked from issues in that repository) that encouraged making the parts there true primitives instead of tightly coupled with templating.

(Also, please try to consolidate your replies a bit. Each new comment triggers a new notification for some and there's over a hundred people watching this repository. When in doubt, edit an existing reply.)

There's another meeting coming up, and I hope @justinfagnani and @rniwa can make the current iteration a bit more concrete by then, as referencing it in this issue as if it's a thing everyone should be aware of is a lil weird.

[ryansolid]

The context you're missing is https://github.com/w3c/webcomponents/blob/gh-pages/proposals/Template-Instantiation.md and some F2F discussion (probably somewhere in minutes linked from issues in that repository) that encouraged making the parts there true primitives instead of tightly coupled with templating.

Thank you, I'm glad I read the second part of that comment about true primitives. I had read that proposal before and admittedly a little terrified that the DOM spec would include such an opinionated implementation. That spec reads like designing a render framework. I mean I'm sure we could do worse, but I'm encouraged to know that simpler proposals are under consideration.

What I like about this proposal is the transparent use with Node API's appendChild/removeChild since it can be treated interchangeably with actual DOM nodes when say returned from some sort of Template instantiation.

I think ownership is the challenge with a lot of value comes from having a clear single ancestor(whether single node or fragment). It lets JS context be tied to specific parent nodes without making more elements in the DOM. But by their very nature I don't see how you'd avoid nesting. Like a loop over these fragments with a top level conditional in it that also returns a fragment. In the end the DOM structure would be pretty flat but you'd have persistent fragments in persistent fragments. Since they aren't part of the actual rendered tree it becomes harder to understand what falls under each since nested dynamic content in this case could change what is in the top fragment. I would absolutely love to see a solution in this space having spent a lot of time figuring out tricks to accomplish similar things. Everyone writing a render library where they don't virtualize the tree hits this issue sooner or later.

[WebReflection]

you'd have persistent fragments in persistent fragments

which is fine, and since appending a fragment to itself breaks, there's nothing different on the DOM.

Current implementation / proposal allows shared nodes between fragments, which is the same as creating a fragment on the fly and append any node found in the wild: nothing stops your from doing that, everything works, no error thrown.

The current idea is that keeping it simple is the only way to quickly move forward, while any ownership concept would require bigger, non backward compatible, changes.

Libraries and frameworks authors won't worry about that anyway, 'cause they are the one creating nodes too, and they are those virtualizing thee in trees.

Having a mechanism to move N nodes at once, accordingly with any DPF appended live, it's also a feature that would simplify state-machine driven UIs.

Last, but not least, hyperHTML has this primitive since long time and it works already, but it doesn't play super nice with the DOM if thrown there as is, and it requires special treatment when used right away.

This proposal would cover that case can much more.

[Jamesernator]

I'm not sure how much this matters, but this would be the first node type in the tree (but not tree-of-trees) that can't survive a serialize/parse round-trip.

Actually CData sections and processing instruction nodes are simply completely broken in text/html because they don't deserialize (instead they just become text nodes) but do serialize. Not that anyone really cares about them.

[jhpratt]

Is there any continued interest in this proposal? Would a persistent fragment have access to some DOM methods like replaceWith()? Those could be heavily optimized in situations like element reordering.

[ryansolid]

I like this proposal a lot more after sitting with it a bit longer. At first I was thinking this was about library code managing the moving and managing of ranges of elements, but this is more. This helps with giving the users of said libraries the equivalent of React JSX Fragments. Like consider:

const div = html`<div></div>`
const frag = html`<div></div><div></div>`

// further down
const view = html`<div>${ condition ? div : frag }</div>`

I had a user ask why the div always worked, but the second time they attached the fragment why did it not render anything. The answer of course was that div.appendChild(frag) removed the childNodes from the fragment so on the second attach there were no childNodes to append. Now one could argue I could make frag an array (slice the childNodes) but that doesn't handle dynamic top level changes in the fragment.

In a library based on KVO Observables top level dynamic updates that execute independently of top down reconciliation having a standardized DOM API that works the same whether attached or not is hugely helpful in this scenarios.

Beyond that all these libraries have a similar concept. Being just a DOM node works very consistently with the whole

// tagged templates
const el = html`______` //or
// jsx
const el = <______ />

way of approaching rendering which has been gaining steam (looking at the API's on the top end of performance in the JS Frameworks Benchmark). More and more non-virtual DOM libraries are picking this approach and showing it is performant.

[WebReflection]

Hey @vitaly-t, how is that related to this proposal?

[jimfranke]

@WebReflection What is the current status of this proposal, is it still going forward? I think it would be an elegant solution to the various fragment implementations in most virtual-dom libraries.

[WebReflection]

@jimfranke I like the openness of this group but what I'm experiencing is that me proposing stuff never lands anywhere, I guess 'cause I don't work for big Co. and also developers needing these things don't bother helping out bumping issues or repeatedly asking for the current state (so that vendors don't forget developers need it).

In few words, thanks for bumping this up, but unfortunately I have no idea what's the status of this proposal, or if anyone worked on it at all, but I know for sure this would solve tons of libraries issues out there, but we need developers asking for this more prominently, imho.

[rniwa]

@jimfranke I like the openness of this group but what I'm experiencing is that me proposing stuff never lands anywhere, I guess 'cause I don't work for big Co. and also developers needing these things don't bother helping out bumping issues or repeatedly asking for the current state (so that vendors don't forget developers need it).

It's not so much that you don't work for a big corporation. As I've clarified before, what you're suggesting is very much similar to NodeTemplatePart we proposed. So this isn't going anywhere as so far as we're not putting any effort into making NodeTemplatePart reality now because we're putting more energy into fixing issues with the existing web components features like accessibility, focusing order, server side rendering, etc... basically things that should have been fixed years ago. As unbelievable as it may sound, a lot of very basic things with regards to shadow DOM are still in limbo even after years upon years of discussion.

I'm sure we'd get back to this sometime in the future. There is a TPAC virtual meeting suggested. If you'd like to present this idea and discuss, I think we'd to be more than happy to do that.

[wycats]

@rniwa as somebody who's pretty interested in this proposal, and who has given a bunch of ad-hoc feedback to people about it when I was asked, I have to agree with @WebReflection that it's really hard to follow the progress of any aspect of this proposal and continue to provide useful feedback.

I don't think that's because anyone is acting nefariously at all, and I agree with you that it's possible to piece together what's going on if sufficiently motivated. But on the other hand, this proposal is not structured to elicit feedback from the people who would be most affected by it, and I think that could be improved.

[rniwa]

@rniwa as somebody who's pretty interested in this proposal, and who has given a bunch of ad-hoc feedback to people about it when I was asked, I have to agree with @WebReflection that it's really hard to follow the progress of any aspect of this proposal and continue to provide useful feedback.

Are you talking about the proposal in this issue, or are you talking about the template instantiation proposal we made?

FWIW, we're not actively soliciting any feedback for our template instantiation proposal at the moment because we're still working through the last round of feedback we got a few years ago. I agree we should put together a new document & designate an avenue for active discussion at some point.

[wycats]

Are you talking about the proposal in this issue, or are you talking about the template instantiation proposal we made?

I'm talking about the proposal in this issue, as it relates to the template instantiation proposal.

From my perspective, the fact that there's an active, but private template instantiation proposal, as well as active but more limited interest in DOM placeholders makes it very hard to make progress as someone who's not actively working on the design.

EDIT: I would personally be very interested in fleshing out and helping to champion a placeholder proposal. I would be able to easily use the feature in Glimmer (Ember's template engine) and I believe that the same would be true of other web frameworks. I would also be willing to do some legwork to verify that hypothesis.

If there's any interest, I'd happily do so.

[wycats]

For further clarification, I understand that the OP here doesn't specifically propose precisely placeholders, but since I have the use-case the OP is talking about, I feel equipped to say that the solution I have in mind (and have discussed with @annevk and a few others in the past) addresses the use-cases envisioned by the original proposal.

[WebReflection]

I understand that the OP here doesn't specifically propose precisely placeholders

FWIW my proposal goal is more or less about having these placeholders, as I believe this is an extremely common use-case for most used modern frameworks/libraries/helpers, and as much as that use case is covered, I don't really care much how such use case is implemented in details, as long as there is a primitive to help us covering it.

The only concern I have is that templates have nothing to do with this proposal/use case, they can surely benefit from it, but I hope these won't be coupled to this new feature, as it looks like it was driven by a single team/library, instead of a wider audience of developers behind various DOM rendering engines.