BUGFIX: Reflect discard of node move changes correctly in the UI

[grebaldi]

fixes: #3184

NOTE: This PR needs to be tested in combination with its companion PR over at neos-development-collection: https://github.com/neos/neos-development-collection/pull/4291

Buckle up for collisions with https://github.com/neos/neos-ui/pull/3569

The overall problem

This one's a bit complex and it stretches over two repositories. The basic scenario is:

1. You move some (document) node(s) to a position below a different parent node (so, just sorting them within their own hierarchical level doesn't count)
2. You discard those changes
3. Chaos ensues

https://github.com/neos/neos-ui/issues/3184 describes the issues that happen only partially. During my investigation I found several problems that I need to disect piece by piece.

(I) The UI does not (entirely) recognize that nodes have been moved to a position below a different parent node

The problem

https://github.com/neos/neos-ui/assets/2522299/9ce3bdb9-35d7-423d-90af-b2c55a009c9d

In the video you can see that after the two document nodes have been moved, the change is not made visible by the usual orange change indicator on the left hand side of the tree nodes. Also, on the attempt to discard the current set of changes, an error occurs that reads:

Call to a member function isRemoved() on null - Check logs for details

Both phenomena are related, because - as it turns out - the UpdateWorkspaceInfo feedback object, that is supposed to inform the UI about pending changes, delivers the wrong node context paths for the nodes that have just been moved (the context paths are still the old ones).

Because the context paths are now out of sync, the UI is unable to associate the pending change with the respective tree node. It then also uses the stale workspace information as payload to the discard command, which leads to the above error at the following line:

https://github.com/neos/neos-ui/blob/5c52e08b8a1effc985911390a7124579c4018c25/Classes/Controller/BackendServiceController.php#L272

How come the context paths are incorrect after the nodes have been moved?

After some investigation I found that the Neos\ContentRepository\Domain\Factory\NodeFactory class memoizes stale data - as opposed to e.g. ContentContext which gets its in-memory cache flushed when nodes were moved:

https://github.com/neos/neos-development-collection/blob/0e28ef208111f5b1576df9078bb9abfedf12500c/Neos.ContentRepository/Classes/Domain/Factory/NodeFactory.php#L81

When the PublishingService is asked for unpublished nodes via getUnpublishedNodes, it receives cache hits within NodeFactory->createFromNodeData for the nodes that have just been moved - thus the old context paths.

EDIT: Not at all true :sweat_smile:! I investigated further after the functional tests over at neos-development-collection failed. The actual reason is as follows:

The Neos UI API uses a slightly different content context configuration than getUnpublishedNodes. So, NodeFactory actually keeps track of two variants of the moved nodes. The ones that getUnpublishedNodes receives are not the ones that the move operation has been performed on.

The solution

I modified the way the content context for node change operations is build. It is now ensured that the content context sees everything by setting the invisibleContentShown, removedContentShown and inaccessibleContentShown flags all to true.

Related Commit(s): https://github.com/neos/neos-ui/pull/3503/commits/f300e933836926bb563f7283bf702634b2feb98c

(II) Discarded move changes are not properly reflected in the document tree

The problem

Problem (I) can be circumvented by hard-reloading the UI (after that, the workspace info will be correct again). But, there's still some strangeness going on...

https://github.com/neos/neos-ui/assets/2522299/9f4e6d55-9c92-4590-8b1a-362532f7f799

In the video you can see that the tree actually reflects the discarded changes correctly for a brief moment there. It then quickly jumps to a broken state in which the nodes that should be at their original positions just disappear.

This problem persists even if you hard-reload the UI after discarding:

https://github.com/neos/neos-ui/assets/2522299/7085ee21-91a3-428a-a5e0-55bc0a7899a6

Now, if you use the reload button of the document tree to manually reload the tree, the nodes reappear:

https://github.com/neos/neos-ui/assets/2522299/c7b2dafd-8113-4a28-a3a8-ac3bc56675f8

(But also: If you hard-reload the UI again, the nodes will once again flash briefly and then disappear)

How does this happen?

In those videos, the parent document that originally contained the moved nodes is focused and open in the guest frame. After discard, it should contain those nodes again. The UI reloads the guest frame, but the document is now rendered with stale node metadata. After the guest frame finished loading, the stale metadata (which still thinks the nodes have been moved elsewhere) overwrites the node data in the UI redux store.

This is why the correct state shows up for a brief moment. It gets overwritten after a short delay when the guest frame is loaded. (Also: The nodes do not disappear if you focus a different document and hard-reload the UI).

Looking at the cache configuration for the node metadata:

https://github.com/neos/neos-ui/blob/5c52e08b8a1effc985911390a7124579c4018c25/Resources/Private/Fusion/Prototypes/Page.fusion#L26-L46

... one should actually assume that the data shouldn't be stale (Neos.Caching.descendantOfTag(documentNode) should do the trick). It turns out though, that the Neos\Neos\Fusion\Cache\ContentCacheFlusher class - in case of discard - will only flush tags related to a node's current workspace. We actually need to have all tags flushed in the base workspace as well to cover the DescendantOf_*-tag of the original parent node.

The solution

The solution for this problem is implemented over here: https://github.com/neos/neos-development-collection/pull/4291

(III) Discarding a node move while having a moved document node open in the guest frame results in an error page

The problem

https://github.com/neos/neos-ui/assets/2522299/7ddda073-47b7-4587-bf3f-ad5c50bdf411

The video shows that when you're currently editing a moved document and then discard the move change, the guest frame reloads and shows a misleading fusion error. This is because the guest frame tries to render a document node that doesn't exist anymore.

A similar situation would be a document that has just been created. If you stay on that document and then discard it, the UI behaves correctly and redirects you to the next-higher document:

https://github.com/neos/neos-ui/assets/2522299/bce265f5-507e-4dd2-bb05-9c72d53564a0

Here, the problem lies within the discardAction of the Neos\Neos\Ui\Controller\BackendServiceController which does not recognize discarded move changes and thus misses to inform the UI that it needs to remove the nodes at their former positions and re-insert them at their original positions.

The solution

I modified the body of the discardAction method of the BackendServiceController. It now explicitly detects node move changes and adds the corresponding RemoveNode and UpdateNodeInfo feedbacks.

Related Commit(s): https://github.com/neos/neos-ui/pull/3503/commits/c4bad624dd40000aaad1003f3e3a883926d4b768

All solutions combined

Here's what it looks like when the PRs in neos-ui and neos-development-collection are combined:

https://github.com/neos/neos-ui/assets/2522299/2904f731-e101-4033-b617-5c196a44bdec

[grebaldi]

@crydotsnake Have you tested this in combination with https://github.com/neos/neos-development-collection/pull/4291?

[crydotsnake]

@crydotsnake Have you tested this in combination with neos/neos-development-collection#4291?

No, i didnt. I have totally overread that 🤦🏽 will try it again..

[grebaldi]

No, i didnt. I have totally overread that 🤦🏽 will try it again..

Yeah, I probably should have pointed that one out more clearly :sweat_smile: Sorry for the trouble! I'll put up a note at the top of the PR description.

[crydotsnake]

No, i didnt. I have totally overread that 🤦🏽 will try it again..

Yeah, I probably should have pointed that one out more clearly 😅 Sorry for the trouble! I'll put up a note at the top of the PR description.

No! you described everything as good as possible! 💙 was definitely not your fault.

[grebaldi]

Hi @markusguenther,

Would it be beneficial to enhance the existing drag-and-drop (DND) end-to-end test with these scenarios?

Oh yes, definitely. I'll look into that.

I can use the patch-technique I've used in https://github.com/neos/neos-ui/pull/3569 to get a green PR and make it easier to reproduce the E2E results locally. We only need to remind ourselves to remove the patch before merge :)

EDIT:

Unfortunately, the Neos pull request is currently facing some blockers.

What are the blockers?

[markusguenther]

What are the blockers?

Maybe blocker was a wrong word ... meant the performance concerns of @Sebobo regarding the context creation.

[markusguenther]

Thank you for your pull request. I appreciate your hard work on this fix.

After discussing this with @mhsdesign, we have decided to postpone merging this change. The main concern is that merging a change this late in the development cycle of 7.3 could potentially introduce regressions.

We would like to thank you for your understanding. Meanwhile, please feel free to continue working on this fix, and we will target Neos-UI 8.3 then.

Thank you again for your contribution 💙

[neos-bot]

🎥 End-to-End Test Recordings

• Recording 1
• Recording 2

These videos demonstrate the end-to-end tests for the changes in this pull request.

[grebaldi]

UPDATE:

I've rebased this branch on 8.3 and added 3 E2E test cases (see: cca575705915704648e54f0fbc37a4357c4c354e) that will fail on 8.3 and turn green with this PR.

To make those test cases run in CircleCI, I've temporarily added a patch to the test distribution, so that https://github.com/neos/neos-development-collection/pull/4291 is incorporated during the run (see: 302da1d18074efeae17d67a0cc584f60d49c410c). That commit must be reverted before this PR can be merged.

To ease verification, here's a summary of the test cases plus before/after videos:

Scenario #1: Moving nodes and then discarding that change does not lead to an error

This corresponds to (I) The UI does not (entirely) recognize that nodes have been moved to a position below a different parent node.

The test takes two document nodes, moves them into another node, discards all changes and asserts that no error message has appeared.

Test run on 8.3

https://github.com/neos/neos-ui/assets/2522299/23ce9a9b-1e8f-491c-901c-a42c88308578

Test run on bugfix/3184/document-move-discard (incl. patch)

https://github.com/neos/neos-ui/assets/2522299/39a5a314-f5d1-45fd-b045-afbbbc9efa40

Scenario #2: Moved nodes do not just disappear after discarding the move change

This corresponds to (II) Discarded move changes are not properly reflected in the document tree and requires the patch for https://github.com/neos/neos-development-collection/pull/4291.

Like Scenario #1, this test takes two document nodes and moves them into another node. Afterwards it focueses the old parent, reloads the UI and discards all changes. It then asserts that the two moved documents have not disappeared.

Test run on 8.3

https://github.com/neos/neos-ui/assets/2522299/4f621102-0672-40cf-bcba-0b86ca4c5349

Test run on bugfix/3184/document-move-discard (incl. patch)

https://github.com/neos/neos-ui/assets/2522299/8f68bb4d-458f-4fbe-aac7-459a14dbe71f

Scenario #3: Discarding a move change while being on a moved node does not lead to an error in the guest frame

This corresponds to (III) Discarding a node move while having a moved document node open in the guest frame results in an error page.

Like the other two scenarios, this test takes two document nodes and moves them into another node. It then reloads the UI, focuses one of the moved nodes and discards all changes. It then asserts that no error is shown in the guest frame and that the now focused node is the next-higher document node (in reference to the previously focused node).

Test run on 8.3

https://github.com/neos/neos-ui/assets/2522299/67a68806-7e0d-47e6-97cc-16dcbe8e1492

Test run on bugfix/3184/document-move-discard (incl. patch)

https://github.com/neos/neos-ui/assets/2522299/f9cc7a79-7a22-4995-b601-81f30c321c16

[Sebobo]

Should we merge this fix asap?

[mhsdesign]

@Sebobo and me wondered how much this change is dependant on https://github.com/neos/neos-development-collection/pull/4291 and what are the implications if this is merged and release first, or the other way around.

FIX #3184: Discarded node move changes are reflected correctly in the document tree

Description	plain 8.3	only Neos fix	only Ui fix	both fixes	error
Scenario 1: Moving nodes and then discarding that change does not lead to an error	❌	❌	✅	✅	Call to a member function isRemoved() on null
Scenario 2: Moved nodes do not just disappear after discarding the move change	❌	✅	❌	✅	Document nodes disappeared in tree
Scenario 3: Discarding a move change while being on a moved node does not lead to an error in the guest frame	❌	❌	✅	✅	Sorry, the page you requested was not fount

... but the rest of the ui continues to function