[Discussion] Syncing System tiddlers from Server to Client

[Arlen22]

Describe the bug

https://github.com/Jermolene/TiddlyWiki5/blob/7ec6d3703150469e73dabcf25655192767092a77/core/modules/server/routes/get-tiddlers-json.js#L31-L33

GET tiddlers.json (aka LIST tiddler) has an option to exclude system tiddlers from the list, in the spirit of restricting syncing of system tiddlers to the server, but GET tiddler and PUT tiddler do not check whether system tiddlers are allowed to be synced before saving or serving system tiddlers. This seems a little strange to me, and I would think that this should rather be checked in PUT tiddler to actually prevent a system tiddler from being modified, and the client should be able to detect on its own whether it is supposed to be changing system tiddlers or not.

In other words LIST and GET should have identical behaviour, but PUT should definitely have a check to prevent tiddlers from being written. I don't know where this feature is used, but I'm assuming it is used in cases where there is a custom index file instead of the autogenerated one from $:/core/save/all.

This also seems like a sort of strange feature as it seems like config options would not be saved back to the server. Is the idea here that system tiddlers on the client would be saved into a separate user store? I can certainly see this heading that direction.

[saqimtiaz]

@Arlen22 I can provide a little background on what this config setting does and why it was introduced.

The new syncer introduced in 5.1.22 caused several problems and broke workflows for multi-user and multi-client situations by syncing from the server to the client, tiddlers such as $:/StoryList which previously were only synced from the client to the server.

It was decided to introduce this config setting as a temporary stop-gap measure to allow restoring the behaviour from pre 5.1.22, until a proper solution for configuring what system tiddlers to sync and how is formulated. Per user storage was indeed discussed but is not related to this config setting. See #4882 and #4868

When $:/config/SyncSystemTiddlersFromServer is set to "yes", system tiddlers are synced from the client to the server, but not from the server to the client. This restores the expected behaviour from before the syncer changes in 5.1.22, and means that one user's $:/StoryList does not get synced to other clients.

It may be helpful moving forwards if you give this issue a title that makes it easier to find.

[Arlen22]

@saqimtiaz So what would be the scenario where you would want to save a system tiddler back to the server but not have other clients see that change until they reload the page. $:/StoryList makes sense, because the browser overwrites that when the page loads, but can't it be added to a list of ignored tiddlers somehow? I'm just asking to try to understand better.

[saqimtiaz]

Hardcoding the storylist to be excluded was one of the first proposed fixes. See #4987 for the discussion around this.

[Arlen22]

So I think my concern is that this seems to be implemented somewhat inconsistently. My main question is why do changes still need to be propagated to the server? Or is that not the case? Does eliminating these tiddlers from the list effectively prevent them from being synced back to the server?

[Arlen22]

To be clear, I don't have a problem with the feature, I'm just trying to understand what it is accomplishing because I'm looking into several things related to this.

[Arlen22]

So I think my concern is that this seems to be implemented somewhat inconsistently. My main question is why do changes still need to be propagated to the server? Or is that not the case? Does eliminating these tiddlers from the list effectively prevent them from being synced back to the server?

Actually, I think I answered the question by further reading #4987 .

[saqimtiaz]

@arlen22 It is late for me and I am short on time for TiddlyWiki at the moment overall, having prioritized it above other things to facilitate the 5.1.23 release, so please pardon the brevity of the responses.

Firstly this is a stop gap fix to a bug introduced by the new syncer in 5.1.22. Secondly, an example of a tiddler that needs to be synced to the server, but not back to the client is the $:/StoryList as it is often used in DefaultTiddlers to load the wiki with the tiddlers last open.

Hopefully @Jermolene can help fill in any blanks.

[Arlen22]

I think I had a draft yesterday and then closed the tab. In brief, I propose several changes.

https://github.com/Jermolene/TiddlyWiki5/blob/7ec6d3703150469e73dabcf25655192767092a77/core/modules/server/routes/get-tiddlers-json.js#L23-L24

• Here we should change $:/config/Server/ExternalFilters/ to refer to the title of a tiddler rather than the text of the tiddler. This will make it easier for the user to change the filter as needed for their particular situation.
• Create two filters $:/config/SyncServerFilter and $:/config/SyncClientFilter.
• The $:/config/SyncServerFilter would be requested by the tiddlyweb adapter as the tiddlers.json filter. It would also be used in the changes listener to filter any server-side push-based events, like SSE or websockets.
• The $:/config/SyncClientFilter would be used by the client to filter change events, as is already the case now. In the client, I think it should be appended to $:/config/SyncServerFilter and then filter all the tiddlers. I'm not sure how that plays out exactly but I think it's correct.

[Arlen22]

• Here we should change $:/config/Server/ExternalFilters/ to refer to the title of a tiddler rather than the text of the tiddler. This will make it easier for the user to change the filter as needed for their particular situation.

Because of the programmatic nature of this it seems to make the most sense to store the filter in a tiddler and then whitelist the title of that tiddler. I understand the nature of this but if we really need this functionality we could add a raw parameter for random filters which would only be allowed if AllowAllExternalFilters is yes.

[Arlen22]

While that takes care of the server tiddler list, it would be used for lazy-loading as well, so do we need a separate filter to change the list of tiddlers the server allows to be saved, which could certainly in theory be different if there are readonly tiddlers? I guess this could be SyncClientFilter if it is used on both sides and adds client tiddlers and removes readonly tiddlers. Then the client and server would not try to save readonly tiddlers but would save client-push tiddlers also, but client-push tiddlers cannot be lazy loaded. But I guess that is already the intended nature of this design, because if the syncer sees the tiddler at all it will update it.

[Arlen22]

So SyncServerFilter would be served in tiddlers.json, and also be used to filter the list of changes in any event emitters using server sent events or websockets.

SyncClientFilter would be used by the client to filter the list of tiddlers that are to be saved by the server and would be chained with SyncServerFilter. It appears that the function getSyncedTiddlers is where this would be applied, but I'm not sure. It would also be used in the PUT tiddler and DELETE tiddler routes to validate the tiddler before saving it or deleting it, which I believe is possible using a source function. The SyncClientFilter could certainly subtract itself if needed, which would require it to be updated manually via text editor.

https://github.com/Jermolene/TiddlyWiki5/blob/7ec6d3703150469e73dabcf25655192767092a77/core/modules/syncer.js#L162-L167

The one function I wasn't sure about is the lazy load function. If it is a readonly tiddler (client filter removed from the server filter) it would not be lazy loaded, so I guess this function would need to use the server filter.

https://github.com/Jermolene/TiddlyWiki5/blob/dde4182830f11bfdbcca558065f432556ccf4e65/core/modules/syncer.js#L400-L402

https://github.com/Jermolene/TiddlyWiki5/blob/cc3462999b80461fe30b8f4b4f272ccfbbb78b35/core/modules/wiki.js#L1270-L1278

[Arlen22]

Obviously once we implement bags and recipes in the Wiki class we're going to get a little more complicated, but not much. I think this lays the groundwork pretty well.

[pmario]

Obviously once we implement bags and recipes in the Wiki class we're going to get a little more complicated, but not much. I think this lays the groundwork pretty well.

As soon as recipes and bags are introduced, we would have a "multi-user" environment, similar to TiddlySpace.

There would be no need to restrict any syncing as implemented here. Different users will be able to save their settings back to their "private" bag, which won't effect other users.

[Jermolene]

Hi @Arlen22 could you perhaps summarise your proposals? I'm having trouble following this thread. As @saqimtiaz explained, the fact that the $:/config/SyncSystemTiddlersFromServer config tiddler only affects changes coming from the server is to match the pre-v5.1.22 behaviour.

[Arlen22]

Basically, SyncServerFilter and SyncClientFilter work in tandom. SyncServerFilter would affect which tiddlers get synced to the client through all of the several sync mechanisms, and SyncClientFilter would affect which tiddlers get synced back to the server. I'm thinking that it would be chained to SyncServerFilter so it would add or remove tiddlers to either make tiddlers readonly from the server or write-only to the server.

Here is an example of what I'm talking about.

SyncServerFilter: [all[tiddler]] -[[$:/core]] -[[$:/StoryList]]
SyncClientFilter: [[$:/StoryList]] -[[$:/ServerStatus]] +[tag[Only Save Tiddlers Tagged With This!]]

I'm not sure exactly how that would get executed, but this is what I'm imagining. Additional tiddlers can be added or removed to SyncClientFilter as shown above. As you can see, adding a plus sign should modify everything prior to it in the filter, if I understand right.

let tiddlersToServe = `${SyncServerFilter}`;
let tiddlersToSave = `${SyncServerFilter} ${SyncClientFilter}`;

tiddlersToServe would be used to filter the list of tiddlers that goes from the server to the client (and also to filter the list of lazy-load requests), and tiddlersToSave would be used to filter which changes are saved from the client to the server. Obviously both the server and the client would have this check (the server in the PUT and DELETE tiddlers route and the client wherever needed).

[Arlen22]

• Here we should change $:/config/Server/ExternalFilters/ to refer to the title of a tiddler rather than the text of the tiddler. This will make it easier for the user to change the filter as needed for their particular situation.

I made an attempt at this just now and concluded that it is way easier to create a button in wikitext to change both tiddlers or change it manually in the file system.