Developer demo of alternate wiki store implementation

[Jermolene]

This PR introduces two alternate implementations of the tiddler store object:

• [x] A smallest possible plain JS wiki store implementation, sharing as much implementation as possible with the existing core implementation with as little code duplication as possible
• [ ] An implementation based on the WebAssembly build of sqlite3
  • [x] Wasm usage from a file URI without requiring external files
  • [x] Sql implementations of main wiki store methods
  • [x] Pass test suite with sqlite3 store
  • [x] Tag indexer optimisations
  • [ ] Explore further optimisations

Testing with the current prerelease wiki, the sqlite3 implementation is currently significantly slower than the standard plain JavaScript wiki store implementation in boot.js.

• Plain JS store:
  • mainRender: 89.20ms
  • styleRefresh: 4.00ms
  • mainRefresh: 12.00ms
• sqlite3 store:
  • mainRender: 300.00ms
  • styleRefresh: 380.30ms
  • mainRefresh: 175.80ms

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Dec 11, 2023 8:43pm

[pmario]

I always thought, that the TW concept could have "additional" $tw.wiki stores if needed. ...

With the implementation like this there can only be 1 type of store. Is this intended or is it just an experiment?

[linonetwo]

In plugin, we always call $tw.wiki, so a singleton is better.

Just need some way to hook into existing $tw.wiki

[Jermolene]

I always thought, that the TW concept could have "additional" $tw.wiki stores if needed. ...

I think that's an orthogonal goal, which would be accomplished by reclassifying the existing $tw.wiki instance as the root admin wiki, and allow for a dynamic set of child wikis that can be created and deleted without a restart.

With the implementation like this there can only be 1 type of store. Is this intended or is it just an experiment?

An alternate store along these lines could still expose its own constructor so that further instances could be created.

[joshuafontany]

Thanks for starting to sketch this out. I think this would allow tighter integration between TW and data-stores like the Yjs CRDT document(s). :)

[linonetwo]

About how to use sqlite and if it should be a plugin https://github.com/Jermolene/TiddlyWiki5/discussions/7531

[linonetwo]

How to support multiple plugins?

I want to create a sqlite plugin that works on nodejs server side, loading tiddler and indexer from sqlite instead of fs, then do swr from fs. I also want to create a plugin that work on client side, to get indexer&tiddlers using ipc, so we can skip most of parsing on client booting. And there might be a plugin you want to create, work on client side, using wasm sqlite to cache content...

[Jermolene]

Hi @linonetwo the "demo-alternate-store" plugin in this PR is intended to be a starting point for people wanting to write their own custom alternate store plugins. The current implementation only works in the browser but will be extended to work under Node.js as well.

To support multiple alternate store plugins being loaded at the same time we will also need a mechanism for selecting between them.

[Jermolene]

I've updated the PR to include the start of a sqlite3 wiki store plugin. At this point, the sqlite3 engine is initialised and tested with some test queries. The next step is to wire it up into the wiki object.

[btheado]

At this point, the sqlite3 engine is initialised and tested with some test queries. The next step is to wire it up into the wiki object.

Nice work. I especially like the code showing how to embed wasm into a tiddlywiki. If someone else wants to embed something wasm, then using your code as a starting point should make it easy.

It looks like you are using in memory database. Do you plan to experiment with persistence?

An alternate wiki store which has persistent backing seems like it would be useful, but I'm confused about how it would work. All tiddlers in the html store area will be loaded into the store. If that store persists, then you wouldn't want to load the tiddlers from the html store the next time the page is loaded. Otherwise they might overwrite any changes which were made.

Or am I on the wrong track here? Maybe an automatically persisting store is not what you have in mind for these alternate stores?

[linonetwo]

Otherwise they might overwrite any changes which were made.

@btheado we can use swr, try search stale-while-revalidate in HTTP RFC 5861

I don't have clear idea about how sqlite work with tw store, so I'm working on other project, also hoping to learn from this implementation...

BTW, TidGi 0.8.0 just released, it has a server-side sqlite for you to play around.

[Jermolene]

Nice work. I especially like the code showing how to embed wasm into a tiddlywiki. If someone else wants to embed something wasm, then using your code as a starting point should make it easy.

Thank you! I am indeed interested in making it as easy as possible to incorporate Wasm functionality. It would be useful to have another example – perhaps something similar to https://squoosh.app/ for image compression.

It looks like you are using in memory database. Do you plan to experiment with persistence?

At this point I am exploring using sqlite3 as a volatile store just for its querying and indexing capabilities. I am also deliberately starting by integrating sqlite in the browser where the persistence options are more limited than under Node.js

An alternate wiki store which has persistent backing seems like it would be useful, but I'm confused about how it would work. All tiddlers in the html store area will be loaded into the store. If that store persists, then you wouldn't want to load the tiddlers from the html store the next time the page is loaded. Otherwise they might overwrite any changes which were made.

Yes, exactly. It would perhaps be more of a matter of importing an external TiddlyWiki HTML file, rather than opening it in place in the traditional way.

Or am I on the wrong track here? Maybe an automatically persisting store is not what you have in mind for these alternate stores?

I do absolutely think there are some interesting options there. For example, we might establish a new format based on sqlite3 for TiddlyWiki files that only contain tiddlers, and rely on an external app to work.

[joshuafontany]

@Jermolene Thank you for picking this up & advancing this demo. I got a new phone last week & almost lost access to my GitHub account (deleted my Auth app & the other 2FA setting were not correct). Back now!

[pmario]

@Jermolene ... I did switch to this branch and I did create an empty.html which is a 3.992KB in size. From your comments I saw that's expected.

Starting the file wiki with FF 115 needed about 24 seconds. .. Is that expected.

The console showed the following. Is that expected?

Ignoring inability to install OPFS sqlite3_vfs: Cannot install OPFS: Missing SharedArrayBuffer and/or Atomics. The server must
emit the COOP/COEP response headers to enable those. See https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep

The wiki loaded, but then everything as in every key-stroke needs about 4-5 seconds to do anything. Clicking the NewTiddler button about 10 seconds. It shows a tiddler. Editing every keystroke 3-5 seconds.

I did test with Windows Edge. Exactly the same thing. Except Edge pops up a window to kill the process, where I have to select "Wait".

I think that behaviour is not expected?

[Jermolene]

Starting the file wiki with FF 115 needed about 24 seconds. .. Is that expected.

Yes. At the moment everything is super slow because I'm using a custom JS collation function so that we can precisely match the ordering performed by the original wiki store. The plan is to have a switch to turn that off.

Also, there's currently no indexing happening, so getting all tiddlers with a particular tag does a full scan of the wiki.

The console showed the following. Is that expected?

Yes as far as I can tell.

The wiki loaded, but then everything as in every key-stroke needs about 4-5 seconds to do anything. Clicking the NewTiddler button about 10 seconds. It shows a tiddler. Editing every keystroke 3-5 seconds.

I did test with Windows Edge. Exactly the same thing. Except Edge pops up a window to kill the process, where I have to select "Wait".

I think that behaviour is not expected?

Yes that is expected at this point.

[pmario]

OK. Thx for the info. I only did skim the code. ... So every future performance improvement will actually feel like one. .. clever ;)

[Jermolene]

Thanks @pmario – and no problem, those were good questions that I was happy to answer, and hopefully will be helpful for others too.

So every future performance improvement will actually feel like one.

That's the hope! The ultimate goal is to compile TW filter expressions into SQL queries...

[Jermolene]

I've updated the OP with the latest progress on the sqlite3 store

[Jermolene]

I've done some work to simplify the schema that we use. At this point everything works, but performance is terrible because it doesn't yet have the tag optimisations that we previously had.

[Jermolene]

I've now completed a major refactoring consisting of the following steps:

• c52014c66ffbf460392ae9ed5b824ad4153265e0 simplifies the SQL schema by shifting more of the work of handling shadow tiddlers from read operations to write operations like saving/deleting tiddlers. This update also temporarily removed most of the optimisations for reading tiddlers by tag
• d4dec0ca65d349c0f02aa6de997dc9e0cd6b2cea switches from using NULL as a special plugin title indicating a non-shadow tiddler to using the empty string. Using NULL was working against the grain of SQL, and using strings consistently allows the code to be much simpler
• f49b9faab0730234883a938aac386842fe42ba11 restores the additional tables used for more efficient tag handling
• 12c6cb35a0336ee44a61a26a643db368b019e45e adds indexes to all the columns used in joins. I didn't notice much impact on performance

Performance is better than before, but not decisively so. I plan now to spend some time profiling the code to find the bottlenecks.

I'd welcome any help with any analysis or optimisations. As ever, questions and suggestions welcome.