Introduce multi-valued variables and let filter run prefix

[Jermolene]

This PR introduces a new filter run prefix :let that assigns the result of the filter run to a variable that is made available for the remaining filter runs of the filter expression.

It solves the problem that otherwise it is impossible to compute values for filter operator parameters; parameters can only be a literal string, text reference or variable reference. The :let filter run prefix provides a solution:

[my.custom.function<param>,[2]] :let[[index]] [<data>jsonget[top],<index>]

The shortcut syntax => can also be used:

[my.custom.function<param>,[2]] =>index [<data>jsonget[top],<index>]

Using the shortcut syntax also avoids having to use double square brackets around the name of the variable.

This example first sorts all the input tiddlers by length, and then uses the length of the shortest title as the second index of a JSON lookup:

[all[tiddlers]] :sort:number[get[text]length[]] :let[[index]] [<data>jsonget[top],<index>]

This PR also introduces the ability for multi-valued variables which contain list of results, not just single strings. They can be assigned with the new :let filter run prefix, or the existing <$let> widget. The full list of values can be retrieved using round brackets instead of the usual angle brackets. In all other contexts only the first item in the list is used as the variable value.

Note that this PR is intended to be a more generic replacement for the experiment :apply filter operator that is currently part of #8702 - see the code and the docs

[github-actions[bot]]

Confirmed: Jermolene has already signed the Contributor License Agreement (see contributing.md)

[netlify[bot]]

✅ Deploy Preview for tiddlywiki-previews ready!

Name	Link
🔨 Latest commit	ac7837a98f0811e9273a3e969063ef137d4b3cb2
🔍 Latest deploy log	https://app.netlify.com/projects/tiddlywiki-previews/deploys/690d250cf32d6000080385af
😎 Deploy Preview	https://deploy-preview-8972--tiddlywiki-previews.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[pmario]

I do get an RSOD if I do :let:test in the AdvancedSearch Filter tab.

Show screenshot

[pmario]

I am not overly happy with special handling of a :let:test[all[]] -- all[] usually means all tiddlers. Now it means only the first one. IMO that's inconsistent and confusing.

May be

• all[first] -- a b :let:index[all[first]] [<data>jsonget[top],<index>]

IMO first would also make sense in general, so we could remove the "special" handling?!

---

---

I did play with: [tag[HelloThere]format:titlelist[]join[ ]] :let:test[all[]] [enlist<test>] to test several elements. It's complex

I could imagine an all[prerun] which would give us a much shorter version of the above test filter [tag[HelloThere]] :let:test[all[prerun]] [<test>enlist-input[]]

[pmario]

The RSOD only triggers if the input is typed. Not if copy / pasted. :letabc also triggers an RSOD

[Jermolene]

I do get an RSOD if I do :let:test in the AdvancedSearch Filter tab.

Thanks @pmario that should be fixed now.

I am not overly happy with special handling of a :let:test[all[]] -- all[] usually means all tiddlers. Now it means only the first one. IMO that's inconsistent and confusing.

I've extended the implementation of variables so that they can store a result list as well as a single value. This allows the let filter run prefix to assign the entire results to the variable. The new varlist[name] operator retrieves a variable as a result list.

[pmario]

@Jermolene --- There is still a problem with: btest :let:test[prefix[b]] [varlist<test>] Should create a RSOD if copy/pasted into Advanced Search -> Filter tab.

[pmario]

The following filter causes an RSOD: a :let:var[get[aa]] [varlist[va]]

When I do type [varlist[va]] see va is not var yet it gets an undefined -> ROSD with

Also creates a RSOD: [varlist[var]]

uncaught exception: Linked List only accepts string values, not undefined

title: a
aa: aaa
bb: bbb

+
title: b
aa: xxx
bb: yyy

[Jermolene]

@Jermolene --- There is still a problem with: btest :let:test[prefix[b]] [varlist<test>] Should create a RSOD if copy/pasted into Advanced Search -> Filter tab.

Thanks @pmario, fixed in https://github.com/TiddlyWiki/TiddlyWiki5/commit/a31df85894236c2f4de193848213b23f6b1d19c7

You're probably aware that in the example you're actually attempting to retrieve the multivalued variable called "btest", which doesn't exist. You might have intended btest :let:test[prefix[b]] [varlist[test]].

[Jermolene]

a :let:var[get[aa]] [varlist[va]]

Thanks @pmario this problem is also fixed by https://github.com/TiddlyWiki/TiddlyWiki5/commit/a31df85894236c2f4de193848213b23f6b1d19c7

[pmario]

You're probably aware that in the example you're actually attempting to retrieve the multivalued variable called "btest", which doesn't exist. You might have intended btest :let:test[prefix[b]] [varlist[test]].

At first I did try the later filter. Then for testing purposes I did also test varlist<test>.

[Jermolene]

Thank you for the feedback. I have just pushed an update that dispenses with the need for the special <$letlist> widget. Instead, we update the <$let> widget so that it always assigns the complete list of results from evaluating the attributes. The <<var>> syntax and the filtered transclusion {{{filter}}} syntax are both able to generate a list of results.

This arrangement is backwards compatible because all the existing usages of variables only look at the first value, and ignore the rest of the values.

I was very surprised that this was possible, and actually only realised that it was as the result of an accident during coding. Good to reduce the number of moving parts.

[Jermolene]

I've pushed a further update that simplifies the let filter run prefix. It now assigns the input list to a multi-valued variable whose name is taken from the first result of the filter run. This highly contrived example shows the syntax:

[all[tiddlers]] :let[[varname]] [varlist[varname]sort[]join[,]]

With this reversal of the ordering of the elements of the assignment I do wonder if we can do better than "let" as a name.

[xcazin]

Hi @Jermolene, wouldn't :push and pop work as respective replacements for :let and varlist ? Likeso:

[all[tiddlers]] :push[[varname]] [pop[varname]sort[]join[,]]

It may confuse programmers in languages where the argument of push() and pop() is the content, but in the context of filters, it reads rather well.

[Jermolene]

I've added a shortcut syntax => for the let filter run prefix:

[my.custom.function<param>,[2]] =>index [<data>jsonget[top],<index>]

Using the shortcut syntax also avoids having to use double square brackets around the name of the variable.

[pmario]

@Jermolene -- I think there is a problem somewhere. See: https://deploy-preview-8972--tiddlywiki-previews.netlify.app/#Filter%20Operators

No operators are listed. I'm not sure where this could come from.

[Jermolene]

@Jermolene -- I think there is a problem somewhere. See: https://deploy-preview-8972--tiddlywiki-previews.netlify.app/#Filter%20Operators

No operators are listed. I'm not sure where this could come from.

Thanks @pmario I am looking at it now.

[Jermolene]

Thanks @pmario I am looking at it now.

Hi @pmario this was fixed in f914aa6605ba11aa62ba7bd72bab126e3ff3a5b3. I think there are quite likely to be similar issues since these are intricate changes. So I appreciate any help in tracking issues down.

[btheado]

On the preview site, I notice if I edit a tiddler which has a tag, the tag does not appear in edit mode. If I add a tag to the tiddler, the text field is replace by that tag.

[pmario]

@Jermolene -- I can confirm the problem that @btheado reported at: https://github.com/TiddlyWiki/TiddlyWiki5/pull/8972#issuecomment-2746682036

[btheado]

If I add a tag to the tiddler, the text field is replace by that tag.

Here is a simplified reproducer of the issue:

\procedure p1(a:"tags")
<<a>>
\end

*<<p1>>
*<$macrocall $name=p1/>
*<$macrocall $name=p1 a=""/>
*<$macrocall $name=p1 a=<<nonexistentvar>>/>
*<$transclude $variable=p1 a=<<nonexistentvar>>/>

On https://tiddlywiki.com/, the output is

• tags
• tags
• tags
• tags

But on the preview site, the output is

• tags
• tags

[Jermolene]

Thanks @btheado @pmario, there's a fix in b6a171cf5ecc4fd8e57e840f0047932c14c67bf9

[pmario]

@Jermolene None of the following code creates any outputs.

Does varlist or something similar exist? IMO it should for the usecase below

\function enlist.multi(var)  [varlist<var>] +[join[, ]]

<$let test={{{ [tag[HelloThere]] }}}> 

call ''enlist.multi'' function

<<enlist.multi var:test>>

---

transclude enlist.multi function

<$transclude $variable="enlist.multi" var="test"/>

[pmario]

What's about the enlist<multi-var> and <multi-var>enlist-input[] operators. IMO they will also need to know about multi-value variables ?!

[github-actions[bot]]

📊 Build Size Comparison: empty.html

Branch	Size
Base (master)	2438.2 KB
PR	2445.6 KB

Diff: ⬆️ Increase: +7.4 KB