Add support for string literal attributes with textual substitution

[Jermolene]

A common use case for macros is to construct a string from component parts. For example:

\define my-url()
https://$(host)$/$(path)$
\end

<$let host="example.com" path="/files">
  <a href=<<my-url>>>
    ...
  </a>
</$let>

It is proposed to introduce a shortcut syntax to cover these simple use cases.

The shortcut syntax uses backticks to quote the string, with the following substitutions:

• $(variable-name)$ is replaced with the value of the variable
• ${filter}$ is replaced with the first result of evaluating the filter

The example above could be expressed as follows using the new syntax:

<$let host="example.com" path="/files">
  <a href=`https://$(host)$/$(path)$`>
    ...
  </a>
</$let>

An example using filters:

<div style.width=`${ [<width>multiply[3]] }$px`>
...

An example of transcluding a tiddler:

<div style.width=`${ [{mytiddler!!myfieldfield}] }$px`>

[CodaCodr]

style.widget? Is that doc'd somewhere?

[Jermolene]

style.widget? Is that doc'd somewhere?

New in v5.2.2. See https://tiddlywiki.com/#HTML%20in%20WikiText, scroll down to the heading "Style Attributes".

[CodaCodr]

Oh, did you mean style.width? "widget" was my confusion. Looking at the example, I assume it's width.

[Jermolene]

Thanks @CodaCodr it was a typo 🙇

[AnthonyMuscio]

I just want to add the ability to include field content, or from a text reference, into "textural substitutions" is also important.

[pmario]

One question popped up, while reading the spec. How can we dynamically create the substitution string.

<div style.width=`${ [<width>multiply[3]] }$px`>
...
<div style.width=`this text should constructed using variables`>

[Jermolene]

I just want to add the ability to include field content, or from a text reference, into "textural substitutions" is also important.

For example:

<div style.width=`${ [{mytiddler!!myfieldfield}] }$px`>

How can we dynamically create the substitution string.

I'm not sure what you mean.

[AnthonyMuscio]

I just want to add the ability to include field content, or from a text reference, into "textural substitutions" is also important.

For example:

<div style.width=`${ [{mytiddler!!myfieldfield}] }$px`>

that would be the way, within the filter.

I could not see it illustrated. Thanks

[pmario]

I think a simple example would be as follows, where \define is just a placeholder for a more complex function, that returns a string. ... May be I'm only not seeing it.

\define action() multiply

<div style.width=`${ [<width>${action}$[3]] }$px`>

[Jermolene]

I think a simple example would be as follows, where \define is just a placeholder for a more complex function, that returns a string. ... May be I'm only not seeing it.

\define action() multiply

<div style.width=`${ [<width>${action}$[3]] }$px`>

Hi @pmario that's a very interesting example – you've got two layers of substitution nested inside each other. I'm not sure that we'd want to recursively apply the substitutions because of the ease of setting up an infinite loop.

[AnthonyMuscio]

A passing depending on how you handle this would the use of text references be possible?

All one needs to test is there is a leading !!, internal or trailing"!!" It could be included in the syntax, the question is can it be processed?

$(!!fieldname)$

$(tiddlername!!fieldname)$

$(tiddlername!!)$ defaults to text, separates it from variable names.

For example;

<$let host="example.com" path="/files">
  <a href=`https://$(host)$/$(path)$#$(!!tiddler-name)$`>
    ...
  </a>
</$let>

Rather than needing the filter as follows.

<$let host="example.com" path="/files">
  <a href=`https://$(host)$/$(path)$#$([{!!tiddler-name}])$`>
    ...
  </a>
</$let>

This above example $([{!!tiddler-name}])$ causes a form of dyslexia where it is hard for the eyes to parse the braces.

[Jermolene]

Hi @AnthonyMuscio

A passing depending on how you handle this would the use of text references be possible?

Using the presence of !! (or presumably ##) to distinguish between text references and variables is not consistent with how we resolve similar problems elsewhere in the design of TiddlyWiki. Generally, we use different types of quotes to distinguish the contents.

EDIT 1st May 2022: added the critical word "not" above

It would be more consistent to recognise the existing {{title!!field}} syntax, but there are a couple of good reasons not to do that:

• The double curly braces are a wikitext syntax, which might give the impression that the templated string is wikified, which of course it is not (and there is no intention to make it do so for performance reasons)
• Using double curly braces would make it impossible to construct templated strings for attributes that happen to contain wikitext including transclusions. For example, something like this would be ambiguous:

<$list ... emptyMessage=`The answer is {{answertiddler}}`

The problem is that the transclusion would be handled by the textual substitution mechanism, and the result would be passed to the emptyMessage attribute, and hence get wikified when rendered. That would mean that we'd end up wikifying the result of the text reference, which would not be intended.

[AnthonyMuscio]

Jeremy,

Regardless of the validity of the approach I suggested to permit fields to be included as references, inside these concatinations, is there a way to address this need "using field values" without forcing the use of filters such as $( [{!!fieldname}] )$ or $( [all[current]getfieldname]] )$ ?

Or is there already a way I have missed?

[Jermolene]

Hi @AnthonyMuscio the proposal at the moment doesn't include provision for textual substitution of text references. The rationale is to keep things simple with a single new mechanism for substituting filter results, only keeping the variable substitution syntax for backwards compatibility.

As it stands, the syntax for the examples you give would be ${ [{!!fieldname}] }$ and ${ [all[current]get[fieldname]] }$.

There is a way to address the need that you raise, which would be to introduce a third substitution syntax (perhaps ${textreference}$ and ${{filter}}$). But I am raising the concern that introducing a third syntax makes things more complicated because it means that users will have to think further about which kind of substitution they are doing.

So, the question is whether it is worth having a special syntax for textual substitution of text references, and the considerations are the complexity that it brings.

[AnthonyMuscio]

As it stands, the syntax for the examples you give would be ${ [{!!fieldname}] }$ and ${ [all[current]get[fieldname]] }$.

This is sufficient and quite powerful as it stands where we can effectively use "any filter" to generate a "string" that will be substituted, and through that even a "text reference" $( [{tiddlername!!fieldname}] )$.

As a result I agree lets not complicate it but instead document it as fields and text references can be substituted through the filter $( filter )$ form.

From my naïve perspective can we say effectively that

any variable can have its value substituted using $(varname)$ with this change not only inside macros (as before) but anywhere

Filters can now be used inside the substitution $( filter )$ and its result will be substitute in its place.

[saqimtiaz]

any variable can have its value substituted using $(varname)$ with this change not only inside macros (as before) but anywhere

Not anywhere, only inside string literal attributes.

Filters can now be used inside the substitution $( filter )$ and its result will be substitute in its place.

Please play closer attention to the syntax. Filter substitutions use curly braces: ${ filter }$

[AnthonyMuscio]

A question; in this manufactured case it only works if you use the wikify widget

\define test() <$text text={{!!fieldname}}/>
<$wikify name=test text="<<test>>">
<$list filter="[<test>match[CamelCase]]">
  go it
</$list>
</$wikify>

If I follow the new substitution's correctly will this or something similar work?

\define test() <$text text={{!!fieldname}}/>
<$list filter="[[${test}$]match[CamelCase]]">
  go it
</$list>

If so that would be wonderful.

[saqimtiaz]

> \define test() <$text text={{!!fieldname}}/>
> <$list filter="[[${test}$]match[CamelCase]]">
>   go it
> </$list>

@AnthonyMuscio no, that will not work. Firstly your example does not use string literal attributes. Secondly, you are passing wikitext where a filter is expected. As Jeremy mentioned earlier in this thread, there is no intention to support automatic wikification due to performance considerations. Wikifying some wikitext to return a string will always be slower than generating it directly via a filter, where possible.

[pmario]

It is proposed to introduce a shortcut syntax to cover these simple use cases.

The shortcut syntax uses backticks to quote the string, with the following substitutions:

• $(variable-name)$ is replaced with the value of the variable
• ${filter}$ is replaced with the first result of evaluating the filter

I think $(filter)$ isn't needed and the limitation to only use the first result shouldn't be there. Same reasoning as in " [IDEA] :foreach filter run prefix" #6781

IMO <$set name=myVar filter="[[a b]] my filter +[nth[2]]"> would do just fine.

It should be possible to even tell the macro, that it should treat the variable as a tiddlywiki list.

<$set name=myVar filter="[[a b]] my filter">
<$text text=<<jsstring "some new text $(myVar)$. " isList=yes>>/>
</$set>

the output may be: (I'm not sure if the macro should iterate over an array )

some new text [[a b]]. some new text my. some new text filter. 

isList defaults to "no", so the string is treated as simple string.

If brackets are a problem the filter can look like this: [[a my]] filter +[join[ ]], which will remove them.

<$set name=myVar filter="[[a b]] my filter">
<$text text=<<jsstring "[[$(myVar)]]$\n" isList=yes>>/>
</$set>

would create

[[a b]]
[[a]]
[[b]]

and so on. ... No code written yet. Just some thoughts.

[Jermolene]

Thanks @pmario

I think $(filter)$ isn't needed

Why?

The idea is to make it easy to insert expressions like today's date is ${ [<now>] }$.

The overall goal here is to make a better alternative to macros for textual substitution tasks, hence the desire to make it as powerful as is appropriate.

the limitation to only use the first result shouldn't be there. Same reasoning as in " [IDEA] :foreach filter run prefix" #6781

I don't see how the reasoning in #6781 applies here. The point there is that there is no workaround available to give the ability to return multiple results from the "filter" prefix. That doesn't apply here.

But it still may be reasonable to accept the entire list that the ${x}$ syntax returns. Presumably we'd just concatenate them.

IMO <$set name=myVar filter="[[a b]] my filter +[nth[2]]"> would do just fine.

Here we're trying to improve on that syntax; it's indirect and verbose.

It should be possible to even tell the macro, that it should treat the variable as a tiddlywiki list.

<$set name=myVar filter="[[a b]] my filter">
<$text text=<<jsstring "some new text $(myVar)$. " isList=yes>>/>
</$set>

I don't think that looks very good. Generally if we've got a choice between two apparently mutually exclusive choices, the worst option is to support both. That means that users have to grapple with the complexity of both. Good system design is about keeping things conceptually simple for users.

I think a macro is not the best approach for what you're trying to do because we can't use computed parameters within a filter.

The approach I was thinking was that the string literal syntax would be a shortcut for a filter operator alternative. For example:

<div title=`a string $(sub)$`>

would be a shortcut for:

<div title={{{ [substitute[a string $(sub)$]] }}}>

(Important to note that the shortcut syntax would be able to cope with double square brackets and other character combinations that would be illegal in the filter operator form).

In the filter operator form, it might be nice to also support $1, $2 for optional positional parameters:

<div title={{{ [substitute[a $1 string],<adverb>] }}}>

Mind you, there would be nothing to stop us supporting parameters with the shortcut syntax too:

<div title=`a $1 string`,<adverb>>

[saqimtiaz]

In the filter operator form, it might be nice to also support $1, $2 for optional positional parameters:

I would very much like to see this implemented. I do something similar with my custom printf filter operator which is a must in every single one of my wikis.

[AnthonyMuscio]

I understand this is to be part of TW 5.3.0?

• Will it be in the pre-release?

I understand the back tick format is also used for inline code highlighting.

• Will previous uses to highlight and stop wikification be backwardly compatible?

Use the `$(currentTiddler)$`variable in macros
<$let var=`$(currentTiddler)$`>
now <<var>>
</$let>

[Jermolene]

I understand this is to be part of TW 5.3.0?

No, this enhancement is not planned for v5.3.0. This ticket is part of showing the roadmap for how we will eventually be able to deprecate macros.

• Will previous uses to highlight and stop wikification be backwardly compatible?

No, the context is different. Here the backticks are used as special quotation symbols for attributes, not as freestanding syntax like code sections.

[pmario]

I wanted to solve exactly that problem with \function and \procedure using PR #6666 as follows. It would work fine if the link-widget would have the same functionality as the wikitext links eg:

[[local link]] .. [ext[external link]]

<$link to="local link" /> .. <$link ext="external link" />

Proposal

Add an ext parameter to the link-widget so it would look like this

\procedure URL(protocol:"https", host, path, tiddler)
<$link ext=<<f-URL>> />
\end

or even better

<$link ext=<<dynURL host:"tiddlywiki.com" tiddler:"HelloThere">> />

Example

A very common usecase is the dynamic creation of URLs, where our users always have problems and the naive implementation creates several technical problems

Code with Function Definition

\function dynURL(protocol:"https", host, path, tiddler) [<protocol>][[://]][<host>][<path>][[#]][<tiddler>] +[join[]]

Eg: create a dynamic link to the HelloThere tiddler at tiddlywiki.com

<<dynURL host:"tiddlywiki.com" tiddler:"HelloThere">> -> Creates a concatenated string

Rendered as: https://tiddlywiki.com#HelloThere -> Creates a concatenated string

.But it should be a link. We need to create external links using the A HTML tag, which is complicated and error-prone, because the HTML code needs additional parameters like "class", "rel" and "target"

\function f-URL() [<protocol>][[://]][<host>][<path>][[#]][<tiddler>] +[join[]]

\procedure URL(protocol:"https", host, path, tiddler)
<a class="tc-tiddlylink-external" href=<<f-URL>> rel="noopener noreferrer" target="_blank"><<f-URL>></a>
\end

[pmario]

Edit: I did add the following intro text to the first post

I wanted to solve exactly that problem with \function and \procedure using PR #6666 as follows. It would work fine if the link-widget would have the same functionality as the wikitext links eg:

[[local link]] .. [ext[external link]]

<$link to="local link" /> .. <$link ext="external link" />

[Jermolene]

Hi @pmario as things stand, the link widget is only used for tiddler links, and is not involved in rendering external links, even if they were made using the same double square bracket syntax as pretty links. This is useful; it means, for example, that it is easy to scan a parse tree looking for tiddler links. The relationship between tiddler links and external links is that tiddler links are layered over the top of the implementation of external links.

Adding the facility to make external links would be smooshing together functionality that is really quite different, making a frankenstein widget with two only tenuously related areas of functionality.

I think the red flag here is that it is an attempt to optimise for brevity at the widget layer. I think that that is almost always a bad idea because it inevitably leads to more complex semantics for the widgets that are the primitives in our system. The more complex that those semantics become the harder it is to reason about how the primitives combine together.

In this case, it sounds like your use case could be served with a custom <$$extlink> widget that hides the messy details of the class, rel and target attributes.

[pmario]

In this case, it sounds like your use case could be served with a custom <$$extlink> widget that hides the messy details of the class, rel and target attributes.

That's right. I'll test that

[pmario]

Just to be sure we need to be able to do stuff a shown in an even simpler way. https://talk.tiddlywiki.org/t/how-text-substitution-can-look-like-with-v5-3-0-prerelease/6607

\function sub() "[" [<operator>] ":" [<suffix>] "<term>]" +[join[]]

\procedure search-macro(f, operator, suffix, term)
  debug: <$text text=<<sub>>/>
  <$list filter="[subfilter<f>] +[subfilter<sub>] +[limit[1]]"/>
\end

<<search-macro "[all[tiddlers]!is[system]]" "regexp" "title" "\.[a-zA-Z]{2,4}$">>

----

<<search-macro "[!is[system]limit[250]]" "search" "text" "operator">>

---

slightly related: https://github.com/Jermolene/TiddlyWiki5/issues/6826

[AnthonyMuscio]

I think $(filter)$ isn't needed and the limitation to only use the first result shouldn't be there. Same reasoning as in " [IDEA] :foreach filter run prefix" #6781

From @pmario

With 5.3.0 pending, and this feature "Add support for string literal attributes with textual substitution #6663" possibly for the next release, I would like to restate the desire not to return only the first result

an example may be;

<div class=${ [subfilter] }$>

• Allowing a parameter to be fed a list

Although perhaps {{{ [subfilter<class-list>] }}} would work?

• but it too, is limited to one value when in use as a parameter.

I understand the motivation for the first value, but it seems an unnecessary handicap.

• Or perhaps an override? An uninformed guess

<div class=`${ =[subfilter<class-list>] }$`>

[saqimtiaz]

@Jermolene A couple of questions as I work on an implementation:

1. Can you think of a good name for the new style widget attributes denoted by backticks? The working name I have is substituted attributes but I think we might need something more intuitive.

2. Similarly, the operator equivalent has a working name of substitute[] and I wonder if we can do better.