Parameterised transclusions

[Jermolene]

Introduction

This PR introduces a number of improvements and new features related to some of TiddlyWiki's most fundamental components: macros, widgets, operators, transclusion and globals.

The motivation is to fix one of TiddlyWiki 5's early design flaws: the reliance on macros using textual substitution as the primary way to modularise and reuse wikitext and filters.

Experience has shown that while macros are a good match for a small number of tasks, they are brittle and error prone for many common operations. Over the years we have introduced mitigations for the worst problems but these have come at a cost of increased complexity.

The changes in this PR provide powerful new ways to achieve common tasks, and unlock completely new capabilities that were previously impossible in wikitext.

• Procedures, which are essentially what macros should have been; they work in exactly the same way except that parameters are exposed as simple variables (without the double underscores) and no textual substitution takes place
• Custom widgets, allowing the creation of widgets in wikitext, and the redefinition of built-in widgets
• Functions, a new way to encapsulate filter expressions with named parameters
• Custom Filter Operators, allowing functions to be used as custom filter operators
• Parameterised transclusions, allowing strings and wikitext trees to be passed to transclusions
• Global definitions that do not clutter up the variable namespace, and do not have to be imported before use

Some smaller improvements are also needed to enable these new features:

• ~~New filter operators for reading JSON data~~ – superseded by #6936
• A new <$genesis> widget that permits any other widget to be dynamically created, with dynamic attributes
• A new <$error> widget for displaying core error messages
• Conditional definitions that only occur if the target variable does not already have a value
• More effective protection against infinitely recursive functions and procedures

All of these changes are intended to be backwards compatible, and should not affect existing functionality. While they represent a new field of opportunities for wikitext authors, equally it is entirely possible for authors to ignore all these new features and continue to use TiddlyWiki 5 in the way that they have always done.

Background

TiddlyWiki 5 macros were originally based on the technique we call "textual substitution": the string values of the parameters provided when calling a macro would be plugged into the macro definition before it was wikified in the usual way.

A typical example of the approach in early versions of TiddlyWiki 5:

\define mymacro(title)
<$codeblock code={{$title$}}/>
\end

The technique worked well enough to get the basics of the TiddlyWiki 5 user interface up and running, but it was clear from the start that it was annoyingly brittle. For example, the macro above would fail with tiddler titles containing double closing curly braces. Trying to use it with the title foo}}bar would lead to the macro being expanded to the following invalid syntax:

<$codeblock code={{foo}}bar}}/>

As a result, for a long time, the TiddlyWiki 5 user interface failed if a variety of combinations of special characters were found in tiddler titles. Long time users will remember a warning that popped up in the edit template whenever a potentially troublesome character was detected.

Over the years we've mitigated almost all of these issues, particularly by providing access to the macro parameters as variables. For backwards compatibility, this was done without affecting the existing syntax, which required us to adopt the clumsy protocol of wrapping the parameter name in double underscores to get the name of the corresponding variable.

This has all worked well enough for us to fix the UI issues with special characters in tiddler titles, but is very inconsistent and complex, requiring users to grasp multiple mutually exclusive conceptual models for what is going on.

New Features and Improvements

The approach taken by this PR is to add new functionality by extending and augmenting the system without disturbing existing functionality.

This lays the groundwork for macros and related features to be deprecated, which is the point at which users are advised not to use old features, and instead given clear pointers to the equivalent modern functionality.

The new transclusion architecture is not by itself sufficient to enable us to fully deprecate macros yet. To handle the remaining use cases we propose a new backtick quoted attribute format that allows for the substitution of variable values. See https://github.com/Jermolene/TiddlyWiki5/issues/6663 for details.

Procedures

Procedures are the modern replacement for macros. The key difference is that the parameters are only available as wikitext variables (without requiring double underscores), and that textual substitution does not occur.

The syntax for defining a procedure closely resembles the existing syntax for defining a macro:

\procedure hello(when:"afternoon")
<$list filter="[<when>match[morning]]">
	Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
	Good afternoon or evening!
</$list>
\end

Note that unlike macros, the parenthesis can be omitted when there are no parameters:

\procedure name value

Procedures are invoked using the same shortcut syntax that is used for macros:

<<hello>>
<<hello "morning">>
<<hello when:"afternoon">>

Procedures can also be invoked as transcluded variable attributes using the syntax <div class=<<hello>>>. Note that the plain, unwikified text of the procedure will be used, without any parameter processing.

Custom Widgets

Custom widgets are defined with the \widget pragma, which works analogously to the \procedure pragma.

The names of user defined widgets must start with the prefix $$.

Built-in widgets can be overridden by using a single $. Note that it is not possible to create new widgets named with a single $, only to override existing built-in widgets. This is to prevent users creating widgets whose names clash with future core widgets.

Invoking a custom widget transcludes the contents of the variable containing the definition. The widget definition can transclude the content of the calling widget by using the construction <$slot $name="ts-raw"/>.

For example:

\widget $$hello(name)
<div class="greeting">
    <h1>Hello <$text text=<<name>>/></h1>
    <$slot $name="ts-raw"/>
</div>
\end

The custom widget is invoked in the usual way:

<$$hello name="Jeremy">
    This is the greeting
</$$hello>

That example would render as:

<div class="greeting">
    <h1>Hello Jeremy</h1>
    This is the greeting
</div>

Functions

Functions are analogous to procedures except that they encapsulate filter expressions rather than wikitext. They are defined in the same way:

\function add-tax(rate:"10")
[multiply<rate>]
\end

Functions can be invoked via the new function operator:

Total: <$text text={{{ [<total>function[add-tax],[12.5]] }}}/>

Functions can also be invoked as transcluded variable attributes. For example:

<div class=<<myfunction "foobar">>>
...
</div>

Custom Filter Operator Functions

If a user defined function name starts with a period then it can also be invoked directly as a custom filter operator:

\function .add-tax(rate:"1.10") [multiply<rate>]

...

Total: <$text text={{{ [<total>.add-tax[1.125]] }}}/>

Transclude Widget Updates

Many of these improvements depend for their implementation on a significantly upgraded <$transclude> widget. To accommodate the new functionality without breaking backwards compatibility, the widget now operates in two modes:

• Modern mode enables all the new functionality. It is engaged when at least one attribute name starts with a $ character such as $tiddler, $field, $index etc.
• Legacy mode is fully backwards compatible, using attribute names such as tiddler, field, index. It is engaged when none of the attributes start with a $ character

Like macros, procedures are implemented as a special type of variable. The syntax for invoking procedures using the <$transclude> widget is as follows:

<$transclude $variable="hello" when="morning"/>

Note how the attributes $variable, $tiddler, $field, $index etc. that start with a single dollar sign are used to define what to transclude, while the plain attributes are used as parameters to be passed to the procedure.

The transclude widget can be used to pass named parameters to the transclusion. The transcluded content can reference the parameters via the <$parameters> widget.

<$transclude $tiddler="MyTiddler" param1="HelloThere"/>

Parameterised Transclusion

The transclusion shortcut syntax has been extended to allow parameters to be passed to all types of transclusion, not just transclusions of macros/procedures/custom widgets.

The parameters are passed according to their position in the corresponding parameters declaration, not by name.

For example, with the text below in a tiddler titled "FooBar":

\parameters (a,b,c,d)
(definition of the transclusion goes here...)

The shortcut syntax for invoking the parameterised transclusion uses a single vertical bar to separate them from the title of the tiddler being transcluded:

{{FooBar|first|second|third|fourth}}
{{FooBar||template|first|second|third}}

Parameters can be omitted to skip them. For example:

{{FooBar|first||third|fourth}}

Note that omitting the first parameter {{FooBar||second|third}} creates an ambiguity because the resulting double vertical bar causes it to be interpreted as {{title||template|param1}}.

Slotted Parameters

"Slots" provide a way to pass complex structured data to a transclusion instead of the simple string values supported by string parameters.

Slots are named areas that are defined within transcluded text using the <$slot> widget. They are replaced with custom content provided by a corresponding <$fill> widget inside the transclusion invocation. For example:

\procedure hello
<div class="frame">
    <$slot $name="greeting"/>
</div>
\end

The syntax for invoking the procedure using the <$transclude> widget uses the <$fill> widget to pass the content for the slot:

<$transclude $variable="hello">
    <$fill $name="greeting">
        <h1>A heading</h1>
        <p>A paragraph</p>
    </$fill>
</$transclude >

The result would be equivalent to:

<div class="frame">
    <h1>A heading</h1>
    <p>A paragraph</p>
</div>

The contents of the <$slot> widget provide the default contents that are used in case a value is not provided for a named slot. For example:

\procedure hello
<div class="frame">
    <$slot $name="greeting">
        Default text for the named slot "greeting"
    </$slot>
</div>
\end

The entire contents of the transclude widget invocation is available as the special slot fill value ts-raw.

Specifying Content to Display When Target is Missing

Up until now, the content of the transclude widget has been used as the fallback content to display when the target of the transclusion is missing.

To allow room for the new <$fill> widget, the new behaviour is to use the special slot value ts-missing if present, and if there are no <$fill> widgets present, then falling back to the content of the transclude widget.

Global Definitions

Definitions of procedures, widgets, functions and macros are made available globally in TidddlyWiki 5 by tagging them with $:/tags/Macro. However, this mechanism suffers from inherently poor performance because every $:/tags/Macro tiddler has to be parsed and loaded at the top of the widget tree, regardless of whether the definitions within it are actually used.

TiddlyWiki now offers an alternative way to define global rocedures, widgets, functions and macros: placing them in tiddlers titled with the name of the global prefixed with $:/global/.

For example, the global variable foo would be defined in a tiddler called $:/global/foo. Accessing the variable <<foo>> then acts as a shortcut for accessing the underlying global variable tiddler.

The following special fields are used to define the behaviour of the global:

• _parameters defines the parameters expected by procedures, widgets and functions
• _is_procedure, _is_widget, _is_function, _is_macro are set to yes to indicate the type of the definition

Note that global definitions can include local variables that are defined before the body of the global. These local variables will not be visible externally.

For example:

title: $:/globals/foo
_is_procedure: yes
_parameters: (param1:"value",param2:"value")

\procedure renderTitle(title)
<div class="mytitle"><$text text=<<title>>/></div>
\end

\function myfn(a)
[[a]getvariable[]addprefix[!]]
\end

<$list filter=<<param1>>>

<<renderTitle "first">>: <$text text=<<param2>>/>

<<renderTitle "second">>: <$text text=<<myfn param2>>/>

</$list>

It is possible to allow the caller to override these local definitions by using the new syntax for conditional definitions. For example, here we only define the function myfn if the variable myfn is not already defined:

\?function myfn(a)
[[a]getvariable[]addprefix[!]]
\end

Open Questions

These need to be settled before merging:

• Do custom widgets really need their own type of definition – they could just be implemented as procedures
• Review naming of "procedures" and "functions"
• Consider adding support for custom filter run prefixes. While it could be added later there's a chance that the design might influence the design of custom filter operators
• Is $$ the best prefix for custom widgets? We have a (very) loose guideline that $ stands for "system", and it is used as a kind of warning sign for system-y stuff that generally doesn't need to bother casual users. But we don't have an equivalent prefix for a user defined namespace. Perhaps ., as we're using with custom operators?
• Is <$fill> a good name? The rationale is that it pairs logically with <$slot> (which in turn reflects the W3C <slot> element). The trouble is that end users won't see the <$slot> widget until they start writing their own relatively complex procedures/widgets, and until that point <$fill> won't make much sense
• Consider making transcluding tiddler text fields understand fields like _parameters, _is_procedure etc

Future Possibilities

These can be addressed after merging:

• Now that the entire parse tree of the contents of the transclude widget is passed to the transclusion as a JSON blob, it might be useful to be able to render a parse tree. In particular, a new flag for the transclude widget that causes it to interpret the target as a JSON parse tree
• Consider adding custom commands, analogous to procedures but for action strings. We may be able to retcon the existing commands into wikitext implementations too
• Refactor core icons to parameterise the width and height (the calendar icon can also parameterise the date that is shown)

Progress

This PR is fully functional, and very, very nearly complete.

Incomplete:

• [x] Disable widget overrides in safe mode
• [x] Function operator should return the input list if the function is missing
• [ ] Review performance impact of these changes
• [ ] Documentation (see below)
• [ ] Use the existing tiddler cache mechanism to cache the processing for globals eg macro params stuff
• [ ] Refactor isFunctionDefinition/isProcedureDefinition/isWidgetDefinition flags into a single field
• [ ] Make transcluding tiddler text fields understand fields like _parameters, _is_procedure etc
• [ ] Review whether other operators that have a fake widget.getVariable() handler (eg $:/core/modules/filters/filter.js) should also implement widget.getVariableInfo()
• [ ] Check that transclude widget refresh optimisations still work
• [ ] Review and rename test cases
• [ ] Extend set widget to set the hidden parse tree properties like isprocedure, and to be able to specify parameters
• [ ] Finish (or defer) visible transclusion demo

Completed

• [x] Recursion detection for functions
• [x] ubertransclude widget
• [x] parameters widget
• [x] slot widget
• [x] values widget
• [x] genesis widget
• [x] widget -> ubertransclusion mapping
• [x] use ubertransclude widget for wikitext transclude syntax
• [x] improved framework for wiki-based rendering tests
• [x] extend wikitext transclusion syntax with positional parameters
• [x] wikitext parser rule for new-style \function definitions
• [x] wikitext parser rule for \parameters construction
• [x] importvariables should skip the parameters widget
• [x] parse trees for transcluded variables should be cached
• [x] custom filter operators
• [x] genesis widget shouldn't evaluate attributes of child widget, leave it to the child widget
• [x] improve efficient of recursion detection (currently the $transclusion variable gets excessively long)
• [x] Support for $:/tags/Global alongside $:/tags/Macro

Documentation progress

• [ ] Procedure Definitions
• [ ] Function Definitions
• [ ] Widget Definitions
• [x] Macro Definitions retcon
• [ ] Transclusion in WikiText update to add parameters
• [x] Globals
• [x] SlotWidget
• [x] ParametersWidget
• [x] MacroCallWidget updates – to note that the transclude widget is preferred now
• [x] ImportVariablesWidget updates to skip parameters widgets
• [x] FillWidget
• [x] ErrorWidget
• [x] SetWidget updates to add conditional attribute
• [x] Unknown Operator
• [x] Function Operator
• [x] TranscludeWidget updates
• [x] Format:Json Operator
• [x] JSON Operators Docs
• [x] Visible Transclusion How To
• [x] LetWidget update noting that dollars are now allowed
• [x] GenesisWidget

Notes for documentation

• [ ] What we've previously referred to as "JavaScript Macros" are actually now better thought of as "JavaScript functions", because they work more like functions than macros
• [x] Global procedures and widgets should use an intrinsic \parameters declaration, and not use the _parameters field. This ensures that the parameter variables are available when the tiddler is viewed directly
• [x] Note the difference between the subfilter and function operators (the former interprets the operand as the filter while the latter interprets the operand as the name of the variable containing the filter)
• [ ] Note that first parameter of user defined filter operator functions cannot be missing, because the empty double square brackets are required by the operator syntax (in other words, the operator syntax doesn't permit operators to be called with zero operands, just one or more)
• [ ] Note that macros, procedures, widgets and functions are all variables, and all share the same namespace. Definitions with the same name will overwrite one another
• [x] Fill widget $name attribute must be specified as a literal string, not a transcluded attribute
• [ ] Note that with the new architecture, updating tiddlers containing a global definition does not trigger a full page refresh

Other tickets that can be closed when this is merged

• #3750

[joshuafontany]

Really interesting stuff, and I can see the appeal of simplifying the wikitext and shortcut syntaxes.

[tw-FRed]

@Jermolene, just for clarity and if I understand the doc correctly, the render of the Widget Syntax Invocation example quoted below misses a line and should read:

<div class="greeting">
		<h1>Hello Jeremy</h1>
    This is the greeting
</div>

Fred

Invoking Macros via Widget Syntax

Macros whose names conform to a special format can also be invoked via the standard widget syntax. When invoking a widget called foo the core looks for a variable called <$foo> and if it finds it, transcludes the contents of the variable instead of invoking the widget in the usual way. It also passes the content of the widget as the named slot value body.

For example:

\macro <$hello>(name)
<div class="greeting">
		<h1>Hello <$text text=<<name>>/></h1>
    <$slot $name="body"/>
</div>
\end

The custom widget is invoked in the usual way:

<$hello name="Jeremy">
    This is the greeting
</$hello>

Would render as:

<div class="greeting">
    This is the greeting
</div>

[AnthonyMuscio]

@Jermolene I applaud the objectives of this piece of work and will do what I can to help, text and support this initiative. A couple of points about what I see as critical to keeping this understandable to new and average users.

• As it stands the explanation and names used are possibly too complex to average users to easily adopt. This is understandable at this stage of development. I would suggest allowing time and feedback from the community to help craft appropriate terms. eg other than slot.
• Perhaps the use of the $ variables in the transclude widget could invoke the new behaviour rather than introduce uberwidget (or by another name). Perhaps a parameter such as $content=varormacroname is what results in the use of the other form?
  • <$transclude $tiddler= /> vs <$transclude $content= />
  • In other discussions of the standard transclude, we felt an alias for the "tiddler parameter" of a "template parameter" would allow code to look more readable. <$transclude $template=tiddlername/> because currentTiddler is maintained.

As you suggest I think the Add support for string literal attributes with textual substitution #6663 is an essential addition. I mention fields there.

[Jermolene]

@Jermolene, just for clarity and if I understand the doc correctly, the render of the Widget Syntax Invocation example quoted below misses a line and should read:

Thanks @tw-FRed, fixed.

• As it stands the explanation and names used are possibly too complex to average users to easily adopt. This is understandable at this stage of development. I would suggest allowing time and feedback from the community to help craft appropriate terms.

Yes @AnthonyMuscio, I think we do need to review all the associated terminology. I've had to invent some new terms and some of the old terms might no longer be such a good fit. For example, I'm not sure that "macros" makes sense now; we might consider "function".

eg other than slot.

The term "slot" is taken from the web custom elements specification:

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/slot

Generally, the implementation of custom widgets is very similar to the custom elements spec, so it seems to make sense to share terminology.

• Perhaps the use of the $ variables in the transclude widget could invoke the new behaviour rather than introduce uberwidget (or by another name). Perhaps a parameter such as $content=varormacroname is what results in the use of the other form?

I think you're suggesting that the transclude widget use the presence of a $ attribute to trigger the new behaviour. That would mean that <$transclude $tiddler="title"/> would trigger the new behaviour (and allow parameters), while <$transclude tiddler="title"/> would still work, but not support passing parameters.

I don't think that will work; the $tiddler attribute is optional, so how would one know whether <$transclude tiddler="title"/> was intending to transclude a tiddler called "title", or to transclude the current tiddler passing a parameter called "tiddler".

• <$transclude $tiddler= /> vs <$transclude $content= />
• In other discussions of the standard transclude, we felt an alias for the "tiddler parameter" of a "template parameter" would allow code to look more readable. <$transclude $template=tiddlername/> because currentTiddler is maintained.

That won't work. A key change here is that we have a single widget that can transclude variables or tiddlers. So we need to have the explicit attribute names so that we know which is which. It would be ambiguous whether <$transclude $content="foobar"/> was transcluding a tiddler or a variable.

[xcazin]

Hi @Jermolene, this feature looks very powerful indeed! FWIW, a name that came to mind was <$paraclude>.

In the spirit of grand unification, I was wondering whether we couldn't do without the $variable, $tiddler, $field and $index attributes, in favour of a single $content attribute that would expect a wikitext value instead of a content source name. For instance:

• <$paraclude $content=<<hello when:"morning">> /> would transclude the content of the macro hello with a morning value for the param when;
• <$paraclude $content=<<hello>> when="morning" /> would transclude the content of the macro/variable hello, passing the morning value to any when <$parameter> widget present in the hello content.
• <$paraclude $content={{hello}} when="morning"/> would transclude the content of the text field of the hello tiddler, passing the morning value to any <$parameter> widget present in the hello tiddler
• <$paraclude $content={{{ [[hello]get[when]] }}} /> would transclude the content of the field when in the hello tiddler
• <$paraclude field="morning"/> would use current tiddler's text field as the default transcluded content, and pass it the morning value through any field attribute present in the <$parameter> widget.
• <$paraclude tiddler="hello"/> would use current tiddler's text field as the default transcluded content, and pass it the hello value through any tiddler attribute present in the <$parameter> widget.

[pmario]

I think you're suggesting that the transclude widget use the presence of a $ attribute to trigger the new behaviour.

I don't think, the idea is too far off. ...

I don't think that will work; the $tiddler attribute is optional,

That's right <$transclude /> will do the same thing as it does now. <$transclude tiddler=x /> <$transclude $tiddler=x /> are the the same thing. IMO it doesn't matter if it triggers new or old behaviour, since there are no parameters.

... so how would one know whether <$transclude tiddler="title"/> was intending to transclude a tiddler called "title", or to transclude the current tiddler passing a parameter called "tiddler".

Since there is no $ it's the old behaviour. ... done.

There could be a possibility for an empty $ parameter, that triggers the new behaviour eg: <$transclude tiddler=x $/> or <$transclude $="" tiddler=x /> for a more verbose writing. .. I would prefer the first version

Just some thoughts.

[Jermolene]

I've pushed some significant changes, and made substantial edits to the original post at the top of this PR.

The changes are:

• New wikitext shortcut syntax for passing parameters to transclusions (eg {{tiddler|param|another}}) – the syntax is borrowed from Wikipedia
• Merging the ubertransclude widget into the transclude widget (using the technique suggested by @AnthonyMuscio and @pmario above)

In the spirit of grand unification, I was wondering whether we couldn't do without the $variable, $tiddler, $field and $index attributes, in favour of a single $content attribute that would expect a wikitext value instead of a content source name. For instance:

Thanks @xcazin I have been considering that possibility. The advantage of the present arrangement is that because the widget is working with tiddler titles (or variable names) it can easily cache the results of parsing. With a generic $content attribute the widget would only see a plain string, without knowing whether it came from a variable or a tiddler. We might be able to reconstruct that information by looking at the parse tree, but it might get difficult for complex cases.

[pmario]

I'll check out the code and have a closer look soon. Looks very interesting.

[AnthonyMuscio]

I am in the bush so cant review with a fine tooth comb. So friendly feed back.

Do you actualy intend to drop the use of parameter:"value" and use "=" in both function definition and calls?

If so this is added complexity going forward so could we do one of the following?

• also allow = to be an alias for : in macro definitions and calls
• or instead use : in the new function definitions and calls not =

Can we use $macrocall for functions and if we do what happens?

Because I think it relates to the handling in this functions;

Personaly I have long felt it would help if we made available in wikitext to be able to convert between keyword="value" pairs and variables with values and the reverse, generate a set of keyword value pairs and handling the delimiters ' " """.

An example may be programatically building a set of keyword value pairs and passing them to actioncreatetiddler or taking a set of variables and save them as settings in a tiddler field or text.

Another example would be uri's do not permit spaces so a link field could contain a link plus more info eg https://tiddlywiki.com pretty='tiddlywiki' target="tw" Then the link field value can be used to get the url split[ ]first[] then use the rest[] to get the keyword value pairs and convert them to variables to use in the link.

Please note I am not trying to abuse by going off topic. Functions as proposed will be at least translating keyword/value pairs to variables if not in the reverse. This seems to me importiant to concider now, if not ultimately spawning another issue.

[btheado]

@AnthonyMuscio, I know the answer to this one:

Can we use $macrocall for functions and if we do what happens?

Using $macrocall for functions is like calling it without passing any arguments since the parameter mechanism is completely different for functions vs. macros.

The first 3 macro calls below all return "hi" only. The transclude returns "hi a b c":

\function myfunc(a, b, c)
hi <<a>> <<b>> <<c>>
\end

<<myfunc>>

<<myfunc a b c>>

<$macrocall $name=myfunc a=a b=b c=c/>

<$transclude $variable=myfunc a=a b=b c=c/>

Edit to add: while this is the behavior of the current code, the "Backwards Compatibility" section makes it sound like this is an open question still.

[Jermolene]

Do you actualy intend to drop the use of parameter:"value" and use "=" in both function definition and calls?

My mistake. I've amended the examples above to use parameter:value for \function definitions. There is no proposed shortcut syntax for calling functions at the moment, just the <$transclude> widget, which will continue to use attribute=value, like all widgets.

• also allow = to be an alias for : in macro definitions and calls

I'd rather that we decided on one or the other. For the moment I've gone for backwards compatibility with the \function pragma, but it really would be better to focus on internal consistency.

• or instead use : in the new function definitions and calls not =

That's the option I prefer. It distinguishes between an attribute assignment and a default declaration.

Can we use $macrocall for functions and if we do what happens?

Yes. The macrocall widget will render the function as usual, but will ignore the parameters.

Because I think it relates to the handling in this functions;

I'm not sure what you mean here?

Personaly I have long felt it would help if we made available in wikitext to be able to convert between keyword="value" pairs and variables with values and the reverse, generate a set of keyword value pairs and handling the delimiters ' " """.

We have quite a few capabilities for dealing with lists of name/value pairs: the action-setmultiplefields widget, the action-sendmessage widget, the setmultiplevariables widget. They all work with separate, coordinates lists of names and values that match up.

I think you're asking for primitives to parse name/value pairs? Given the brittleness of quoting systems, I would expect JSON to be more robust, but perhaps it would be helpful to understand more about the use cases you have in mind.

An example may be programatically building a set of keyword value pairs and passing them to actioncreatetiddler or taking a set of variables and save them as settings in a tiddler field or text.

I think those are all covered by the widgets I mentioned above.

Another example would be uri's do not permit spaces so a link field could contain a link plus more info eg https://tiddlywiki.com pretty='tiddlywiki' target="tw" Then the link field value can be used to get the url split[ ]first[] then use the rest[] to get the keyword value pairs and convert them to variables to use in the link.

The web already has a syntax for passing name/value pairs as part of a URI: https://tiddlywiki.com?pretty=tiddlywiki&target=tw – note that it doesn't use quotes for values, but rather requires the characters within them to be escaped so that the terminating & can be unambiguously identified. These strings can be read via $:/info/url/search

Please note I am not trying to abuse by going off topic. Functions as proposed will be at least translating keyword/value pairs to variables if not in the reverse. This seems to me importiant to concider now, if not ultimately spawning another issue.

As I say, I am pretty sure that we've been thinking on similar lines, and we already have most or all of what you need.

Edit to add: while this is the behavior of the current code, the "Backwards Compatibility" section makes it sound like this is an open question still.

As things have gone, I'm increasingly confident that we can introduce the new features without disturbing backwards compatibility, but that does assume that we can find an alternative shortcut syntax to <<name param>> for function calls.

[pmario]

The first thing I tested based on:

• The ability to invoke functions using widget syntax (in other words, the ability to define custom widgets in wikitext)

and

\function hello(when:"afternoon")
<$list filter="[<when>match[morning]]">
	Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
	Good afternoon or evening!
</$list>
\end

leads to my assumption that I can call it like this:

<$hello />

Which results in: Undefined widget 'hello' ...

Is it intended to work that way, or am I completely wrong?

[Jermolene]

Hi @pmario only functions whose names conform to the pattern <$widgetname> can be invoked using the widget syntax. So you'd need to do something like this:

\function <$hello>(when:"afternoon")
<$list filter="[<when>match[morning]]">
	Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
	Good afternoon or evening!
</$list>
\end

<$hello/>
<$hello when="yesterday"/>

[Jermolene]

@pmario that is documented in the section "Invoking Functions via Widget Syntax" in the OP.

[pmario]

@pmario that is documented in the section "Invoking Functions via Widget Syntax" in the OP.

Sorry. Will read the whole docs first before I do more testing. ....

[pmario]

The following transclusion with a typo creates a RSOD

<$transclude $variable="helloX" when="morning"/>

[Jermolene]

The following transclusion with a typo creates a RSOD

Thanks @pmario, fixed in a10106a4a6fae8f5ccaf419b73f23778eb41e71a

[pmario]

@Jermolene .. I think this one https://github.com/Jermolene/TiddlyWiki5/pull/6666#issuecomment-1114240153 needs to be addressed. ... I didn't see much discussion about that yet.

From a "naive" users point of view I'd expect: <<hello when=morning>> to be valid since <<hello>> also does something.

[pmario]

At the moment a function call looks like this using $variable

<$transclude $variable="hello" when="morning"/>

I think it should be $function

<$transclude $function="hello" when="morning"/>

Reasoning:

If it is $variable the below code needs to work too, because macros are technically variables.

\define asdf(when:test) x: <<__when__>>

\function hello(when:"afternoon")
<$list filter="[<when>match[morning]]">
	Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
	Good afternoon or evening!
</$list>
\end

<$transclude $variable="hello" when="morning"/>

<$transclude $variable="asdf" when="morning"/>

[Jermolene]

@Jermolene .. I think this one #6666 (comment) needs to be addressed. ... I didn't see much discussion about that yet.

From a "naive" users point of view I'd expect: <<hello when=morning>> to be valid since <<hello>> also does something.

I don't quite understand. The comment you reference is noting that macro calls can't be used to invoke functions because they are different things.

Perhaps what you mean is that as mentioned in the OP under "Backwards Compatibility", there is as yet no wikitext shortcut syntax for transcluding a variable?

I think it should be $function

The terminology here is tricky. The rationale for the transclude widget using the "$variable" is (a) consistency with eg the "variable" attribute in the list widget and (b) because the target of the transclusion is not necessarily a function; it could just be a plain variable.

This hinges on there being a distinction between a function and a variable. The position I've adopted so far is that a function is a variable with the extra machinery for passing parameters. In other words, "functions" are a subset of "variables".

[AnthonyMuscio]

I have returned from some time in the bush and a hot spot that had a broken usb connector so no Internet.

I returned to review, thanks for your answers @Jermolene

I need to return to the top of the thread because it is hard to follow, this a also something we need to consider for future users, will everyday users be able to use it or is it a little too hard to follow?

on key-value pairs, Yes, I would like easy to use Primitives to parse name/value pairs! BECAUSE OF the brittleness of quoting systems, perhaps I miss understand but I am not looking to URI encode to allow this because it is not plain text and we do not type in encoded forms.

To keep things simple I raise a separate issue [IDEA] native support for handling key=value and key:value pairs in tiddlywiki #6674

However if such a facility was available it would support the possibilities raised here "Parameterised transclusions" and perhaps we could even pass a string of key value pairs into a function?

[Jermolene]

I need to return to the top of the thread because it is hard to follow, this a also something we need to consider for future users, will everyday users be able to use it or is it a little too hard to follow?

There's a lot going on in this PR, but if you read the OP carefully I think you'll find it easy to follow; from the perspective of an experienced end user there are a small number of significant changes: parameterised transclusions and widget overrides.

on key-value pairs, Yes, I would like easy to use Primitives to parse name/value pairs! BECAUSE OF the brittleness of quoting systems, perhaps I miss understand but I am not looking to URI encode to allow this because it is not plain text and we do not type in encoded forms.

To keep things simple I raise a separate issue [IDEA] native support for handling key=value and key:value pairs in tiddlywiki #6674

I'll respond to the points about key/value pairs there.

However if such a facility was available it would support the possibilities raised here "Parameterised transclusions" and perhaps we could even pass a string of key value pairs into a function?

We can already do that. The way to pass a string of key value pairs into a function is as a pair of lists or filters, one being the names and one being the values. That technique already pervades the core and works well.

[saqimtiaz]

For reference, the $genesis widget addresses both use cases discussed in https://github.com/Jermolene/TiddlyWiki5/discussions/6491.

[Jermolene]

I've added a lot of new functionality since the last update of the OP which I shall document soon.

If you're following, though, you might be interested to have a look at the latest improvement. It is now possible to create user defined filter operators that take named parameters. Here's an example of how an operator is defined:

\function [multiplybysomething[]](factor:2)
[multiply[2]multiply<factor>]
\end

And then it can be called just like a built-in operator:

<$text text={{{ [[123]multiplybysomething[3]] }}}/>

There's more work to be done on this, but I'm confident that this is one of the last missing pieces of our filter language. Breaking down filters into custom operators allows complex operations to be structured into readable chunks, and brings the filter language up to being a proper functional language.

[Jermolene]

By the way I intend to make a post about all of this work to talk.tiddlywiki.org as soon as things are a little more stable.

[Jermolene]

Another interesting new feature is a component that overrides the <$transclude> widget itself to make transclusions visible. It renders block transclusions as red blocks, and inline transclusions as green spans:

There's a button in the docs tiddler Visible Transclusions that enables visible transclusions.

[btheado]

Another interesting new feature is a component that overrides the <$transclude> widget itself to make transclusions visible. It renders block transclusions as red blocks, and inline transclusions as green spans

Wow, this is fantastic! My first thought when I saw the "Invoking functions via widget syntax" feature was how to override transclude to make some sort of "x-ray" functionality.

Just this morning I was idly wondering how to implement it and now you've already done it.

Really jawdropping! Great work.

[btheado]

Another way to use the visible transclude is to put it in a tiddler tagged with $:/tags/EditPreview

<$importvariables filter="$:/core/ui/VisibleTransclude">

<$transclude/>
</$importvariables>

Then it can be used on-demand as a preview mode for any tiddler.

And I think for the "tiddler" parameter, I think it would be better to render it as a link so it is easy to navigate to all the transcluded tiddlers. I might try that and see if it works out.

[btheado]

It would really useful to make macro calls visible in the same way you did for transclusions. I took a stab at it, but it seems macrocalls using wiki syntax do not populate the widget tree with the attributes. IOW, <<now format:YYYY-0MM-0DD>> does not result in the same widget tree as <$macrocall $name=now format=YYYY-0MM-0DD/>. The latter has an attributes field, but the former does not.

As a result, when I override the macrocall widget, I'm unable to access the input parameters and therefore cannot perform the original macrocall.

[Jermolene]

Hi @btheado

It would really useful to make macro calls visible in the same way you did for transclusions. I took a stab at it, but it seems macrocalls using wiki syntax do not populate the widget tree with the attributes

Ouch! I had forgotten about that unsightly scar tissue from the past. See 55d479c540f5e0a05e6c59ef7eb14606b89857c3 for some background.

It's tricky to fix it; the macrocall widget uses the params array to manage positional parameters and so it isn't straightforward to map those broken parsetrees into ordinary widget parse tree nodes. So I'd be inclined to leave fixing that for a separate PR – and indeed would welcome help on that if anyone is able.