HypertextLiteral.jl A way to avoid interpolating escaped $s (e.g `htl" \$NOinterp "`) to match Julia interpolation

Julia does not interpolate escaped $ in strings, so @htl(" \$NOinterp ") and htl" \$NOinterp " have different behavior. Replacing findnext('$' with a regexp for the next $ that is not preceded by an odd number of \, I think brings htl" " closer to the expected Julia string interpolation behavior.

Existing behavior

Note the extra interpolations for the 3rd and 4th.

julia> using HypertextLiteral
julia> NOinterp = "YES_I'm interpolated! ";

julia> @htl("$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \\NOinterp")
YES_I&apos;m interpolated!  YES_I&apos;m interpolated!  $(NOinterp) $NOinterp \YES_I&apos;m interpolated!  \NOinterp

julia> htl"$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \\NOinterp"
YES_I&apos;m interpolated!  YES_I&apos;m interpolated!  \YES_I&apos;m interpolated!  \YES_I&apos;m interpolated!  \\YES_I&apos;m interpolated!  \\NOinterp

New behavior

Note that the escaped dollar signs still show the raw \$ -- this is the expected difference to the cooked string from @htl.

julia> htl"$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \NOinterp"
YES_I&apos;m interpolated!  YES_I&apos;m interpolated!  \$(NOinterp) \$NOinterp \\YES_I&apos;m interpolated!  \NOinterp

Jun 23 '22 02:06 snoeyink

Jack,

Thanks for this suggestion. I struggled a lot with this issue and opted to not implement what you have suggested at that time:

Julia a string literals already have their own escape peculiarities and I wasn't sure if I could implement something that would deal with all of the edge cases.
The lack of '' as an escape character may be advantageous, for example, usage within a path.
Escaping '$' but not other characters seems a bit strange to me. What should '\$' mean, for example. In early releases I escaped everything. Alas, this hit a few unsettling edge cases with non-standard string literal's escaping.
In the end, I think I settled with "SQL" style escaping, e.g. double up the '$' to escape. This seemed the least-worst option.

I'm not opposed to what you suggest in lieu of doubling up '$'. I think edge cases may need some work (have not looked at PR code yet). Importantly, what exactly is the use case where this particular compromise would be advantageous?

Cheers,

Clark

On Wed, Jun 22, 2022, at 9:06 PM, Jack Snoeyink wrote:

Julia does not interpolate escaped $ in strings, so @.***(" $NOinterp ")andhtl" $NOinterp "have different behavior. By replacing findnext('$' with a regexp for the next $ that is not preceded by an odd number of , I think bringshtl" "` get closer to the expected Julia string interpolation behavior.

Existing behavior

Note the extra interpolations for the 3rd and 4th.

julia> using HypertextLiteral julia> NOinterp = "YES_I'm interpolated! ";

julia> @htl("$(NOinterp) $NOinterp $(NOinterp) $NOinterp \$(NOinterp) \NOinterp") YES_I'm interpolated! YES_I'm interpolated! $(NOinterp) $NOinterp \YES_I'm interpolated! \NOinterp

julia> htl"$(NOinterp) $NOinterp $(NOinterp) $NOinterp \$(NOinterp) \NOinterp" YES_I'm interpolated! YES_I'm interpolated! \YES_I'm interpolated! \YES_I'm interpolated! \YES_I'm interpolated! \NOinterp New behavior

Note that the escaped dollar signs still show the raw $ -- this is the expected difference from the cooked string from @htl https://github.com/htl.

julia> htl"$(NOinterp) $NOinterp $(NOinterp) $NOinterp \$(NOinterp) \NOinterp" YES_I'm interpolated! YES_I'm interpolated! $(NOinterp) $NOinterp \YES_I'm interpolated! \NOinterp

You can view, comment on, or merge this pull request online at:

https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30

Commit Summary

0a00742 https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/commits/0a00742d2b026366dbe664fa42eb224930544017 debugging test fucn fh for escaping $ for htl_str.

7621db1 https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/commits/7621db1c66b3106854cfb897de1b3549d290f646 Interpolate only unescaped $ by findnext regexp.

b81e47c https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/commits/b81e47c2b76414755a626d6eb6d6c50c955d6658 htl_str interpolate only unesc $ : rm debug code File Changes

(1 file https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/files)

M src/macro.jl https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/files#diff-5e84de38fb08a2615207f16bf896e93ccb8c58bbfa50e6420fc8d54d88daca84 (8) Patch Links:

https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30.patch

https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30.diff

— Reply to this email directly, view it on GitHub https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQ7IUWMVVF65ENAEMSKDLVQPBEBANCNFSM5ZSTHIZQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

Jun 24 '22 17:06 clarkevans

Summary

The scenario is MarkdownLiteral.jl:

CommonMark cm" " allows interpolation into markdown, but not code, HTML, or latex.
@fonsp's @markdown (alias @mdx) macro takes a Julia string and applies @htl then a CommonMark parser to make markdown with interpolated HTML.
I wanted to take raw strings through this pipeline, since the Julia string escape processing gives errors for backslashes used in \LaTeX.
My PR MarkdownLiteral#6 aims to allow mdx" " to replace cm" " by using htl" " to work with raw strings.
The one problem is that htl" " interpolates all single $, even if escaped as \$, before passing to CommonMark. My PR suggests to not interpolate $ preceded by an odd number of \, but to pass both characters unchanged. (Note: this is not escaping \, but just deciding to skip some $. You can independently decide whether the $$ escape remains beneficial.)

Responses to your points

(Julia escape peculiarities:) Fully on board with not trying to implement all escapes for htl" ", since that is supposed to be a raw string.
(Good not to have \ as escape:) Agreed, and that is the whole point, because I want it passed through for \LaTeX.
My PR actually keeps the string raw -- doesn't use \ as an escape character, but simply decides not to do htl" " $ interpolation when the $ is preceded by an odd number of \.
(Escaping $ strange:) I don't think of it as escaping (since all \ are passed unchanged), but as tamping down the aggressiveness which $s htl" "will interpolate to be compatible with Julia strings, CommonMark cm" " and Markdown md" ". I want to be able to switch CommonMark cm" " to MarkdownLiteral mdx" " without having htl interpolating $ that are preceded by an odd number of \ .
(SQL-style $$ escape:) I confess that I didn't realize that htl" " uses $$ to escape $ because $$ throws an error in Julia, CommonMark, and Markdown strings. All my experiments were based on converting valid cm" " to mdx" ". You may want to keep $$ as an escape producing a single $, since for me \$ passes both characters (which is different from CommonMark, but works great if the htl" " is followed by CommonMark parsing, as in my MarkdownLiteral PR.)

Details

I've used $\TeX/\LaTeX$ since 1985, but Julia only recently, so apologies if I am coming from a different context on processing of $ and \. Markdown handles latex poorly; CommonMark does considerably better (double backtick for inline latex), but doesn't allow interpolation into code, HTML, or latex environments. I'm impressed with how well @htl does, and liked @fonsp's MarkdownLiteral combining CommonMark and HypertextLiteral to allow interpolation everywhere.

I'm converting slides & textbook for my Discrete Mathematic course to Julia CommonMark, and want to be able to switch cm" slide " to mdx" slide " whenever there are cool HTML goodies I want to add to make the slides more dynamic. Since I want to use htl" " in defining mdx" ", I ask you to reconsider which $ trigger interpolation in htl" ".

Note first that Julia strings, cm" "and md" " all choose not to interpolate cm"\$", and produce parse errors for cm"$$"
As written htl"$$" escapes $, and all other $ in a string are interpolated, including htl"\$".
My suggestion is for htl" " to not interpolate any $ immediately preceded by an odd number of \, but to pass all characters along. I think that this behavior is easy to describe, and less surprising that finding that htl"\$NOinterp" actually does interpolate.
The behavior of passing the raw characters is different from Julia strings, cm" "and md" ", which consume one \ in escaping the $. Passing all characters allows any escape processing to happen downstream, while keeps the string raw at this stage.
This suggestion is orthogonal to the treatment of $$ as an escaped $ in htl" " (Personally, I would take that escape out so htl"$$" would give the same error as cm"$$", but then my goal is for mdx" " to be a drop-in replacement for cm" ".)
An alternative is to consume one \ and escape the $, but I haven't tested if that plays well in MarkdownLiteral.

So, your options:

Leave htl" " as is: I'll have to suggest a different implementation for @mdx_str in MarkdownLiteral.
Accept suggestion to not interpolate $ preceded by an odd number of \:
1. Leave the option of $$ escaping $: current PR.
2. Remove all escapes by deleting lines macro.jl:71-75 (if length(expr) >= start && expr[s ... end)
Modify the suggestion to make \$ escape a dollar sign. (I can check the interaction with MarkdownLiteral, if you want to go this route.)

Examples

using CommonMark, HypertextLiteral, MarkdownLiteral
NOinterp = "INTERP"
"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> "INTERP \$NOinterp \\INTERP \$INTERP"
"$$NOinterp" #-> syntax: invalid interpolation syntax: "$$"
   	cm"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> INTERP $NOinterp \INTERP $INTERP
cm"$$NOinterp" #-> UndefVarError: $ not defined
md"$NOinterp" #-> INTERP
md"\$NOinterp" #-> $NOinterp
md"\\$NOinterp" #-> \\INTERP
md"\$$NOinterp" #-> \$INTERP
md"$$NOinterp" #-> UndefVarError: $ not defined
# existing behavior
htl"$NOinterp \$NOinterp \\$NOinterp $$NOinterp \$$NOinterp" #-> INTERP \INTERP \\INTERP $NOinterp \$NOinterp
# desired behavior
    htl"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> INTERP \$NOinterp \\INTERP \$INTERP
    mdx"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> INTERP $NOinterp \INTERP $INTERP

Jun 27 '22 15:06 snoeyink

Jack,

Your proposal sounds very well thought out. It's good with me. There are few string literal users, so, I think there is little community impact. So, let's merge the pull request then and release?

Did you want to keep $$ as escape (is it even still implemented). I like it because it is an error in the regular macro context so there is no confusion.

Clark

Jun 28 '22 18:06 clarkevans

That would be great. I'm fine with the $$ remaining as an escape, since that is different behavior from \$ which prevents interpolation, but preserves both characters. The $$ escape won't affect my replacement of cm" " with mdx" " built on htl" ". Thanks!

Jun 29 '22 14:06 snoeyink