[JavaScript] HTML and CSS syntax highlighting in tagged template strings

[jrburke]

Is it possible to support HTML/CSS syntax highlighting inside a tagged template string? I am looking to do the equivalent of this atom editor change.

My first attempt at this did not work out so well. I just tried modifying the existing literal-quasi (but locally renamed it to literal-template-string everywhere in the file), just to see if I could get the HTML referencing correctly, before creating a new kind of match just for the "ending in html" tagged template function case:

  literal-template-string:
    - match: '([a-zA-Z$_][\w$_]*)?(`)'
      captures:
        1: entity.template-string.tag.name.js
        2: punctuation.definition.template-string.begin.js
      push:
        - meta_content_scope: source.html.embedded.js
        - include: 'scope:text.html.basic'
      with_prototype:
        - match: "`"
          captures:
            0: punctuation.definition.template-string.end.js
          pop: true
        - match: '\${'
          captures:
            0: punctuation.template-string.element.begin.js
          push:
            - meta_scope: entity.template-string.element.js
            - match: "}"
              captures:
                0: punctuation.template-string.element.end.js
              pop: true
            - include: expression

But I get an Apparent recursion within a with_prototype action: 25000 context sanity limit hit error. I am very new to these kinds of syntax files, so I expect I am making some very simple mistakes. I appreciate receiving any pointers on how to fix the issue.

My guess is that since JavaScript.sublime-syntax's expression has include: literal-template-string and literal-template-string is referencing source:text.html.basic, which inside of it references scope:source.js, maybe some pathway with all of that is causing the recursion.

Related context:

• This does not take into account the changes in #165, I expect to redo some of it once that changeset lands.
• Previous discussion in Benvie/JavaScriptNext.tmLanguage#134, but I would want to scope the use of HTML to tagged template functions ending in (?i)html, to allow alternate highlighting for other content, like CSS for functions ending in (?i)css, as the Atom changeset does.
• GitHub's source display now does this kind of higlighting: example test.js.
• That test file is part of a tts-syntax-highlight repo to track this sort of enhancement in editors now that ES2015 template strings are used more.

[orizens]

There's also a suggested YML solution in this thread https://forum.sublimetext.com/t/javascript-es6-template-literals-syntax-for-html/18242

[Rayraegah]

I made it work by extending the default JavaScript Syntax.

Edited to include suggestions from comments below.

%YAML 1.2
---
# See http://www.sublimetext.com/docs/3/syntax.html
name: JavaScript Custom Element
file_extensions:
  - element.js
  - tag.js
  - jsx
  - js
scope: source.js.tag
contexts:
  main:
    - match: ""
      push: Packages/JavaScript/JavaScript.sublime-syntax
      with_prototype:
      - match: '([a-zA-Z$_][\w$_]*)?(`)'
        push:
          - meta_content_scope: text.html.basic.embedded.js
          - include: 'scope:text.html.basic'
          - match: '`'
            pop: true
          - match: '\$\{'
            captures:
              0: punctuation.definition.template-expression.begin.js
            push:
              - meta_scope: meta.template.expression.js
              - include: 'scope:source.js'
              #- meta_content_scope: source.js.embedded.expression
              - match: '\}'
                scope: punctuation.definition.template-expression.end.js
                pop: true

[3mcd]

@Rayraegah Your solution doesn't work with tagged template literals.

Replace - match: '`\n' with - match: '([a-zA-Z$_][\w$_]*)?(`)' and they will work.

[qodesmith]

@Rayraegah where did you save that file and with what name?

[Rayraegah]

@qodesmith Just create a new syntax file and dump that code in there. Also, use @ericmcdaniel 's suggestion.

Then, in your editor settings, set open all files with the new syntax instead of the default.

[Thom1729]

The JS Custom package lets you add user-defined template tags. In my opinion, this is a better approach than adding them to the core syntax.

[Slakinov]

@Thom1729 would you mind explaining in a bit more detail how the JS Custom package should be set up to show html syntax within `` backticks in JS files?

[Thom1729]

JS Custom version 1.0.13 is out, updating the package for Sublime 3.1.

More importantly, I've just fixed a really egregious bug in the documentation. Try this:

{
    "configurations": {
        "Vue Backticks": {
            "custom_template_tags": {
                "html": "scope:text.html.basic"
            },
        },
    },
}

[Slakinov]

@Thom1729 Thanks for the info! Just updated the package and tried that config. I'm now getting the recursion warning someone mentioned above. Any idea how to fix? Do I need to make an alternative html syntax definition that doesn't attempt to parse js again within itself? Sorry if these are dumb questions.

[Thom1729]

Are you running Sublime 3.1? The new version of the core HTML syntax uses embed/escape instead of push/with_prototype, and that should avoid the infinite recursion.

[Slakinov]

I'm running build 3170, which is 3.1 according to the sublime site, but 3.0 in my about window.

Had to remove some user package files after removing the JS Custom package to stop the recursion error when viewing any .html file.

I've now reinstalled JS Custom, and added your config above, no recursion error now, but no backtick highlighting either...

Any more ideas?

[keith-hall]

my guess is you need to write html before the opening backtick.

[Thom1729]

Yes, you'd need to write:

template: html`
    <div class="playlist">...</div>
`

The JS Custom extension isn't designed to apply arbitrary highlighting to regular, untagged templates. You probably wouldn't want all template strings to be highlighted as HTML, because template strings are used for many purposes.

However, there may be an opportunity for extension here. The Ecmascript Sublime lets you specify a syntax in a comment, like so:

template: /* syntax: html */ `
    <div class="playlist">...</div>
`

A similar feature might be useful for JS Custom. Of course, like custom_template_tags, the feature would be fully configurable. Would this be useful to you?

[Slakinov]

Hmm. This isn't straightforward is it :( Putting html before a backtick makes the JS invalid, Vue can't execute it.

[Slakinov]

I personally would find a syntax highlighter useful that assumes all backtick blocks in JS contain html, since I never use them for anything else in JS files.

[kleinfreund]

Putting html before a backtick makes the JS invalid

That’s not correct. Those are called tagged template literals.

I personally would find a syntax highlighter that assumes all backtick blocks in JS to contain html, since I never use them for anything else in JS files.

You might not use them for anything else, but a lot of people use them for everything but HTML. The default should definitely be no extra highlighting.

[Slakinov]

I'm not saying this should be default JS highlighting at all, I'm just trying to get a solution to work for my own case, and would imagine other people would want this behaviour as an option, too.

Haven't tried running my code through Babel in this case, but Chrome throws an error with tagged template literals:

[Thom1729]

The MDN reference explains in more detail. The "tag" of a template literal is a function that processes the literal's contents. So in this context, html should be a function that takes in the literal's text and produces some value.

As a workaround for now, you could implement it with const html = String.raw.

[Slakinov]

Aha! That works, thanks :)

[Slakinov]

...And now the recursion errors are back when reopening Sublime 😭
Regular HTML syntax highlighting appears to be hosed.

Sublime build 3176 Latest JSCustom, with the settings suggested above Have tried 'rebuild syntaxes'

[Slakinov]

https://github.com/bathos/Ecmascript-Sublime works great, I can live with those syntax declaration comments.

Thanks again @Thom1729

[Thom1729]

I've figured it out.

The underlying problem is that the HTML syntax is embedding the source.js scope, while the JS Custom syntax is embedding the HTML syntax using with_prototype. This is causing the recursion error. I had thought that if at least one of those two syntaxes used embed, then it would work. Apparently, I was wrong.

Ecmascript-Sublime works around this by accident. Because of the way that syntax is written, the HTML syntax won't embed it; it will embed another JavaScript syntax instead, such as the core syntax. This means that you can't use any of Ecmascript-Sublime's features in embedded JavaScript, and so the error never comes up.

The HTML syntax will embed JS Custom syntaxes, though. Specifically, it will embed the syntax for the last configuration in lexicographic order. If that syntax embeds HTML using with_prototype, the error you've found will occur. I couldn't reproduce the error because the configuration I was testing all of this on was not the last one in lexicographic order.

I plan to solve this in a systematic manner. However, there is a simple workaround that will solve the problem for you today. Add the following configuration to your JS Custom.sublime-settings:

        "~default": {
            "hidden": true,
            "custom_template_tags": null,
        },

This will create a new, hidden syntax that will be used by the HTML syntax and other syntaxes for embedded JavaScript. It will use all of your default settings, except that it won't use any custom template tags. This should eliminate those recursion errors once and for all.

Sorry that this ended up being a hassle!

[Slakinov]

Super-delayed thanks for that last post :) Have been using JS Custom with these settings for months, working perfectly for my needs. Just posting here for reference:

[niksy]

@Rayraegah your suggestion doesn’t seem to be working properly when you have JS inside template literal. Is this up to date?

[Rayraegah]

@niksy I didn't have a case where I had to handle JS inside a template literal (other than template expressions). Can you post a snippet of your code where syntax highlight fails? (screenshot would help as well, along with the name of the sublime theme)

I did try some edge cases: can you check with the following commented line uncommented?

            push:
              - meta_scope: meta.template.expression.js
              - include: 'scope:source.js'
              #- meta_content_scope: source.js.embedded.expression
              - match: '\}'
                scope: punctuation.definition.template-expression.end.js
                pop: true

[peterVG]

It's alive!! I went the JS Custom package route (preferred over custom syntax files) but it took me way longer than I'd like to admit to get it fully working. I would have been completely lost if not for this thread. However, I think it can do with a few updates to hopefully save the next person some time.

My setup:

• Sublime Text v. 3.2.2.
• JS Custom v. 2.4.1

Firstly, don't include the following setting referred to as a work-around above:

"~default": { "hidden": true, "custom_template_tags": null, },

I suppose this workaround was fixed and made reduant in a later JS Custom version. As far as I can tell, it prevented me from seeing the JS Custom option in the View > Syntax menu. Only when I removed it and rebuilt the JS Custom syntax file did the menu option finally appear. Also, JS Custom is now listed directly in that menu, rather than under a User submenu as suggested somewhere further up in the thread.

The HTML in Javascript highlighting did work in Sublime when I used the html`<div>... directive BUT I'm using this for VueJS and that convention breaks its Component parsing. Fortunately JS Custom now has a comments configuration setting that allows for the highlighting of untagged template literals based on a preceding block comment. So this is my configuration:

Which now enables the flagging of my template literals using the /* html */ directive:

It plays nice with VueJS setup and my SublimeLinter StandardJS settings, which continue to work with JS Custom in place. Note however, that I did have to move my Javascript syntax preferences from Javascript.sublime-settings to Vue Backticks.sublime-settings (namely to enforce 2-space tabs in JS files).

[Thom1729]

I can confirm that the ~default workaround is obsolete. The current version of JS Custom avoids the infinite recursion issues, so there's no need to manually create a ~default configuration.

I'd appreciate any suggestion you might have to improve the documentation.

[peterVG]

Hey @Thom1729! Thanks for this great package. It's a pleasure to use now that I have it up and running. So glad I don't have to give up syntax highlighting in template literals as it's so fundamental to VueJS which I'm trying to implement ATM.

As for docs, you might have picked up from my comment above that, because I've only defined one configuration in the User settings, it appears directly under the View > Syntax menu (which is nice):

However, if I add another configuration then, yes, it does appear under View > Syntax > User > JS Custom > [config-name] as you have documented. Not sure you if you want to be more specific about that.

Also, while your "style": "scope:source.css" example value for the comments configuration option is useful, perhaps most users will want to use this option with html comments like I did? At the risk of making the documentation too verbose you may want to provide the html example as well as it's not totally straightforward what scope value to use if you're not familiar with the inner workings of the syntax highlighting functions.

{
    "configurations": {
        "Vue Backticks": {
            "custom_templates": {
                "comments": {
                    "html" : "scope:text.html.basic"
                },
            },
        },
    },
}

[Thom1729]

Huh. So apparently if you have exactly one configuration, it will appear directly in the menu, but if you have several they will appear in a subfolder. I did not know this (though in hindsight it seems obvious). This is definitely confusing; I'll open an issue.

[kireerik]

I am using the following configuration:

• https://github.com/kireerik/refo/tree/master/example#sublime-text

 

This way I don't need to use comments.