[Go] Highlight format verbs inside Printf strings

[akinsho]

Describe the highlighting problem

In Sublime, format verbs in printf like strings in go e.g %v, %d, %s etc. are highlighted differently in the string, so it's clear when looking at a string where the format verbs are/where the substitutions will be. This currently isn't the case for the Treesitter go highlights. I'm currently trying to implement this with no luck. I've tried to add the highlight in the TS playground, but this seems to error with an Invalid Node error.

Example snippet that causes the problem

(interpreted_string_literal "%d" @punctuation.special) @none

Tree-sitter parsing result

call_expression [46, 14] - [48, 13]
  function: selector_expression [46, 14] - [46, 24]
    operand: identifier [46, 14] - [46, 17]
    "." [46, 17] - [46, 18]
    field: field_identifier [46, 18] - [46, 24]
  arguments: argument_list [46, 24] - [48, 13]
    "(" [46, 24] - [46, 25]
    interpreted_string_literal [46, 25] - [46, 82]
      """ [46, 25] - [46, 26]
      """ [46, 81] - [46, 82]
    "," [46, 82] - [46, 83]
    identifier [47, 3] - [47, 8]
    "," [47, 8] - [47, 9]

Example screenshot

Expected behavior

In the screenshot above, %v and %d etc. are specially highlighted, so it's clear where the replacement values are in a string.

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()
========================================================================
## Installation
  - OK: `tree-sitter` found 0.20.6 (parser generator, only needed for :TSInstallFromGrammar)
  - OK: `node` found v17.8.0 (only needed for :TSInstallFromGrammar)
  - OK: `git` executable found.
  - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
    Version: cc (Homebrew GCC 11.2.0_3) 11.2.0
  - OK: Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

## Parser/Features H L F I J
  - scss           ✓ . . ✓ . 
  - erlang         . . . . . 
  - cmake          ✓ . ✓ . . 
  - elixir         ✓ ✓ ✓ ✓ ✓ 
  - cpp            ✓ ✓ ✓ ✓ ✓ 
  - javascript     ✓ ✓ ✓ ✓ ✓ 
  - help           ✓ . . . . 
  - surface        ✓ . ✓ ✓ ✓ 
  - pioasm         ✓ . . . ✓ 
  - bash           ✓ ✓ ✓ . ✓ 
  - eex            ✓ . . . ✓ 
  - html           ✓ ✓ ✓ ✓ ✓ 
  - ocaml_interface✓ ✓ ✓ . ✓ 
  - tsx            ✓ ✓ ✓ ✓ ✓ 
  - ocaml          ✓ ✓ ✓ . ✓ 
  - hcl            ✓ . ✓ ✓ ✓ 
  - llvm           ✓ . . . . 
  - org            . . . . . 
  - swift          ✓ ✓ . . . 
  - latex          ✓ . ✓ . ✓ 
  - c_sharp        ✓ ✓ ✓ . ✓ 
  - foam           ✓ ✓ ✓ ✓ ✓ 
  - markdown       ✓ . ✓ . ✓ 
  - typescript     ✓ ✓ ✓ ✓ ✓ 
  - hack           ✓ . . . . 
  - scheme         ✓ . ✓ . ✓ 
  - scala          ✓ . ✓ . ✓ 
  - supercollider  ✓ ✓ ✓ ✓ ✓ 
  - proto          ✓ . ✓ . . 
  - m68k           ✓ ✓ ✓ . ✓ 
  - slint          ✓ . . ✓ . 
  - json           ✓ ✓ ✓ ✓ . 
  - vim            ✓ ✓ . . ✓ 
  - haskell        ✓ . . . ✓ 
  - lalrpop        ✓ ✓ . . . 
  - rego           ✓ . . . ✓ 
  - wgsl           ✓ . ✓ . . 
  - solidity       ✓ . . . . 
  - tlaplus        ✓ ✓ ✓ . ✓ 
  - rasi           ✓ ✓ ✓ ✓ . 
  - clojure        ✓ ✓ ✓ . ✓ 
  - cooklang       ✓ . . . . 
  - vala           ✓ . . . . 
  - toml           ✓ ✓ ✓ ✓ ✓ 
  - elvish         ✓ . . . ✓ 
  - lua            ✓ ✓ ✓ ✓ ✓ 
  - norg           ✓ . ✓ . ✓ 
  - astro          ✓ ✓ ✓ ✓ ✓ 
  - hocon          ✓ . . . ✓ 
  - cuda           ✓ ✓ ✓ ✓ ✓ 
  - pug            ✓ . . . ✓ 
  - d              ✓ . ✓ ✓ ✓ 
  - prisma         ✓ . . . . 
  - vue            ✓ . ✓ ✓ ✓ 
  - http           ✓ . . . ✓ 
  - make           ✓ . . . ✓ 
  - glsl           ✓ ✓ ✓ ✓ ✓ 
  - hjson          ✓ ✓ ✓ ✓ ✓ 
  - dockerfile     ✓ . . . ✓ 
  - elm            ✓ . . . ✓ 
  - json5          ✓ . . . ✓ 
  - dot            ✓ . . . ✓ 
  - fortran        ✓ . ✓ ✓ . 
  - yang           ✓ . ✓ ✓ . 
  - rust           ✓ ✓ ✓ ✓ ✓ 
  - zig            ✓ . ✓ ✓ ✓ 
  - ninja          ✓ . ✓ ✓ . 
  - jsonc          ✓ ✓ ✓ ✓ ✓ 
  - ocamllex       ✓ . . . ✓ 
  - ledger         ✓ . ✓ ✓ ✓ 
  - r              ✓ ✓ . ✓ ✓ 
  - ruby           ✓ ✓ ✓ ✓ ✓ 
  - dart           ✓ ✓ . ✓ ✓ 
  - ql             ✓ ✓ . ✓ ✓ 
  - python         ✓ ✓ ✓ ✓ ✓ 
  - rst            ✓ ✓ . . ✓ 
  - verilog        ✓ ✓ ✓ . ✓ 
  - fennel         ✓ ✓ . . ✓ 
  - jsdoc          ✓ . . . . 
  - gomod          ✓ . . . ✓ 
  - go             ✓ ✓ ✓ ✓ ✓ 
  - gowork         ✓ . . . ✓ 
  - gdscript       ✓ ✓ . ✓ ✓ 
  - nix            ✓ ✓ ✓ . ✓ 
  - graphql        ✓ . . ✓ ✓ 
  - query          ✓ ✓ ✓ ✓ ✓ 
  - pascal         ✓ ✓ ✓ ✓ ✓ 
  - c              ✓ ✓ ✓ ✓ ✓ 
  - godot_resource ✓ ✓ ✓ . . 
  - perl           ✓ . ✓ . . 
  - comment        ✓ . . . . 
  - regex          ✓ . . . . 
  - turtle         ✓ ✓ ✓ ✓ ✓ 
  - fish           ✓ ✓ ✓ ✓ ✓ 
  - devicetree     ✓ ✓ ✓ ✓ ✓ 
  - teal           ✓ ✓ ✓ ✓ ✓ 
  - php            ✓ ✓ ✓ ✓ ✓ 
  - sparql         ✓ ✓ ✓ ✓ ✓ 
  - svelte         ✓ . ✓ ✓ ✓ 
  - java           ✓ ✓ . ✓ ✓ 
  - fusion         ✓ ✓ ✓ ✓ . 
  - commonlisp     ✓ ✓ ✓ . . 
  - kotlin         ✓ ✓ ✓ . ✓ 
  - beancount      ✓ . ✓ . . 
  - gleam          ✓ ✓ ✓ ✓ ✓ 
  - heex           ✓ ✓ ✓ ✓ ✓ 
  - julia          ✓ ✓ ✓ ✓ ✓ 
  - todotxt        ✓ . . . . 
  - glimmer        ✓ . . . . 
  - bibtex         ✓ . ✓ ✓ . 
  - css            ✓ . ✓ ✓ ✓ 
  - yaml           ✓ ✓ ✓ ✓ ✓ 

  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang}

Output of nvim --version

:version                                                                                           
NVIM v0.7.0                                                                                        
Build type: Release                                                                                
LuaJIT 2.1.0-beta3                                                                                 
Compiled by [email protected]                                                          
                                                                                                   
Features: +acl +iconv +tui                                                                         
See ":help feature-compile"                                                                        
                                                                                                   
   system vimrc file: "$VIM/sysinit.vim"                                                           
  fall-back for $VIM: "/opt/homebrew/Cellar/neovim/0.7.0/share/nvim"                               
                                                                                                   
Run :checkhealth for more info

Additional context

No response

[theHamsta]

I don't know why we can't capture escape_literal anymore: https://github.com/tree-sitter/tree-sitter-go/blob/372d3241b099e189406475a9445cbb29dac2e054/grammar.js#L842 but this should be fixed https://github.com/fsouza/nvim-treesitter/blob/7bdee273486f80c5ddce4d9600d1ed5e506b704d/queries/go/highlights.scm#L189

[akinsho]

@theHamsta I can't see the link to the highlights.scm you are referring too, and I can't find an associated PR either

[bab014]

I'd like to add on to this. It would be nice to get this working. When capturing under the cursor I get this. Seems like it's identified in the syntax, how would we go about mapping that to a color?

And yeah, that second link goes nowhere.

[akinsho]

That's interesting @bab014 I don't see goFormatSpecifier on my end, I just see goString

[bab014]

Oh, weird. I mean, I don't know where that Syntax section comes from but you'd think it'd be the same.

[akinsho]

@bab014 🤦🏾‍♂️ wasn't paying close enough attention. The syntax block is because you likely are using a regex based syntax plugin for go, as well as the go treesitter highlights. The treesitter bit doesn't identify the format specifier, only the regex does.

[bab014]

@akinsho Ah, that makes more sense.

[bab014]

@theHamsta So what can we do to get the format verb in the grammar file? I'm new to treesitter and would love to help, but would need a small push in the right direction

[theHamsta]

Just have a look how it is implemented in grammar.js in tree-sitter-c and do the same in tree-sitter-go. The rule for parsing strings should be more less the same in both languages. You can test using : TSIstallFrom Grammar setting the tree-sitter-go Url to a local path in parser.lua

[bab014]

Thanks @theHamsta I'll give it a try

[bab014]

@theHamsta Been running into an error when trying to install. I just want to test that the following works

format_verb: $ => token.immediate(seq(
  '%',
  choice(/(?i)[a-z]/)
))

I just want to test if we can capture '%/[a-z]/' just to see if it works and fine tune later.

Do you think this would work?

The error is when I try and install from grammar.

E5108: Error executing lua ...er/start/nvim-treesitter/lua/nvim-treesitter/install.lua:55: files: expected table, got nil
stack traceback:
        [C]: in function 'error'
        vim/shared.lua: in function 'validate'
        ...er/start/nvim-treesitter/lua/nvim-treesitter/install.lua:55: in function 'get_parser_install_info'
        ...er/start/nvim-treesitter/lua/nvim-treesitter/install.lua:373: in function 'install_lang'
        ...er/start/nvim-treesitter/lua/nvim-treesitter/install.lua:419: in function 'run'
        [string ":lua"]:1: in main chunk

[theHamsta]

Did you forget the files entry?

list.go = {
  install_info = {
    url = "path-to-local/tree-sitter-go",
    files = { "src/parser.c" },
  },
  maintainers = { "@theHamsta", "@WinWisely268" },
}

it fails here: https://github.com/theHamsta/nvim-treesitter/blob/934409208d9f0014d43396d64666051361a02743/lua/nvim-treesitter/install.lua#L57

[bab014]

So what I did was clone the tree-sitter-go repository, and created a branch for myself. I made the change to that top level grammar.jsfile then tried running :TSInstallFromGrammar pointing to that file.

When I look at the nvim-treesitter repo on my machine I see the lines of code you're pointing me to.

Would it be easier to just manually create the parser from the changed grammar file?

[bab014]

@theHamsta Ok, so I forked the repo and modified the grammar.js file https://github.com/bab014/tree-sitter-go/blob/format-verbs/grammar.js#L852. I built it with npm run build and tried parsing this file https://github.com/bab014/tree-sitter-go/blob/format-verbs/tests/main.go. It doesn't catch my format_verb token.

Not sure what I'm doing wrong.

[theHamsta]

It's because of token. Token makes everything inside of it a single token ("atom" of the syntax tree, so you can't see anything inside it) https://github.com/bab014/tree-sitter-go/blob/cc15e8eaada53b1e410d570d5c41613cef34395b/grammar.js#L832

tree-sitter-c does not use token: https://github.com/tree-sitter/tree-sitter-c/blob/e348e8ec5efd3aac020020e4af53d2ff18f393a9/grammar.js#L960-L967

[bab014]

@theHamsta you're looking at 'raw_string_literal' I'm using the normal string

[bab014]

@theHamsta So am I to understand there is no way of identifying format verbs for Go???

[theHamsta]

I guess you need to do just the same as for escape_sequence: https://github.com/tree-sitter/tree-sitter-go/blob/c03e250fe4b4021b0a0c81cf63b143371987ad40/grammar.js#L800-L818. Don't allow % in a normal string and add a rule in parallel to escape sequence just starting with % . You can see here that \ is forbidden in the string rule https://github.com/tree-sitter/tree-sitter-go/blob/c03e250fe4b4021b0a0c81cf63b143371987ad40/grammar.js#L803-L803. The escape sequence starts with a \ https://github.com/tree-sitter/tree-sitter-go/blob/c03e250fe4b4021b0a0c81cf63b143371987ad40/grammar.js#L810-L810 In your rule you missed to ad % here https://github.com/tree-sitter/tree-sitter-go/blob/c03e250fe4b4021b0a0c81cf63b143371987ad40/grammar.js#L800-L818

Can you please try whether this works?

[bab014]

@theHamsta That did the trick. Thank you. I can test this now.

[bab014]

@akinsho @theHamsta Here is my change in action

[akinsho]

That looks great @bab014 🙌🏾 thanks for working on it

[bab014]

Still has some work to do. Right now it only accounts for format verbs with a letter. The float and precision ones will require more regex work, and I'm not great with regex.

[akinsho]

@bab014 maybe just as the simplest ones first and tackle more complex stuff in a follow-up PR as and when you have time 🤷🏾‍♂️

[BacchusJackson]

Is this still an issue? Finding it in May 2023

[IlyasYOY]

Hello, I faced the same problem. Highlighting works for some methods but not for Errorf.

Here is the simple thing you can do in your dot files to fix the problem: https://github.com/IlyasYOY/dotfiles/blob/59db6b5469aef63bec6b82eb08564b9753432f55/config/nvim/after/queries/go/injections.scm#L3-L12

[lucario387]

Hello, I faced the same problem. Highlighting works for some methods but not for Errorf.

Well it's impossible to follow with all the default specifications of all languages. If you believe we have missed some stuffs that is in an stdlib of a language, make a PR for it

[clason]

Since the parser request for this was closed, there's nothing we can do here, so I'm closing this. Query improvements are always welcome; no need for an issue tracking this.