tree-sitter-hl-mode assumption of one major mode per buffer breaks polymode

[wyuenho]

Polymode is a fairly popular multiple major mode framework these days. The way it works is it looks at clearly demarcated code blocks, say in a Github Flavored Markdown or restructuredText file, and invokes the appropriate major modes to highlight the code blocks and respond to commands differently when the point is in these code blocks. This means when the user moves point into one of these code blocks, the major mode switches to a language that tree-sitter-language recognizes, and then the entire buffer is highlighted as if all text is written in that language.

Reproduction:

1. Install poly-rst with (require 'poly-rst)
2. Visit a new test.rst file
3. Type some lorum ipsum
4. Open a new line and then type

.. code-block:: python print('hello world')

5. Now the entire buffer is highlighted as if the restructuredText was Python.

Incidently, after you've saved and revisited the file, the entire buffer is highlighted correctly, presumably because now poly-rst has taken back control. But as soon as you start editing, and maybe deleted some Python code in the code block, this error occurs:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  tree-sitter-hl--extend-regions((3637 . 3661) (3637 . 3661))
  tree-sitter-hl--highlight-region(3637 3661 nil)
  apply(tree-sitter-hl--highlight-region (3637 3661 nil))
  #f(advice-wrapper :override #f(compiled-function (&rest args) #<bytecode 0x4cf746c9>) tree-sitter-hl--highlight-region)(3637 3661 nil)
  #f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x416772f7>)(3637 3661)
  apply(#f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x416772f7>) (3637 3661))
  polymode-inhibit-during-initialization(#f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x416772f7>) 3637 3661)
  apply(polymode-inhibit-during-initialization #f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x416772f7>) (3637 3661))
  font-lock-fontify-region(3637 3661)
  #f(compiled-function (fun) #<bytecode 0x4961e1ed>)(font-lock-fontify-region)
  run-hook-wrapped(#f(compiled-function (fun) #<bytecode 0x4961e1ed>) font-lock-fontify-region)
  jit-lock--run-functions(3637 3661)
  #f(compiled-function (span) #<bytecode 0x49364895>)((body 3610 3661 #<pm-inner-auto-chunkmode rst::python-mode>))
  pm-map-over-spans(#f(compiled-function (span) #<bytecode 0x49364895>) 3637 3662)
  poly-lock-fontify-now(3637 3662)
  poly-lock-function(3637)
  pos-visible-in-window-p()
  pm--select-existing-buffer-visibly(#<buffer  README.rst[python]>)
  pm-select-buffer((body 3610 3661 #<pm-inner-auto-chunkmode rst::python-mode>) visibly)
  pm-switch-to-buffer()
  polymode-post-command-select-buffer()

[ubolonton]

Now the entire buffer is highlighted as if the restructuredText was Python.

I could reproduce this with poly-markdown, not poly-rst.

The cause is probably the same: tree-sitter-mode and tree-sitter-hl-mode currently ignore narrowing. They don't have a setting similar to font-lock-dont-widen. The proper fix is probably for tree-sitter-mode (not tree-sitter-hl-mode) to provide such a setting, which can then be used by polymode.

Can you also provide steps to reproduce the signaled error?

[wyuenho]

How would polymode use the new setting? The steps I've provided is how to reproduce with poly-rst. I can reproduce the exact same issue with poly-markdown using the exact same steps.

[ubolonton]

How would polymode use the new setting?

By setting a variable in its indirect buffers, probably similar to how it sets font-lock-dont-widen.

The steps I've provided is how to reproduce with poly-rst. I can reproduce the exact same issue with poly-markdown using the exact same steps.

I couldn't. This can be config-specific. A minimal and reproducible config would be helpful (preferably with specific commits, e.g. through straight.el).

[wyuenho]

By lorum ipsum, I mean, type some random sentences that have python keywords in them, such as

For a long time in wonderland...

[wyuenho]

1. Put this in your init file

(use-package tree-sitter-langs
  :config
  (add-hook 'tree-sitter-after-on-hook 'tree-sitter-hl-mode))
(use-package poly-rst)

2. Visit a new buffer called ~/test.rst
3. Type For a long time in wonderland
4. RET
5. Type .. code-block:: python
6. C-j (RET won't work because there's a bug in polymode that interferes with indentation, you'll be backed up one line)
7. C-j again

Now polymode should have turned on python-mode within the code block, and you should see For and in are highlighted as if they are python keywords.

[wyuenho]

The behavior on markdown is a little different, here's the reproduction:

1. Put this in your init file

(use-package tree-sitter-langs
  :config
  (add-hook 'tree-sitter-after-on-hook 'tree-sitter-hl-mode))
(use-package poly-markdown)

2. Visit a new buffer called ~/test.md
3. Type For a long time in wonderland
4. RET
5. Type ` `
6. Then I get the following error.
7. If I move my point back to the buffer and continue typing ```python print("hello world") ```

for and in, the last 2 backticks of the opening triple backticks, and ("hello world are highlighted.

Debugger entered--Lisp error: (args-out-of-range 32 35)
  add-text-properties(32 35 (invisible markdown-markup))
  #f(compiled-function (highlight) "Apply HIGHLIGHT following a match.\nHIGHLIGHT should be of the form MATCH-HIGHLIGHT, see `font-lock-keywords'." #<bytecode 0x41a37c89>)((1 markdown-markup-properties))
  font-lock-fontify-keywords-region(1 34 nil)
  font-lock-default-fontify-region(1 34 nil)
  #f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x41a372ff>)(1 34)
  apply(#f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x41a372ff>) (1 34))
  polymode-inhibit-during-initialization(#f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x41a372ff>) 1 34)
  apply(polymode-inhibit-during-initialization #f(compiled-function (beg end &optional loudly) "Fontify the text between BEG and END.\nIf LOUDLY is non-nil, print status messages while fontifying.\nThis works by calling `font-lock-fontify-region-function'." #<bytecode 0x41a372ff>) (1 34))
  font-lock-fontify-region(1 34)
  #f(compiled-function (fun) #<bytecode 0x4be20919>)(font-lock-fontify-region)
  run-hook-wrapped(#f(compiled-function (fun) #<bytecode 0x4be20919>) font-lock-fontify-region)
  jit-lock--run-functions(1 34)
  poly-lock-fontify-now(1 34)
  poly-lock-function(1)
  redisplay_internal\ \(C\ function\)()

To fix both poly-rst and poly-markdown, I have to do this:

(use-package tree-sitter-langs
  :config
  (add-hook 'tree-sitter-after-on-hook
            (lambda ()
              (tree-sitter-hl-mode (if polymode-mode -1 1)))))

[vincentbernat]

I am running into the same error, but with consult-line. It replaces isearch and display excerpts of the current buffer while searching, so I suppose the problem can be similar.

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  tree-sitter-hl--extend-regions((64543 . 367742) (64543 . 367742))
  tree-sitter-hl--highlight-region(64543 367742 nil)
  tree-sitter-hl--highlight-region-with-fallback(#f(compiled-function (&rest args) #<bytecode -0x5ff386f63a3adf6>) 64543 367742 nil)
  apply(tree-sitter-hl--highlight-region-with-fallback #f(compiled-function (&rest args) #<bytecode -0x5ff386f63a3adf6>) (64543 367742 nil))
  #f(advice-wrapper :around #f(compiled-function (&rest args) #<bytecode -0x5ff386f63a3adf6>) tree-sitter-hl--highlight-region-with-fallback)(64543 367742 nil)
  font-lock-fontify-region(64543 367742)
  #f(compiled-function (fun) #<bytecode 0x199283b729558dfd>)(font-lock-fontify-region)
  jit-lock--run-functions(64543 367742)
  jit-lock-fontify-now()
  consult--fontify-all()
  consult--line-candidates(nil 3)
  consult-line(nil nil)
  funcall-interactively(consult-line nil nil)
  call-interactively(consult-line)
  (if (and start end (not multiline-p)) (consult-line (replace-regexp-in-string " " "\\\\ " (rxt-quote-pcre (buffer-substring-no-properties start end)))) (call-interactively #'consult-line))
  (cond ((or nil nil) (call-interactively (if (and start end (not multiline-p)) #'swiper-isearch-thing-at-point #'swiper-isearch))) (t (if (and start end (not multiline-p)) (consult-line (replace-regexp-in-string " " "\\\\ " (rxt-quote-pcre (buffer-substring-no-properties start end)))) (call-interactively #'consult-line))))
  (save-restriction (if (region-active-p) (progn (setq start (region-beginning) end (region-end) multiline-p (/= (line-number-at-pos start) (line-number-at-pos end))) (deactivate-mark) (if multiline-p (progn (narrow-to-region start end))))) (cond ((or nil nil) (call-interactively (if (and start end (not multiline-p)) #'swiper-isearch-thing-at-point #'swiper-isearch))) (t (if (and start end (not multiline-p)) (consult-line (replace-regexp-in-string " " "\\\\ " (rxt-quote-pcre (buffer-substring-no-properties start end)))) (call-interactively #'consult-line)))))
  (let (start end multiline-p) (save-restriction (if (region-active-p) (progn (setq start (region-beginning) end (region-end) multiline-p (/= (line-number-at-pos start) (line-number-at-pos end))) (deactivate-mark) (if multiline-p (progn (narrow-to-region start end))))) (cond ((or nil nil) (call-interactively (if (and start end (not multiline-p)) #'swiper-isearch-thing-at-point #'swiper-isearch))) (t (if (and start end (not multiline-p)) (consult-line (replace-regexp-in-string " " "\\\\ " (rxt-quote-pcre ...))) (call-interactively #'consult-line))))))
  +default/search-buffer()
  funcall-interactively(+default/search-buffer)
  command-execute(+default/search-buffer)