forward-sexp: Scan error: "Unbalanced parentheses", 121867, 1 with Emacs 29

[fredericgiquel]

Steps to reproduce

• Launch Emacs 29 with the following minimal init.el:

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

(use-package helpful
  :ensure t)
(use-package lsp-mode
  :ensure t)

• M-x helpful-callable
• Enter lsp--workspace-server-capabilities

Further information

The error only happens with Emacs 29. It works well with Emacs 28.

Debug trace:

Debugger entered--Lisp error: (scan-error "Unbalanced parentheses" 121867 1)
  scan-sexps(121868 -1)
  forward-sexp(-1 nil)
  backward-sexp()
  find-function--search-by-expanding-macros(#<buffer lsp-mode.el> lsp--workspace-server-capabilities nil)
  find-function-search-for-symbol(lsp--workspace-server-capabilities nil "/home/fred/.config/emacs-profiles/bug-helpful/elpa...")
  helpful--definition(lsp--workspace-server-capabilities t)
  helpful-update()
  helpful--update-and-switch-buffer(lsp--workspace-server-capabilities t)
  helpful-callable(lsp--workspace-server-capabilities)
  funcall-interactively(helpful-callable lsp--workspace-server-capabilities)
  command-execute(helpful-callable record)
  execute-extended-command(nil "helpful-callable" "helpful-ca")
  funcall-interactively(execute-extended-command nil "helpful-callable" "helpful-ca")
  command-execute(execute-extended-command)

Because of the error, lsp-mode.el is not closed. So we can swith to this buffer (opened in fundamental-mode) and execute check-parens. This command returns "Unmatched bracket or quote" in this case. But if we open the same lsp-mode.el with find-file (so with emacs-lisp major-mode), check-parens detects no problem.

Last thing, not binding auto-mode-alist to nil in helpful--open-if-needed fixes the problem. But it's probably too radical.

[fredericgiquel]

Another way to reproduce the error. This time lsp-mode package is not needed. Instead I had an unbalanced parentheses inside comments directly in init.el.

• Launch Emacs 29 with the following minimal init.el:

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

(use-package helpful
  :ensure t)

(cl-defstruct test-struct
  ;; just a smiley :) inside comments
  (test-slot nil))

• M-x helpful-callable
• Enter test-struct-test-slot

[Jake-Moss]

I can reproduce this on 29.1 (a9b28224af0f73d1fe0f422e9b318c5b91af889b) with emacs -Q using the built in variable mode-line-format

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

(use-package helpful
  :ensure t)

(helpful-variable 'mode-line-format)

However, opening the buffer.c buffer and switching to M-x c-mode and re-evaluating (helpful-variable 'mode-line-format) works.

Debugger entered--Lisp error: (scan-error "Unbalanced parentheses" 161071 205895)
  scan-lists(161053 1 0)
  forward-list()
  helpful--source(mode-line-format nil #<buffer buffer.c> 161053)
  helpful-update()
  helpful--update-and-switch-buffer(mode-line-format nil)
  helpful-variable(mode-line-format)
  (progn (helpful-variable 'mode-line-format))
  eval((progn (helpful-variable 'mode-line-format)) t)
  elisp--eval-last-sexp(nil)
  eval-last-sexp(nil)
  funcall-interactively(eval-last-sexp nil)
  call-interactively(eval-last-sexp nil nil)
  command-execute(eval-last-sexp)

[purplg]

I have this problem to. This seemingly innocuous commit (66ba816b26b68dd7df08e86f8b96eaae16c8d6a2) is the first one that has the error for me.

It doesn't error on every variable, but this consistently errors:

(helpful-variable 'load-path)

But this does not:

(helpful-variable 'user-emacs-directory)

[astratagem]

I got here because of the same error on the default value of default-frame-alist as defined in frame.c. I am running Emacs version 29.1.90.

[LemonBreezes]

Hi. Please let me know if #335 fixes this issue for you. @montchr @purplg @Jake-Moss @fredericgiquel.

[purplg]

@LemonBreezes Unfortunately this did not fix the issue for me.

I tested with emacs -Q using Emacs 29.1.90 (commit 77678244b83455b34a92e90ddeea4c68799d1d58)

Configured with:

./configure --with-imagemagick --with-json --with-native-compilation=aot --with-pgtk --with-tree-sitter --without-x

I'd be happy to provide any more details. I'm not sure what information would be useful for this kind of error.

[LemonBreezes]

@LemonBreezes Unfortunately this did not fix the issue for me.

I tested with emacs -Q using Emacs 29.1.90 (commit 77678244b83455b34a92e90ddeea4c68799d1d58)

Configured with:

./configure --with-imagemagick --with-json --with-native-compilation=aot --with-pgtk --with-tree-sitter --without-x

I'd be happy to provide any more details. I'm not sure what information would be useful for this kind of error.

Okay. You're right. It works if we do (funcall major-mode) but not if we just let-bind the major mode. I'm thinking of how to make this run faster or if we should just enable the major-mode.

EDIT: I think I'm going to leave it as-is calling the major mode. Let-binding the syntax table doesn't work for this.

[fredericgiquel]

@LemonBreezes The error described in first and second post is still present when I test with your PR.

The first problem has been fixed in lsp-mode but I revert the commit to test.

[LemonBreezes]

@LemonBreezes The error described in first and second post is still present when I test with your PR.

The first problem has been fixed in lsp-mode but I revert the commit to test.

You've tried it with the current version where we call the major mode as well? I'm thinking we should just call the major mode or not let-bind auto-mode-alist.

[fredericgiquel]

@LemonBreezes As far as I understand the PR, your modification only applies to symbols defined in C (condition in line 1255) whereas the problem described happens in Emacs Lisp files.

I also did a test by adding (debug-on-entry 'helpful--source) to the init.el file and there is no debugger invocation before the "Unbalanced parentheses" error. So helpful--source function is not involved in the problem.

[LemonBreezes]

@LemonBreezes As far as I understand the PR, your modification only applies to symbols defined in C (condition in line 1255) whereas the problem described happens in Emacs Lisp files.

I also did a test by adding (debug-on-entry 'helpful--source) to the init.el file and there is no debugger invocation before the "Unbalanced parentheses" error. So helpful--source function is not involved in the problem.

Ohhh, makes sense. The fix involving setting the major-mode was specifically because I got an error using describe-variable with mode-line-format. I get the feeling we need to contact the Emacs maintainers to ask for more information about scan-sexps and why it's giving that error.

[LemonBreezes]

@LemonBreezes Unfortunately this did not fix the issue for me.

I tested with emacs -Q using Emacs 29.1.90 (commit 77678244b83455b34a92e90ddeea4c68799d1d58)

Configured with:

./configure --with-imagemagick --with-json --with-native-compilation=aot --with-pgtk --with-tree-sitter --without-x

I'd be happy to provide any more details. I'm not sure what information would be useful for this kind of error.

I think I finally figured out the right fix in https://github.com/Wilfred/helpful/pull/336.

[roshanshariff]

@LemonBreezes, I'm not sure if the change in #336 is the way to go. For one, using the emacs-lisp-mode syntax table isn't correct for C code, when you're finding the definition of a function or variable that's defined in a .c file.

In fact, when Helpful opens a source buffer, it uses the helpful--open-if-needed function, which takes care of setting either the emacs-lisp-mode syntax table or a C syntax table as appropriate. But I'm not able to figure out why this syntax table isn't taking effect, leading to the scan errors.

[LemonBreezes]

@LemonBreezes, I'm not sure if the change in #336 is the way to go. For one, using the emacs-lisp-mode syntax table isn't correct for C code, when you're finding the definition of a function or variable that's defined in a .c file.

In fact, when Helpful opens a source buffer, it uses the helpful--open-if-needed function, which takes care of setting either the emacs-lisp-mode syntax table or a C syntax table as appropriate. But I'm not able to figure out why this syntax table isn't taking effect, leading to the scan errors.

Yeah, the error occurs with the c-mode-syntax-table but not with the emacs-lisp-mode-syntax-table basically.

[roshanshariff]

Manually running the following code in the buffer returned by helpful--open-if-needed still doesn't fix the issue:

(let ((syntax-table (make-syntax-table)))
  (c-populate-syntax-table syntax-table)
  (set-syntax-table syntax-table))

This is effectively what helpful--open-if-needed does, hence the scan errors.

However, manually enabling c-mode in the buffer and then calling Helpful does work. So we need to figure out how c-mode sets the syntax table. Unfortunately, that code is very complex, and I haven't been able to figure out what it does in addition to that snippet above.

[purplg]

@LemonBreezes Unfortunately this did not fix the issue for me. I tested with emacs -Q using Emacs 29.1.90 (commit 77678244b83455b34a92e90ddeea4c68799d1d58) Configured with:

./configure --with-imagemagick --with-json --with-native-compilation=aot --with-pgtk --with-tree-sitter --without-x

I'd be happy to provide any more details. I'm not sure what information would be useful for this kind of error.

I think I finally figured out the right fix in #336.

This seems to work for me now! :)

[dm701]

I actually get a similar error with this package, enabled as per the OP, and calling C-h v load-path.

It returns:

forward-list: Scan error: "Unbalanced parentheses", 157780, 170875

[LemonBreezes]

Just updated my PR #336 to add a test case. Hopefully that will be merged as a temporary fix until the day someone decides to grok the C syntax table code enough to set that up properly.