issues with syntax propertization of |escaped symbols|

[tgbugs]

Two issues related to symbols escaped with |.

1. unquoted escaped symbols are not propertized correctly so things like parens leak out.
2. It seems like the priority for | and " is backwards, so |foo " bar| incorrectly starts a quote.
3. A block comment followed by an escaped symbol incorrectly propertizes everything until the end of the next block comment as a comment.

;; Issue 482
'(|[| |]|)
|foo " bar|
#|yes-comment|# 'no-comment '|disaster-strikes| 'no-comment #|this fixes it|# 'no-comment

It looks to me like the fixes that were put in place for https://github.com/greghendershott/racket-mode/issues/298 broke the |foo " bar| case since removing ?\" from the various rx nots fixes that issue but reintroduces the problem described in 298.

I have tried a variety of potential solutions by modifying the existing rx based propertization for both of these and all my attempts include another error somewhere else.

``` ((alist-get 'racket-mode package-alist) #s(package-desc racket-mode (20200623 1549) "Racket editing, REPL, and more" ((emacs (25 1)) (faceup (0 0 2)) (pos-tip (20191127 1028))) nil nil "/home/tom/.emacs.d/elpa/racket-mode-20200623.1549" ((:url . "https://www.racket-mode.com/") (:maintainer "Greg Hendershott") (:authors ("Greg Hendershott")) (:commit . "5eb31a284c8db5102bc71b1e2c6ce3a095ae085b")) nil)) ((emacs-version "28.0.50") (system-type gnu/linux) (x-gtk-use-system-tooltips t) (major-mode emacs-lisp-mode) (racket--el-source-dir "/home/tom/.emacs.d/lisp/racket-mode/") (racket--rkt-source-dir "/home/tom/.emacs.d/lisp/racket-mode/racket/") (racket-program "racket") (racket-command-timeout 10) (racket-xp-after-change-refresh-delay 1) (racket-xp-highlight-unused-regexp "^[^_]") (racket-repl-buffer-name-function nil) (racket-memory-limit 2048) (racket-error-context medium) (racket-history-filter-regexp "\\`\\s *\\S ?\\S ?\\s *\\'") (racket-images-inline t) (racket-images-keep-last 100) (racket-images-system-viewer "display") (racket-images-system-viewer "display") (racket-use-repl-submit-predicate nil) (racket-pretty-print t) (racket-indent-curly-as-sequence t) (racket-indent-sequence-depth 0) (racket-pretty-lambda nil) (racket-smart-open-bracket-enable nil) (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)") (racket-logger-config ((cm-accomplice . warning) (GC . info) (module-prefetch . warning) (optimizer . info) (racket/contract . error) (sequence-specialization . info) (* . fatal))) (racket-show-functions (racket-show-echo-area))) (enabled-minor-modes (async-bytecomp-package-mode) (auto-composition-mode) (auto-compression-mode) (auto-encryption-mode) (auto-fill-mode) (auto-revert-mode) (auto-save-mode) (column-number-mode) (csv-field-index-mode) (display-line-numbers-mode) (eldoc-mode) (electric-indent-mode) (electric-pair-mode) (evil-local-mode) (evil-mode) (file-name-shadow-mode) (font-lock-mode) (global-display-line-numbers-mode) (global-eldoc-mode) (global-font-lock-mode) (global-git-commit-mode) (global-magit-file-mode) (global-undo-tree-mode) (global-visual-line-mode) (highlight-numbers-mode) (hl-todo-mode) (hs-minor-mode) (line-number-mode) (magit-auto-revert-mode) (magit-file-mode) (menu-bar-mode) (mouse-wheel-mode) (override-global-mode) (rainbow-delimiters-mode) (savehist-mode) (shell-dirtrack-mode) (show-paren-mode) (tooltip-mode) (transient-mark-mode) (undo-tree-mode) (url-handler-mode) (visual-line-mode) (which-function-mode)) (disabled-minor-modes (abbrev-mode) (archive-subfile-mode) (auto-complete-mode) (auto-fill-function) (auto-revert-tail-mode) (auto-save-visited-mode) (bibtex-completion-notes-global-mode) (bibtex-completion-notes-mode) (blink-cursor-mode) (buffer-face-mode) (buffer-read-only) (bug-reference-mode) (bug-reference-prog-mode) (cider--debug-mode) (cider-auto-test-mode) (cider-enlighten-mode) (cider-mode) (cider-popup-buffer-mode) (cl-old-struct-compat-mode) (compilation-minor-mode) (compilation-shell-minor-mode) (completion-in-region-mode) (csv-align-mode) (cua-mode) (datum-comments-mode) (defining-kbd-macro) (desktop-save-mode) (diff-auto-refine-mode) (diff-minor-mode) (dired-hide-details-mode) (doc-view-minor-mode) (doc-view-presentation-mode) (electric-layout-mode) (electric-quote-mode) (elpy-django) (elpy-mode) (eshell-arg-mode) (eshell-command-mode) (eshell-proc-mode) (eshell-var-mode) (eval-sexp-fu-flash-mode) (evil-leader-mode) (evil-magit-toggle-text-minor-mode) (evil-org-mode) (evil-paredit-mode) (flycheck-mode) (flymake-mode) (flyspell-mode) (git-commit-mode) (global-auto-complete-mode) (global-auto-revert-mode) (global-evil-leader-mode) (global-flycheck-mode) (global-hl-line-mode) (global-hl-todo-mode) (global-linum-mode) (global-prettify-symbols-mode) (global-reveal-mode) (gnuplot-context-sensitive-mode) (gnus-dead-summary-mode) (gnus-undo-mode) (goto-address-mode) (goto-address-prog-mode) (helm--minor-mode) (helm--remap-mouse-mode) (helm-autoresize-mode) (helm-display-line-numbers-mode) (helm-migemo-mode) (helm-popup-tip-mode) (hl-line-mode) (horizontal-scroll-bar-mode) (html-autoview-mode) (ido-everywhere) (image-minor-mode) (isearch-mode) (ispell-minor-mode) (jit-lock-debug-mode) (julia-repl-mode) (linum-mode) (macrostep-mode) (magit-blame-mode) (magit-blame-read-only-mode) (magit-blob-mode) (magit-wip-after-apply-mode) (magit-wip-after-save-local-mode) (magit-wip-after-save-mode) (magit-wip-before-change-mode) (magit-wip-initial-backup-mode) (magit-wip-mode) (mail-abbrevs-mode) (markdown-live-preview-mode) (mml-mode) (next-error-follow-minor-mode) (ob-ipython-mode) (org-capture-mode) (org-cdlatex-mode) (org-src-mode) (org-table-follow-field-mode) (orgtbl-mode) (outline-minor-mode) (overwrite-mode) (paragraph-indent-minor-mode) (paredit-mode) (prettify-symbols-mode) (pycoverage-mode) (pyvenv-mode) (pyvenv-tracking-mode) (racket-smart-open-bracket-mode) (racket-xp-mode) (rainbow-delimiters-org-mode) (recentf-mode) (rectangle-mark-mode) (reftex-mode) (reveal-mode) (server-mode) (sgml-electric-tag-pair-mode) (sh-electric-here-document-mode) (shell-command-with-editor-mode) (size-indication-mode) (slime-autodoc-mode) (slime-edit-value-mode) (slime-editing-mode) (slime-fuzzy-target-buffer-completions-mode) (slime-macroexpansion-minor-mode) (slime-mode) (slime-popup-buffer-mode) (slime-repl-map-mode) (slime-repl-read-mode) (slime-trace-dialog-autofollow-mode) (slime-trace-dialog-hide-details-mode) (slime-trace-dialog-minor-mode) (smerge-mode) (tab-bar-history-mode) (tab-bar-mode) (table-fixed-width-mode) (table-mode-indicator) (temp-buffer-resize-mode) (text-scale-mode) (tool-bar-mode) (transient-resume-mode) (undo-tree-visualizer-selection-mode) (unify-8859-on-decoding-mode) (unify-8859-on-encoding-mode) (use-hard-newlines) (vc-parent-buffer) (vdiff-3way-mode) (vdiff-mode) (vdiff-scroll-lock-mode) (view-mode) (visible-mode) (vterm-copy-mode) (window-divider-mode) (with-editor-mode) (xref-etags-mode)) ```

[greghendershott]

Thanks for the report and for attempting to figure out a fix!

I'll take a look, too.

[Digression: As it happens, I've been spending the last few weeks experimenting with using a #lang supplied 'color-lexer, including for #lang racket/base. This is probably the only way to get 100% correct handling of everything all the time. The "pile of regexps" approach, tangled with Emacs' own char syntax propertization rules for strings and comments, seems vulnerable to situations like your report -- where you push down a bump in the carpet to fix bug A... and it just pops up a bump elsewhere as bug B. OTOH using a lang lexer has its own challenges, e.g. performance when typing fast and/or on large files.]

Anyway it might be possible to fix this bug w/o introducing another bug, and I'll try.

[tgbugs]

re: digression: I was going to add a comment at the end of my initial report speculating that a proper parser might be the only way to get this to work correctly. One possibility might be to have a pass that only did the parts that were troublesome, I tried to look into how the lisp-mode highlighting dealt with this, but couldn't grok what it was doing in the time allotted, but I couldn't spot the normal propertization function, so they seem to be doing something else. Another possibility that crossed my mind was https://github.com/ubolonton/emacs-tree-sitter, though requiring loadable module support is another giant can of worms, especially if it leads to having to maintain multiple implementations for syntax propertization/font locking.