`evil-search` fails to obey search-invisible in `org-mode` buffers in recent versions of Org

[hlissner]

Issue type

• Bug report

Environment

Emacs version: tested in 28.1 and 29.0.50 Operating System: NixOS 22.11.20220530.f1c1676 Evil version: 1.15.0 Evil installation type: Straight (but tested with manual install too) Graphical/Terminal: Graphical Tested in a make emacs session (see CONTRIBUTING.md): No (requires org; see sandbox script below)

Reproduction steps

• Set up evil, org, and test files:

  # Create test directory
  mkdir ~/eviltest/ && cd eviltest
  
  # Setup dependencies
  git clone https://github.com/emacs-evil/evil
  git clone https://github.com/emacs-straight/org-mode   # faster mirror
  cd org-mode
  git checkout e9da29b6fafe63abbc2774e9d485ac13d2811b65
  make
  
  # Set up test files
  cat << EOF > test.org
  #+STARTUP: fold
  
  * This is a test
  ** This is a test
  ** This is a test
  ** This is a test
  ** This is a test
  * This is a test
  EOF
  
  cat << EOF > test.el
    (require 'evil)
    (require 'org)
    (evil-mode +1)
    (evil-select-search-module 'evil-search-module 'evil-search)
    (setq search-invisible 'open)
  EOF
  
  # Launch Emacs
  emacs -Q -L evil -L evil/lib -L org-mode/lisp -l ./test.el test.org
  

• Observe that headings are folded.
• Invoke evil-search (press /) and search for test.
• Use n/N to repeat the search several times.
• Observe that the (invisible) matches in the folded regions are ignored.

Expected behavior

evil-search should respect a non-nil value of search-invisible and jump to the matches within. If search-invisible is set to 'open (its default value on 28.1), it should open the folded region and close it upon leaving it.

Actual behavior

evil-search skips folded regions, regardless of the value of search-invisible, with only a slight variation of cursor positioning depending on search-invisible's value:

If search-invisible == 'open or nil:

(The {N} markers denote where the cursor stops on each invokation of n/N, and in what order)

* This is a {1}test [...]
* This is a {2}test

Else if search-invisible == t:

* This is a {1}test {2}[...]{3}
* This is a {4}test

Further notes

• In https://github.com/bzg/org-mode/commit/88f86237c504a2762f4a65e691ea8e8dc3a3757a, org-mode introduced a fast, new implementation for folding text in org-mode buffers using text properties. This library is called org-fold.
• The introduction of org-fold does not affect isearch or the isearch module for evil-search, only the evil-search module.
• This change does not affect the evil-search module if (setq org-fold-core-style 'overlays) is set, instructing org-fold to use overlays instead of text properties.

[polaris64]

Here's a discussion on the emacs-orgmode mailing list in case that helps in any way

[natejlong]

There's a lot of relevant discussion in https://github.com/minad/consult/issues/563 between Ihor Radchenko (@yantar92) and @minad on how to fix consult's identical issue.

Especially worth noting: Ihor made a 2022-08-13 commit to org-mode to add an API for other modules to use. The consult folks seemed to think would be enough for them to fix the issue on their end, perhaps it's the same here.

[yantar92]

Nate Long @.***> writes:

Especially worth noting: Ihor made a 2022-08-13 commit to org-mode to add an API for other modules to use. The consult folks seemed to think would be enough for them to fix the issue on their end, perhaps it's the same here.

I think that evil may simply respect the isearch settings used in `org-fold-core--isearch-setup', as I mentioned in @.***/. In addition to fixing evil search in Org, it will make the evil search more complaint with Emacs' isearch.

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at https://orgmode.org/. Support Org development at https://liberapay.com/org-mode, or support my work at https://liberapay.com/yantar92

[natejlong]

@yantar92 github censored the link text there with its email filter

as I mentioned in ***@***.***/.

[yantar92]

https://list.orgmode.org/87y1ymcpzp.fsf@localhost/

[lazzalazza]

Is there any progress on this issue or any possible workaround came up after october? Just switched to org 9.6 and find this rather disrupting for my workflow...

[decadent]

@lazzalazza As mentioned in the issue text, if you set org-fold-core-style to 'overlays, Org works the same as before, so Evil has no problems.

[betaprior]

Many thousands of users (including myself) find both org mode and evil to be indispensable. While I appreciate the fact that a workaround exists, it's quite unfortunate that two of the most popular emacs packages don't play well with each other out of the box.

[lazzalazza]

@decadent I know about the proposed org-fold-core-style solution, which has proven to be causing problems to many users because unfortunately it brakes org-cycle (see the somewhat related discussion on the org-roam github: https://github.com/org-roam/org-roam/issues/2198). I totally agree with @betaprior about the necessity of finding some sort of integrated or out-of-the-box solution.

[yantar92]

Andrea @.***> writes:

@decadent I know about the proposed org-fold-core-style solution, which has proven to be causing problems to many users because unfortunately it brakes org-cycle

This is because some users might be setting `org-fold-core-style' after loading Org. It must not be done, as mentioned in the variable docstring.

(see the somewhat related discussion on the org-roam github: https://github.com/org-roam/org-roam/issues/2198). I totally agree with @betaprior about the necessity of finding some sort of integrated or out-of-the-box solution.

As I mentioned above, the main issue is coming from the fact that Evil re-implements isearch functionality, but only partially. So, Org mode's settings for isearch are only in effects partially; thus creating the observed broken state.

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at https://orgmode.org/. Support Org development at https://liberapay.com/org-mode, or support my work at https://liberapay.com/yantar92

[stites]

Is (setq org-fold-core-style 'overlays) supposed to fix evil-search so that it will fold and unfold the region when search-invisible is set to 'open? I modified @hlissner 's code block to include the following test.el

  (setq org-fold-core-style 'overlays)
  (require 'evil)
  (require 'org)
  (evil-mode +1)
  (evil-select-search-module 'evil-search-module 'evil-search)
  (setq search-invisible 'open)

Now the behavior I observe is:

{2,4}* This is a {1,3,5}test [...]
* This is a test

As before, the {N} markers denote where the cursor stops on each invocation of n/N, and in what order. Both of the tests and the ellipsis [...] are highlighted, but the hidden region is not unfolded. Continuing to invoke n, I can never get to the second visible text

[yantar92]

Sam Stites @.***> writes:

As before, the {N} markers denote where the cursor stops on each invocation of n/N, and in what order. Both tests and [...]s are highlighted, but the hidden region is not unfolded.

Please update to the latest bugfix commit. We had a bug in this area recently. Fixed, but not yet on ELPA.

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at https://orgmode.org/. Support Org development at https://liberapay.com/org-mode, or support my work at https://liberapay.com/yantar92

[stites]

Thank you! It's possible it was fixed earlier. There turned out to be a bug in the script above: the test.el and test.org are placed in the org-mode dir, so the local emacs [...] -L org-mode/lisp [...] was not getting loaded correctly.

I can confirm that it works, here is an update script from above to replicate the expected behavior

# Create test directory, removing one if it already exists
rm -rf ~/eviltest
mkdir ~/eviltest/ && cd eviltest

# Setup dependencies
git clone https://github.com/emacs-evil/evil --depth 1
git clone https://git.savannah.gnu.org/git/emacs/org-mode.git --depth 1 # drop the mirror
cd org-mode
make
cd ..

# Set up test files
cat << EOF > test.org
#+STARTUP: fold

* This is a test
** This is a test
** This is a test
** This is a test
** This is a test
* This is a test
EOF

cat << EOF > test.el
  (setq org-fold-core-style 'overlays)
  (require 'evil)
  (require 'org)
  (evil-mode +1)
  (evil-select-search-module 'evil-search-module 'evil-search)
  (setq search-invisible 'open)
EOF

# Launch Emacs
emacs -Q -L evil -L evil/lib -L org-mode/lisp -l ./test.el test.org

[betaprior]

So it seems that with the latest org mode, there are two working configurations:

;; Configuration A
(setq org-fold-core-style 'overlays)
(evil-select-search-module 'evil-search-module 'evil-search)

;; Configuration B
(setq org-fold-core-style 'text-properties)
(evil-select-search-module 'evil-search-module 'isearch)

Can anyone shed light on what the trade-offs are? What is the behavioral difference between evil-search and isearch? Is there a reason to pick one configuration over the other?

[polaris64]

The reason I switched evil-search-module to evil-search instead of isearch is so that I could use operators like /e to move the point to the end of the match. e.g. running /text-to-find/e in Vim and with evil-search will leave the point on the "d" of "find" but this won't work when using isearch.

[yantar92]

I hope that I managed to fix this with an advice for isearch-clean-overlays. https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=7dee2c07f459680eacbc3afdccff2f1a0d930086

Please check.

[bobf32]

For me, in org-mode 9.6.1, even with org-fold-core-style set to overlays, neither :s/$/ foo/ nor regexp-replace RET $ RET foo appends text to just the visible line/s.