Swiper freezes on certain search expressions with long line files

[Mihara]

I think this might be related to #227, but I can't say for sure, so I decided to open a new one.

1. I have an org file written in visual line mode, with long paragraphs.
2. I'm using swiper to search for a phrase containing a space.

The moment I enter a space, swiper freezes emacs, requiring me to killall -SIGUSR2 to get it to unstick.

I adjust my ivy-re-builders-alist to contain (swiper . ivy--regex) and the problem stops -- the two-word phrase gets found and there is no appreciable slowdown. But when, with the two-word phrase typed in, I C-o m to switch the matcher -- that hydra shows the wrong matcher to start with when you do that, but I think you knew that -- it freezes instantly.

I'm not sure where exactly is it sticking, but the trace seems to suggest it's stuck trying to highlight characters in the minibuffer that are not even on screen.

[abo-abo]

I think it's best to accept that Emacs isn't designed to handle long lines, and try to adjust to it. visual-line-mode is cute, but comes with a huge performance penalty for large files.

There was talk on emacs-devel of improving the text engine wrt long lines. But even optimistically, that will come out in Emacs 26 or so. So I suggest to use C-q more.

If you want, attach the long file; I'll look at it. But I can only try to fix swiper, I can't fix visual-line-mode.

[Mihara]

The file in question is ~80kb and contains ~650 lines. If that is to be considered "large" I wonder how small is small supposed to be. There is no problem with using the visual line mode in it per se.

The problem only turns up when swiper is attempting to search with fuzzy matching, presumably while highlighting characters in very long lines -- and it does not turn up when the matching is not fuzzy, but pure regex.

I'll see about making a minimal replication kit and get back to you.

[abo-abo]

Thanks. Also, I meant to write M-q, not C-q.

[Mihara]

Clarification: Using swiper in conjunction with flx seems to cause the problem. Using swiper without flx does not. Turning visual-line-mode on is in fact not required, the problem manifests without it.

Example file: https://gist.github.com/d2fb23f5efe232b953ea9fd44a193836

The following config (assuming emacs -Q and that the required packages are installed) demonstrates the problem:

(package-initialize)
(require 'ivy)
(require 'flx)
(setq-default ivy-use-virtual-buffers t
              ivy-count-format "(%d/%d) "
              ivy-display-style 'fancy
              ivy-re-builders-alist '((ivy-switch-buffer . ivy--regex-plus)
                                      (t . ivy--regex-fuzzy)))

(define-key ivy-mode-map (kbd "C-s") 'counsel-grep-or-swiper)

Presumably, some of these lines are extraneous still.

The use of flx is as described in this blog post.

Loading the offending org file, C-s, and starting to search for "Lorem ipsum..." progressively adding words from the beginning of the first paragraph inevitably freezes emacs after at most three words.

[unhammer]

This is not limited to swiper; I also see it with e.g. counsel-rg if and only if I use flx. Running counsel-rg in a project after forgetting to ignore node_modules is an easy way to reproduce it :-) And it's very annoying since a single C-g doesn't stop it, you have to repeatedly mash the keyboard or kill emacs :-/

error in process filter: Quit [8 times]
Quit emacsclient request [2 times]
Quit [5 times]
error in process sentinel: Quit [2 times]
Quit emacsclient request
Quit
Error in post-command-hook (ivy--exhibit): (quit)

[abo-abo]

@unhammer I was greatly annoyed by this as well. Until I found out that ripgrep supports limiting too long lines:

(setq counsel-grep-base-command
      "rg -i -M 120 --no-heading --line-number --color never '%s' %s")
(setq counsel-rg-base-command
      "rg -i -M 120 --no-heading --line-number --color never %s .")

Kill your Emacs no more!

[unhammer]

Nice :-) It's typically minified stuff with tens of thousands of characters per line that mess things up for me at least, maybe it'd be good to have a default value for -M for the ripgrep base command that's relatively high but less than thousands?

[abo-abo]

@unhammer Good idea, -M 140 is probably fine; I can't imagine why someone would want to read a tweet's worth for a single line of grep output. But I wonder if every version of ripgrep supports -M.

[unhammer]

https://github.com/BurntSushi/ripgrep/blob/e10544f8199487c2f01975d683240e0d686983bf/CHANGELOG.md#050-2017-03-12 seems to be the first version supporting -M, so less than half of rg's lifetime

[noctuid]

I have issues when visual line mode is enabled even in small files without long lines. Swiper becomes extremely slow even in python files and in dired buffers, for example (e.g. 3-15 seconds just to appear vs. instantly). I've disabled globabl visual line mode, but it would be nice if there was a version of swiper-use-visual-line that could be set by the user. Has anyone else encountered this issue?

[phoenixanimations]

@noctuid I have the same issue. I can replicate this with a new emacs -Q as well.

(package-initialize)
(require 'swiper)
(global-visual-line-mode)

If I go into a directory with 180+ files and dired-hide-details-mode and M-x swiper on I notice some small lag. You can get a lot of lag if you go into scratch do C-6000 C-o M-x swiper the culprit seems to be

- swiper                                             7,276,337  60%
 - swiper--candidates                                6,515,763  54%
  - #<compiled 0x4244b0b9>                           4,591,121  38%
   - line-move                                       4,546,769  37%
    - default-line-height                            2,983,231  24%
     - default-font-height                           2,983,231  24%
        frame-char-height                               48,900   0%
    - line-move-visual                               1,147,228   9%
     - posn-at-point                                   512,828   4%
      - eval                                           508,124   4%
         if                                            461,948   3%
         unless                                         17,952   0%
    + window-inside-pixel-edges                        272,038   2%
    swiper--line                                        20,064   0%
  + swiper-font-lock-ensure                             15,180   0%
    put-text-property                                    1,702   0%
 + swiper--ivy                                         758,462   6%

Line-move. Personally I just disable global-visual-line-mode as it seems to cause trouble in other places too unfortunately.

[abo-abo]

@phoenixanimations Thanks for the simple test case. If you use elp-instrument-function on vertical-motion, you'll see that most of the time is spent there. This is a C function that's used eventually by visual-line-mode. I don't know how to optimize it.

However, I'll now limit the use of swiper-use-visual-line to files below 400 lines. My reasoning is, since it takes 0.0002482364s to call vertical-motion, and we call it 3 times during the course of swiper--candidates, the start-up time is:

(* 0.0002482364 400 3)
;; => 0.29788368

We could do some optimizations here to reduce the number of vertical-motion calls from 1200 to 400 for a 400 line file. But that would potentially complicate our code without a lot of gain. Still, if someone wants to do it without complicating the code too much, I'd accept a PR.

Maybe someone knows of an API that's better than vertical-motion for our purposes.

[diamond-lizard]

If problem isn't with ripgrep or the other greps searching long lines, nor with them returning long lines, but with emacs displaying long lines, couldn't the long lines that these greps return be split in to shorter lines before displaying them?

[slk500]

Clarification: Using swiper in conjunction with flx seems to cause the problem. Using swiper without flx does not. Turning visual-line-mode on is in fact not required, the problem manifests without it.

Example file: https://gist.github.com/d2fb23f5efe232b953ea9fd44a193836

The following config (assuming emacs -Q and that the required packages are installed) demonstrates the problem:

(package-initialize)
(require 'ivy)
(require 'flx)
(setq-default ivy-use-virtual-buffers t
              ivy-count-format "(%d/%d) "
              ivy-display-style 'fancy
              ivy-re-builders-alist '((ivy-switch-buffer . ivy--regex-plus)
                                      (t . ivy--regex-fuzzy)))

(define-key ivy-mode-map (kbd "C-s") 'counsel-grep-or-swiper)

Presumably, some of these lines are extraneous still.

The use of flx is as described in this blog post.

Loading the offending org file, C-s, and starting to search for "Lorem ipsum..." progressively adding words from the beginning of the first paragraph inevitably freezes emacs after at most three words.

Thx. That resolve problem with freezing but it dosen't work on encrypted file. How fix that?