[mu4e bug] Displaying the headers sometimes takes a lo,g time

[Chris00]

Describe the bug

Displaying the headers buffer after fetching mail sometimes takes several seconds (15 in the screenshot below):

How to Reproduce

Unfortunately I do not have a clear way to reproduce that (it does not happen systematically when fetching email).

Environment OS: Debian GNU/Linux Emacs: 27.1 mu/mu4e: commit 6fdc20ae

Checklist

• [x] you are running either the latest 1.6.x release, or a 1.8.x release (otherwise, please upgrade)
• [x] you are running mu4e without any third-party extensions (otherwise, make sure you can reproduce without those)
• [x] you have read all of the above

[djcb]

Likely, this is during indexing? If so, that's somewhat expected.

[dcolascione]

Is there anything that would be helpful in debugging this problem?

[djcb]

Well, there are two things I suppose:

1. why does your indexing take so long? After the initial indexing, it should be quite fast (a few seconds) ... do you have a enormous amount of messages, or slow hardware perhaps? Or.... there is some problem with the indexing being unnecessarily slow? (ie., try with command-line mu index))
2. now even when it takes a long time, doing other things in parallel (in mu4e) should still be quite fast; but apparently it is not. The indexing (from mu4e) is running in worker thread, (index_handler, in mu-server.cc); perhaps the main thread is being starved and/or there's a bunch of work queued waiting on the Xapian lock.

[djcb]

One thing that might help diagnosing further (and assuming journalctl here, change as needed):

• run a recent 1.9 version (>= 1.9.17 or so should do)
• run mu4e with (setq mu4e-mu-debug t)
• after using mu4e for a while, including some case where things take longer than expected, do:

$ journalctl -b _COMM="mu" | grep "sw:"
Found existing alias for "journalctl". You should use: "jc"
Jan 29 08:12:32 evergrey mu[4674]: sw: ran query 'maildir:"/inbox"'; related: yes; threads: yes; max-size: 500: finished after 473 us
Jan 29 08:12:32 evergrey mu[4674]: sw: find_handler (indexing: no): finished after 613 us
[...]
Jan 29 08:17:17 evergrey mu[4674]: sw: indexing: finished after 2,7 s
[...]
Jan 29 10:18:03 evergrey mu[26439]: sw: view_handler (indexing: no): finished after 7,3 ms
[...]
Jan 29 16:30:34 evergrey mu[103688]: sw: ran query 'flag:unread AND NOT flag:trashed'; related: yes; threads: yes; max-size: 500: finished after 283,9 ms
Jan 29 16:30:35 evergrey mu[103688]: sw: find_handler (indexing: yes): finished after 1389,6 ms

An interesting case would be when you're indexing and finding/viewing at the same time in the above you see such a case at 16:30:35, finding is quite a bit slower, but in absolute terms, 1.3s is still reasonable I think

[djcb]

Note, I've pushed a bunch of optimizations to master which make querying / showing results a lot faster. Of course, this ticket is about worst case performance.

Anyway, if this still happens: please check according the Jan 29 notes. A reindexing job shouldn't take very long in most cases, and and queries not much longer. But I can't reproduce this, so I need a bit of analysis, in particular how long re-indexing takes.

[djcb]

I tested with doing a full indexing run (about 1.5 min) and during that time, I can use mu4e with queries etc., with hardly noticeable delay.

[Chris00]

I am running it and it is noticeably faster (way < 1ms/msg). Thanks!

[djcb]

Overall, mu 1.11.x searching is quite a bit faster than 1.10; and I cannot reproduce any of the cases where it suddenly slows down; anyway, there's not much more to do here, so closing this.

If you still see slowdowns, feel free to open a new ticket, but please with the details as mentioned in the Jan 6 / Jan 29 comments here- thanks!