Speed up howm-menu and howm-list-recent for large note collections

[bitic]

(Prepared with the help of Codex CLI because I lack the technical depth to analyze the issue thoroughly myself. Tell me if using an LLM is not correct behaviour in this forum)

Summary

When working with large note collections (for example, ~25,000 notes), the commands howm-menu (howm-menu.el) and howm-list-recent (howm-mode.el) feel much slower than howm-list-todo (howm-reminder.el). In practice, opening the main menu or the list of recent notes can take several seconds (23 on my laptop), which makes day-to-day navigation rather sluggish. On the other hand, C-c , t is almost instantaneous.

Observed behaviour

• Both howm-menu and howm-list-recent call howm-folder-items howm-directory t, so every invocation enumerates the entire note tree and materializes one item structure per file. With tens of thousands of notes, that is expensive.
• Immediately afterwards, howm-filter-items-by-mtime loops over those items and asks file-attributes for each one, so we end up hitting the filesystem tens of thousands of times per refresh.
• howm-recent-menu sorts the whole set of notes by modification time before taking the first N entries, which means the full sort cost scales with the total collection size.
• By contrast, howm-list-todo delegates filtering to the grep layer (howm-reminder-search → howm-view-search-folder-items), which limits the search to the matching files and leverages grep’s output to grab titles/context. That pathway is noticeably faster.

Proposed direction

• Introduce a recent-items cache (e.g., howm-recent-cache) that stores the mtime (and optionally the first line) for each note. Populate it once, then keep it up to date incrementally.
• Update the cache from existing hooks (howm-after-save, note creation/deletion) so we do not need to rescan the entire tree each time.
• Rewrite howm-menu-recent and howm-list-recent to consume the cache: pick the top N recent files from the cached metadata, then reuse the current howm-view-search-items flow (which already benefits from grep) just on those candidates.
• Optionally persist the cache to disk so Emacs sessions start fast, and provide a manual rebuild command.

This would mirror the efficient approach used by howm-list-todo, making the main menu and recent list responsive again even for very large collections.

Thanks for considering it!

[kaorahi]

(Off-topic)

Your post was a great trigger! It prompted me to profile the code, and I found howm-list-recent runs 2x faster after rewriting a function. The bottleneck was the repeated calls to the slow howm-normalize-file-name through howm-subdirectory-p.

I'll test this new code for a while before pushing. To try it right away, add the following code after howm is loaded.

(defun howm-files-in-directory (path)
  "Return a list of files under PATH if PATH is a directory, searching
subdirectories recursively.  If PATH is a file, return a single-element
list containing it.  Files and subdirectories matching
`howm-exclude-p' are ignored, but PATH itself is always included.
This is because some users like to store their notes in ~/.howm but
still ignore the dot files inside it."
  (let ((full-path (expand-file-name path))
        (include-p (lambda (d) (not (howm-exclude-p d)))))
    (if (file-directory-p full-path)
        (seq-remove #'howm-exclude-p
                    (directory-files-recursively full-path "" nil include-p t))
      (and (file-exists-p full-path) (list full-path)))))

[kaorahi]

(On-topic)

I only read the Summary so far. It would be useful to talk about what and why we need something before planning how to implement it. :-)

First, let me clarify the situation:

1. Could you run M-x benchmark RET (message "%s" (length (directory-files-recursively howm-directory ""))) RET and then paste the result of M-x honest-report RET? You can skip the "Screen shot" part.
2. How many files does M-x howm-list-recent list?
3. Is C-u 0 M-x howm-list-recent still slow? (At "No recent notes. Create? (y or n)", just type "n".)
4. Is the following "Side Notes" enough for you? If not, why?

(If speeding up howm-list-recent is really important, another approach could be to use an external command like find, similar to how we already use grep for searches.)

Side Notes

For the menu cache, please see the "Quick start" section in README.md.

If you set the variable howm-recent-excluded-files-regexp, you can exclude certain files from M-x howm-list-recent.

Personally, I use the following settings:

• Keep todo/schedule/recent list buffers once opened.
• Add key bindings to switch to existing todo/schedule/recent lists.

;; Give different buffer names to the todo/schedule/recent lists.
(setq howm-view-summary-name "*howmS:%s*")
(setq howm-view-contents-name "*howmC:%s*")

;; Prevent "q" key from closing todo/schedule/recent lists.
(defvar my-howm-protected-buffer-names
  '("*howmS:{todo}*" "*howmS:{schedule}*" "*howmS:*" "*howmC:*"))
(defadvice howm-view-kill-buffer (around keep-todo activate)
  (if (member (buffer-name) my-howm-protected-buffer-names)
      (bury-buffer)
    ad-do-it))

;; Key bindings to open the existing todo/schedule/recent list.
(global-set-key "\C-c,!" (lambda () (interactive) (switch-to-buffer "*howmS:{todo}*")))
(global-set-key "\C-c,@" (lambda () (interactive) (switch-to-buffer "*howmS:{schedule}*")))
(global-set-key "\C-c,_" (lambda () (interactive) (switch-to-buffer "*howmS:*")))

[kaorahi]

related? #84

[bitic]

Yes, It is related.