File watches are created over-zealously

[hpdeifel]

It seems that lsp-mode now creates file-watches for all the things in the projects root directory, which uses up resources and isn't always needed. For example, rls registers delete/change/create watchers explicitly only for Cargo.lock, Cargo.toml plus a deletion watcher for the target/ directory. Instead of creating only three file watches, lsp-mode decides to recursively watch files and directories from e.g. the target/ directory, which leads to errors like:

directory-files-recursively: Opening directory: No such file or directory, /somewhere/target/rls/debug/deps/rmeta31dHOO

Please correct me if I'm wrong, I haven't yet understood the file-watcher code in its entirety.

[yyoncho]

I agree that it is not the optimal solution, but at this point, this is the simplest solution having in mind that the language servers that are running in the folder are not defined and it is not trivial to deduplicate the paths. e. g. jdt-ls and boot-ls combined run watch 10+ file patterns and most of them are like **/*.java src/main/**. Also, they could unregister/register a file watch at any point. I agree that we might think for some optimizations, e. g. if the paths do not have wildcards - watch only them.

There is a similar issue for vscode: https://github.com/Microsoft/vscode/issues/45295

directory-files-recursively: Opening directory: No such file or directory, /somewhere/target/rls/debug/deps/rmeta31dHOO

This is a bug that has to be fixed. You may ignore the target folder via lsp-file-watch-ignored for now.

[hpdeifel]

I agree that creating only the required watches is a hard problem.

This is a bug that has to be fixed. You may ignore the target folder via lsp-file-watch-ignored for now.

Does that also ignore deletion events for the folder itself? Because that would be one of the three things that rls explicitly watches.

[yyoncho]

Does that also ignore deletion events for the folder itself?

The deletion events comes from the parent folder so it is fine to ignore the folder itself. I will fix the deletion event today, so you may ignore it.

[yyoncho]

I have pushed https://github.com/emacs-lsp/lsp-mode/commit/1f7e807d27e6b3e41f3a51b3d509a555d2692b34 which fixes the directory-files-recursively issue and I have documented the "Limitations" in the README page. I am not planning to address the issue related to optimizing the watches since it will become irrelevant if, for example, RLS starts to monitor **.rs files.

[uqix]

From README:

In case your project contains a lot of files you might want to disable file monitoring via lsp-enable-file-watchers (you may use dir-locals).

Is dir-locals for project root or any dir in the project?

[yyoncho]

@uqix the dir local solution will be per project. But you may use lsp-file-watch-ignored to skip a particular folder from being monitored.

[axelson]

Is there a chance for this to be re-visited? Or is there any way to work around this limitation from a language server implementation side?

[yyoncho]

Is there a chance for this to be re-visited? Or is there any way to work around this limitation from a language server implementation side?

No.

I am looking at https://github.com/elixir-lsp/elixir-ls/issues/448 and given the fact that you are interested in **/*.<ext> there is even no room for optimization in elisp since we should watch everything...

In general, 1000 threshold limit is very aggressive. I think that emacs will perform fine after the initial watch establishment even with 10k files to watch.

[jturner314]

As an idea, it may be helpful to add a hook so that users can more easily customize which files to ignore (with more sophisticated logic than just regexps). Personally, I'd like to ignore everything according to .gitignore (via git-check-ignore) except for Cargo.lock.

[aldanor]

@jturner314 That would be the cleanest default, indeed - anything that's already in your .gitignore has to no reason to be watched.

[nbfalcon]

@aldanor how about source code that is generated as part of the build process and can as such change any time? However, this might be more of an edge-case, and so if this were customizable, that should be good enough.

[aldanor]

@nbfalcon As you've said it's more of an edge case - it would be nice if you could prevent ignoring (or rather force-watching folders or files), bonus points if it can also be done on a per-project basis.

As for .gitignore, here's another simple example - the target/ folder will be present in most rust projects (and it doesn't have to be at project root, technically) and it will almost certainly be in .gitignore but in other projects there may be a valid target/ folder that has to be watched.

Cargo.lock may be another exception that will often be in .gitignore but would have to be watched.

[clayrisser]

Is this related to the fact that emacs keeps hanging on me with he message ". . . so watching the Repo may slow emacs down. Do you want to watch all files in . . ."?

[yyoncho]

Is this related to the fact that emacs keeps hanging on me with he message ". . . so watching the Repo may slow emacs down. Do you want to watch all files in . . ."?

@clayrisser here you are supposed to press y or n

[borkdude]

Personally, I'd like to ignore everything according to .gitignore

+1

[ericdallo]

@yyoncho, what about a defcustom to make lsp-file-watch-ignored follow .gitignore?

[borkdude]

The reverse would also work for me: watch only files that are part of the git index (possibly using git itself for detecting the relevant directories).

[ordnungswidrig]

The reverse would also work for me: watch only files that are part of the git index (possibly using git itself for detecting the relevant directories).

What about all the freshly created file you did not commit yet?

[aldanor]

What about all the freshly created file you did not commit yet?

Yep, indeed. I'd rather just have it follow .gitignore by default (and maybe have some option of un-ignoring certain folders that may have code generated by the build process; although that would be a rare case; .gitignore path should work out of the box most of the time for most projects).

[Dieken]

Is this related to the fact that emacs keeps hanging on me with he message ". . . so watching the Repo may slow emacs down. Do you want to watch all files in . . ."?

@clayrisser No idea why lsp-mode picks some parent directory as workspace root directory, that will make lsp-mode traverse a lot of files.

https://github.com/emacs-lsp/lsp-mode/blob/2d1732a3b9bc27c1eeb4e69805bb6a8527ae7893/lsp-mode.el

Functions lsp--directory-files-recursively and lsp-watch-root-folder should exit early when number of files is greater than lsp-file-watch-threshold. At least lsp--directory-files-recursively really should exit early.

@yyoncho

[yyoncho]

Functions lsp--directory-files-recursively and lsp-watch-root-folder should exit early when number of files is greater than lsp-file-watch-threshold. At least lsp--directory-files-recursively really should exit early.

We could do that if we change the wording in the warning dialog.

As a side note, I have a prototype utilizing emacs threads which avoids the hanging.

[bhipple]

Every time I open a python file in any git repo, emacs hangs for ~20 seconds while I mash C-g, then I try to open the file a second time and am greeted with a y/n prompt that adding file watches to everything in my home dir will be slow. I've fixed it on my end by adding this:

(with-eval-after-load 'lsp-mode (add-to-list 'lsp-file-watch-ignored-directories (getenv "HOME")))

I'm wondering if watching $HOME is the intended behavior, or if it's a bug in either lsp-mode or spacemacs, or if it's something weird that only I'm seeing? My git clones don't have symlinks back to $HOME so I'm not sure how it "escaped".

[yyoncho]

I'm wondering if watching $HOME is the intended behavior, or if it's a bug in either lsp-mode or spacemacs,

most likely you have incidentally added the home folder as project root. Use M-x lsp-describe-session to check what are your project roots.

[Dieken]

yes, projectile or some other elisp plugins may set wrong project root.

I have added (setq lsp-enable-file-watchers nil) to my emacs config, haven't noticed any severe impact on user experience.

[Anton-Latukha]

Spitballing:

It is not widely known, Linux kernel & filesystems for many years ship the inotify watcher system with a directory mode, there is no need to create trigger on file basis - filesystems can be asked to notify when there would be some changes inside the directory: old system was dnotify, then now it is inotify, LWN docs on it: Part 1, Part 2, ArchWiki section, Gentoo Wiki had me to mention inotify is Emacs USE flag since 24.4, dk if it is default one (probably is or should be on Linux), and lsp-mode most probably indirectly leverages it.

[Nathan-Furnal]

I don't understand what's in lsp-file-watched-ignored-directories ? I want to ignore my home folder my/home/ because anytime I start a random file without a project, lsp freezes Emacs and asks if I want to watch my entire home directory. It's never the case and I don't know how to stop it? @yyoncho

[yyoncho]

@Nathan-Furnal check whether you have accidentally added your home dir as a root folder. Do M-x lsp-describe-session and then use M-x lsp-workspace-folders-remove.

FWIW I will change the code that counts the file in workspace folders to stop when it hits the file limit.

[Nathan-Furnal]

Thanks that worked for me! Is there any way i can tell lsp to use the current folder when there's no project? I don't write files in my home folder and that'd be a useful alternative when i just want to write a short file.

[yyoncho]

@Nathan-Furnal press I when you are asked to select project root.

[Nathan-Furnal]

Fair enough, I didn't know it was that obvious. Thanks for taking the time!

[mjkramer]

For now, here's some advice for ignoring anything in .gitignore. At some point I'll try to submit a PR with per-workspace variables for toggling this behavior and manually un-ignoring specific directories.

(defun ++git-ignore-p (path)
  (let* (; trailing / breaks git check-ignore if path is a symlink:
         (path (directory-file-name path))
         (default-directory (file-name-directory path))
         (relpath (file-name-nondirectory path))
         (cmd (format "git check-ignore '%s'" relpath))
         (status (call-process-shell-command cmd)))
    (eq status 0)))

(defun ++lsp--path-is-watchable-directory-a
    (fn path dir ignored-directories)
  (and (not (++git-ignore-p (f-join dir path)))
       (funcall fn path dir ignored-directories)))

(advice-add 'lsp--path-is-watchable-directory
            :around #'++lsp--path-is-watchable-directory-a)

[dschrempf]

Has there been any work on this? Whenever I run computations which create a lot of output within a project, Emacs becomes unusable. The profiler report shows that lsp--folder-watch-callback is to blame :).

[yyoncho]

@dschrempf for now the plan is to move the establishment of the watches in a separate thread and use thread-yield. Thus we will have a responsive UI with the same number of file watches.

[jdelStrother]

For now, here's some advice for ignoring anything in .gitignore. At some point I'll try to submit a PR with per-workspace variables for toggling this behavior and manually un-ignoring specific directories.

(defun ++git-ignore-p (path)
  (let* (; trailing / breaks git check-ignore if path is a symlink:
         (path (directory-file-name path))
         (default-directory (file-name-directory path))
         (relpath (file-name-nondirectory path))
         (cmd (format "git check-ignore '%s'" relpath))
         (status (call-process-shell-command cmd)))
    (eq status 0)))

(defun ++lsp--path-is-watchable-directory-a
    (fn path dir ignored-directories)
  (and (not (++git-ignore-p (f-join dir path)))
       (funcall fn path dir ignored-directories)))

(advice-add 'lsp--path-is-watchable-directory
            :around #'++lsp--path-is-watchable-directory-a)

Nice idea, though it does make lsp take forever to start in my project, as it seems to run git check-ignore hundreds/thousands of times.