bbatsov
projectile-project-dirs very slow after upgrading package
Expected behavior
Jumping to next compiler error should be instant.
Actual behavior
Emacs freezes for several seconds.
Steps to reproduce the problem
Open a project with a lot of files, >420,000. Introduce a deliberate error then try to compile, then run M-x next-error.
If you profile it you should see something like this:
- command-execute 7976 86%
- call-interactively 7976 86%
- funcall-interactively 7976 86%
- next-error 7929 85%
- compilation-next-error-function 7929 85%
- apply 7924 85%
- compilation-find-file 7924 85%
- apply 7924 85%
- compilation-find-file-projectile-find-compilation-buffer 7924 85%
- projectile-current-project-dirs 7372 79%
- projectile-project-dirs 7371 79%
mapcar 7135 77%
delete-dups 113 1%
Looking at the implementation of the function it's not super surprising, it loops over every single file in the project checking if each one is a directory:
(defun projectile-project-dirs (project)
"Return a list of dirs for PROJECT."
(delete-dups
(delq nil
(mapcar #'file-name-directory
(projectile-project-files project)))))
Maybe projectile-project-files could already take care of separately returning which entries are directories? delq and delete-dups I think are also going to iterate over the entire list, it would probably be more efficient to employ a real set data structure.
Environment & Version information
Projectile version information
Not kidding here is the real output I get from the version command:
Projectile version nil
Emacs version
27.2
Operating system
Linux
version information from list-packages:
Status: Installed in ‘projectile-20211220.1144/’ (unsigned).
Version: 20211220.1144
Commit: fab2c546fdf990f010acb0755004a96510dc07db
Summary: Manage and navigate projects in Emacs easily
Requires: emacs-25.1
Required by: counsel-projectile-20211004.2003
Homepage: https://github.com/bbatsov/projectile
Keywords: project convenience
Maintainer: Bozhidar Batsov <[email protected]>
Author: Bozhidar Batsov <[email protected]>
Other versions: 20211220.1144 (melpa).
Confirmed this works around the issue:
;; without this projectile became incredibly slow when opening files
;; might make things slightly incorrect but it expires every two hours
(memoize 'projectile-project-dirs)
Using memoize.el found here: https://github.com/skeeto/emacs-memoize
Hmm, but there haven't been changes to this code in ages, so I wonder why things suddenly became slow for you. I totally agree that the current implementation is slow, but I don't think memoization is a very good idea. Perhaps we can implement this in terms of some faster external command that would return the dirs directly without the need to filter all files in Emacs Lisp?
I don't have anything setup to automatically update my packages, so it's very possible before I was running something months or even a year old.
Agreed external command would probably help, shelling out to find . -type d would probably be much faster although I assume it has to be more involved than that to take into account ignores in .projectile.
That file is ignored if you're using alien indexing anyways, so we can just make this conditional on the type of the project. Something like:
git ls-files | xargs -n 1 dirname | uniq
Should get the job done, but someone should test if it's really a lot faster.
I tried to use that on Chromium codebase. And it takes few seconds to open a single file.
Do you have any ideas how to improve that?
I see there is a caching on the list of files for the current project, maybe it make sense to create separate caching for the list of directories?
