bbatsov
Projectile makes new file creation extremely slow
Expected behavior
New file creation (or opening of a "new to projectile" file) is similar to vanilla emacs (virtually instantaneous).
Actual behavior
New file creation (and opening of files that are "new to projectile") takes several seconds.
Steps to reproduce the problem
Create a new file within a projectile directory, or open a file generated by another program that projectile hasn't indexed.
Profile of Opening the File
- command-execute 3852 93%
- call-interactively 3852 93%
- funcall-interactively 3852 93%
- counsel-find-file 3787 92%
- counsel--find-file-1 3787 92%
- ivy-read 3787 92%
- ivy-call 3760 91%
- counsel-find-file-action 3760 91%
- find-file 3760 91%
- find-file-noselect 3760 91%
- find-file-noselect-1 3759 91%
- after-find-file 3758 91%
- run-hooks 3749 91%
- projectile-find-file-hook-function 3745 91%
- projectile-cache-files-find-file-hook 3742 91%
- projectile-cache-current-file 3742 91%
- projectile-serialize-cache 3733 91%
- projectile-serialize 3684 89%
- write-region 149 3%
+ select-safe-coding-system 149 3%
Environment & Version information
Projectile version: 2.6.0-snapshot
Emacs: 27.2
Operating System: Fedora 35
I haven't noticed such a slowdown myself so I assume it happens only in certain cases (e.g. in your stacktrace it seems you have caching enabled; if you're cache is big serializing it is quite expensive). You can just use remove-hook to remove the problematic function from find-file-hook.
Upon examination, it looks like my cache has grown to nearly 200MB. Maybe this would be nightmarish, but perhaps each project should have its own file/file set?
My rational is I don't really use the global projectile search, I just jump to the project, and search there. Most of my projects are small, but I have a lot of projects checked out, and a few very large ones.
Other ideas:
- Debounce the writes (that way a huge file can't end up being written for every file opened that isn't in cache)
- Use quick append writes to a "catch up log", upon startup if there's a catch up file, merge it into the in memory cache to create a the fast in memory cache, and update the on disk cache. (I like this the best from a perspective of keeping the editor as responsive as possible, despite cache sizes)
Example large projects:
- llvm-project/llvm
Even my "core" projects are enough to start to notice the performance hit with a roughly 6MB file (~150k files).
Yeah, I agree that the global cache was a mistake, but I haven't had the time to make it project local (e.g. a projectile.cache file in the root of each project). Perhaps someone would be me to this eventually - shouldn't be a complex change, as the global search doesn't really depend on having a global cache.
Sleeping on this I'm not sure that per-project caches is enough. With projects like LLVM there are 100k+ files, writing a cache with 100k+ entries everytime a new file is added/opened isn't great.
Having some kind of append log, and then merging that into the startup (after so long idle?), seems like not only the better thing for performance, but for the disk health.