sublimehq
SSHFS long refresh waits
Summary
Very slow refresh of file tree in the sidebar when using SSHFS remote mounts. Using the Project -> Refresh Folders or right click context to refresh the tree can take anywhere from 10 seconds to a couple of minutes. Closing Sublime and re-opening, the tree is updated almost instantly.
Additional details: If making a change on the remote system (via SSH) the changes appear immediately in the Finder (OSX). Making a change locally in OSX the changes are visible immediately on remote. There is no delay in SSHFS replicating changes between remote and local and vice versa. Only Sublime seems to have issues seeing them and really only when using the "refresh" tools.
Even when using the sidebar context menus to create the file or folder the changes appear in the FS immediately, but Sublime takes forever to update the tree view, again even when requested.
Expected behavior
Updates to the tree happen in real time, or even when requested using the refresh tools provided.
Actual behavior
Long delays in updating the tree, anywhere up to a couple of minutes. Quick solution is to close the app and reopen it as that always provides the changes immediately.
Steps to reproduce
Install Sublime (build doesn't seem to matter as I've had this issue since at least V3). Install osxfuse (https://osxfuse.github.io/) Install SidebarEnhancements (pretty sure this doesn't affect anything but provides the context menus).
Create a file/folder either with context in sidebar, via ssh or locally with Finder. Ask Sublime to refresh the tree.
NOTE: Some of my trees are pretty big (full python project including virtualenv files), that said, again, on reopening Sublime the entire tree is refreshed instantly.
Environment
OSX 10.11.6 SublimeText3 build 3125 (dev channel) sshfs -V SSHFS version 2.5 (OSXFUSE SSHFS 2.5.0) OSXFUSE 3.5.2 FUSE library version: 2.9.7
Any updates on this?
Is there an option that can be provided to tell the sidebar to refresh at a faster interval?
Still an issue with updates:
macOS 10.12.4 ST3 build 3128 OpenSSH_7.4p1, LibreSSL 2.5.0 OSXFUSE 3.5.6
I've looking for a solution since 2016... Still an issue for me and I have to restart sublime every time to refresh directories.
macOS 10.13.3 ST3 build 3143 OpenSSH_7.6p1, LibreSSL 2.6.2 OSXFUSE 3.7.1
come join the pity party... :) https://forum.sublimetext.com/t/file-tree-reload/22649/20
Still experiencing this issue. The directory tree takes a long time to load, and new files created on remote don't show up in sublime.
Same here. It also seems to persist even after I quit and restart sublime, which was not the case for @volksman.
i've tried many SSHFS mount settings, toying with ST3 settings, nothing.
Today I saw this https://github.com/mutagen-io/mutagen but it's going to take time to get into it and see if it's good.
@volksman try https://mutagen.io ...it's nuts. I can't believe there's finally a real solution for remote development which is as fast as local. Cya SSHFS!
thanks! not the place to ask questions but it seems like mutagen will make local copies of the remote and I don't want any local files unless they are strictly temporary.
I have the same issues as @volksman. I make heavy use of SSHFS's ability to navigate file structures while only transferring the data that needs to be accessed.
Sublime Text is very explicitly designed to scan and cache lots of info, to make things works "instantly".
Goto File is powered by creating an in-memory representation of the entire folder tree(s) opened in the side bar, and allowing the use to filter through those with no lag by fuzzy mapping on the file paths.
The symbol index scans the contents of your entire folder tree(s) and records every location of a symbol (function name, class name, etc) so users can jump to the definition of a symbol without having to know what file it is in, or what line it is on.
The side bar keeps itself up to date, and reflects changes in the folder tree(s) by listening to filesystem events, and updating the in-memory representation of the tree.
In ST4 builds, the crawler also indexes file contents to provide auto complete suggestions based on existing code.
More or less, that major features of Sublime Text are built to heavy cache things in memory so the user can immediately navigate around. This is fundamentally opposed with the desire to "only transferring the data that needs to be accessed".
All that to say, there are likely some edge cases we can fix. However, remote FS issues are probably going to be lower priority because slow filesystems and filesystems that don't reliably send FS notifications aren't things we are every going to be able to solve ourselves.
Something like mutagen that creates a local cache and keeps things in sync seems like it is going to be more successful than trying to work around the edge cases of using a slow remote filesystem.
One other thing you could try is https://www.expandrive.com/desktop/. Mostly just to see if alternative implementations of remote access suffer from the same symptoms. I don't believe it caches an entire tree locally, however I can't imagine it can provide filesystem notifications over an SSH connection.
Comparing different client programs (such as Sublime Text versus Finder) is problematic since they have different problems to solve. Finder needs to keep the current folder refreshed, whereas Sublime Text needs to keep entire trees refreshed. Finder may fall back to polling, whereas we don't want to be constantly polling every folder in a tree.
The long refreshes would need more time to pin down. It could easily be that queued operations on the folder tree are preventing the refresh from executing quickly.
Hi @wbond thanks for the detailed response. I completely understand the issue you are explaining however:
-
As mentioned in the report, if I quit ST3 and reopen it the tree is drawn as I expect (with recent changes etc) and I assume the GotoFile cache is updated as it should be as well. This isn't an issue of the "auto-refresh" not working, that's an issue too but not the issue at hand. The issue at hand is that if I right click or use the menu item to "Refresh" (IE explicitly ask to refresh) the side bar it basically hangs for upwards of 5 minutes as it tries to refresh it. The fastest way to get a fresh tree or reflect changes is to quit ST3 and re-open it immediately and that saves me time but causes disruption to process flow.
-
I have tried using other mounting mechanism apps like ExpanDrive and one other I forget the name of now. Both used the same underlying Fuse plugin to mount the remote system and provides no additional help in this issue.
What I would like to see is that if I select a subfolder of a project and hit "Refresh" only that subfolder is scanned for changes not the entire project as I suspect that is part of the problem on large projects.
I think it would also be of benefit to have a "Rebuild" option that would do the same thing as what happens when I close ST3 and re-open it, under which I'm assuming the cache is destroyed and rebuilt rather than scanned for changes, which again under more assumptions however, is part of the issue.
If this was really just an issue of slow FS then quitting and re-opening ST3 would suffer from the same/similar delay, it does not, so I'm left to believe this is a problem with how the cache is managed.
Thanks again for your attention on this. Would love to help fix this one!
As mentioned in the report, if I quit ST3 and reopen it the tree is drawn as I expect (with recent changes etc) and I assume the GotoFile cache is updated as it should be as well. This isn't an issue of the "auto-refresh" not working, that's an issue too but not the issue at hand. The issue at hand is that if I right click or use the menu item to "Refresh" (IE explicitly ask to refresh) the side bar it basically hangs for upwards of 5 minutes as it tries to refresh it. The fastest way to get a fresh tree or reflect changes is to quit ST3 and re-open it immediately and that saves me time but causes disruption to process flow.
It may be that the queue of remote FS commands is preventing an immediate refresh.
I have tried using other mounting mechanism apps like ExpanDrive and one other I forget the name of now. Both used the same underlying Fuse plugin to mount the remote system and provides no additional help in this issue.
That leads me to believe the nature of using a non-caching remote FS is probably not going to work "well".
What I would like to see is that if I select a subfolder of a project and hit "Refresh" only that subfolder is scanned for changes not the entire project as I suspect that is part of the problem on large projects.
If the reason the subfolder isn't being refreshed is a backlogged queue, I suspect this won't help.
If this was really just an issue of slow FS then quitting and re-opening ST3 would suffer from the same/similar delay, it does not, so I'm left to believe this is a problem with how the cache is managed.
Not if the FS layer is what is backlogged.
If you could look into getting debug info as to what the FS layer is doing at a given time, so see what ST is asking for and what commands the FS layer is sending to the remote system, that would probably help pinpoint if the issue is that we are being given results and mishandling them, or if the FS layer isn't giving us an answer in a timely manner.
Not if the FS layer is what is backlogged.
Definitely know very little if anything at all about FS queues, however, if I have 1 file open for edit and ST3 isn't trying to sync, what traffic/backlogs would be present? I don't have any other apps accessing these trees (other than the OS) and from what you have told me SSHFS doesn't do update notification or things of that nature, so what else would cause a delay in there (and keeping in mind this is a significant delay...into minutes, not just a few seconds)?
If you could look into getting debug info as to what the FS layer is doing at a given time, so see what ST is asking for and what commands the FS layer is sending to the remote system, that would probably help pinpoint if the issue is that we are being given results and mishandling them, or if the FS layer isn't giving us an answer in a timely manner.
Would love to! Any idea how? :)
Have you disabled indexing? That reads the entire contents in your folder tree, which could easily cause a backlog.
I personally don’t know how the fuse SSHFS works, so I would just be using Google like anyone else.
I found the -d flag for SSHFS to provide some output, so I'll try to get some details with that.
Disabled indexing where, ST3?
Disabled indexing where, ST3?
Yes. Open the Preferences and change index_files to false.
Gotta run for a bit but I've made that change and quick switched to a large project. No real difference. However with SSHFS debug on I see that it's not doing anything other than receiving the odd DELETE/FORGET call. I'm going to let this run its course and see if it starts rendering the tree but so far I only see the top node and the loading animation.
Also just an additional note: As ST3 is having issues rendering the tree, I opened a Finder and started browsing around the same tree. Absolutely no delay or issues, opened a file or two for editing in another app, no problems.
I believe I also tested this in Atom ages ago but Atom made me sad. That said I don't believe it suffered from this issue if memory serves, which, sometimes does not.
At the very least, we are on a path to determining what is going on, so that is good!
Ok one last update....I just noticed that the LOOKUP calls that SSHFS is making is for the project I'm switching AWAY from not the project I'm switching to. So as I sit with Project2 tree not displaying (but showing the loading icon), it seems ST3 is trying to update its cache for Project1 which is no longer in scope.
So I let it run it's course and it took about 25 minutes to show me the tree for Project2. All the while in the SSHFS debug I was seeing things like this:
LOOKUP /realtive/path/to/file
getattr /realtive/path/to/file
NODEID: 67762
unique: 190, success, outsize: 160
unique: 276, opcode: FORGET (2), nodeid: 66633, insize: 48, pid: 0
FORGET 66633/1
DELETE: 66633
unique: 278, opcode: FORGET (2), nodeid: 66634, insize: 48, pid: 0
FORGET 66634/1
DELETE: 66634
unique: 96, opcode: FORGET (2), nodeid: 66635, insize: 48, pid: 0
FORGET 66635/1
DELETE: 66635
unique: 98, opcode: FORGET (2), nodeid: 66636, insize: 48, pid: 0
FORGET 66636/1
DELETE: 66636
unique: 188, opcode: FORGET (2), nodeid: 66637, insize: 48, pid: 0
FORGET 66637/1
DELETE: 66637
Sorry but the path includes my clients name so I've omitted the actual path. It is important and strange to note that the paths were ALL for Project1 not Project2 and at no time did I see anything in the SSHFS logs for Project2, so I think I'm still only seeing half the story here.
I would expect a debug log to mention listing any dirs that exist in the sidebar of Sublime Text, or else the debugging must have been started after Sublime Text, or the debug level isn't high enough.
My bad. I didn't enable SSHFS debug on the mount for Project2. Both projects are on different servers so different SSHFS mounts.
From all the debug output this still seems to me to be an ST3 issue and not a queuing issue. The fact that when switching projects the old project continues to poll the FS for hours after tells me there is something else going on here. I setup debug on both mounts and there are no errors and seemingly no delays. The OS can access all mounts and any changes as soon as they are made on the remote (no blocking).
I had noticed anecdotally in the past that the more I switched between projects the laggier it got, leading to a quit and restart to solve all problems. The reality is however that it's there after the very first switch or "open" of a project after another project was already open.
So are you suggesting that there is a thread limit within ST3 that may create a block? Otherwise I'm not sure where an FS level block or queue would come into play here.