Improve handling of remote/TRAMP-related buffers (e.g. remote Dired)

[indigoviolet]

Thanks for bufler, it's now my daily-driver buffer menu.

Perhaps related to #41, if I have a remote dired buffer, ~C-x b~ [edit: C-x C-b, bufler] seems to do some tramp-related stuff which is either slow or impossible based on my remote login status. Can this be configured off?

[alphapapa]

Thanks, I'm glad it's useful to you.

What do you mean by C-x b?

[indigoviolet]

Sorry, I meant bufler, bound to C-x C-b

[alphapapa]

Ok, well, Bufler doesn't do anything specific with regard to TRAMP. You can disable the option mentioned in the issue you linked (which is disabled by default, anyway). If that doesn't help, then I don't know what's causing the slowdown.

I'd suggest removing columns from the Bufler list one-by-one and see if you can find one that causes the slowdown. If one of them does, maybe we can work around the problem in it. If none of them seem to make a difference, then I'd suggest closing all buffers except a Dired-TRAMP buffer that causes the problem, then using the Emacs profiler to see what takes the longest when refreshing the Bufler list.

[indigoviolet]

Thanks for the guidance.

For context, my test was as follows:

1. log into a remote server
2. Open a remote dired buffer
3. Open bufler and switch buffers
4. turn off the remote server --> You can also just kill the `*tramp*` buffer
5. Open bufler again --> Here we see the "Opening connection" etc prompts from Tramp.

If the remote server is turned off, step 5 will hang. If you kill the remote dired buffer, step 5 will be responsive again.

Removing column names did not help; the tramp prompts showed even with just "Path" turned on.

Profiling showed the following (pointing to bufler-group-auto-project as the origin of the tramp stuff):

     340  92% - ...                                                
     294  80%  - bufler-group-auto-project                         
     294  80%   - project-current                                  
     294  80%    - let                                             
     294  80%     - project--find-in-directory                     
     294  80%      - run-hook-with-args-until-success              
     294  80%       - project-try-vc                               
     294  80%        - let*                                        
     294  80%         - condition-case                             
     294  80%          - progn                                     
     294  80%           - vc-responsible-backend                   
     282  77%            - file-directory-p                        
     282  77%             - tramp-file-name-handler                
     282  77%              - apply                                 
     282  77%               - tramp-sh-file-name-handler           
     282  77%                - apply                               
     282  77%                 - tramp-sh-handle-file-directory-p   
     282  77%                  - tramp-run-test                    
     282  77%                   - tramp-send-command-and-check     
     282  77%                    - tramp-send-command              
     281  76%                     + tramp-maybe-open-connection    
       1   0%                     + tramp-get-connection-process   
      12   3%            - mapcar                                  
      12   3%             + #<compiled -0x1960a484f730a096>        

        ...

Note that my bufler-vc- variables are all set to nil:

[alphapapa]

Thanks for investigating! That's exactly what was needed.

I'm not sure what is the best way to handle this. Here are a few initial ideas:

1. Similar to the vc-related fix, use that option to control whether remote buffers are allowed to be grouped by auto-project. That would mean that, when the option is disabled, remote Dired buffers wouldn't be grouped properly, but it would be better than an indefinite delay.
2. Group remote buffers separately, earlier in the default groups, and don't do any additional grouping on them. i.e. have an auto-remote group.
3. Combine ideas 1 and 2 by having the auto-remote group do nothing unless the aforementioned option is disabled.
4. Try to implement some kind of caching for auto-project. I'd strongly prefer not to do that.
5. Try to improve project upstream to do some kind of caching or handle remote buffers smartly. I'm probably not interested in doing that myself, but it would be a good thing to do, anyway. I hardly use TRAMP, myself, so regardless of what we do here, I'd suggest that you bring up this issue on emacs-devel, if you have the energy to deal with that conversation. :)

In the meantime, as a workaround, you could, of course, remove auto-project from your groups. Or you could copy its definition and add a file-remote-p check as the first step, or something like that.

What do you think? Thanks.

[indigoviolet]

I like the idea of the auto-remote group (2), and I will try your suggestions for workarounds in the meantime.

Have you considered the idea of caching all buffers' positions in the tree? I know you didn't want to do caching for auto-project specifically, but perhaps this version is more palatable. This scenario would be like having an auto-cache group at the top that returned the cached position except for new buffers; you could then have Refresh clear this cache (or even have two separate actions Refresh local and Refresh all including remote).

[alphapapa]

Bufler already has caches, including for the computed groups: see https://github.com/alphapapa/bufler.el/blob/097f4349920215bdd829fceabc1afdbba172c32a/bufler.el#L97. The cache is cleared when the user uses a prefix argument to bufler-list or when the sxhash of buffer-list changes between calls to bufler-buffers.

you could then have Refresh clear this cache (or even have two separate actions Refresh local and Refresh all including remote).

That's an interesting idea, but IIUC that would entail a significant amount of work to either invert the buffer groups' data structure, or special-case certain buffers and insert them into groups in an additional step after the other buffers have been grouped. I generally don't think it would be worth the complexity that would entail. But it might be something to consider at a later stage in the project's development, e.g. after the grouping library has matured and been used in some other projects I have in mind.

[skeezix]

Adding some color (without the detailed debugging done above.. nice!); new Bufler user here

Invoke bufler and..

• if all local buffers, it comes up instantly
• if some Tramp buffers are open, Bufler 'hangs' for awhile depending on number of remote buffers ; a single Tramp buffer open can suddenly make it take 5 or 10s for Bufler to come up

list-buffers -> instant ibuffer -> slowed by Tramp buffers, but not so much as bufler bufler -> large delay

Sorry, I didn't profile to see where the delay is coming from; presumably Bufler is just asking for some details about the file, and then Tramp is waking up and doing remote calls to get that info.

I had a similar problem in Marginalia during completion, where a Tramp buffer being open completely killed completion; in the end, I think Marginalia just checked for 'is it a remote file?' and if so, made wild assumptions (don't check file size, permissions, etc, just say 'remote file' or something.)

So maybe less can be done by Bufler for a remote file, to limit slowdown

Of note: Bufler caches, so if you repeatly invoke 'bufler' to go back to an existing window, the 'hang' is on first hit, and lessened later (didn't explore this much.)

[WorldsEndless]

I have had similar problems (with ibuffer as well) involving hanging when tramp has gone inactive. For a long time my work-around is to close Tramp-using buffers aggressively when I'm done, so they don't fall asleep. If I want a persisting ssh login, I use vterm and not M-x shell. Finally, something I did in both ibuffer and now bufler, was bind a key to tramp-cleanup-all-buffers so I can regularly close connections that I know would kill me when they fall asleep on their server.

I viewed this as a tramp issue more than a bufler one.