Scimax issue with tramp on shell for results >20 lines

[shrysr]

Vanilla Scimax appears to have some kind process that seems to interfere with the response received over shell (M-x shell) over Tramp. There are no helpful error messages. I've also tried setting some tramp and ssh options like chunk size and default ssh protocol.

In particular, this occurs when one tries to cat more than ~20 lines from any file, or basically - any command (including psql) result over ~20 lines.

Step wise description:

1. Start a tramp connection and open and org file on the remote machine.
2. C-u M-x shell (and use /bin/sh and specify the tramp directory).
3. Run commands.
4. If result is > ~15-20 lines, the process will hang. Hitting C-g repeatedly will eventually terminate the process, and spit most (if not all the results). However, the longer the result, the more times one has to hit C-g.

I have also tried the above with no other processes or buffers open. Atleast the buffer-list indicated so regarding the processes.

Note : for results <20 lines of code : this problem is not encountered. However, I would say, there is a slight sluggishness in response.

OTOH the above process followed on: emacs -Q results in breathtakingly fast results, even for 100+ lines. THe responsiveness is actually awesome. Also - this works without any tramp chunk size, compression size adjustments.

Steps being taken to resolve:

This veers a little off-course and is somewhat connected to your idea in #344, but here goes:

The above has forced me to re-think my approach. I've spent a lot of time in trying to extract portions of Scimax that I actually use (which apparently seem to be more than I realised) using the straight.el package which is both wonderful and annoying. I'm not entirely sure whether straight.el does what I think it does - but essentially - it builds (byte compiles) packages pulled from repos and symlinks all the dependencies to the repo and provides a scope to hack on them with improved management. It does come with caveats. For example the build process of ob-ipython does not symlink client.py.

I hope to come up eventually with some kind of blue print and share that. The primary driving force, is to somehow reach an Emacs that is as responsive as it can be, while still using all I can of scimax, and maintaining a strong(er) control over the config and process with the ability to freeze packages and hack on them.

I may still be wrong - but I believe the following packages / scripts are not part of the problem. IIRC : adding scimax to the load path by itself triggers this tramp problem and led me to straight.el and extraction.

1. packages.el (or atleast most of it)
2. org-ref
3. scimax-org (and the functions within it)
4. scimax-journal
5. scimax-notebook
6. scimax-hydras
7. scimax-yas
8. scimax-ivy
9. ob-ipython upstream (including scimax-org-babel-ipython-upstream and scimax-ob)

I have in general got ipython working, however, there are still issues with the overlays and the little kernel reference notifications, and it is far from complete.

The config reached so far is temporarily at https://github.com/shrysr/.emacs.d, and a patch for scimax is attached here. Some changes in the patch are arbitrary, but still - they are not yet substantial and mostly concern calling org-plus-contrib and org and using straight-use-package.

Another substantial target: I've usually had trouble building scimax and connecting to MELPA everytime I've setup Scimax on a new Linux machine. Doom Emacs started up pretty much flawlessly the first time on the same machine in this occassion. I hope with this straight.el based configuration, I can achieve something similar.

Init time so far, without any garbage collection and 100 blocks of code and a reasonable bit of scimax is 7 seconds. i.e there is potential to reduce it.

The tramp issue has not yet rared it's head with my current config.

I'm happy for feedback and advice on the above.

[jkitchin]

I don't know what could cause this, but there are a few other funny things that happen in scimax for me. The way to track this down is to bisect the scimax initialization, or run each bit manually until you see where it causes this. It is probably some package, as I don't think I do anything to cause it intentionally.

I don't have the bandwidth to track it down now, but lets leave this open for a future time when we can.

[jkitchin]

I was able to reproduce your specific example above. I think it is caused by:

(use-package undo-tree
  :diminish undo-tree-mode
  :config (global-undo-tree-mode))

if you comment that out in packages.el, do you get good performance again?

[shrysr]

I will try this with vanilla scimax and revert. However - in my alternate config, I do have undo tree enabled globally (without diminish, though I guess that makes no difference), and this has not resulted in the same problem.

[jkitchin]

I got to that conclusion basically by running the init files manually, line by line, loading one package at a time. tramp seemed fine up to that point for me, but maybe it is another package.

[shrysr]

I just tried a rebuild of the latest version of scimax, and before launching emacs -q -l ~/scimax/init.el. (and with no user files) - I commented out the undo tree snippet in packages.el. I'm afraid the behavior was exactly the same. I am now using Emacs 27 (as opposed to 26.3 when I posted the issue) and on the same Mac OS.

[shrysr]

As a separate note, I've just pushed the updated alternate config that I've been using since raising this issue ( https://github.com/shrysr/.emacs.d). I am using a limited version of Scimax, (i.e I added scimax packages and config one by one rather than start with a fork - https://git.sr.ht/~shrysr/scimax-subset). I will have to try consolidate and report.

Other than the overlays of Latex rendering (which I would like to incorporate soon) - I think I do have a decent chunk of Scimax stuff in my config though including the excellent ipython enhancements that I'm using everyday. The Tramp issue has not occurred so far. It may be worth noting that the startup time (with some basic gc optimisation) is at ~7.7 seconds of which 2.7 s go to pdf-tools).