Symex 2.0 Release Prep

[countvajhula]

Summary of Changes

All the changes needed for a 2.0 release, including polishing the tree-sitter functionality, improvements and bug fixes.

Remaining Work

See the ✨ Kanban Board ✨

WIP

• [ ] Reduce reliance on evil (incl. eliminating evil-surround) (#118 )
• [ ] Idempotence of null operations (#98 )

Already Done

(reverse-chronological, most recently completed at the top)

• [x] Streamline basic computation model (#125 )
• [x] Add effect form to be able to specify side effects to traversals (#124 )
• [x] Distinguish UX abstraction level (#123 )
• [x] Ability to do transformations in the DSL (#104 )
• [x] More gardening 2 (#107 )
• [x] More gardening (#96 )
• [x] Trim dependencies (#94 )
• [x] Low-level UX improvements (#93 )
• [x] streamline definition of symex commands behind a common set of macros (#69)
• [x] fully deprecating hydra (#70)
• [x] Migrate old kanban board and convert the draft tickets to real issues
• [x] fix all links to point to the drym-org account instead of countvajhula

Public Domain Dedication

• [x] In contributing, I relinquish any copyright claims on my contribution and freely release it into the public domain in the simple hope that it will provide value.

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

[countvajhula]

Hey all, hope everyone's doing great. I'm getting the ball rolling on this effort for anyone who's around to work on it 😊 I'll aim to document outstanding items more thoroughly in the coming days, but I think the Kanban board from last time should still roughly be current if you want to take a crack at anything in particular. I think we made an effort last time to keep the tasks pretty bite-sized. I'll review them more closely and update them as needed soon. Get pumped for Symex 2.0! 😉 👍

@markgdawson @anonimitoraf @pepperblue @doyougnu @dcostaras @devcarbon-com

[countvajhula]

Thanks for reviewing the changes already in the branch @j-shilling ! Just to clarify, this branch is going to serve as the integration branch for pre-2.0 changes (including the WIP PRs #69 and #70 which will be merged here as opposed to master). This is partially as a way to organize for collaboration, and partially so that we can move faster as we only need to worry about testing everything once, at the end, rather than in depth for every PR.

[countvajhula]

I've migrated and updated the Kanban board (added in the PR description) and converted all tickets into actual issues on the repo, and also added any relevant user bug reports to the board. I think the tasks are all ready to be worked on. For the numerous tree-sitter features, there are no descriptions in the issues as they should function analogously to the Lisp features. Happy to answer any questions or help in any way! I'm not a huge chat user nowadays but I'll aim to hang out in #emacs IRC as another communication option in case that's easier for anyone. If you are interested in contributing, feel free to shoot me a message there!

[countvajhula]

For anyone who wants to play with the latest and greatest in this branch (there are lots of improvements already, and of course more exciting work still planned 😸 ), you could use config resembling this:

(use-package symex
  :straight
  (symex
   :type git
   :host github
   :repo "drym-org/symex.el")
   :branch "2.0-integration"
   :config (symex-initialize))

If you're interested in working on anything in particular, comment here (or find me on IRC if I happen to be there) and I'll assign the ticket to you. Help would especially be appreciated on:

• any tree-sitter item -- e.g. shift-forward/backward etc. Each of the features should be bite-sized and doable in a single sitting (and if not, that would be good feedback) cc @polaris64
• #67 -- highlight sometimes doesn't keep up with actual cursor position
• #68 -- probably this is just that shift-forward/backward hasn't been implemented
• #86 -- using the (evil) dot operator causes the scroll margins from the top/bottom of the screen to become deactivated cc @j-shilling
• #84 -- could warrant a new minor mode or two, cc @tommy-mor

[devcarbon-com]

Is the plan to replace tsc with the built-in treesit library?

[countvajhula]

@devcarbon-com That is the eventual goal, yes. It isn't currently planned to be changed as part of the 2.0 release, but I know @polaris64 has some WIP on this front so if he champions it or if it turns out to be a simple change we might include it.

[devcarbon-com]

Okay, sounds good. A precursory look seems like it will be pretty straight forward, except for maybe tsc-changed-ranges, I haven't seen a equivalent for that yet.

[countvajhula]

Hey folks! In case you've been using this branch, there were some bugs that had been introduced a little while back (e.g. in emit and capture) that were recently fixed in #123 , and there were a number of performance improvements in #124 and #125 , so you should update to the latest to get these 🙂

One of the performance improvements is in the "leap branch" feature, which now executes in O(n) where formerly it was O(nlogn). This should make it pretty seamless to use and in my experience it doesn't cause noticeable delays in most cases as it used to formerly. Some other performance improvements were in (1) making traversal execution tail-recursive, which allows it to be more memory efficient (since the in-progress results are passed into recursive calls instead of retained in the calling scope to be combined with the result after recursion). And (2) the "recursive" features like recursive tidy/indent (M-tab), recursive eval (M-e) were formerly causing some delays in execution as well, but now they are pretty zippy! In fact, when I tried recursive tidy on the master branch it actually caused Emacs to crash 😅 . Not sure if that's just me or if it's actually doing that at the moment - that would be bad, lol. In any case, these issues are fixed in this branch as it is more memory-efficient all around. The DSL is also more expressive now, with the ability to express conditional recursion via a new loop form, and arbitrary side effects via effect, and that should allow us to implement features (including existing features) more conveniently going forward.

There are a couple of introduced bugs, however -- (1) counts/quantifiers don't work for deletion at the moment, and (2) "soar" across trees (M-} M-{) operates like "leap" (} {) within a tree, instead of always crossing trees. I'm working on it and will hopefully have fixes soon, though I need to take a short break to finish some other stuff.

Enjoy, and please let me know if you run into any issues while using this branch.

[devcarbon-com]

Hurray! :tada:

Also, one of the the "bugs" sounds like a feature I want - a leap-or-soar command :D

[countvajhula]

Good to know! One of the reasons leap doesn't already soar is because accidentally crossing trees formerly had a high cost. Since the cost isn't as high now, we could potentially make leap-or-soar the default behavior, with soar being an explicit "leave this tree."

[dcostaras]

Hi @countvajhula :)

Apologies for being so quiet, it hasn't been because I'm not excited by Symex and Rigpa, work has just been nuts for the last 2 years.

For better or worse that has now come to an abrupt end and I'd love to be at least a bit more involved in the projects. I'm going to start by just catching up with the developments of the last year and a half.

Cheers, d

[countvajhula]

Hi @dcostaras . Congrats! This is the first step on the road to liberation from the reductive capitalist imperatives that imprison us all! 😆

Less flippantly, I'm glad to hear that. Welcome back. I'm currently taking a short break from Symex to focus on Jewel, but I'll be returning to work on 2.0 soon.

Re: what you could do, I think directly helping on 2.0 release work (e.g. tree-sitter, package decomposition, DSL, refactoring, etc.) would be tricky to coordinate in the immediate timeframe, but here are some broad things on my mind that are more decoupled and could be good to work on:

• https://github.com/drym-org/symex.el/issues/67
• https://github.com/drym-org/symex.el/issues/53
• https://github.com/drym-org/symex.el/issues/49
• any ideas on idempotence? Editing the undo stack sounds risky to me, so I'm wondering if there is a clever and simple solution, or a way to use overlays to simulate a change without actually making it until the user continues typing -- or something.
• Re: Rigpa, it is in need of a modal interface provider that is global (like Hydra) and not buffer-specific (like Evil), but which provides clean mode entry and exit hooks (pre-entry, post-entry, pre-exit, post-exit) that are reliable. Hydra is not really designed to be a full fledged modal interface and wrestling with these hooks to try to force it to behave like a modal interface has been a losing battle. I'd like to transition Buffer Mode, Window Mode, and other such "global" modes to using a lightweight modal interface provider whose entire purpose is to track entry and exit from a global "mode" such that its keybindings generally take precedence over "most" other things. I don't know how big of an effort this would be but I suspect it will be not that big. It's important for it to be lightweight. See this issue for some adjacent information: https://github.com/countvajhula/rigpa/issues/10 . Depending on the solution here, it may make sense to use this modal interface provider even for buffer-local states like Symex, but I'm not sure. I don't really have a clear idea of the considerations here, at the present time. I have a feeling @devcarbon-com might have a few (or many 😆 ) thoughts this.
• Here's another Rigpa item -- I don't fully understand keymap precedence and the utilities provided by Emacs for this, and I feel there must surely be a better solution for this: https://github.com/countvajhula/rigpa/issues/13 . This is a minor but everpresent annoyance in e.g. Magit buffers, Dictionary buffers, Dired, etc. which require custom fixes each time (and sometimes haven't been fixed, like for Magit the one-off approach used for Dired etc doesn't work).

I realize that's a lot of stuff to mull over but in case you can help with any of it, that would be helpful and appreciated. If you decide to work on any component, please do reach out to anyone else who may be on the relevant threads in case they are interested in collaborating or if they have any thoughts or WIP on it. Let me know if I can help clarify anything in particular!

[dcostaras]

Hi @countvajhula haha, don't get me started on capitalism, the ultimate local maximum!

Thanks for the detailed response, in reading the mentioned threads and other issues your thorough communication is beneficial :)

Due to a rising sense that my Doom Emacs config and Evil/Symex hacks (which would be almost entirely solved by using Rigpa) are starting to grow a life of their own (maybe time to declare emacs config bankruptcy again!) and some issues with RSI, I've been thinking about how to do things differently. I've never managed to get Rigpa setup properly on my config; so, I think, that is where I'm going to start. But also I've been looking at other modal paradigms such as Kakoune and Helix and other Emacs model systems like Meow and Boon (@devcarbon-com, it looks like you're also currently looking at Meow specifically). This all makes me quite interested in the work towards a cleaner modal system for use in Rigpa and Symex. I know that's not a good first goal as I don't have the required experience with Evil, Symex and Rigpa and it's not an immediate priority for Symex/Rigpa anyway but that's what's tickling my pure aspirational and intellectual interest.

Once I've got Rigpa and Symex WIP 2.0 working I'll have a look at the above smaller, more immediate tasks.

[dcostaras]

BTW, is general project communication happening on GH issues/PRs and the Symex 2.0 board?

[countvajhula]

Yes, general project communication is happening on GitHub Issues/PRs and there are no other channels in use at the moment.

If you have any difficulties setting up Rigpa or if the instructions are inaccurate / non-ideal / don't work out of the box for any reason, please report that as it's a bug 🙂

[RomeoV]

Excited to see this come to life soon! For now, what I really only want is some vim-like navigation of the syntax tree for non-lisp languages with emacs-29 treesit, e.g. for org mode. For example, moving in around the headings in org document using hjkl. Is this already viable if I have an org-treesit grammar installed?

[countvajhula]

Hi @RomeoV ! Being able to navigate org documents with hjkl would be great, and I'm curious what would happen if you tried that right now with org-treesit together with Symex. Assuming tree-sitter-mode is true, it should trigger structural navigations in the alpha support Symex already has for tree-sitter in the main branch, and it's possible it does work already (I've tried this with navigating JSON documents in the past and that has worked fine).

Now re: the 2.0 effort, it's high time for a status update so here it is:

I believe the 2.0 work is reasonably close, but there's only so much time in the day and lately I've been focusing on Attribution Based Economics (which Symex is one of the pilot projects for! I'm very excited about the possibilities that ABE offers for open source economic viability as there have been some very interesting breakthroughs lately that I believe resolve some questions I myself have felt were unanswered by ABE thus far), and recently gave a talk at the Foresight Institute on that subject. I will be giving another talk related to ABE at RacketCon in October, so I don't think I will have a lot of time for a deep push on 2.0, but it's silly for so many improvements to be sitting in a branch not being used.

So here's what I'm thinking: I can aim to do a triage of the 2.0 work in the near future and retain anything that is a clear improvement over master, while being more conservative with changes that aren't quite there yet. The goal would be to merge as much of this into main as possible so that further work can continue on the main branch.

What isn't done yet:

1. The initial goal was to have the Symex DSL be a level of abstraction above the primitive tree operations (i.e. tree-sitter and native ELisp), so that most/all structural features would be implemented at a high level of abstraction in Symex, and this would allow the primitives implemented by tree-sitter and Lisp to be relatively small instead of having large sets of primitive operations. We aren't especially close on this front since drawing the new abstraction boundaries will take some time, as Tree-sitter in some ways accomplishes what was initially unique about Symex -- that is, being able to assume an explicit tree representation in implementations of features. I still think it would be good to aim for this since Symex is a higher level language than tree-sitter, and thus easier even for casual users to add nontrivial features in. But as I said, we aren't especially close yet on this front and it will take more work than I can do in the immediate future, and having to use a large set of primitives to provide various features may be OK for now.

2. Tree-sitter UX polish. There are still a number of edge cases in the UX with respect to tree-sitter languages, and it would be good to tackle these one at a time as they come up.

3. Using Emacs's native Tree Sitter either directly or via tree-edit. This wasn't explicitly a goal with this release but it would be a nice one to tackle soon as many folks have brought it up.

4. Package decomposition. Symex is a big package. We want to decompose it into several small packages (e.g. DSL, interface, runtime, etc. in separate packages). We just might be able to do this as part of this release.

5. One big improvement I wanted to make was actually in Rigpa, to use the dynaring package in ways that would make the UX with Symex and Normal state very natural. But it does not seem likely I'll be able to get to it this time around.

6. Reducing reliance on Evil. This one's on @devcarbon-com 🙂

On next steps, I will aim to triage this branch and get it into a reasonably mergeworthy form. At that stage, we could use help with testing on both Lisp and especially Tree Sitter. I will post about that when we're there.

Thanks again for your interest and support.

[RomeoV]

Thanks for the detailed answer. Just one thought from my side: I do think with the inclusion of treesit in emacs, many people are looking to switch away from having both tree-sitter-mode and treesit installed. For me personally, any package focussing on tree-sitter-mode is a non-starter, as it's most likely not the future.

I think a triage for 2.0 would be useful indeed, there can always be a 3.0 release ;) Good luck also with your other work!

[countvajhula]

Heads up: I will be rebasing this branch sometime today to bring it up to date with master.

[countvajhula]

This branch has been rebased! If you happened to have work in progress that was based on this branch, you could use these instructions to bring it up to date and reflect the updated base branch correctly.

[countvajhula]

Rebasing now to get recent fixes on the main branch including #144

[countvajhula]

Hey all, I've merged the change to use Lithium as the modal UI! Please note you'll now need this dependency which isn't yet on MELPA:

(use-package lithium
  :straight
  (lithium
    :type git
    :host github
    :repo "countvajhula/lithium"))

If you have time, it would be very helpful if you could try out this branch and report any bugs you encounter --- just an issue with a quick note would be a great start, or even just a comment here if you want to discuss first. This would give us small, concrete items to focus on and take steps towards release 🙂

[devcarbon-com]

I'll take a look tomorrow and hopefully get it setup in time to try it out while I'm working. Thanks!

[countvajhula]

I made a few small fixes last week, and also just fixed the Hydra compatibility issue in Lithium. Aside from those, I haven't noticed any other remaining issues. Now would be a good time to pull the latest 2.0-integration branch and also the latest Lithium 🙂

Next, I'm going to look at our tree-sitter status and see what it would take to migrate to Emacs's built-in Tree Sitter support. I think you had started on a prototype migration a while ago @polaris64 ? If you have a link handy do send it over, or if you have any thoughts or suggestions on it I'd be glad to hear them!

I think migrating to built-in tree-sitter is the last blocker for merging this (unless any other bugs are reported). Fully removing evil would be nice, too @devcarbon-com 😉

[polaris64]

Hi Sid,

I made a few small fixes last week, and also just fixed the Hydra compatibility issue in Lithium. Aside from those, I haven't noticed any other remaining issues. Now would be a good time to pull the latest 2.0-integration branch and also the latest Lithium 🙂

I'm glad to hear things are progressing nicely!

Next, I'm going to look at our tree-sitter status and see what it would take to migrate to Emacs's built-in Tree Sitter support. I think you had started on a prototype migration a while ago @polaris64 ? If you have a link handy do send it over, or if you have any thoughts or suggestions on it I'd be glad to hear them!

I actually only just merged this branch a couple of weeks ago to my development branch, so the timing is good. My commit was to remove the dependency on the old Elisp Tree Sitter library as I thought it probably doesn't make a lot of sense to support the old library now. Either people will use Symex with Emacs 29+ and have Tree Sitter support, or they'll use it with an older of version of Emacs and will just have the Lisp-like language support.

To summarise where I am so far: I have replaced all of the existing functionality to support the built-in Tree Sitter system except for one element, the `symex-ts--handle-tree-modification' macro. The purpose of this macro was to get a difference between the trees before and after an edit and to use this to move the point to a new current node. In the previous implementation, code which modified the tree could be wrapped by this macro to ensure that the state was relatively sane after an edit. Currently this just signals a "not implemented" error when using the built-in Tree Sitter library.

The problem is that the built-in Tree Sitter library does this differently. It instead uses a system of notifiers which are parser-specific, and buffers can contain more than one parser. Notifiers can be added and removed for a given parser and the list of notifiers for a parser can be obtained via `treesit-parser-notifiers'.

Given all of the other changes to Symex since I last worked on it I'm not sure if this macro is even still needed. If not then great, the conversion should be complete! If it is however then it'll need to be modified in order to work with this notifier system. This is the only task remaining before I would say the conversion is complete.

In any case, my own branch https://github.com/polaris64/symex.el/tree/scratch/builtin-treesitter contains all of this work so far, hopefully that will help.

Kind regards,

-- Simon Pugnet https://www.polaris64.net/ PGP key fingerprint: 3BF7 85DE 162C 00C8 FB4D A6FD BA13 59A8 2C0B 3EF9

[countvajhula]

That's fantastic news @polaris64 ! I just checked and it looks like we do still use symex-ts--handle-tree-modification (looks like it's only used in delete and in paste, both in the same file), so it sounds like we will need to migrate that. Excited to take a look at your code soon 😄👍

[countvajhula]

Hi all, as you may have noticed, we've merged the tree-sitter updates! 🌳 🐿️ This drops support for the older tsc and elisp-tree-sitter libraries and instead uses Emacs 29+'s built-in Treesitter. Thank you @polaris64 for the awesome work!

Note that this now means that older versions of Emacs will not be able to use Treesitter, but Symex should still work with regular Lisp code, and shouldn't raise any errors if you attempt to use it in a non-Lisp buffer. If anyone here is on an older version of Emacs (< Emacs 29), it would be helpful if you could verify that this is actually the case.

The treesitter features are still not as complete as the Lisp features, and the full matrix of features that are working/not working is listed in this PR description. We've already merged that PR of course, and will continue making improvements here on the integration branch 👍

[countvajhula]

Hey @polaris64 , good news! With your timely tip re: notifiers and the root cause of the "outdated node" errors, I was able to get things working with selecting a node from point after every change to the buffer! Just empirically, some features that were formerly not working are now working, but I haven't done a comprehensive pass of testing yet.

To give a status update of where it is now: I think there are many features in symex-transformations.el that simply are hardcoded to assume Lisp (e.g. symex-shift-, symex-wrap, ...), and it just remains to: (1) write a wrapper that checks if TS is available and then delegate to either a lisp or TS implementation (like many of the functions in the same module), and then (2) provide a treesitter implementation for that feature in symex-transformations-ts.el (after moving the original implementation into the corresponding lisp module).

There may be a few isolated cases where a feature could be implemented in the symex DSL (like delete and paste, and all the motions are done this way) --- if that's possible, then the same implementation would work for both Lisp as well as TS, and could just be done in the transformations.el file itself! But my impression is that, at least currently, mutating operations implemented in the DSL generally work inconsistently in TS 🤷 , so we probably need separate TS implementations for most of them.

In any case, I'll resume working on this soon (sometime in the next few days as I have some other obligations coming up). If you're inspired to tackle any of the features in the meantime, feel free to (but also no pressure --- just letting you know where it's at 🙂 ).

[countvajhula]

Hullo folks, another status update:

Things are coming along nicely. I've updated the matrix of features in the PR description of the merged treesitter PR (it's a convenient place to track things, for now). As you can see, we're getting closer on the remaining Symex features for treesitter!

Now is a good time to pull the latest 2.0-integration branch and also the latest Lithium, and any feedback would be helpful.

A few observations:

A couple of the remaining features use evil-surround (symex-change-delimiter) or paredit (symex-raise). These obviously aren't expected to work in treesitter/non-Lisp languages, and I'm thinking it would be ideal if we can implement them in a common way without these dependencies. I've recently upgraded an old utility macro called symex--transform-in-isolation. This was formerly being used in a few features, but I think it could actually be really useful to implement a lot of features without worrying about too many messy details of transforming the source buffer, so I've made some improvements to it towards this goal.

The problem it aims to solve is that mutating the source buffer can be messy, as we need to worry about changing offsets and positions, stale point locations, indented vs non-indented code, etc.

What this macro does is, it copies the region to be transformed into a temporary buffer using the same major mode, and then executes whatever code we write, and then pastes the resulting contents of the temporary buffer back into the source buffer at the end, neatly replacing the original region. This seems to make things easier, like not having to worry about messing up indentation of surrounding forms in the source buffer, etc. I'm not sure if it can be used always, but in cases where it can be, it seems quite handy.

We might be able to use this macro to implement some of the remaining features in a Lisp/Treesitter agnostic way, as we have a number of primitives now that work for both lisp and treesitter, like (symex--get-end-point count), which can give bounds of expressions, or --go-up, --go-forward etc. So, if we can express the operation in terms of these primitive operations, they should work across both Lisp and Treesitter with a common implementation :)

So with these primitives and the transform-in-isolation macro, we just might be able to eliminate a few dependencies like paredit and evil-surround. FYI @devcarbon-com .

And finally, I've noticed that after moving to Lithium, evil-repeat (dot) no longer works 😭 . It might be because Lithium wraps every command with error handling, something like:

(lambda ()
  (interactive)
  (unwind-protect
      (call-interactively <original-command>)
    <error handling, lithium exit hooks, etc>)

My guess is that because of this wrapping interactive lambda, it is no longer being recognized as a repeatable command by evil-repeat, as only the original-command was declared as repeatable. Do you think this is a likely explanation @j-shilling ? If you have any thoughts on wrapping repeatable commands in such a way that the repeatability is not obscured (ideally without declaring the containing lambda as repeatable, to avoid introducing an evil-repeat dependency in Lithium), I'd love to hear them.

Alright that's it for now. Enjoy and please let me know of any feedback. Thanks!

[countvajhula]

Hullo all! We just merged @devcarbon-com 's work to remove Symex's hard dependency on evil. Now, it should work whether you have Evil installed or not. This takes us a long way towards making Symex lean and lightweight.

In principle if you do have evil installed, it should give Symex a couple of additional features, especially evil-repeat via ., but this feature is currently broken, as I explained in my previous comment. I hope to take a closer look at this soon (input welcome!).

If you are using this branch, then now is a good time to git pull. I'm having trouble testing tree-sitter functionality since there are so many languages to try (Python, Clojure, JavaScript, ...), so if you happen to want to try Symex for one of these non-Lisp languages, if you could report any issues you see, that would be especially helpful. Remember that you will need to use a tree-sitter enabled mode for this, like python-ts-mode or clojure-ts-mode.

Enjoy, and thank you @devcarbon-com !