Feature Suggestion: Emulate Stepping Through Code with Stickers

[artforlife]

After working with SLY for a couple of weeks, I really like it. However, new feature ideas keep popping up.

One of such ideas is to introduce the ability to step through code similar to how one can do it in Java, C and other languages of that sort.

At the moment, if one wants to debug something, he needs to set a Sticker at a spot of which he is aware. However, there are times when one debugs a large code-base which he does not understand very well. That is, it may not be obvious what function calls another function and so on.

I see a couple of ways to address that. One is to introduce a kind of general Sticker that would cover an entire function. Then, if that function contains several Lisp forms, the Sticker would break after (or after and before) each form's evaluation. This would allow one to step through a function.

This idea can be generalized. Suppose that we say that we do not want to break on each form of the original function, but also (or only) on the forms that are one level lower than that. Then, this Sticker would also do that. Depending on the code's depth and/or complexity, it may require this depth setting to be anywhere from n=1 to n=1000 or more to reach the forms and expressions of Common Lisp's lowest level.

There are probably other way of doing this. However, if one could have the ability to step through code, combined with Common Lisp's other powerful debugging tools, for which -- I am sure -- we all love it, it would be an unstoppable combination of debugging flexibility and power.

[joaotavora]

What you are requesting is an approximation to a generic, portable CL stepper.

It's an elusive thing in Common Lisp, and while I'm probably not aware of all of the problems associated with it, one of them is macros. Think: a form in a function can be something that will be evaluated and therefore it makes sense to put a sticker there. But it may be just a bit of syntax in a macro, and a sticker there is useless, and quite possibly confuse the macroexpander.

So what you need is something known as a code walker, which is more or less what themacroexpand family already is. But, vitally, one that keeps the connection between the input position in the file/buffer and the actual form being walked, which is what matters to IDE's wanting to place stickers, or other kinds of instrumentation.

Such a vital component, and a clear view of how to integrate it with an IDE, is still lacking. I will page @luismbo here, since I vaguely remember that he had the beginnings of something like this.

Unfortunately, lately I have only minimal time to hack on Lisp, so you'll have to find the manpower for implementation elsewhere.

[artforlife]

I have noticed that SBCL now provides Single Stepping functionality as documented in section 5.10 here: http://sbcl.org/manual/sbcl.pdf

Are you aware of SLIME or SLY interfacing with this functionality? If it has not been implemented, this is something I may be interested in looking into.

[joaotavora]

SBCL has been doing that for a while. SLIME does it, but if I remember, it doesn't work great and, obsviously, is not portable.

[mmgeorge]

This seems like it is a bit finicky but works decently well on sly/slime on sbcl, but you need to make sure that the function(s) you want to step are compiled with (debug 3) and that sly/slime is compiled without debugging. If you don't do the latter, sldb will attempt to step itself which causes all kinds of issues.

To use the stepper, when you hit a breakpoint and enter the debugger (while ensuring that the cursor is under the backtrace portion of the debugger window), press "s" for "sly-db-step". The first time this is pressed it will start stepping. Afterwards it will behave as a "step-into".

Also note that if you try to "step-out" when you are at the top-level of where you began stepping, this will currently cause the debugger to get a bit messed up.

Might be nice if this was improved to make it a bit more user friendly ... could make sure that the relevant slynk functions are always compiled without debugging, more gracefully handling top-level "step-out", and maybe a persistent indicator / easy variable access when going through the code? I can try and take a stab at this.

[joaotavora]

Hello @mmgeorge you might want to read a paper that I recently wrote on the subject. https://zenodo.org/record/3742759

And check out https://github.com/joaotavora/sly-stepper. The implementation is fairly "green" but is workable (and uses stickers as @artforlife suggested). I would love to have some feedback of that.

The SBCL stepper, in my opinion, in not the way to go. We want a portable stepper.

[mmgeorge]

@joaotavora I took a look at your paper and have been playing around with sly-stepper a bit. Great to see some work on this! I would love to see a truly portable stepper, however I do worry that it will be very hard to create an experience that rivals that of other IDEs without without compiler support. Granted, that may not be the goal here.

Some initial impressions:

Benefits:

• Portability 🎉
• Code is highlighted as you step through.
• I kind of like the whole capture-and-replay workflow. I feel like this provides a nice workflow for debugging issues where you have a good grasp of the code is called throughout.

Cons/limitations

• ~Currently it seems like it isn't really possible to observe side-effects/state changes, because value when you replay the sticks is the final value. Maybe this can be fixed by storing the printed value of the expression?~

~When I replay foo below, I get back (list 1 2) instead of the expected (list nil nil) when stepping through the third line:~

(defun foo ()
  (declare (optimize (debug 3)))
  (let* ((out (list nil nil))
         (x (fiz out 2 1))
         (y (baz out 2 1)))
    (baz out y x)))
  
(defun baz (out a b)
  (declare (optimize (debug 3)))   
  (setf (cadr out) 2)
  (+ a b))

(defun fiz (out a b)
  (declare (optimize (debug 3)))
  (setf (car out) 1)
  (fiz-inner a b))

(defun fiz-inner (a b)
  (declare (optimize (debug 3)))
  (- a b))

• ~Stepper is not integrated into the debugger. At each step of a stepper, it is often useful to know what variables are relevant for a given frame, and what their values are at a given step. Again this is very useful for debugging objects/side-effects.~

• All steps for the stepper must be annotated upfront. This does seem like it is a bit more tedious than just setting breakpoints where you want them to be, though perhaps not a deal breaker. Where it is problematic is when one uses a stepper to understand (perhaps not even debug) a large codebase. Also for debugging non-local exists, a major benefit of having stepper is also being able to use it to find where the exit is.

• Stickers are leaked in the call-stack, though this can probably be somewhat easily fixed when reporting this back to the user? vs

That said, I'm definitely looking forward to seeing where this goes!

[joaotavora]

Currently it seems like it isn't really possible to observe side-effects/state changes, because value when you replay the sticks is the final value. Maybe this can be fixed by storing the printed value of the expression?

You should have a look at (setq slynk-stickers:*break-on-stickers* '(:after))

I'll read the rest of your feedback tomorrow.

Stickers are leaked in the call-stack, though this can probably be somewhat easily fixed when reporting this back to the user?

This is by design. I could severely complicate the code and eliminate those stack frames, but the variables would be there anyway, just pretty unreadable.

Stepper is not integrated into the debugger.

It is, it is: have a look at *break-on-stickers*. It just needs more (and better restarts). Do read the paper more carefully, I talk about exactly this expectation at one point.

[mmgeorge]

It is, it is: have a look at break-on-stickers. It just needs more (and better restarts). Do read the paper more carefully, I talk about exactly this expectation at one point.

Ah let me take another look and get back to you

[joaotavora]

Third and fourth paragraphs of section 4.

[mmgeorge]

Sorry @joaotavora, not sure how I missed that 🤦‍♂️. This is much, much closer to what I was hoping for! Aside from the obvious WIP things like you mentioned (e.g., the restarts - would be very nice if, like the paper mentioned, the annotations were aware of nesting and had corresponding step-next/step-out restarts), I would still say I still find myself coming back to a couple of the points I mentioned above:

1. Needing to pre-annotate something you would like to step into on paper feels a bit cumbersome. However, that said, I'm not sure how much it actually matters in practice -- some annotation is expected in any stepper to set a breakpoint to start with. I could see it being less than ideal in finding where a very far non-local exit occurred, as you would have to annotate these upfront.
2. The stack trace is quite a bit harder to read. I understand that there's a tradeoff here between this and code complexity, but not seeing the stickers in the stack trace is something that would definitely be on my wishlist down the road. At least for SBCL, I feel that the stack trace is already overly chatty, as there are just so many irrelevant frames captured. That said, from a functional standpoint, the main reason for wanting a clean stack trace at a given step is that I would want to inspect the value of some variable at the latest (relevant) frame associated with that step. Since inspecting variables in 90% of the reason as to why I would want to do this, I wonder if there is a way this could be done in an automated way, e.g., by decorating the source code directly with values?

Other than that, I'm not quite sure how I feel about stepping atoms. On the one hand, it makes the code flow through the stepper very expected, but it also takes much longer to get to where I want to be. Literals don't seem like they should be stepped.

For this and the above two points I think I would need to actually spend some time using the stepper to debug something real to really know what I feel about these / how much these actually matter.

On the plus side, versus the SBCL stepper, the way things step seems vastly more intuitive, and I don't have to worry about the stepper suddenly behaving differently due to the way the compiler optimized certain things. This is a huge benefit and I think the interaction of how some things become unsteppable after compilation is a major part of the reason why the SBCL stepper can feel almost buggy at times.

[joaotavora]

Matt George [email protected] writes:

Sorry @joaotavora, not sure how I missed that 🤦‍♂️. This is much, much closer to what I was hoping for! Aside from the obvious WIP things like you mentioned (e.g., the restarts - would be very nice if, like the paper mentioned, the annotations were aware of nesting and had corresponding step-next/step-out restarts), I would still say I still find myself coming back to a couple of the points I mentioned above:

1 Needing to pre-annotate something you would like to step into on paper feels a bit cumbersome. However, that said, I'm not sure how much it actually matters in practice -- some annotation is expected in any stepper to set a breakpoint to start with. I could see it being less than ideal in finding where a very far non-local exit occurred, as you would have to annotate these upfront.

Well that is discussed in the paper, and in fact it was already discussed in the 80's. I favor annotation-based steppers, precisely because they give me the power to choose what i want to step on. But I admit sometimes it's more confortable to know that everything is being stepped. For that, you need an evaluator-based stepper and/or to compile all the code with stepping instrumentation (which is what SBCL does).

2 The stack trace is quite a bit harder to read. I understand that there's a tradeoff here between this and code complexity, but not seeing the stickers in the stack trace is something that would definitely be on my wishlist down the road. At least for SBCL, I feel that the stack trace is already overly chatty, as there are just so many irrelevant frames captured. That said, from a functional standpoint, the main reason for wanting a clean stack trace at a given step is that I would want to inspect the value of some variable at the latest (relevant) frame associated with that step. Since inspecting variables in 90% of the reason as to why I would want to do this, I wonder if there is a way this could be done in an automated way, e.g., by decorating the source code directly with values?

The classical debugger with the stacktrace is an relatively inferior interface to stepping. You current have 3: replay, breaking debugger and in-source fetching, which you may have missed. But It's not unthinkable to have a fourth interface, which is like the breaking debugger, but simplified for common stepping operations. SLY/SLIME makes this possible: you develop an interface to your taste in Elisp that shows only the needed information from the Common Lisp side. In fact, if you notice, the SLY debugger is alreayd a bit tweaked visually when it "knows" it is stepping on a sticker. The idea could be advanced greatly, cleaning up the stack trace below, or preceding it with another more valuable section of "just variables".

Other than that, I'm not quite sure how I feel about stepping atoms. On the one hand, it makes the code flow through the stepper very expected, but it also takes much longer to get to where I want to be. Literals don't seem like they should be stepped.

This is reasonably easy to do, or to make configurable. I may also note that you can manually adjust the stickers after placing them with C-c C-s P

For this and the above two points I think I would need to actually spend some time using the stepper to debug something real to really know what I feel about these / how much these actually matter.

Yep. I do that occasionally. The main thing I miss are nice step-in/step-out restarts.

On the plus side, versus the SBCL stepper, the way things step is vastly more intuitive, and I don't have to worry about the stepper suddenly behaving differently due to the way the compiler optimized certain things. This is a huge benefit and I think the interaction of how some things become unsteppable after compilation is a major part of the reason why the SBCL stepper can feel almost buggy at times.

I admit I haven't tried the SBCL stepper very much. Same with every other stepper, and for the same reason: not portable.

João