Add ClojureScript support to the debugger

[Malabarba]

The debugger simply fails if used in ClojureScript code, it would be nice for it to work.

Here's a list of things to achieve that. If anyone can help with any of these items, it will likely save me a lot of researching down the road. So please don't be shy.

• [ ] (trivial) Change the debug middleware from cljs/expects-piggieback to cljs/requires-piggieback.
• [ ] Ensure that the #dbg and #break reader tags are active when the cljs code is read. This is done in Clojure via our data_readers.clj file at the classpath root, is there a similar file for ClojureScript?
• [ ] Currently instrumentation is done by ensuring that tools.nrepl uses our instrument-and-eval function instead of the plain eval (see debug.clj). I'm not sure how to do this in piggieback.
• [ ] instrument.clj should work as is. It parses the code (which is still Clojure data even with cljs code), and it wraps parts of it in some code of our own.
• [ ] The debugger then works by running some CIDER code inside the user's code. This code therefore needs to be defined in the cljs environment.

The last item is hardest. The code in question is just one macro and a few functions in debug.clj. These needs to be defined in the cljs environment, which probably means they'll have to be moved to a cljc file (or duplicated in a cljs file).

The problem is that this code interacts with tools.nrepl, which (IIUC) is impossible to do from the cljs environment. So it'll likely involved a deep rethinking of how the debugger works. Reimplementing the first version of the debugger in cljs might be the solution (it prompted the user directly instead of using the nrepl).

[divs1210]

Hey, @Malabarba ! Thanks for the list of things to start off from. I'm new to Emacs and CIDER, but would like to give this a shot.

[sfrdmn]

Any thoughts on a setting for simply ignoring the fact you're in a CLJS file and just interpreting the forms as plain Clojure? I guess it'll work fine in a lot of cases and if it's documented + opt-in, shouldn't be too problematic?

[Malabarba]

I think M-x clojure-mode might work.

[sfrdmn]

So it does! Thanks 👍

[crinklywrappr]

2nd time this week Cider has prompted me to visit this page.

[NightMachinery]

How does Cursive manage this?

[bbatsov]

No idea. It's not open-source so we can't easily check.

[cursive-ide]

@NightMachinary @bbatsov Very simple - it doesn't :-). Cursive can only debug JVM clojure right now.

[bendlas]

So, the biggest problem with having a debugger based on instrumented code in JavaScript, is, that unlike the JVM, you can't block your thread to wait for input from the debugging frontend. This leaves two likely approaches: Debugging via the runtime's debugger interface (e.g. via debugger.html) and relying on source maps to recover the cider experience, or, doing a CPS transformation on the source.

I honestly can't decide, what feels to be more work, but ultimately, CPS feels more in-line with the spirit in the existing debugger, of instrumenting the source. A remote-control debugger would totally be worth doing, though, and I'm sure that an RDP-based debugger, targetting Clojure and ClojureScript could be a great success as well.

That said, let me try to sell you on an even crazier idea, than doing CPS on ClojureScript code: Doing CPS on JavaScript code. That would (theoretically) enable debugging callbacks coming from javascript frameworks.

I decided to explore existing CPS solutions for JavaScript, and there are some, most of them seem to focus on providing some syntax for explicit passing, but jwacs stood out, not just because it takes its job of doing the heavy lifting for providing true continuations, seriously, but also because it's written in CommonLisp, so working with it reminded me of how Clojure with Emacs should feel ;-)

Initial results on some snippets and even a 2.4M advanced minified JS seemed promising, after fixing some minor issues

I so far found three features missing:

• finally clauses are documented as missing https://github.com/chumsley/jwacs/issues/4
• geters, seters, necessary to deal with in some way, because they can't be transpiled away https://github.com/chumsley/jwacs/issues/5
• ~breaks from labelled-statement-blocks~, a rather obscure language feature, used by gclosure advanced

After implementing labelled statement blocks to get acquainted with the code base, I'm pretty confident, that I (and you too) can make it work for everything we need, including the above issues.

If you're interested in helping with this, I'm currently in the process of getting acquainted with cider-debug-middleware's internals. In particular, I'd like to figure out, how to run some hand-crafted code in a clojurescript runtime, to contact cider-debug-middleware with a break-point.

TLDR; A jwacs-based cps-transformer could be run in a separate server, processing JavaScript, instrumented with breakpoints based on function_continuation, suspend and resume, as per jwacs-doc.

[bbatsov]

@bendlas It's nice to see someone interested in fixing this! Let me know if you need any assistance from me!

I don't know almost anything about ClojureScript, but I know a thing or two about CIDER. :-)

[bendlas]

@bbatsov theat's great, thanks for offering! I'm currently trying to understand how piggieback communicates with the rest of nrepl. For some reason, on a piggieback-ed cljs connection, I don't see any more messages coming through the debug middleware ... we can discuss more interactively on the #cider slack ...

[divs1210]

This could be helpful in some way: https://github.com/philoskim/debux

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution and understanding!

[rksm]

These are my 2¢ from a recent Slack chat, @bbatsov suggested to post these for posteriors' sake.

The comment that @bendlas left last year is spot on about how the last item in the "todo list" can be approached. However having done a JS debugger using the CPS approach myself once, I can say that this approach is more complicated as it seems. Again, the fundamental problem is you can't stop / blockingly wait in JavaScript. What you basically do to get around that is to a) instrument all code so that it can record and externalize stackframes / state on the stackframes and some sort of handmade PC (program counter) so you know where you are. This part is actually not that hard except for one issue: it requires instrumentation of ALL code, at least all code that can call code you want to debug -> so you have the stackframes to step/continue. JS has the concept of native funcions (e.g. iterators) that might very well call your code so this is the tricky part about it. There are ways to hack around that issue.

"Stopping" happens by unwinding the program using exceptions, that, step-by-step record the state that is present b/c of the instrumented code. This is easy.Problem three to solve is how to resume execution. The state you gathered in step 2 is all you need, but there is no built-in method to resume, of course. In the past I've used a JavaScript-in-JavaScript interpreter to "fast-forward" to the position in the stack frames and then run natively from there on. Again, this needs instrumentation of all code leading to the code you want to debug. Instead of an interpreter one can use a "skip" implementation similar to what the cider debugger uses right now, I think this would be a bit nicer.

Let's say this all can be build, the remaining issue is to integrate the code rewriting with the ClojureScript compiler itself. Ideally the rewriting should happen as part of the compilation step.From my humble experience this is a whole lot of pain and more time intensive than a voluntary project will allow.

So there are two alternatives, I think. On the one hand, it would be possible to do a "CPS-light" debugger, just for a single function, maybe with a step into for code that gets called. That would only require to instrument code that the user actually wants to debug but you can't access the stack. I haven't thought too deeply about that and there might be issues I overlook but I think that could be made to work reasonably fast.

The other solution is to use Chrome DevTools Protocol to remote control a chrome / v8 vm, e.g. via https://github.com/tatut/clj-chrome-devtools. The advantage of this approach is that it definitely works and since devtools are very widely used now will remain pretty stable. The only problem I see is that devtools are JS not CLJS tooling. This means, even with the use of source maps, there will be a certain mismatch between what the chrome debugger actually works on and what the user has written. E.g. Certain local vars made not be accessible, the debugger does not know about macros and such. Other than that the effort to integrate that is to make it work with the bazillions of ways of using ClojureScript. What's basically needed there is to create a "side-channel" browser environment that connects to the chrome-dev instance (similar to https://github.com/hagmonk/cobalt but prepped for debugging tasks). It needs to share the compiler / browser state with the tool the user actually uses, e.g. figwheel or shadow or whatever. So the main work is in getting those configs right, "multiplex" the repl and keep everything in sync

[bendlas]

The other solution is to use Chrome DevTools Protocol to remote control a chrome / v8 vm, e.g. via https://github.com/tatut/clj-chrome-devtools.

I found a solution for this by way of wanting to repl into a sandboxed JS RT: No network connectivity means having to tunnel through CDP for resource loading, and that means implementing blocking calls back through CDP.

https://github.com/webnf/cdp-repl is a repl-env with some structured support for getting a debugger connection from an extension + blueprint of how to do blocking calls with that. I believe it should be possible to port the cider debugger to this.

[bendlas]

Specifically, the next step on this would be to get cljs versions of the #dbg, #break and #light reader tags. Maybe @vspinu has some notes on this?

[Bad3r]

Any updates on this issue?

[vemv]

https://github.com/jpmonettas/cider-storm probably is a nice tool you can use today!

I'm going to close this issue because the chances someone will spontaneously pick it up and implement something seem very low.

Obviously, the doors are always open to someone who brought in a specific design and plan. Else better not to promise anything by keeping the issue open.

[divs1210]

If cider-storm was made the default debugger for CIDER, then we'd have a working CLJ(S) debugger in CIDER by default.

Would it make sense to delegate all of CIDER's debugging functionality to cider-storm?

[bbatsov]

I'm quite attached to our debugger, so I don't plan to replace it any time soon.

[vemv]

I'll try to mention Cider Storm in our docs.

Will also be interesting to see how that project evolves. At 14 commits it probably has some rough edges, but FlowStorm's underlying work is very solid.

[bbatsov]

Yeah, it's a good idea to mention it in the docs. I had never heard of it until yesterday.

[divs1210]

I believe I have found a solution!

Stopify:

Stopify is a JavaScript-to-JavaScript compiler that makes JavaScript a better target language for high-level languages and web-based programming tools. Stopify enhances JavaScript with debugging abstractions, blocking operations, and support for long-running computations.

Compiling debug builds with Stopify will give us all the debugging abstractions we need!

They have even tested it successfully with ClojureScript!

I'm going to try it out and post my results.

EDIT:

1. Unfortunately the project seems to be unmaintained
2. I have verified that it does work in the browser with simple JS
3. Running on node is currently unsupported / has bit rotted
4. For compiling using Stopify in the browser, we need the entire program stored in a string. I'm assuming CLJS tooling saves this string somewhere in memory before sending it to the browser. We'd have to compile that string with Stopify before it is run in the browser, but I haven't looked into this yet. To see if this would work, I tried to copy paste a compiled cljs program into the console as a template string but it keeps giving syntax errors.