Auto completion for lexical bound variables?

[kchanqvq]

It seems that currently only global variables appears in the auto completion list (in fact seems to be just all interned symbol I guess). I've lived with this for a long time but it would be really good if it can include variables introduced by LET, LET* etc, before executing the DEFUN and interning those symbols.

I wonder if the feature is already there and I missed it and is there anyway to enable this?

[joaotavora]

I wonder if the feature is already there and I missed it and is there anyway to enable this?

There is not that feature yet, but it's a very interesting feature.

It seems that currently only global variables appears in the auto completion list (in fact seems to be just all interned symbol I guess).

First some clarification: symbols appear in the list. Some of those are special (and this is what you call global variables, I think), but some are functions, macros, and some aren't anything of the sort, they're just symbols. Not all interned symbols appear in the list, most of the times, just the exported ones. Also, the very same symbol might appear more than once if it can be designated in multiple ways (by way of package nicknames and cross-package importations).

IF I understand what you are requesting, then it would make some sense to me to collect some symbols in the current lexical scope (exported or not) and sort them to the top of the list. That is because when writing functions or other constructs, one is primarily interested in the symbols bindings (value or function) established in the tightest lexical block (of course one may also be interested in any symbols, secondarily).

Anyway, as you're typing, new names the symbols that appear may or may not have been interned by the lisp reader. They will when you C-c C-c the definition, so they will appear in the list the next time, but probably sorted way to the bottom, and not identified as lexically established symbols in any way.

And, in my view, that's the main difficulty: discovering which symbols are lexically established at the current point.

Recently, I've been working on a stepper tool for SLY (very alpha, but should be working). Within that tool is a "source-tracking form reader" that could be used to this effect: discovering a superset of lexically established symbols. But the difficulties don't stop there because not all symbols mentioned lexically are really "established" as variable or function bindings. As you yourself noted, only LET and its friends are interesting. Symbols used in macros for other purposes are probably not interesting at all. Think of LOOP: do you want the completion system to think initially, for and with are lexically established symbols? No, right? But you do want whatever came after the for or with to be identified, right?

That's a more difficult problem, but it also has a solution... Anyway I've been rambling long enough.

[kchanqvq]

And, in my view, that's the main difficulty: discovering which symbols are lexically established at the current point.

Suppose you can discover a superset of symbols which could mean lexical variable, this will be helpful for filtering them: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html This is not in ANSI Common Lisp, but there's a portability package: http://quickdocs.org/cl-environments/

[kchanqvq]

This might have some performance issue however. I can imagine to account for exported symbol one only need to refresh when a package is (re)defined, but to account for lexical variables, the most naive way is constantly calling macro-expand on the current form being edited and collect the information, and will involve full code walking. It might also turn out to be a non-issue though, nowadays computers are fast.

[joaotavora]

Thanks for the pointer to environments: I was going to point to something else, but indeed for this particular problem, environments seem useful in principle.

This might have some performance issue however.

Yes, fully macro-expanding on every completion request seem problematic.

nowadays computers are fast.

The problem is that arbitrary user macros can take a long time to run. Is it a good decision to burden the debugger of some horrible code with a long wait for completions just because whoever wrote that macro made it extra-slow to expand?

I'm thinking we could do this on a best-effort without full macro-expansion, i.e. recognize some special forms (let, multiple-value-bind) and maybe some macros (say, loop) and only deal with those.

[kchanqvq]

I haven't got chance to hack into SLY internals so don't take what I'm going to say seriously, especially the highly opinionated "best"/"better" etc...

Recognizing some special form will do part of the job, but it seems like reimplementing an incomplete code walker. This does not look like the best thing to do for me. I think one better way is to use the LISP implementation's code walker but in a separate thread, and keep local variable candidates and interned symbol candidates separate. We can then asynchronously poll and refresh the lexical variable candidates. If at some point it doesn't work out then nothing is harmed except the candidate list is not refreshed. A timeout can also be setup to kill the thread and force restart code walking on current point to avoid dead-looping macros.

I'm not sure how hard to implement this in SLY but at least it saves the effort of writing an incomplete code walker. And it is guaranteed to generate all candidates except the edge case that some macro expansion hangs. Hanging or super slowing macro must be very rare because user will soon notice it while they try to evaluate any code using it. On the other hand, since refreshing is done asynchronously (and it refreshed FREQUENTLY enough when user is editing, instead of refreshing every time user edits), though normal macro expansion may take dozens of milliseconds it won't be noticeable.