[Lisp] Highlighting seems very gone

[JellyWX]

The highlighting on Scheme dialect of Lisp seems very dead.

1: Minus symbols as variable names are highlighted inconsistently 2: Some internal functions like map are highlighted, whereas others like pair are not (maybe this depends what Lisp dialect is in use) 3: define isn't highlighted

Another note I just found: null? highlights the null section purple but the qmark remains grey/white +: rest is highlighted as a function like map

[michaelblyons]

(define (square x)
    (* x x))

(define (square-tree tree)
    (map (lambda (sub-tree)
        (if (pair? sub-tree)
            (square-tree sub-tree)
            (square sub-tree)))
    tree)

Lisp (mostly) still has a "highlight the keywords" structure to its syntax definition file, rather than the stack of contexts that most of the recent language rewrites have used. On the plus side, Lisp's elegance probably makes it relatively easy to rewrite.

[rotty]

I've noticed this issue as well, and especially the dashes-are-operators-even-in-identifiers problem is extremely annoying. I'm not a Sublime user, but the Rust ecosystem syntax highlighting support is based on Sublime's syntax definitions, via the syntect crate, which in turn is used by the static site generator I happen to be using.

@michaelblyons Could you give some guidance on what "modern" syntax definitions to base a Scheme/CL syntax definition on? I'd be interested at giving it a go, probably focusing on Scheme first.

[michaelblyons]

I don't know what syntax definition might be closest to Lisp/Scheme, mostly because I don't use any languages that look like them. :wink: In my opinion, Javascript is the syntax definition that is most meticulously maintained. Perl had a fairly recent rewrite. There's an overhaul of Haskell in PR #2225.

The Sublime Text Discord is happy to help with questions, and the pointers in #757 may help you out. (I believe someone specifically mentions syntect --debug as being helpful to them.)

[jrappen]

[Thom1729]

I'd also recommend taking a look at the Lua syntax definition, which has a similar structure to the JavaScript definition but is much simpler.

The Lisp definition should be easy to rewrite because Lisp's syntax is so simple. You could probably write a no-frills highlighter in an hour. A complication is that basic features like control flow and function definitions are implemented as (essentially) library functions, and they vary from dialect to dialect. I'm not sure whether the existing Lisp definition was written for Scheme, Common Lisp, both, or neither, or to what extent a single definition could cover a variety of Lisp dialects.

[rotty]

@michaelblyons Thanks for the pointers -- I was looking for some maintained/modern syntax definition, not something I could directly base a Scheme one on, so answer is indeed quite helpful.

@Thom1729 So it seems Lua would be my go-to place when I get stuck. Thanks for the hint. I agree that a basic syntax definition should be pretty easy, but at least for Scheme, I'll go for covering a superset of R6RS, with some common non-conflicting extensions. While this is probably very simple as far as languages go, I suspect it's not trivial, if you want to cover all the syntax that can be reasonably reflected in a Sublime syntax definition. Regarding "keywords": they indeed do not exist in Scheme: there are just identifiers, some of which happen to be bound to syntax (aka macros), and they can be imported under different names, so the highlighting is necessarily an approximation. However, that's good enough™, and I think even die-hard Scheme programmers will not expect more -- even Emacs, a major Lisp IDE, relies on an approximate approach for highlighting.

Regarding sharing syntax definitions between Common Lisp (CL) and Scheme, I'm pretty sure the existing "Lisp" definition is for CL, and it would be a bad fit for Scheme, even if ignoring the obvious flaws that apply to both dialects. The names and semantics of available syntax ("keywords" in the above Sublime sense) vary wildly between them, and that's not the only difference. Some basic stuff is totally common, e.g. round brackets for structure, and the use of the semicolon for comments, but, just off the top of my head (and note that I know CL only from hearsay):

• In standard Scheme, there are no keywords (note that these are not your ususal keywords), but some Scheme implementations have them anyway, e.g., CL has :foo and Guile Scheme supports #:foo, and depending on reader settings, :foo and foo: as well.
• The way square brackets are used differs between standard Scheme -- if allowed at all (R6RS), they are used equivalently to round brackets, used by some authors/communtities for stylistical purposes, and eschewed by others. Emacs Lisp uses them to denote vectors, and in CL they might be re-defined by the user by modifying the read syntax, but are not part of the "core syntax".
• Scheme has a separate boolean type, and literals #t and #f, while CL just has nil, every other value counts as true.

This list is for sure not exhaustive.

So, it would be helpful for me, and the next person to come along, who, like me, may have zero experience with Sublime syntax definitions, to give some hints how to structure the support for the Lisp family, and if it seems sensible to share code (YAML, code is data yadayada :wink:) between the definitions at all, and if so, how.

Thinking about it, I will probably aim for the following in a first shot:

• Create a separate definition for Scheme, and leave the "Lisp" definition alone, save for removing the file suffixes commonly associated with Scheme, and not CL and other Lisps (Emacs Lisp being the third major branch of Lisp relevant today).
• Submit a PR, which will (given the above) will not fix this issue, at least for CL and Emacs Lisp.
• Based on review feedback, and feedback (hopefully!) given here, try to rework the "Lisp" syntax definition to something more reasonable, while glossing over details, as I'm not familiar with CL (althouh I am with Emacs Lisp).

How does that sound?

[rotty]

I'm not sure what the intent of this comment was, but FWIW, I've put discord on my personal blacklist of services to never use, due to a very bad signup experience, which (would have) required me to reveal my phone number in order to sign up.

[michaelblyons]

I think @jrappen was linking Discord, which I failed to do when I mentioned it. The same people also frequent a Discourse-based forum if that's your preference.

---

Your plan sounds good to me. If there are matches or contexts shared between Lisp dialects that are self-contained (or do not need to push or set any dialect-specific bits), you can centralize them in a single file and refer to them in the various dialects' syntax definitions. I'm not sure quite how relevant to Lisp this is, though, since my perception is that almost everything will push the same context that contains everything.

Also, if Thom contradicts anything I've said, go with Thom. 😉

[rotty]

I think @jrappen was linking Discord, which I failed to do when I mentioned it. The same people also frequent a Discourse-based forum if that's your preference.

Ah yes, it's clear in retrospect, sorry for knee-jerking. Thanks for pointing out the forum, the existence of discourse as an alternative is indeed much appreciated.

[gekoke]

See also https://github.com/dandavison/delta/issues/1580

[michaelblyons]

Feel free to fork #2387. I don't know when I'll be able to return to it.