[JSON] Rewrite syntax

[jrappen]

• Using inheritance split up JSON.sublime-syntax into:
  • JSON.sublime-syntax with scope:source.json
  • JSONC.sublime-syntax with scope:source.json.jsonc
  • JSON5.sublime-syntax with scope:source.json.json5
  • JSON_dotNET.sublime-syntax with scope:source.json.json-dotnet
• Add many more file extensions for JSON & JSONC:
  • add doc links to extensions where applicable as a reference to be able to more quickly verify that they (still) use said syntax flavor
• JSON:
  • Make use of newer syntax features including those only available in version: 2 syntaxes
  • Make use of variables (with optimizations provided by @deathaxe and regex patterns provided by @Thom1729)
  • Context names now more closely match the naming scheme of other (recently re-written) default syntaxes
  • (correctly formatted) JSON code can now be prettified or minified via the context menu or the command palette. JSON code can optionally be auto-prettified on pre save events.
  • highlight leading, trailing & multiple commas as invalid
  • only allow exactly one structure (object, array) or value (constant, number, string) at top level (thanks to @keith-hall)
  • links (meta.link.inet) and email addresses (meta.link.email) are scoped the same as in Markdown (thanks to @deathaxe)
• JSONC:
  • highlight some files by default as JSONC (as decided by @jskinner in sublimehq/Packages#285)
  • highlight leading & multiple commas as invalid, trailing as valid
  • scope empty block comments as such
  • support syntax based folding of ST4131+, compare sublimehq/Packages#3291
• JSON5:
  • explicitly pos numbers, hexadecimal ints, Infinity and NaN
  • single quoted strings
  • more escape chars for strings
  • ECMA identifierName as object keys (regexes thanks to @Thom1729)
    • scoped as plain unquoted strings (thanks to @Thom1729)
    • support string interpolation (thanks to @deathaxe)
  • line continuation in strings (with tests thanks to @keith-hall)
• JSON.NET:
  • support requested by @keith-hall, built with feedback from @michaelblyons
• Objects:
  • Highlighting speed improvements for empty objects (thanks to @FichteFoll)
  • Make mapping.* contexts more modular
• Arrays:
  • Highlighting speed improvements for empty arrays (thanks to @FichteFoll)
• Numbers:
  • Correctly scope number signs with constant.numeric.sign instead of keyword.operator.arithmetic
  • Significantly improve number highlighting (thanks to @deathaxe)
• Completions:
  • completions have been added for language constants, including kind info and details (with links to docs)
    • null, false, true for JSON
    • Infinity and NaN for JSON5
• Settings:
  • a default_extension is now set for all JSON flavors
• Symbol index:
  • with an object structure at the top-level, only top-level keys within now show up in the index (including tests for symbols and syntax)
• Tests:
  • test files now test the base scope
  • Significantly extend tests to cover more parts of the syntaxes
  • Split original test file into logical parts
  • Add indentation tests for:
    • json, json5 & jsonc
    • mapping (objects), sequence (arrays)
  • Add symbols tests for:
    • top-level keys of object structures (thanks to deathaxe)
    • languages: json, json5 & jsonc
  • Fix tests for meta.mapping meta.mapping.*
• Leave JSON headers in Markdown as json only, but split up fenced code blocks into json, json5 & jsonc to behave similarly to GitHub Flavored Markdown

BREAKING CHANGES:

• JSON does not have values that can be set via an inline calculation with the help of operators, but only simple number values. Scopes for number signs have changed from being keyword.operator.arithmetic to constant.numeric.sign. Color scheme authors should add this, should it be missing.

• The JSON.sublime-syntax now marks comments as invalid, third party plugin authors should instead target JSONC.sublime-syntax to keep the user experience as-is.

• Indexed symbols (i.e. top-level keys in JSON object structures) are scoped as source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key). Color scheme authors should add special highlighting to differentiate them from other keys.

• fix sublimehq/Packages#285

• address sublimehq/Packages#421 (thanks to @FichteFoll)

• address sublimehq/Packages#481 to remove incompatible regex patterns according to @wbond

• address sublimehq/Packages#757 to fix line comments for JSONC (thanks to @keith-hall)

• address sublimehq/Packages#2430 using sort-order (as requested by @deathaxe)

• address sublimehq/Packages#2711 with regards to constant.language.null vs. constant.language.empty (thanks to @FichteFoll)

• address sublimehq/Packages#2852 to fix scopes of curly braces & square brackets in JSON (thanks to @Thom1729)

• address sublimehq/Packages#3228 to fix punctuation.separator scopes, compare sublimehq/Packages#3270

• address sublimehq/sublime_text#3154 and add symbol tests

Co-authored-by: Ashwin Shenoy [email protected] Co-authored-by: Jack Cherng [email protected] Co-authored-by: Janos Wortmann [email protected] Co-authored-by: Jon Skinner [email protected] Co-authored-by: FichteFoll [email protected] Co-authored-by: Keith Hall [email protected] Co-authored-by: Michael B. Lyons [email protected] Co-authored-by: Rafał Chłodnicki [email protected] Co-authored-by: Thomas Smith [email protected] Co-authored-by: Will Bond [email protected] Co-authored-by: deathaxe [email protected]

[jrappen]

⚠️ This is a first step to show my current work in progress.

There is still a lot of work to do. Please comment below with suggestions.

[jrappen]

As @jskinner requested in #285, this PR applies JSONC to *.json files by default to not confuse beginners too much who do not know how to set a syntax for a view.

[michaelblyons]

What do you think about using source.json and source.json.jsonc? I worry a little that moving the primary *.json association to source.jsonc will break a bunch of people and packages' configuration scopes.

[jrappen]

Sounds reasonable.

I haven't looked at other packages that either use JSON or are affected by this change, yet.

[jfcherng]

I haven't looked at other packages that either use JSON or are affected by this change, yet.

Like I have one in my custom keybindings via selector context.

    // plugin: JsPrettier
    {
        "keys": ["ctrl+alt+f"],
        "command": "js_prettier",
        "context": [
            {
                "key": "selector",
                "operator": "equal",
                "operand": "text.html.basic | text.html.markdown | text.html.vue | source.js | source.json | source.css | source.scss | source.less | source.ts | source.tsx | source.yaml"
            }
        ]
    },

[jrappen]

Ok, follow up question, should I change stuff like:

scope: comment.line.double-slash.jsonc

to

scope: comment.line.double-slash.json.jsonc

for JSONC.sublime-syntax?

[michaelblyons]

Ok, follow up question, should I change stuff like:

scope: comment.line.double-slash.jsonc

to

scope: comment.line.double-slash.json.jsonc

for JSONC.sublime-syntax?

That's disputed. One camp says "yes." One camp says "no." One camp actually says "Change the last . to a -."

In this instance, I am in the "keep it as .jsonc" camp. Closest thing in default packages is the GitHub markdown extensions.

[deathaxe]

Agree. The backward compatibility concerns are mainly tarteting the syntax's main scope here.

If a derived/extending syntax adds more patterns, it's ok to only add it's trailing part of a main scope.

So I am also in the "keep it as .jsonc" camp.

[deathaxe]

JSON with Comments (jsonc) is just an extension of JSON. I don't therefore see any value in adding a JSON (Basic) which JSON is derived from by just adding some file extensions.

[jrappen]

JSON with Comments (jsonc) is just an extension of JSON. I don't therefore see any value in adding a JSON (Basic) which JSON is derived from by just adding some file extensions.

I'm still undecided if I should add more flavors in this step ... or do that later. This was just some preparation for (possibly) later.

[deathaxe]

I'm still undecided if I should add more flavors in this step ... or do that later. This was just some preparation for (possibly) later.

Haven't investigated any other flavours nor being aware of them, but does that change anything?

The default JSON.sublime-syntax could still be the base for everything with its source.json main scope, no? It shouldn't even hurt to add the .basic and keep it as is.

So we'd end up in:

file	scope
JSON.sublime-syntax	source.json.basic
JSONC.sublime-syntax	source.json.jsonc
JSON (<flavor>).sublime-syntax	source.json.<flavor>

[jrappen]

Probably not. Unless I have to extract bits from JSON (Basic) to JSON. It's easier to merge than separate them later.

[michaelblyons]

It shouldn't even hurt to add the .basic and keep it as is.

What happens when another syntax - include: scope:source.json if you change the root scope to source.json.basic? Will something be included? If so, what?

[jrappen]

If so, what?

In that case I'd assume nothing.

[deathaxe]

You are right @michaelblyons. Nothing is included then. So it's not an option.

[jrappen]

Could use this instead, though:

- include: scope:Packages/JSON/JSON.sublime-syntax

unless I'm mistaken. I have a vague memory seeing this somewhere.

[michaelblyons]

Could use this instead, though:

- include: scope:Packages/JSON/JSON.sublime-syntax

unless I'm mistaken. I have a vague memory seeing this somewhere.

I know you can do this within a package. I believe Git Formats does so. You may be able to do it between packages (e.g. Markdown front matter).

But switching away from source.json would still break other people's packages all over the internet.

[jrappen]

Sometimes, breaking stuff is the way forward. No need for that here, though.

[jrappen]

On the other hand, the package you linked from across the internet I will definitely break, cause he'd have to switch to scope:source.json.jsonc#prototype as JSON won't have that (i.e. comments) anymore.

[jrappen]

Does anybody know of any JSON flavors that do accept decimal floats with leading or trailing periods in the base?

In basic JSON they apparently are illegal (and weren't scoped as such, yet).

[jrappen]

⚠️ I have addressed and resolved all comments above, except for the one about meta_include_prototype.

---

Decimal floats are now forced to a more explicit format.

[jrappen]

Thoughts on embed: scope:source.json in:

• LaTeX
• Markdown
• Perl
• PHP Source

and push: scope:source.json in:

• PHP Source

anyone?

[jrappen]

For Markdown I believe (1) the frontmatter can be JSON and (2) the fenced blocks can be split up into json & jsonc.

GitHub flavored Markdown code blocks behaves similar (i.e. json dislikes comments).

[jwortmann]

Decimal floats are now forced to a more explicit format.

I would remove the rule with "invalid.illegal" highlighting for numbers with trailing decimal points. Otherwise you will get a short flash of that "illegal" highlighting each time you write a floating point number. I would suggest to use "invalid.illegal" only if an expression is in an unrecoverable state. Maybe even better to leave such things for linters in general. I think a good solution for this case would be to leave trailing dots after numbers unscoped.

[jrappen]

I would suggest to use "invalid.illegal" only if an expression is in an unrecoverable state. Maybe even better to leave such things for linters in general. I think a good solution for this case would be to leave trailing dots after numbers unscoped.

@jwortmann So you are basically suggesting, I add a look-ahead for a separator (,) or end of array (]) or end of object (}) to "make sure we are done typing"?

[deathaxe]

He suggests not to highlight incomplete numbers illegal, but keep trailing punctuation unscoped instead. I fully support his arguments. Syntax definitions should handle incomplete expressions/statements/tokens gracefully as it is probably the most common state while typing.

All the over eager illegal highlighting results in poor and annoying user experience. Of course a number may have a trailing dot, if a user has not finished writing it. Causing the whole number to flash red feels just terrible.

It's better to not highlight those parts if in doubt.

The old behavior to just highlight the decimal sign illegal is more than enough, IMHO.

[jwortmann]

So you are basically suggesting, I add a look-ahead for a separator (,) or end of array (]) or end of object (}) to "make sure we are done typing"?

This still won't work, because of auto-pairing brackets, i.e. you often have one of these characters to the right of the caret while typing. Better remove that rule for illegal trailing decimal points entirely.

[jrappen]

This still won't work, because of auto-pairing brackets, i.e. you often have one of these characters to the right of the caret while typing. Better remove that rule for illegal trailing decimal points entirely.

Yeah ok, I forgot about those. I first thought the new branching capabilities would've been usefule here. A solution to have both would be nice, though.

[jrappen]

⚠️ I have addressed and resolved all comments above.

[jrappen]

With regards to:

• embed: scope:source.json
• push: scope:source.json

in Packages other than JSON, I have:

• LaTeX:
  • left as is, because minted uses pygments / JsonLexer
• Markdown:
  • left JSON headers as json
  • split up fenced code blocks into json and jsonc (similar to GitHub Flavored Markdown)
    • added tests
• Perl:
  • left as is
• PHP:
  • left as is