Expose scope comments in IToken

[Copilot]

TextMate grammars support a comment field on rules for documentation purposes. This field was being parsed but not exposed to consumers, making it inaccessible to tools like VS Code's scope inspector.

Changes

Type System

• Added optional comment field to IRawRule interface
• Added scopeComments: (string | null)[] to IToken interface, parallel to existing scopes array

Rule Storage

• Updated Rule base class and all subclasses (CaptureRule, MatchRule, IncludeOnlyRule, BeginEndRule, BeginWhileRule) to store and expose comments via getComment() method
• Modified RuleFactory to extract comment field from grammar rules during compilation

Scope Stack

• Extended ScopeStack to track scopeComment alongside scopeName, added getComments() method
• Updated AttributedScopeStack.pushAttributed() to accept and propagate comment parameter
• For multi-scope pushes (space-separated scopes), comment is assigned to first scope only

Tokenization

• Modified tokenization logic to retrieve comments from rules and pass through pushAttributed() calls
• Token creation now includes scope comments array matching scope hierarchy

Example

// Grammar with comments
{
  "match": "\\b(hello)\\b",
  "name": "keyword.test",
  "comment": "Matches the hello keyword"
}

// Resulting token
{
  scopes: ["source.test", "keyword.test"],
  scopeComments: [null, "Matches the hello keyword"]
}

Comments default to null when not present, maintaining backward compatibility.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Expose scope comment in IToken</issue_title> <issue_description>There's a feature request in VS Code to include grammar scope comments in https://github.com/microsoft/vscode/issues/206192. First, the scope comments need to be exposed in IToken in vscode-textmate. </issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes microsoft/vscode-textmate#227

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.