tm4e
tm4e copied to clipboard
eclipse
IGrammar.tokenizeLine not considering TextMate format strings properly
The output of the given test is currently:
{startIndex: 0, endIndex: 2, scopes: [text.html.markdown, markup.heading.${1/(#)(#)?(#)?(#)?(#)?(#)?/${6:?
6:${5:?5:${4:?4:${3:?3:${2:?2:1}}}}}/}.markdown, punctuation.definition.heading.markdown]}
{startIndex: 2, endIndex: 3, scopes: [text.html.markdown, markup.heading.${1/(#)(#)?(#)?(#)?(#)?(#)?/${6:?6:${5:?5:${4:?4:${3:?3:${2:?2:1}}}}}/}.markdown]}
{startIndex: 3, endIndex: 4, scopes: [text.html.markdown, markup.heading.${1/(#)(#)?(#)?(#)?(#)?(#)?/${6:?6:${5:?5:${4:?4:${3:?3:${2:?2:1}}}}}/}.markdown, entity.name.section.markdown]}
But it should actually replace
${1/(#)(#)?(#)?(#)?(#)?(#)?/${6:?6:${5:?5:${4:?4:${3:?3:${2:?2:1}}}}}/}
by a 2 in this case (actually, the number of # in the parsed text) -- i.e.: markup.heading.2.markdown
The TextMate manual says that to support this
In Format Strings
$0-n
\\U, \\L, \\E, \\u, \\l
\\t, \\r, \\n, \\x{HHHH}, \\xHH
<variables>
(?<var>:<if>:<else>}
(?<var>:<if>}
In Both
Variables
${<var>:?<if>:<else>}
${<var>:+<if>}
${<var>:-<else>}
${<var>:<else>}
${<var>/<regexp>/<format>/<options>}
${<var>:[/upcase][/downcase][/capitalize][/asciify]}
Code to reproduce below (grammar gotten from https://github.com/textmate/markdown.tmbundle).
It's possible that I'm just misreading the API and there's some post-processing to do after the match. If this is the case, please let me know.
import java.io.ByteArrayInputStream;
import org.eclipse.tm4e.core.grammar.IGrammar;
import org.eclipse.tm4e.core.grammar.ITokenizeLineResult;
import org.eclipse.tm4e.core.registry.Registry;
import junit.framework.TestCase;
public class Tm4eIntegrationTest extends TestCase{
public void testTM4eMarkdownIntegration() throws Exception {
Registry registry = new Registry();
ByteArrayInputStream stream = new ByteArrayInputStream(("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
"<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" +
"<plist version=\"1.0\">\n" +
"<dict>\n" +
" <key>fileTypes</key>\n" +
" <array>\n" +
" <string>md</string>\n" +
" <string>mdown</string>\n" +
" <string>markdown</string>\n" +
" <string>markdn</string>\n" +
" </array>\n" +
" <key>keyEquivalent</key>\n" +
" <string>^~M</string>\n" +
" <key>name</key>\n" +
" <string>Markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#block</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>repository</key>\n" +
" <dict>\n" +
" <key>block</key>\n" +
" <dict>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#separator</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#heading</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#blockquote</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#lists</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#raw_block</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-def</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#html</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#paragraph</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>repository</key>\n" +
" <dict>\n" +
" <key>blockquote</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(^|\\G)(>) ?</string>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.quote.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>name</key>\n" +
" <string>markup.quote.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#block</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>while</key>\n" +
" <string>(^|\\G)(>) ?</string>\n" +
" </dict>\n" +
" <key>heading</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(?:^|\\G)(#{1,6})\\s*(?=[\\S[^#]])</string>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.heading.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>contentName</key>\n" +
" <string>entity.name.section.markdown</string>\n" +
" <key>end</key>\n" +
" <string>\\s*(#{1,6})?$\\n?</string>\n" +
" <key>name</key>\n" +
" <string>markup.heading.${1/(#)(#)?(#)?(#)?(#)?(#)?/${6:?6:${5:?5:${4:?4:${3:?3:${2:?2:1}}}}}/}.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#inline</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <key>heading-setext</key>\n" +
" <dict>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>match</key>\n" +
" <string>^(={3,})(?=[ \\t]*$\\n?)</string>\n" +
" <key>name</key>\n" +
" <string>markup.heading.setext.1.markdown</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>match</key>\n" +
" <string>^(-{3,})(?=[ \\t]*$\\n?)</string>\n" +
" <key>name</key>\n" +
" <string>markup.heading.setext.2.markdown</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <key>html</key>\n" +
" <dict>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(?i)(^|\\G)(?=<(p|div|h[1-6]|blockquote|pre|table|dl|ol|ul|script|noscript|form|fieldset|iframe|math|ins|del).*</\\2\\s*>\\s*$)</string>\n" +
" <key>end</key>\n" +
" <string>$</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>text.html.basic</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(?i)(^|\\G)(?=<(p|div|h[1-6]|blockquote|pre|table|dl|ol|ul|script|noscript|form|fieldset|iframe|math|ins|del))</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>text.html.basic</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>while</key>\n" +
" <string>\\G(?!</\\2\\s*>)</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <key>link-def</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.markdown</string>\n" +
" </dict>\n" +
" <key>10</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>11</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.title.markdown</string>\n" +
" </dict>\n" +
" <key>12</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>13</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>constant.other.reference.link.markdown</string>\n" +
" </dict>\n" +
" <key>3</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.separator.key-value.markdown</string>\n" +
" </dict>\n" +
" <key>5</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" <key>6</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>markup.underline.link.markdown</string>\n" +
" </dict>\n" +
" <key>7</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" <key>8</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.title.markdown</string>\n" +
" </dict>\n" +
" <key>9</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>^(?x:\n" +
" \\s* # Leading whitespace\n" +
" (\\[)(.+?)(\\])(:) # Reference name\n" +
" [ \\t]* # Optional whitespace\n" +
" (<?)(\\S+?)(>?) # The url\n" +
" [ \\t]* # Optional whitespace\n" +
" (?:\n" +
" ((\\().+?(\\))) # Match title in quotes…\n" +
" | ((\").+?(\")) # or in parens.\n" +
" )? # Title is optional\n" +
" \\s* # Optional whitespace\n" +
" $\n" +
" )</string>\n" +
" <key>name</key>\n" +
" <string>meta.link.reference.def.markdown</string>\n" +
" </dict>\n" +
" <key>list_paragraph</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(^|\\G)(?=\\S)(?![*+-]\\s|[0-9]+\\.\\s)</string>\n" +
" <key>name</key>\n" +
" <string>meta.paragraph.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>text.html.basic</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#heading-setext</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>while</key>\n" +
" <string>(^|\\G)(?!\\s*$|#|[ ]{0,3}([-*_][ ]{2,}){3,}[ \\t]*$\\n?|>|[ ]{0,3}[*+-]|[ ]{0,3}[0-9]+\\.)</string>\n" +
" </dict>\n" +
" <key>lists</key>\n" +
" <dict>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(^|\\G)([ ]{0,3})([*+-])([ ]{1,3}|\\t)</string>\n" +
" <key>beginCaptures</key>\n" +
" <dict>\n" +
" <key>3</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.list.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>comment</key>\n" +
" <string>Currently does not support un-indented second lines.</string>\n" +
" <key>name</key>\n" +
" <string>markup.list.unnumbered.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#list_paragraph</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#block</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>while</key>\n" +
" <string>\\G([ ]{4}|\\t|$)</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(^|\\G)([ ]{0,3})([0-9]+\\.)([ ]{1,3}|\\t)</string>\n" +
" <key>beginCaptures</key>\n" +
" <dict>\n" +
" <key>3</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.list.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>name</key>\n" +
" <string>markup.list.numbered.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#list_paragraph</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#block</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>while</key>\n" +
" <string>\\G([ ]{4}|\\t|$)</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <key>paragraph</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(^|\\G)(?=\\S)</string>\n" +
" <key>name</key>\n" +
" <string>meta.paragraph.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>text.html.basic</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#heading-setext</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>while</key>\n" +
" <string>(^|\\G)(?!\\s*$|#|[ ]{0,3}([-*_][ ]{2,}){3,}[ \\t]*$\\n?|\\s*\\[.+?\\]:|>)</string>\n" +
" </dict>\n" +
" <key>raw_block</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(^|\\G)([ ]{4}|\\t)</string>\n" +
" <key>name</key>\n" +
" <string>markup.raw.block.markdown</string>\n" +
" <key>while</key>\n" +
" <string>(^|\\G)([ ]{4}|\\t)</string>\n" +
" </dict>\n" +
" <key>separator</key>\n" +
" <dict>\n" +
" <key>match</key>\n" +
" <string>(^|\\G)[ ]{0,3}([-*_])([ ]{0,2}\\2){2,}[ \\t]*$\\n?</string>\n" +
" <key>name</key>\n" +
" <string>meta.separator.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>inline</key>\n" +
" <dict>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#ampersand</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#bracket</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#bold</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#italic</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#raw</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#escape</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#image-inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#image-ref</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-email</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-inet</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-ref</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-ref-literal</string>\n" +
" </dict>\n" +
" </array>\n" +
" <key>repository</key>\n" +
" <dict>\n" +
" <key>ampersand</key>\n" +
" <dict>\n" +
" <key>comment</key>\n" +
" <string>\n" +
" Markdown will convert this for us. We match it so that the\n" +
" HTML grammar will not mark it up as invalid.\n" +
" </string>\n" +
" <key>match</key>\n" +
" <string>&(?!([a-zA-Z0-9]+|#[0-9]+|#x[0-9a-fA-F]+);)</string>\n" +
" <key>name</key>\n" +
" <string>meta.other.valid-ampersand.markdown</string>\n" +
" </dict>\n" +
" <key>bold</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(?x)\n" +
" (\\*\\*|__)(?=\\S) # Open\n" +
" (?=\n" +
" (\n" +
" <[^>]*+> # HTML tags\n" +
" | (?<raw>`+)([^`]|(?!(?<!`)\\k<raw>(?!`))`)*+\\k<raw>\n" +
" # Raw\n" +
" | \\\\[\\\\`*_{}\\[\\]()#.!+\\->]?+ # Escapes\n" +
" | \\[\n" +
" ( \n" +
" (?<square> # Named group\n" +
" [^\\[\\]\\\\] # Match most chars\n" +
" | \\\\. # Escaped chars\n" +
" | \\[ \\g<square>*+ \\] # Nested brackets\n" +
" )*+\n" +
" \\]\n" +
" (\n" +
" ( # Reference Link\n" +
" [ ]? # Optional space\n" +
" \\[[^\\]]*+\\] # Ref name\n" +
" )\n" +
" | ( # Inline Link\n" +
" \\( # Opening paren\n" +
" [ \\t]*+ # Optional whtiespace\n" +
" <?(.*?)>? # URL\n" +
" [ \\t]*+ # Optional whtiespace\n" +
" ( # Optional Title\n" +
" (?<title>['\"])\n" +
" (.*?)\n" +
" \\k<title>\n" +
" )?\n" +
" \\)\n" +
" )\n" +
" )\n" +
" )\n" +
" | (?!(?<=\\S)\\1). # Everything besides\n" +
" # style closer\n" +
" )++\n" +
" (?<=\\S)\\1 # Close\n" +
" )\n" +
" </string>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.bold.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>end</key>\n" +
" <string>(?<=\\S)(\\1)</string>\n" +
" <key>name</key>\n" +
" <string>markup.bold.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>applyEndPatternLast</key>\n" +
" <integer>1</integer>\n" +
" <key>begin</key>\n" +
" <string>(?=<[^>]*?>)</string>\n" +
" <key>end</key>\n" +
" <string>(?<=>)</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>text.html.basic</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#escape</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#ampersand</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#bracket</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#raw</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#italic</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#image-inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-inet</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-email</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#image-ref</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-ref-literal</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-ref</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <key>bracket</key>\n" +
" <dict>\n" +
" <key>comment</key>\n" +
" <string>\n" +
" Markdown will convert this for us. We match it so that the\n" +
" HTML grammar will not mark it up as invalid.\n" +
" </string>\n" +
" <key>match</key>\n" +
" <string><(?![a-z/?\\$!])</string>\n" +
" <key>name</key>\n" +
" <string>meta.other.valid-bracket.markdown</string>\n" +
" </dict>\n" +
" <key>escape</key>\n" +
" <dict>\n" +
" <key>match</key>\n" +
" <string>\\\\[-`*_#+.!(){}\\[\\]\\\\>]</string>\n" +
" <key>name</key>\n" +
" <string>constant.character.escape.markdown</string>\n" +
" </dict>\n" +
" <key>image-inline</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>10</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.title.markdown</string>\n" +
" </dict>\n" +
" <key>11</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.markdown</string>\n" +
" </dict>\n" +
" <key>12</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.markdown</string>\n" +
" </dict>\n" +
" <key>13</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.title.markdown</string>\n" +
" </dict>\n" +
" <key>14</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.markdown</string>\n" +
" </dict>\n" +
" <key>15</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.markdown</string>\n" +
" </dict>\n" +
" <key>16</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.metadata.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>5</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>invalid.illegal.whitespace.markdown</string>\n" +
" </dict>\n" +
" <key>6</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.metadata.markdown</string>\n" +
" </dict>\n" +
" <key>7</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" <key>8</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>markup.underline.link.image.markdown</string>\n" +
" </dict>\n" +
" <key>9</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(?x:\n" +
" \\! # Images start with !\n" +
" (\\[)((?<square>[^\\[\\]\\\\]|\\\\.|\\[\\g<square>*+\\])*+)(\\]) \n" +
" # Match the link text.\n" +
" ([ ])? # Space not allowed\n" +
" (\\() # Opening paren for url\n" +
" (<?)(\\S+?)(>?) # The url\n" +
" [ \\t]* # Optional whitespace\n" +
" (?:\n" +
" ((\\().+?(\\))) # Match title in parens…\n" +
" | ((\").+?(\")) # or in quotes.\n" +
" )? # Title is optional\n" +
" \\s* # Optional whitespace\n" +
" (\\))\n" +
" )</string>\n" +
" <key>name</key>\n" +
" <string>meta.image.inline.markdown</string>\n" +
" </dict>\n" +
" <key>image-ref</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>5</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.markdown</string>\n" +
" </dict>\n" +
" <key>6</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>constant.other.reference.link.markdown</string>\n" +
" </dict>\n" +
" <key>7</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>\\!(\\[)((?<square>[^\\[\\]\\\\]|\\\\.|\\[\\g<square>*+\\])*+)(\\])[ ]?(\\[)(.*?)(\\])</string>\n" +
" <key>name</key>\n" +
" <string>meta.image.reference.markdown</string>\n" +
" </dict>\n" +
" <key>italic</key>\n" +
" <dict>\n" +
" <key>begin</key>\n" +
" <string>(?x)\n" +
" (\\*|_)(?=\\S) # Open\n" +
" (?=\n" +
" (\n" +
" <[^>]*+> # HTML tags\n" +
" | (?<raw>`+)([^`]|(?!(?<!`)\\k<raw>(?!`))`)*+\\k<raw>\n" +
" # Raw\n" +
" | \\\\[\\\\`*_{}\\[\\]()#.!+\\->]?+ # Escapes\n" +
" | \\[\n" +
" ( \n" +
" (?<square> # Named group\n" +
" [^\\[\\]\\\\] # Match most chars\n" +
" | \\\\. # Escaped chars\n" +
" | \\[ \\g<square>*+ \\] # Nested brackets\n" +
" )*+\n" +
" \\]\n" +
" (\n" +
" ( # Reference Link\n" +
" [ ]? # Optional space\n" +
" \\[[^\\]]*+\\] # Ref name\n" +
" )\n" +
" | ( # Inline Link\n" +
" \\( # Opening paren\n" +
" [ \\t]*+ # Optional whtiespace\n" +
" <?(.*?)>? # URL\n" +
" [ \\t]*+ # Optional whtiespace\n" +
" ( # Optional Title\n" +
" (?<title>['\"])\n" +
" (.*?)\n" +
" \\k<title>\n" +
" )?\n" +
" \\)\n" +
" )\n" +
" )\n" +
" )\n" +
" | \\1\\1 # Must be bold closer\n" +
" | (?!(?<=\\S)\\1). # Everything besides\n" +
" # style closer\n" +
" )++\n" +
" (?<=\\S)\\1 # Close\n" +
" )\n" +
" </string>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.italic.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>end</key>\n" +
" <string>(?<=\\S)(\\1)((?!\\1)|(?=\\1\\1))</string>\n" +
" <key>name</key>\n" +
" <string>markup.italic.markdown</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>applyEndPatternLast</key>\n" +
" <integer>1</integer>\n" +
" <key>begin</key>\n" +
" <string>(?=<[^>]*?>)</string>\n" +
" <key>end</key>\n" +
" <string>(?<=>)</string>\n" +
" <key>patterns</key>\n" +
" <array>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>text.html.basic</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#escape</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#ampersand</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#bracket</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#raw</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#bold</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#image-inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-inline</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-inet</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-email</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#image-ref</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-ref-literal</string>\n" +
" </dict>\n" +
" <dict>\n" +
" <key>include</key>\n" +
" <string>#link-ref</string>\n" +
" </dict>\n" +
" </array>\n" +
" </dict>\n" +
" <key>link-email</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>markup.underline.link.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(<)((?:mailto:)?[-.\\w]+@[-a-z0-9]+(\\.[-a-z0-9]+)*\\.[a-z]+)(>)</string>\n" +
" <key>name</key>\n" +
" <string>meta.link.email.lt-gt.markdown</string>\n" +
" </dict>\n" +
" <key>link-inet</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>markup.underline.link.markdown</string>\n" +
" </dict>\n" +
" <key>3</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(<)((?:https?|ftp)://.*?)(>)</string>\n" +
" <key>name</key>\n" +
" <string>meta.link.inet.markdown</string>\n" +
" </dict>\n" +
" <key>link-inline</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>10</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.title.markdown</string>\n" +
" </dict>\n" +
" <key>11</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>12</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>13</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.description.title.markdown</string>\n" +
" </dict>\n" +
" <key>14</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>15</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>16</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.metadata.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.title.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>5</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>invalid.illegal.whitespace.markdown</string>\n" +
" </dict>\n" +
" <key>6</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.metadata.markdown</string>\n" +
" </dict>\n" +
" <key>7</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" <key>8</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>markup.underline.link.markdown</string>\n" +
" </dict>\n" +
" <key>9</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.link.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(?x:\n" +
" (\\[)((?<square>[^\\[\\]\\\\]|\\\\.|\\[\\g<square>*+\\])*+)(\\]) \n" +
" # Match the link text.\n" +
" ([ ])? # Space not allowed\n" +
" (\\() # Opening paren for url\n" +
" (<?)(.*?)(>?) # The url\n" +
" [ \\t]* # Optional whitespace\n" +
" (?:\n" +
" ((\\().+?(\\))) # Match title in parens…\n" +
" | ((\").+?(\")) # or in quotes.\n" +
" )? # Title is optional\n" +
" \\s* # Optional whitespace\n" +
" (\\))\n" +
" )</string>\n" +
" <key>name</key>\n" +
" <string>meta.link.inline.markdown</string>\n" +
" </dict>\n" +
" <key>link-ref</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.title.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>5</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.begin.markdown</string>\n" +
" </dict>\n" +
" <key>6</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>constant.other.reference.link.markdown</string>\n" +
" </dict>\n" +
" <key>7</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.end.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(\\[)((?<square>[^\\[\\]\\\\]|\\\\.|\\[\\g<square>*+\\])*+)(\\])[ ]?(\\[)([^\\]]*+)(\\])</string>\n" +
" <key>name</key>\n" +
" <string>meta.link.reference.markdown</string>\n" +
" </dict>\n" +
" <key>link-ref-literal</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.begin.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>string.other.link.title.markdown</string>\n" +
" </dict>\n" +
" <key>4</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.string.end.markdown</string>\n" +
" </dict>\n" +
" <key>5</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.begin.markdown</string>\n" +
" </dict>\n" +
" <key>6</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.constant.end.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(\\[)((?<square>[^\\[\\]\\\\]|\\\\.|\\[\\g<square>*+\\])*+)(\\])[ ]?(\\[)(\\])</string>\n" +
" <key>name</key>\n" +
" <string>meta.link.reference.literal.markdown</string>\n" +
" </dict>\n" +
" <key>raw</key>\n" +
" <dict>\n" +
" <key>captures</key>\n" +
" <dict>\n" +
" <key>1</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.raw.markdown</string>\n" +
" </dict>\n" +
" <key>2</key>\n" +
" <dict/>\n" +
" <key>3</key>\n" +
" <dict>\n" +
" <key>name</key>\n" +
" <string>punctuation.definition.raw.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>match</key>\n" +
" <string>(`+)([^`]|(?!(?<!`)\\1(?!`))`)*+(\\1)</string>\n" +
" <key>name</key>\n" +
" <string>markup.raw.inline.markdown</string>\n" +
" </dict>\n" +
" </dict>\n" +
" </dict>\n" +
" </dict>\n" +
" <key>scopeName</key>\n" +
" <string>text.html.markdown</string>\n" +
" <key>uuid</key>\n" +
" <string>0A1D9874-B448-11D9-BD50-000D93B6E43C</string>\n" +
"</dict>\n" +
"</plist>").getBytes("utf-8"));
IGrammar grammar = registry.loadGrammarFromPathSync(".tmLanguage",stream);
ITokenizeLineResult result = grammar.tokenizeLine("## 1");
for (int i = 0; i < result.getTokens().length; i++) {
System.err.println(result.getTokens()[i]);
}
}
}
@fabioz you consume the API correctly. I had written quicly that at https://github.com/eclipse/tm4e/wiki/Core
In short answern this issue seems to be a bug.
But VSCode textmate seems have the same problem. In their VSCode installation, they use a markdown grammar with simple declaration markup.heading.markdown https://github.com/Microsoft/vscode/blob/master/extensions/markdown/syntaxes/markdown.tmLanguage#L289
instead of markup.heading.${1/(#)(#)?(#)?(#)?(#)?(#)?/${6:?6:${5:?5:${4:?4:${3:?3:${2:?2:1}}}}}/}.markdown
https://github.com/textmate/markdown.tmbundle/blob/master/Syntaxes/Markdown.tmLanguage#L105
To be honnest with you, I'm not a big expert with TextMate. I have just translated the vscode-textmate written in TypeScript to Java to have the same feature than VSCode (I have almost the same tests).
The best thing to fix this issue is to create an issue at https://github.com/Microsoft/vscode-textmate/issues My API is the same than vscode-textmate.
Hope you will understand.
