C# file panics (too many pops)

[robinst]

How to reproduce:

1. Create a file Test.cs with the following content:

   $"test {"\"foo\""} test \"{bar}\""
   

2. Run cargo run --example syncat Test.cs

Expected: It works. Actual: It crashes, backtrace:

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:335:21
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::begin_panic
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
             at libstd/panicking.rs:537
   6: std::panicking::try::do_call
             at libstd/panicking.rs:521
   7: std::panicking::try::do_call
             at libstd/panicking.rs:497
   8: <core::ops::range::Range<Idx> as core::fmt::Debug>::fmt
             at libcore/panicking.rs:71
   9: <core::ops::range::Range<Idx> as core::fmt::Debug>::fmt
             at libcore/panicking.rs:51
  10: <core::option::Option<T>>::unwrap
             at /Users/travis/build/rust-lang/rust/src/libcore/macros.rs:20
  11: <core::cell::RefMut<'b, T> as core::ops::deref::DerefMut>::deref_mut
             at src/highlighting/highlighter.rs:118
  12: <core::cell::RefMut<'b, T> as core::ops::deref::DerefMut>::deref_mut
             at src/highlighting/highlighter.rs:142
  13: <alloc::vec::Vec<T>>::extend_desugared
             at /Users/travis/build/rust-lang/rust/src/liballoc/vec.rs:1921
  14: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::spec_extend
             at /Users/travis/build/rust-lang/rust/src/liballoc/vec.rs:1818
  15: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter
             at /Users/travis/build/rust-lang/rust/src/liballoc/vec.rs:1813
  16: <alloc::vec::Vec<T> as core::iter::traits::FromIterator<T>>::from_iter
             at /Users/travis/build/rust-lang/rust/src/liballoc/vec.rs:1713
  17: core::iter::iterator::Iterator::collect
             at /Users/travis/build/rust-lang/rust/src/libcore/iter/iterator.rs:1303
  18: <syntect::highlighting::style::_IMPL_DESERIALIZE_FOR_FontStyle::<impl serde::de::Deserialize<'de> for syntect::highlighting::style::FontStyle>::deserialize::__Visitor<'de> as serde::de::Visitor<'de>>::visit_seq
             at src/easy.rs:65
  19: syncat::main
             at examples/syncat.rs:110
  20: std::rt::lang_start::{{closure}}
             at /Users/travis/build/rust-lang/rust/src/libstd/rt.rs:74
  21: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:479
  22: panic_unwind::dwarf::eh::read_encoded_pointer
             at libpanic_unwind/lib.rs:102
  23: rust_panic
             at libstd/panicking.rs:458
             at libstd/panic.rs:358
             at libstd/rt.rs:58
  24: std::rt::lang_start
             at /Users/travis/build/rust-lang/rust/src/libstd/rt.rs:74
  25: syncat::main

Note that this seems to be caused by getting too many pops (no corresponding push). I wrote the following test that fails if you add it to parser.rs:

    #[test]
    fn pop_on_empty_stack() {
        let ps = SyntaxSet::load_from_folder("testdata/Packages").unwrap();
        let mut state = {
            let syntax = ps.find_syntax_by_name("C#").unwrap();
            ParseState::new(syntax)
        };

        let mut basic_stack = Vec::new();
        let line = r#"$"test {"\"foo\""} test \"{bar}\"""#;
        let mut stack = ScopeStack::new();
        for (_, op) in ops(&line, &mut state) {
            stack.apply_with_hook(&op, |basic_op, _| {
                match basic_op {
                    BasicScopeStackOp::Push(scope) => {
                        basic_stack.push(scope)
                    },
                    BasicScopeStackOp::Pop => {
                        basic_stack.pop().expect("Pop of empty stack!");
                    },
                }
            })
        }
    }

(Haven't tried to figure out where the bug is coming from yet.)

[keith-hall]

In case it helps, I cut down the C# syntax definition into what I believe is (almost) the smallest necessary for correct scoping of the test, and the panic didn't occur... So it seems likely to be an interplay of something in the rest of the syntax definition that is causing it somehow.

It's interesting that syncat doesn't panic when one of the interpolations is removed from the test, and that it doesn't panic with synhtml.

%YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
# Copyright (c) 2016 Sublime Text HQ Pty, @gwenzek,
#     Matthew Winter @wintermi, Adam Lickel @lickel
# MIT license: https://opensource.org/licenses/mit-license.php

name: C# String Interpolation Test
scope: source.cstest
file_extensions: [cstest]

variables:
  # characters
  unicode_char: '(?:\\u\h{4}|\\U\h{8})'
  escaped_char: '(?:\\[abfnrtv"''\\]|{{unicode_char}}|\\x[0-9a-fA-F]{1,4}|\\[0-9]{1,3})'

  visibility: \b(?:public|private|protected|internal|protected\s+internal)\b

  name: '(?:{{name_normal}})'

  start_char: '(?:{{unicode_char}}|[_\p{L}])'
  other_char: '(?:{{unicode_char}}|[_0-9\p{L}])'
  name_normal: '{{start_char}}{{other_char}}*\b'
  cap_name: '(\p{Lu}{{other_char}})'

contexts:
  main:
    # allows coloration of code outside a class
    - match: (?=\S)
      push:
        - match: (?={{visibility}}|\b(?:class|delegate|interface|namespace|readonly)\b)
          pop: true
        - include: line_of_code

  code_block_in:
    - match: (?=\S)
      push: line_of_code

  line_of_code:
    - match: '(?=\S)'
      set: line_of_code_in

  line_of_code_in:
    - include: line_of_code_in_no_semicolon
    - match: ';'
      scope: punctuation.terminator.statement.cs
      pop: true

  line_of_code_in_no_semicolon:
    # interpolated strings
    - match: '\$"'
      scope: punctuation.definition.string.begin.cs
      push: format_string
    # multi-line strings
    - match: '{{name}}'
      scope: variable.other.cs
    - include: literals
    - match: (?=\}|\)|>|\]|,)
      pop: true

  # bools, numbers, chars, simple strings
  literals:
    # strings
    - match: '"'
      scope: punctuation.definition.string.begin.cs
      push: string

  string:
    - meta_include_prototype: false
    - meta_scope: string.quoted.double.cs
    - include: escaped
    - match: '"'
      scope: punctuation.definition.string.end.cs
      pop: true
    - include: string_placeholders
    - match: $\n?
      scope: invalid.illegal.unclosed-string.cs
      pop: true

  format_string:
    - meta_include_prototype: false
    - meta_scope: meta.string.interpolated.cs string.quoted.double.cs
    - match: '"'
      scope: punctuation.definition.string.end.cs
      pop: true
    - include: escaped
    - include: string_placeholder_escape
    - match: \{
      scope: punctuation.section.interpolation.begin.cs
      push:
        - meta_scope: meta.string.interpolated.cs source.cs
        - clear_scopes: 2
        - match: $
          pop: true
        - include: string_interpolation
    - match: $\n?
      scope: invalid.illegal.unclosed-string.cs
      pop: true

  string_placeholder_escape:
    - match: '\{\{|\}\}'
      scope: constant.character.escape.cs

  string_placeholders:
    - include: string_placeholder_escape
    - match: '(\{)(\d+)(?=")'
      scope: constant.other.placeholder.cs
      captures:
        1: punctuation.definition.placeholder.begin.cs
        2: constant.numeric.cs invalid.illegal.unclosed-string-placeholder.cs
    - match: '(\{)(\d+)'
      captures:
        1: punctuation.definition.placeholder.begin.cs
        2: constant.numeric.cs
      push: string_placeholder

  string_placeholder:
    - meta_scope: constant.other.placeholder.cs
    - match: '(\})(\}(?!\}))?'
      captures:
        1: punctuation.definition.placeholder.end.cs
        2: invalid.illegal.unescaped-placeholder.cs
      pop: true
    - match: '(?=[}"])'
      pop: true
    - include: string_placeholder_format
    - match: '[^"}]+'
      scope: invalid.illegal.unexpected-character-in-placeholder.cs

  string_placeholder_format:
    - match: '\s*(?:(,)\s*(-?\d+)\s*)?'
      captures:
        1: punctuation.separator.arguments.cs
        2: constant.numeric.formatting.cs
    - match: ':(?="(?!"))'
      scope: invalid.illegal.unclosed-string-placeholder.cs
      pop: true
    - match: ':'
      scope: punctuation.separator.cs
      push:
        - meta_scope: constant.other.format-spec.cs
        - include: string_placeholder_escape
        - include: escaped
        - match: '(?=\})'
          pop: true
        - match: '([^}"\\]+(\\.)*)+(?="(?!"))'
          scope: invalid.illegal.unclosed-string-placeholder.cs
          pop: true
        - match: '\{'
          scope: invalid.illegal.unescaped-placeholder.cs

  string_interpolation:
    - include: string_placeholder_format
    - match: '\}'
      scope: punctuation.section.interpolation.end.cs
      pop: true
    - include: line_of_code_in

  escaped:
    - match: '{{escaped_char}}'
      scope: constant.character.escape.cs
    - match: \\
      scope: invalid.illegal.lone-escape.cs

[keith-hall]

It may help to know that it doesn't panic with $"test {"\"foo\""} test {bar}\"" (i.e. \" removed before {bar}), and the highlighting it receives could give us a clue where it is going wrong.

it looks to me like the slash before the quote is being scoped as invalid...

EDIT: after applying https://github.com/sublimehq/Packages/commit/51a737d56dd0c9830f3fecec3efdd96fd8eb4dbe, syntest gets it right but it still panics in syncat... my guess would be that syncat isn't handling clear_scopes correctly... or is somehow matching the regex patterns wrongly to get the invalid.illegal.lone-escape scope when it should get constant.character.escape?

[keith-hall]

ah I forgot to make the pack dumps after applying the fix to C#.sublime-syntax, that makes syncat work as expected - scoping correctly with no panics. But I guess it still shouldn't panic parsing dodgy syntax definitions if all the regexes compile?