JuliaSyntax.jl icon indicating copy to clipboard operation
JuliaSyntax.jl copied to clipboard

Failure to parse some octal escape sequences

Open pfitzseb opened this issue 3 years ago • 2 comments

julia> Meta.parse(raw""" "\777" """)
"\xff"

julia> JuliaSyntax.parse(Expr, raw""" "\777" """)
ERROR: ArgumentError: octal escape sequence out of range
Stacktrace:
  [1] unescape_julia_string(io::IOBuffer, str::SubString{String})
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/OawBx/src/value_parsing.jl:172
  [2] unescape_julia_string(str::SubString{String}, is_cmd::Bool, is_raw::Bool)
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/OawBx/src/value_parsing.jl:203
  [3] JuliaSyntax.SyntaxNode(source::JuliaSyntax.SourceFile, raw::JuliaSyntax.GreenNode{JuliaSyntax.SyntaxHead}, position::UInt32)
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/OawBx/src/syntax_tree.jl:53
  [4] JuliaSyntax.SyntaxNode(source::JuliaSyntax.SourceFile, raw::JuliaSyntax.GreenNode{JuliaSyntax.SyntaxHead}, position::UInt32)
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/OawBx/src/syntax_tree.jl:89
  [5] build_tree(::Type{JuliaSyntax.SyntaxNode}, stream::JuliaSyntax.ParseStream; filename::Nothing, kws::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/OawBx/src/syntax_tree.jl:189
  [6] build_tree
    @ ~/.julia/packages/JuliaSyntax/OawBx/src/syntax_tree.jl:186 [inlined]
  [7] #build_tree#90
    @ ~/.julia/packages/JuliaSyntax/OawBx/src/expr.jl:208 [inlined]
  [8] build_tree
    @ ~/.julia/packages/JuliaSyntax/OawBx/src/expr.jl:207 [inlined]
  [9] #parse#83
    @ ~/.julia/packages/JuliaSyntax/OawBx/src/parser_api.jl:124 [inlined]
 [10] parse(::Type{Expr}, input::String)
    @ JuliaSyntax ~/.julia/packages/JuliaSyntax/OawBx/src/parser_api.jl:120
 [11] top-level scope
    @ REPL[81]:1

pfitzseb avatar Aug 23 '22 12:08 pfitzseb

I think it's the flisp parser which is in error here. \777 is 0x1ff, not 0xff which the flisp parser silently saturates to.

However we shouldn't throw so I think this is still a bug. I think what we need to do here instead is emit an error token and not try to convert to a numeric form.

c42f avatar Aug 24 '22 01:08 c42f

https://github.com/JuliaLang/JuliaSyntax.jl/pull/83 caused

julia> "\777"
"\xff"

julia> using JuliaSyntax; JuliaSyntax.enable_in_core!()

julia> "\777"
"✘"

We should probably throw a ParseError when encountering an ErrorVal in to_expr. Or just throw that error directly in unescape_julia_string.

pfitzseb avatar Sep 19 '22 11:09 pfitzseb