parser
parser copied to clipboard
- lexer.rl: handle CLRF as a line separator
Closes https://github.com/whitequark/parser/issues/1020.
3.3.1 :004 > Parser::CurrentRuby.parse("1\r\n2\r\n3").children[2].loc
=> #<Parser::Source::Map::Operator:0x00000001222f0f80 @expression=#<Parser::Source::Range (string) 6...7>, @node=s(:int, 3), @operator=nil>
3.3.1 :005 > Parser::CurrentRuby.parse("1\r\n2\r\n3").children[2].loc.expression.source
=> "3"
A few notes:
- If
\r\nis a line separatorparserstill emitstNLtoken with location of the\ncharacter -
tSTRING_CONTENTtokens now have proper locations, but the content doesn't include\rpart of\r\n(becauseeval(%{"\r\n"})is just"\n"), so.sourceof their locations doesn't match string content. I guess it's fine, the same happens with all escape sequences anyway.
If it doesn't break Rubocop's test suite I guess it's safe to merge it as is.
@kddnewton Could you take a look at this please? Does it fix Prism's translator?
This gets close, but runs into issues with escaped \r and literal \n then getting grouped, as in:
<<EOS
foo\rbar
baz\r
EOS
(There are regular newline characters after each line.) In this PR, it groups that last \r\n because the gsub is happening after escape sequences are resolved.