p5-Compiler-Lexer icon indicating copy to clipboard operation
p5-Compiler-Lexer copied to clipboard

{line} value incorrect after mulit-line qw()

Open hyperbole opened this issue 3 years ago • 0 comments

This issue occurs in Compiler::Lexer->tokenize().

If there is a statement in the scanned code using a qw() which extends over multiple lines, the {line} member of the resulting token is 1 too many.

For example, my code has something like the following:

14   use Module1   qw( function1 );
15
16   use Module2  qw(
17                                 function2
18                                 function3
19                               );
20
21   use Module3 qw( function4 );

The tokens for 'use', 'Module1', 'qw', '(', 'function1', ')', and ';' all show {line} => 14

The tokens for 'use', 'Module2', 'qw', and '(' show {line} => 16.

The token for ' function2 function3 . shows {line} 20.

Tokens for ')', and ';' also show {line} => 20. These last three tokens should show {line} => 19.

This causes all following tokens to have {line} 1 greater than they should be. For example, the tokens for 'use', 'Module3', 'qw', '(', 'function4', ')', ';' all show {line} => 22.

I'm guessing at this point, but, I think tokenize() is counting the number of '\n' characters in the string that covers lines 16-19, but the first '\n' was counted when line 16 was first read, then it's re-counted when parsing the string.

I've noticed this increase in the line number each time a multi-line qw() is used in the scanned program.

hyperbole avatar Dec 28 '22 04:12 hyperbole