cadence icon indicating copy to clipboard operation
cadence copied to clipboard

Make the lexer operate on `[]byte` instead of `string` for source-code

Open SupunS opened this issue 2 years ago • 1 comments

Issue To Be Solved

from @pattyshack (https://axiomzen.slack.com/archives/C024LNQE5N1/p1658887300660419)

i notice the parser/lexer always copy the source code since they expect the input to be a string, but the real input is always a []byte regardless of where the source came from. could you file an issue to make the lexer operate on []bytes instead of string? (it's probably low priority)

SupunS avatar Jul 27 '22 15:07 SupunS

i haven't looked into the details, but my guess is we don't need to materialize strings except for identifiers (and maybe a few other specialized tokens).

If you want to get really fancy, you can use an intern string pool to dedup copies of the same token string to further reduce memory usage.

pattyshack avatar Jul 27 '22 17:07 pattyshack

Done in #1930

turbolent avatar Sep 23 '22 23:09 turbolent