cadence
cadence copied to clipboard
Make the lexer operate on `[]byte` instead of `string` for source-code
Issue To Be Solved
from @pattyshack (https://axiomzen.slack.com/archives/C024LNQE5N1/p1658887300660419)
i notice the parser/lexer always copy the source code since they expect the input to be a string, but the real input is always a []byte regardless of where the source came from. could you file an issue to make the lexer operate on []bytes instead of string? (it's probably low priority)
i haven't looked into the details, but my guess is we don't need to materialize strings except for identifiers (and maybe a few other specialized tokens).
If you want to get really fancy, you can use an intern string pool to dedup copies of the same token string to further reduce memory usage.
Done in #1930