effekt icon indicating copy to clipboard operation
effekt copied to clipboard

Proper String lexing

Open dvdvgt opened this issue 2 years ago • 0 comments

Following up on the recently closed #290, we actually decided to have proper string lexing and support across all backends. Ideally, we would have thorough string lexing/parsing without regex but with a custom "parser" (new Parser[String]) as it is currently done for the insertion of semicolons here:

https://github.com/effekt-lang/effekt/blob/cf3b7d1fa7eda3a92f74351d99fcd3de159c900f/effekt/shared/src/main/scala/effekt/Parser.scala#L595-L640

Currently, the all (potentially invalid) escape sequences are permitted by the lexer

https://github.com/effekt-lang/effekt/blob/cf3b7d1fa7eda3a92f74351d99fcd3de159c900f/effekt/shared/src/main/scala/effekt/Parser.scala#L737

but instead, we want to only permit these escape sequences for strings. This also entails implementing potential conversions of source language strings to strings that are semantically equivalent but supported by the respective backends (see https://github.com/effekt-lang/effekt/pull/290#issuecomment-1773157642).

dvdvgt avatar Nov 08 '23 20:11 dvdvgt