Consider exposing the lexer

[dabrahams]

Description

I am able to use the parser to get the tokens, but the tokens are all I really need and building syntactic structure on top of it is wasteful. Not the end of the world for my application, but it would be cleaner if I could just get a token stream.

[ahoppen]

I don’t think you would want to use the tokens out of the lever straight away because the parser may modify the lexemes, eg.

• The parser performs state transitions in the lever during string interpolation parsing https://github.com/swiftlang/swift-syntax/blob/ef367f7370c51200ed95b384e03d545f59c7fc98/Sources/SwiftParser/StringLiterals.swift#L566-L576
• The parser re-classifies trivia during string interpolation parsing https://github.com/swiftlang/swift-syntax/blob/ef367f7370c51200ed95b384e03d545f59c7fc98/Sources/SwiftParser/StringLiterals.swift#L215-L217
• The parser may decide to only consume the prefix of a lexeme, eg. during parsing of generics https://github.com/swiftlang/swift-syntax/blob/ef367f7370c51200ed95b384e03d545f59c7fc98/Sources/SwiftParser/Nominals.swift#L347

[dabrahams]

Interesting. I just need to skip past balanced pairs of {...} to look for the next “}” in my input, but if the parser is needed to deduce where the bounds of a string literal are that would definitely require the parser. Hmm, how does SwiftSyntax express a string literal with interpolations in it? Is the literal still a token?

[ahoppen]

I think matching braces should be doable in the lever alone but the distinction about what the lexer can do by itself vs for what it needs the parser are too subtle for me to be confident to expose the lexer by itself.

swift-syntax parses the string literal into string and interpolation segments: https://swiftpackageindex.com/swiftlang/swift-syntax/main/documentation/swiftsyntax/stringliteralexprsyntax