Parsing Fails only on Windows

[erayerdin]

I have a strange issue where logos fails tests only on Windows builds.

This is how I define the token types. I test it across this custom script.

While MacOS and Linux builds pass the tests, Windows build oddly fails while parsing a token.

Troubleshooting

Windows Paths

At first, I have thought maybe it is related Windows paths when I include_str here since it might require backwards slashes. However, it is not the case, the file is read successfully.

Windows Whitespace Encoding

This is the only thing I am suspicious about. I define there might be a whitespace at this line as [ ]. Maybe the encoding in Windows is not the same. What do you say?

---

Environment

• Rust 1.43.1
• logos 0.11.4

[erayerdin]

I actually shared this on Rust Users Forum, this is the solution.

Apparently, include_str was the problem and, as far as I could understand, it was including the related text file with some Windows specific setting/encoding (?).

https://github.com/maciejhirsz/logos/blob/5c391c615cec44f8a7158f27e0ba37620e6a9c17/logos/src/lib.rs#L43

I believe this line better contain \r unless there is a better reason to keep it that way. See below:

#[regex(r"[ \t\r\n\f]+", logos::skip)]

Feel free to close the issue.

[maciejhirsz]

Thanks, I'll keep this open so I remember to note that in the handbook!

[jeertmans]

Closing this as the solution has been found and is not really related to Logos, but more to file encoding where Windows uses \r\n instead of the classical \n on other platforms. Feel free to re-open this issue if you think this should be documented in Logos, or else.