\n is a valid preprocessing token

[jyn514]

Expected behavior

\n should be a valid preprocessor token so that it can be used in token pasting. Found in /usr/include/sys/sysmacros.h. glibc is cursed.

Code

#define __SYSMACROS_DM(symbol) __SYSMACROS_DM1 (a b \n c)
#define __SYSMACROS_DM1(...) __glibc_macro_warning (#__VA_ARGS__)
__SYSMACROS_DM(a)
<stdin>:1:53 error: invalid token: unknown token: '\'
#define __SYSMACROS_DM(symbol) __SYSMACROS_DM1 (a b \n c)
                                                    ^

Output from clang -E:

__glibc_macro_warning ("a b \n c")

http://port70.net/~nsz/c/c11/n1570.html#6.4p3:

The categories of preprocessing tokens are: header names, identifiers, preprocessing numbers, character constants, string literals, punctuators, and single non-white-space characters that do not lexically match the other preprocessing token categories.

"If it's not whitespace, it's valid!" :facepalm:

[jyn514]

Fixing this long-term will be difficult, since \n is not a valid lexer token. I came up with an idea to split apart the lexer and preprocessor: preprocessor tokens are a superset of lexer tokens, but should be turned into lexer tokens by the time preprocessing finishes. Then there would be two options for a lexer: either the preprocessor or the lexer, but not both.

[hdamron17]

So is '