Fix parsing of doc comments which are turned into raw strings

[kraktus]

PR prompted by https://github.com/dtolnay/syn/issues/771

macro_rules! m {
    (#[doc = $tt:tt]) => { stringify!($tt) };
}

fn main() {
    println!(m! {
        ///test
    });
   // output: r"test"
}

Will also open a PR for rustdoc

[ehuss]

Thanks for the PR! I don't think this is entirely accurate. For example, a doc-comment passed to a proc-macro will receive an escaped string, such as #[doc = "comment with \"quotes\" inside"] (see proc_macro_server.rs). Also, the encoding form for a token in a macro_rules tt will use sufficient # to encode properly (see desugar).

I'm not sure if these exact behaviors are something we want to enshrine in the documentation just yet, as I suspect they have not had much focus. Perhaps you can engage with the lang team to see if this difference in behavior is something they want to commit to?

Given that these are mostly just illustrations of how a doc comment gets converted to an attribute, I don't think it is trying to exhaustively specify the exact method of that conversion. I think it would be best to leave it as-is, and have a separate paragraph discussing the conversions and how they interact with macros of different kinds.

[kraktus]

Thanks for the insight. The issue with the reference as is, is that user could think unicode encoding would be interpreted while it's not.

Maybe just make it clear they're turned into raw-literal strings without weighting on the implementation?

[ehuss]

Sorry, I'm a bit confused. I'm not sure what is meant by "unicode encoding". All Rust strings are unicode.

I'm also not sure what is meant by waiting (I assume) on the implementation. Waiting for what implementation?

As I mentioned, I don't think it is correct to say they are raw string literals, since in some circumstances (proc-macros) they are not, so it would be incorrect for that case.

[kraktus]

Sorry, I meant that a user writing /// foo\u{2122} could expect from reading the reference that \u{2122} would be turned into ™