Format markdown files and rust code blocks in markdown files

[ytmimi]

trafficstars

Closes #2036

This would be a first step to resolving #5782, though this PR doesn't change anything about how rustfmt currently rewrite doc comments. Assuming this is a worthwhile change we could eventually update our doc comment rewriting code to call into these markdown formatting additions. I also want to note that these changes don't implement wrapping when wrap_comments=true is set. Given that this is more of an experimental change I didn't implement everything our current doc comment formatting handles, though these changes should handle markdown formatting more accurately since rustfmt would use pulldown_cmark to parse and format markdown documents if these changes are merged.

In the distant future I could also see this enabling rustfmt to format markdown files that are included in doc attributes, e.g #[doc = include_str!("path/to/docs.md")]

I know this looks like a massive addition, and it is, but 21k+ lines are related to JSON files and test cases that get autogenerated from those files if that makes this any more approachable 😅. @calebcartwright I know you have a preference for smaller PRs, but I figured I'd keep it all together to at least show the final vision of what I've been hacking on. I'd be happy to break this up into smaller PRs and rebase this one as those smaller PRs get merged.

[calebcartwright]

I know this looks like a massive addition, and it is, but 21k+ lines are related to JSON files and test cases that get autogenerated from those files if that makes this any more approachable 😅. @calebcartwright I know you have a preference for smaller PRs, but I figured I'd keep it all together to at least show the final vision of what I've been hacking on. I'd be happy to break this up into smaller PRs and rebase this one as those smaller PRs get merged.

Yeah that diff number made my eyes pop out like a cartoon character. At first blush it definitely feels too big to try to do in one pass, so decomposing it into the smallest atomic sets of changes will be necessary.

I do just want to re-emphasize what I said in https://github.com/rust-lang/rustfmt/issues/5782#issuecomment-1595567794 though, namely that I don't anticipate being able to make time to dig into this, at least not until we've got things bundled up and ready to go for the 2024 style edition

[ytmimi]

Yeah that diff number made my eyes pop out like a cartoon character. At first blush it definitely feels too big to try to do in one pass, so decomposing it into the smallest atomic sets of changes will be necessary.

Yup, I more or less expected that

I do just want to re-emphasize what I said in https://github.com/rust-lang/rustfmt/issues/5782#issuecomment-1595567794 though, namely that I don't anticipate being able to make time to dig into this, at least not until we've got things bundled up and ready to go for the 2024 style edition

@calebcartwright Understood. The first 3 commits are all pretty small, and lay some of the foundation for the later commits. Can we land those when broken out into their own PRs?

[tgross35]

@ytmimi I assume you haven't had a chance to get back to this - any way to help? I know it's not much but I could do something like create individual PRs from your smaller commits and try to chunk up the markdown crate.

Your commits really are very nicely atomic, only beef9c5a (Add markdown crate...) is big enough to possibly merit being split somehow.

[ytmimi]

Hey @tgross35, I haven't put more work into this PR, but I've recently been trying to expand on what I've got here in https://github.com/ytmimi/markdown-fmt. Feel free to take a look!