MagicComments should be applied after Prism tokenization to properly track line numbers

[PoetaKodu]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

Prerequisites

• [X] I'm using the latest version of Docusaurus.
• [ ] I have tried the npm run clear or yarn clear command.
• [ ] I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
• [X] I have tried creating a repro with https://new.docusaurus.io.
• [ ] I have read the console error message carefully (if applicable).

Description

The way magic comments are currently applied (before passing the code to Prism) interferes with Prism after-tokenize hook (and possibly other hooks too). I implemented a way to manually specify type names for proper highlighting:

(this is a screenshot from the webpage)

This requires me to erase each line with a specific pattern:

// custom-syntactic-tweak-that-adds-types:Vehicle

This results in magic comments being shifted.

Reproducible demo

https://codesandbox.io/s/docusaurus-magiccomments-breaks-lines-n78f0r?file=/docusaurus.config.js

Steps to reproduce

To see the problem:

1. Click on "Tutorial" to go to the intro doc.

How to reproduce it starting from a fresh installation:

1. Using prism-include-languages.js hook to Prism after-tokenize and erase any token that contains a new line
2. Test how // highlight-next-line works in a code block

Expected behavior

Magic comments are applied after the after-tokenize hook so that user-provided hooks work if lines are added or removed.

Actual behavior

Magic comments are applied before passing the code to the Prism tokenizer and this results in highlighted lines being shifted if a hook is used to add/remove tokens that contains new lines.

Your environment

• Public source code: https://codesandbox.io/s/docusaurus-magiccomments-breaks-lines-n78f0r?file=/docusaurus.config.js
• Public site URL: https://n78f0r.sse.codesandbox.io/docs/intro
• Docusaurus version used: 2.2.0
• Environment name and version (e.g. Chrome 89, Node.js 16.4): Brave v1.47.171 (Jan 12, 2023), Node.js v16.15.1
• Operating system and version (e.g. Ubuntu 20.04.2 LTS): CodeSandbox (?)

Self-service

• [ ] I'd be willing to fix this bug myself.

[Josh-Cena]

Not sure how easy it is to implement, but accepting a PR to experiment. We can even move this logic to a Prism hook if it works.