[v5] New code highlighter

[yhatt]

Related: marp-team/marp#103, #194

Let's start to think new architecture for code highlight toward Marp Core.

CSS variables based coloring

highlight.js based highlighting has high flexibility that came from stylabillity based on CSS class.

On the other hand, it is hard than expected to overload exist color scheme by the context. If the theme author wanted to use different highlight colors in section.invert, required reverting all styles about highlight.js and adding styles for changed colors.

When using theme inheritance by @import, need to know all of styles using for highlight in the parent theme and revert them surely.

I think it's painful for the custom theme author, so providing much better coloring system in code highlight would be best. https://github.com/orgs/marp-team/discussions/103#discussioncomment-702465

If Marp adopted CSS variables for code highlighting, changing color scheme by class context would be drastically simpler. It also means to deny detailed styling for highlighted keywords through CSS properties, but it's a corner case in the syntax highlight.

highlight.js -> Shiki

https://github.com/marp-team/marp-core/issues/194#issuecomment-1122164223 has suggested a modern highlighter Shiki, made on TextMate json and Oniguruma RegExp engine.

Concerns

In future, this concern may become to resolve by adopting JavaScript RegExp instead of Oniguruma.

• Deasync for loading highlight engine (markdown-it/Marpit does not support Promise in rendering for now)
  • Node.js: synckit (similar to deasync but requires no native bindings or node-gyp)
  • Browser: nothing
    • Inject an inline Web Worker when initializing Marp for deasync
    • Or consider to return thenable proxy object in Marp.render()
  • (We are already planning the next-gen engine of Marpit to support async/await while rendering)
• Oniguruma WASM
  • Browser requires to fetch WASM from CDN that is supported CORS (jsDelivr, unpkg, etc)

Extended syntax for code fence

CommonMark defines the info string for fenced code block, like ```javascript. It leaves to Markdown processors about how to use this string, so may add more useful features to Marp.

• Line-based highlight: ```lang 1,3-4
  • https://github.com/marp-team/marp-vscode/issues/146
  • #168
• Line numbers ```lang line-numbers (```lang line-numbers=3 to begin from the middle of line numbers)
  • marp-team/marp#164

We need work to them carefully because complexity of Marp Core theme CSS will increase at the same time.

[yhatt]

The latest Shiki v1.15.0 has been introduced experimental JavaScript RegExp Engine. This may become a huge step for this task.

An concern of using Shiki highilghter is depending on Oniguruma RegExp engine to recognize TextMate JSON. That is provided by WASM, and not supporting non-asynchronous function that is only supported by markdown-it API. The JavaScript RegExp engine could become to remove that dependency, support sync function, and make simplify the architecture.

[isBatak]

Hi @yhatt, I managed to use Shiki highlighter like this:

// engine.js

import Shiki from "@shikijs/markdown-it";

const shiki = await Shiki({
  themes: {
    light: "one-light",
    dark: "one-dark-pro",
  },
});

export default ({ marp }) => marp.use(shiki);

Would it be possible to also incorporate magic-move ? Do you have any ideas how?

[yhatt]

@isBatak Yes, this is the correct way to use Shiki with the current version of Marp Core. In an environment where there are no issues with using WASM and where a Promise can be resolved before initializing Marp, this would be the best approach.

(We are currently waiting for the stabilization of synchronous API and JavaScript RegExp engine to maintain maximum compatibility with other environments)

The application of magic-move is challenging for some reasons:

• Lack of built-in syntax for step-based code animations
• Server-side JS orientation on Marp Core and its plugins (magic-move should render in the client-side)

However, it should not be impossible if implemented custom syntax extension and a tailored slide display HTML for Marp Core slides.