Upgrade rubygem versions.

[yilinwei]

• This upgrades from ruby 2.7 to 3.3 since 2.7 is eol.
• Upgrades to jekyll 4.3.3 since 3.x is only supporting maintenance fixes at this point.
• Drops redcarpet since jekyll has dropped support for it.
• Moves the dependencies to the Gemfile

Notes

Syntax highlighting

We don't use Jekyll's syntax highlighting (rouge) at all. As far as I can tell this is due to:

1. It doesn't support EBNF
2. We do a "two-step" render process where the latex pre-preprocessor spat out the html and highlight.js used to support merging of html nodes.

Because we embed math elements into the snippets - something more complicated like using the tree-sitter-based grammar is a no-go; it is not nearly permissive enough to deal with what we're doing. We could use a strategy like replacing the math elements with placeholder identifiers (making sure to respect Scala's naming rules) and substitute them back in - but treesitter doesn't support EBNF either, which is a large part of the spec.

However since highlight-js has removed support for this feature, we could potentially look at doing this server-side or using rouge. This has the added benefit that it would be more inline with Github's own syntax highlighting I think. Since I don't know ruby or rouge at all, I haven't done a thorough investigation of this option.

I do, on the other hand, know Javascript so I've added started to add the merge behaviour back in like before, so we can now upgrade the highlight.js version successfully.

[bishabosha]

I've noticed this is based on a very old branch (previous commit is from September 2023), do you mind rebasing?

[bishabosha]

The latex handling also only seems to be working in specific places, e.g. only one E here is substituted correctly:

for comparison with the current deployed docs:

[yilinwei]

@bishabosha

Yup, I have a local branch which is in-progress and I'll undraft/rebase this once it's done. Basically I don't deal with the case where the backtick extends across multiple highlight spans (I deal with the inverse, where one span contains multiple backtick expansions). Some of issues here are also because it infers the wrong language as well - for some reason it really likes to infer as perl...

[bishabosha]

That is quite tricky. Perhaps your other idea of escaping the backticks at build time makes more sense?

Maybe each backtick is subbed by id___35 (index 35) and then each decoded fragment is stored in a json array possibly

(I have no idea how this works for disclaimer)

[yilinwei]

@bishabosha

I'd love that to be that simple, but we backtick across delimiters such as T_0, ..., T_n, so there would have to be an adhoc ruleset about how we introduce the backtick segments (i.e. you want id__xx per T) so that the syntax highlighting would still highlight the correct spans. Not saying we couldn't do, just that it isn't completely straightforward, and I would be more in favour of preserving the current behaviour if possible.

I have some time to look at this, this weekend and I think that it's pretty much there on my own local branch.

[bishabosha]

I have some time to look at this, this weekend and I think that it's pretty much there on my own local branch.

awesome, thanks!