Add Codeblock Syntax Highlighting using Pygments

[Rohitth007]

This PR implements syntax highlighting for markdown code blocks by creating a padded "box" which would visually separate as code. Currently, only looks good in dark theme.

Commit 1: themes: codehilite: Adding Pygments style to the palette.

• This commit uses Pygments internal dictionaries and Pygments Material Style and adds it into the palette after the Screen is initialized. This is done inside add_codehilite_style and called in Controller.__init__.

Commit 2: boxes: codehilite: Implement syntax highlighting functionality.

• Basic syntax highlighting functionality was implemented in soup2markup by extracting the css_style of each span tag in the soup and using it as urwid style attributes. The remaining NavigableStrings are given a style attribute of w which is the whitespace css class in Pygments.

Commit 3: boxes: codehilite: Create a visual "box" by padding whitespaces.

• This commit moves the previous code into parse_codeblock, a static method which creates a visual "box" by:
  • Finding the longest line using code_soup.text.
  • Appending span text into markup
  • Adding code_indent part of the NavigableString to markup.
  • Adding right_padding to each line.
  • Making sure multiple empty lines are padded using padded_line.
  • Adding the remaining NavigableStrings not contributing to line breaks into markup.
• The NavigableString is added in parts and not all at once because though it would produce the same output, it would later help while implementing hl_lines preventing the case where half the previous line is highlighted along with the line expected.

Commit 4: boxes: codehilite: Implement padding for Plain Codeblocks.

• Plain Codeblock have to be treated differently as they don't come in the usual multiple span's in a pre tag package. The whole code is a NavigableString which does not start with \n.
• Hence, if this String has multiple lines it is split and padded accordingly.

Commit 5: boxes: codehilite: Restrict padding to avoid wrapping.

• This commit sets a limit to how much padding is to be added. Many a times most of the code has smaller lines and there is one line which is very long which causes all the lines to wrap and this makes the "box" look very messy. To avoid this a MAX_NO_WRAP_WIDTH constant of 80 is used which is a reasonable limit for in a full screen no autohide mode.

[neiljp]

@Rohitth007 This doesn't run for me - you don't seem to have added the dependency information in a commit? I could guess, but please add it as a separate commit for now based on what you have installed locally.

[Rohitth007]

@neiljp Strange, can I know what the error was because I haven't installed anything separately. Pygments installs as a dependency to pudb

[Rohitth007]

Currently I have added the colors into the palette after the screen is initialized. I think it's better to do it inside run.py like how monochrome is generated.

[Rohitth007]

I have added Pygments as a dependency into setup.py and also fixed the edge case for old messages without code tag.

[Rohitth007]

@zulipbot add "feedback wanted"

[Rohitth007]

@neiljp PR updated, tests added and ready for another review. I have simplified the code a bit and added comments and doc-strings to help understand the HTML structure.

Minor addition: Used Pygments style for inline code

[zulipbot]

Heads up @Rohitth007, we just merged some commits that conflict with the changes your made in this pull request! You can review this repository's recent commits to see where the conflicts occur. Please rebase your feature branch against the upstream/main branch and resolve your pull request's merge conflicts accordingly.

[zulipbot]

Heads up @Rohitth007, we just merged some commits that conflict with the changes your made in this pull request! You can review this repository's recent commits to see where the conflicts occur. Please rebase your feature branch against the upstream/main branch and resolve your pull request's merge conflicts accordingly.

[neiljp]

@Rohitth007 I'm thinking of close this, since some of these commits were merged already. I'd suggest you open a new PR with the remaining work after fetching and rebasing, so we can focus on those.

That said, I know it wasn't clear what to do regarding long lines and we ended up with various options based on spaces, which wraps badly. Do you think a widget-based approach might work better?

[neiljp]

@Rohitth007 Thanks for all the work on the syntax highlighting! :+1:

I've just merged #1346 with a version of the last commit from here, which sets the inline style to match codeblocks, though I adjusted the existing theme style name rather than removing it, since we still use it elsewhere.

We discussed flattening the edges of the code blocks in a separate PR, but I don't believe anything came of that, so I've opened #1347 for that remaining element. That points back here for reference, so I'm going to close this now, since I recall that was the remaining element we hadn't merged.