Allow Markdown configuration for Mattermost and Slack reporters

[jamesquilty]

I have configured urlwatch to report to an incoming webhook on my workplace's Mattermost server using the Slack reporter (#596). At present it appears that the reporter allows only text configuration and because this text is interpreted as Markdown by Mattermost there are several unintended formatting artefacts in the formatted post which frequently render the urlwatch output — particularly diff output — unreadable. It would be helpful if Markdown could be added as a configuration option for the Mattermost and Slack reporters, as it is for the Matrix reporter.

[scottmac]

You can use a filter on the URL turn the content into markdown

filter:
- css: ul#monitor
- html2text:
    method: pyhtml2text
    ul_item_mark: '-'
    strong_mark: '*'

[thp]

Patches to improve this gladly accepted. Alternatively, maybe just a documentation of the filters in "Advanced" section could help?

Alternatively, there's also https://github.com/thp/urlwatch/pull/572 which improves reporting with Markdown for Matrix, maybe this can be repurposed so that Slack + Mattermost use the MarkdownReporter instead of the TextReporter?

[jamesquilty]

@scottmac That's good to know for the input side of things!

@thp I'll take a look to see if a patch would be within my Python skills, but those skills are rudimentary so I'm not hopeful. I do appreciate your attention on this bug.

An example of what's going on might help, as there are a couple of things that are problematic. The lines starting with - and + (note the space) that are interpreted as a list by Markdown renderers. The separating text --------------------------------------------------------------------------- is rendered as a horizontal rule when preceded by a blank line and as H1 or H2 heading (depending on the renderer) when preceded by text. Thus,

---------------------------------------------------------------------------
CHANGED: Title ( URL )
---------------------------------------------------------------------------
--- @	Wed, 02 Dec 2020 16:00:01 +1300
+++ @	Tue, 08 Dec 2020 16:00:02 +1300
@@ -13,7 +13,7 @@
    Unchanged document (328.9 KB)
   </a>
   <a data-assetid="1798728" href="https://www.example.com/etc/etc/some-document.pdf" id="d5">
-   Previous document (362.4 KB)
+   Current document (108.0 KB)
   </a>
  </body>
 </html>

renders as:

---

CHANGED: Title ( URL )

--- @ Wed, 02 Dec 2020 16:00:01 +1300 +++ @ Tue, 08 Dec 2020 16:00:02 +1300 @@ -13,7 +13,7 @@ Unchanged document (328.9 KB)

• Previous document (362.4 KB)

• Current document (108.0 KB)

or, something like that depending on the renderer.

I think that these problems are already addressed by MarkdownReporter(ReporterBase)?

[dkasak]

@jamesquilty Yes, this is already handled by the MarkdownReporter. The trick is to enclose the diff into a Markdown fenced code block.

```diff
Diff lines
Go here
```

[knutwannheden]

AFAICT Slack doesn't support the language as indicated by Markdown code blocks :-( Also using the Slack desktop application the only way I know of is to add a "text snippet" to the message, where syntax highlighting can be specified. I took a quick look at the documentation, but I couldn't see how an incoming webhook can be used to include a "text snippet".

[knutwannheden]

It seems like the Slack reporter would have to upload the diff as a file to get the diff rendering with colors. That is then however not supported by the incoming webhooks and the reporter would instead have to use a REST API with an OAuth token.

I will probably look into implementing a custom reporter for myself. If there is interest in this I could consider creating a PR for this.

[jamesquilty]

It seems that Mattermost and Slack have divergent capabilities. Perhaps this is a case for splitting the urlwatch reporters?

[knutwannheden]

I implemented my own reporter now which uploads files instead, which allows colored rendering of diffs in Slack. A drawback is that this doesn't work on Android (no idea about iOS), but good enough for me.