Draft: naive approach for managing markdown flavour (PoC).

[thibsy]

Hi folks,

This PR merely lays some groundwork for further discussion and analysis in regards to markdown flavours in ILIAS. The code changes provide a naive approach to demonstrate how markdown syntax can be restricted or extended by some higher order system with minimal changes to the UI framework itself. Please note this is not a proposal of any kind, these changes are only for demo purposes. The examples should be runnable, so feel free to checkout this branch and try it out yourselves.

Let me quickly summarise the current situation of ILIAS:

• we have introduced support for markdown with the ILIAS\UI\Component\Input\Field\Markdown input and the ILIAS\Refinery\Transformation\MarkdownFormattingToHTML transformation, for converting markdown syntax into HTML.
• we are using the thephpleague\commonmark php library for converting markdown syntax to HTML.
• the converter is currently wrapped by the ILIAS\Refinery transformation, which uses the default syntax and follows the CommonMark specification.
• the ILIAS\UI\Component\Input\Field\Markdown editor implements actions for inserting the according markdown syntax: unordered list, ordered list, heading, italic, bold and link.
• there is currently no concept or way for adding new functionality to the markdown editor, besides implementing new actions or mechanisms inside the UI framework itself. This couples the UI framework to the refinery transformation implicitly, because submitted syntax needs to be supported by the transformation, or converter to be concrete.
• the thephpleague\commonmark library implements and supports an extension eco-system, which allows to add custom markdown flavours (dialects) in form of code-extensions which can be registered in the converter environment. There is also a broad set of existing extensions and a community around it.
• providing a ILIAS\UI\Component\Input\Field\MarkdownRenderer is the responsibility of the consumers of the markdown input. This could be a problem if we want to establish an ILIAS-wide markdown flavour, because consumers remain in control over which converter with what supported syntax they are using for their scenario.
• there is a paper about text handling in ILIAS, which needs to be considered when working on the process and concept of maintaining a markdown flavour inside ILIAS. It e.g. talks about specific kinds of markdown string representations, which may prohibit certain syntax for specific locations.

During the investigation of how ILIAS could incorporate and maintain a markdown flavour, I have focused on the following aspects:

• is it possible to implement a custom mechanism for uploading and using images and other media inside the markdown editor? E.g. drag-and-dropping an image inside the editor will upload the image to the IRSS, which would store the image accordingly and return an URL for consumption of the image.
• is it possible to extend the markdown converter and editor, so tables can be written using a custom syntax? There already is an established convention for structuring tables using markdown syntax with | and - characters to divide columns and rows.
• is it possible to extend the markdown converter and editor, so further tailored functionality can be incorporated? E.g. using == as opening and closing characters for gap-questions in the ILIAS T&A service.
• is it possible to extend the markdown converter and editor, so latex markup can be embedded? ILIAS does so in other places using custom blocks ([tex]...[/tex]) to mark sections which should be compiled using a latex library on the client.
• how can we maintain such extensions or such a markdown flavour for ILIAS in the future? Should markdown flavours be maintained globally or locally (one flavour for all editors and converters, or one flavour for one editor and converter)?

During my investigation and while working on some sort of proof-of-concept for incorporating a markdown flavour, I have came to the following conclusions:

• inserting images is supported by default (syntax-wise), so implementing a custom approach for uploading and inserting images is not exactly related to the markdown flavour - the syntax doesn't care from what source the image is being loaded. This mechanism would need to be tightly coupled to the markdown input though, so it must be implemented in the UI framework directly.
• a mechanism for uploading images within the markdown editor could currently be implemented using a similar approach as for file inputs, using an upload handler for storing resources and providing URLs for consumption. This however will put the responsibility on the consumer again. A mechanism like this would also have the same problem as file inputs currently have: file deletions.
• tables have a well established markdown syntax and the converter already provides an according extension, so it can be easily incorporated. An action for inserting tables and/or table columns and rows would be more complex than other actions, and possibly requires separate actions.
• since the markdown converter allows to implement custom syntax, it would be possible to support syntax for any kind of transformations. However, we should keep in mind that markdown is designed for formatting text - I am not sure if supporting gap-questions really fits into this category. If we have established a process to incorporate additional syntax, it would be worth a discussion then.
• the PoC for implementing custom markdown syntax showed me, that markdown might not be the best tool for creating gap-questions in the T&A service. To actually implement this, two different kinds of converters need to be used: one for converting the syntax to inputs, so users can fill in the gaps, and one for collecting the actual values which serve as answers for these inputs.
• latex could technically be incorporated into markdown as well. There is a FR for the centralisation of latex, which IMO could be implemented inside the UI framework as well. The project certainly has some correlation with the outcome of this project, since the underlying issue could be addressed in a similar manner.
• a markdown syntax could be implemented similar to approaches currently used within ILIAS, using [tex]...[/tex] blocks to mark latex code for later processing. We could incorporate such syntax, but it should certainly be discussed when we have established a process as well.
• implementing markdown syntax like [tex]...[/tex] could potentially open doors for incorporating various other complex HTML structures like UI components, so e.g. [ui_form]...[/ui_form] blocks could feature [ui_input ...] inlines, which are converted to actual UI component forms. This may be something we want to avoid, since markdown is designed for text formatting, not creating entire page layouts. However, there are projects like WordPress out there, which are using similar text based approaches for embedding complex HTML structures inside their pages.
• syntax can also be suppressed by not registering certain core extensions in the converter environment. This could be necessary for scenarios in which e.g. only emphasised text should be allowed. The paper about text handling in ILIAS also talks about restricted markdown "shapes", which represent scenarios like this with according classes (e.g. EmphasisedMarkdownShape).
• for both, the restriction and the extension of syntax, we need to find ways to clearly communicate the supported syntax of the editor to the user, especially if we are maintaining local markdown flavours. It could otherwise be confusing for users, if certain syntax only works in certain places, and users could find themselves frustrated if the editor preview always seems to only convert an arbitrary set of syntax. For restriction, this could be permanently disabling the according action buttons or hiding them altogether, and for extensions this could simply be to provide an additional action button, or a dropdown menu featuring multiple additional actions.
• we will most likely need to support local markdown flavours, since we run into similar use-cases like we do with the file input regarding upload limits: certain restrictions or additions will be tailored to their scenario. E.g. object titles may only allow emphasised text, or the T&A service is required to implement additional syntax for structuring their questions in a unique way, which is not meant for global usage.
• the paper about text handling in ILIAS already suggests an approach for implementing data classes as a way to talk about text, and transforming text from one to another format using refinery transformations. This will ultimately be the determining factor when it comes to a markdown flavour. Following this approach we would need to think about a markdown flavour in a very modular way, since supported syntax needs to be tailored to the formats determined by such a text class.
• when incorporating markdown flavours into the UI framework, these text classes could be used to communicate the supported syntax. The transformation for both sync and async content could be delegated to the refinery, as long as the syntax is not custom. Custom syntax needs to be incorporated differently, since the UI framework will not be able to build the according editor actions or perform the desired transformations on its own.
• I am currently not sure how we would incorporate custom syntax (registering editor actions and injecting a converter) into the UI framework. The PoC implements a naive approach which primarily demands three things from the consumer:
  • the consumer must provide a glyph component to visually represent the client side action button.
  • the consumer must provide a JavaScript function (as string) which will be mapped to the client side action button.
  • the consumer must implement their own markdown converter which supports the additional syntax.

I am happy to discuss these findings and open to hear your feedback.

Kind regards, @thibsy

[kergomard]

Thank you very much @thibsy for your analysis!

Just a quick first reaction on a conceptual level.

You write:

the PoC for implementing custom markdown syntax showed me, that markdown might not be the best tool for creating gap-questions in the T&A service. To actually implement this, two different kinds of converters need to be used: one for converting the syntax to inputs, so users can fill in the gaps, and one for collecting the actual values which serve as answers for these inputs.

I don't think we should go on doing this: We do not need to be able to represent the correct answer in the representation in the gap, we should just be able to retrieve a value from the input and then it would be the question's problem how to deal with it. This will very much lead to:

implementing markdown syntax like [tex]...[/tex] could potentially open doors for incorporating various other complex HTML structures like UI components, so e.g. [ui_form]...[/ui_form] blocks could feature [ui_input ...] inlines, which are converted to actual UI component forms.

I believe you are right that we should keep a tight handle on extensions like this, but, looking from the view point of a developer of the Test & Assessment, I don't see much of a way around it (The reasoning in short: We should provide a single way to edit text, thus we should get rid of the tiny editor. I don't see a viable way to use the ILIAS Page Edtior here, as I would see it as the way to go for creating a page, but not for editing text. There are very valid - and important - use cases for certain formatting options in the corresponding input fields, so plain text is not an option either.). So maybe creating processes to manage this, would be the right way to go.

For me the same applies to

we will most likely need to support local markdown flavours, since we run into similar use-cases like we do with the file input regarding upload limits: certain restrictions or additions will be tailored to their scenario. E.g. object titles may only allow emphasised text, or the T&A service is required to implement additional syntax for structuring their questions in a unique way, which is not meant for global usage.

I think we should not make everything configurable. Maybe we can define different "functionality-sets", but maybe we should also keep this simply under wraps and decide that we should provide a very minimal set of extensions and just decide that the base set is a take-it-or-leave-it-proposition.

I'm very much looking forward to the discussions around this.

Thanks again and best, @kergomard

[kergomard]

Hi all

When going through the PRs that have been around without activity for the TB, I found this one.

We will need to get back to this soon in the context of refactoring the test questions, so we keep this open for the time being.

Thank you very much and best, @kergomard