feat: extension point for myst options

[tavin]

trafficstars

[github-actions[bot]]

:point_left: Launch a Binder on branch tavin/jupyterlab-myst/feat/myst-options

[tavin]

@rowanc1 I got all the checks passing except "PR label" which it seems I'm not allowed to edit.

[tavin]

@rowanc1 Does this need a deeper discussion with stakeholders to move it along? I am not sure where or with whom to pursue that... but I think I can explain why these changes make sense :)

[agoose77]

At a high level, this looks reasonable, thanks @tavin! Some points I'm not clear on:

• why does the interface accept a notebook widget? Is that a use case you already have a need for?
• @rowanc1 what's the benefit from defining directives without defining the renderer? Is it just to map new directives onto existing mdast so that they can be rendered?

[tavin]

@agoose77

• why does the interface accept a notebook widget? Is that a use case you already have a need for?

I can imagine several ways of choosing myst options:

• through a jupyterlab settings page, handled by a standard labextension (maybe part of jupyterlab-myst, or in any case maintained by executablebooks?) ... this being the use case that doesn't need the notebook reference
• notebook metadata (why not, if someone wants to write a labextension for it?)
• reading a _config.yml file reachable from the notebook file (another labextension possibility which I might attempt)

When the markdown preview integration is ready I am thinking my approach would be duplicated with some kind of document object in place of the notebook object. Then for me the 3rd bullet point would be great for working with jupyter books "live" i.e. without constantly rebuilding and reloading the output.

To be fair I'm not 100% sure how to use the notebook (document) object for these purposes but I read over some code and API docs and it seemed to me that it should be possible.

A simpler argument is that it's probably wrong to assume that globally fixed myst options are correct for every project/file a user edits. The notebook (document) object is the obvious thing to switch on.

[agoose77]

OK, I circled back around to look at this!

I'm a bit hesitant in merging this because I have not yet thought about what the extension picture should look like. In part, that's because the question of extensibility goes beyond just making jupyterlab-myst work, and into defining these extensions alongside the document.

Furthermore, there are questions as to whether extensions should be per-document, or per-environment (I think the former).

Maybe we can start by having discussions with @stevejpurves @rowanc1 et al. to figure out how we want to capture this information.

I can see a fairly straightforward solution, which involves defining a frontmatter entry that contains a table of reserved extension names. Then, the frontend can be in charge of ensuring that those extensions are loaded (i.e. some kind of plugin system). There's also a question of versioning, but this extends beyond plugins to the MyST distribution itself.

What happens for third-party extensions is less clear; I am worried about the fragmentation that comes from this, but it's also important to help people use MyST as they need it.

[tavin]

I think this particular PR should be viewed as merely adding a technical path forward so that someone could build a jupyterlab settings page where people can opt into colon fences or whatever.

For me and @meldefon well we just wanna be able to edit documents with sphinx proof directives and see something that looks like an admonition in the markdown preview. If this PR is merged I can very easily hack that into my local environment with a labextension -- that I can share -- while we wait for sphinx proof to land in mystjs.

Also I've hinted at wanting to be able to detect what myst options are set in my jupyter book build and mimic those. Why not?

I am not trying to create a myst-markdown extension/plugin architecture right now :)

[agoose77]

For me and @meldefon well we just wanna be able to edit documents with sphinx proof directives and see something that looks like an admonition in the markdown preview. If this PR is merged I can very easily hack that into my local environment with a labextension -- that I can share -- while we wait for sphinx proof to land in mystjs.

OK, this is a good argument. Right now, it's really helpful that you're testing these things out. Long term, I don't think this PR is the right way forward, but I don't want to hold things back in the near term. So, I propose that we merge it as an experimental interface that may be removed in future. Does this work for you? You can always pin to the version of jupyterlab-myst that contains this code to ensure it always works for you.

[tavin]

I'd be happy with an experimental interface and indeed I think it should be clearly documented as experimental.

In the future I'll be delighted if it's replaced with something better. What matters is the philosophy:

• that labextension authors have some way to tune myst options
• that it's possible to tune myst options based on the file or project (so not only from frontmatter -- e.g. we don't want to repeat the same _config.yml boilerplate in every markdown file in a project)

Thank you!

[agoose77]

@tavin I'll get to this early this week :)