Consider how users can extend roles/directives

[agoose77]

Context

If a user wants to add a custom role or directive, I think they would need to inject these definitions into the list of roles / directives that the markdown-it-docutils plugin accepts in the options argument. This may not be very easy - the options can be modified from JupyterLab, but to pass in JS objects would require some extension to set these options.

Proposal

Perhaps it would be better to expose a token from jupyterlab-myst that lets users write extensions to provide their own roles / directives. These extensions could be authored using https://github.com/jupyterlab/jupyterlab-plugin-playground to make it easy to play around with them from JupyterLab.

Tasks and updates

No response

[tavin]

In jupyter book (well really docutils I guess) you can use a directive to define a custom role. I wonder if that could be supported and also maybe a simple way to define a custom directive, e.g.

```{directive} myname(nameofbase)
:class: adefaultclass
```

I'm not sure what else might be doable beyond setting key-value parameters but that would already be useful.

Of course this concept leads to repetition of boilerplate in markdown files and then you need an include system... so I dunno. It sounds like you have other approaches in mind, and it will be interesting to see what happens.

[nthiery]

Hello!

Reviving this thread.

I would like to use custom roles to use semantic markup rather than
presentation markup in my course notes. E.g. in definitions write:

A {definiendum}`function` is ... 

rather than

A *function* is ...

And have the role generate appropriate presentation. This is a common
pattern for e.g. people coming from the latex world.

I have this up and running in my Jupyter book with custom roles, but these
roles don't render yet in the notebooks themselves which the students
are going to manipulate; this is a show stopper.

Do you have a timeline for custom roles in jupyterlab-myst?
I would love to be able to use them for the next semester starting in
early September.

A first approximation that would be good enough in a first step is if

{myrole}`foo <bar>`

would be rendered as foo.

Having an analog of latex's newcommand for the simple cases, as alluded to above,
would be a very nice second step.

Happy to beta test anything!

Thanks a lot in advance!
Nicolas

More context and motivation

The Kwarc group at Erlangen is developing cool online technology so that,
if you add some semantic information to your course notes (e.g. this is a
definition, this is the concept being defined; these are the concepts that
it depends on), then automatically you get a web interface with flash cards,
guided tours, all generated automatically and taylored to the learner's current
competencies, thanks to learning analytics / learner models.

At this stage, the technology works for course notes written in LaTeX. We
are currently experimenting bringing this technology to Jupyter which would
be very cool.

[rowanc1]

Thanks @nthiery! Excited to get to extensibility, I think it might be possible to get to a beta version of this over the summer as it is coming up much higher on the list in a bunch of places.

Right now unknown roles do render as text: {definiendum}`function` --> function, I think you are also asking for a default of removing the angle-bracket content if it exists? For example: {myrole}`foo <bar>` --> foo. Is that correct?

I think that is probably a sensible default for any unknown roles (maybe with some improved styling as well), and we could do that really fast.

From the extensibility side, things are getting better, from a code perspective the effort involved is:

• Create a role: https://github.com/executablebooks/mystjs/blob/main/packages/myst-roles/src/smallcaps.ts
• Add a renderer: https://github.com/executablebooks/myst-theme/blob/ddf09f5ed5deba04d732aaed4246448d35ef0405/packages/myst-to-react/src/basic.tsx#L94
• Add renderers here: https://github.com/executablebooks/jupyterlab-myst/blob/29de3e79cce06290621f66bbbc79210c3fbcc926/src/renderers.tsx#L8
• Add roles here: https://github.com/executablebooks/jupyterlab-myst/blob/29de3e79cce06290621f66bbbc79210c3fbcc926/src/myst.ts#L45

We then need some mechanism to extend jupyterlab-myst, e.g. using a new Jupyter plugin (I think?). @tavin has done something similar to this in #115, and that might be a pattern to use? I think @agoose77 also has some thoughts on how to do this.

Right now @agoose77 is putting a bunch of work into making things compatible with JupyterLab 4.0, which has changed a lot with regard to how we are structuring the extension. Our plan after that was to focus on citations/bibliography, but probably things will have settled down at an architecture level to maybe tackle both of these at once -- especially if @tavin's approach is possible, or @agoose77 has some thoughts/ideas/time! Curious on both of your thoughts there!

@nthiery, let me know if a quick improvement to the rendering of unknown roles would be helpful, and we can get that done right away.

[nthiery]

Thank @rowanc1 for the quick feedback! Yes, a quick improvement to the rendering of unknown roles would definitely be helpful.

Will read in detail your post later today; sounds exciting. And indeed I can see that jupyterlab 4 be a priority :-)

[tavin]

#115 needs to be partially reworked now that #102 has been merged.

Perhaps @rowanc1 or @agoose77 can weigh in on whether it's worth spending time on this?

It would make it easy to distribute a labextension that renders custom roles/directives implemented in ecmascript. You could experiment with custom roles/directives written in myst inside such a labextension.

[rowanc1]

I think there are some major changes coming in #142, and any changes should be based on that. I believe that @agoose77 also has some other restructuring in mind that he wants to do before that PR merges. Currently it is closer to #102 in how it works across everything.

Supporting JupyterLab 4.0 required quite a few changes to the way the whole extension works, but hopefully we are getting to a slightly better pattern. Thanks both for your patience as we sort this out. :)

[ashwinvis]

@rowanc1 and @agoose77 , now that JupyterLab 4.0 support has landed, could your revive this conversation? I am working on building an EU supported training platform and we are strongly considering using Jupyter Notebooks + MyST markup (with some custom directives from the sphinx-lesson plugin) for authoring course materials.

The missing element is to preview those custom directives which now appears as "unknown directives". It is nice that we see something, but it could be much better. I want to know if you have any approach planned out to add support for such custom directives?

[nthiery]

Same here, for the same use case!

@ashwinvis: happy to exchange best practices :-) In case this can help for now, here is a potential workaround: in my workflow -- and this is a standard practice -- the notebooks for the students are generated automatically from notebooks for the instructors that contains additional material like solutions, instructor notes, etc. I used to do that with nbgrader, but now am using a custom script for more flexibility. For simple things, this script could replace custom roles and admonitions with plain MyST counter parts (e.g. replace :::{exercise} Foo by :::{admonition} Exercise: Foo.

(Script available here: https://gitlab.com/travo-cr/travo-prepare/)

The caveat is that the instructor version is not practical to read at this stage with jupyterlab-myst, because undefined directives are replaced by just an error message "undefined directive"; you have to look at the sources to actually see the content. So an easy and helpful improvement to jupyterlab-myst would be that undefined directives get displayed as if they were (say) a plain admonition, with just a side note "undefined directive: foo".

[ashwinvis]

@nthiery Thanks for sharing your solution. Of course scripting can easily solve these issues. However I am hoping for a less hacky solution, either something which is:

• implemented into jupyterlab-myst, or
• packaged as an extension, or
• can be configured on a per-project basis

[nthiery]

Same here :-) The workaround is just to survive until then, e.g. for courses starting next week :-)

[ashwinvis]

Trying to follow up as a discussion: https://github.com/orgs/jupyter-book/discussions/2273