Overview of the MyST stack

This section provides a high-level overview of the main concepts and tools in the MyST ecosystem. See the guiding principles of the MyST ecosystem for some more high-level context.

Overview of the MyST build process¶

The MyST Build process takes input documents from authors and converts them into outputs that are meant for consumption by readers. The MyST CLI contains logic to use the MyST Document Engine, renderers, and themes to carry out this entire process. Here’s a high-level breakdown of the major steps of that process:

Here is a brief overview of these steps:

How does parsing work?¶

The MyST Document engine knows how to parse many kinds of documents into MyST AST. This is most-commonly done with markdown files (.md) or Jupyter Notebooks (.ipynb) written in MyST Markdown, a flavor of markdown that was designed for the MyST Document Engine.

However, the MyST Engine knows how to parse other kinds of syntax into MyST AST as well. For example, for admonition compatibility with GitHub Markdown. This is because we see the MyST AST as the primary point of standardization for the MyST ecosystem, not the Markdown flavor. Input documents may have many different forms, but once it is parsed into MyST AST, the document should have a standardized structure defined by the MyST Specification.

What is the MyST AST and Specification?¶

The MyST AST is a version of parsed MyST content that follows a specific structure so that Renderers can convert into different outputs. MyST AST conforms to the MyST Specification. The MyST Specification is a superset of the mdast specification (meaning that MyST includes all of the syntax that mdast supports, plus some extra ones like Directives).

Here’s an example of MyST content, and the same content parsed as MyST AST:

- type: block
children:
  - type: paragraph
    children:
      - type: strong
        children:
          - type: text
            value: Hello there!

The easiest way to understand how MyST Markdown gets converted to MyST AST is to explore the MyST Sandbox. Click the AST tab to see the underlying AST structure for anything that you type. The PRE tab represents the initial parsing phase, and the POST tab represents the AST after the resolving phase.

How do MyST Transformers work?¶

MyST Transformers are a way to convert a MyST AST node into another type of node. Transformers operate on AST rather than on raw Markdown because AST has more standardized structure to work with.

For example, consider a Markdown link like [some text](#a-label). In MyST Markdown, this defines a cross-reference to #a-label, but it uses Markdown link syntax. We use a MyST Transformer to convert that Markdown to a cross-reference AST node like so:

• First parse the Markdown [some text](#a-label).
• The result is a MyST AST node for a Markdown link.
• Next, search the document AST for any Markdown link nodes with a target that starts with #. Check if there’s a target for the link. If so, it is meant to be a cross reference.
• If so, converts the Markdown Link node into a Cross Reference node.

Some other uses for Transformers include:

• Lifting metadata from code-cells to their parent structures
• Check that figures have alt-texts
• Convert non-standard AST nodes (e.g., ones generated from a custom user directive) into ones that MyST knows how to render^[1].

See Concepts: MyST Transformers for more details about how transforms operate on an AST and what intermediate steps look like.

How do MyST themes and templates work?¶

MyST Themes and MyST Templates are both related to exporting a resolved MyST AST into one or more final output formats. They differ in the amount of customizability that a user has in altering their behavior, but the end-result is the same (generating an export artifact).

For a more technical explanation see the developer guide on themes, templates, and renderers.

Are there other MyST engines?¶

The MyST Specification and MyST Markdown Syntax was designed so that others could implement their own parsers, document engines, and renderers using MyST. Although there are presently no independent examples of a true MyST Document Engine, the MyST Parser for Sphinx project fulfills a similar role in the Sphinx ecosystem.

The MyST Parser for Sphinx is an extension for Sphinx that can parse MyST Markdown into Sphinx’s internal document structure. It was created to allow users to parse MyST Markdown syntax into Sphinx for V1 of Jupyter Book. It now exists as an independent extension for the Sphinx community, as Jupyter Book now uses the MyST Document Engine.

How does Jupyter Book relate to MyST?¶

The MyST Markdown Syntax was originally created to allow Jupyter Book V1 to use the Sphinx Document engine with Markdown content^[2], rather than the less well-known ReStructuredText language.

Over time, the Jupyter Book team decided that the most sustainable path forward was to maintain its own document engine, the MyST Document Engine. An early version of the MyST Document Engine was created by Curvenote. To promote its long-term sustainability, the project was open sourced and further developed in collaboration with the Jupyter Book team. It serves as the engine behind Jupyter Book V2.

You can think of Jupyter Book as a distribution of the MyST Document Engine. In other words, Jupyter Book wraps the MyST Document Engine application, with out-of-the-box configuration that supports a multi-page community knowledge base or documentation site. MyST is created and maintained by the Jupyter Book team.

Currently, Jupyter Book and the MyST Document Engine share much of the same functionality. Over time, we imagine that the MyST Document Engine will be a more flexible tool that makes fewer assumptions about the end use-case that it’s being used for. Jupyter Book will then have “more opinions” above the base configuration of the MyST Document Engine. That said — we are still figuring this out! We aren’t sure the exact relationship the two will have, and will revisit this as we learn more. See next.jupyterbook.org for Jupyter Book V2 documentation, and see this section of the Contributing Guide for our approach to documenting the two projects.

How do MyST and Sphinx compare?¶

Sphinx is an open-source documentation system used in many software projects, especially in the Python ecosystem. It builds an internal representation of technical documents as a tree (see docutils) similar to the MyST AST. Whilst Sphinx partially defines a specification of sorts^[3], a MyST Document Engine explicitly publishes its MyST AST for other tools to consume.

The Sphinx ecosystem has excellent support for Python documentation, referencing content, as well as externally providing an inventory of references known as intersphinx. You can link to Sphinx documentation with a MyST Document Engine by using intersphinx references, and the official MyST Document Engine automatically exposes the information required to allow Sphinx documentation to reference your MyST project.

At this time, the official MyST Document Engine does not support the authoring of software API documentation^[4]. As such, if your project is documenting Python software we suggest that you use Sphinx. If your project is primarily tutorials, educational textbooks (including with Jupyter Notebooks), a presentation, or scientific paper, we hope that you find a better fit with the MyST stack!

Sphinx and MyST take very different approaches to publishing to the web. In Sphinx, themes are used to customize the generation of HTML and JavaScript. Sites built using these themes can be deployed to static web servers like ReadTheDocs. In the MyST world, the equivalent to a Sphinx theme is a MyST Renderer that consumes the MyST AST and outputs HTML and JavaScript for your browser. Formally separating the MyST Document Engine that generates MyST AST and the MyST Renderer that consumes it makes it possible to create rich new experiences with MyST, such as the SciPy Conference Proceedings website that renders MyST AST on the fly.

Although the official engine also supports static HTML outputs, implementing themes as applications makes it easier to build more powerful MyST viewing experiences.