Developer Guide

These sections help you get started with a development environment for mystmd and learn how to contribute to the codebase. Note that mystmd is written in TypeScript.

We start by discussing the project architecture, and give a birds eye view of how the various parts fit together. Then, we delve into the specific tools you need, and the developer workflow for each part. Finally, we go over requirements for contributing code back, via GitHub Pull Requests.

Architecture underlying a MyST build¶

From an author’s perspective, mystmd can be thought of as a tool which compiles files in MyST Markdown format, a variant of Markdown with several extensions, into books, articles, and websites. Like other such tools, it can produce static output, such as PDF, docx, or HTML. See Overview of the MyST build process for a conceptual overview.

However, for a developer it is important to understand that mystmd consists of two parts:

An engine that converts input documents into an AST (abstract syntax tree).
A renderer, that converts that AST into a given output format.

This model is equally applicable to exporting/converting static documents and rich web applications. Here’s a workflow for static documents:

Here’s a workflow for producing a rich web application:

Walking through this last example, we see that invoking myst start --headless converts a set of input files into an AST, and serves the resulting AST via a content server. The theme server is a React app that knows how to style data, which it pulls in from the content server. The user interacts with the React app, which may trigger new fetches from the content server.

Project-specific concepts¶

mystmd is built on top of well known tools in the JavaScript ecosystem, such as unist from unified, mdast, and citation.js for the myst CLI. The default MyST web themes use React, Remix (potentially vite soon), and Tailwind CSS for the theme server.

If you are familiar with these tools, you should not find many surprises in the codebase.

That said, there are a couple of concepts used only in this project, that won’t be familiar. These are detailed below:

Concepts: Renderers, themes, and templates¶

In the diagram above, we saw that mystmd produces websites by:

Parsing a set of documents to an AST.
Transforming the AST into a resolved AST.
Rendering the AST into components that can be used by a template or theme.
Exporting into a final output by a template / theme.

This section describes a bit more how Rendering and Exporting work using Themes and Templates.

For an introduction to themes and templates, see How do MyST themes and templates work?. In addition, below we’ll define what a Renderer is:

Renderer: Converts MyST AST into components that Themes and Templates can use to export final outputs. For example, the myst-to-react renderer converts MyST AST into a number of React components that the book and article React themes use to generate websites.

For example, in the case of the book theme, a MyST Document engine serves MyST AST via a Content Server, the MyST React renderer ingests that content and output React components, and the book theme converts those components into HTML outputs.

Where to find renderers, themes, and templates¶

MyST has multiple renders, themes, and templates that allow it to transform MyST AST into final output formats. The MyST team maintains a few that are worth noting:

github.com/jupyter-book/mystmd: Has several out-of-the-box renderers in addition to the core document engine.
- A collection of Renderers, look for the myst-to-* folders in mystmd/packages. These render MyST AST into components that themes can consume.
github.com/jupyter-book/myst-theme: The core React-based renderer and theme.
- A collection of MyST Rendering Packages that define various React UI components for default MyST themes to use.
- The MyST React Renderer generates React components out of MyST AST for use by the default MyST Themes. It provides a <MyST /> component which can render MyST AST into a React tree.
- The source code of the default MyST Themes, each of which use the React renderer. These themes are built and then published at the myst-templates GitHub organization for consumption by users.
- A React context, named ThemeContext (defined here in the myst-theme repository), is used to push state deeply into the tree, without having to pass it via props.
myst-templates: An index of templates that convert rendered components into final outputs. These are similar to MyST Themes, but follow a more standard “template” structure to product static outputs.

Example: Adding an “edit this page” button¶

Here’s a brief example to illustrate a common development pattern. Let’s say we want to add a new button to each page of the MyST Theme that includes an “edit link” for the page.

To accomplish this, we need to make three contributions.

First, update jupyter-book/mystmd, so that we can expose the “edit link” for each page as a new piece of metadata. The main logic for adding the edit URL is here.

Second, update the React theme infrastructure at jupyter-book/myst-theme. Here’s a pull request that implements this. Note how it:

Defines a new React component called <EditLink />.
Pulls the new edit URL metadata from the MyST AST.
Adds the new <EditLink /> component to the <FrontMatterBlock /> component.
Because the book theme and the article theme already use the <FrontMatterBlock />, they inherit this functionality automatically.

Concepts: MyST Transformers¶

See How do MyST Transformers work? for a higher-level overview of transforms. This section provides a few concrete examples of how transforms modify the AST throughout the transforms process.

Example: Parsing an admonition¶

Let’s say you’ve got MyST Markdown that defines an admonition, like this:

page1.md

:::{note} Here's an admonition
:::

The result of Parsing phase will be raw MyST AST (normally, this is .json but we’ll show it in YAML so it is a bit more readable):

page1.json

- type: mystDirective
  name: note
  args: Here's an admonition
  children:
    - type: admonition
      kind: note
      children:
        - type: paragraph
          children:
            - type: text
              value: Here's an admonition

The AST simply encodes that there’s a Directive present, with the name note.

After initial parsing, all Directive nodes are run, triggering the Admonition Directive logic. This converts the Directive Node into an Admonition Node.

During the transform phase, the Admonition Transforms is applied to each Admonition node. These do things like double-check that the admonition has all the necessary information to be rendererd.

The final output has more admonition-specific metadata defined, like admonitionTitle.

page1.json

- type: admonition
  kind: note
  children:
    - type: admonitionTitle
      children:
        - type: text
          value: Note
    - type: paragraph
      children:
        - type: text
          value: Here's an admonition

Example: Parsing a cross-reference¶

Let’s say you have a page that labels some content, and cross-references it elsewhere on the page:

page1.md

(label)=
**A labeled paragraph**.

A reference to [my label](#label)

The initial parse of this page nodes where labels are present, and treats our markdown link syntax as a regular URL.

page1.json

- type: mystTarget
  label: label
- type: paragraph
  children:
    - type: strong
      children:
        - type: text
          value: A labeled paragraph
- type: paragraph
  children:
    - type: text
      value: 'A reference to '
    - type: link
      url: '#label'
      children:
        - type: text
          value: my label

During the Transformations phase, a number of enumeration transforms are run, including one that detects and resolves cross-reference links. At the end of these transforms, the AST now correctly encodes that we have a cross reference rather than a “normal” URL link. This can now be rendered into various output formats.

page1.json

- type: paragraph
  children:
    - type: strong
      children:
        - type: text
          value: A labeled paragraph
  label: label
  identifier: label
  html_id: label
- type: paragraph
  children:
    - type: text
      value: 'A reference to '
    - type: crossReference
      children:
        - type: text
          value: my label
      urlSource: '#label'
      identifier: label
      label: label
      kind: paragraph
      template: Paragraph
      resolved: true
      html_id: label

Tools used in development¶

mystmd is built and developed using:

NodeJS and npm, which provides a JavaScript runtime and package management;
TypeScript for static type checking;
ESLint for code linting;
Prettier for code formatting; and
Changesets for versioning and changelogs.

Developer workflow: myst CLI¶

The mystmd libraries and command line tools are written in TypeScript, and require NodeJS and npm for local development.

To do local development, clone the repository:

git clone git@github.com:jupyter-book/mystmd.git
cd mystmd

Then, install dependencies via npm. You need to do this each time you pull from upstream, since dependency versions may change:

npm install

Now, build mystmd:

npm run build

Optionally, you can link the built executable as your globally installed mystmd:

npm run link

These commands allow you to use the myst CLI from any directory; source code changes are picked up after each npm run build (executed in the top-level source directory).

Developer workflow: myst-theme¶

The myst-theme README provides a more detailed overview of the components of that package.

Recall from the architecture overview that myst-theme is a React web application. It provides theming, and requires a separate content server for data. When developing, the steps are therefore to:

Launch a content server
Launch the myst-theme web application server (this is what you browse to)
Run a process to monitor changes and rebuild myst-theme

Content server¶

We need some example data to test our theme against, such as the example landing page. Clone this example content repository and start the content server:

git clone https://github.com/jupyter-book/example-landing-pages
cd example-landing-pages
myst start --headless

The --headless flag tells myst not to start a theme server; we want to do that ourselves in the next step.

When you start a content server without a theme server, you can still “visit” the pages in your site (often on port 3100). If you do so, you will see raw JSON and images. These represent the AST that the content server produces, and that a theme server uses to render a website (in the next step).

myst-theme server¶

We now fire up the myst-theme React app. This app server fetches the AST JSON from the content-server, then converts it to HTML, and serves it to the client where it is hydrated.

First, clone the myst-theme repository and install dependencies:

git clone https://github.com/jupyter-book/myst-theme/
cd myst-theme
npm install

Then, see if you can build the package:

npm run build

After the build succeeds, launch the theme server:

npm run theme:book

After a while, it should tell you that the server has been started at http://localhost:3000. Browse there, and confirm that you can see the landing-page content.

Each time you change the myst-theme source, you must recompile by re-running npm run build. This allows you to preview the changes locally.

To automatically watch for changes and reload, use the following command:

npm run dev

Note that you can run npm run dev from within any folder if you’d like to watch individual packages instead of the entire directory structure.

Infrastructure we run¶

The MyST API server¶

We run a lightweight server at api.mystmd.org to help users resolve and download templates. The code for this exists at the myst-templates/templates repository.

For example, to get a list of template types you can GET this URL:

https://api.mystmd.org/templates

And to see a list of available templates that can be resolved for Typst, you can GET this URL:

https://api.mystmd.org/templates/typst

Hwo to make a release¶

Make a release of `mystmd`¶

To publish a new release of mystmd, we do two things:

Publish to NPM.
Publish a GitHub release.

We describe each below.

Publish a `mystmd` release to NPM¶

Find the changesets PR. This contains a list of the version updates that will be included with this release. Here’s an example of a release PR.
Double-check the changes that have been merged to main and make sure nothing particularly complex or breaking has been merged. Bias towards action, but use your best judgment on whether to move forward.
After merging that PR, this GitHub action will make a release.
- It calls npm run version to generate the changelog (to review the changelog, you can run that command locally too).
- It then publishes the updated packages to the mystmd npm registry (it calls npm run publish:ci, which calls changeset publish).
- It creates a git version tag (which you’ll use in making the GitHub release).
Next, make a release on GitHub.

Make a release on GitHub¶

When we publish a new release to NPM, we also make a release on GitHub and share it for our user community. Here’s a brief process for what to do:

Confirm a release has been made. Go to the Tags page and look for a tag from the latest release.
Create a release on GitHub. Go to the Releases page and then click Draft a new release.
- The title should be the version from the tag. So if the tag was mystmd@1.3.26, the title is v1.3.26.
- Click Choose a tag and link it against the tag for the latest release to NPM (the one you discovered in the first step).
- Click Generate release notes so that GitHub automatically generates a list of the merged PRs and contributors.
- Categorize the PRs into Features, Fixes, Documentation, and Maintenance as you wish. (this is optional)
- For any major changes or new functionality, write a brief description that helps a user understand why it’s important and how to learn more. (this is optional)
- Write a one or two sentence summary of the big ideas in this release at the top. (this is optional).
Publish the release. Do this by clicking the Publish release button at the bottom.

Write a brief post for sharing the release. This helps others learn about it, and follow the links for more information. Here’s a snippet you can copy/paste:

TITLE: 🚀 Release: MySTMD v1.3.26

BODY:
The [Jupyter Book project](https://compass.jupyterbook.org) recently made a new release! 👇

[MySTMD v1.3.26](https://github.com/jupyter-book/mystmd/releases/tag/mystmd%401.3.26)

See the link above for the release notes on GitHub! Many thanks to the [Jupyter Book team](https://compass.jupyterbook.org/team) for stewarding our development and this release.

Share the release post in Jupyter-adjacent spaces. Here are a few places that are worth sharing (you can just copy/paste the same text into each):
- The MyST Discord
- The Jupyter Zulip Forum
- The Jupyter Discourse
- Social media spaces of your choosing.

Make a release of the `myst-theme`¶

The process for releasing myst-theme infrastructure is similar to the release process for mystmd. Here’s a brief overview:

Find the changesets PR in myst-theme and merge it, similar to the mystmd release process. Here’s an example PR in myst-theme.
Double-check the changes that have been merged to main and make sure nothing particularly complex or breaking has been merged. Bias towards action, but use your best judgment on whether to move forward.
Merge that PR. This will trigger the release process by running our release action. Here’s an example run of that action.
- The action will build the latest version of the theme infrastructure, and update the template files in the myst-templates GitHub organization. Here are the lines that update this template.
Make a release on GitHub, by following the same process in Make a release on GitHub.

Practices we follow¶

Build system¶

We use Turbo to manage our testing and build system.

Testing¶

Tests help ensure that code operates as intended, and that changes do not break existing code. You can run the test suite using:

npm run test

If you are working in a particular package, change your working directory to that specific package, and run the tests there. To run in “watch mode” (runs each time a change is saved), use npm run test:watch.

Linting¶

When contributing code, continuous integration will run linting and formatting on your pull request. You can also run npm run lint and npm run lint:format locally to catch errors early. To automate that process for each commit, install the git pre-commit hook: npm run install-pre-commit.^[1] If you are using the VSCode editor, it can be setup to show changes in real time and fix formatting issues on save (extensions: eslint and prettier).

Running in live-changes mode: depending on the package you are working in we have also setup live changes which can be faster than the npm run build; this can be run using npm run dev. If your changes aren’t showing up, use npm run build as normal.

Versioning and changesets¶

We use changesets for tracking changes to packages and updating versions. To learn about changesets, see this introductory guide to changesets. We use this changesets GitHub Action when we publish a release.

Before submitting your Pull Request, please add a changeset using npm run changeset, which will ask you questions about the package and ask for a brief description of the change. Commit the changeset file to the repository as a part of your pull request. You can use npm run version to preview the generated changelog.

Our current versioning procedure is a little loose compared to strict semantic versioning; as mystmd continues to mature, this policy may need to be updated. For now, we try to abide by the following rules for version bumps:

major: Backward incompatible change to the underlying supported MyST data. These would be cases where a non-developer MyST user’s project or site built with major version N would not work with major version N+1. Currently, we never intentionally make these changes.
minor: Backward incompatible change to the Javascript API, for example, changing the call signature or deleting an exported function. These can be a headache for developers consuming MyST libraries, but they do not break MyST content.
patch: For now, everything else is a patch: bug fixes, new features, refactors. This means some patch releases have a huge, positive impact on users and other patch releases are basically invisible.

Packages in the mystmd repository¶

All packages used to build mystmd live in the https://github.com/jupyter-book/mystmd repository.

These packages are ESM modules^[2], written in TypeScript, and live at https://github.com/jupyter-book/mystmd/tree/main/packages.

Command Line Tools:

mystmd: provides CLI functionality for myst start.
mystmd-py: a Python wrapper around mystmd, which makes it easy to install the tool in a Python environment. It should not be used for development.

Core Packages:

myst-cli: provides CLI functionality for mystmd. It does not export the CLI directly.
jtex: a templating library (see docs).
myst-frontmater: definitions and validation for scientific authorship/affiliation frontmatter (see docs).
myst-config: validation and reading of configuration files.
myst-templates: types and validation for templates (LaTeX, web, and word).

Markdown Parsing

markdown-it-myst: markdown-it plugin to handle tokenizing roles and directives.
myst-directives: core directives for MyST.
myst-roles: core roles for MyST.
myst-parser: converts a markdown-it token stream to Markdown AST (mdast).

Readers

tex-to-myst: convert LaTeX to MyST AST.
jats-to-myst: convert JATS XML, a standard for representing papers, to MyST AST.

Transformers

myst-transforms: transformations for use with MyST AST to transform, e.g., links, citations, cross-references, and admonitions (see here for more information](#develop:transforms)).

Export Tools

myst-to-docx: convert MyST documents to MS Word format.
myst-to-jats: convert MyST to JATS, for use in scientific archives.
myst-to-tex: convert MyST to LaTeX, to be used in combination with our jtex template compiler, to create stand alone LaTeX documents.
myst-to-html: convert MyST to HTML.

Extensions:

myst-ext-card: Card directives.
myst-ext-grid: Grid directives.
myst-ext-tabs: Tab directives.
myst-ext-reactive: Reactive directives.

Miscellaneous:

myst-common: common utilities for working with ASTs.
myst-spec: JSON Schema that defines the MyST AST.
myst-spec-ext: temporary development extensions to myst-spec.
citation-js-utils: utility functions to deal with citations.
myst-cli-utils: shared utils between jtex, and myst-cli.
simple-validators: validation utilities, that print all sorts of nice warnings.

Footnotes¶

Uninstall the pre-commit hook with git config --unset core.hooksPath.
↩
ESM modules are imported using the JavaScript import syntax, and cannot be used with CommonJS require().
↩

Architecture underlying a MyST build¶

Project-specific concepts¶

Concepts: Renderers, themes, and templates¶

Where to find renderers, themes, and templates¶

Example: Adding an “edit this page” button¶

Concepts: MyST Transformers¶

Example: Parsing an admonition¶

Example: Parsing a cross-reference¶

Tools used in development¶

Developer workflow: myst CLI¶

Developer workflow: myst-theme¶

Content server¶

myst-theme server¶

Infrastructure we run¶

The MyST API server¶

Hwo to make a release¶

Make a release of mystmd¶

Publish a mystmd release to NPM¶

Make a release on GitHub¶

Make a release of the myst-theme¶

Practices we follow¶

Build system¶

Testing¶

Linting¶

Versioning and changesets¶

Packages in the mystmd repository¶

Make a release of `mystmd`¶

Publish a `mystmd` release to NPM¶

Make a release of the `myst-theme`¶