MyST engine internals

This page explains how a build moves through the engine and which objects carry state between the stages. It is for people changing the engine or calling myst-cli from their own code. If you are writing a plugin, start with MystPlugin instead. Each section links to the source on main, which may be newer than the version you have installed.

Build pipeline¶

Directives and roles run inside the parser, so they see no other file and no execution output. Document transforms then run once per file in transformMdast, which ends by building that file’s RendererData. Once every file is loaded, MyST collects cross-reference targets from all of them. Finally postProcessMdast, in the same module, runs once per file. It resolves links and cross-references in place and then runs the project transforms. See Concepts: MyST Transformers for what the AST looks like before and after.

Every engine function in these stages takes an ISession. Plugins never receive a session, as explained in Session cache: ISessionWithCache.

Session¶

Every stage shares one session object. It carries the logger, the HTTP client, the Redux store, the build paths, and the loaded plugins. Two types share the name.

• The minimal ISession in myst-cli-utils has only log and fetch(). Use it for utilities and link transformers.

• The full ISession in myst-cli adds the Redux store (see Where the other types live), path helpers such as sourcePath() and buildPath(), the loaded plugins, and helpers for notebook execution. The source lists the rest.

Session cache: ISessionWithCache¶

At runtime the session also caches per-file data such as parsed trees, citation renderers, and execution outputs. ISessionWithCache is the wider type that exposes these caches. Call castSession(session) to get the wider type. The two types are separate so that code which only needs ISession does not have to import every myst-* package that the cache fields depend on. Both types and castSession are exported from myst-cli, but neither is a stable public API.

• $mdast maps each file to its pre and post data (see Document data). Read and write entries through $getMdast(file) and $setMdast(file, data).

• The other caches are $citationRenderers (keyed by file), $doiRenderers (keyed by DOI), $externalReferences, $outputs, and $siteTemplate.

Plugin transforms never see the session. If you drive the build yourself, you can pass extraTransforms to transformMdast or processProject instead. MyST calls each transform as transform(session, opts) once per file, right after that file’s RendererData is stored. The opts argument is the same options object that transformMdast received.

Document data¶

• A VFile wraps one document while transforms run. Its path is the absolute source path. Messages attached to the file become build warnings, as described in Warnings and errors.

• PreRendererData is created when a file is loaded. It holds the file path, its location in the project, the parsed mdast, the kind (Article, Notebook, or Part), and optional frontmatter, identifiers, and widgets.

• RendererData extends PreRendererData with sha256, slug, the resolved frontmatter, references, and dependencies. transformMdast builds the RendererData, postProcessMdast updates the object in place, and the site content JSON is the serialised form.

AST node types¶

• GenericNode and GenericParent are the loosely typed shapes most engine code uses. They have type, optional children, value, label, identifier, and position, and allow any other field.

• The canonical node types are in myst-spec. MyST-only additions are in myst-spec-ext.

• Traverse with unist-util-visit and query with unist-util-select:

import { visit } from 'unist-util-visit';
import { selectAll } from 'unist-util-select';

visit(tree, 'code', (node) => {
  if (!node.lang) node.lang = 'text';
});
const figures = selectAll('container[kind=figure]', tree);

Link transformers¶

DOIs, arXiv IDs, and internal cross-references each need their own handling. Link resolution is therefore a chain of LinkTransformer objects, tried in order during post-processing.

• test(uri) returns whether this transformer handles the URI.

• transform(link, vfile) mutates the Link node in place and returns true on success.

• protocol names the scheme. Set formatsText to true if your transformer sets the link text itself, so that MyST leaves it alone.

The built-in transformers live in myst-transforms/src/links/. To add your own, pass them as extraLinkTransformers to postProcessMdast or processProject. They run before the built-ins. There is no way to add them from myst.yml.

Where the other types live¶

• ProjectConfig and SiteConfig are the validated forms of myst.yml. Read them from the Redux store with the selectors exported from myst-cli.

• ICache and LocalDiskCache in myst-execute cache cell outputs on disk, one JSON file per key, so that rebuilds skip unchanged cells.

• Logger is Pick<typeof console, 'debug' | 'info' | 'warn' | 'error'>. Use session.log for build-level messages and fileWarn from myst-common for anything tied to a document, as described in Warnings and errors.