Testing a Tutorial

CI runs every tutorial against the pinned Horopter release, using the harness in scripts/tutorial/. The same harness runs on your machine. The markers it reads are in the style guide.

PrerequisitesCopy Anchor Link

• mise, activated in your shell, which installs and selects the Node and pnpm versions pinned in mise.toml.
• Docker, for a tutorial that runs containers.

StepsCopy Anchor Link

Set Up a CloneCopy Anchor Link

git clone https://github.com/horopter-dev/horopter-docs.git
cd horopter-docs
mise install
pnpm install

Run One TutorialCopy Anchor Link

Pass the page to pnpm tutorial:

pnpm --silent tutorial scripts/tutorial/fixtures/echo.mdx

scripts/tutorial/fixtures/echo.mdx: every run block passed
scripts/tutorial/fixtures/echo.mdx: untested: manual block at line 57 (bash)

echo.mdx is one of the harness's fixtures, under scripts/tutorial/fixtures/; tutorial pages live under content/docs/tutorials/. The page runs in a throwaway directory, which is removed afterwards. A passing page's report goes to stdout and a failing page's to stderr. The command exits 0 when every page passes, 1 when one fails, and 2 when no page is given. Without --silent, pnpm also prints the command it runs; with or without it, pnpm ends a failing run with an [ELIFECYCLE] Command failed line, left out of the reports below.

Run Every TutorialCopy Anchor Link

pnpm tutorial takes several pages, runs them one at a time, and reports on each, so one failing page does not stop the rest. This runs every tutorial page under content/docs/, as CI lists them, and does nothing when there are none:

git grep -lE '^```bash run|^<include' -- 'content/docs/*.mdx' | xargs -r pnpm --silent tutorial

Bump a PinCopy Anchor Link

Renovate opens the pull request when a pin in versions has a new version. Each pin has a regex manager in renovate.json5, which updates the pin's line in versions and, for a pin a page has the reader set by hand, each page's export KEY= line with it. Add one when you add a pin to versions: pnpm test fails on a pin no manager updates. To move a pin by hand:

1. Change the pin's line in versions, such as HOROPTER_VERSION=v0.8.0, and each page's export line for that pin, such as export HOROPTER_VERSION=v0.8.0, with it.
2. Run every tutorial against it, as above.
3. Open a pull request.

The value reaches every page that renders it, with <HoropterVersion/> or <Version name="…"/>, and every setup, step and cleanup that reads the variable of the same name. Each line is KEY=value, the key in capitals, digits and underscores, the value not empty, and no key twice. HOROPTER_VERSION is required and must be a release tag, v and three numbers with an optional pre-release suffix. Otherwise the build and the harness both stop with an error naming the file and the line.

On the pull request, the Tutorials workflow's Plan job first checks that ghcr.io/horopter-dev/horoctl exists at the new tag, and fails if it does not. Each tutorial then runs against the new release in its own job. A push to main that changes versions runs the workflow again, and the docs site, rebuilt on the same push, shows the new pins.

Reading a Failure ReportCopy Anchor Link

This is the wrong-expect.mdx fixture's report, whose second expect block is wrong:

scripts/tutorial/fixtures/wrong-expect.mdx: run block at line 14: line 1: expected "three", got "two"
  command:
    echo two
  expected (expect block at line 18):
    three
  output:
    two

The first line names the page, the run block that failed by the line its fence is on, and why. A block from an included snippet is named by its line in the snippet, as run block at line N of content/snippets/…, and so is its expect block. Below it are the block's command, its expect block with that block's line, and what the block printed. The report names only the first failing block. The reason is one of:

• line N: expected "…", got "…": an output line does not match.
• line N: expected "…", but the output ended: the block printed fewer lines than its expect block holds.
• line N: unexpected "…": the block printed more lines, and its expect block has no final ... line.
• exited with status N, killed by SIGNAL, or exceeded its timeout of Ns: the block stopped the session. Its output is what it printed before stopping, and for a retry block that of its last attempt that printed anything.

A page with setup: reports it first: page: setup passed, or, when it failed, page: setup exited with status N; no run block ran or page: setup exceeded its timeout of Ns; no run block ran, each followed by the setup command and what it printed. A failed setup reports no run block, since none ran.

A line page: untested: manual block at line N (lang) lists each manual block, pass or fail, with of and the snippet after the line number for one from a snippet. A failing cleanup: adds page: cleanup and its reason, with the command and its output. A page the harness cannot read, such as one with a run block that is not bash, gets one line naming the page and what is wrong with it.

What CI RunsCopy Anchor Link

The Tutorials workflow, .github/workflows/tutorials.yaml, runs on every pull request, weekly, on a push to main that changes versions, and when started by hand. Its four jobs:

• Harness fixtures runs echo.mdx, which must pass and list its manual block as untested; setup.mdx, which must pass, its steps reading what its setup script exported; and wrong-expect.mdx, which must fail naming its second run block.
• Plan checks that horoctl's image exists at the pinned HOROPTER_VERSION, so a wrong pin fails once, and lists the pages under content/docs/ with a line starting ```bash run or an <include>, whose snippet may hold run blocks.
• Tutorial runs each listed page in its own job on Linux, with the runner's Homebrew on PATH.
• Tutorial on macOS runs the install how-to on a macOS arm64 runner, testing the macOS release download and Homebrew install. It runs no other page: they need Docker, which macOS runners lack.

The weekly run catches drift from outside this repository, such as an image the tutorials pull changing under the same tag.