Migrate legacy Sphinx technical notes#

Introduced in December 2023, Rubin Observatory has a new technical note format for Markdown and reStructuredText documents, based on the Sphinx and Technote Python packages. If you started writing your technical note before December 2023, it is likely in the legacy Sphinx format. You can tell if your technical note is in the legacy format if it has a metadata.yaml file. If you are continuing to write such a technical note, you should migrate it to the new format to take advantage of new features. This page provides a guide to migrating your technical note to the new format.

To get started on your migration, be sure to have your technote repository checked-out locally. Work on a ticket branch, per the Developer Guide, so you can check your work with a pull request before merging to the main branch.

Running the migration tool#

Documenteer provides a migration tool, documenteer technote migrate, that can help automate this process.

If you cannot use this tool, you can follow the detailed steps below. For example, if you have a customized build process, you may want to manually migrate your technote. Alternatively you can run the migration tool, and then use Git to unstage any changes you don’t want to commit.

Preparation#

To run the migration command in an isolated Python environment, without affecting other projects, you can use the pipx tool. First, install pipx:

pip

python -m pip install pipx

conda-forge

conda install -c conda-forge pipx

homebrew

brew install pipx

Important

Documenteer, and therefore pipx, need Python 3.11 or later.

Run the migration#

From the root of the cloned technote repository, you can run the migration tool:

pipx run documenteer technote migrate

Tip

You can also add author information during the migration. Open authordb.yaml on GitHub and find the keys that identify the technote’s authors. For example, sickj is the key for Jonathan Sick. Then run the migration command with a -a option for each author:

pipx run documenteer technote migrate -a sickj -a economouf

When the migration runs, it shows a summary of the files changed and deleted. Use git diff to see the changes in detail. You may want to tweak those changes before committing them.

If you want to learn more about the changes made by the migration tool, you can review the detailed steps below.

Detailed steps#

Step 1. Create a technote.toml file#

The technote.toml file replaces the original metadata.yaml file. This new file provides both metadata and Sphinx configuration for your document.

Here is a simple technote.toml file to get you started:

[technote]
id = "EXAMPLE-000"
series_id = "EXAMPLE"
canonical_url = "https://example-000.lsst.io/"
github_url = "https://github.com/lsst/example-000"
github_default_branch = "main"
date_created = 2015-11-18
date_updated = 2023-11-01

[[technote.authors]]
name = {given = "Drew", family = "Developer"}
internal_id = "example"
orcid = "https://orcid.org/0000-0001-2345-6789"
[[technote.authors.affiliations]]
name = "Rubin Observatory Project Office"
internal_id = "RubinObs"

When you’re done, add this file to your repository and commit it:

git add technote.toml
git commit -m "Add technote.toml file"

Note

The schema for this file is described in the Technote package documentation, and elsewhere in the Documenteer documentation for Rubin technotes. For now, some pointers on important metadata:

id is the technote’s handle. A lower-cased version of the handle is the subdomain of the technote’s website. For example, the handle for https://sqr-000.lsst.io/ is SQR-000.
series_id is the technote’s series handle. At Rubin, this is the handle’s prefix. Common series include RTN, DMTN, SQR, and SITCOMTN.
canonical_url is the URL of the technote’s website.
github_url is the URL of the technote’s GitHub repository.
date_created is an optional field that specifies when the technote was first created.
date_updated is an optional field that specifies when the technote was last updated. If you omit this field, the current date is used.
Each author is specified with a [[technote.authors]] table (in TOML, the double brackets represent a table in an array of tables). Use the mamke add-author command to add an author to this file using data from authordb.yaml. It’s important to use the internal_id field to identify authors with their corresponding key in authordb.yaml. This enables Documenteer to update author information with the make sync-authors command.

Step 2. Update conf.py#

The conf.py file directly configures the Sphinx build process. New technotes use a different configuration set provided by Documenteer that uses technote.toml to customize the Sphinx configuration. For most technotes, the conf.py file should be a single line:

from documenteer.conf.technote import *  # noqa: F401, F403

Commit any changes:

git add conf.py
git commit -m "Update conf.py to new technote format"

If your conf.py file has additional content, some of that configuration may be migrated to technote.toml. Reach out to #dm-docs-support on Slack for advice.

Step 3. Update the index.rst file#

The index.rst file is the main content file for your technical note. The new technote format requires some changes to this file: the title is now part of the content, the abstract is marked up with a directive, status information is now part of technote.toml, and the configuration for the reference section is dramatically simplified.

Tip

Besides these changes, your technote can also be written in Markdown (index.md). If you wish to switch from ReStructuredText to Markdown, you can use MyST Parser’s migration tool:

pipx install "rst-to-myst[sphinx]"
rst2myst convert index.rst

This procedure uses pipx to install the rst-to-myst tool in an isolated Python environment (you might have already installed pipx to run the main migration tool).

Finally, delete the original index.rst file and edit index.md to fix any formatting issues.

git rm index.rst
git add index.md
git commit -m "Switch to Markdown format"

Add the title#

The title is now part of the content, not the metadata. Add the title to the top of the content:

rst

index.rst#

######################
Example technical note
######################

[... content below ...]

index.md#

# Example technical note

Move document status information to technote.toml#

The original technote format used a note directive to describe whether the document was a draft or deprecated. Now this status metadata is structured in technote.toml. Delete the note directive and add the status information to technote.toml following Describing the document status (draft, deprecated, or stable).

Format the abstract/summary with the abstract directive#

Legacy technotes either provided an abstract or summary through the description field in metadata.yaml, in a note directive in index.rst, or in a content section in index.rst. The new technote format uses an abstract directive to mark up the abstract/summary.

rst

index.rst#

######################
Example technical note
######################

.. abstract::

   This is a summary of the technical note.

Introduction
============

[... content below ...]

index.md#

# Example technical note

```{abstract}
This is a summary of the technical note.
```

## Introduction

[... content below ...]

Simplify the reference section#

If your technote makes references to other documents with roles like cite, you’ll need a reference section to display the bibliography. In the new technote format, this section is simplified:

rst

index.rst#

[... content above ...]

References
==========

.. bibliography::

index.md#

[... content above ...]

## References

```{bibliography}
```

Specifically:

The references section should be a regular section, not a “rubric.”
The bibliography directive no longer requires any configuration; all configuration is provided by Documenteer.

Commit any changes#

rst

git add index.rst
git commit -m "Reformat index.rst for new technote format"

git add index.md
git commit -m "Reformat index.md for new technote format"

Step 4. Delete metadata.yaml#

At this point, all relevant metadata about the technote is in technote.toml or index.rst/index.md. Delete the deprecated metadata.yaml file:

git rm metadata.yaml
git commit -m "Remove metadata.yaml file"

Step 5. Delete the lsstbib/ directory#

The legacy technote format vendored Rubin BibTeX bibliography files from lsst/lsst-texmf. The new technote format automatically downloads and caches these files so that you no longer need to commit them into your repository. Delete the lsstbib directory:

git rm -r lsstbib
git commit -m "Remove lsstbib/ directory"

Step 6. Update .gitignore#

The new technote format introduces additional directories that should be ignored by Git. Ensure at least the following paths are included in the .gitignore file:

.gitignore#

_build/
.technote/
.tox/
venv/
.venv/

git add .gitignore
git commit -m "Update .gitignore file"

Step 7. Set up pre-commit hooks#

Pre-commit is a Python package that runs validation and formatting checks on your technote’s repository before you commit. Although it is not required, it’s highly recommended that you set up pre-commit hooks for your technote. To start, add a .pre-commit-config.yaml file:

.pre-commit-config.yaml#

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      # - id: trailing-whitespace
      - id: check-yaml
      - id: check-toml

Commit any changes:

git add .pre-commit-config.yaml
git commit -m "Add pre-commit configuration"

Tip

You can add additional pre-commit hooks to this file to suite your needs. See Pre-commit’s directory of available hooks for ideas.

Step 8. Update requirements.txt#

The Python dependencies for your technote are listed in a requirements.txt file that should now look like this:

requirements.txt#

documenteer[technote]>=1.0,<2.0

Commit any changes:

git add requirements.txt
git commit -m "Update requirements.txt file"

Note

If your technote has additional dependencies listed, you can reach out to #dm-docs-support on Slack if you are unsure whether they are part of the Sphinx build process or separate packages needed for any custom document preprocessing.

Step 9. Add a tox.ini file#

Tox is a tool for running Python programs in dedicated virtual environments. This makes your local technote builds more reproducible by separating the technote’s dependencies from your system and other projects.

This is the recommended tox configuration to start with:

tox.ini#

[tox]
environments = html
isolated_build = True

[testenv]
skip_install = true
deps =
    -rrequirements.txt

[testenv:html]
commands =
   sphinx-build --keep-going -n -W -T -b html -d _build/doctrees . _build/html

[testenv:linkcheck]
commands =
   sphinx-build --keep-going -n -W -T -b linkcheck -d _build/doctrees . _build/linkcheck

[testenv:lint]
deps = pre-commit
commands =
   pre-commit run --all-files

[testenv:add-author]
commands =
    documenteer technote add-author

[testenv:sync-authors]
commands =
    documenteer technote sync-authors

Step 10. Update the Makefile#

The Makefile file provides a simple entrypoint for building your technote and performing other common tasks. This is the suggested content for your Makefile that works with the tox and pre-commit configurations:

Makefile#

.PHONY:
init:
	pip install tox pre-commit
	pre-commit install

.PHONY:
html:
	tox run -e html

.PHONY:
lint:
	tox run -e lint,linkcheck

.PHONY:
add-author:
	tox run -e add-author

.PHONY:
sync-authors:
	tox run -e sync-authors

.PHONY:
clean:
	rm -rf _build
	rm -rf .technote
	rm -rf .tox

Step 11. Update GitHub Actions workflows#

Recent technotes have already migrated their GitHub Actions workflows to use the reusable workflow from lsst-sqre/rubin-sphinx-technote-workflows. Check the .github/workflows/ci.yaml file to make sure it looks like this:

.github/workflows/ci.yaml#

name: CI

'on': [push, pull_request, workflow_dispatch]

jobs:
  call-workflow:
    uses: lsst-sqre/rubin-sphinx-technote-workflows/.github/workflows/ci.yaml@v1
    with:
      handle: example-001
    secrets:
      ltd_username: ${{ secrets.LTD_USERNAME }}
      ltd_password: ${{ secrets.LTD_PASSWORD }}

Replace example-001 with your technote’s handle (the subdomain of lsst.io).

If necessary, commit any changes:

git add .github/workflows/ci.yaml
git commit -m "Update GitHub Actions workflow"

Note

The original Rubin technotes used Travis CI for continuous integration and deployment, but we no longer use that service. In that case, you will need to create the .github/workflows directory and add the above ci.yaml workflow. GitHub Actions will automatically start using this workflow.

If your technote has a .travis.yml file, you should delete it:

git rm .travis.yml
git commit -m "Remove Travis configuration"

Step 12. Add dependabot support#

Dependabot is a service provided by GitHub that generates pull requests when there are new versions of your technote’s dependencies. Set up Dependabot by adding a .github/dependabot.yml file:

.github/dependabot.yml#

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"

Commit the changes:

git add .github/dependabot.yml
git commit -m "Add dependabot configuration"

Step 13. Update the README#

The README for a legacy-format technote likely has outdated information about how to build the technote. Here is a suggested README template for technotes in the new format:

rst

README.rst#

.. image:: https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg
   :target: https://example-000.lsst.io
.. image:: https://github.com/lsst/example-000/workflows/CI/badge.svg
   :target: https://github.com/lsst/example-000/actions/

##############
Document Title
##############

EXAMPLE-000
===========

A short description of this document.

**Links:**

- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/

Build this technical note
=========================

You can clone this repository and build the technote locally if your system has Python 3.11 or later:

.. code-block:: bash

   git clone https://github.com/lsst/example-000
   cd example-000
   make init
   make html

Repeat the ``make html`` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run ``make clean``.

The built technote is located at ``_build/html/index.html``.

Publishing changes to the web
=============================

This technote is published to https://example-000.lsst.io whenever you push changes to the ``main`` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.

Editing this technical note
===========================

The main content of this technote is in ``index.rst`` (a reStructuredText file).
Metadata and configuration is in the ``technote.toml`` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.

Commit any changes:

git add README.rst
git commit -m "Update README for new technote format"

README.md#

[![Website](https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg)](https://example-000.lsst.io)
[![CI](https://github.com/lsst/example-000/actions/workflows/ci.yaml/badge.svg)](https://github.com/lsst/example-000/actions/workflows/ci.yaml)

# Document Title

## EXAMPLE-000

A short description of this document.

### Links

- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/

## Build this technical note

You can clone this repository and build the technote locally if your system has Python 3.11 or later:

```bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
```

Repeat the `make html` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run `make clean`.

The built technote is located at `_build/html/index.html`.

## Publishing changes to the web

This technote is published to https://example-000.lsst.io whenever you push changes to the `main` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.

## Editing this technical note

The main content of this technote is in `index.md` (a Markdown file).
Metadata and configuration is in the `technote.toml` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.

Commit any changes:

git add README.md
git commit -m "Update README for new technote format"

Step 14. Merge the migration#

At this point, you should have a working technote in the new format. If you haven’t already, push your branch to the GitHub repository and open a pull request. GitHub Actions will build the technote and publish a preview version that is linked from the /v path of your technote’s website.

If the build works, you can merge the pull request.

If there are build errors, you can reach out to #dm-docs-support on Slack for help. Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed.