Migrate legacy reStructuredText/Sphinx technical notes¶
Introduced in December 2023, Rubin Observatory has a new technical note format for Markdown and reStructuredText documents. This update provides many new features included a responsive and branded page design, Markdown support, improved build configuration, automated bibliography management, and citation-ready metadata.
This page shows how to migrate your existing technotes with a command-line tool that largely automates the process. This page also provides file-by-file notes in case you need to revise the migration.
Tip
Your technote is in the legacy format if it has a metadata.yaml file at the top level of its repository.
LaTeX-format technotes are unaffected by this migration.
Step 1. Install pipx¶
The migration tool needs Python 3.11 or later.
You can verify this by running python --version from your shell.
This official migration procedure uses the pipx tool to install and run the migration tool in an isolated Python environment.
You can verify that pipx is installed by running pipx --version from your shell and checking for a version number.
If you don’t have the pipx command-line tool already, you can install pipx several ways:
python -m pip install pipx
conda install -c conda-forge pipx
brew install pipx
Step 2. Clone your technote and create a branch¶
Since the migration is performed locally, you need to clone the repository:
git clone https://github.com/{org}/{repo}
Then create a branch for the migration (see the Developer Guide for more information on branching):
git switch -c {...}
Working from a branch allows you to create a pull request to verify the migration before merging it into the main branch.
Step 4. Run the migration tool¶
From the root of the cloned technote repository, run the migration tool, listing any identified authors that are relevant to your technote:
pipx run documenteer technote migrate -a sickj -a economouf
When the migration runs, it shows a summary of the files changed and deleted.
Use git diff to review the changes in case you need to make tweaks before committing.
If you want to learn more about the changes made by the migration tool, you can review the detailed changes, below.
Step 5 (optional). Migrate to Markdown¶
The new technote format supports Markdown as well as reStructuredText. If you wish to use Markdown, you can use MyST Parser’s migration tool:
pipx install "rst-to-myst[sphinx]"
rst2myst convert index.rst
The Markdown flavor used by the new technote format is MyST Markdown, which is a superset of CommonMark Markdown with support for Sphinx roles and directives. See the MyST Parser documentation for more information.
Step 6 (optional). Build the technote locally¶
You can build the technote locally to verify that the migrated technote compiles successfully.
GitHub Actions will test the technote build when you open a pull request (next step), so you can skip this step. However, you can get faster feedback by building locally.
make init
make html
The technote’s website will be in the _build/html directory.
Open the index.html file in your web browser to view the technote:
open _build/html/index.html
You can repeat the make html command to rebuild the technote after making changes.
You might also want to run the linter to check links and find common issues:
make lint
If you have any questions or issues about the build, you should still proceed with committing and creating a pull request (see next step). This way you can link to the repository when you reach out for help in #dm-docs-support on Slack.
Step 7. Commit the migration, pull request, and merge¶
At this point, you should have a working technote in the new format.
If you haven’t already, commit your work, push your branch to the GitHub repository, and open a pull request.
GitHub Actions will build the technote and publish a preview version that is linked from the /v path of your technote’s website.
If the build works, you can merge the pull request.
If there are build errors, you can reach out to #dm-docs-support on Slack for help. Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed.
Migration details¶
The migration is automated by the documenteer technote migrate command, as described above. This section describes the steps performed by the migration tool in detail in case you need to make adjustments or understand the changes made to your technote.
technote.toml file (added)¶
The technote.toml file replaces the original metadata.yaml file.
This new file provides both metadata and Sphinx configuration for your document.
Here is a simple technote.toml file:
[technote]
id = "EXAMPLE-000"
series_id = "EXAMPLE"
canonical_url = "https://example-000.lsst.io/"
github_url = "https://github.com/lsst/example-000"
github_default_branch = "main"
date_created = 2015-11-18
date_updated = 2023-11-01
organization.name = "Vera C. Rubin Observatory"
organization.ror = "https://ror.org/048g3cy84"
license.id = "CC-BY-4.0"
[[technote.authors]]
name = {given = "Drew", family = "Developer"}
internal_id = "example"
orcid = "https://orcid.org/0000-0001-2345-6789"
[[technote.authors.affiliations]]
name = "Rubin Observatory Project Office"
internal_id = "RubinObs"
Note
The schema for this file is described in the Technote package documentation, and elsewhere in the Documenteer documentation for Rubin technotes. For now, some pointers on important metadata:
idis the technote’s handle. A lower-cased version of the handle is the subdomain of the technote’s website. For example, the handle for https://sqr-000.lsst.io/ isSQR-000.series_idis the technote’s series handle. At Rubin, this is the handle’s prefix. Common series includeRTN,DMTN,SQR, andSITCOMTN.canonical_urlis the URL of the technote’s website.github_urlis the URL of the technote’s GitHub repository.date_createdis an optional field that specifies when the technote was first created.date_updatedis an optional field that specifies when the technote was last updated. If you omit this field, the current date is used.Each author is specified with a
[[technote.authors]]table (in TOML, the double brackets represent a table in an array of tables). Use the make add-author command to add an author to this file using data from authordb.yaml. It’s important to use theinternal_idfield to identify authors with their corresponding key in authordb.yaml. This enables Documenteer to update author information with the make sync-authors command.
conf.py file (updated)¶
The conf.py file directly configures the Sphinx build process.
New technotes use a different configuration set provided by Documenteer that uses technote.toml to customize the Sphinx configuration.
For most technotes, the conf.py file should be a single line:
from documenteer.conf.technote import * # noqa: F401, F403
If your conf.py file has additional content, some of that configuration may be migrated to technote.toml.
Reach out to #dm-docs-support on Slack for advice.
index.rst file (updated)¶
The index.rst file is the main content file for your technical note.
The new technote format requires some changes to this file: the title is now part of the content, the abstract is marked up with a directive, status information is now part of technote.toml, and the configuration for the reference section is dramatically simplified.
Additionally, the new technote format supports Markdown as well as reStructuredText. See migrate to Markdown to learn how to switch to Markdown.
Tip
Besides these changes, your technote can also be written in Markdown (index.md).
If you wish to switch from ReStructuredText to Markdown, you can use MyST Parser’s migration tool:
pipx install "rst-to-myst[sphinx]"
rst2myst convert index.rst
This procedure uses pipx to install the rst-to-myst tool in an isolated Python environment (you might have already installed pipx to run the main migration tool).
Finally, delete the original index.rst file and edit index.md to fix any formatting issues.
git rm index.rst
git add index.md
git commit -m "Switch to Markdown format"
Title¶
The title is now part of the content, not the metadata. Add the title to the top of the content:
Document status¶
The original technote format used a note directive to describe whether the document was a draft or deprecated.
Now this status metadata is structured in technote.toml.
Delete the note directive and add the status information to technote.toml following Describing the document status (draft, deprecated, or stable).
Abstract¶
Legacy technotes either provided an abstract or summary through the description field in metadata.yaml, in a note directive in index.rst, or in a content section in index.rst.
The new technote format uses an abstract directive to mark up the abstract/summary.
######################
Example technical note
######################
.. abstract::
This is a summary of the technical note.
Introduction
============
[... content below ...]
# Example technical note
```{abstract}
This is a summary of the technical note.
```
## Introduction
[... content below ...]
Reference section¶
If your technote makes references to other documents with roles like cite, you’ll need a reference section to display the bibliography.
In the new technote format, this section is simplified:
[... content above ...]
References
==========
.. bibliography::
[... content above ...]
## References
```{bibliography}
```
Specifically:
The references section should be a regular section, not a “rubric.”
The bibliography directive no longer requires any configuration; all configuration is provided by Documenteer.
metadata.yaml file (deleted)¶
At this point, all relevant metadata about the technote is in technote.toml or index.rst/index.md.
Delete the deprecated metadata.yaml file:
git rm metadata.yaml
lsstbib/ directory (deleted)¶
The legacy technote format vendored Rubin BibTeX bibliography files from lsst/lsst-texmf.
The new technote format automatically downloads and caches these files so that you no longer need to commit them into your repository.
Delete the lsstbib directory:
git rm -r lsstbib
.gitignore file (updated)¶
The new technote format introduces additional directories that should be ignored by Git.
Ensure at least the following paths are included in the .gitignore file:
_build/
.technote/
.tox/
venv/
.venv/
.pre-commit-config.yaml file (added)¶
Pre-commit is a Python package that runs validation and formatting checks on your technote’s repository before you commit.
Although it is not required, it’s highly recommended that you set up pre-commit hooks for your technote.
To start, add a .pre-commit-config.yaml file:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
# - id: trailing-whitespace
- id: check-yaml
- id: check-toml
Tip
You can add additional pre-commit hooks to this file to suite your needs. See Pre-commit’s directory of available hooks for ideas.
requirements.txt file (updated)¶
The Python dependencies for your technote are listed in a requirements.txt file that should now look like this:
documenteer[technote]>=1.0,<2.0
Note
If your technote has additional dependencies listed, you can reach out to #dm-docs-support on Slack if you are unsure whether they are part of the Sphinx build process or separate packages needed for any custom document preprocessing.
tox.ini file (added)¶
Tox is a tool for running Python programs in dedicated virtual environments. This makes your local technote builds more reproducible by separating the technote’s dependencies from your system and other projects.
This is the recommended tox configuration to start with:
[tox]
environments = html
isolated_build = True
[testenv]
skip_install = true
deps =
-rrequirements.txt
[testenv:html]
commands =
sphinx-build --keep-going -n -W -T -b html -d _build/doctrees . _build/html
[testenv:linkcheck]
commands =
sphinx-build --keep-going -n -W -T -b linkcheck -d _build/doctrees . _build/linkcheck
[testenv:lint]
deps = pre-commit
commands =
pre-commit run --all-files
[testenv:add-author]
commands =
documenteer technote add-author
[testenv:sync-authors]
commands =
documenteer technote sync-authors
Makefile file (updated)¶
The Makefile file provides a simple entrypoint for building your technote and performing other common tasks.
This is the suggested content for your Makefile that works with the tox and pre-commit configurations:
.PHONY:
init:
pip install tox pre-commit
pre-commit install
.PHONY:
html:
tox run -e html
.PHONY:
lint:
tox run -e lint,linkcheck
.PHONY:
add-author:
tox run -e add-author
.PHONY:
sync-authors:
tox run -e sync-authors
.PHONY:
clean:
rm -rf _build
rm -rf .technote
rm -rf .tox
.github/workflows/ci.yaml file (added/updated)¶
Recent technotes have already migrated their GitHub Actions workflows to use the reusable workflow from lsst-sqre/rubin-sphinx-technote-workflows.
Check the .github/workflows/ci.yaml file to make sure it looks like this:
name: CI
'on': [push, pull_request, workflow_dispatch]
jobs:
call-workflow:
uses: lsst-sqre/rubin-sphinx-technote-workflows/.github/workflows/ci.yaml@v1
with:
handle: example-001
secrets:
ltd_username: ${{ secrets.LTD_USERNAME }}
ltd_password: ${{ secrets.LTD_PASSWORD }}
Replace example-001 with your technote’s handle (the subdomain of lsst.io).
Note
The original Rubin technotes used Travis CI for continuous integration and deployment, but we no longer use that service.
In that case, you will need to create the .github/workflows directory and add the above ci.yaml workflow.
GitHub Actions will automatically start using this workflow.
If your technote has a .travis.yml file, you should delete it:
git rm .travis.yml
.github/dependabot.yml file (added)¶
Dependabot is a service provided by GitHub that generates pull requests when there are new versions of your technote’s dependencies.
Set up Dependabot by adding a .github/dependabot.yml file:
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
README.rst or README.md file (updated)¶
The README for a legacy-format technote likely has outdated information about how to build the technote. Here is a suggested README template for technotes in the new format:
.. image:: https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg
:target: https://example-000.lsst.io
.. image:: https://github.com/lsst/example-000/workflows/CI/badge.svg
:target: https://github.com/lsst/example-000/actions/
##############
Document Title
##############
EXAMPLE-000
===========
A short description of this document.
**Links:**
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
Build this technical note
=========================
You can clone this repository and build the technote locally if your system has Python 3.11 or later:
.. code-block:: bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
Repeat the ``make html`` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run ``make clean``.
The built technote is located at ``_build/html/index.html``.
Publishing changes to the web
=============================
This technote is published to https://example-000.lsst.io whenever you push changes to the ``main`` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
Editing this technical note
===========================
The main content of this technote is in ``index.rst`` (a reStructuredText file).
Metadata and configuration is in the ``technote.toml`` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.
[](https://example-000.lsst.io)
[](https://github.com/lsst/example-000/actions/workflows/ci.yaml)
# Document Title
## EXAMPLE-000
A short description of this document.
### Links
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
## Build this technical note
You can clone this repository and build the technote locally if your system has Python 3.11 or later:
```bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
```
Repeat the `make html` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run `make clean`.
The built technote is located at `_build/html/index.html`.
## Publishing changes to the web
This technote is published to https://example-000.lsst.io whenever you push changes to the `main` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
## Editing this technical note
The main content of this technote is in `index.md` (a Markdown file).
Metadata and configuration is in the `technote.toml` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.