Migrate legacy Sphinx technical notes#
Introduced in December 2023, Rubin Observatory has a new technical note format for Markdown and reStructuredText documents, based on the Sphinx and Technote Python packages.
If you started writing your technical note before December 2023, it is likely in the legacy Sphinx format.
You can tell if your technical note is in the legacy format if it has a metadata.yaml
file.
If you are continuing to write such a technical note, you should migrate it to the new format to take advantage of new features.
This page provides a guide to migrating your technical note to the new format.
To get started on your migration, be sure to have your technote repository checked-out locally.
Work on a ticket branch, per the Developer Guide, so you can check your work with a pull request before merging to the main
branch.
Running the migration tool#
Documenteer provides a migration tool, documenteer technote migrate, that can help automate this process.
If you cannot use this tool, you can follow the detailed steps below. For example, if you have a customized build process, you may want to manually migrate your technote. Alternatively you can run the migration tool, and then use Git to unstage any changes you don’t want to commit.
Preparation#
To run the migration command in an isolated Python environment, without affecting other projects, you can use the pipx
tool.
First, install pipx
:
python -m pip install pipx
conda install -c conda-forge pipx
brew install pipx
Important
Documenteer, and therefore pipx, need Python 3.11 or later.
Run the migration#
From the root of the cloned technote repository, you can run the migration tool:
pipx run documenteer technote migrate
Tip
You can also add author information during the migration.
Open authordb.yaml on GitHub and find the keys that identify the technote’s authors.
For example, sickj
is the key for Jonathan Sick.
Then run the migration command with a -a
option for each author:
pipx run documenteer technote migrate -a sickj -a economouf
When the migration runs, it shows a summary of the files changed and deleted.
Use git diff
to see the changes in detail.
You may want to tweak those changes before committing them.
If you want to learn more about the changes made by the migration tool, you can review the detailed steps below.
Detailed steps#
Step 1. Create a technote.toml file#
The technote.toml
file replaces the original metadata.yaml
file.
This new file provides both metadata and Sphinx configuration for your document.
Here is a simple technote.toml
file to get you started:
[technote]
id = "EXAMPLE-000"
series_id = "EXAMPLE"
canonical_url = "https://example-000.lsst.io/"
github_url = "https://github.com/lsst/example-000"
github_default_branch = "main"
date_created = 2015-11-18
date_updated = 2023-11-01
[[technote.authors]]
name = {given = "Drew", family = "Developer"}
internal_id = "example"
orcid = "https://orcid.org/0000-0001-2345-6789"
[[technote.authors.affiliations]]
name = "Rubin Observatory Project Office"
internal_id = "RubinObs"
When you’re done, add this file to your repository and commit it:
git add technote.toml
git commit -m "Add technote.toml file"
Note
The schema for this file is described in the Technote package documentation, and elsewhere in the Documenteer documentation for Rubin technotes. For now, some pointers on important metadata:
id
is the technote’s handle. A lower-cased version of the handle is the subdomain of the technote’s website. For example, the handle for https://sqr-000.lsst.io/ isSQR-000
.series_id
is the technote’s series handle. At Rubin, this is the handle’s prefix. Common series includeRTN
,DMTN
,SQR
, andSITCOMTN
.canonical_url
is the URL of the technote’s website.github_url
is the URL of the technote’s GitHub repository.date_created
is an optional field that specifies when the technote was first created.date_updated
is an optional field that specifies when the technote was last updated. If you omit this field, the current date is used.Each author is specified with a
[[technote.authors]]
table (in TOML, the double brackets represent a table in an array of tables). Use the mamke add-author command to add an author to this file using data from authordb.yaml. It’s important to use theinternal_id
field to identify authors with their corresponding key in authordb.yaml. This enables Documenteer to update author information with the make sync-authors command.
Step 2. Update conf.py#
The conf.py
file directly configures the Sphinx build process.
New technotes use a different configuration set provided by Documenteer that uses technote.toml
to customize the Sphinx configuration.
For most technotes, the conf.py
file should be a single line:
from documenteer.conf.technote import * # noqa: F401, F403
Commit any changes:
git add conf.py
git commit -m "Update conf.py to new technote format"
If your conf.py
file has additional content, some of that configuration may be migrated to technote.toml
.
Reach out to #dm-docs-support on Slack for advice.
Step 3. Update the index.rst file#
The index.rst
file is the main content file for your technical note.
The new technote format requires some changes to this file: the title is now part of the content, the abstract is marked up with a directive, status information is now part of technote.toml
, and the configuration for the reference section is dramatically simplified.
Tip
Besides these changes, your technote can also be written in Markdown (index.md
).
If you wish to switch from ReStructuredText to Markdown, you can use MyST Parser’s migration tool:
pipx install "rst-to-myst[sphinx]"
rst2myst convert index.rst
This procedure uses pipx to install the rst-to-myst
tool in an isolated Python environment (you might have already installed pipx to run the main migration tool).
Finally, delete the original index.rst
file and edit index.md
to fix any formatting issues.
git rm index.rst
git add index.md
git commit -m "Switch to Markdown format"
Add the title#
The title is now part of the content, not the metadata. Add the title to the top of the content:
Move document status information to technote.toml#
The original technote format used a note
directive to describe whether the document was a draft or deprecated.
Now this status metadata is structured in technote.toml
.
Delete the note
directive and add the status information to technote.toml
following Describing the document status (draft, deprecated, or stable).
Format the abstract/summary with the abstract directive#
Legacy technotes either provided an abstract or summary through the description
field in metadata.yaml
, in a note
directive in index.rst
, or in a content section in index.rst
.
The new technote format uses an abstract
directive to mark up the abstract/summary.
######################
Example technical note
######################
.. abstract::
This is a summary of the technical note.
Introduction
============
[... content below ...]
# Example technical note
```{abstract}
This is a summary of the technical note.
```
## Introduction
[... content below ...]
Simplify the reference section#
If your technote makes references to other documents with roles like cite
, you’ll need a reference section to display the bibliography.
In the new technote format, this section is simplified:
[... content above ...]
References
==========
.. bibliography::
[... content above ...]
## References
```{bibliography}
```
Specifically:
The references section should be a regular section, not a “rubric.”
The bibliography directive no longer requires any configuration; all configuration is provided by Documenteer.
Commit any changes#
git add index.rst
git commit -m "Reformat index.rst for new technote format"
git add index.md
git commit -m "Reformat index.md for new technote format"
Step 4. Delete metadata.yaml#
At this point, all relevant metadata about the technote is in technote.toml
or index.rst
/index.md
.
Delete the deprecated metadata.yaml
file:
git rm metadata.yaml
git commit -m "Remove metadata.yaml file"
Step 5. Delete the lsstbib/ directory#
The legacy technote format vendored Rubin BibTeX bibliography files from lsst/lsst-texmf.
The new technote format automatically downloads and caches these files so that you no longer need to commit them into your repository.
Delete the lsstbib
directory:
git rm -r lsstbib
git commit -m "Remove lsstbib/ directory"
Step 6. Update .gitignore#
The new technote format introduces additional directories that should be ignored by Git.
Ensure at least the following paths are included in the .gitignore
file:
_build/
.technote/
.tox/
venv/
.venv/
git add .gitignore
git commit -m "Update .gitignore file"
Step 7. Set up pre-commit hooks#
Pre-commit is a Python package that runs validation and formatting checks on your technote’s repository before you commit.
Although it is not required, it’s highly recommended that you set up pre-commit hooks for your technote.
To start, add a .pre-commit-config.yaml
file:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
# - id: trailing-whitespace
- id: check-yaml
- id: check-toml
Commit any changes:
git add .pre-commit-config.yaml
git commit -m "Add pre-commit configuration"
Tip
You can add additional pre-commit hooks to this file to suite your needs. See Pre-commit’s directory of available hooks for ideas.
Step 8. Update requirements.txt#
The Python dependencies for your technote are listed in a requirements.txt
file that should now look like this:
documenteer[technote]>=1.0,<2.0
Commit any changes:
git add requirements.txt
git commit -m "Update requirements.txt file"
Note
If your technote has additional dependencies listed, you can reach out to #dm-docs-support on Slack if you are unsure whether they are part of the Sphinx build process or separate packages needed for any custom document preprocessing.
Step 9. Add a tox.ini file#
Tox is a tool for running Python programs in dedicated virtual environments. This makes your local technote builds more reproducible by separating the technote’s dependencies from your system and other projects.
This is the recommended tox configuration to start with:
[tox]
environments = html
isolated_build = True
[testenv]
skip_install = true
deps =
-rrequirements.txt
[testenv:html]
commands =
sphinx-build --keep-going -n -W -T -b html -d _build/doctrees . _build/html
[testenv:linkcheck]
commands =
sphinx-build --keep-going -n -W -T -b linkcheck -d _build/doctrees . _build/linkcheck
[testenv:lint]
deps = pre-commit
commands =
pre-commit run --all-files
[testenv:add-author]
commands =
documenteer technote add-author
[testenv:sync-authors]
commands =
documenteer technote sync-authors
Step 10. Update the Makefile#
The Makefile
file provides a simple entrypoint for building your technote and performing other common tasks.
This is the suggested content for your Makefile
that works with the tox and pre-commit configurations:
.PHONY:
init:
pip install tox pre-commit
pre-commit install
.PHONY:
html:
tox run -e html
.PHONY:
lint:
tox run -e lint,linkcheck
.PHONY:
add-author:
tox run -e add-author
.PHONY:
sync-authors:
tox run -e sync-authors
.PHONY:
clean:
rm -rf _build
rm -rf .technote
rm -rf .tox
Step 11. Update GitHub Actions workflows#
Recent technotes have already migrated their GitHub Actions workflows to use the reusable workflow from lsst-sqre/rubin-sphinx-technote-workflows.
Check the .github/workflows/ci.yaml
file to make sure it looks like this:
name: CI
'on': [push, pull_request, workflow_dispatch]
jobs:
call-workflow:
uses: lsst-sqre/rubin-sphinx-technote-workflows/.github/workflows/ci.yaml@v1
with:
handle: example-001
secrets:
ltd_username: ${{ secrets.LTD_USERNAME }}
ltd_password: ${{ secrets.LTD_PASSWORD }}
Replace example-001
with your technote’s handle (the subdomain of lsst.io
).
If necessary, commit any changes:
git add .github/workflows/ci.yaml
git commit -m "Update GitHub Actions workflow"
Note
The original Rubin technotes used Travis CI for continuous integration and deployment, but we no longer use that service.
In that case, you will need to create the .github/workflows
directory and add the above ci.yaml
workflow.
GitHub Actions will automatically start using this workflow.
If your technote has a .travis.yml
file, you should delete it:
git rm .travis.yml
git commit -m "Remove Travis configuration"
Step 12. Add dependabot support#
Dependabot is a service provided by GitHub that generates pull requests when there are new versions of your technote’s dependencies.
Set up Dependabot by adding a .github/dependabot.yml
file:
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
Commit the changes:
git add .github/dependabot.yml
git commit -m "Add dependabot configuration"
Step 13. Update the README#
The README for a legacy-format technote likely has outdated information about how to build the technote. Here is a suggested README template for technotes in the new format:
.. image:: https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg
:target: https://example-000.lsst.io
.. image:: https://github.com/lsst/example-000/workflows/CI/badge.svg
:target: https://github.com/lsst/example-000/actions/
##############
Document Title
##############
EXAMPLE-000
===========
A short description of this document.
**Links:**
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
Build this technical note
=========================
You can clone this repository and build the technote locally if your system has Python 3.11 or later:
.. code-block:: bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
Repeat the ``make html`` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run ``make clean``.
The built technote is located at ``_build/html/index.html``.
Publishing changes to the web
=============================
This technote is published to https://example-000.lsst.io whenever you push changes to the ``main`` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
Editing this technical note
===========================
The main content of this technote is in ``index.rst`` (a reStructuredText file).
Metadata and configuration is in the ``technote.toml`` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.
Commit any changes:
git add README.rst
git commit -m "Update README for new technote format"
[![Website](https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg)](https://example-000.lsst.io)
[![CI](https://github.com/lsst/example-000/actions/workflows/ci.yaml/badge.svg)](https://github.com/lsst/example-000/actions/workflows/ci.yaml)
# Document Title
## EXAMPLE-000
A short description of this document.
### Links
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
## Build this technical note
You can clone this repository and build the technote locally if your system has Python 3.11 or later:
```bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
```
Repeat the `make html` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run `make clean`.
The built technote is located at `_build/html/index.html`.
## Publishing changes to the web
This technote is published to https://example-000.lsst.io whenever you push changes to the `main` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
## Editing this technical note
The main content of this technote is in `index.md` (a Markdown file).
Metadata and configuration is in the `technote.toml` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.
Commit any changes:
git add README.md
git commit -m "Update README for new technote format"
Step 14. Merge the migration#
At this point, you should have a working technote in the new format.
If you haven’t already, push your branch to the GitHub repository and open a pull request.
GitHub Actions will build the technote and publish a preview version that is linked from the /v
path of your technote’s website.
If the build works, you can merge the pull request.
If there are build errors, you can reach out to #dm-docs-support on Slack for help. Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed.