Migrate legacy Sphinx technical notes#
Introduced in November 2023, Rubin Observatory has a new technical note format for Markdown and reStructuredText documents, based on the Sphinx and the Technote Python packages.
If your started writing your technical note before November 2023, it is likely in the legacy Sphinx format.
You can tell if your technical note is in the legacy format if it has a metadata.yaml file.
If you are continuing to write such a technical note, you should migrate it to the new format to take advantage of new features.
This page provides a guide to migrating your technical note to the new format.
To get started on your migration, be sure to have your technote repository checked-out locally.
Work on a ticket branch, per the Developer Guide, so you can check your work with a pull request before merging to the main branch.
Step 1. Create a technote.toml file#
The technote.toml file replaces the original metadata.yaml file.
This new file provides both metadata and Sphinx configuration for your document.
Here is a simple technote.toml file to get you started:
[technote]
id = "EXAMPLE-000"
series_id = "EXAMPLE"
canonical_url = "https://example-000.lsst.io/"
github_url = "https://github.com/lsst/example-000"
github_default_branch = "main"
date_created = "2015-11-18"
date_updated = "2023-11-01"
[[technote.authors]]
name = { given_names = "Drew", family_names = "Developer" }
orcid = "https://orcid.org/0000-0001-2345-6789"
affiliations = [
{ name = "Rubin Observatory", ror = "https://ror.org/048g3cy84" }
]
Add this file to your repository and commit it:
git add technote.toml
git commit -m "Add technote.toml file"
Step 2. Update conf.py#
The conf.py file directly configures the Sphinx build process.
New technotes use a different configuration set provided by Documenteer that uses technote.toml to customize the Sphinx configuration.
For most technotes, the conf.py file should be a single line:
from documenteer.conf.technote import * # noqa: F401, F403
Commit any changes:
git add conf.py
git commit -m "Update conf.py to new technote format"
If your conf.py file has additional content, some of that configuration may be migrated to technote.toml.
Reach out to #dm-docs-support on Slack for advice.
Step 3. Update the index.rst file#
The index.rst file is the main content file for your technical note.
The new technote format requires some changes to this file: the title is now part of the content, the abstract is marked up with a directive, status information is now part of technote.toml, and the configuration for the reference section is dramatically simplified.
Tip
Besides these changes, your technote can also be written in Markdown (index.md).
If you wish to switch from ReStructuredText to markdown, install pandoc and run:
pandoc -f rst -t markdown -o index.md index.rst
Then, delete the original index.rst file and edit index.md to fix any formatting issues.
git rm index.rst
git add index.md
git commit -m "Switch to Markdown format"
Add the title#
The title is now part of the content, not the metadata. Add the title to the top of the content:
Move document status information to technote.toml#
The original technote format used a note directive to describe whether the document was a draft or deprecated.
Now this status metadata is structured in technote.toml.
Delete the note directive and add the status information to technote.toml following Describing the document status (draft, deprecated, or stable).
Format the abstract/summary with the abstract directive#
Legacy technotes either provided an abstract or summary through the description field in metadata.yaml, in a note directive in index.rst, or in a content section in index.rst.
The new technote format uses an abstract directive to mark up the abstract/summary.
######################
Example technical note
######################
.. abstract::
This is a summary of the technical note.
Introduction
============
[... content below ...]
# Example technical note
```{abstract}
This is a summary of the technical note.
```
## Introduction
[... content below ...]
Simplify the reference section#
If your technote makes references to other documents with roles like cite, you’ll need a reference section to display the bibliography.
In the new technote format, this section is simplified:
[... content above ...]
References
==========
.. bibliography::
[... content above ...]
## References
```{bibliography}
```
Specifically:
The references section should be a regular section, not a “rubric.”
The bibliography directive no longer requires any configuration; all configuration is provided by Documenteer.
Commit any changes#
git add index.rst
git commit -m "Reformat index.rst for new technote format"
git add index.md
git commit -m "Reformat index.md for new technote format"
Step 4. Delete metadata.yaml#
At this point, all relevant metadata about the technote is in technote.toml or index.rst/index.md.
Delete the deprecated metadata.yaml file:
git rm metadata.yaml
git commit -m "Remove metadata.yaml file"
Step 5. Delete the lsstbib/ directory#
The legacy technote format vendored Rubin BibTeX bibliography files from lsst/lsst-texmf.
The new technote format automatically downloads and caches these files so that you no longer need to commit them into your repository.
Delete the lsstbib directory:
git rm -r lsstbib
git commit -m "Remove lsstbib/ directory"
Step 6. Update .gitignore#
The new technote format introduces additional directories that should be ignored by Git.
Ensure at least the following paths are included in the .gitignore file:
_build
.technote
.tox
venv
.venv
git add .gitignore
git commit -m "Update .gitignore file"
Step 7. Set up pre-commit hooks#
Pre-commit is a Python package that runs validation and formatting checks on your technote’s repository before you commit.
Although it is not required, it’s highly recommended that you set up pre-commit hooks for your technote.
To start, add a .pre-commit-config.yaml file:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: check-yaml
- id: check-toml
Commit any changes:
git add .pre-commit-config.yaml
git commit -m "Add pre-commit configuration"
Tip
You can add additional pre-commit hooks to this file to suite your needs. See Pre-commit’s directory of available hooks for ideas.
Step 6. Update requirements.txt#
The Python dependencies for your technote are listed in a requirements.txt file that should now look like this:
documenteer[technote]>=1.0.0,<2.0.0
Commit any changes:
git add requirements.txt
git commit -m "Update requirements.txt file"
Note
If your technote has additional dependencies listed, you can reach out to #dm-docs-support on Slack if you are unsure whether they are part of the Sphinx build process or separate packages needed for any custom document preprocessing.
Step 7. Add a tox.ini file#
Tox is a tool for running Python programs in dedicated virtual environments. This makes your local technote builds more reproducible by separating the technote’s dependencies from your system and other projects.
This is the recommended tox configuration to start with:
[tox]
environments = html
isolated_build = True
[testenv]
skip_install = true
deps =
-rrequirements.txt
[testenv:html]
commands =
sphinx-build --keep-going -n -W -T -b html -d {envtmpdir}/doctrees . _build/html
[testenv:linkcheck]
commands =
sphinx-build --keep-going -n -W -T -b linkcheck -d {envtmpdir}/doctrees . _build/linkcheck
[testenv:lint]
commands =
pre-commit run --all-files
Step 7. Update the Makefile#
The Makefile file provides a simple entrypoint for building your technote and performing other common tasks.
This is the suggested content for your Makefile that works with the tox and pre-commit configurations:
.PHONY:
init:
pip install tox pre-commit
pre-commit install
.PHONY:
html:
tox run -e html
.PHONY:
lint:
tox run -e lint,link-check
.PHONY:
clean:
rm -rf _build
rm -rf .technote
rm -rf .tox
Step 8. Update GitHub Actions workflows#
Recent technotes have already migrated their GitHub Actions workflows to use the reusable workflow from lsst-sqre/rubin-sphinx-technote-workflows.
Check the .github/workflows/ci.yaml file to make sure it looks like this:
name: CI
'on': [push, pull_request, workflow_dispatch]
jobs:
call-workflow:
uses: lsst-sqre/rubin-sphinx-technote-workflows/.github/workflows/ci.yaml@v1
with:
handle: example-001
secrets:
ltd_username: ${{ secrets.LTD_USERNAME }}
ltd_password: ${{ secrets.LTD_PASSWORD }}
Replace example-001 with your technote’s handle (the subdomain of lsst.io).
If necessary, commit any changes:
git add .github/workflows/ci.yaml
git commit -m "Update GitHub Actions workflow"
Note
The original Rubin technotes used Travis CI for continuous integration and deployment, but we no longer use that service.
In that case, you will need to create the .github/workflows directory and add the above ci.yaml workflow.
GitHub Actions will automatically start using this workflow.
If your technote has a .travis.yml file, you should delete it:
git rm .travis.yml
git commit -m "Remove Travis configuration"
Step 9. Add dependabot support#
Dependabot is a service provided by GitHub that generates pull requests when there are new versions of your technote’s dependencies.
Set up Dependabot by adding a .github/dependabot.yml file:
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
Commit the changes:
git add .github/dependabot.yml
git commit -m "Add dependabot configuration"
Step 10. Update the README#
The README for a legacy-format technote likely has outdated information about how to build the technote. Here is a suggested README template for technotes in the new format:
.. image:: https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg
:target: https://example-000.lsst.io
.. image:: https://github.com/lsst/example-000/workflows/CI/badge.svg
:target: https://github.com/lsst/example-000/actions/
##############
Document Title
##############
EXAMPLE-000
===========
A short description of this document.
**Links:**
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
Build this technical note
=========================
You can clone this repository and build the technote locally if your system has Python 3.11 or later:
.. code-block:: bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
Repeat the ``make html`` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run ``make clean``.
The built technote is located at ``_build/html/index.html``.
Publishing changes to the web
=============================
This technote is published to https://example-000.lsst.io whenever you push changes to the ``main`` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
Editing this technical note
===========================
The main content of this technote is in ``index.rst`` (a reStructuredText file).
Metadata and configuration is in the ``technote.toml`` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.
Commit any changes:
git add README.rst
git commit -m "Update README for new technote format"
[](https://example-000.lsst.io)
[](https://github.com/lsst/example-000/actions/workflows/ci.yaml)
# Document Title
## EXAMPLE-000
A short description of this document.
### Links
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
## Build this technical note
You can clone this repository and build the technote locally if your system has Python 3.11 or later:
```bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
```
Repeat the `make html` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run `make clean`.
The built technote is located at `_build/html/index.html`.
## Publishing changes to the web
This technote is published to https://example-000.lsst.io whenever you push changes to the `main` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
## Editing this technical note
The main content of this technote is in `index.md` (a Markdown file).
Metadata and configuration is in the `technote.toml` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.
Commit any changes:
git add README.md
git commit -m "Update README for new technote format"
Step 11. Merge the migration#
At this point, you should have a working technote in the new format.
If you haven’t already, push your branch to the GitHub repository and open a pull request.
GitHub Actions will build the technote and publish a preview version that is linked from the /v path of your technote’s website.
If the build works, you can merge the pull request.
If there are build errors, you can reach out to #dm-docs-support on Slack for help. Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed.