Update a technote#
A technote repository falls behind Rubin’s current standard in two different ways.
One is age: a technote written before December 2023 keeps its metadata in a metadata.yaml file and has to be converted to the current format.
The other is drift: a technote that has been converted still carries the tooling files — the CI workflow, the tox and pre-commit configuration, the Makefile — that whichever Documenteer generated them wrote, and the standard has moved on since.
documenteer technote update covers both.
documenteer technote update
Run it from the root of the technote’s repository, or point it elsewhere with the -d/--dir option.
It inspects the directory, does whichever of the two jobs the technote needs, and leaves both kinds of technote in the same state.
Running it a second time changes nothing.
Tip
Your technote is in the legacy format if it has a metadata.yaml file at the top level of its repository.
LaTeX-format technotes are unaffected by this page.
Note
documenteer technote migrate is a deprecated alias for this command: it takes the same options, behaves identically, and prints a notice pointing here. See Migrate a legacy technote.
Refresh a technote’s tooling#
A technote that already has a technote.toml file is refreshed.
Documenteer rewrites the standard tooling files from its current templates, built from the metadata technote.toml declares — the technote’s handle, its repository URL, and its canonical URL.
These are the files it writes:
.github/dependabot.yml.github/workflows/ci.yaml.pre-commit-config.yaml.gitignoreMakefilerequirements.txttox.ini
Each of these seven files is replaced, not merged: Documenteer renders the template and writes the result over whatever the repository had.
Anything of your own in one of them — an extra tox environment, an added pre-commit hook, a comment explaining why the CI workflow does what it does — goes with the old copy of the file, and the report calls that updated like any other change.
If you need to keep such a file, hold it back with –ignore-file.
Merging a customization into the refreshed file, so that you need not choose between the template and your own copy, is tracked in documenteer#514.
Your own writing is never touched: README.rst and the content file (index.rst or index.md) are yours.
So is conf.py, the moment you put anything of your own in it — it is rewritten only when it holds nothing but the import line Documenteer generates, and is otherwise reported and left alone.
Every file is reported, whether or not it changed:
.github/dependabot.yml: updated
.github/workflows/ci.yaml: unchanged
.pre-commit-config.yaml: unchanged
.gitignore: unchanged
Makefile: updated
requirements.txt: unchanged
tox.ini: updated
conf.py: differs from every Documenteer template; left unchanged
Updated 3 files. Review the changes with 'git diff'.
Review the changes with git diff, then commit them and open a pull request as you would for any other change to the technote.
Keep a file you have customized#
If you have deliberately customized one of the standard files — an extra pre-commit hook, or a tox environment of your own — hold it back with --ignore-file, naming the file by the path the report prints:
documenteer technote update --ignore-file tox.ini --ignore-file .pre-commit-config.yaml
The option is repeatable, and an ignored file is reported as skipped (--ignore-file).
Caution
The Makefile and tox.ini are a pair: the Makefile’s targets are thin wrappers around the tox environments tox.ini defines.
Holding tox.ini back while the Makefile is refreshed leaves the refreshed targets calling environments the retained tox.ini may not define — make lint calls technote-lint and make sync-cff calls sync-cff — and each such target fails.
Hold both files back, or neither, and bring a retained tox.ini forward by hand.
Check without writing#
--check writes nothing and reports what would change, exiting non-zero when any file is out of date:
documenteer technote update --check
That is how to find out whether a technote has fallen behind without changing it — across a series of technotes at once, or from a technote’s own continuous integration.
A file held back with --ignore-file, and a conf.py of your own, are not out of date: neither one fails the check.
Convert a legacy technote#
A technote that still has a metadata.yaml file, and no technote.toml yet, is converted to the current format instead.
This update provides many new features including a responsive and branded page design, Markdown support, improved build configuration, automated bibliography management, and citation-ready metadata.
The steps below walk through the conversion, and file-by-file notes follow in case you need to revise it. The conversion requires:
Python 3.12 or later
The uvx command from the uv package
Step 1. Clone your technote and create a branch#
Since the conversion is performed locally, you need to clone the repository:
git clone https://github.com/{org}/{repo}
Then create a branch for the conversion (see the Developer Guide for more information on branching):
git switch -c technote-migration
Working from a branch allows you to create a pull request to verify the conversion before merging it into the main branch.
Step 3. Run the update command#
From the root of the cloned technote repository, run the update command, listing any identified authors that are relevant to your technote:
uvx documenteer technote update -a sickj -a economouf
Details for installing uv/uvx…
The update command needs Python 3.12 or later.
You can verify this by running python --version from your shell.
This official conversion procedure uses the uvx command from uv to install and run the update command in an isolated Python environment.
If you don’t have the uvx command-line tool already, you can install uv several ways:
python -m pip install uv
brew install uv
When the conversion runs, it shows a summary of the files changed and deleted.
Use git diff to review the changes in case you need to make tweaks before committing.
If you want to learn more about the changes the conversion makes, you can review the detailed changes, below.
Step 4 (optional). Migrate to Markdown#
The new technote format supports Markdown as well as reStructuredText. If you wish to use Markdown, you can use MyST Parser’s migration tool:
uvx --from "rst-to-myst[sphinx]" rst2myst convert index.rst
The Markdown flavor used by the new technote format is MyST Markdown, which is a superset of CommonMark Markdown with support for Sphinx roles and directives. See the MyST Parser documentation for more information.
Step 5 (optional). Build the technote locally#
You can build the technote locally to verify that the migrated technote compiles successfully.
GitHub Actions will test the technote build when you open a pull request (next step), so you can skip this step. However, you can get faster feedback by building locally.
make init
make html
The technote’s website will be in the _build/html directory.
Open the index.html file in your web browser to view the technote:
open _build/html/index.html
You can repeat the make html command to rebuild the technote after making changes.
You might also want to run the linter to check links and find common issues:
make lint
If you have any questions or issues about the build, you should still proceed with committing and creating a pull request (see next step). This way you can link to the repository when you reach out for help in #square-docs-support on Slack.
Step 6. Commit the conversion, pull request, and merge#
At this point, you should have a working technote in the new format.
If you haven’t already, commit your work, push your branch to the GitHub repository, and open a pull request.
GitHub Actions will build the technote and publish a preview version that is linked from the /v path of your technote’s website.
If the build works, you can merge the pull request.
If there are build errors, you can reach out to #square-docs-support on Slack for help. Include the repository URL and ideally a link to the pull request or GitHub Actions workflow run that failed.
Conversion details#
The conversion is automated by the documenteer technote update command, as described above. This section describes the steps it performs in detail, in case you need to make adjustments or understand the changes made to your technote.
technote.toml file (added)#
The technote.toml file replaces the original metadata.yaml file.
This new file provides both metadata and Sphinx configuration for your document.
Here is a simple technote.toml file:
[technote]
id = "EXAMPLE-000"
series_id = "EXAMPLE"
canonical_url = "https://example-000.lsst.io/"
github_url = "https://github.com/lsst/example-000"
github_default_branch = "main"
date_created = 2015-11-18
date_updated = 2023-11-01
organization.name = "Vera C. Rubin Observatory"
organization.ror = "https://ror.org/048g3cy84"
license.id = "CC-BY-4.0"
[[technote.authors]]
name = {given = "Drew", family = "Developer"}
internal_id = "example"
orcid = "https://orcid.org/0000-0001-2345-6789"
[[technote.authors.affiliations]]
name = "Rubin Observatory Project Office"
internal_id = "RubinObs"
Note
The schema for this file is described in the Technote package documentation, and elsewhere in the Documenteer documentation for Rubin technotes. For now, some pointers on important metadata:
idis the technote’s handle. A lower-cased version of the handle is the subdomain of the technote’s website. For example, the handle for https://sqr-000.lsst.io/ isSQR-000.series_idis the technote’s series handle. At Rubin, this is the handle’s prefix. Common series includeRTN,DMTN,SQR, andSITCOMTN.canonical_urlis the URL of the technote’s website.github_urlis the URL of the technote’s GitHub repository.date_createdis an optional field that specifies when the technote was first created.date_updatedis an optional field that specifies when the technote was last updated. If you omit it, the technote is dated by the commit it is published from; declaring it pins the date, including the date its citation states (see Where the date comes from).Each author is specified with a
[[technote.authors]]table (in TOML, the double brackets represent a table in an array of tables). Use the make add-author command to add an author to this file using data from authordb.yaml. It’s important to use theinternal_idfield to identify authors with their corresponding key in authordb.yaml. This enables Documenteer to update author information with the make sync-authors command.
conf.py file (updated)#
The conf.py file directly configures the Sphinx build process.
New technotes use a different configuration set provided by Documenteer that uses technote.toml to customize the Sphinx configuration.
For most technotes, the conf.py file is the template documenteer technote update writes — a comment pointing at these docs, and one import:
# See the Documenteer docs for how to customize conf.py:
# https://documenteer.lsst.io/technotes/
from documenteer.conf.technote import * # noqa: F403
If your conf.py file has additional content, some of that configuration may be migrated to technote.toml.
Reach out to #square-docs-support on Slack for advice.
index.rst file (updated)#
The index.rst file is the main content file for your technical note.
The new technote format requires some changes to this file: the title is now part of the content, the abstract is marked up with a directive, status information is now part of technote.toml, and the configuration for the reference section is dramatically simplified.
Additionally, the new technote format supports Markdown as well as reStructuredText. See migrate to Markdown to learn how to switch to Markdown.
Title#
The title is now part of the content, not the metadata. Add the title to the top of the content:
Document status#
The original technote format used a note directive to describe whether the document was a draft or deprecated.
Now this status metadata is structured in technote.toml.
Delete the note directive and add the status information to technote.toml following Describing the document status (draft, deprecated, or stable).
Abstract#
Legacy technotes either provided an abstract or summary through the description field in metadata.yaml, in a note directive in index.rst, or in a content section in index.rst.
The new technote format uses an abstract directive to mark up the abstract/summary.
######################
Example technical note
######################
.. abstract::
This is a summary of the technical note.
Introduction
============
[... content below ...]
# Example technical note
```{abstract}
This is a summary of the technical note.
```
## Introduction
[... content below ...]
Reference section#
If your technote makes references to other documents with roles like cite, you’ll need a reference section to display the bibliography.
In the new technote format, this section is simplified:
[... content above ...]
References
==========
.. bibliography::
[... content above ...]
## References
```{bibliography}
```
Specifically:
The references section should be a regular section, not a “rubric.”
The bibliography directive no longer requires any configuration; all configuration is provided by Documenteer.
metadata.yaml file (deleted)#
At this point, all relevant metadata about the technote is in technote.toml or index.rst/index.md.
Delete the deprecated metadata.yaml file:
git rm metadata.yaml
lsstbib/ directory (deleted)#
The legacy technote format vendored Rubin BibTeX bibliography files from lsst/lsst-texmf.
The new technote format automatically downloads and caches these files so that you no longer need to commit them into your repository.
Delete the lsstbib directory:
git rm -r lsstbib
.gitignore file (updated)#
The new technote format introduces additional directories that should be ignored by Git.
Ensure at least the following paths are included in the .gitignore file:
_build/
.technote/
.tox/
venv/
.venv/
.pre-commit-config.yaml file (added)#
Pre-commit is a Python package that runs validation and formatting checks on your technote’s repository before you commit.
Although it is not required, it’s highly recommended that you set up pre-commit hooks for your technote.
To start, add a .pre-commit-config.yaml file:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
# - id: trailing-whitespace
- id: check-yaml
- id: check-toml
Tip
You can add additional pre-commit hooks to this file to suit your needs. See Pre-commit’s directory of available hooks for ideas.
requirements.txt file (updated)#
The Python dependencies for your technote are listed in a requirements.txt file that should now look like this:
documenteer[technote]>=2.5.0,<3
Note
If your technote has additional dependencies listed, you can reach out to #square-docs-support on Slack if you are unsure whether they are part of the Sphinx build process or separate packages needed for any custom document preprocessing.
tox.ini file (added)#
Tox is a tool for running Python programs in dedicated virtual environments. This makes your local technote builds more reproducible by separating the technote’s dependencies from your system and other projects.
This is the recommended tox configuration to start with:
[tox]
environments = html
isolated_build = True
[testenv]
skip_install = true
deps =
-rrequirements.txt
[testenv:html]
commands =
sphinx-build --keep-going -n -W -T -b html -d _build/doctrees . _build/html
[testenv:linkcheck]
passenv = OOK_TOKEN
commands =
sphinx-build --keep-going -n -W -T -b linkcheck -d _build/doctrees . _build/linkcheck
[testenv:lint]
deps = pre-commit
commands =
pre-commit run --all-files
[testenv:add-author]
commands =
documenteer technote add-author
[testenv:sync-authors]
commands =
documenteer technote sync-authors
[testenv:sync-cff]
commands =
documenteer technote sync-cff
[testenv:technote-lint]
commands =
documenteer technote lint
documenteer technote sync-cff --check
Makefile file (updated)#
The Makefile file provides a simple entrypoint for building your technote and performing other common tasks.
This is the suggested content for your Makefile that works with the tox and pre-commit configurations:
.PHONY:
init:
pip install tox pre-commit
pre-commit install
.PHONY:
html:
tox run -e html
.PHONY:
lint:
tox run -e lint,technote-lint,linkcheck
.PHONY:
add-author:
tox run -e add-author
.PHONY:
sync-authors:
tox run -e sync-authors
.PHONY:
sync-cff:
tox run -e sync-cff
.PHONY:
clean:
rm -rf _build
rm -rf .technote
rm -rf .tox
.github/workflows/ci.yaml file (added/updated)#
Recent technotes have already migrated their GitHub Actions workflows to use the reusable workflow from lsst-sqre/rubin-sphinx-technote-workflows.
Check the .github/workflows/ci.yaml file to make sure it looks like this:
name: CI
'on': [push, pull_request, workflow_dispatch]
jobs:
call-workflow:
uses: lsst-sqre/rubin-sphinx-technote-workflows/.github/workflows/ci.yaml@v1
with:
handle: example-001
secrets:
ltd_username: ${{ secrets.LTD_USERNAME }}
ltd_password: ${{ secrets.LTD_PASSWORD }}
Replace example-001 with your technote’s handle (the subdomain of lsst.io).
Note
The original Rubin technotes used Travis CI for continuous integration and deployment, but we no longer use that service.
In that case, you will need to create the .github/workflows directory and add the above ci.yaml workflow.
GitHub Actions will automatically start using this workflow.
If your technote has a .travis.yml file, you should delete it:
git rm .travis.yml
.github/dependabot.yml file (added)#
Dependabot is a service provided by GitHub that generates pull requests when there are new versions of your technote’s dependencies.
Set up Dependabot by adding a .github/dependabot.yml file:
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
README.rst or README.md file (updated)#
The README for a legacy-format technote likely has outdated information about how to build the technote. Here is a suggested README template for technotes in the new format:
.. image:: https://img.shields.io/badge/example--000-lsst.io-brightgreen.svg
:target: https://example-000.lsst.io
.. image:: https://github.com/lsst/example-000/workflows/CI/badge.svg
:target: https://github.com/lsst/example-000/actions/
##############
Document Title
##############
EXAMPLE-000
===========
A short description of this document.
**Links:**
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
Build this technical note
=========================
You can clone this repository and build the technote locally if your system has Python 3.12 or later:
.. code-block:: bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
Repeat the ``make html`` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run ``make clean``.
The built technote is located at ``_build/html/index.html``.
Publishing changes to the web
=============================
This technote is published to https://example-000.lsst.io whenever you push changes to the ``main`` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
Editing this technical note
===========================
The main content of this technote is in ``index.rst`` (a reStructuredText file).
Metadata and configuration is in the ``technote.toml`` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.
[](https://example-000.lsst.io)
[](https://github.com/lsst/example-000/actions/workflows/ci.yaml)
# Document Title
## EXAMPLE-000
A short description of this document.
### Links
- Publication URL: https://example-000.lsst.io
- Alternative editions: https://example-000.lsst.io/v
- GitHub repository: https://github.com/lsst/example-000
- Build system: https://github.com/lsst/example-000/actions/
## Build this technical note
You can clone this repository and build the technote locally if your system has Python 3.12 or later:
```bash
git clone https://github.com/lsst/example-000
cd example-000
make init
make html
```
Repeat the `make html` command to rebuild the technote after making changes.
If you need to delete any intermediate files for a clean build, run `make clean`.
The built technote is located at `_build/html/index.html`.
## Publishing changes to the web
This technote is published to https://example-000.lsst.io whenever you push changes to the `main` branch on GitHub.
When you push changes to a another branch, a preview of the technote is published to https://example-000.lsst.io/v.
## Editing this technical note
The main content of this technote is in `index.md` (a Markdown file).
Metadata and configuration is in the `technote.toml` file.
For guidance on creating content and information about specifying metadata and configuration, see the Documenteer documentation: https://documenteer.lsst.io/technotes.