Overview of the pipelines.lsst.io build system#

This page is a high-level overview of the tools and processes for building the pipelines.lsst.io documentation.

To learn how to build the pipelines.lsst.io site in practice, see Building the pipelines.lsst.io site locally. For more background on contributing to pipelines.lsst.io documentation, see Overview of the Stack documentation system.

Documenteer#

Documenteer is the Python package that provides tooling for LSST DM’s Sphinx-based documentation projects, including Stack user guides such as pipelines.lsst.io. Documenteer plays four roles:

  • Documenteer manages and versions the Python dependencies needed to build Sphinx documentation. The requirements.txt file in the pipelines_lsst_io repository refers to a specific release of Documenteer, which in turn specifies its own Python dependencies.

  • Documenteer provides centralized Sphinx configurations for both the pipelines_lsst_io repository and individual packages.

  • Documenteer provides Sphinx extensions built specifically for LSST DM needs.

  • Documenteer provides the stack-docs command-line app that runs the Sphinx documentation build process (Documenteer also provides the related package-docs app for developer:single-package documentation builds).

Stack build and set up#

The prerequisite for the Stack documentation build is that the Stack is already built and set up. Sphinx imports Python packages to generate documentation for their APIs.

The main documentation repository (pipelines_lsst_io) has an EUPS table file. This table file defines all the Stack packages that should be set up and that are available for inclusion in the Stack documentation build. See How to add a package to pipelines.lsst.io for details.

Doxygen build#

Doxygen extracts information from the source code to generate the C++ API reference. Documenteer runs Doxygen as a step in the stack-docs command.

First, Documenteer identifies which packages contain a doc/doxygen.conf.in file. Any package containing a Doxygen configuration stub file, even if empty, is presumed to potentially contain source material the C++ API reference.

Second, Documenteer constructs a Doxygen configuration. By default, the include directory of each relevant package is included in the INPUT Doxygen configuration tag. Individual packages can also add other paths to the INPUT and IMAGE_PATH tags, remove paths (EXCLUDE or EXCLUDE_PATTERNS tags), or exclude symbols (EXCLUDE_SYMBOLS tag).

Finally, Documenteer uses this combined Doxygen configuration to run the doxygen command to generate an HTML site and tag file that exclusively documents the C++ API reference. Sphinx copies the Doxygen site into the cpp-api directory during its build so that the Doxygen-generated API reference effectively becomes an sub-site of the Sphinx-rendered site. The pipelines.lsst.io documentation project has a special lsstcc role, created through doxylink extension, using the Doxygen tag file, that allows reStructuredText content to link to C++ API reference pages in the Doxygen site.

For more information about Documenteer’s built-in Doxygen build, see the documenteer.stackdocs.doxygen module, and the run_doxygen and DoxygenConfiguration APIs in particular. From a documentation writer’s perspective, see also: Cross-linking the Doxygen-built C++ API reference with the lsstcc role.

Package documentation linking#

The next step of the build is to run stack-docs, from Documenteer.

The stack-docs app begins by discovering packages that are set up by EUPS and that also have doc/manifest.yaml files. Following the doc/manifest.yaml file, stack-docs symlinks the module and package documentation directories into the modules/ and packages/ directories of the pipelines_lsst_io repository.

The Sphinx build#

Next, stack-docs runs the Sphinx build on the pipelines_lsst_io repository. This is different from most projects that use a Makefile and Sphinx’s builtin sphinx-build app. stack-docs is an all-in-one front-end designed and engineered for building the pipelines.lsst.io site.

The conf.py file in pipelines_lsst_io configures the Sphinx build, which in turn uses Documenteer’s centralized Sphinx configuration API.

Since the documentation content from all the packages is symlinked into the pipelines_lsst_io repository, a single Sphinx build generates the entire pipelines.lsst.io site.

Python API reference#

The automodapi directives in module homepages generate the Python API reference documentation. Specifically, automodapi introspects the APIs in the given Python module and creates stub files in the py-api directory that contain autodoc directives that generate the documentation page for each API. The numpydoc extension transforms docstrings before they are read by autodoc from the numpydoc format to Sphinx’s native docstring markup.

The py-api directory is entirely transient. In fact, the stack-docs clean command will delete it.

See Module homepage topic type for more information on the automodapi directives.