Validation¶

Every brick in the catalogue cites the paper it implements. This section shows the check behind the citation: a number printed in that paper, or a laboratory rest wavelength, next to the number the brick measures, with a tolerance and the reason for it. The catalogue is generated when the documentation is built by running every case; the same results are served as JSON for tools and agents.

The registry is one file, tests/reference/cases.py. The reference test suite (pytest tests/reference) asserts the same cases: a value drifting outside its tolerance fails CI there. The page only reports what this build measured.

Two families of cases¶

Literature values on synthetic inputs. A synthetic spectrum is built so that the answer is known by construction, and the brick must reproduce the value the paper prints: Hα 6562.79 Å air becomes 6564.603 Å vacuum with Morton's relation, [O III] (4959 + 5007) / 4363 = 100 gives T_e = 12 961 K with Osterbrock & Ferland's Eq. 5.4, the Kasten & Young airmass at z = 60° is 1.994293, and so on. These are the acceptance values the unit tests already assert, gathered in one place with their citation.

Real data. The CALSPEC spectrophotometric standards tracked under tests/reference/data/ (Git LFS) are measured with the catalogue bricks and compared with NIST rest wavelengths or MK standard types. Nothing here is synthetic: the Balmer lines of Vega are fitted on the STIS composite after vacuum_to_air and a radial-velocity correction, and their centres are compared with the laboratory values.

Reading the tolerances¶

A tolerance is either absolute (±x in the quantity's unit) or relative (±x of the published value), and every case says where it comes from:

• for a literature value, the rounding of the published number or the known accuracy of the approximation (Olivero & Longbothum's Voigt FWHM is good to 0.02 %, so the case allows 0.1 %);
• for a fit on synthetic data, the sensitivity of the result to the fitted quantity (at R = 100 the [O III] temperature moves by 51 K per unit of R, so 100 K allows a 2 % error on the ratio);
• for real data, the resolution of the fixture: half the FWHM column of the CALSPEC table at the line (2.75 Å for Hβ, Hγ, Hδ on the STIS G430L segment, 4.9 Å for Hα on G750L). The measured deviations are shown next to the status, so a case that passes by a wide margin reads differently from one that passes at the edge.

A tolerance is never tightened to make a case pass, and never loosened without a reason written in the note field.

What the fixtures can and cannot check¶

The solar fixture does not resolve the Na I D doublet (5.97 Å apart) nor the 1.09 Å cores of Ca II H and K, so the doublet separation and the S-index are recorded on the catalogue page as not measurable rather than as cases. The spectral-type cases (Vega → A0 V, Sun → G2 V with classify_template_chi2) need the Pickles 1998 atlas, which is neither bundled nor downloaded: they run when SPECTRO_PICKLES_ATLAS_DIR points at a local copy and are shown as not run in this build otherwise.

Adding a case¶

Edit tests/reference/cases.py only. A case is a ValidationCase with an id, the algorithm under test, a dataset (a synthetic recipe or one of the fixtures), the quantity with its unit, the published value, its verbatim source, a measured_fn that builds the input and returns the measured value, a tolerance and the note that justifies it. A real-data case raises CaseSkipped with the reason when its input is unavailable. Then:

.venv/bin/python -m pytest tests/reference -q          # the case is now a test
.venv/bin/python tools/docs/generate_validation.py     # and a row of the page

The published value must come from the cited paper or standard, never from the brick's own output. If no published number exists for a quantity, it belongs in the not measurable list with the reason, not in the table.

Why real data¶

A synthetic test proves that a brick does what its formula says. Only a real spectrum, with its resolution, sampling and calibration, shows that the formula was the right one to apply and that the surrounding steps (frame conversion, continuum, windowing) were composed correctly. Reference standards such as the CALSPEC set are what an observatory pipeline is checked against; publishing the same check, with the numbers, is what lets a professional user decide whether the kernel's measurement can stand next to their own.