xpectra.pipeline.build_db
Build the three clean processed route files covering all eight datasets.
- Two sources feed the routes:
The five xpectrass reference datasets, loaded raw from the
xpectrasspackage loaders (the published, Zenodo-archived dataset) and preprocessed per dataset on the native grid (denoise -> baseline -> atmospheric-interpolate -> normalize), then interpolated onto the common 680-3100 cm^-1 grid withcombine_datasets.The three external augmentation datasets (
openspecy,simple,awi2024), whose already-normalized common-grid spectra are preserved in the package catalog DB (xpectra/data/xpectra_catalog.sqlite) and appended to the combined frame.
Savitzky-Golay derivatives are computed ONCE, on the full combined frame, after every dataset
sits on the identical grid in the identical (descending) column order. This is deliberate:
the derivative parameters are index-based (delta per point, window_length in points),
so taking derivatives on native grids or on differently ordered columns would give
per-dataset scale/sign artifacts. Combining first makes consistency automatic.
The result is exactly three files (norm / deriv1 / deriv2), each holding all eight datasets
distinguished by the study column; downstream code selects the labeled core studies or
the external pools via PreprocessConfig.labeled_studies / external_studies.
By default duplicates are removed per dataset (both ambiguous duplicate sample_id values and
exact-duplicate spectra, keep-first), so the matrices are clean at source. Identical spectra
appearing in two different sources (e.g. Primpke-2018 spectra present in both Open Specy and
siMPle) are legitimately kept — uniqueness is a within-study property. This module is the
single implementation shared by scripts/process_raw_data.py and the step-0 notebook.
Functions
|
End-to-end: xpectrass raw + catalog externals -> the three route DataFrames. |
|
Per-dataset preprocess -> combine to common grid -> append externals -> derivatives. |
|
Per-route counts of rows, unique (study, spectrum) and (study, sample_id) pairs. |
|
Load one dataset's raw spectra from the xpectrass loader. |
|
Load the external augmentation datasets from the catalog DB. |
|
Load every study's raw spectra from xpectrass (deduplicated by default). |
|
Write the route DataFrames as compressed CSVs. |
- xpectra.pipeline.build_db.load_dataset_frame(dataset_id, dedupe=True)[source]
Load one dataset’s raw spectra from the xpectrass loader.
When
dedupeis set, drop both ambiguous duplicatesample_idvalues and exact-duplicate spectra (keep first), so the output has unique ids and unique spectra.
- xpectra.pipeline.build_db.read_frames(dedupe=True, only=None, verbose=True)[source]
Load every study’s raw spectra from xpectrass (deduplicated by default).
- xpectra.pipeline.build_db.read_external_frames(catalog_path=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/xpectra/envs/latest/lib/python3.12/site-packages/xpectra/data/xpectra_catalog.sqlite'), dedupe=True, only=None, verbose=True)[source]
Load the external augmentation datasets from the catalog DB.
The catalog stores them already normalized on the common wavenumber grid, so they need no preprocessing — only appending to the combined frame.
studyis set to the dataset_id so route files can be filtered per source.
- xpectra.pipeline.build_db.build_routes(frames, config=None, externals=None, verbose=True)[source]
Per-dataset preprocess -> combine to common grid -> append externals -> derivatives.
- xpectra.pipeline.build_db.build_processed_routes(dedupe=True, only=None, config=None, include_external=True, verbose=True)[source]
End-to-end: xpectrass raw + catalog externals -> the three route DataFrames.
- xpectra.pipeline.build_db.write_routes(routes, out_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/xpectra/checkouts/latest/processed_data'), verbose=True)[source]
Write the route DataFrames as compressed CSVs.
- xpectra.pipeline.build_db.duplicate_report(routes)[source]
Per-route counts of rows, unique (study, spectrum) and (study, sample_id) pairs.
Uniqueness is keyed within study: the same spectrum appearing in two different sources (e.g. a Primpke-2018 spectrum in both Open Specy and siMPle) is a legitimate property of the sources, not a leak, and is never collapsed. All three counts should match.