Preprocessing routes & the route-prefix convention

The three routes

xpectra works over three preprocessing routes, all produced in a single pass by xpectra-process-raw and stored under processed_data/:

Route	File	Content
`norm`	`combined_norm_data.csv.xz`	`spectral_moments`-normalized absorbance
`deriv1`	`combined_norm_deriv1_data.csv.xz`	first Savitzky–Golay derivative
`deriv2`	`combined_norm_deriv2_data.csv.xz`	second Savitzky–Golay derivative

The defaults reproduce the reference preprocessing: wavelet denoising, aspls baseline correction, atmospheric interpolation with zero, spectral_moments normalization, and Savitzky–Golay derivatives with window_length=15, polyorder=3. Derivatives are computed once, after combining all datasets on the shared 680–3100 cm⁻¹ grid, so every spectrum receives identical derivative scaling and sign.

Note

The carbonyl index and other physical band ratios are only meaningful on the norm route. Derivative routes amplify high-frequency noise and discard absolute band areas.

Selecting a route

Library loaders take the route as their first argument:

lab = load_labeled("deriv1", trust)

The study CLIs take a --route flag (default norm):

xpectra-study deep-ssl --route deriv1
xpectra-nb3-analysis --route norm

The route-prefix convention

Two output conventions coexist, by design:

Single-route outputs are prefixed with the route name. Any file that represents one route’s result carries a norm_ / deriv1_ / deriv2_ prefix, so runs on different routes never overwrite one another:

results/norm_deep_ssl_loso.csv
results/deriv1_deep_ssl_loso.csv
figures/norm_figure3_main_contrastive_transfer.png
.cache/xpectra/norm_contrastive_emb_labeled.npy

Multi-route sweeps carry a route column instead. The classical sweeps (calibrate, loso, ood, ssl) evaluate all three routes in one run and write a single unprefixed file with a route column:

results/loso_folds.csv          # rows for norm, deriv1, deriv2
results/calibration_metrics.csv

To consume a specific route from a sweep file, filter the column:

import pandas as pd
folds = pd.read_csv("results/loso_folds.csv")
norm_folds = folds[folds["route"] == "norm"]