Supervised Diffsky Prior Learning

Purpose

The supervised prior workflow learns a population density directly from truth parameters:

\[L_\mathrm{prior} = -\frac{1}{N}\sum_i \log p_\beta(x_{\mathrm{true}, i})\]

where theta_true is mapped to unconstrained x_true with the same bounded logistic transforms used by amortized latent variables. This workflow does not use photometry, an encoder, or the DSPS decoder.

alpha_sed is not part of this workflow. It is a global decoder calibration nuisance parameter used by photometric likelihood paths, not a galaxy physical parameter. The supervised prior therefore does not add alpha_sed to theta_true, does not sample it per galaxy, and does not compare it to object-level ground truth.

It is separate from:

same-parameter forward closure, which tests whether theta_true can reproduce the catalog photometry;
photometric amortized inference, which learns q(theta | flux).

A good photometric fit is not evidence of physical recovery. Physical claims require same-parameter forward closure, supervised prior-vs-truth diagnostics, posterior calibration, and comparison of derived physical quantities rather than only raw latent parameters.

Schemas

Production closure schema

diffsky_dsps_closure_full is the production schema for the synthetic FENIKS/DSPS closure dataset. It requires one truth column for every fitted diffsky_basic parameter and uses missing_policy: fail in configs/prior_diffsky_synthetic_feniks_full_realnvp.yaml.

redshift_true                         -> z_obs
logsm_true                            -> log10_stellar_mass
diffstar_lgmcrit_true                 -> diffstar_lgmcrit
diffstar_lgy_at_mcrit_true            -> diffstar_lgy_at_mcrit
diffstar_indx_lo_true                 -> diffstar_indx_lo
diffstar_indx_hi_true                 -> diffstar_indx_hi
diffstar_lg_qt_true                   -> diffstar_lg_qt
diffstar_qlglgdt_true                 -> diffstar_qlglgdt
diffstar_lg_drop_true                 -> diffstar_lg_drop
diffstar_lg_rejuv_true                -> diffstar_lg_rejuv
diffmah_logm0_true                    -> diffmah_logm0
diffmah_logtc_true                    -> diffmah_logtc
diffmah_early_index_true              -> diffmah_early_index
diffmah_late_index_true               -> diffmah_late_index
diffmah_t_peak_true                   -> diffmah_t_peak
log10_stellar_metallicity_true        -> log10_stellar_metallicity
dust_av_true                          -> dust_av
dust_delta_true                       -> dust_delta

Train the production prior after closure validation passes:

python -m euclid_dsps.cli \
  --config configs/diffsky_synthetic_feniks_260617_50k.yaml \
  diffsky-plan-prior-workflow \
  --out outputs/reports/feniks_prior_workflow

python -m euclid_dsps.cli \
  --config configs/prior_diffsky_synthetic_feniks_full_realnvp.yaml \
  diffsky-train-supervised-prior \
  --out outputs/runs/prior_diffsky_synthetic_feniks_full_realnvp

The workflow plan is a cheap preflight. It reads the configured parquet schemas and row counts, verifies that diffsky_dsps_closure_full can resolve all 18 truth parameters, and writes the expected supervised-prior, NN+DSPS+NF, MAP-under-prior, MCLMC-baseline, and post-hoc inferred-prior commands.

Configs

configs/prior_diffsky_synthetic_feniks_full_realnvp.yaml

Train

python -m euclid_dsps.cli \
  --config configs/prior_diffsky_synthetic_feniks_full_realnvp.yaml \
  diffsky-train-supervised-prior \
  --out outputs/runs/prior_diffsky_synthetic_feniks_full_realnvp

The run writes:

prior_training_log.csv
prior_validation_loglike.csv
learned_prior_samples.parquet
truth_theta_samples.parquet
truth_x_samples.parquet
supervised_prior_summary.json
supervised_prior_vs_truth_report.md
prior_vs_truth_metrics.csv
checkpoints/best.eqx
checkpoints/last.eqx

Sample

python -m euclid_dsps.cli \
  --config configs/prior_diffsky_synthetic_feniks_full_realnvp.yaml \
  diffsky-sample-supervised-prior \
  --checkpoint outputs/runs/prior_diffsky_synthetic_feniks_full_realnvp/checkpoints/best.eqx \
  --n-samples 50000 \
  --out outputs/runs/prior_diffsky_synthetic_feniks_full_realnvp_samples

Report

python -m euclid_dsps.cli \
  --config configs/prior_diffsky_synthetic_feniks_full_realnvp.yaml \
  diffsky-supervised-prior-report \
  --run outputs/runs/prior_diffsky_synthetic_feniks_full_realnvp \
  --dataset Data/diffsky/synthetic/feniks_260617_dsps_closure/test.parquet

Diagnostics include per-parameter histogram comparisons, KS distance, Wasserstein distance, mean/std/median residuals, a z/logM/logSFR pair plot when those parameters are present, and a corner plot when corner is installed.