- Running Cosmolike projects (Basic instructions)
- Baryonic feedback on EXAMPLE_EVALUATE1
- Running ML emulators
- Running Hybrid Cosmolike-ML emulators
- Training Roman ML emulators
- Unit tests
- Computing covariances
- Exploring notebooks
- Appendix: Which accuracy settings are available?
From Cocoa/Readme instructions:
Note
setup_cocoa.sh and compile_cocoa.sh install the cosmolike projects that set_installation_options.sh selects: a commented IGNORE_*_CODE key enables a project, and an active key skips it. The shipped file enables roman_real:
[Adapted from Cocoa/set_installation_options.sh shell script]
#export IGNORE_COSMOLIKE_LSST_Y1_CODE=1
export IGNORE_COSMOLIKE_DES_Y3_CODE=1
#export IGNORE_COSMOLIKE_DESXPLANCK_CODE=1
export IGNORE_COSMOLIKE_ROMAN_FOURIER_CODE=1
#export IGNORE_COSMOLIKE_ROMAN_REAL_CODE=1
export IGNORE_COSMOLIKE_ROMAN_KL_CODE=1
(...)
export ROMAN_REAL_URL="https://github.com/CosmoLike/cocoa_roman_real.git"
export ROMAN_REAL_NAME="roman_real"
export ROMAN_REAL_GIT_TAG="v5.05"
Each released project is pinned to a tag. To select another revision, set
only one of its GIT_COMMIT, GIT_BRANCH or GIT_TAG keys: a commit takes
precedence over a branch, and a branch over a tag.
Note
If users want to recompile cosmolike, there is no need to rerun the Cocoa general scripts. Instead, run the following three commands:
source start_cocoa.sh
and
source ./installation_scripts/setup_cosmolike_projects.sh
and
source ./installation_scripts/compile_all_projects.sh
or (in case users just want to compile roman_real project)
source ./projects/roman_real/scripts/compile_roman_real.sh
Tip
Assuming Cocoa is installed on a local (not remote!) machine, type the command below after step 2️⃣ to run Jupyter Notebooks.
jupyter notebook --no-browser --port=8888
The terminal will then show a message similar to the following template:
(...)
[... NotebookApp] Jupyter Notebook 6.1.1 is running at:
[... NotebookApp] http://f0a13949f6b5:8888/?token=XXX
[... NotebookApp] or http://127.0.0.1:8888/?token=XXX
[... NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Now go to the local internet browser and type http://127.0.0.1:8888/?token=XXX, where XXX is the previously saved token displayed on the line
[... NotebookApp] or http://127.0.0.1:8888/?token=XXX
The project roman_real contains jupyter notebook examples located at projects/roman_real.
Note
The example notebooks load their shared support functions from
Cocoa/external_modules/code/cosmolike_core/cosmolike_notebook_utils/:
the CAMB run packaged for cosmolike, the data-vector plots, and the
Fisher-forecast helpers. The notebooks keep only what is specific to
this project: fiducial values, the compiled-interface calls, and thin
wrappers binding them to the shared functions.
To run the example
Step 1️⃣: activate the cocoa Conda environment, and the private Python environment
conda activate cocoa
and
source start_cocoa.sh
Step 2️⃣: Select the number of OpenMP cores (below, we set it to 8).
-
Linux
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1 -
macOS (arm)
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
Step 3️⃣: The folder projects/roman_real contains examples. So, run the cobaya-run on the first example following the commands below.
Warning
(Linux only) In some HPC nodes, numa can cause you problems. If that is the case,
replace numa with slot
-
One model evaluation:
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 1 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ cobaya-run ./projects/roman_real/EXAMPLE_EVALUATE1.yaml -f -
macOS (arm)
mpirun -n 1 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_EVALUATE1.yaml -f
-
-
MCMC (Metropolis-Hastings Algorithm):
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 4 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ cobaya-run ./projects/roman_real/EXAMPLE_MCMC1.yaml -f -
macOS (arm)
mpirun -n 4 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_MCMC1.yaml -f
-
EXAMPLE_EVALUATE1.yaml can apply an external baryonic feedback suppression to the
matter power spectrum via the bfmt theory block (SP(k), BCEmu, Flamingo, BACCOemu,
or BCemu2025). By default, the example runs without feedback.
Step 1️⃣: ensure the lines below are commented out in set_installation_options.sh
before running setup_cocoa.sh and compile_cocoa.sh. By default, these lines should
be commented out, but it is worth checking.
[Adapted from Cocoa/set_installation_options.sh shell script]
#export IGNORE_PYSPK_CODE=1 # SP(k)
#export IGNORE_BCEMU_CODE=1 # BCEmu
#export IGNORE_FBRE_CODE=1 # FlamingoBaryonResponseEmulator
#export IGNORE_BACCOEMU_CODE=1 # BACCOemu
#export IGNORE_BFMT_CODE=1 # Baryon Feedback Theory Block
Step 2️⃣: in EXAMPLE_EVALUATE1.yaml, uncomment the bfmt theory block and select
the model:
theory:
bfmt:
baryon_model: 2 # 1 = SP(k), 2 = BCEmu, 3 = FlamingoEmulator, 4 = BACCOemu, 5 = BCemu2025
Step 3️⃣: set external_baryon_suppression: True on the roman_real.cosmic_shear
likelihood block.
Step 4️⃣: uncomment the selected model's parameters in the params block and in
the sampler: evaluate: override block (the example carries a commented block for each
model).
Tip
For the sampled parameters of each model, their validity ranges, and the bfmt
options, see Cocoa/external_modules/code/baryon_suppression/README.md.
Note
EXAMPLE_EVALUATE3.ipynb runs six of these methods
(the three SP(k) relations, BCEmu, Flamingo and BCemu2025) through the bfmt
block on the 3x2pt data vector. It needs the packages of the first step above;
see Exploring notebooks.
Cocoa contains a few transformer- and CNN-based neural network emulators capable of simulating the CMB, cosmolike outputs, matter power spectrum, and distances. We provide a few scripts that exemplify their API. To run them, users ensure the following lines are commented out in set_installation_options.sh before running the setup_cocoa.sh and compile_cocoa.sh. By default, these lines should be commented out, but it is worth checking.
[Adapted from Cocoa/set_installation_options.sh shell script]
# insert the # symbol (i.e., unset these environmental keys on `set_installation_options.sh`)
#export IGNORE_EMULTRF_CODE=1 #SaraivanovZhongZhu (SZZ) transformer/CNN-based emulators
#export IGNORE_EMULTRF_DATA=1
#export IGNORE_LIPOP_LIKELIHOOD_CODE=1 # to run EXAMPLE_EMUL_(EVALUATE/MCMC/NAUTILUS/EMCEE1).yaml
#export IGNORE_LIPOP_CMB_DATA=1
#export IGNORE_ACTDR6_CODE=1 # to run EXAMPLE_EMUL_(EVALUATE/MCMC/NAUTILUS/EMCEE1).yaml
#export IGNORE_ACTDR6_DATA=1
#export IGNORE_NAUTILUS_SAMPLER_CODE=1 # to run PROJECTS/EXAMPLE/EXAMPLE_EMUL_NAUTILUS1.py
#export IGNORE_POLYCHORD_SAMPLER_CODE=1 # to run PROJECTS/EXAMPLE/EXAMPLE_EMUL_POLY1.yaml
#export IGNORE_GETDIST_CODE=1 # to run EXAMPLE_TENSION_METRICS.ipynb
#export IGNORE_TENSIOMETER_CODE=1 # to run EXAMPLE_TENSION_METRICS.ipynb
Tip
What if users have not configured ML-related keys before sourcing setup_cocoa.sh?
Answer: Comment the keys below before rerunning setup_cocoa.sh.
[Adapted from Cocoa/set_installation_options.sh shell script]
# These keys are only relevant if you run setup_cocoa multiple times
#export OVERWRITE_EXISTING_ALL_PACKAGES=1
#export OVERWRITE_EXISTING_COSMOLIKE_CODE=1
#export REDOWNLOAD_EXISTING_ALL_DATA=1
Now, users must follow all the steps below.
Step 1️⃣: Activate the private Python environment by sourcing the script start_cocoa.sh
source start_cocoa.sh
Step 2️⃣: Ensure OpenMP is OFF.
export OMP_NUM_THREADS=1
Step 3️⃣ Run cobaya-run on the first emulator example following the commands below.
-
One model evaluation:
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 1 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL_EVALUATE1.yaml -f -
macOS (arm)
mpirun -n 1 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL_EVALUATE1.yaml -f -
MCMC (Metropolis-Hastings Algorithm):
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 4 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL_MCMC1.yaml -r -
macOS (arm)
mpirun -n 4 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL_MCMC1.yaml -r
-
-
Halofit Comparison
The scripts that generated the plots below are provided at
scripts/EXAMPLE_PLOT_COMPARE_CHAINS_EMUL[1-4].py.
For multi-node runs, see MPI across nodes.
Note
Note on core oversubscription: an MPI process that is waiting still burns 100% of its
core, checking for messages in a loop. With more processes than cores, this stalls the
processes doing real work. Open MPI usually detects this and makes waiting processes give
up the CPU, but its detection can be fooled. Adding --mca mpi_yield_when_idle 1 forces
that behavior; it is harmless otherwise.
-
PolyChord:
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 90 --oversubscribe \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x OMP_DYNAMIC \ -x ROOTDIR -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES \ -x CLIK_PLUGIN -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH \ -x CLIK_DATA --mca mpi_yield_when_idle 1 --rank-by slot --map-by slot \ --mca pml ob1 --mca btl vader,tcp,self --bind-to core:overload-allowed \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 --report-bindings \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL_POLY1.yaml -r -
macOS (arm)
mpirun -n 12 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL_POLY1.yaml -r
-
-
Nautilus:
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 90 --oversubscribe \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x OMP_DYNAMIC \ -x ROOTDIR -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES \ -x CLIK_PLUGIN -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH \ -x CLIK_DATA --mca mpi_yield_when_idle 1 --rank-by slot --map-by slot \ --mca pml ob1 --mca btl vader,tcp,self --bind-to core:overload-allowed \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 --report-bindings \ python -m mpi4py.futures ./projects/roman_real/EXAMPLE_EMUL_NAUTILUS1.py \ --root ./projects/roman_real/ \ --outroot "EXAMPLE_EMUL_NAUTILUS1" \ --maxfeval 750000 \ --nlive 2048 \ --neff 15000 \ --flive 0.01 \ --nnetworks 5 -
macOS (arm)
mpirun -n 12 --oversubscribe \ python -m mpi4py.futures ./projects/roman_real/EXAMPLE_EMUL_NAUTILUS1.py \ --root ./projects/roman_real/ \ --outroot "EXAMPLE_EMUL_NAUTILUS1" \ --maxfeval 750000 \ --nlive 2048 \ --neff 15000 \ --flive 0.01 \ --nnetworks 5
-
-
Emcee:
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 51 --oversubscribe \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x OMP_DYNAMIC \ -x ROOTDIR -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES \ -x CLIK_PLUGIN -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH \ -x CLIK_DATA --mca mpi_yield_when_idle 1 --rank-by slot --map-by slot \ --mca pml ob1 --mca btl vader,tcp,self --bind-to core:overload-allowed \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 --report-bindings \ python ./projects/roman_real/EXAMPLE_EMUL_EMCEE1.py \ --root ./projects/roman_real/ \ --outroot "EXAMPLE_EMUL_EMCEE1" \ --maxfeval 1000000 -
macOS (arm)
mpirun -n 12 --oversubscribe \ python ./projects/roman_real/EXAMPLE_EMUL_EMCEE1.py \ --root ./projects/roman_real/ \ --outroot "EXAMPLE_EMUL_EMCEE1" \ --maxfeval 1000000
The number of steps per MPI worker is
$n_{\rm sw} = {\rm maxfeval}/n_{\rm w}$ , with the number of walkers being$n_{\rm w}={\rm max}(3n_{\rm params},n_{\rm MPI})$ . For proper convergence, each walker should traverse 50 times the autocorrelation length ($\tau$ ), which is provided in the header of the output chain file. A reasonable rule of thumb is to assume$\tau > 200$ and therefore set${\rm maxfeval} > 10,000 \times n_{\rm w}$ . Finally, our code sets burn-in (per walker) at$5 \times \tau$ .With these numbers, users may ask when
Emceeis preferable toMetropolis-Hastings? Here are a few numbers based on ourPlanck CMB (l < 396) + SN + BAO + LSST-Y1test case.-
MHachieves convergence with$n_{\rm sw} \sim 150,000$ (number of steps per walker), but only requires four walkers. -
Emceehas$\tau \sim 300$ , so it requires$n_{\rm sw} \sim 15,000$ when running with$n_{\rm w}=114$ .
Conclusion:
Emceerequires$\sim 3$ more evaluations in this case, but the number of evaluations per MPI worker (assuming one MPI worker per walker) is reduced by$\sim 10$ . Therefore,Emceeseems well-suited for chains where the evaluation of a single cosmology is time-consuming (and there is no slow/fast decomposition).What if the user runs an
Emceechain withmaxevalinsufficient for convergence?Emceesaves the chain checkpoint atchains/outroot.h5. -
-
Sampler Comparison
The scripts that generated the plots below are provided at
scripts/EXAMPLE_PLOT_COMPARE_CHAINS_EMUL[1-4].py. -
Global Minimizer:
Our minimizer is a reimplementation of
Procoli, developed by Karwal et al (arXiv:2401.14225)-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 51 --oversubscribe \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x OMP_DYNAMIC \ -x ROOTDIR -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES \ -x CLIK_PLUGIN -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH \ -x CLIK_DATA --mca mpi_yield_when_idle 1 --rank-by slot --map-by slot \ --mca pml ob1 --mca btl vader,tcp,self --bind-to core:overload-allowed \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 --report-bindings \ python ./projects/roman_real/EXAMPLE_EMUL_MINIMIZE1.py \ --root ./projects/roman_real/ \ --outroot "EXAMPLE_EMUL_MIN1" \ --nstw 450 -
macOS (arm)
mpirun -n 12 --oversubscribe \ python ./projects/roman_real/EXAMPLE_EMUL_MINIMIZE1.py \ --root ./projects/roman_real/ \ --outroot "EXAMPLE_EMUL_MIN1" \ --nstw 450
The number of steps per Emcee walker per temperature is
$n_{\rm stw}$ , and the number of walkers is$n_{\rm w}={\rm max}(3n_{\rm params},n_{\rm MPI})$ . The minimum number of total evaluations is$3n_{\rm params} \times n_{\rm T} \times n_{\rm stw}$ , which can be distributed among$n_{\rm MPI} = 3n_{\rm params}$ MPI processes for faster results.The scripts that generated the plots below are provided at
scripts/EXAMPLE_PLOT_MIN_COMPARE_CONV.pyIn our testing,
$n_{\rm stw} \sim 250$ worked reasonably well up to$n_{\rm param} \sim \mathcal{O}(10)$ . -
-
Profile:
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 51 --oversubscribe \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x OMP_DYNAMIC \ -x ROOTDIR -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES \ -x CLIK_PLUGIN -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH \ -x CLIK_DATA --mca mpi_yield_when_idle 1 --rank-by slot --map-by slot \ --mca pml ob1 --mca btl vader,tcp,self --bind-to core:overload-allowed \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 --report-bindings \ python ./projects/roman_real/EXAMPLE_EMUL_PROFILE1.py \ --root ./projects/roman_real/ \ --cov 'chains/EXAMPLE_EMUL_MCMC1.covmat' \ --outroot "EXAMPLE_EMUL_PROFILE1" \ --factor 3 --nstw 450 --numpts 10 \ --profile 1 \ --minfile="./projects/roman_real/chains/EXAMPLE_EMUL_MIN1.txt" -
macOS (arm)
mpirun -n 12 --oversubscribe \ python ./projects/roman_real/EXAMPLE_EMUL_PROFILE1.py \ --root ./projects/roman_real/ \ --cov 'chains/EXAMPLE_EMUL_MCMC1.covmat' \ --outroot "EXAMPLE_EMUL_PROFILE1" \ --factor 3 --nstw 450 --numpts 10 \ --profile 1 \ --minfile="./projects/roman_real/chains/EXAMPLE_EMUL_MIN1.txt"
The argument
factorspecifies the start and end of the parameter being profiled:start value ~ minimum value - factor*np.sqrt(np.diag(cov)) end value ~ minimum value + factor*np.sqrt(np.diag(cov))We advise
${\rm factor} \sim 3$ for parameters that are well constrained by the data when a covariance matrix is provided. Ifcovis not supplied, the code estimates one internally from the prior. If a parameter is poorly constrained orcovis not given, we recommend${\rm factor} \ll 1$ .The script of the plot below is provided at
projects/roman_real/scripts/EXAMPLE_PLOT_PROFILE2.pyProfile 1:
Cosmic Shear only (plus weak Gaussian priors) -
Warning
When running Profiles, you should not set flat priors on parameters that are not well constrained by the data.
By doing that, you then risk having the minimizer select values near the boundary of parameter space. This is a big problem when using emulators, as volume near the
boundary will be inevitable outside the training range. You can convert a flat prior to a Gaussian one by setting the standard deviation to be
prior:
# These priors are meant to prevent the sampler to wander far off training
g1: "lambda As_1e9: stats.norm.logpdf(As_1e9, loc=2.35, scale=1.6)"
g2: "lambda ns: stats.norm.logpdf(ns, loc=0.96, scale=0.05)"
g3: "lambda H0: stats.norm.logpdf(H0, loc=70, scale=10.0)"
g4: "lambda omegab: stats.norm.logpdf(omegab, loc=0.045, scale=0.012)"
g5: "lambda omegam: stats.norm.logpdf(omegam, loc=0.3 , scale=0.25)"
g6: "lambda w0pwa: stats.norm.logpdf(w0pwa, loc=-1.0 , scale=1.44)"
g7: "lambda w: stats.norm.logpdf(w, loc=-1.0, scale=1.44)"
g8: "lambda roman_A1_1: stats.norm.logpdf(roman_A1_1, loc=0, scale=2.5)"
g9: "lambda roman_A1_2: stats.norm.logpdf(roman_A1_2, loc=-1.7, scale=2.5)"
Running Profile also requires emulators trained on larger volumes of the parameter space.
Note
These hybrid examples remain experimental. The checks below verify the workflow; assess emulator accuracy and posterior convergence for your analysis.
The EXAMPLE_EMUL2 examples emulate the background expansion and matter
power spectra. CosmoLike still computes the survey projections, bias and
intrinsic-alignment contributions. Changing n(z) or nuisance parameters does
not require retraining a survey data-vector network.
The shared theory networks live in external_modules/data/emultrf. Install
them through the main Cocoa emulator recipe.
These networks assume mnu = 0.06 eV; do not sample neutrino mass. Their
cold-matter power approximation is not a calibrated massive-neutrino halo
model. Check their training range before widening cosmological priors.
We assume Cocoa and this project are installed, the Cocoa Conda environment
is active, the shell is Bash, and the current folder is cocoa/Cocoa/.
Step 1️⃣: activate Cocoa.
source start_cocoa.shStep 2️⃣: Select the number of OpenMP cores (below, we set it to 4).
-
Linux
export OMP_NUM_THREADS=4; export OMP_PROC_BIND=close; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1 -
macOS (arm)
export OMP_NUM_THREADS=4; export OMP_PROC_BIND=disabled; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
Step 3️⃣: remove GPU access on Linux; these examples use the CPU.
export CUDA_VISIBLE_DEVICES=""Step 4️⃣: evaluate the first hybrid example.
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 1 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL2_EVALUATE1.yaml --force -
macOS (arm)
mpirun -n 1 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL2_EVALUATE1.yaml --force
The YAML selects the CPU for the distance emulator. Keep BLAS at one thread
per MPI rank (OPENBLAS_NUM_THREADS=1, MKL_NUM_THREADS=1); on macOS also
use VECLIB_MAXIMUM_THREADS=1. The Python sampler entry points set these
BLAS limits before importing numerical libraries.
| Example | Configuration 1 | Configuration 2 |
|---|---|---|
| Fixed evaluation | EXAMPLE_EMUL2_EVALUATE1.yaml | EXAMPLE_EMUL2_EVALUATE2.yaml |
| Cobaya MCMC | EXAMPLE_EMUL2_MCMC1.yaml | EXAMPLE_EMUL2_MCMC2.yaml |
| Annealed minimization | EXAMPLE_EMUL2_MINIMIZE1.py | EXAMPLE_EMUL2_MINIMIZE2.py |
| Parameter profile | EXAMPLE_EMUL2_PROFILE1.py | EXAMPLE_EMUL2_PROFILE2.py |
| Nautilus sampling | EXAMPLE_EMUL2_NAUTILUS1.py | EXAMPLE_EMUL2_NAUTILUS2.py |
Configuration 1 uses roman_real.cosmic_shear, NLA, and example1.dataset.
Configuration 2 uses roman_real.combo_3x2pt, NLA, and example1.dataset.
The minimization, profile and Nautilus scripts read the corresponding
EXAMPLE_EMUL2_EVALUATE1.yaml or 2.yaml; --input selects another evaluate
YAML. They require cocoa_hybrid_sampling.py from the matching shared core
revision. They do not maintain separate embedded cosmologies. --check evaluates
the specified fiducial and prints the sampled parameter order without sampling.
Use a new --outroot for each run; these scripts refuse to overwrite results.
With Cocoa activated in Bash from cocoa/Cocoa/, run the first MCMC
example. Use configuration
2 for the second likelihood listed above. Check chain convergence before
interpreting posterior constraints.
Step 1️⃣: Select the number of OpenMP cores (below, we set it to 8).
-
Linux
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1 -
macOS (arm)
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
Step 2️⃣: remove GPU access on Linux; these examples use the CPU.
export CUDA_VISIBLE_DEVICES=""Step 3️⃣: start Cobaya's hybrid MCMC.
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 4 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL2_MCMC1.yaml -
macOS (arm)
mpirun -n 4 --oversubscribe \ cobaya-run ./projects/roman_real/EXAMPLE_EMUL2_MCMC1.yaml
We assume Cocoa and this project are installed, the Cocoa Conda environment
is active, the shell is Bash, and the current folder is cocoa/Cocoa/.
Step 1️⃣: Select the number of OpenMP cores (below, we set it to 8).
-
Linux
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1 -
macOS (arm)
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
Step 2️⃣: remove GPU access on Linux; these examples use the CPU.
export CUDA_VISIBLE_DEVICES=""Step 3️⃣: check the hybrid setup before a long run.
python ./projects/roman_real/EXAMPLE_EMUL2_MINIMIZE1.py --checkStep 4️⃣: search for a minimum.
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 21 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x ROOTDIR \ -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \ -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH -x CLIK_DATA \ -x CLIK_PLUGIN -x CUDA_VISIBLE_DEVICES --mca mpi_yield_when_idle 1 \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ python ./projects/roman_real/EXAMPLE_EMUL2_MINIMIZE1.py --nstw 200 --outroot hybrid_min1 -
macOS (arm)
mpirun -n 12 --oversubscribe \ python ./projects/roman_real/EXAMPLE_EMUL2_MINIMIZE1.py --nstw 200 --outroot hybrid_min1
Step 5️⃣: profile the first sampled parameter using that saved minimum.
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 21 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x ROOTDIR \ -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \ -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH -x CLIK_DATA \ -x CLIK_PLUGIN -x CUDA_VISIBLE_DEVICES --mca mpi_yield_when_idle 1 \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ python ./projects/roman_real/EXAMPLE_EMUL2_PROFILE1.py --profile 0 --nstw 200 --numpts 11 --factor 1 --minfile ./projects/roman_real/chains/hybrid_min1.json --outroot hybrid_profile1 -
macOS (arm)
mpirun -n 12 --oversubscribe \ python ./projects/roman_real/EXAMPLE_EMUL2_PROFILE1.py --profile 0 --nstw 200 --numpts 11 --factor 1 --minfile ./projects/roman_real/chains/hybrid_min1.json --outroot hybrid_profile1
Step 6️⃣: run Nautilus as an independent sampling example.
-
Linux
"${CONDA_PREFIX}"/bin/mpirun -n 90 --oversubscribe \ --mca pml ob1 --mca btl vader,tcp,self \ -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x ROOTDIR \ -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \ -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH -x CLIK_DATA \ -x CLIK_PLUGIN -x CUDA_VISIBLE_DEVICES --mca mpi_yield_when_idle 1 \ --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 \ --bind-to core:overload-allowed --report-bindings \ --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \ python ./projects/roman_real/EXAMPLE_EMUL2_NAUTILUS1.py --nlive 1000 --neff 10000 --maxfeval 100000 --outroot hybrid_nautilus1 -
macOS (arm)
mpirun -n 12 --oversubscribe \ python ./projects/roman_real/EXAMPLE_EMUL2_NAUTILUS1.py --nlive 1000 --neff 10000 --maxfeval 100000 --outroot hybrid_nautilus1
The annealed Emcee search follows the DES × Planck template. Its objective
is −2 log posterior, including nuisance and cosmological priors; the
profile is therefore a penalized profile, not a pure likelihood profile.
--nstw sets steps per walker per temperature. More steps and independent
starts are needed to assess whether a minimum is reliable.
--profile accepts a sampled-parameter name or its printed zero-based index.
--factor gives the half-width in proposal standard deviations, clipped to
the prior bounds. --cov accepts a covariance whose header lists the sampled
parameters in order; without it, the prior covariance sets the proposal.
The minimum JSON must come from the same evaluate YAML and parameter order.
Older plain-text minimum files are not accepted. Set any additional priors
in the input YAML; these scripts do not insert hidden cosmological priors.
Nautilus writes weighted GetDist-compatible rows and a JSON convergence
record. Reaching --maxfeval is not convergence. If the budget ends before
any posterior samples are retained, only the checkpoint and a JSON record
with converged: false are saved. Its prior transform uses
Cobaya's one-dimensional prior distributions; external prior factors enter
once as additional log weight. Evidence with unnormalized external priors
has that normalization limitation. These examples do not certify emulator
accuracy or posterior convergence.
Details on the matter power spectrum emulator designs will be presented in the emulator_code repository.
Basically, we apply standard neural network techniques to generalize the syren-new Eq. 6 of arXiv:2410.14623 formula for the linear power spectrum (w0waCDM with a fixed neutrino mass of
Note
Users can decide not to correct the syren-new formula for the linear power spectrum (flag in the yaml). Although we have not conducted extensive studies of the caveats of the syren-new approximation, it appears sufficient for w0waCDM forecasts when combined with the Euclid Emulator to compute the nonlinear boost.
For back-of-the-envelope LCDM calculations (e.g., to test cosmolike features), users can also choose not to correct the syren-Halofit formula for the LCDM nonlinear boost (see figure below). In this case, the overhead on top of cosmolike computations is minimum, at the order of
-
Emulator Comparison
The scripts that generated the plots below are provided at
scripts/EXAMPLE_PLOT_COMPARE_CHAINS_EMUL[1-4].py.
Note
Running on more than one node.
- Network interface. TCP must use an interface routable between compute
nodes. A common exclusion list is
--mca btl_tcp_if_exclude lo,docker0,virbr0,ib0; adapt it to the cluster. Keepib0if routable IP-over-InfiniBand is the intended network. These examples exchange parameter vectors and scalar scores, so communication volume is small; actual scaling still depends on the machine. - Environment. Remote ranks need the same Cocoa paths and libraries:
ROOTDIR,PATH,LD_LIBRARY_PATH,PYTHONPATH,CONDA_PREFIX, OpenMP and BLAS settings, andCLIK_PATH/CLIK_DATA/CLIK_PLUGINwhen used. Slurm normally exports the submitting environment (--export=ALL). Explicit-xoptions also forward these variables with SSH launchers. Activate Cocoa before launching; build/download flags do not replace runtime paths. All nodes must see the same files at the same paths. - Slurm geometry. Keep
ntasks-per-node × cpus-per-taskwithin the allocated physical cores per node. SetOMP_NUM_THREADStoSLURM_CPUS_PER_TASKand use--map-by numa:pe=${OMP_NUM_THREADS}. The minimization, profile and Nautilus pool reserves one MPI rank as coordinator; the remaining ranks evaluate the model.
Within a Slurm allocation, first activate Cocoa in Bash on the launch node. The following steps assume Open MPI 4 and shared installation/data paths. Omit optional CLIK exports if those variables are not set.
Step 1️⃣: match OpenMP threads to the scheduler allocation.
export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}"; export OMP_PROC_BIND=close; \
export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
Step 2️⃣: launch the hybrid minimizer across the allocated ranks.
"${CONDA_PREFIX}"/bin/mpirun -n "${SLURM_NTASKS}" \
--mca pml ob1 --mca btl vader,tcp,self \
--mca btl_tcp_if_exclude lo,docker0,virbr0 \
--map-by numa:pe=${OMP_NUM_THREADS} --bind-to core --report-bindings \
-x ROOTDIR -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX \
-x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \
-x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CUDA_VISIBLE_DEVICES \
python ./projects/roman_real/EXAMPLE_EMUL2_MINIMIZE1.py --nstw 200 --outroot hybrid_multinodeFor a Planck likelihood add -x CLIK_PATH -x CLIK_DATA -x CLIK_PLUGIN when
those variables are defined. Follow the cluster's MPI module and Slurm
launch policy; do not oversubscribe a production allocation. Outside Slurm,
supply the hosts and slots with the cluster's --hostfile or --host recipe.
Emulators for Roman are stored on emulators/, we usually provide the YAML files used to train them. For example, datavectors required to train an emulator
with settings similar to
nla_cosmic_shear/w0wa_takahashi_cs_cnn.h5
nla_cosmic_shear/w0wa_hmcode2020_cs_cnn.h5
can be found at
nla_cosmic_shear/w0wa_takahashi_cs_cnn.yaml
The repository emulators_code provides the script dataset_generator_lensing.py that can generate data vectors for any cosmological project.
The script below computes data vectors for cosmic shear (NLA,
"${CONDA_PREFIX}"/bin/mpirun -n 10 --oversubscribe \
-x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x OMP_DYNAMIC \
-x ROOTDIR -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES \
-x CLIK_PLUGIN -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH \
-x CLIK_DATA --mca mpi_yield_when_idle 1 --rank-by slot --map-by slot \
--mca pml ob1 --mca btl vader,tcp,self --bind-to core:overload-allowed \
--mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 --report-bindings \
python external_modules/code/emulators/emultrf/emultraining/dataset_generator_lensing.py \
--root projects/roman_real/ \
--fileroot emulators/nla_cosmic_shear/ \
--nparams 10000 \
--yaml 'w0wa_takahashi_cs_cnn.yaml' \
--datavsfile 'w0wa_takahashi_dvs_train' \
--paramfile 'w0wa_takahashi_params_train' \
--failfile 'w0wa_takahashi_params_failed_train' \
--chain 0 \
--unif 0 \
--temp 64 \
--maxcorr 0.15 \
--freqchk 2000 \
--loadchk 0 \
--append 1
-
The requested number of data vectors is given by the
--nparamsflag. -
There are two possible samplings.
- The option
--unif 1sets the sampling to follow a uniform distribution (respecting parameter boundaries set in the YAML file) - The option
--unif 0sets the sampling to follow a Gaussian distribution with the following options- The covariance matrix is set in the YAML file (keyword
params_covmat_fileinside thetrain_argsblock). For example, our provided YAML selects the Fisher-based w0wa_fisher_covmat.txt covariance matrix - Temperature reduces the curvature of the likelihood (
cov = cov/T) and is set by--tempflag - The correlations of the original covariance matrix are reduced to be less than
--maxcorr.
- The covariance matrix is set in the YAML file (keyword
- The option
-
For visualization purposes, setting
--chain 1sets the script to generate the training parameters without computing the data vectors. -
The output files are
# Distribution of training points ready to be plotted by GetDist w0wa_params_train_cs_64.1.txt w0wa_params_train_cs_64.covmat w0wa_params_train_cs_64.paramnames w0wa_params_train_cs_64.ranges #Corresponding data vectors w0wa_takahashi_nobaryon_dvs_train_cs_64.npy # Training parameters in which the data vector computation failed w0wa_params_failed_train_cs_64.txt -
The flags
--freqchk,--loadchk, and--appendare related to checkpoints.- The option
--freqchksets the frequency at which the code saves checkpoints (chk). - The options
--loadchkand--appendspecify whether the code loads the parameters and data vectors from a chk. In the two cases below, the code determines which remaining data vectors to compute based on the flags saved in the--failfilefile.- Case 1 (
--loadchk 1and--append 1): the code loads params from the chk and appends~nparamsmodels to it. - Case 2 (
--loadchk 1and--append 0): the code loads the params.
- Case 1 (
- The option
The tests/ folder compares likelihood predictions against stored reference
values, checks for race conditions from OpenMP threading, and measures the
numerical error of the default accuracy settings.
tests/README.md describes the checks, how to run them,
the input data snapshots, and how to refresh them.
Armadillo was chosen to make a convenient Python API for notebook exploration. This C++ library provides vectors, matrices and three-dimensional arrays called cubes. A small interface layer connects them to NumPy through pybind11, with CARMA handling array conversion. The notebooks expose intermediate quantities; production calculations use the CLI interfaces.
We assume Cocoa and this project are installed, the Cocoa Conda environment
is active, the shell is Bash, and the current folder is cocoa/Cocoa/.
Compile the project first; the covariance notebook also needs the optional covariance build described below.
Step 1️⃣: activate Cocoa.
source start_cocoa.shStep 2️⃣: Select the number of OpenMP cores (below, we set it to 8).
-
Linux
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1 -
macOS (arm)
export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \ export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \ export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
Step 3️⃣: start Jupyter.
jupyter notebook --no-browser --port=8888Step 4️⃣: open the printed URL and choose a notebook below.
Step 5️⃣: select Kernel → Restart Kernel and Run All Cells.
| Notebook | Contents |
|---|---|
| EXAMPLE_EVALUATE1.ipynb | Cosmic shear of EXAMPLE_EVALUATE1.yaml: how |
| EXAMPLE_EVALUATE2.ipynb | Galaxy–galaxy lensing and clustering of the 3x2pt example EXAMPLE_EVALUATE2.yaml: |
| EXAMPLE_EVALUATE3.ipynb | Baryonic feedback from the bfmt theory block: six methods applied to the 3x2pt data vector, the chains/EXAMPLE_EVALUATE3.datavectors, and one-parameter sweeps of each method's suppression. |
| EXAMPLE_EVALUATE_COVARIANCE.ipynb | G, SSC, cNG, total, separate 1h–4h matter trispectra and matrix diagnostics. |
Note
EXAMPLE_EVALUATE3.ipynb needs the bfmt packages of the
baryonic feedback section. Its parameter
sweeps run CAMB about 130 times and take tens of minutes.
The three data-vector notebooks call the wrappers in
interface/cosmolike_roman_real_notebook_wrappers.py; the covariance
notebook calls the survey adapter covariance/roman_real_covariance.py.
Read the notebooks in this order:
flowchart TD
A["EXAMPLE_EVALUATE1: cosmic shear"] --> B["EXAMPLE_EVALUATE2: 3x2pt"]
B --> C["EXAMPLE_EVALUATE3: bfmt feedback"]
B --> D["EXAMPLE_EVALUATE_COVARIANCE: G, SSC, cNG"]
W["cosmolike_roman_real_notebook_wrappers.py"] --> A
W --> B
W --> C
S["roman_real_covariance.py"] --> D
Choose the Python kernel from the activated Cocoa environment and restart it after recompiling. The covariance guide explains the forecast files, figures and refinement workflow.
The covariance tools compute Gaussian (G), super-sample (SSC), connected non-Gaussian (cNG) and total matrices before scale cuts. covariance/README.md explains compilation, the production CLI, exploratory notebooks, accuracy settings and output files.
Data-vector options belong to the selected likelihood block. Covariance
options belong to the evaluate YAML's covariance block. They use separate
names and settings; changing one does not refine the other.
| Data-vector setting | What it changes |
|---|---|
accuracyboost |
Overall interpolation-table resolution. |
integration_accuracy |
Quadrature resolution; levels 5/6/7 allow 2048/4096/8192 points per Limber panel. Refine independently of interpolation. |
internal_accuracyboost |
C-FAST-PT convolution grid. |
nonlimber_accuracyboost |
Non-Limber distance sampling. |
pk_z_refinement |
Nested redshift refinement of matter-power inputs. |
lmax |
Real-space angular-transform cutoff, where a real-space transform is used. |
kmax_boltzmann |
Requested Boltzmann power range; coordinate it with the theory settings. |
adopt_limber_gg and adopt_limber_gs choose a projection approximation.
photoz_interpolation_type chooses how n(z) is interpolated, while
photoz_zmid_convention describes the input coordinates. These are modeling
or input-convention choices, not interchangeable accuracy boosts.
photoz_histogram_akima: 0 retains the original midpoint n(z) reconstruction; 1 selects histogram preparation followed by Akima for source and lens samples. photoz_histogram_akima_factor chooses 3 (default), 5 or 7 preparation cells per original uniform bin. See the reconstruction comparisons and area plots. The selected first factor never changes with an accuracy boost.
The default data-vector power grid has 1,500 wavenumbers at boost 1.
Covariance alone uses power_accuracyboost: 8 to prepare 11,993 nodes by
natural cubic interpolation before C linear lookup. Its accuracy_boost
refines tables and cutoffs; its integration_accuracy independently selects
quadrature levels 0–4. See the complete covariance accuracy table.
CAMB's theory.camb.extra_args.AccuracyBoost controls CAMB, not CosmoLike.
Check interpolation, quadrature, input-power sampling and transform cutoffs
separately at fixed cosmology and measurement bins. Narrow n(z) overlaps
particularly require a quadrature check; increasing accuracyboost alone
is not that check. The data-vector test guide
and covariance test guide state what each set of
tests actually verifies. A passing regression or a larger boost is not a general
claim of survey or Fisher convergence.




