Skip to content
nikolay327Public

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

SCARF-ML

Research code for representation learning and latent-space density-ratio estimation on high-purity germanium detector waveforms.

Research-code status

This repository is an archive of machine-learning methods developed during and after a Master’s research project for the LEGEND experiment. It contains the main thesis models alongside later architectural experiments, distributed-training implementations, abandoned approaches, and analysis utilities.

The repository should not currently be treated as a standalone reproducible package. Experiment data, model configurations, metadata, trained checkpoints, scheduler scripts, and the original HPC environment are not included.

Overview

SCARF-ML investigates machine-learning methods for extracting useful representations from high-purity germanium detector waveform time series and applying those representations to detector-event discrimination.

The main workflow developed for the Master’s project consists of two stages:

  1. Self-supervised representation learning: a convolutional adversarial autoencoder learns a compact representation of detector waveforms.
  2. Latent-space density-ratio estimation: a downstream classifier compares encoded detector events with samples drawn from the autoencoder’s Gaussian reference distribution.

A later branch of the project replaces the convolutional autoencoder with a larger Transformer-based adversarial autoencoder and scales its training synchronously across 32 GPUs with PyTorch Distributed Data Parallel.

The remaining modules document experiments with alternative Transformer autoencoders, autoregressive models, normalizing flows, MCMC methods, calibration procedures, and statistical-inference utilities.

Main research workflow

Unlabelled HPGe waveforms
          |
          v
Convolutional adversarial autoencoder
          |
          v
Gaussian-regularised latent representation
          |
          v
Latent-space density-ratio estimator
          |
          v
Detector-event discrimination scores

The autoencoder and density-ratio estimator were trained separately. The downstream estimator loads a pretrained autoencoder checkpoint and uses the encoder as a fixed representation model.

1. Convolutional adversarial autoencoder

The convolutional adversarial autoencoder implemented in models/AdversarialAutoencoder.py is the representation model used for the Master’s thesis.

Architecture

The model contains:

  • a one-dimensional convolutional encoder
  • residual convolutional blocks
  • progressive temporal downsampling
  • a compact latent representation
  • a transposed-convolutional decoder
  • a sigmoid output layer for waveform reconstruction

The encoder maps a waveform (x) to a latent representation (z):

waveform x
    |
    v
1D convolutional encoder
    |
    v
latent representation z

The decoder maps the latent representation back to a reconstructed waveform:

latent representation z
    |
    v
1D transposed-convolutional decoder
    |
    v
reconstructed waveform x_rec

The autoencoder itself is paired with the separate latent discriminator implemented in models/discriminator.py.

Denoising objective

During training, the model receives a corrupted waveform and attempts to reconstruct the corresponding clean waveform:

clean waveform x ----------------------+
                                       |
                                       v
corruption or noise             reconstruction loss
       |                               ^
       v                               |
corrupted waveform x_corr -> encoder -> z -> decoder -> x_rec

This allows the encoder to learn waveform features without requiring class labels for every training sample.

The convolutional training implementation uses an L1 reconstruction loss:

L_rec = |x_rec - x|

Adversarial latent regularisation

The latent discriminator distinguishes between:

  • samples produced by the waveform encoder and
  • samples drawn from a standard Gaussian distribution
encoded waveform z --------> latent discriminator
                                  ^
                                  |
standard-normal sample -----> latent discriminator

The discriminator is trained to classify the two sources correctly. The autoencoder is simultaneously trained to make its encoded latent representations appear Gaussian to the discriminator.

The complete autoencoder objective combines reconstruction and adversarial terms:

L_AE = lambda * L_reconstruction + L_adversarial

This regularisation provides a simple reference distribution in latent space, which is subsequently used by the density-ratio estimator.

Training implementation

The primary convolutional training path is contained in:

src/scarf_ml/trainer/cnn_trainer.py

It includes:

  • PyTorch Distributed Data Parallel
  • NCCL-based multi-GPU synchronisation
  • bfloat16 autocast
  • gradient scaling
  • separate autoencoder and discriminator optimisers
  • rank-aware data sharding
  • job-local shared-memory data staging
  • epoch-level checkpointing
  • training continuation from saved optimiser and scaler states

This is research training code and contains experiment-specific assumptions, scaling constants, checkpoint compatibility paths, and cluster-environment dependencies.

2. Latent-space density-ratio estimation

The downstream method used for the thesis is implemented through:

src/scarf_ml/models/DREstimator.py
src/scarf_ml/trainer/DRE_trainer.py
src/scarf_ml/evaluate/DRE_tester.py
src/scarf_ml/evaluate/dre_evaluator.py

Motivation

Adversarial training encourages the aggregated latent distribution of the autoencoder to resemble a standard Gaussian distribution. Individual detector-event populations can nevertheless occupy different regions of that latent space.

The downstream estimator compares:

  • encoded detector-event samples and
  • proposal samples drawn from the Gaussian reference distribution
detector waveform
       |
       v
frozen autoencoder encoder
       |
       v
encoded data sample z_data -----+
                                 |
                                 v
                          DRE classifier
                                 ^
                                 |
standard-normal sample z_ref ----+

Training procedure

For every training batch, the pipeline:

  1. Passes detector waveforms through the pretrained autoencoder.
  2. Extracts their latent representations.
  3. Optionally reconstructs and re-encodes each waveform to obtain a denoised latent representation.
  4. Draws proposal samples from a standard Gaussian distribution.
  5. Concatenates the proposal and encoded samples.
  6. Trains a classifier using cross-entropy loss.

The Gaussian proposal distribution acts as the reference class. Encoded detector populations form the remaining classes.

Under the classifier-based density-ratio-estimation interpretation, differences between the classifier logits provide scores related to the density ratios between encoded detector populations and the Gaussian reference distribution.

DRE model

DREstimator.py implements a residual multilayer perceptron consisting of:

  • a projection from the autoencoder latent dimension
  • multiple fully connected residual blocks
  • batch normalisation
  • dropout
  • a final multiclass logit projection

A temperature-scaled variant is also included for post-hoc calibration experiments.

Validation and diagnostics

The training and testing utilities evaluate:

  • classification accuracy
  • validation cross-entropy
  • top-class expected calibration error
  • per-class expected calibration error
  • global expected calibration error
  • confidence-bin statistics
  • effective sample sizes calculated from estimated density-ratio weights
  • uncertainty intervals for calibration curves

These diagnostics were used to study whether the classifier outputs could be interpreted reliably as density-ratio scores rather than only as class predictions.

Experiment-data evaluation

evaluate/dre_evaluator.py applies trained models to experiment-tier LH5 files.

For each event, the evaluator can write:

  • the latent representation
  • DRE logits
  • first-pass reconstruction error
  • second-pass reconstruction error
  • waveform-alignment information
  • timestamp
  • event identifier

The outputs are written to LH5 tables for use in the subsequent detector-data analysis.

3. Transformer-based DAAE experiment

models/daae.py and trainer/daae.py contain a later adversarial-autoencoder implementation developed after the convolutional thesis model.

This model was trained synchronously across 32 GPUs using PyTorch Distributed Data Parallel.

It is a separate research branch and was not the convolutional model used for the original Master’s-thesis downstream analysis.

Encoder

The DAAE encoder combines:

  • convolutional waveform downsampling
  • residual convolutional blocks
  • feature-wise linear modulation for positional information
  • Transformer self-attention blocks
  • optional FlashAttention
  • mean pooling over the encoded sequence
  • projection into a compact latent vector
waveform
   |
   v
convolutional downsampling
   |
   v
position-modulated tokens
   |
   v
Transformer encoder
   |
   v
pooled latent representation

Decoder

The decoder:

  • repeats the latent representation across the requested output sequence
  • adds positional information
  • processes the sequence with Transformer blocks
  • uses rotary positional embeddings
  • projects each token to a reconstructed waveform sample

A separate multilayer discriminator regularises the latent representation toward a Gaussian reference distribution.

Distributed training

The distributed trainer includes:

  • NCCL process-group initialisation
  • synchronous PyTorch DDP
  • synchronised batch normalisation
  • torch.compile
  • bfloat16 model execution
  • mixed-precision gradient scaling
  • fused LAMB optimisation
  • job-local shared-memory data staging
  • rank- and worker-aware data partitioning
  • distributed aggregation of training metrics
  • checkpoint recovery for the model, discriminator, optimisers, and gradient scalers

The code reflects the original HPC environment and contains hard-coded cluster paths and assumptions that must be adapted before reuse elsewhere.

Data processing

The src/scarf_ml/data package contains the data-curation, batching, splitting, and evaluation-time preprocessing code used by the training workflows.

The pipeline was developed around LEGEND LH5 detector data and experiment-specific metadata.

LH5-to-memmap curation

data/dataloader.py contains the main dataset-curation implementation.

The MMAPBuilder:

  • discovers waveform and physical-parameter data through the experiment file database
  • reads selected detector channels and LH5 groups
  • applies configurable physical event cuts before loading waveform samples
  • retrieves detector- and run-dependent pole-zero correction parameters
  • performs baseline subtraction
  • applies pole-zero or double-pole-zero correction
  • normalises waveforms by reconstructed energy
  • determines the waveform position corresponding to 50% amplitude
  • extracts fixed-length aligned waveform windows
  • renormalises waveforms with large overshoots
  • appends reconstructed energy, baseline, and target information
  • writes the resulting samples to NumPy memory-mapped files

Files are counted and processed in parallel using ProcessPoolExecutor. The output arrays are intended to provide efficient sequential or randomly indexed access during large-scale training.

Each curated waveform sample has the form:

[waveform samples, reconstructed energy, baseline mean, target]

The exact waveform length, event selection, detector channels, and target definition are controlled by the experiment configuration.

Distributed pretraining data

MemmapBatchDataset is an iterable dataset for reading fixed-size batches directly from a NumPy memory map.

It supports:

  • deterministic epoch-dependent shuffling
  • reuse of a global permutation over configurable training cycles
  • reconstruction of the appropriate permutation when training resumes from a checkpoint
  • sharding across PyTorch Distributed Data Parallel ranks
  • further sharding across DataLoader workers
  • removal of incomplete batches so that every rank performs the same number of training steps
  • optional filtering for latent-space datasets

The shuffled indices are stored in Python shared memory so that DataLoader workers can access the same global ordering without independently duplicating it.

DynamicCollateFn performs the final runtime preprocessing before GPU transfer. It separates the waveform and metadata fields, optionally adds Gaussian waveform corruption, and uses rank-, worker-, and epoch-dependent random seeds.

This is the data path used by the adversarial-autoencoder training workflows.

DRE training data

The downstream density-ratio-estimation workflow uses DRETrainValTestSplitter and DREDataset.

DRETrainValTestSplitter:

  • reads the configured labelled memory maps
  • performs deterministic dataset mixing using a fixed random seed
  • constructs reproducible training, validation, and test partitions
  • calculates class-specific batch sizes from the configured sampling priors
  • loads the selected partition into host memory

DREDataset constructs stratified batches containing the configured number of samples from each detector-event population. The remaining positions in each batch are reserved for samples from the Gaussian proposal distribution, which are generated by the DRE trainer.

DRECollateFn separates waveform values, labels, reconstructed energy, baseline information, and auxiliary targets before the batch is passed to the model.

Experiment-data evaluation

DREEvaluatorDataset and DREEvaluatorCollateFn provide the evaluation-time data path for applying a trained autoencoder and DRE model to LH5 experiment files.

The evaluator:

  • streams each LH5 file in fixed-size batches
  • reads waveform, energy, timestamp, and event-number columns
  • obtains the appropriate pole-zero parameters from experiment metadata
  • performs baseline subtraction and pole-zero correction
  • normalises each waveform by its reconstructed energy
  • aligns the waveform using its 50% amplitude position
  • extracts the configured fixed-length input window
  • returns the original event metadata together with the processed waveform

The downstream evaluator then writes latent representations, model logits, reconstruction diagnostics, timestamps, and event identifiers to new LH5 analysis tables.

Additional utilities

The data package also contains:

  • StreamingMemmapDataset for sequential memory-map evaluation
  • IndexSampler for random sampling without replacement
  • generic dataset-splitting utilities
  • class-specific DataLoader construction
  • MultiClassLoader for combining batches from multiple class-specific loaders

Some of these utilities belong to earlier experimental workflows and are retained for compatibility with the research archive.

Exploratory and legacy code

A large part of the repository records model variants and approaches investigated during development.

These include:

  • alternative Transformer autoencoders
  • rotary-embedding and causal-decoder experiments
  • autoregressive teacher models
  • autoregressive normalizing flows
  • normalizing-flow density-ratio estimators
  • direct neural-ratio-estimation variants
  • latent-space flow models
  • PCA-based baselines
  • Langevin and other MCMC experiments
  • learning-rate range searches
  • calibration experiments
  • event-ranking utilities

These modules are retained as research history. They should not be interpreted as equally mature, mutually compatible, or part of one final pipeline.

Some contain incomplete implementations, superseded interfaces, hard-coded parameters, or dependencies on files that were not included in the public repository.

Repository structure

.
├── README.md
├── pyproject.toml
└── src/
    └── scarf_ml/
        ├── data/
        │   ├── dataloader.py              # LH5 curation and memmap construction
        │   ├── dataset.py                 # Distributed, DRE, and evaluation datasets
        │   ├── collate_fn.py              # Runtime waveform preprocessing
        │   ├── datasampler.py             # Legacy random index sampler
        │   └── __init__.py
        │
        ├── models/
        │   ├── AdversarialAutoencoder.py  # CNN thesis representation model
        │   ├── discriminator.py           # CNN latent discriminator
        │   ├── DREstimator.py             # Main downstream DRE model
        │   ├── daae.py                    # Transformer-based DAAE
        │   ├── DREFlow.py                 # Experimental flow-based DRE
        │   ├── flows.py                   # Normalizing-flow components
        │   └── ...                        # Additional model experiments
        │
        ├── trainer/
        │   ├── cnn_trainer.py             # CNN adversarial-autoencoder training
        │   ├── DRE_trainer.py             # Main downstream DRE training
        │   ├── daae.py                    # Distributed Transformer DAAE training
        │   └── ...                        # Experimental training paths
        │
        ├── evaluate/
        │   ├── DRE_tester.py              # DRE testing and calibration metrics
        │   ├── dre_evaluator.py            # Inference on experiment LH5 data
        │   ├── daae.py                    # DAAE reconstruction evaluation
        │   └── ...                        # Additional evaluation experiments
        │
        ├── lr_search/                      # Learning-rate experiments
        ├── mcmc/                           # Latent-space MCMC experiments
        ├── LBI/                            # Likelihood-based inference utilities
        ├── logging.py
        └── utils.py

Installation

The package metadata specifies Python 3.9 or newer.

git clone https://github.com/nikolay327/scarf_ml.git
cd scarf_ml

python -m venv .venv
source .venv/bin/activate

pip install -e .

The research environment used packages including:

  • PyTorch with CUDA and NCCL support
  • NumPy
  • SciPy
  • pandas
  • lgdo
  • dspeed
  • FlashAttention
  • NVIDIA Apex
  • einops
  • iminuit
  • optionally Weights & Biases

FlashAttention and Apex must be compiled against compatible PyTorch and CUDA versions.

Historical entry points

The commands below describe the original script interfaces. They are provided as documentation and are not expected to run without the experiment data, configurations, metadata, trained checkpoints, and original cluster environment.

Train the convolutional thesis autoencoder

torchrun \
    --standalone \
    --nproc_per_node=<number_of_gpus> \
    --module scarf_ml.trainer.cnn_trainer \
    <starting_epoch> \
    <final_epoch> \
    <model_name> \
    <working_dir> \
    <mmap_dir> \
    <training_config.json> \
    <num_dataloader_workers>

Train the Transformer DAAE

torchrun \
    --standalone \
    --nproc_per_node=<number_of_gpus> \
    --module scarf_ml.trainer.daae \
    <starting_epoch> \
    <final_epoch> \
    <model_name> \
    <working_dir> \
    <mmap_dir> \
    <training_config.json> \
    <num_dataloader_workers>

For multi-node execution, the standalone launch must be replaced by the rendezvous and rank configuration supplied by the scheduler.

Train the downstream DRE

torchrun \
    --standalone \
    --nproc_per_node=<number_of_gpus> \
    --module scarf_ml.trainer.DRE_trainer \
    <starting_epoch> \
    <final_epoch> \
    <model_name> \
    <working_dir> \
    <mmap_dir> \
    <training_config.json>

Evaluate DRE test data

torchrun \
    --standalone \
    --nproc_per_node=<number_of_gpus> \
    --module scarf_ml.evaluate.DRE_tester \
    <checkpoint_id> \
    <model_name> \
    <working_dir> \
    <mmap_dir> \
    <training_config.json>

Apply the DRE to LH5 files

python -m scarf_ml.evaluate.dre_evaluator \
    <detector_channel> \
    <input_tier> \
    <model_name> \
    <checkpoint_id> \
    <working_dir> \
    <training_config.json> \
    <batch_size>

Required external inputs

The public repository does not include:

  • LEGEND waveform data
  • LH5 detector files
  • experiment file-database configurations
  • excluded-timestamp definitions
  • pole-zero correction parameters
  • calibration-to-physics mappings
  • training JSON files
  • evaluation configurations
  • NumPy memory maps
  • trained model checkpoints
  • cluster job scripts

Consequently, the repository currently documents the implementation and research history rather than providing a complete reproduction package.

Outputs

Depending on the selected workflow, the scripts produce:

  • PyTorch model checkpoints
  • discriminator checkpoints
  • optimiser states
  • gradient-scaler states
  • reconstruction-loss histories
  • adversarial-loss histories
  • DRE training and validation losses
  • classification and calibration metrics
  • effective-sample-size diagnostics
  • latent representations
  • reconstruction errors
  • detector-event scores
  • LH5 analysis tables

Model checkpoints, generated data products, images, notebooks, training configurations, and logs are excluded from version control.

Reproducibility

The code contains explicit seeds and deterministic splitting logic for several data-loading and training operations. However, complete reproduction additionally requires:

  • the original waveform datasets
  • the exact experiment metadata
  • the original JSON configurations
  • compatible dependency versions
  • the model checkpoints
  • the original HPC launch configuration
  • the experiment-specific file-database and preprocessing metadata

The code should therefore be treated as a record of the methodology rather than a fully reproducible release.

Intended use

This repository is primarily intended to:

  • document the machine-learning implementation used during the research project
  • preserve the convolutional adversarial autoencoder used for the Master’s thesis
  • preserve the downstream latent-space density-ratio estimator
  • document the later 32-GPU Transformer DAAE experiment
  • provide a technical reference for related scientific machine-learning work
  • retain exploratory approaches that informed the final methodology

It is not currently intended to provide a plug-and-play waveform-analysis package.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages