Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ jobs:
-w /workspace \
"${COVERAGE_IMAGE}" \
bash -lc '
python -m pip install --root-user-action=ignore -e . pytest pytest-cov coverage[toml]
python -m pip install --root-user-action=ignore -e ".[pimm]" pytest pytest-cov coverage[toml]
mkdir -p /tmp/spine-sitecustomize
printf "import h5py\n" > /tmp/spine-sitecustomize/sitecustomize.py
NUMBA_DISABLE_JIT=1 PYTHONPATH="/tmp/spine-sitecustomize:${PYTHONPATH:-}" pytest test \
Expand Down
6 changes: 3 additions & 3 deletions docker/spine/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -184,9 +184,9 @@ RUN wget -O "${ICARUS_PHOTONLIB_PATH}" "${ICARUS_PHOTONLIB_URL}" && \
COPY pyproject.toml MANIFEST.in README.md ./
COPY src/ ./src/

# Install SPINE with the optional visualization stack. The compiled model
# dependencies are installed explicitly above to keep their versions aligned.
RUN pip install --no-cache-dir --no-build-isolation .[viz]
# Install SPINE with PIMM readers and the optional visualization stack.
# Compiled model dependencies are installed above to keep versions aligned.
RUN pip install --no-cache-dir --no-build-isolation ".[viz,pimm]"

# Verify that the default container environment and the explicit fallback setup
# recipe both provide the compiled dependency imports SPINE needs at runtime.
Expand Down
143 changes: 143 additions & 0 deletions docs/source/api/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,151 @@ File Readers
:toctree: generated

read.HDF5Reader
read.PIMMReader
read.LArCVReader
read.CacheReader

.. _pimm-input:

PIMM detector input
-------------------

Use ``provider: pimm`` for paired JAXTPC ``hits`` and ``labl``
files in the PIMM layout. ``PIMMReader`` delegates decoding, event alignment,
per-hit overrides and particle-ID normalization to ``pimm_data.JAXTPCDataset``.
SPINE consumes its untransformed NumPy output and retains its own parsers,
collation and augmentation.

Install ``spine[pimm]``. The optional extra includes PyTorch, ``pimm-data`` and
HDF5 compression filters; importing base SPINE does not import ``pimm-data``.
The Docker image installs this extra. The provisional dependency is pinned to
``doraemon-final-layout`` commit ``23dc94f3ac1134936891b281e4408a975d0097ed``;
an upstream release is required before this integration becomes official.
See the `upstream examples
<https://github.com/OmarAlterkait/pimm-data/tree/23dc94f3ac1134936891b281e4408a975d0097ed/examples/doraemon_final>`_.

.. code-block:: yaml

io:
loader:
minibatch_size: 1
num_workers: 0
dataset:
provider: pimm
data_root: /path/to/jaxtpc_pixel
dataset_name: sim_pixel
runs: "j*/*"
schema:
data: {provider: tensor}
seg_label: {provider: segmentation}
clust_label: {provider: cluster_label}
particles: {provider: particle}
coord_label: {provider: particle_coords}
ppn_label: {provider: particle_points}
meta: {provider: meta}
graph_label: {provider: particle_graph}
event_info: {provider: event_info}

For separately located shards, omit ``runs`` and provide ``files`` with
``hits`` and ``labl`` paths or glob patterns. The upstream loader requires
matching task and event identities and rejects partial pairs. Standard SPINE
entry selection (``n_entry``, ``n_skip``, ``entry_list``) is applied afterward.
Use the published final-layout corpus or prepare paired files using the
upstream PIMM data-preparation tools before starting training.

Coordinates, charge and supervision
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The 3D parsers currently require **pixel hits**. Upstream can read wire hits,
but a wire/tick pair does not define a unique 3D point; these parsers reject
wire input instead of interpreting its indexes as detector coordinates.

Pixel indexes ``(py, pz, tick)`` are converted using ``supera_pixel_geometry``
from the ``labl/config`` attributes. The transform uses pixel centers and
nominal drift coordinates, including the stored reference tick and drift
sign. ``supera_hit_x_from=nominal`` and ``supera_hit_energy=charge`` are required.
No interaction-time correction is inferred from truth.

All spatial parsers share ``voxel_size_mm`` (default 3) and ``origin_mm``
(default zero). Each event's integer grid is translated to nonnegative
coordinates; the shared ``Meta`` retains its absolute position in cm.
Conflicting parser grids are rejected.

* The tensor contains **summed hit charge**, in the source hit units, not MeV.
Signed charge is retained. Coincident hits share one voxel.
* Semantic labels come directly from ``segment_motif``: shower, track, Michel,
delta, LE. Overlap winners use track, shower, delta, Michel, LE priority,
followed by source row order; every voxel product shares that selection.
* Voxels without any labelled contribution are excluded by default because
generic SPINE segmentation requires targets in 0--4. Charge from all hits
in retained voxels is preserved. Set ``include_unlabeled: true`` on every
spatial parser to retain target ``-1`` for inspection, not generic training.
LE remains class 4 with its upstream ownership references
(``le_is_stuff=False``); downstream objectives retain their normal selection.
* Fragment, instance, ancestor and graph references use event-local rows of
the PIMM particle table. SPINE collation applies the batch offsets.
* Coordinate labels contain one start/end pair per particle-table row. Stored
collected-step positions use the shared transform; missing endpoints remain
NaN. No deposit-derived or track-creation substitute is used. PPN includes
finite, in-image instance starts and track ends.
* Particle PID is mapped from PDG to SPINE's classes. Primary flags and
genealogy use the upstream table; interaction vertices use the host
instance's interaction. Missing ancestry is never indexed through ``-1``.
* Particle deposited energy is unavailable from these hit/label products and
remains NaN. Initial energy and momentum retain MeV and MeV/c. Generator
neutrino information is unavailable. The ``event_info`` product preserves
the upstream event ID when present and the source entry, without invented
run/subrun values.

Truth construction uses the complete schema above with:

.. code-block:: yaml

build:
mode: truth
units: cm

Known limitations and validation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Missing fragment endpoints can reduce primary/orientation supervision.
Drift timing can also displace nominal hit coordinates relative to true
endpoints. Unavailable endpoints remain NaN.

Synthetic tests cover alignment, charge conservation, ignored labels, empty
clouds, missing endpoints, event selection, worker serialization, batching and
truth HDF5 round trips. Optional production checks use paired ``hits``/``labl``
directories via ``SPINE_PIMM_ROOT`` and assert nonzero supervision and gradients
for segmentation, PPN, Graph-SPICE and shower/track/interaction GrapPA. Their
availability is not a claim that every objective has eligible targets in every
production.

.. code-block:: bash

PYTHONPATH=src pytest -q test/test_io/test_pimm
SPINE_PIMM_ROOT=/path/to/paired/data PYTHONPATH=src \
pytest -q -s test/test_io/test_pimm

.. autosummary::
:toctree: generated

parse.pimm.PIMMTensorParser
parse.pimm.PIMMSegmentationParser
parse.pimm.PIMMClusterLabelParser
parse.pimm.PIMMParticleParser
parse.pimm.PIMMParticlePointsParser
parse.pimm.PIMMParticleCoordsParser
parse.pimm.PIMMParticleGraphParser
parse.pimm.PIMMMetaParser
parse.pimm.PIMMEventInfoParser
dataset.pimm
read.pimm
parse.pimm.base
parse.pimm.cluster
parse.pimm.metadata
parse.pimm.particle
parse.pimm.tensor

Entry Filtering
---------------

Expand Down Expand Up @@ -234,6 +376,7 @@ with raw input through the mixed dataset.
dataset.LArCVDataset
dataset.CacheDataset
dataset.HDF5Dataset
dataset.PIMMDataset
dataset.MixedDataset
dataset.JointDataset

Expand Down
7 changes: 7 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ Documentation = "https://spine.readthedocs.io/"
"Bug Tracker" = "https://github.com/DeepLearnPhysics/spine/issues"

[project.optional-dependencies]
# Detector-hit readers and compression filters used by PIMM productions.
# Provisional doraemon-final-layout commit; require a release before adoption.
pimm = [
"pimm-data @ git+https://github.com/OmarAlterkait/pimm-data.git@23dc94f3ac1134936891b281e4408a975d0097ed",
"hdf5plugin>=4",
]

# Visualization dependencies
viz = [
"matplotlib>=3.3.0",
Expand Down
2 changes: 2 additions & 0 deletions src/spine/io/dataset/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@
from .joint import JointDataset
from .larcv import LArCVDataset
from .mixed import MixedDataset
from .pimm import PIMMDataset

__all__ = [
"CacheDataset",
"HDF5Dataset",
"JointDataset",
"LArCVDataset",
"MixedDataset",
"PIMMDataset",
]
3 changes: 2 additions & 1 deletion src/spine/io/dataset/hdf5.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ class HDF5Dataset(BaseDataset):

name: ClassVar[str] = "hdf5"
supports_stages: ClassVar[bool] = False
parser_dict: ClassVar[dict[str, Any]] = PARSER_DICT
parsers: dict[str, Any]
reader: ReaderBase

Expand Down Expand Up @@ -102,7 +103,7 @@ def __init__(
parser_cfg = dict(parser_cfg)
parser_stage = parser_cfg.pop("stage", stage)
parser = instantiate(
PARSER_DICT, parser_cfg, alt_name="parser", dtype=dtype
self.parser_dict, parser_cfg, alt_name="parser", dtype=dtype
)
self.parsers[data_product] = parser
for key in parser.tree_keys:
Expand Down
59 changes: 59 additions & 0 deletions src/spine/io/dataset/pimm.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
"""PIMM detector-hit dataset using the common HDF5 parsing machinery."""

from __future__ import annotations

from collections.abc import Mapping
from typing import Any, ClassVar

from spine.config.factory import module_dict

from ..parse import pimm
from ..read.pimm import PIMMReader
from .hdf5 import HDF5Dataset

__all__ = ["PIMMDataset"]


class PIMMDataset(HDF5Dataset):
"""Parse aligned PIMM hits and truth into aligned SPINE training products."""

name: ClassVar[str] = "pimm"
parser_dict = module_dict(pimm)

def _build_reader(
self, stage: str | None, stage_map: Mapping[str, str], kwargs: Mapping[str, Any]
) -> PIMMReader:
"""Select the native reader and enforce a common voxelization grid.

Parameters
----------
stage : str, optional
Reserved cache-stage selection; always None for this dataset.
stage_map : mapping
Reserved cache product routing; empty for this dataset.
kwargs : mapping
Physical-reader and entry-selection options.

Returns
-------
PIMMReader
Native reader projected onto the products required by the schema.

Raises
------
ValueError
If parsers request inconsistent grid settings.
"""
del stage, stage_map

# Every supervision product must use the same voxel coordinates.
grids = {p.grid for p in self.parsers.values() if hasattr(p, "grid")}
if len(grids) > 1:
raise ValueError(
"PIMM parsers must use the same voxel size, origin and unlabelled policy."
)

# Reuse the parent dataset's schema-driven physical projection.
return PIMMReader(
keys=None if self.keys is None else tuple(self.keys), **kwargs
)
6 changes: 6 additions & 0 deletions src/spine/io/parse/pimm/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Parsers for aligned PIMM detector hits and truth tables."""

from .cluster import *
from .metadata import *
from .particle import *
from .tensor import *
Loading
Loading