Add zarr v3 support across image loading, dashboard, and finetuning - #92
Merged
Conversation
zarr-python 2.x cannot open zarr.json (v3) stores, so add a dedicated zarr_v3 helper module and route dataset discovery, scale-pyramid/ multiscale detection, and finetune crop loading through it whenever a v3 container is detected.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed correctness issues in the new v3 path handling (multiscale chunk-shape contract and closest-scale detection for v3 scale paths) and in crop resampling that can drop data when shapes aren’t divisible by downsample factors.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR adds first-class handling for Zarr v3 (zarr.json) datasets across dataset inspection/loading, neuroglancer/dashboard utilities, and finetuning crop ingestion—while preserving the existing Zarr v2 code paths for backward compatibility.
Changes:
- Introduces
cellmap_flow/utils/zarr_v3.pyto read v3 metadata directly and open v3 arrays via TensorStore’szarr3driver. - Routes dataset discovery, multiscale detection, closest-scale selection, and finetune crop loading through the v3 helper when a v3 container is detected.
- Improves dashboard crop import behavior by resampling mismatched voxel sizes and fixing overlay chunk/bbox overlap logic.
File summaries
| File | Description |
|---|---|
cellmap_flow/utils/zarr_v3.py |
New helper module for Zarr v3 container detection, metadata parsing, multiscale interpretation, and TensorStore-backed array reads. |
cellmap_flow/utils/ds.py |
Detects v3 containers and dispatches get_ds_info to the v3 helper; adjusts TensorStore driver selection for v3. |
cellmap_flow/image_data_interface.py |
Adds v3-aware multiscale/closest-scale resolution during dataset initialization. |
cellmap_flow/utils/scale_pyramid.py |
Updates multiscale detection to check v3 multiscales metadata when applicable. |
cellmap_flow/utils/neuroglancer_utils.py |
Adds v3-aware closest-scale resolution for viewer overlay scaling. |
cellmap_flow/finetune/crop_loader.py |
Adds v3 support for reading voxel size/offset metadata and opening v3 arrays for crop loading. |
cellmap_flow/dashboard/routes/finetune/yaml_crops.py |
Adds resampling (majority-vote downsample + zoom fallback) when crop voxel size differs from target volume voxel size. |
cellmap_flow/dashboard/routes/finetune/overlay.py |
Fixes chunk classification by using overlap instead of full-containment against imported crop bounding boxes. |
tests/utils/test_zarr_v3.py |
Adds unit tests for v3 metadata reading and array access behavior. |
tests/finetune/test_crop_loader_zarr.py |
Verifies v2 and v3 crop loading produce identical voxel size/offset and data reads. |
tests/finetune/test_write_crop_into_volume.py |
Regression tests for voxel-size mismatch resampling and half-voxel translation correction. |
tests/utils/test_overlay_chunk_bbox.py |
Regression tests for bbox overlap logic in overlay chunk classification. |
tests/utils/__init__.py |
Adds package marker for utils tests. |
.gitignore |
Ignores daisy_logs/. |
Review details
- Files reviewed: 12/14 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+271
to
+288
| integer_factors = eff_output_vs / src_voxel_size_nm | ||
| if np.all(scale_ratio <= 1.0) and np.allclose( | ||
| integer_factors, np.round(integer_factors), atol=1e-6 | ||
| ): | ||
| # Exact integer downsample: majority-vote (mode) over each | ||
| # block, rather than picking one arbitrary corner sample. | ||
| remapped = _majority_vote_downsample(remapped, integer_factors) | ||
| else: | ||
| from scipy.ndimage import zoom | ||
|
|
||
| # grid_mode=True aligns to pixel *centers* rather than the | ||
| # default's array-endpoint alignment (wrong, and increasingly | ||
| # so toward the edges) -- but it still samples a single fixed | ||
| # corner of each block, used here only as a fallback for | ||
| # non-integer ratios / upsampling where block-voting doesn't | ||
| # apply. | ||
| remapped = zoom(remapped, scale_ratio, order=0, grid_mode=True, mode="nearest") | ||
|
|
Comment on lines
+26
to
+30
| v3_container = zarr_v3.find_v3_container(dataset_path) | ||
| if v3_container is not None and zarr_v3.multiscales_from_group(v3_container) is not None: | ||
| _, resolutions, _ = zarr_v3.get_scale_info_v3(v3_container) | ||
| target_scale, _, _ = zarr_v3.find_closest_scale_v3(v3_container, target_resolution) | ||
| return tuple(resolutions[target_scale]) |
Comment on lines
+211
to
+223
| if spatial_indices is not None: | ||
| voxel_size = Coordinate(scale[i] for i in spatial_indices) | ||
| offset = Coordinate(translation[i] for i in spatial_indices) | ||
| shape = Coordinate(arr_meta["shape"][i] for i in spatial_indices) | ||
| axes_names = [axes[i]["name"] for i in spatial_indices] | ||
| else: | ||
| voxel_size = Coordinate(scale) | ||
| offset = Coordinate(translation) | ||
| shape = Coordinate(arr_meta["shape"]) | ||
| axes_names = ["z", "y", "x"][-len(shape):] | ||
| chunk_shape = tuple(arr_meta["chunk_grid"]["configuration"]["chunk_shape"]) | ||
| roi = Roi(offset, voxel_size * shape) | ||
| return voxel_size, chunk_shape, shape, roi, axes_names, "zarr" |
Comment on lines
+225
to
+229
| best_count = np.zeros(block_dims, dtype=np.int32) | ||
| result = np.zeros(block_dims, dtype=labels.dtype) | ||
| for val in np.unique(labels): | ||
| count = (flat_blocks == val).sum(axis=-1) | ||
| better = count > best_count |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
zarr.json(v3) stores, so this adds a dedicatedcellmap_flow/utils/zarr_v3.pyhelper module for reading v3 container metadata/arrays directly.image_data_interface.py,utils/ds.py), scale-pyramid/multiscale detection (utils/scale_pyramid.py,utils/neuroglancer_utils.py), and finetune crop loading (finetune/crop_loader.py) through the v3 helper whenever a v3 container is detected, falling back to the existing zarr-python 2.x path otherwise.dashboard/routes/finetune/yaml_crops.py) gains resampling when a crop's voxel size differs from the target volume's: majority-vote downsampling for exact integer factors, nearest-neighborzoomfallback otherwise, plus a half-voxel offset correction matching OME-NGFF's own multiscale translation convention.dashboard/routes/finetune/overlay.py) that required chunks to be fully inside an import bbox (misclassifying boundary chunks) to use overlap instead.