BigStitcher Apptainer image writes Zarr v3 output by default; Fiji cannot open stitched results or intermediate datasets

[celiayaojingyi]

Description of the bug

Hi,
I am testing the EASI-FISH / BigStitcher workflow using the current BigStitcher container:
ghcr.io/janeliascicomp/bigstitcher:2.4.1-spark4.1.2-scala2.13-java21-ubuntu24.04-zarrv3-0ac2f8b，

The pipeline runs successfully and produces a stitched output container,however, the output appears to be written as Zarr v3 even when the container name uses the .zarr extension. I also noticed that the intermediate BigStitcher resave output is also written as Zarr v3.

The problem is that I cannot open or visualize either the final stitched.zarr result or the intermediate dataset.zarr / dataset.xml-associated data in Fiji / BigStitcher / BigDataViewer. Fiji does not seem to recognize these Zarr v3 outputs, so I cannot inspect the stitched result interactively.

I tried removing sharding-related options --useSharding from the BigStitcher config, but the output is still not recognized. So the issue does not appear to be only related to sharding.

My current question is:

1. Is Zarr v3 the expected default output format for this BigStitcher image?
2. Is there a way to force BigStitcher / SparkFusion / CreateFusionContainer to write Zarr v2 / OME-Zarr v2 instead?
3. What is the recommended way to visualize the final stitched result in Fiji?
4. If Fiji cannot currently read these Zarr v3 outputs, is the recommended workaround to output N5 or HDF5 instead?
5. Is there a known Fiji / BigDataViewer plugin version that can open the Zarr v3 output produced by this container?

I expected .zarr to produce an OME-Zarr v2-style output based on the pipeline documentation, but the actual result is Zarr v3.
Could you please clarify the intended workflow for visualizing BigStitcher Spark outputs in Fiji, or how to configure the output format to something Fiji can open?
Thanks!

Command used and terminal output

Relevant files

.nextflow.log

System information

No response

[cgoina]

@celiayaojingyi here are the answers to your questions:

1. Yes zarr v3 is the default output format for bigstitcher. Zarr V3 creates fewer files on the file system when sharding is used and it still can offer decent size chunks for viewing the data in tools like Neuroglancer (and I think even Fiji - the latest version)
2. BigStitcher still supports zarr v2 - simply name your output container using a '.zarr2' extension. Now you probably have "stitching_result_container": "stitched.zarr" - use "stiched.zarr2" instead. For the other tools used in EASI-FISH pipeline there is a specific parameter for the output zarr format bigstream_zarr_format, segmentation_zarr_format that you can set but I will make a change to follow the same convention as bigstitcher for zarr2 extensions.
3. You can view results in Fiji - but it is true my version is patched to support ZARR v3. However I know for sure BigStitcher's latest release was postponed until zarr v3 was available and supported in Fiji. BigStitcher github has a script update_fiji_deps.py that could be used to update the dependencies for zarrv3 support. Another way to visualize it is in Neuroglancer which also supports zarr3.
4. No I would not recommend using N5 because downstream tools do not fully support N5 yet. The other tools are python based and N5 support was removed from the python zarr library. We are still working (testing) the N5 but there are still bugs that have not been fixed yet.
5. I think BigDataViewer also supports zarr v3 but I really don't know what needs to be done for that - check with the BDV team what you need to do to view zarr3 containers. With that patch that I mention I can use the import file option in Fiji to view a specific scale of the zarr container but I have not used BDV except as the one packaged with BigStitcher.

I hope this answers all your questions

[celiayaojingyi]

@cgoina Thank you very much for the detailed reply. This is very helpful.

I tried the method you suggested and changed the BigStitcher output container name from "stitching_result_container": "stitched.zarr" to "stitching_result_container": "stitched.zarr2" but this directory still appears to be a Zarr v3 container, and the actual fused image data seems to be missing / empty.

For example, instead of getting a Zarr v2-style container with .zgroup / .zarray metadata, I still see Zarr v3-style metadata. The output folder exists, but it does not seem to contain the expected stitched image datasets.

Do you have any idea what might cause this? Is .zarr2 expected to work with the current BigStitcher Spark image ghcr.io/janeliascicomp/bigstitcher:2.4.1-spark4.1.2-scala2.13-java21-ubuntu24.04-zarrv3-0ac2f8b

or does it require a different config / BigStitcher version?

Thanks again for your help.

[cgoina]

@celiayaojingyi you are correct it still creates a zarr3 even if the extension is zarr2. The tools aren't quite consistent and in some cases it uses the extension in other it takes the default and the createContainer if the storageType was not set - it used a default value. I am changing this now but until then in your local checkout directory you should see a file <EASIFISH-checkout-dir>/conf/bigstitcher_config.yml. Look for createContainer: and add this to the advancedArgs: - '-s ZARR2' - make sure you use the correct indentation and no tabs - just spaces:
I think the one that is committed does not have the datatype either and BigStitcher will save your data as Float32. If that is OK with you, you don't need to worry but if your data is UINT16 you should set it accordingly - otherwise the fused zarr may be much larger than your input czi.

createContainer:
  classname: net.preibisch.bigstitcher.spark.CreateFusionContainer
  advancedArgs:
    - '--multiRes'
    - '--preserveAnisotropy'
    - '--dataType=UINT16'
    - '-s ZARR2'
    - '--compressionLevel=5'