Skip to content

Optional: write zarr alongside netCDF for direct-from-S3 reads #117

Description

@joglekara

Not needed for v1 of the Thomson analysis browser — filed so the option stays visible when weighing how much server we want to run long-term.

The browser's backend will download .nc artifacts and slice them server-side. If fits also wrote the same datasets as consolidated zarr, a web client could read chunks straight from S3 (zarr-js / zarrita) and most server-side slicing endpoints become optional — the visualizer trends toward a static frontend plus the MLflow REST API.

Scope

  • Config flag (default off) to write to_zarr copies with consolidated metadata next to the existing netCDF in binary/
  • Chunking chosen for the access patterns: whole-lineout reads and downsampled overview reads
  • Entry in the artifact manifest (see the manifest/contract issue)
  • Check artifact-size impact on public-ergodic-continuum before ever defaulting it on

netCDF stays canonical either way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions