Skip to content

Avoid unnecessary copies in aligned single-component mosaics - #488

Merged
timbeccue merged 4 commits into
mainfrom
feature/mosaic-single-component
Oct 12, 2026
Merged

timbeccue merged 4 commits into
mainfrom
feature/mosaic-single-component

Conversation

@timbeccue

Copy link
Copy Markdown
Contributor

Description

MosaicCreator rebuilds the science, mask, and uncertainty arrays even when a single component already matches the output geometry. For large QHY frames, this adds unnecessary memory copying and temporary-file allocation.

This PR adds a shortcut for single-component images that already match the output geometry. It reuses existing arrays while keeping the same header updates, uncertainty precision, and memory-map behavior in later stages.

The shortcut is used for inputs with native float64 science data, uint8 masks, and writable, contiguous arrays. All other inputs use the existing mosaic code.

Validation

  • 438 unit tests passed, with five end-to-end tests excluded.
  • Regression coverage includes array and header equivalence, allocation counts, separate primary headers, binned windows, cropping, flipping, dtype handling, and read-only or strided inputs.
  • A single local synthetic full-frame benchmark with float64 uncertainties decreased from 2.85 s to 0.076 s, with matching output arrays and headers.

@timbeccue
timbeccue requested review from cmccully and drhaz September 15, 2026 23:22
@timbeccue

Copy link
Copy Markdown
Contributor Author

Some more details on the work we save here:

AWS banzai currently takes around 2 minutes to run this stage, and reductions on site deployments take about 8 minutes.

The following is work that we are able to skip for single images:

  1. Constructing the empty mosaic destination image
Array Bytes written into its mapping
Science 493 MB
Mask 62 MB
Uncertainty 493 MB
Duplicate uncertainty copy 493 MB
Total 1.54 GB
  1. trim() creates another copy of the above, adding 1.54 GB

  2. the final assignment loop copies the intermediate image into a final destination, adding 1.05 GB (the duplicate uncertainty array is not copied)

This is a total of 4.13 GB of writes into mapped arrays for full frame images. It's hard to tell how much of this is written to disk, but timing measurements imply at least some is being flushed.

With this change, the site reduction mosaic stage is reduced from over 8 seconds to near 0. I have not tested the AWS reduction timing.

Comment thread banzai/mosaic.py Outdated
Comment thread banzai/mosaic.py
@timbeccue
timbeccue merged commit fb199d4 into main Oct 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants