Skip to content

Preserve and recover archive frame IDs for calibrations - #490

Merged
timbeccue merged 2 commits into
mainfrom
fix/calibration-registration-frameids
Sep 29, 2026
Merged

timbeccue merged 2 commits into
mainfrom
fix/calibration-registration-frameids

Conversation

@timbeccue

@timbeccue timbeccue commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Archived calibrations can have calimages.frameid = NULL because two registration commands omit the archive ID when constructing their DataProduct:

  • Importing BPMs with banzai_populate_bpms --db-address "$DB_ADDRESS".
  • Uploading a manually prepared calibration with banzai_add_super_calibration /path/to/readnoise.fits --db-address "$DB_ADDRESS" --upload-to-archive.
  • Re-importing an archived calibration can also overwrite an existing ID with NULL.

A reduction can find these calibrations through the Archive basename lookup, but the existing save-back condition never persists the recovered ID: database records already include a frameid key, and opening the file updates that dictionary in place.

Changes

  • Pass the Archive response's id into DataProduct in add_bpms_from_archive.
  • Pass the ingester response's frameid into DataProduct in add_super_calibration, removing the ineffective cal_image.frameid assignment.
  • Remember whether a master calibration's ID was missing before opening it, then use the existing database helper to save a successfully recovered ID.

Recovery writes to cal_db_address, the same database used to select the calibration. This defaults to db_address in the main AWS configuration; an explicit calibration database remains the target at sites.

Registration without upload still stores NULL, including when replacing an existing filename. Existing NULL rows can be repaired through corrected re-imports or basename fallback during calibration use.

@timbeccue timbeccue changed the title Preserve archive frame IDs during calibration registration Preserve and recover archive frame IDs for calibrations Sep 24, 2026
@cmccully
cmccully requested a lite review from Copilot September 28, 2026 17:50
Comment thread banzai/calibrations.py
# If the frame id was not included originally but we were able to pull it from the archive,
# we store it for future use
if 'frameid' not in master_calibration_file_info and master_calibration_image.frame_id is not None:
if frame_id_was_missing and master_calibration_image.frame_id is not None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add if frame_id_was_missing case that then populates master_calibration_image.frame_id in the database?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The db update is just below in dbs.update_calibration_frameid(...). The old condition prevented it from running because it just looked for whether the key existed and not whether it was populated vs None, which is what this change fixes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Add coverage for re-importing an existing filename and updating its frame ID instead of NULL.

Review effort: Lite
Findings: None

What changed in this PR

This PR preserves archive frame IDs during calibration registration and recovery.

Changes:

  • Passes archive and ingester IDs into calibration records.
  • Persists IDs recovered through archive lookup.
  • Adds database tests for registration and recovery.
File Description
banzai/​tests/​test_dbs.py Tests registration and recovery behavior.
banzai/​main.py Preserves frame IDs during calibration registration.
banzai/​calibrations.py Saves IDs recovered through archive lookup.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@cmccully cmccully left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks fine. I do want to make sure that we store the newly found frame id in the database and I'm not finding the code to do that. We should probably also add a change log note and bump the version (though we can wait to tag the final version until we've merged all the prs).

@timbeccue
timbeccue merged commit d6c55fb into main Sep 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants