Skip to content

Add daily run (and reply) modules - #30

Merged
olivialynn merged 13 commits into
mainfrom
olivia/daily-run
Nov 19, 2026
Merged

olivialynn merged 13 commits into
mainfrom
olivia/daily-run

Conversation

@olivialynn

@olivialynn olivialynn commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Change Description

Adds:

  • the daily run: ingests whatever's new in landing/ into the working catalog)
  • the replay: mainly a one-off, this feeds the rest of the parquet dump through the daily run one night at a time. Gets the catalog to the same state as the v1 frozen catalog, and also helps us test out the daily increment code.

This is step 7 (Daily run + replay) in the design doc's implementation plan.

Solution Description

New Things

  • daily/run.py: the nightly run. Under the lock:
    1. checks for format-change flag: stops pipeline (and emails me) if flag is set. (The flag would have been set by a previous run, and indicates a human still needs to come and figure out how we want to handle the new schema format.)
    2. cleans up orphan files: from a crashed or preempted run
    3. finds new chunks in landing/: misnamed ones (e.g. 17005__*) are skipped and reported
    4. checks every chunk's format from its parquet footer (before importing anything): a format change writes nothing, sets the format-change flag, emails.
    5. imports with hats-import: import into the existing pixels (existing_pixels), writes <run_id>.parquet per pixel, rebuilds skymaps + metadata
    6. logs the run: and emails a summary to me (and anyone else who adds their email to the script 🙇 )
  • mock/replay.py: ingests one alertId prefix (~one night) per daily run.
    • copies the prefix's chunks into landing/ (note: currently pointing to a diff dir temporarily, since I lacked enough permissions for landing/), then runs the daily pipeline, so it takes the same path as live alerts
    • one Dask cluster for all runs, one summary email at the end
    • skips prefixes already ingested, so it can be stopped and resubmitted
  • scripts/replay.sbatch: runs the replay; MAX_NIGHTS=N for a trial.

Changes

  • notify.py:
    • new email for runs that stop because the flag is already set ("not run, flag set since ...")
    • the format-change email now says nothing was ingested and daily runs stop too (it predated the switch from best-effort ingest)
  • Design doc: replay + daily-run bullets, footer format check, minor updates.

Notes

  • Replay results: verified against v1 catalog in this notebook (replay_vs_frozen notebook)
    • 72 runs, 5,694,257 alerts, 163 chunks; nothing skipped
    • working = frozen: same 16,466,969 diaSourceIds, same count on every night, sampled values match (except the solar-system diaObjectId: null vs 0, as in the baseline PR)
    • median 1.6 min per run on 8 Dask workers
  • Partitions need the weekly re-import.
    • daily runs never split a pixel, and the baseline (Jun 26 only) had a sparse layout with 3 order-0 pixels
    • after the replay: 6 partitions over 200k, the largest 2.17M rows; median 24 files per partition
    • frozen, cut at 200k from the same alerts: 227 partitions, max 198,913
    • nothing breaks, but reads of those pixels are slow; a strong argument for doing the re-import soon
  • rubin:default on milano is preemptable.
    • two replay jobs got preempted; resubmitting resumed cleanly (each night is all or nothing)
    • a preempted job leaves state/lock behind, which has to be deleted by hand
    • matters for scheduling (step 9): auto-resubmit, stale-lock detection, or a smaller-than-whole-node request

@olivialynn olivialynn changed the title Olivia/daily run Add daily run (and reply) modules Oct 31, 2026
@olivialynn
olivialynn added this pull request to stack #28 October 31, 2026 22:01
@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.52%. Comparing base (38dfd77) to head (7f99276).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/alert_archive/mock/replay.py 97.22% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #30      +/-   ##
==========================================
- Coverage   99.65%   99.52%   -0.13%     
==========================================
  Files          12       14       +2     
  Lines         858     1057     +199     
==========================================
+ Hits          855     1052     +197     
- Misses          3        5       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@olivialynn
olivialynn force-pushed the olivia/daily-run branch 2 times, most recently from 99c8fc7 to 3f4a3ec Compare November 4, 2026 16:15
Base automatically changed from olivia/baseline to main November 4, 2026 16:47
@olivialynn
olivialynn marked this pull request as ready for review November 12, 2026 23:09

@camposandro camposandro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks great! 🙌

Comment thread src/alert_archive/daily/run.py Resolved
Comment thread src/alert_archive/mock/replay.py Resolved
@olivialynn
olivialynn merged commit 5f260da into main Nov 19, 2026
4 of 6 checks passed
@olivialynn
olivialynn deleted the olivia/daily-run branch November 19, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants