Skip to content

Add weekly re-import pipeline - #52

Open
camposandro wants to merge 11 commits into
mainfrom
sandro/weekly-reimport
Open

camposandro wants to merge 11 commits into
mainfrom
sandro/weekly-reimport

Conversation

@camposandro

@camposandro camposandro commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Add a weekly pipeline to re-import the alert catalog as a new HATS collection.

The collection is repartitioned, contains a 10 arcsec margin, and a diaSourceId index. This PR also adds scripts/weekly.sbatch (to be scheduled for Saturdays with 8 workers × 50 GiB) and updates the docs.

Design details

Today the catalog is working/, which the daily runs write to and the public symlink points to. This PR turns it into a HATS collection, collections/, with one directory per version, named by the UTC date it was made. Each weekly writes a whole new version next to the current one, then points collections/collection.properties to it:

collections/
├── collection.properties   points into the current version
├── 20261101/               previous version, kept for users
└── 20261107/               current version: main catalog (daily runs write here), margin, index

Users open collections/, which always gives them the current version, so the symlink (http://slac.data.lsdb.io:10942/lsdb/rubin_alert_archive/) only has to be repointed once: from working/ (the current one) to collections/. The first version is created from working/ itself (see details in the follow-up section).

A weekly run holds the pipeline lock and:

  1. skips, with an email, if the format-change flag is set;
  2. deletes orphaned daily files and rebuilds stale metadata;
  3. re-imports into <YYYYMMDD>/ with hats-import, which checks the schema and row count;
  4. marks the merged runs consolidated, so they can't be re-ingested;
  5. points collection.properties at the new version, in one atomic write;
  6. deletes all but the new and previous versions.

Users always see either the previous version or the new one. A failure leaves the previous version as the current one, and a retry the same day deletes the half-built version first.

Follow-up

The first weekly will need some follow-up work outside this PR. I will need to:

  1. Upgrade hats and hats-import to v0.11.2 on USDF. These pins are necessary because the reimport of catalogs with leaf pixel directories requires fixes hats#782 and hats-import#747.
  2. Between two daily runs, point the daily run to current_catalog_dir, move working/ to a first collection with adopt_working and repoint the symlink (as demonstrated in the runbook).
  3. Add the cron line to run weekly.sbatch on sdfcron001.

Thanks for reviewing, and sorry for the size (once again)! A lot of it is tests and docs. The main logic lives in collection.py, weekly/reimport.py and weekly/run.py, in case that's useful.

Closes astronomy-commons/lsdb#1536.

@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.24242% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.59%. Comparing base (38dfd77) to head (c8e2029).

Files with missing lines Patch % Lines
src/alert_archive/collection.py 98.24% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #52      +/-   ##
==========================================
- Coverage   99.65%   99.59%   -0.06%     
==========================================
  Files          12       15       +3     
  Lines         858      987     +129     
==========================================
+ Hits          855      983     +128     
- Misses          3        4       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@olivialynn olivialynn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

Comment thread scripts/weekly.sbatch
Comment on lines +33 to +35
export SHELL=/bin/bash
export PYTHONUNBUFFERED=1
setup lsst_distrib

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export SHELL=/bin/bash
export PYTHONUNBUFFERED=1
setup lsst_distrib
set -e
export SHELL=/bin/bash
export PYTHONUNBUFFERED=1
source /sdf/group/rubin/sw/loadLSST.sh
setup lsst_distrib

If you've already run this on the crontab node and it works, ignore the following - but otherwise, if it doesn't, two things that might be helpful:

  • adding a set -e to the top of the file, so the script will halt if the setup fails, instead of proceeding to launch the python
  • first sourcing setup as a command from loadLSST.sh, before running setup lsst_distrib

Comment on lines +292 to +295
def mark_consolidated(self, run_ids):
"""Mark the runs a weekly re-import merged as :attr:`~RunRecord.consolidated`.

:meth:`reingest_runs` then refuses them, even with ``missing_files_ok``,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def mark_consolidated(self, run_ids):
"""Mark the runs a weekly re-import merged as :attr:`~RunRecord.consolidated`.
:meth:`reingest_runs` then refuses them, even with ``missing_files_ok``,
def _mark_consolidated(self, run_ids):
"""Mark the runs a weekly re-import merged as :attr:`~RunRecord.consolidated`.
Only the weekly re-import should call this, holding the lock.
:meth:`reingest_runs` then refuses them, even with ``missing_files_ok``,

Nit: maybe make this a private member, and mention in the docstring that the caller must hold the pipeline lock (since it's rewriting the whole log). Should be fine without the change, but it would be nice to idiot-proof our future selves 😎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Weekly re-import of alert archive with atomic promote and rollback

2 participants