Repository navigation
Add weekly re-import pipeline - #52
Open
camposandro wants to merge 11 commits into
Open
camposandro wants to merge 11 commits into
camposandro wants to merge 11 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #52 +/- ##
==========================================
- Coverage 99.65% 99.59% -0.06%
==========================================
Files 12 15 +3
Lines 858 987 +129
==========================================
+ Hits 855 983 +128
- Misses 3 4 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
olivialynn
approved these changes
Nov 20, 2026
Comment on lines
+33
to
+35
| export SHELL=/bin/bash | ||
| export PYTHONUNBUFFERED=1 | ||
| setup lsst_distrib |
Member
There was a problem hiding this comment.
Suggested change
| export SHELL=/bin/bash | |
| export PYTHONUNBUFFERED=1 | |
| setup lsst_distrib | |
| set -e | |
| export SHELL=/bin/bash | |
| export PYTHONUNBUFFERED=1 | |
| source /sdf/group/rubin/sw/loadLSST.sh | |
| setup lsst_distrib |
If you've already run this on the crontab node and it works, ignore the following - but otherwise, if it doesn't, two things that might be helpful:
- adding a
set -eto the top of the file, so the script will halt if the setup fails, instead of proceeding to launch the python - first sourcing
setupas a command from loadLSST.sh, before runningsetup lsst_distrib
Comment on lines
+292
to
+295
| def mark_consolidated(self, run_ids): | ||
| """Mark the runs a weekly re-import merged as :attr:`~RunRecord.consolidated`. | ||
|
|
||
| :meth:`reingest_runs` then refuses them, even with ``missing_files_ok``, |
Member
There was a problem hiding this comment.
Suggested change
| def mark_consolidated(self, run_ids): | |
| """Mark the runs a weekly re-import merged as :attr:`~RunRecord.consolidated`. | |
| :meth:`reingest_runs` then refuses them, even with ``missing_files_ok``, | |
| def _mark_consolidated(self, run_ids): | |
| """Mark the runs a weekly re-import merged as :attr:`~RunRecord.consolidated`. | |
| Only the weekly re-import should call this, holding the lock. | |
| :meth:`reingest_runs` then refuses them, even with ``missing_files_ok``, |
Nit: maybe make this a private member, and mention in the docstring that the caller must hold the pipeline lock (since it's rewriting the whole log). Should be fine without the change, but it would be nice to idiot-proof our future selves 😎
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a weekly pipeline to re-import the alert catalog as a new HATS collection.
The collection is repartitioned, contains a 10 arcsec margin, and a
diaSourceIdindex. This PR also addsscripts/weekly.sbatch(to be scheduled for Saturdays with 8 workers × 50 GiB) and updates the docs.Design details
Today the catalog is
working/, which the daily runs write to and the public symlink points to. This PR turns it into a HATS collection,collections/, with one directory per version, named by the UTC date it was made. Each weekly writes a whole new version next to the current one, then pointscollections/collection.propertiesto it:Users open
collections/, which always gives them the current version, so the symlink (http://slac.data.lsdb.io:10942/lsdb/rubin_alert_archive/) only has to be repointed once: fromworking/(the current one) tocollections/. The first version is created fromworking/itself (see details in the follow-up section).A weekly run holds the pipeline lock and:
<YYYYMMDD>/with hats-import, which checks the schema and row count;consolidated, so they can't be re-ingested;collection.propertiesat the new version, in one atomic write;Users always see either the previous version or the new one. A failure leaves the previous version as the current one, and a retry the same day deletes the half-built version first.
Follow-up
The first weekly will need some follow-up work outside this PR. I will need to:
current_catalog_dir, moveworking/to a first collection withadopt_workingand repoint the symlink (as demonstrated in the runbook).weekly.sbatchonsdfcron001.Thanks for reviewing, and sorry for the size (once again)! A lot of it is tests and docs. The main logic lives in
collection.py,weekly/reimport.pyandweekly/run.py, in case that's useful.Closes astronomy-commons/lsdb#1536.