Skip to content

Add temp fix for new schema 1102 (just drop new cols for now) - #44

Draft
olivialynn wants to merge 15 commits into
mainfrom
olivia/format-1102
Draft

olivialynn wants to merge 15 commits into
mainfrom
olivia/format-1102

Conversation

@olivialynn

Copy link
Copy Markdown
Member

Change Description

Rubin started sending alerts in format 1102 on Nov 11, which set the format-change flag. This lets the daily run ingest 1102 alerts into the catalog's current 1101 columns, dropping 1102's seven new fields until we re-import with them. Design doc updated.

Solution Description

  • schemas/1102.json: copy of the archive's 1102 schema. vs 1101: seven new fields in diaSource (and so prvDiaSources), nothing removed or retyped.
  • schema.py:
    • CATALOG_SCHEMA_ID = 1101 pins the catalog's columns, instead of "newest known format"
    • fields of a known format the catalog doesn't have go in ConformReport.dropped_known, and don't count as a format change
    • anything else dropped still sets the flag
  • Tests: 1102 vs 1101 diff, conforming 1102 alerts, an unknown field next to 1102's still flags, and an end-to-end daily run on an 1102 chunk
  • Design doc: new "Format 1102 is ingested without its new fields, for now" bullet

Notes

  • We're dropping real data for now
    • mixedInstruments, pixelFlags_spike, templateInstrument, templateMjdMin/Mean/Max, templateVersion
    • Still in the raw Avro alerts (and landing chunks), so a re-import can bring them back
    • This is tracked in Full reimport with support for schema 1102 #43
  • Adding them = raise CATALOG_SCHEMA_ID + full re-import
    • Older alerts would get nulls, same as the 1101 fields today
    • Sandro's reconverted 1101 chunks have the new columns, but all null, so no extra data there
  • Checked against the real chunks that set the format-change flag (171193, 171194): all 1102, only the 14 known new fields dropped, no casts or nulls

@olivialynn
olivialynn added this pull request to stack #28 November 12, 2026 22:06
@codecov

codecov Bot commented Nov 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.52%. Comparing base (e9b71ce) to head (d0be29a).

Additional details and impacted files
@@                Coverage Diff                @@
##           olivia/daily-run      #44   +/-   ##
=================================================
  Coverage             99.51%   99.52%           
=================================================
  Files                    14       14           
  Lines                  1041     1056   +15     
=================================================
+ Hits                   1036     1051   +15     
  Misses                    5        5           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Base automatically changed from olivia/daily-run to main November 19, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant