Conversation
- MIGRATION_STRATEGY.md: High-level approach, tech stack, phases - MIGRATION_OVERVIEW.md: Complete checklist and status - PYTHON_MIGRATION_PLAN.md: Detailed technical implementation guide - PYTHON_MIGRATION_PLAN_ERROR_LOGGING.md: Error tracking strategy - ARCHITECTURE_PER_TRACKER.md: Per-tracker processing design - ARCHITECTURE_STATELESS_GCP.md: GCP stateless deployment with BigQuery state - LOGGING_COMPARISON.md: loguru vs structlog comparison - CLAUDE.md: Documentation for AI assistance
- Create a4d-python/ subfolder for Python implementation - Set up project with uv/pyproject.toml - Configure dependencies: polars, duckdb, pydantic, loguru, etc. - Create package structure: extract, clean, tables, gcp, state - Add Dockerfile for containerization - Add basic configuration with Pydantic Settings - Add README with quick start guide Technology stack: - Polars (dataframes), DuckDB (SQL), Pydantic (config/validation) - loguru (logging), pytest (testing), uv (dependencies) - Google Cloud SDK (BigQuery/GCS integration)
- Add Python CI workflow using Astral's complete stack - ruff for linting and formatting - ty for type checking - uv for dependency management - Update .gitignore to exclude .serena/ and secrets/ - Configure CI to run on migration branch and PRs - Only triggers when Python code changes
- Merge 8 separate docs into 1 comprehensive MIGRATION_GUIDE.md - Move docs to a4d-python/docs/migration/ (better organization) - Update CLAUDE.md for Python project (moved to a4d-python/docs/) - Remove scattered docs from root directory What's included in MIGRATION_GUIDE.md: - Strategy & architectural decisions - Technology stack (Astral toolchain) - Architecture (per-tracker, BigQuery state) - Key migration patterns (R → Python) - Phase-by-phase checklist - Code examples for critical components - Success criteria Single source of truth for the migration, easier to maintain.
- Points to both R (legacy) and Python (active) projects - Links to detailed Python documentation in a4d-python/docs/ - Warns about shared reference_data/ used by both - Ensures AI assistance can find guidance at repository root
- Add justfile with common development commands (test, lint, format, check, ci) - Add Docker commands (docker-build, docker-run) - Add utility commands (sync, clean, update, info) - Update README to showcase justfile commands as primary workflow - Replace mypy reference with ty in README - Reorganize Technology Stack section to highlight Astral toolchain 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add ColumnMapper class to standardize column names from tracker files using YAML-based synonym definitions. This is the first component of Phase 1 (Core Infrastructure) and is essential for Script 1 (data extraction). Features: - Load synonyms from YAML files (synonyms_patient.yaml, synonyms_product.yaml) - Build reverse lookup for fast column name resolution - Rename Polars DataFrame columns to standardized names - Support strict mode to validate all columns are mapped - Helper methods for column validation and missing column detection - Robust path finding using Path(__file__).parents[4] to locate reference_data Tests: - 19 tests with 99% code coverage - Unit tests for all mapper functionality - Integration tests with actual reference_data YAML files - Tests for edge cases (duplicates, unmapped columns, missing files) Documentation: - REFERENCE_DATA_MIGRATION.md with detailed migration plan for all reference data files (synonyms, provinces, data_cleaning.yaml, clinic_data.xlsx) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…tching Create reusable utilities for loading reference data and implement province validation module. Refactor synonyms mapper to use shared code. New utilities (utils/reference_data.py): - find_reference_data_dir() - Locate reference_data directory from package - load_yaml() - Common YAML loading with optional relative path support - get_reference_data_path() - Build paths to reference data files Province validation (schemas/provinces.py): - load_allowed_provinces() - Load and flatten all provinces (lowercased) - load_provinces_by_country() - Load provinces organized by country - is_valid_province() - Case-insensitive province validation - get_country_for_province() - Lookup country for a province - All province data lowercased for case-insensitive matching - Results cached with @lru_cache for performance Refactoring: - Updated ColumnMapper to use shared load_yaml() and get_reference_data_path() - Simplified loader functions by removing duplicate path-finding logic - Removed custom reference_data_dir parameter (use shared utilities instead) Tests: - 26 tests for province validation with 100% coverage - Case-insensitive validation tests (Bangkok/BANGKOK/bangkok all valid) - Integration tests with actual allowed_provinces.yaml file - Unicode province name support (Vietnamese, etc.) - Updated synonyms tests to match new error messages 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Move all reference data loaders into a cohesive reference/ package,
improving code organization and making the purpose of each module clearer.
Package reorganization:
- utils/reference_data.py → reference/loaders.py (shared YAML loading)
- synonyms/mapper.py → reference/synonyms.py (column mapping)
- schemas/provinces.py → reference/provinces.py (province validation)
Test reorganization:
- tests/test_synonyms/ → tests/test_reference/test_synonyms.py
- tests/test_schemas/ → tests/test_reference/test_provinces.py
New structure:
```
src/a4d/
├── reference/ # All reference data loaders
│ ├── __init__.py # Clean exports
│ ├── loaders.py # Shared utilities
│ ├── synonyms.py # Column name mapping
│ └── provinces.py # Province validation
tests/test_reference/ # Tests mirror package structure
├── test_synonyms.py
└── test_provinces.py
```
Benefits:
- Clear purpose: Everything in reference/ loads from reference_data/
- Co-location: All reference data handling in one package
- Cleaner imports: `from a4d.reference import load_patient_mapper`
- Better test organization: Tests mirror src/ structure
- Removed old synonyms/ and schemas/ directories
All 43 tests pass with 80% coverage.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Eliminate two-pass workbook loading (structure + read-only modes) - Implement forward-fill logic for horizontally merged cells - Achieve 72% average speedup (2.4s → 0.4s per sheet) - Update profiling scripts and documentation - All tests pass with correct column counts and headers
- 180 lines of clean, well-tested code
- 91% code coverage
- Handles all edge cases from real tracker files (2024, 2019, 2018)
2. Key Features Implemented:
- ✅ Read all month sheets from Excel trackers
- ✅ Extract tracker year from sheet names or filename
- ✅ Merge two-row headers with horizontal fill-forward
- ✅ R-compatible duplicate column merging (concatenate values with commas, like tidyr::unite())
- ✅ Apply synonym mapping for column harmonization
- ✅ Add metadata columns (sheet_name, tracker_month, tracker_year, file_name)
- ✅ Combine sheets with type-safe concatenation
- ✅ Filter invalid patient rows
3. Testing: 25 comprehensive tests covering all edge cases
4. Documentation Updates:
- Updated MIGRATION_GUIDE.md with Phase 2 progress
- Updated CLAUDE.md with current status
- Created memory: r_implementation_check.md - reminder to always verify against R code
Add automatic age correction from date of birth (DOB) to match R pipeline's fix_age() function. This ensures data quality by always calculating age from DOB rather than trusting potentially incorrect Excel values. Changes: - Add _fix_age_from_dob() function in clean/patient.py (step 5.5) - Calculate age: tracker_year - birth_year - (1 if tracker_month < birth_month else 0) - Log warnings and track errors via ErrorCollector for all age corrections - Handle missing ages, mismatched ages, and negative ages (set to error value) Validation: - Tested with 2025_06_CDA tracker: 35 age errors properly corrected and tracked - Results now match R output (e.g., patient KH_CD016: 18 years, not 21) - Improvement over R: structured error tracking instead of logging only Also adds: - compare_r_vs_python.py: Comprehensive comparison tool for validation - fastexcel dependency: Required for Excel reading in comparison scripts Fixes critical data quality issue where incorrect ages from Excel were propagated to final datasets. Now matches R pipeline behavior while providing better error tracking and documentation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implemented two critical data quality fixes to match R pipeline: 1. Date Validation (_validate_dates): - Validates all date columns against tracker_year - Replaces dates beyond December 31 of tracker_year with error date (9999-09-09) - Fixed: Patient KH_CD016 Mar25 fbg_updated_date (3035-03-01 → 9999-09-09) - Logs each invalid date with patient context 2. FBG Text Value Conversion (_fix_fbg_column): - Converts qualitative FBG values to numeric (CDC guidelines) - Mappings: high/hight/bad/hi → 200, medium/med → 170, low/good/okay → 140 - Removes "(DKA)" markers and trims whitespace - Matches R's fix_fbg() function (script2_helper_patient_data_fix.R:551-567) 3. Improved Comparison Script: - Fixed field names: patient_id (not national_id), sheet_name, tracker_date - Implemented approximate float comparison (rel_tol=1e-9, abs_tol=1e-12) - Enhanced error reporting with patient_id and sheet_name context - Shows ALL mismatches (not just first 3) - Fixed join logic to use composite key [patient_id, sheet_name] Results: fbg_updated_date mismatches resolved, only 2 expected differences remain (insulin_total_units: Python extracts correctly; status: minor formatting) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds robust date parsing to handle various formats from legacy trackers (2018-2019), fixing day/month swap issues with DD/MM/YYYY dates. Changes: - Add date_parser.py with parse_date_flexible() for handling: - DD/MM/YYYY and DD-MM-YYYY formats (Southeast Asian standard) - Month-year abbreviations (Mar-18 → 2018-03-01) - Excel serial numbers (days since 1899-12-30) - ISO dates with time components - 4-letter month name truncation - Update converters.py with parse_date_column() wrapper - Integrates flexible parser with ErrorCollector - Detects and logs parsing failures - Update patient.py to use flexible date parser - Replace simple cast with parse_date_column() for Date columns - Add _extract_date_from_measurement() for legacy combined value+date format - Extract dates from "value (Mar-18)" patterns in hba1c_updated, fbg_updated - Strip unit suffixes (mg/dl, mmol/l) from FBG values in legacy trackers - Add VALIDATION_TRACKING.md to track validation progress across 174 files Results for 2018_CDA A4D Tracker: - dob: 0% mismatches (was 52.2%) ✓ - t1d_diagnosis_date: 0% mismatches (was 89.9%) ✓ - recruitment_date: 0% mismatches (was 85.5%) ✓ - age: 0% mismatches (was 21.7%) ✓ - Cleaning errors: 38 (down from 53) The explicit format parsing with strptime() is more reliable than dateutil.parser's dayfirst=True for ambiguous dates like 06/05/2013. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Updates the compare_r_vs_python.py script to only require a filename instead
of full paths to both R and Python parquet files.
Changes:
- Add fixed base path constants for R and Python output directories
- Change CLI to accept --file/-f parameter with just the filename
- Script automatically constructs full paths from base directories
- Display resolved paths for transparency
Before:
uv run python scripts/compare_r_vs_python.py \
-r "/path/to/r/file.parquet" \
-p "/path/to/python/file.parquet"
After:
uv run python scripts/compare_r_vs_python.py -f "file.parquet"
This simplifies the workflow for comparing the 174 tracker files during
validation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Updates the month-year pattern regex to make the separator (hyphen/space) optional, handling legacy data quality variations. Changes: - Regex pattern: `[-\s]` → `[-\s]?` (makes separator optional) - Now handles: "Mar-18", "Mar 18", "Mar18" (all parse to 2018-03-01) This fixes the hba1c_updated_date mismatch in 2018 tracker where the raw value was "May18" instead of "May-18". Results for 2018_CDA A4D Tracker: - hba1c_updated_date: 0% mismatches (was 1.4%) ✓ - Cleaning errors: 37 (down from 38) The remaining fbg_updated_date mismatch (1.4%) is actually Python being correct - it properly parses DD/MM/YY format while R incorrectly interprets it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Documents completion of 2018_CDA A4D Tracker validation with detailed results. Changes: - Move 2018 tracker from "PARTIAL" to "✅ Validated Files" - Document 3 acceptable mismatches (Python more correct than R) - Update cleaning errors: 37 (down from 257 initially) - Update validation procedure to use simplified command - Add new acceptable differences for legacy trackers - Update summary statistics: 2 validated, 0 in progress, 172 pending - Update last modified date to 2025-11-07 Results for 2018_CDA A4D Tracker: - All date fields: 100% match ✓ - FBG extraction: Python correctly extracts, R shows error values - Date parsing: Python handles DD/MM/YY correctly, R has edge case bugs Python implementation is demonstrably more accurate than R for this legacy tracker. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit fixes two critical bugs that prevented processing of the 2021 Phattalung Hospital tracker (and likely other trackers with similar issues). **Bug 1: Extraction - Wrong data start row detection** - Problem: find_data_start_row() stopped at first non-None value in column A, but some sheets have stray spaces/text above patient data - Example: 2021 Phattalung had space " " at row 29, but patient data started at row 48. This caused wrong headers and skipped 7 month sheets (Jun-Dec), losing 58% of data (30/72 records) - Fix: Modified find_data_start_row() to search for first NUMERIC value (patient row IDs: 1, 2, 3...) instead of any non-None value - File: src/a4d/extract/patient.py:116 - Result: Raw extraction now correctly produces 72 records **Bug 2: Cleaning - map_elements() fails on all-null columns** - Problem: map_elements() with return_dtype=pl.Date fails when ALL values are None (e.g., hospitalisation_date column with only 'NA') - Root cause: Polars cannot infer Date type when there are zero non-null examples, even with return_dtype specified - Fix: Replaced map_elements() with list-based approach that creates pl.Series with explicit dtype=pl.Date (doesn't require non-null values) - File: src/a4d/clean/converters.py:151-158 - Result: Cleaning now completes successfully (72 records, 22 errors) **Validation Results:** ✅ Record counts match: R=72, Python=72 ✅ Schema matches: 83 columns ✅ Data quality: All mismatches are known acceptable differences (blood_pressure, insulin_regimen case, bmi precision) **Impact:** - 2021 Phattalung Hospital: FULLY FIXED - Extraction fix likely helps other trackers with stray values - Cleaning fix handles edge case of all-null date columns 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
**Root Cause**: Some Excel trackers have data quality issues where patient rows are missing the row number in column A (which is normally 1, 2, 3...) but still contain valid patient data in column B onwards. Example: 2022 Surat Thani Hospital tracker has patient TH_ST003 with: - Working months (Jan-Apr, Nov-Dec): row number = 3 in column A ✓ - Failing months (May-Oct): row number = None in column A, but patient_id='TH_ST003' in column B ✓ **Previous Logic**: Skipped ALL rows where row[0] (column A / row number) was None → Lost 6 TH_ST003 records from May-Oct sheets (-2.2% data loss) **New Logic**: Only skip rows where BOTH row[0] (row number) AND row[1] (patient_id) are None → Extracts all valid patient rows regardless of missing row numbers → Recovers the 6 missing TH_ST003 records **Impact**: - Fixes 2022 Surat Thani Hospital: Now extracts all 276 records (was 270) - More robust handling of Excel data quality issues - R pipeline handles this correctly (it doesn't rely on row numbers) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
**Status Update**: - 2022 Surat Thani Hospital: ✅ FULLY FIXED (276/276 records) - Resolved record count discrepancies: 3 total (was 2) - Remaining issues: 7 trackers (was 8) - Validation rate: 92.5% (was 92.0%) **Root Cause**: Patient TH_ST003 had missing row numbers in column A for months May-Oct, causing extraction to skip those rows even though valid patient data existed in subsequent columns. **Fix Applied**: Modified read_patient_rows() to only skip rows where BOTH row number AND patient_id are missing, instead of skipping all rows with missing row numbers. **Impact**: - Recovered 6 missing records (TH_ST003 now has all 12 months) - More robust handling of Excel data quality issues - Python output now matches R output perfectly (276 records each) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit fixes two critical extraction bugs found during validation: 1. Handle worksheets with None max_row value - Some Excel files don't have dimension metadata, causing ws.max_row to be None - Added fallback to use 1000 as max_row when None is encountered - Fixes: 2024 Sultanah Bahiyah tracker processing error 2. Filter out Excel error values in patient_id - Excel error values like #REF!, #DIV/0!, etc. should not be extracted as valid patient IDs - Added filtering to remove any patient_id starting with "#" - Applied to all three extraction paths: monthly sheets, Patient List, and Annual - Fixes: 2024 Sultanah Bahiyah had 3 extra records with patient_id="#REF!" Impact: - 2024 Sultanah Bahiyah: Now matches R output (142 records, was 145) - Aligns Python extraction with R pipeline behavior 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds compare_id_overlap (do the same patient_ids / product names appear on both sides, independent of the row-alignment key) and compare_categorical_overlap (same idea, generalized to every categorical column) as new layers in a4d.migration.compare, plus the fixes both needed once run against real data: raw pipeline output isn't schema-normalized like cleaned output, so per-file column presence varies -- compare_totals and compare_categorical_overlap now skip a column missing from either side instead of crashing. Wires all of this into scripts/compare_outputs.py: patient_data_raw and product_data_raw are now compared alongside the cleaned directories (one report per stage, so raw-vs-cleaned divergence can be localized to extraction vs. cleaning), and the console tables are grouped by tracker year (descending) instead of one flat file list.
…vergence Adds build_mismatch_rows to a4d.migration.compare -- flattens id_overlap, categorical_overlap, totals, and cell_mismatches into per-row dicts -- and wires it into scripts/compare_outputs.py as a per-stage Excel workbook (one sheet per measure), using openpyxl (already a dependency, no xlsxwriter needed). HTML stays the aggregate-counts dashboard; Excel carries the actual flagged rows so they can be filtered/sorted/annotated while triaging. Adds --only-mismatches to skip printing files where every measure is clean, threaded through the just recipe (which needed a trailing *ARGS to accept it -- safe here since flags have no spaces, unlike the paths *ARGS broke before). Renames "ID overlap"/"Categorical overlap" to "ID divergence"/"Categorical divergence" everywhere (console, HTML legend, docstrings) -- what's displayed and flagged is the R-only/Python-only divergence, not the shared overlap, so "overlap" was the wrong word. Also fills in the HTML legend's missing categorical-divergence entry and fixes compare.py's module docstring, which still described four layers after two more were added.
Removes render_html_report entirely -- triage means loading results as a dataframe, filtering, sorting, adding columns, which Excel supports and a static HTML page doesn't. Replaces it with build_summary_rows (same per-column/per-cause aggregate data, as plain dict rows) merged into the same per-stage Excel workbook the flagged-row detail already goes into, so each stage now produces exactly one .xlsx instead of an .html + .xlsx pair. Adds compare_row_key_overlap / RowKeyOverlap: counts rows whose full row-alignment key (all key_cols, not just a single identity column) found no partner on the other side at all, or fanned out via a repeated key. This was the missing signal that made raw product files misleading -- 0 cell mismatches there turned out to mean "the join matched zero rows," not "everything agreed," and nothing previously said so. Row-key divergence makes that visible directly: 0 cell divergence + near-100% row-key divergence now reads as "nothing to compare," not "clean." Also does a naming consistency pass: every count-based check is now named "X divergence" (ID, Column, Categorical, Row-key, Totals, Cell), reordered so Totals sits with the other value-comparison checks and Row-key sits immediately before Cell divergence since both depend on the same row-alignment key.
build_mismatch_rows gained the row_key_overlap counts as detail sheets in the CLI and console, but never made it into the actual Excel workbook. Unlike the other sheets (one row per flagged issue), this one lists every file with its matched/r_unmatched/py_unmatched counts, since the point is sorting by unmatched count to find the worst-affected files, not just listing problems. Verified live: raw product files show near-100% row-key divergence (e.g. 560/560 unmatched) alongside near-zero cell divergence, confirming the cell-divergence 0 there is "nothing was paired to compare," not "everything agreed."
…osed Records the substantial follow-on work this session did to compare_outputs.py after ticket 15 formally closed and ticket 17 spawned: dropping HTML for Excel, adding id/categorical/row-key divergence checks, raw-vs-cleaned staging, and the naming consistency pass. Added as an addendum to ticket 15 (same deliverable maturing, not a new decision) and folded into ticket 17's premise, since RowKeyOverlap now gives it a direct way to validate a proposed alignment fix. Redrew the generated views per the wayfinder protocol.
The old key (clinic_id, product, product_sheet_name, product_entry_date) collapsed onto far fewer distinct values than rows exist wherever product_entry_date is null, causing join fan-out that inflated mismatch counts by orders of magnitude and hid product_entry_date itself from classification since it was a join key. Ordinal position within (clinic_id, product_sheet_name), taken in each file's existing row order, validated directly against the real R/Python output pair: row-key divergence drops from near-100% to ~2.8% (all 1330 remaining unmatched rows isolated to one clinic where R's frozen output has a clinic_id typo, NGH vs NOH). product_sheet_name mismatch count now reproduces the parity-presentation PDF's number (201) exactly, confirming the key is sound. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvLJ14sMDzGwDTVyhs7A15
…triage Row-alignment key is fixed and validated against the real R/Python output pair (row-key match 97.2%, product_sheet_name mismatch count reproduces the parity-presentation PDF exactly). Column-by-column triage for both arms did not converge in the same session, per the ticket's own pre-authorization to split -- carried into ticket 18 along with the still-unreconciled 189-vs-155-tracker discrepancy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvLJ14sMDzGwDTVyhs7A15
Persists each run's per-column/per-cause mismatch counts as a small JSON snapshot per stage under compare_history/, and diffs the new run against the most recent prior snapshot -- printed as a delta table and written as extra sheets in the Excel report. Makes a triage fix's actual effect (or a regression) visible directly by count, without needing every cause classified first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvLJ14sMDzGwDTVyhs7A15
--output-dir replaces --report-out: each invocation now writes its Excel reports and history snapshots into its own timestamped subfolder (output/comparison/<timestamp>/), so a run is a self-contained unit on disk instead of scattering same-named files across the cwd every time. Also removed ten stale one-off debug scripts predating this map's work, all hardcoded to a drive layout that no longer exists and superseded by either the pytest suite, ticket 10's profiling approach, or this comparison tool. Fixed the resulting dangling references and one stale hardcoded path in the kept analyze_logs.sql. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvLJ14sMDzGwDTVyhs7A15
Ticket 19's resolution, ticket 18's stale filename references, and the map's own summary all updated to match the same-session --output-dir rename, per-run timestamped subfolder, and scripts/ cleanup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvLJ14sMDzGwDTVyhs7A15
…pawn tickets 20-23 product_category mismatches trace to R's add_product_categories() doing a case/whitespace-sensitive left-join with no normalization, unlike Python's reference/products.py which lowercases and strips first. Added a PRODUCT_CATEGORY_CLASSIFIERS registry (r_category_lookup_miss) to compare.py. Renamed the entry_date typo_rescue classifier to r_value_missing since most cases are plain R extraction gaps, not typos. Resolved the 189-vs-155-tracker discrepancy: no snapshot on the drive reached 189 files against the old frozen R baseline. Then production tracker uploads grew to 248 files, so R was re-run once (no code changes, final capture before ticket 12 retires it) against the current set - output_r/ now holds that fresh output (229/243 files), with the old 155-file baseline preserved as output_r_155_frozen_backup_2025-11-14. That re-run showed most of the original product_category mismatch count was baseline staleness rather than the join bug itself, and pushed the row-alignment key to a 100% match. Remaining product columns, raw-stage columns, and the patient arm did not converge this session - split into tickets 20-23 with fresh premise numbers from the re-run. Also fixed justfile's compare-outputs recipe default (compare_output), which had drifted out of sync with compare_outputs.py's own documented default (output/comparison) since ticket 19's --report-out rename.
R's raw extraction stores unparsed source date text (an Excel serial string for date-formatted cells) while Python's raw extraction already ISO-formats parsed dates for the same value -- a representation difference the comparison tool was treating as a false mismatch on 99.9% of raw-stage product_entry_date rows. normalize_date_column() reuses the cleaning stage's own flexible date parser to bring both sides to a common date representation before diffing, wired in for the Product (raw) stage only. Verified against the real R/Python output on the USB drive: mismatches dropped from 65,743 to 91, landing in the same order of magnitude as other raw product columns. Triaged the residual 91: about half already land in existing seeded classifiers, the rest spread thinly across 14 files with no dominant pattern. Spawned by ticket 18; unblocks nothing on its own (ticket 12 still waits on tickets 21-23), but ticket 6's blocked_by list drops to three open tickets.
Ticket 20's raw-stage product_entry_date fix dropped mismatches from 65,743 to 91, but 50 of those 91 were left unclassified with no tracking, which doesn't meet the map's destination bar of every Python/R difference being explicitly decided. Folds that residual (plus one confirmed real bug: a stray newline-dated row in Python's raw extraction for one Sarawak sheet) into ticket 22's existing scope rather than leaving it stranded in a closed ticket.
…pace representation in raw product comparison remove_header_rows dropped rows where every cell was None, but a formula-emptied Excel cell can surface as "" instead of None, so a row with one stray empty cell survived extraction when R's is.na() check drops it. This was the root cause of a row-count/shift mismatch across several raw-stage product columns for multiple trackers, including the Sarawak stray-row bug ticket 20 had flagged but not chased. Also add normalize_numeric_column and normalize_whitespace_column to the R/Python comparison tool, mirroring ticket 20's normalize_date_column: R's own float-to-string conversion rounds trailing digits differently than Python's, and readxl's trim_ws default plus its line-ending convention diverge from openpyxl's raw extraction. Neither is a real content difference. Verified against the real drive data: raw-stage product mismatches across all 9 columns dropped from 2,007 to 105 (95%). Residual split into ticket 24.
Five of six columns (product_balance, product_received_from, product_released_to, product_remarks, product_units_received) trace to one cause: R's row sort falls back to raw input order whenever product_entry_date fails to parse (near-universal for several major clinics), while Python correctly sorts chronologically -- both follow the same documented algorithm, so this is R's already-known date-extraction gap resurfacing as a sort-order divergence, not a Python bug. Since the row-alignment key is purely positional, the order difference cascades into value-level mismatches on every column compared through it, even though the underlying data is unaffected (verified: 98.3% of affected groups still land on the same end-of-group balance). Added a row_order_divergence classifier to compare.py (backed by a new opt-in order_group_cols check in compare_cells) that explains 4 of the 5 columns fully or almost fully; product_balance shares the cause but under-detects via simple value-membership, left as future work. The sixth column, product, had a distinct cause -- an embedded \r\n-vs-\n line break surviving cleaning unnoticed -- resolved fully by extending ticket 22's whitespace normalization to the cleaned stage. Spawned ticket 25 for product_units_released's cleaned-stage mismatches (2,144), discovered unassigned to any ticket during this session.
The script never calls a4d.logging.setup_logging() (no pipeline run, no output_root), so loguru's default sink had no level filter and every date cell parsed by normalize_date_column logged a DEBUG line to the console. Remove the default sink and add a WARNING-level one instead.
Found while answering a question about the CLI's Column divergence count: compare_columns has been computed and shown since ticket 15 but every triage ticket since worked only off the cell_mismatches sheet, leaving this structural layer (column existence + dtype) unaddressed. Ticket 26 captures the current concrete findings and puts it on the frontier.
…artifact Extended normalize_date_column to the Patient (raw) comparison stage using the cleaned schema's own get_date_columns() helper (derived, not hand-listed). Verified against the real 248-tracker drive comparison: raw-stage mismatches dropped from 564,096 to 46,788 (91.7%) across 70 to 67 columns. Cleaned stage confirmed unaffected as expected. This closes ticket 12's last blocker, putting it on the frontier. Split the raw residual and the untouched cleaned-stage triage into tickets 27 and 28.
…vergences (ticket 28) _fix_t1d_diagnosis_age unconditionally recomputed the value from dob and t1d_diagnosis_date, discarding a patient's real recorded diagnosis age whenever a date failed to parse. Its docstring claimed this matched R, but R's equivalent function is dead code, never called from the pipeline - R always keeps the raw recorded value. Fixed to prefer the raw value and only fall back to date-based calculation when it is missing or an Excel error sentinel. Verified with a real 248-tracker pipeline re-run: column mismatches dropped from 25,968 to 4,807. recruitment_date and insulin_subtype turned out to be genuine, already- correct Python divergences rather than bugs - one confirmed against the real source Excel, the other already documented in code as an R validator bug. Added r_extraction_gap and r_validator_rejects_multivalue classifiers to the comparison tool so these stop showing as unclassified noise. Patient cleaned-stage mismatches: 120,639 -> 99,478. insulin_total_units, fbg_baseline_mg, and 56 untouched columns didn't converge this session - split into ticket 29. Also corrects ticket 12 (retire R): it was marked unblocked prematurely - tickets 24-28 carry the same "might still need R's source" risk ticket 12 already named as its reason to wait. No files were deleted; r-archive/ was briefly staged for removal outside this ticket's scope and fully restored before anything committed.
Add a column_divergence report sheet so compare_columns findings stop being CLI-only. Root-cause every named divergence against real drive data: product's Float64-vs-Int32 and all-null Boolean-vs-String cases are harmless representation artifacts (documented, not normalized); product_returned_by/product_units_returned raw-stage gap is a genuine R extraction bug, confirmed against real source Excel. Patient's raw stage did not converge, split into ticket 30. Also fix a broken except clause a formatter introduced during this session.
…ceived_from mismatches All 105 residual mismatches are comparison-tool representation artifacts, not pipeline bugs. R's readxl coerces a lone Excel date/time-formatted cell in an otherwise-numeric column to that column's numeric type, while openpyxl honors the cell's own format -- new STRAY_DATE_CLASSIFIERS classifier (using openpyxl's from_excel to also replicate the Excel 1900-leap-year serial bug). 20 rows were float-precision formatting differences, fixed by extending normalize_numeric_column to all three columns. The remaining 5 rows (old Mandalay wide-format files) are a comma/hyphen-split ambiguity on messy source notes where Python's extraction is verified more faithful than R's -- new WIDE_FORMAT_FRAGMENT_CLASSIFIERS classifier. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvLJ14sMDzGwDTVyhs7A15
…assify formula-error and Buddhist-era-typo mismatches Extends normalize_numeric_column and normalize_date_column (ticket 20/22 precedent) to the Patient (raw) comparison stage, and adds two new classifiers: r_formula_error (Excel formula-error strings R's raw extraction carries through where Python correctly has no cached value) and buddhist_era_typo (a clinician-entered Thai Buddhist-Era year in a Gregorian date cell, verified against real source Excel and confirmed harmless by the cleaned stage's existing future-date guard). Cuts raw-stage patient mismatches from 46,788 to 18,813 (59.8%). complication_screening (84% of what remains) and the rest of the residual columns are split into ticket 31.
… them in cleaning Extraction was silently converting the source trackers' own formula-error strings (#DIV/0!, #NUM!, ...) to null, so the raw layer misreported what the source file contained and the "this formula could not compute" signal was lost entirely rather than recorded. clean_excel_errors is removed from all four extraction call sites in both arms. normalize_excel_formula_errors (clean/converters.py) now nulls them at the cleaning stage and logs each under a new source_formula_error code. null rather than the 999999 numeric sentinel: that sentinel means "a value was recorded but is invalid", whereas here no value could be computed at all because a required input was never entered. Verified with a full both-arm re-run against the real 248-tracker dataset: cleaned-stage output byte-identical in both arms (no production data moved), raw stage now faithful, source_formula_error entries present across 145 tracker log files. The compare tool's excel_formula_error classifier matches both directions, since R is the inconsistent side: readxl nulls an error cell in a column it guesses numeric but keeps it in one it guesses character. Verified against real source Excel in both directions.
…arser noise Standing rule recorded on the map and propagated into every open triage ticket: explaining a difference and naming a cause is only half the job. Each one must also carry an explicit verdict on whether Python is doing the right thing. Observing "Python has A where R has B" and adding a classifier is not a decision in favour of A. Ticket 32 re-audits all nine existing classifier registries against that bar; off_by_one_day and r_value_missing are named as the clearest failures. Tooling fixes to the comparison script: - summarize_directory (compare.py, unit-tested) plus a per-arm totals table showing files affected and total per measure. The per-year tables say where a divergence is; there was no way to see how much without scrolling and adding up every year. - Silence a4d.clean.* logging. The date parser warns once per unparseable value, which is a real signal in the pipeline but noise here, where raw free-text columns are fed in deliberately; at 18 date columns x ~245 files it drowned the report entirely.
…pets) CI has failed on every push to migration since 2026-08-09. Sole cause is ruff format --check wanting to reformat the Python code blocks inside docs/migration/MIGRATION_GUIDE.md; the step fails in ~15s so the test suite never runs. Not a regression of ticket 4's fix. Diagnosis only, fix deferred to its own session. Wired as a blocker of ticket 6, since the destination requires CI green before promotion.
MIGRATION_GUIDE.md is a working spec document -- its fenced Python is illustrative prose, never imported or executed, so there is nothing for ruff to validate and reformatting only rewraps examples whose line breaks were chosen for readability. Scoped to the whole docs/migration directory rather than the one failing file: both markdown files carrying python fences live there, and the second would have re-broken CI on any future edit. All CI steps reproduced locally and passing: ruff check, ruff format --check, ty check src/, pytest (555 passed), product coverage gate 88%. The guard against local/CI check-set drift is spawned as ticket 34 rather than left as an intention -- just ci exists but still diverges from CI (no coverage gate, different pytest markers) and nothing makes anyone run it.
Build artifact from reproducing CI's coverage step locally; .gitignore covered .coverage but not the xml report CI generates.
CI green again means the suite's output is readable for the first time since 2026-08-09. All 17 warnings come from three sites and each is a real behaviour decision (notably whether an empty str.split fragment should become null or a phantom empty-string product row under Polars 2.0), not a silencing job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.