Fix Snowflake external table 0-row bug with flat JSON array format

[concaf]

Summary

• Snowflake's VARIANT type has a 16MB per-row limit. The current format writes each stage's output as a single JSON object per source document (e.g., 195MB for embeddings), which exceeds this limit and causes external tables to silently return 0 rows despite files being registered in EXTERNAL_TABLE_FILE_REGISTRATION_HISTORY.
• Switch all pipeline stages (converted, chunks, embeddings) to output JSON arrays of flat row objects. With STRIP_OUTER_ARRAY=TRUE already set on the external tables, each array element becomes its own row (~10KB each), well under the 16MB limit.
• Add new row types (ConvertedRow, ChunkRow, EmbeddingRow) and update all controllers to write the new format while reading both old and new formats for backward compatibility.

Companion DPO MR: Updates materialized views to read the flat row format (no more LATERAL FLATTEN needed).

Test plan

• Deploy to sandbox and trigger a pipeline for a data product (e.g., aiftestgdrivear)
• Verify select count(*) from <db>.staging.embeddings_ext_tbl returns > 0
• Verify select count(*) from <db>.staging.chunks_ext_tbl returns > 0
• Verify materialized views (EMBEDDINGS_MV, CHUNKS_MV, CONVERTED_MV) return correct data
• Verify MCP tools (get_chunks, search) still work end-to-end
• Test with an existing data product that has old-format files — verify re-processing generates new format

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}