fix(openai): preserve multimodal chat content

[Rana Singh (ranadeepsingh)]

Fixes #2246

Root cause

OpenAIChatCompletion flattened composite content arrays into joined text before request serialization, discarding non-text parts such as image_url.

Change

• Preserve heterogeneous struct-backed Chat Completions content arrays, including nested image_url.url and optional detail.
• Keep legacy string messages and map-backed text-part collapsing compatible.
• Validate roles plus structured text/image_url shapes with value-free errors.
• Route malformed, empty, and null-part rows into errorCol, skip HTTP, preserve existing errors, and restore the original messages column.
• Reject messagesCol collisions with outputCol or errorCol instead of silently discarding public results.
• Avoid allocating the full serialized payload during validation.
• Add local Scala and Python transformer-path coverage, including exact mixed multimodal wire JSON and negative no-HTTP behavior.

Tests

• JDK 11 cognitive/Test/compile: passed
• JDK 11 cognitive main/test Scalastyle: passed, 0 findings
• JDK 11 ResponseFormatOrderSuite + OpenAIChatCompletionMultimodalSuite + OpenAIV1EndpointSuite: 37 passed
• JDK 11 cognitive/codegen: passed; generated wrapper syntax validated
• Targeted generated Python pytest against the locally built JVM jars: 1 passed
• Black 22.3 project check: passed, 190 files unchanged
• Exact Azure release replay onto spark4.1 tip 1d1d0dadf8574c1558018e6ee30d1851593ecf2a: applied cleanly
• JDK 17 / Scala 2.13 / Spark 4.1 full test:compile: passed
• Spark 4.1 targeted OpenAI suites: 37 passed
• No live Azure resources were created or used locally

Compatibility

• No existing public JVM signatures or response schemas changed.
• Existing content: String, null messages, and map-backed text-part requests retain established behavior.
• Struct-backed multimodal parts survive the real transformer/request path.
• Invalid structured rows produce per-row errors and no outbound request.
• Errors expose structural locations only, never prompts, URLs, keys, or credentials.
• The patch is source-compatible with Scala 2.12/Spark 3.5 and Scala 2.13/Spark 4.1.

Verified revisions

• Upstream master: f7a1dc50d09d400d279d08bf69a1fac322896748
• PR head: a34f595aab7c814cc965abd96ef2a86b92c0ab8b

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Rana Singh (ranadeepsingh)]

/azp run

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR fixes multimodal OpenAI Chat Completions request serialization in the cognitive module by preserving structured content arrays (e.g., image_url) instead of flattening them to text, ensuring image inputs survive to the wire format expected by the OpenAI API.

Changes:

• Add structured-content validation for Chat Completions (text and image_url parts) and preserve content arrays during request serialization.
• Recursively encode Spark struct/array/map values into JSON-safe nested Scala structures for correct request JSON.
• Expand offline regression tests to cover legacy string content, structured image parts, mixed payload JSON, and invalid shapes.

Show a summary per file

File	Description
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIChatCompletion.scala	Stops flattening composite content, adds structured content validation before serialization.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAI.scala	Adds recursive Spark value encoding to preserve nested struct/map content shapes in JSON.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAICoreOfflineSuite.scala	Adds deterministic offline tests for multimodal content preservation and validation behavior.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

CI note: the spark4.1 release compatibility job failed before compilation while replaying the PR patch because that release branch predates OpenAICoreOfflineSuite.scala (does not exist in index) and its OpenAI.scala has Scala 2.13 collection changes. Main-branch GitHub compile/style validation passes, and the Azure OpenAI unit job is still queued. This is release-branch patch-replay divergence rather than a product/test failure.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAI.scala:495

• encodeMessagesToMap currently calls encodeSparkValue(row.get(contentField), arrayType) for array-backed content. If the content field is null (allowed by some inferred schemas), this throws Expected array content but found null before OpenAIChatCompletion's location-based validation runs, yielding inconsistent / less actionable error messages compared to the string-content null case.

  private[openai] def encodeMessagesToMap(messages: Seq[Row]): Seq[Map[String, Any]] = {
    messages.map { row =>
      val role = row.getAs[String]("role")
      val contentField = row.schema.fieldIndex("content")
      val contentType = row.schema.fields(contentField).dataType

      val content = contentType match {
        case StringType =>
          // OpenAIMessage: content is a String
          row.getAs[String]("content")
        case arrayType@ArrayType(elementType, _) =>
          elementType match {
            case _: StructType | _: MapType =>
              encodeSparkValue(row.get(contentField), arrayType)
            case other =>

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAI.scala:23

• The updated messagesCol param doc still doesn’t describe the actual structured content-part schema (required type plus text or image_url.url/detail). Since this string is used for generated setter documentation, it should be explicit enough for users to construct the correct Spark schema for multimodal chat messages.

  val messagesCol: Param[String] = new Param[String](
    this, "messagesCol", "The column messages to generate chat completions for in chat format. " +
      "The column should have type Array(Struct(role: String, content: String or Array of content parts)).")

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Final CI update for 90526d7: Azure UnitTests openai passed all 166 tests (0 failed), and GitHub compile/style, Python style, CodeQL, dependency review, dead-link, title, CLA, and WIP checks pass. The spark4.1 compatibility job remains the only relevant failure: it stops during patch replay before compilation because that release branch has a conflicting OpenAI.scala and does not contain OpenAICoreOfflineSuite.scala; no product tests run in that job. This is the same release-branch divergence documented above.

[Rana Singh (ranadeepsingh)]

Compatibility fix pushed in 948e737. I reproduced the pipeline's aggregate git diff + prerequisite check + git apply --3way --index mechanism locally. The final three-path patch applies cleanly to spark4.1, full JDK 17/Scala 2.13 est:compile passes, and the shared offline suite passes 28/28 on both master and spark4.1. The implementation now localizes struct-backed multimodal encoding to OpenAIChatCompletion, preserves legacy map-backed text collapsing, and keeps tests in a suite present on both branches. Final adversarial review findings for role validation and output/scratch-column collisions are also fixed with row-level no-HTTP regressions.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Final verification for a34f595aab7c814cc965abd96ef2a86b92c0ab8b against upstream master f7a1dc50d09d400d279d08bf69a1fac322896748:

• Azure build 231387843 and all GitHub checks completed green: 77/77, 0 pending, 0 failed.
• Spark 3.5/JDK 11 targeted OpenAI suites: 37/37; compile, main/test Scalastyle, codegen, generated Python syntax, Black 22.3, and targeted Python pytest passed.
• Exact Spark 4.1 replay: full test:compile and targeted OpenAI suites 37/37 passed.
• All 4 review threads are resolved; latest Copilot review reported 0 new comments.
• Worktree is clean and remote head matches the verified SHA.

GitHub still reports REVIEW_REQUIRED; the existing request to Brendan Walsh (@BrendanWalsh) is the only remaining merge gate.