Skip to content

Fix input validation issues and update multiple-choice options handling#2501

Merged
apsonawane merged 7 commits into
mainfrom
asonawane/input
Jun 6, 2026
Merged

Fix input validation issues and update multiple-choice options handling#2501
apsonawane merged 7 commits into
mainfrom
asonawane/input

Conversation

@apsonawane
Copy link
Copy Markdown
Contributor

@apsonawane apsonawane commented Jun 5, 2026

This pull request introduces improvements to the handling of multiple-choice questions and options extraction in the data preprocessing and evaluation pipeline, as well as a minor enhancement to ONNX model input handling. The main changes include switching from numeric to letter-based multiple-choice options, extracting the selected letter during evaluation, and supporting an additional input name in ONNX graph surgeries.

Multiple-choice options handling:

  • Updated pre_process_data.py to format multiple-choice options using letters (A, B, C, ...) instead of numbers, and added a new field extract_option_letter to indicate when letter extraction is needed. [1] [2]
  • Modified olive_evaluator.py to extract the selected letter from model responses for multiple-choice questions, in addition to the previous digit extraction. This ensures compatibility with the new letter-based formatting. [1] [2] [3]

ONNX model input handling:

  • Enhanced the TieWordEmbeddings class in graph_surgeries.py to support both input_embeds and inputs_embeds as valid input names, improving compatibility with different model variants.

Copilot AI review requested due to automatic review settings June 5, 2026 00:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates the ONNX TieWordEmbeddings graph surgery to be more compatible with exported model interfaces by recognizing "inputs_embeds" as an additional valid input name (alongside "input_ids" and "input_embeds").

Changes:

  • Extend TieWordEmbeddings.__call__ input name detection to include "inputs_embeds".
  • Update the inline comment to reflect the supported input-name variants.

Comment thread olive/passes/onnx/graph_surgeries.py
Comment thread olive/evaluator/olive_evaluator.py Fixed
Comment thread olive/evaluator/olive_evaluator.py Fixed
@apsonawane apsonawane changed the title Fix input validation issues Fix input validation issues and update multiple-choice options handling Jun 5, 2026
@apsonawane apsonawane enabled auto-merge (squash) June 5, 2026 21:42
@apsonawane apsonawane requested a review from xiaoyu-work June 5, 2026 21:51
@apsonawane apsonawane merged commit 6da982b into main Jun 6, 2026
13 checks passed
@apsonawane apsonawane deleted the asonawane/input branch June 6, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants