Skip to content

test: make Tool/Agent serialization assertions version-agnostic for Haystack 2.x/3.x#3533

Merged
julian-risch merged 5 commits into
mainfrom
test/tool-serde-2x-3x-agnostic
Jul 3, 2026
Merged

test: make Tool/Agent serialization assertions version-agnostic for Haystack 2.x/3.x#3533
julian-risch merged 5 commits into
mainfrom
test/tool-serde-2x-3x-agnostic

Conversation

@julian-risch

@julian-risch julian-risch commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Related Issues

  • Part of deepset-ai/haystack-private#446

Serialization tests in five integrations compared to_dict() output against hard-coded expected dicts that pin the exact Tool/Toolset/Agent serialization format of one haystack-ai version. Haystack 3.0 adds fields (e.g. Tool.to_dict() now includes async_function; Agent gained tool_concurrency_limit, hooks, tool_streaming_callback_passthrough), which broke these tests. This PR makes them version-agnostic so they pass on both 2.x and the current v3 branch:

Proposed Changes:

  • anthropic, huggingface_api, transformers, watsonx: the to_dict tests no longer pin the haystack-owned tools serialization format at all. They exclude the tools entries from the pinned-dict comparison and instead do a real from_dict(component.to_dict()) round-trip, asserting the recovered tools equal the originals.
  • aimlapi, cometapi, meta_llama, mistral, openrouter, orcarouter, togetherai: the same for their test_serde_in_pipeline tests — the tools entries are popped from the pinned pipeline dict; tool fidelity is covered by the Pipeline.loads(pipeline.dumps()) round-trip (Pipeline.__eq__ compares serialized forms) plus explicit attribute checks on the loaded generator.
  • github (test_pipeline_serialization): the test pins the full pipeline dict including Agent and OpenAIChatGenerator init parameters, which belong to haystack-ai and change between versions. It now copies any init parameters the installed haystack-ai adds beyond the expected baseline before comparing, replacing the two existing single-key compat shims (http_client_kwargs/tool_invoker_kwargs pops and the confirmation_strategies/required_variables/user_prompt loop). The test keeps pinning what the integration owns: the GitHubFileEditorTool serialization. It also now asserts Pipeline.from_dict(pipeline.to_dict()) == pipeline.
  • github: also changed import unittest to import unittest.mock — the test uses unittest.mock.ANY and only worked because haystack-ai 2.x happened to import unittest.mock transitively; with v3 installed it fails with AttributeError: module 'unittest' has no attribute 'mock'.

How did you test it?

Notes for the reviewer

  • Reviewing this uncovered that cometapi's test_serde_in_pipeline never deserialized anything (it compared the original generator to itself), which was hiding a real bug: CometAPIChatGenerator could not be deserialized in a pipeline at all. That fix, together with the missing round-trip, was extracted into fix(cometapi): make CometAPIChatGenerator deserializable in pipelines #3542 (now merged); main has been merged back into this branch, so cometapi's test_serde_in_pipeline here combines both changes.

Checklist

🤖 Generated with Claude Code

@julian-risch @claude
test: make Tool/Agent serialization assertions version-agnostic
a3770ab 
Haystack 3.0 extends the serialized format of Tool (async_function) and
Agent (tool_concurrency_limit, hooks, ...), breaking tests that compare
to_dict() output against hard-coded dicts. Build the expected tool
entries with tool.to_dict()/toolset.to_dict() at runtime and, in the
github pipeline test, accept init parameters added by the installed
haystack-ai version. Also replaces the per-field hasattr/pop compat
shims these tests had accumulated for earlier format changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (github)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Coverage report (watsonx)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Coverage report (huggingface_api)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Coverage report (anthropic)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Coverage report (transformers)

This PR does not seem to contain any modification to coverable code.

This was referenced Jul 2, 2026
Open
Merged
@julian-risch @claude
test: make remaining serde-in-pipeline assertions version-agnostic
8629c75 
Same fix as the first commit for seven more chat-generator suites whose
serde tests were masked by the ToolInvoker collection error when this PR
was opened: build the expected tools entry with tool.to_dict() and drop
the per-field hasattr compat shims.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (orcarouter)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Coverage report (cometapi)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (openrouter)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (mistral)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (aimlapi)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (meta_llama)

This PR does not seem to contain any modification to coverable code.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Coverage report (togetherai)

This PR does not seem to contain any modification to coverable code.

julian-risch added a commit that referenced this pull request Jul 2, 2026
@julian-risch @claude
test: trust test modules in seven more chat-generator suites
7d34f94 
Their serde-in-pipeline tests round-trip tools whose functions are
defined in the test modules, so Pipeline.loads hits the Haystack 3.0
trusted-module allowlist once #3533 fixes the dict comparison and #3535
lets the suites collect.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
julian-risch
julian-risch commented Jul 2, 2026
View reviewed changes
Comment thread integrations/aimlapi/tests/test_aimlapi_chat_generator.py
if not hasattr(pipeline, "_connection_type_validation"):
expected_dict.pop("connection_type_validation")

# add outputs_to_string, inputs_from_state and outputs_to_state tool parameters for compatibility with

@julian-risch julian-risch Jul 2, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no longer needed thanks to "tools": [tool.to_dict()],

@julian-risch julian-risch marked this pull request as ready for review July 2, 2026 15:04
@julian-risch julian-risch requested a review from a team as a code owner July 2, 2026 15:04
@julian-risch julian-risch requested review from sjrl and removed request for a team July 2, 2026 15:04
sjrl
sjrl reviewed Jul 2, 2026
View reviewed changes
Comment thread integrations/aimlapi/tests/test_aimlapi_chat_generator.py Outdated
}
],
# serialize the tool with the installed haystack-ai version so the expected fields match it
"tools": [tool.to_dict()],

@sjrl sjrl Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principal this is okay but it is kind of asserting that pipeline.to_dict equals pipeline.to_dict which isn't necessarily that helpful.

Maybe converting these tests to do Pipeline.from_dict(pipeline.to_dict)) and asserting the newly loaded pipeline equals the old one?

@julian-risch @claude
test: verify tool serialization via round-trips instead of self-compa…
ba4feb9 
…rison

Review feedback: comparing to_dict output against tool.to_dict() partly
asserts to_dict equals itself. Exclude the haystack-ai-owned tools entry
from the pinned-dict comparisons and rely on round-trips instead: the
serde-in-pipeline tests already dump and load the pipeline and assert
equality, and the plain to_dict tests now deserialize the serialized
component and compare the recovered tools with the originals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the type:documentation Improvements or additions to documentation label Jul 3, 2026
@julian-risch julian-risch mentioned this pull request Jul 3, 2026
Merged
@julian-risch @claude
test(github): assert the deserialized pipeline equals the original
0621a99 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@julian-risch julian-risch force-pushed the test/tool-serde-2x-3x-agnostic branch from 1dd8e79 to 0621a99 Compare July 3, 2026 07:27
julian-risch
julian-risch commented Jul 3, 2026
View reviewed changes
Comment thread integrations/aimlapi/tests/test_aimlapi_chat_generator.py
Comment on lines 506 to 508
pipeline_yaml = pipeline.dumps()
new_pipeline = Pipeline.loads(pipeline_yaml)
assert new_pipeline == pipeline

@julian-risch julian-risch Jul 3, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the roundtrip test

sjrl
sjrl approved these changes Jul 3, 2026
View reviewed changes

@sjrl sjrl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@julian-risch julian-risch merged commit b7e2a38 into main Jul 3, 2026
96 of 104 checks passed
@julian-risch julian-risch deleted the test/tool-serde-2x-3x-agnostic branch July 3, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sjrl sjrl sjrl approved these changes

Assignees

No one assigned

Labels

integration:aimlapi integration:anthropic integration:cometapi integration:github integration:huggingface-api integration:meta-llama integration:mistral integration:openrouter integration:orcarouter integration:togetherai integration:transformers integration:watsonx type:documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@julian-risch @sjrl