Skip to content

[Python] Configure object detection benchmark timeout - #39949

Draft
bvolpato wants to merge 2 commits into
apache:masterfrom
bvolpato:bvolpato/fix-inference-benchmark-timeout
Draft

bvolpato wants to merge 2 commits into
apache:masterfrom
bvolpato:bvolpato/fix-inference-benchmark-timeout

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Addresses #38782.

The object-detection example cancels its Dataflow job after a hard-coded 30-minute wait, even though the benchmark workflow allows 180 minutes per step. Recent benchmark runs reached that cancellation after about 1,803 seconds.

Add --timeout_ms to the example, retaining the 30-minute default, and set the shared object-detection benchmark options to 60 minutes. That override applies to all four CPU/GPU and batch/streaming combinations.

This PR remains draft pending an end-to-end Dataflow run. The fixed-batch configuration fix in #39953 is now upstream and may change runtime. The timeout increase should only merge if the corrected benchmark still needs more than 30 minutes.

The September 2 benchmark includes #39953 and still cancels CPU object detection after 1,804 seconds, but it reports repeated SDK harness disconnections well before that deadline, described in the log as likely process crashes. This does not establish how long a healthy run needs or show that 60 minutes would help. The September 3 run failed during setup and skipped the benchmarks. Worker-failure diagnosis and useful end-to-end timing evidence are still needed before merging.

Validation:

  • Isolated parser and orchestration checks pass for the default, an override, streaming execution, and cleanup after a wait failure. These verify timeout forwarding and cancellation without running a model or Dataflow job.
  • Python compilation, YAPF, Ruff, and git diff --check pass.

End-to-end validation requires the Dataflow benchmark infrastructure.

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

The failed Python 3.10 wheel job completed the build and auditwheel repair, then timed out during cibuildwheel's Docker cleanup. I did not find a build failure related to this change.

Could someone with rerun access retry that job? My fork account cannot rerun upstream Actions. Thanks!

@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.19%. Comparing base (76f6e6d) to head (671336d).
⚠️ Report is 101 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #39949      +/-   ##
============================================
+ Coverage     56.16%   56.19%   +0.02%     
  Complexity     2288     2288              
============================================
  Files          1124     1124              
  Lines        178336   178343       +7     
  Branches       1489     1489              
============================================
+ Hits         100170   100220      +50     
+ Misses        75645    75602      -43     
  Partials       2521     2521              
Flag Coverage Δ
python 79.61% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bvolpato
bvolpato force-pushed the bvolpato/fix-inference-benchmark-timeout branch 2 times, most recently from 36dc3f2 to 671336d Compare October 8, 2026 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant