[Python] Return every group from Regex.find_all with Regex.ALL

[anishmehta24]

Regex.find_all(..., group=Regex.ALL) only returns group 0 and group 1, though the docstring says it returns all groups:

beam.Create(['a=1 b=2']) | beam.Regex.find_all(r'(\w)=(\d)', beam.Regex.ALL)
# [[('a=1', 'a'), ('b=2', 'b')]]   group 2 is dropped

A pattern with no groups raises IndexError: tuple index out of range. Each match now gives (m.group(), *m.groups()), so single-group output is unchanged with the default outputEmpty=True; with outputEmpty=False a match is now dropped only when the whole match is empty, like group=0 (noted in CHANGES.md). Added a test for two groups and for no groups; util_test.py regex tests pass.

Written with help from Claude.

---

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

[github-actions]

Assigning reviewers:

R: @damccorm for label python.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

With any(m.groups() or (m.group(), )), if a pattern has multiple capturing groups and one group is empty while another is non-empty (for example, r'(\w)=(\d?)' matching 'c=' in your test, where m.groups() is ('c', '')), outputEmpty=False will still include ('c=', 'c', '') because 'c' is non-empty.

Compare this with the existing single-group CheckAllNonEmptyGroups test ('a(b*)' on 'abb ax abbb' with outputEmpty=False), where 'ax' matches 'a' with an empty group 1 (('a', '')) and is filtered out. With any(...), simply wrapping 'a' in a capturing group ('(a)(b*)') would cause ('a', 'a', '') to be kept despite group 2 being empty.

Should this use all(m.groups() or (m.group(), )) instead so outputEmpty=False filters out matches where any captured group is empty (while behaving identically for 0-group and 1-group patterns)?

Suggested change

	yield [(m.group(), *m.groups()) for m in matches
	if outputEmpty or any(m.groups() or (m.group(), ))]
	yield [(m.group(), *m.groups()) for m in matches
	if outputEmpty or all(m.groups() or (m.group(), ))]

[anishmehta24]

Applied, thanks. all keeps the single-group behaviour.

[damccorm]

Sorry for the back-and-forth here! Thinking this through more, I think we should actually simplify this to just gate on if outputEmpty or m.group() (matching the group=0 behavior) and clearly document it in the docstring:

      if group == Regex.ALL:
        yield [(m.group(), *m.groups()) for m in matches
               if outputEmpty or m.group()]

Otherwise, with all (or any), you can go from having a match when group=0 (outputEmpty=False) to losing that match when group=Regex.ALL (outputEmpty=False) whenever an optional capturing group matches '' (like r'(\w)=(\d?)' on 'c=' or 'a(b*)' on 'ax'), which is pretty surprising behavior.

Note that this will require updating CheckAllNonEmptyGroups in util_test.py (since 'a' is a non-empty full match on 'ax') and is a breaking change, so could you also add a note to CHANGES.md (under the Breaking Changes section) mentioning that Regex.find_all with Regex.ALL now returns all groups and checks the full match for outputEmpty=False?

[anishmehta24]

Done: it gates on the full match now, the docstring says so, CheckAllNonEmptyGroups keeps ('a', ''), and there's a Breaking Changes entry under 2.78.0.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.06%. Comparing base (76f6e6d) to head (7fad717).
⚠️ Report is 110 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #40399       +/-   ##
=============================================
+ Coverage     56.16%   59.06%    +2.89%     
- Complexity     2288    15616    +13328     
=============================================
  Files          1124     2798     +1674     
  Lines        178336   280831   +102495     
  Branches       1489    12470    +10981     
=============================================
+ Hits         100170   165868    +65698     
- Misses        75645   108522    +32877     
- Partials       2521     6441     +3920     

Flag	Coverage Δ
python	79.68% <100.00%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[damccorm]

Thanks, I think we just need a couple more minor comment changes for readability

[anishmehta24]

Applied all three. The two red checks look unrelated: the Coverage job hit a Vertex AI 429 quota error in vertex_ai_it_test, and the Integration job failed on a Dataflow RPC in streaming_wordcount_it_test.

[anishmehta24]

Merged master to fix the CHANGES.md conflict. Both Breaking Changes entries are kept.