Skip to content

[YAML] Add SplunkIO write SchemaTransform and WriteToSplunk - #40481

Open
schizophrenicmaniac wants to merge 2 commits into
apache:masterfrom
schizophrenicmaniac:fix/40470-feature-request-normalize-splunkio-for
Open

schizophrenicmaniac wants to merge 2 commits into
apache:masterfrom
schizophrenicmaniac:fix/40470-feature-request-normalize-splunkio-for

Conversation

@schizophrenicmaniac

Copy link
Copy Markdown
Contributor

Adds a SchemaTransform for SplunkIO writes and exposes it in Beam YAML as WriteToSplunk. It follows the same approach as the Datadog one (#38362). SplunkIO in Java only supports writes, so there is no ReadFromSplunk here.

Each input row becomes a SplunkEvent. event (string) is required. time (int64), host, source, sourcetype, index and fields are optional. fields can be a nested row or a JSON object string, and other columns are ignored. The config covers everything SplunkIO.Write supports (url, token, batch_count, parallelism, disable_certificate_validation, root_ca_certificate_path, enable_batch_logs, enable_gzip_http_compression) plus error_handling. The error output has the same shape as Datadog's: rows that can't be converted come back with failed_row and a 400, and HEC write failures come back with the HTTP status and the payload. Without error_handling, a failed write fails the pipeline.

While testing this I found that SplunkWriteError loses all of its values once it goes through a coder. It has @DefaultSchema(AutoValueSchema.class), but its accessors aren't get-prefixed, so the inferred schema has no fields and every error decodes as null/null/null. SplunkEventWriterTest doesn't catch this because containsInAnyOrder decodes both sides with the same coder. I added a SplunkWriteErrorCoder (same idea as DatadogWriteErrorCoder) and set it on the output of SplunkIO.Write. This changes the output coder of SplunkIO.Write, so I listed it under bugfixes in CHANGES.md.

Changes:

  • SplunkWriteSchemaTransformProvider / SplunkWriteSchemaTransformConfiguration, registered in the IO expansion service
  • WriteToSplunk in standard_io.yaml and the matching entry in standard_external_transforms.yaml
  • SplunkWriteErrorCoder, used by SplunkIO.Write
  • Java tests against a mock HEC server, and a YAML integration test with a fake HEC fixture (tests/splunk.yaml)

Fixes #40470

@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 105 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.75%. Comparing base (76f6e6d) to head (3e1715b).
⚠️ Report is 103 commits behind head on master.

Files with missing lines Patch % Lines
...n/apache_beam/yaml/test_utils/splunk_test_utils.py 0.00% 99 Missing ⚠️
sdks/python/apache_beam/yaml/integration_tests.py 0.00% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40481      +/-   ##
============================================
+ Coverage     56.16%   58.75%   +2.58%     
- Complexity     2288    13917   +11629     
============================================
  Files          1124     2586    +1462     
  Lines        178336   271824   +93488     
  Branches       1489    11243    +9754     
============================================
+ Hits         100170   159708   +59538     
- Misses        75645   106110   +30465     
- Partials       2521     6006    +3485     
Flag Coverage Δ
python 79.59% <0.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.
R: @kennknowles for label java.
R: @derrickaw for label yaml.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Normalize splunkIO for yaml

1 participant