Add private Splunk MCP deployment templates

[Cristhian (cristhianu)]

Summary

• This adds a Bicep and Terraform templates for private Splunk MCP connectivity in cross-region and same-region topologies. It uses Splunk connector with Splunk's MCP which uses the VNet for traffic.
• It deploys Splunk without a public IP using delegated SRE Agent subnets, private DNS, NSGs, NAT, and Global VNet Peering only where required
• It adds Bash and PowerShell workflows for deployment, Splunk configuration, SRE Agent attachment, token minting, validation, and cleanup
• It scrubs and verifies deletion of token-bearing Managed Run Commands before displaying minted tokens
• It documents independent testing, trusted HTTPS versus the optional HTTP test mode, and cleanup procedures

Architecture

Cross-region

flowchart LR
    subgraph AgentRegion[Agent region]
        Agent[SRE Agent]
        AgentSubnet[Delegated agent subnet<br/>Microsoft.App/environments]
        AgentVNet[Agent VNet]
        Agent --> AgentSubnet
        AgentSubnet --- AgentVNet
    end

    subgraph SplunkRegion[Splunk region]
        SplunkVNet[Splunk VNet]
        SplunkSubnet[Private Splunk subnet]
        SplunkVM[Splunk VM<br/>No public IP]
        NAT[NAT Gateway<br/>Outbound bootstrap only]
        SplunkVNet --- SplunkSubnet
        SplunkSubnet --> SplunkVM
        SplunkSubnet -->|Outbound package and image access| NAT
    end

    AgentVNet <-->|Global VNet Peering<br/>Private MCP traffic| SplunkVNet
    DNS[Private DNS zone<br/>Splunk hostname to private IP]
    DNS -. VNet link .-> AgentVNet
    DNS -. VNet link .-> SplunkVNet

Loading

The SRE Agent reaches the private Splunk MCP endpoint across Global VNet Peering; private DNS is linked to both VNets.

Same-region

flowchart LR
    subgraph Region[Shared Azure region]
        subgraph SharedVNet[Shared VNet]
            Agent[SRE Agent]
            AgentSubnet[Delegated agent subnet<br/>Microsoft.App/environments]
            SplunkSubnet[Private Splunk subnet]
            SplunkVM[Splunk VM<br/>No public IP]
            Agent --> AgentSubnet
            AgentSubnet -->|Private MCP traffic| SplunkSubnet
            SplunkSubnet --> SplunkVM
        end
        NAT[NAT Gateway<br/>Outbound bootstrap only]
        SplunkSubnet -->|Outbound package and image access| NAT
    end

    DNS[Private DNS zone<br/>Splunk hostname to private IP]
    DNS -. VNet link .-> SharedVNet

Loading

The SRE Agent and Splunk use isolated subnets in one VNet, so no VNet peering is required.

Validation

• cross-region and same-region targeted dry-run tests passed
• Bicep compilation passed for both topologies
• Terraform init, fmt, and validate passed for both topologies
• Bash syntax and PowerShell parser/runtime checks passed
• Bicep and Terraform deployments completed for both topology variants
• connector discovery and read-only Splunk tool calls completed over private routing
• Bash and PowerShell token minting verified Run Command output scrubbing and resource deletion
• git diff --check passed

The aggregate repository test runner was also attempted; unrelated existing tests require jq and have Windows runner failures outside these examples. The two new targeted suites pass.

[Cristhian (cristhianu)]

The failing check is repository-level, not caused by the Splunk templates. The default-branch workflow passes about 216 KB of changed-file JSON as one jq argument, which exceeds Linux's per-argument limit; the configured webhook also targets a direct SRE Agent endpoint instead of the required Logic App callback. I opened #350 with the tested file-backed payload fix and a fail-fast webhook diagnostic. Because this workflow uses pull_request_target, #350 must land on main and the repository secret must be updated to the Logic App callback before #349 can pass; then synchronizing this branch will rerun the corrected check.