feat: opt-in local connector execution with JIT secret hydration - #46
Draft
Johnny Schmidt (johnny-schmidt) wants to merge 4 commits into
Draft
Johnny Schmidt (johnny-schmidt) wants to merge 4 commits into
Johnny Schmidt (johnny-schmidt) wants to merge 4 commits into
Conversation
Add an opt-in local execution path to `connectors execute`. Hosted execution remains the zero-config default. In local mode the CLI fetches an unhydrated execute bundle from Sonar via `POST .../execute/prepare`, hydrates `secret_coordinate::` values from a customer-owned AWS Secrets Manager in memory, and performs the connector HTTP request from the CLI process over an isolated hardened transport. - Global root flags --execution-mode/--aws-profile/--aws-region with deterministic precedence; SDK-compatible AWS_SECRET_MANAGER_* env set. - internal/secrets + internal/secrets/aws: provider-neutral hydration walker and AWS Secrets Manager provider (strict SecretString, cached-SSO login remediation, redacted typed errors). - internal/localexec: native Go compiler from bundle YAML to an immutable request plan (OpenAPI-3 + x-airbyte-* extensions, restricted JSONPath), auth mapping, isolated HTTPS-only transport (no Airbyte headers, redirect /body/timeout bounds, safe-method-only retries), and response shaping. - connectors execute branches on runtime mode; fail-closed (no hosted fallback), validation before hydration, bundle removed from output. - Stable local error taxonomy mapped to exit codes; docs + skill guidance. Fail-closed by construction: no secret-bearing flags, connector-origin traffic never carries Airbyte credentials, hydrated secrets are invocation-local and never persisted. AWS SDK v2 pinned to Go-1.22-compatible versions; yaml.v3 added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nvelope The prepare endpoint serializes the backend's shared ConnectorExecuteResponse, which always includes `result` (null on the direct/local path) and `context_store_bundle` (added for context-store local execution). The strict DecodeEnvelope (DisallowUnknownFields) rejected both as unknown fields, failing every real local execution with "envelope is not valid JSON". Model both as ignored fields so the decoder accepts the real response while still rejecting genuinely unexpected fields.
…ions Real connector definitions (e.g. Stripe) emit a boolean `required: true` on individual property schemas, alongside the object-level `required: [names]` list. Unmarshaling a bool into Schema.Required ([]string) failed the whole parse with "definition_yaml is not valid YAML". Decode `required` through a tolerant type that yields the name list for a sequence and an empty list for any non-sequence value.
Johnny Schmidt (johnny-schmidt)
requested a review
from Cam Kennedy (cjkenned)
August 28, 2026 20:43
Check the discarded error returns golangci-lint's errcheck flags: the httptest w.Write calls in executor_test.go and the assertType helper returns in the AWS provider tests (the *secrets.Error return was dropped at soft-miss call sites). Bump skills/airbyte-agent/SKILL.md to v0.1.3 for the local- execution docs this PR adds, satisfying the skill-version-bump gate. Claude-Session: https://claude.ai/code/session_01Hyh5zmKZ9p8Z5HmykM3NZU
Johnny Schmidt (johnny-schmidt)
force-pushed
the
feat/local-execution-jit-secret-hydration
branch
from
August 28, 2026 20:53
66b4898 to
3094cc8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds an opt-in local execution path to
airbyte-agent connectors execute. Hosted execution stays the zero-config default and is unchanged. In local mode the CLI:POST .../execute/prepare(same body as hosted/execute, incl.intent) → unhydrated execute bundlesecret_coordinate::values from a customer-owned AWS Secrets Manager in memoryWhat's here
cmd/,internal/config--execution-mode/--aws-profile/--aws-regionflags + deterministic precedence; SDK-compatibleAWS_SECRET_MANAGER_*env setinternal/secrets(+/aws)SecretString, cached-SSO login remediation, redacted typed errors)internal/localexecx-airbyte-*, restricted JSONPath), auth mapping, HTTPS-only transport, response shaping, orchestrating executorinternal/resourcesconnectors executebranches on runtime mode; fail-closed (no hosted fallback), validate-before-hydrate,bundleremoved from outputconnectors-executereference (IAM least-privilege, precedence, fail-closed model)Security properties
Verification
gofmtclean ·go vet ./...·go build -mod=readonly ./...·go test ./...·-raceon localexec/secrets/resources ·go generateidempotent ·go 1.22.0directive preserved.Draft because these need reconciliation before merge/release:
api/app_public.jsonwas hand-edited, NOT synced from the pinned Sonar revision (ff7c970…) — Sonar was unreachable.ConnectorExecuteResponsegot optionalbundle/warningand non-requiredresult;extracted_gen.goregenerated. Reconcile against a real upstream sync.config v1.29.14, etc.).go mod tidybumps thegodirective to 1.24 — don't run it without intent.🤖 Generated with Claude Code