feat(parser_http_server): HTTP+JSON pivot for TVC public ingress

[pepe-anchor]

Why am I making this PR?

Turnkey's TVC public ingress is HTTP only: Cloudflare in front of app-<uuid>.turnkey.cloud rejects gRPC with 403 (verified 2026-05-16). Moving the parse path onto the pivot needs a binary that speaks HTTP and JSON natively, instead of gRPC over vsock. This is the keystone of the PRS-581 stack; two PRs build directly on it (in-enclave X-Stamp auth, real NSM boot proof).

What am I changing?

• New parser_http_server binary opens /health, /v1/parse, /v2/parse (v2 identical to v1 for now); no payment enforcement, no auth, no proto change.
• Handlers take raw request bytes instead of deserializing, so a later X-Stamp check can verify the exact signed bytes.
• Every response, success and error (including 413/404/405), carries a bootProof; the attestation doc stays empty until NSM support lands.
• Manifest fields are re-encoded from JSON to borsh so the Go verifier can read them.
• Parsing runs off the async reactor so it can't starve the health check.
• Adds the stagex CI legs to build and deploy the new image; drops the unrelated parser_grpc_server leg that can't boot.

What is the Linear ticket?

PRS-581

What are the rollback steps?

1. Revert the commits.

This adds a new binary nothing calls yet and changes no existing behavior, so a revert is inert. Nothing is deployed by merging it.

Is this change backwards compatible?

Yes. Only a new binary and a new CI target are added; no proto, wire-format, or exported API changes to existing crates.

Does this require cross-team/service coordination?

No. Self-contained to this repo; the new binary is not wired into any deployment yet.

How do I know it works as designed? Which tests exercise this code?

cargo test -p parser_http_server        -> 7 passed, 0 failed
make -C src test                        -> 1614 passed, 0 failed (all suites)
cargo clippy --all-targets -D warnings  -> clean

CI, Sonar, and Copilot gates are green as of this rebase.

Rebase onto main after #446

#446 merged, so this branch is rebased onto current main and the merge commit is flattened away. Three commits now, no merge.

The rebase itself was textually clean: this branch never edited host_primitives/src/turnkey.rs or parser/gateway/src/main.rs, it only imports from them.

What it did leave behind was a semantic mismatch. The pivot was written against host_primitives::turnkey as of the fork point, which had error_response but not yet success_response (that landed in 87816dff, after the fork). So after the rebase handle_parse was still assembling the success wrapper field by field, which is exactly the second copy of the wire contract #446 existed to delete. Fixed in its own commit: the success path now goes through success_response, matching what parser_gateway does.

No wire change. success_response fills the same three fields and sets error: None, so the emitted JSON is byte-identical, and bootProof stays camelCase with exactly its six camelCase keys. Three separate tests pin that and all still pass (host_primitives::turnkey::boot_proof_wire_shape_is_exactly_six_camel_case_keys, parser_http_server's static_boot_proof_has_the_six_keys_and_a_real_ephemeral_pubkey, and the http_server integration test's key-set assertion on both the success and the 400 response).

Review pass, four more commits

A pre-push review round on top of the rebase. Nothing here changes the wire
shape or the three seams the stacked PRs plug into.

Every response now really does carry bootProof. The PR claimed that and
was wrong for three statuses. axum rejects an oversized body, an unmatched
route and a disallowed method before any handler runs, so 413, 404 and 405
went out as bare axum errors with no envelope. 413 is fixed with a middleware,
since DefaultBodyLimit rejects while reading the body and no handler sees it.
404 and 405 deliberately do not share that middleware: handle_parse
legitimately returns 404 itself for Code::NotFound with the parser's real
message, and a middleware keyed on status alone cannot tell that apart from an
unmatched route, so it would replace a genuine parse failure with "not found".
Those two go through Router::fallback and method_not_allowed_fallback,
which axum invokes only when no handler ran. The integration test asserts the
envelope on all four error paths now.

The parse call no longer blocks the reactor. parse is CPU bound, so
running it on the async task pinned a Tokio worker per concurrent request and
would starve the health check Turnkey polls on a 1 to 2 vCPU replica.
block_in_place matches what parser_app's Processor already does around
this same call on the vsock path.

Error messages stopped reflecting request bodies. serde_json's Display
for a type mismatch embeds the offending value, so a malformed request echoed
up to the full body back to an unauthenticated caller. The client gets a fixed
message; the detail goes to stderr, along with new logging on three internal
error paths that previously failed silently.

The TODO(#231) clippy exemptions are gone, not deferred: the ephemeral
key load propagates its error and SIGTERM registration falls back to ctrl-c,
so the crate-level unwrap/expect/panic allows could be deleted outright.

NEAR is compiled in. The default feature set said it mirrored
parser_app's and did not, so a NEAR request would have succeeded against
parser_app and failed against the pivot. That was a silent divergence, not a
deliberate trim.

Two correctness fixes to the notes this branch generates. The release notes
told an operator that /v2 is TVC-enforced when GATEWAY_SIGNING_PUBKEY_HEX
is set and that parser_gateway performs x402 verify and settle. Neither is
true yet: that variable is read nowhere in src, /v2 routes to the same open
handler as /v1, and there is no payment code in the tree. Both notes now
describe the scaffolding as scaffolding.

Correction to ac9e5a47's commit title. It says "serialize release-note
writes" and the mechanism does not serialize. The retry plus read-back is best
effort: it catches this leg's own write being clobbered before it reads back,
but a later leg writing from a stale read can still land after that read-back
succeeded and drop this leg's block. The in-file comment now says so plainly.
Blast radius is a cosmetic release body, not the build or deploy path, and a
manual re-run restores it. Left as best effort rather than adding If-Match or
per-target release assets, which is a real design change and out of scope here.
Not amending the commit message, since this branch has stacked children.

Also pins the borsh encoding of qosManifestB64 and qosManifestEnvelopeB64
with a round-trip test. Nothing asserted it, so swapping to the JSON that
actually sits on disk would have broken Go-side verification with every test
still green.

Correction: dropped the parser_grpc_server release image. That matrix
leg and images/parser_grpc_server/Containerfile rode in from #447, which was
closed unmerged. It can't boot: the binary resolves its ephemeral key from
EPHEMERAL_FILE, which nothing in the repo sets, so it falls back to
integration/fixtures/ephemeral.secret, a committed test key that doesn't
exist in the image rootfs, and GrpcService::new aborts on start. Nothing in
CI runs the image, so it built green and would only have failed on first
deploy. It's orthogonal to this PR (the HTTP+JSON pivot), so it's dropped here
and lands separately once it has a real key-loading story. The crate, the
src/Makefile dev target, and the wallet-integration docs are untouched;
only the attested release image goes away.

Stacked on #446 (merged). Supersedes the pivot portion of #304.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds an in-process HTTP/JSON ingress for Turnkey TVC, including boot-proof extension points and deployment packaging.

Changes:

• Adds open v1/v2 parse routes and health endpoint.
• Adds static boot-proof generation and HTTP integration coverage.
• Extends container builds and stagex release metadata.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/parser/http-server/src/main.rs	Implements HTTP ingress and parsing.
src/parser/http-server/src/boot_proof.rs	Adds boot-proof abstraction and manifest encoding.
src/parser/http-server/Cargo.toml	Configures server dependencies and features.
src/parser/http-server/build.rs	Injects build version metadata.
src/integration/tests/http_server.rs	Tests HTTP routes and response shape.
src/integration/Cargo.toml	Adds HTTP test dependency.
src/Cargo.toml	Registers the new workspace crate.
src/Cargo.lock	Locks new crate dependencies.
Makefile	Adds server image targets.
images/parser_http_server/Containerfile	Packages the HTTP pivot.
images/parser_grpc_server/Containerfile	Packages the gRPC server.
.github/workflows/stagex.yml	Builds images and publishes deployment notes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.

Suppressed comments (1)

src/parser/http-server/src/main.rs:355

• This only exercises parse_envelope directly, so it still passes if either route changes its extractor from raw Bytes to Json<TurnkeyRequestWrapper>. The handler-level compile regression described in the resolved review thread is absent from the current diff; restore a test that calls parse_v1(State(state), Bytes::from_static(...)) (and/or routes a raw body through the router) so the X-Stamp extension seam is actually pinned.

    fn envelope_is_parsed_from_raw_bytes_not_reserialized() {
        // A later PR verifies an X-Stamp signature over the exact request
        // bytes. If a handler ever takes `Json<T>` and re-serializes, the
        // bytes change (key order, whitespace, unicode escaping) and every
        // stamp fails. Locking the seam here means that PR adds one call and

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/parser/http-server/src/main.rs:135

• Logging {e} still records attacker-controlled values from the unauthenticated request: as noted above, a serde type mismatch can include the offending string verbatim, allowing each 400 to write nearly 64 KiB of request data (including transaction contents) into enclave logs. Log only bounded metadata such as the error location/category while keeping the client message generic.

            eprintln!("invalid request body: {e}");

.github/workflows/stagex.yml:140

• Including parser_gateway here generates the same paste-ready tvc deploy create instructions used for enclave pivots, even though the role note identifies it as host-side. Its image contains only the gateway, which defaults GRPC_ADDR to localhost and reports /health as unavailable without a separate gRPC backend (src/parser/gateway/src/main.rs:69-105,228-237), so following these generated instructions creates an unhealthy TVC deployment. Keep publishing its image, but restrict TVC deployment instructions to the actual pivot binaries (or give the gateway a separate non-TVC note).

        if: matrix.target.name == 'parser_app' || matrix.target.name == 'parser_gateway' || matrix.target.name == 'parser_http_server'

[prasanna-anchorage]

Read the whole crate rather than the diff, and verified the claims that are load-bearing rather than taking the commit messages for them. The HTTP pivot itself is in good shape — this is careful work and the comments explain why in the places that matter. make lint and make test are both green locally on 73a93048 (exit 0, no failures; the http_server integration test runs and passes).

One thing I'd like resolved before approving, and it is not the pivot — it is the parser_grpc_server image that came along from the closed #447.

Verified

• The vsock gate is actually wired. The Cargo.toml comment warns that without it the pivot reads the dev ephemeral path and crash-loops as Healthy / Desired Replicas: 0/N. images/parser_http_server/Containerfile does pass --features "vsock ${CHAIN_FEATURES}", with the rationale inline. This was my first check because it is the failure mode that is invisible until deploy.
• healthCheckType: TVC_HEALTH_CHECK_TYPE_HTTP is the right pairing for this binary, and correctly different from parser_app's gRPC health check.
• The byte-identical response claim holds. TurnkeyPayload::intermediate_output carries #[serde(default, skip_serializing_if = "String::is_empty")], so base64-of-empty ("") really is omitted and existing consumers see an unchanged body.
• The block_in_place precedent is real — parser_app::service::Processor::process does exactly this around the same parse() call (service.rs:34), so the pivot is consistent with the vsock path rather than inventing a policy.
• The layering for the 413 rewrite is correct. envelope_body_limit_rejection is applied after DefaultBodyLimit, so it is the outer layer and does see the limit's rejection on the way out. The reasoning for handling 413 by status but routing 404/405 through fallback/method_not_allowed_fallback instead is right, and the mod tests comment explaining that handle_parse legitimately returns its own 404 is the kind of note that stops someone "simplifying" this later.
• v2-being-open is the status quo, not new exposure. I started to flag registering an open /v2, then found parser_gateway already routes both v1 and v2 to the same open parse_handler and says production Turnkey serves both from one backend. So this matches today's behaviour; withdrawn.
• Integration coverage is genuinely good — health, v1, v2 parity, the include_intermediate_output opt-in on a Solana transfer (the path that actually emits a non-empty blob), 400, 404, 405, and 413, each asserting the six-key bootProof survives. The try_wait fail-fast with the "if built with --features vsock, rebuild without it" message will save someone an afternoon.
• aws_attestation_doc_b64: String::new() — empty, never faked, so a strict verifier rejects outright. Right call, and e3411caf correcting the release notes to stop claiming enforcement that does not exist is the right instinct; notes that overstate what is enforced really are worse than silence.

The one to resolve: the parser_grpc_server image cannot start

images/parser_grpc_server/Containerfile is new here, and the image it produces panics at startup:

1. parser_grpc_server resolves its key from std::env::var("EPHEMERAL_FILE"), defaulting to integration/fixtures/ephemeral.secret (main.rs:79-80).
2. The image rootfs contains only the binary — mv ${RELEASE_DIR}/parser_grpc_server /rootfs/ then COPY --from=build /rootfs/. . onto busybox. That fixture path does not exist in it.
3. GrpcService::new does .expect("Failed to load ephemeral key"), so the process aborts rather than degrading.
4. EPHEMERAL_FILE is set nowhere in the repo — no Containerfile, workflow, or doc mentions it.

So it builds green in CI (nothing runs it) and fails the moment anyone deploys it. Two things compound that: stagex.yml's TVC-deployment-details step is gated on parser_app || parser_gateway || parser_http_server, so this target is the only one in the matrix that publishes an image with no role note and no deployment guidance; and the default it falls back to is a committed test key (src/integration/fixtures/ephemeral.secret, 64 bytes, in git). Nothing here exposes it today because the file is absent from the image — but a default that points at a repo-committed signing key is a bad direction for an enclave binary to fail in, and it should probably be a required arg with no fallback rather than an env var with one.

Given #447 was closed unmerged, my suggestion is to either drop the parser_grpc_server target from this PR and let it land with its own deployment story, or add the role note plus make the key path required. Happy either way — I just do not think a release should carry an attested image that cannot boot.

Non-blocking

1. encode_borsh_b64 swallows a borsh failure into "" (unwrap_or_default()), which is indistinguishable from the "manifest unreadable in local dev" case that its sibling read_manifest_borsh_b64 does eprintln! about. Near-unreachable for these types (a Vec writer), so this is about the asymmetry rather than the risk — worth an eprintln! on the error arm so the two silent-empty paths are distinguishable in a log.
2. The integration test is one function with seven numbered steps, so a failure at step 2 hides 3-7. Understandable given how expensive server startup is, and it matches the other integration tests, so I would only split it if it starts flaking.
3. SignatureScheme::try_from(sig.scheme).unwrap_or(SignatureScheme::Unspecified) silently degrades an unknown scheme. Unreachable while the producer and consumer are the same binary, which they are — noting it only because it stops being true if parse() ever moves back behind a wire boundary here.
4. All diagnostics go through eprintln!. That is genuinely better than the log::warn! situation I flagged on #453 (inert in parser_app/grpc-server, no subscriber installed), so no complaint — but it does mean this binary has no structured logging either, and the pivot is the one process an operator will most want fields from.

[pepe-anchor]

Fixed: dropped the parser_grpc_server release image entirely rather than making the key path required. It rode in from #447 (closed unmerged) and is orthogonal to this PR's HTTP pivot, so no reason to carry its deployment story here. Containerfile deleted, stagex matrix entry and root Makefile target removed. It can land later with a real required-key story instead of the test-key fallback.

Also added the eprintln you flagged on encode_borsh_b64's error arm, so a borsh-encode failure no longer shares the silent-empty-string path with the "manifest file missing" case.

ddf94f4. fmt/clippy/build/test all green locally.

---

Addressed by Claude, code change applied. React 👎 if this doesn't land and I'll flag it for follow-up.

[pepe-anchor]

Two more findings from the latest Copilot review (#450 (review)), listed under "Suppressed comments" rather than as inline threads (Copilot doesn't re-post comments on code unchanged since the previous pass, so there was nothing to reply to directly). Addressing both here:

src/parser/http-server/src/main.rs:135: eprintln!("invalid request body: {e}") logged serde_json's Display for the error, which embeds the offending value verbatim, i.e. up to the full unauthenticated request body (including transaction contents) into enclave logs. Changed to log only bounded metadata (e.classify(), e.line(), e.column()), never the Display string.

.github/workflows/stagex.yml (TVC deployment details step): the step generated the same paste-ready tvc deploy create instructions for parser_gateway as for the enclave pivots, even though the role note identifies it as host-side. Following those instructions for the gateway would produce an unhealthy TVC deployment (it only contains the gateway binary, and defaults GRPC_ADDR to localhost with no /health backend). Restricted the step's if to parser_app / parser_http_server, the two actual TVC pivot binaries, and dropped the now-dead parser_gateway case branch. Image publishing for parser_gateway is untouched.

Verified: cargo build/clippy/test -p parser_http_server all clean, and the workflow YAML parses.

---

Addressed by Claude, code changes applied to both findings. React 👎 if this doesn't land and I'll flag it for follow-up.

[Copilot]

🟡 Changes recommended

Manifest loading is unbounded, and the promised parser_grpc_server Stagex artifact is absent.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/parser/http-server/src/boot_proof.rs:82

• std::fs::read allocates for the entire manifest before validation, so a malformed or misprovisioned /qos.manifest can cause unbounded startup memory use. Use the repository's bounded-reader pattern (for example, src/parser/cli-core/src/mapping_parser.rs:120-140) and reject files beyond a fixed limit before deserializing.
  .github/workflows/stagex.yml:50
• The PR description says this Stagex matrix builds both parser_http_server and parser_grpc_server, but this change adds only the HTTP target; there is also no out/parser_grpc_server rule or images/parser_grpc_server/Containerfile in the repository. Either add the promised gRPC image leg and its build files or update the PR scope so operators do not expect an artifact that this workflow never publishes.

• Files reviewed: 11/12 changed files
• Comments generated: 1
• Review effort level: Balanced

[pepe-anchor]

Addressed Copilot's review summary (review 5095495163):

Item	Verdict	What happened
Json<T> extractor doc accuracy at main.rs:83/394/396 (suppressed comments)	fix	Verified against axum 0.8.8 source (json.rs): Json<T>'s FromRequest impl only deserializes via serde_path_to_error and never serializes anything, so the code comment's claim that it "re-serializes before the handler body runs" was wrong. Reworded both occurrences (the parse_v1 doc comment and the envelope_is_parsed_from_raw_bytes_not_reserialized test comment) to say the extractor deserializes and discards the original bytes, so verification would later have to re-serialize the parsed value to get bytes back, which is what changes key order/whitespace/escaping. Comment-only change, no behavior or assertion changes; cargo check -p parser_http_server passes.

Addressed by Claude. React 👎 on this comment if any of these don't land and I'll flag it for follow-up.

[Copilot]

🟡 Changes recommended

Deployment instructions omit mandatory startup arguments, and parse routes no longer enforce the existing JSON media-type contract.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

.github/workflows/stagex.yml:213

• The generated deployment instructions omit pivotArgs, but parser_http_server exits during startup unless exactly one ABI-trust option is present (main.rs:322-326). Following this release-note block therefore produces an unhealthy HTTP pivot. Add target-specific instructions for pivotArgs that require the operator to choose either --accept-unsigned-abis or one or more --accept-signatures-from-pubkey values.

            echo 'tvc deploy init -o tvc-deploy.json'
            echo '# Edit tvc-deploy.json to set:'
            echo "#   \"pivotContainerImageUrl\": \"${pinned_url}\""
            echo "#   \"pivotPath\": \"/${TARGET_NAME}\""
            echo "#   \"expectedPivotDigest\": \"${digest}\""
            echo "#   \"qosVersion\": \"${qos_version}\""
            echo '#   "appId": "<env-specific>"'
            echo 'tvc deploy create tvc-deploy.json'

src/parser/http-server/src/main.rs:132

• The v2 route has the same media-type regression as v1: raw Bytes accepts JSON under arbitrary or missing Content-Type, unlike the existing gateway's Json<TurnkeyRequestWrapper> contract. Apply the shared header validation here as well so both versions return an enveloped 415 without reserializing the signed bytes.

async fn parse_v2(
    State(state): State<AppState>,
    body: axum::body::Bytes,
) -> (StatusCode, Json<TurnkeyResponseWrapper>) {
    tokio::task::block_in_place(|| handle_parse(&state, &body))

• Files reviewed: 12/13 changed files
• Comments generated: 1
• Review effort level: Balanced

[pepe-anchor]

Addressed Copilot's review summary (review 5134672719):

Item	Verdict	What happened
.github/workflows/stagex.yml:213 deployment instructions omit pivotArgs	fix	Added a pivotArgs line to the generated tvc-deploy.json edit instructions noting the required ABI-trust flag (--accept-unsigned-abis or --accept-signatures-from-pubkey), since both parser_app and parser_http_server fail closed at startup without it (see CLAUDE.md's "Deploy-Time ABI Trust Posture").
src/parser/http-server/src/main.rs:132 v2 route media-type regression	fix	Same fix as the inline main.rs:116 thread: added is_json_content_type and applied it to both parse_v1 and parse_v2, restoring the 415-on-non-JSON behavior axum's Json<T> extractor gives parser_gateway's route.

Addressed by Claude. React 👎 on this comment if any of these don't land and I'll flag it for follow-up.

[Copilot]

🟡 Changes recommended

The generated parser_app deployment arguments omit required host options, and the startup test can stall CI for an extended period.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/integration/tests/http_server.rs:107

• The 100-iteration poll does not actually bound startup: after all polls miss, execution falls through to wait_until_port_is_bound, whose increasing sleeps can total over two hours. A child that remains alive without binding will therefore stall CI until the job timeout instead of failing fast. Record whether this loop observed the listener and panic after its five-second budget when it did not.

• Files reviewed: 12/13 changed files
• Comments generated: 1
• Review effort level: Balanced

[pepe-anchor]

Addressed Copilot's review summary (review 5134732760):

Item	Verdict	What happened
src/integration/tests/http_server.rs:107 startup poll doesn't bound startup	fix	Confirmed: wait_until_port_is_bound's linear backoff can run ~2h14m worst case, and the preceding 100-iteration poll had no flag recording whether it observed the bind, so a child that stays alive without binding fell through into that slow path. Added an observed_bind bool that panics with a clear message within the loop's own ~5s budget when the port was never observed as bound.

Addressed by Claude. React 👎 on this comment if any of these don't land and I'll flag it for follow-up.

[Copilot]

🟡 Changes recommended

Deployment guidance is incorrect for port overrides, and failed integration setup can leave private keys and child processes behind.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 12/13 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The new public enclave ingress and deployment workflow are security-sensitive and require final human validation despite comprehensive coverage.

Review details

• Files reviewed: 12/13 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[pepe-anchor]

Addressed Copilot's review summary (review 5134824013):

Item	Verdict	What happened
(none)	skip	no actionable items found in the review body

Addressed by Claude. React 👎 on this comment if any of these don't land and I'll flag it for follow-up.