Skip to content

grpcclient: unblock processes when exec stream ends - #7300

Open
Zaczero wants to merge 2 commits into
moby:masterfrom
Zaczero:fix-exec-stream-failure
Open

Zaczero wants to merge 2 commits into
moby:masterfrom
Zaczero:fix-exec-stream-failure

Conversation

@Zaczero

@Zaczero Zaczero commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

The gateway client's Container.Start and ContainerProcess.Wait can hang indefinitely when the shared ExecProcess stream ends while their caller contexts remain live. The receive loop stops delivering messages, but the per-process receivers only observe their caller's context and deregistration. Transport errors cancel the forwarder's errgroup context without reaching those receivers; EOF and gRPC cancellation return nil without canceling it.

Cancel the forwarder's context with the receive error on every stream termination, including EOF and cancellation, and have process receivers observe that context. Return the cancellation cause directly from the mailbox, so Start and Wait propagate it through their existing error paths. Keep the existing Release behavior for EOF and cancellation, and use the existing output-handler cleanup to unblock pending stdin.

The regression uses a real gRPC server on a loopback listener. It terminates the stream with EOF, Unavailable, Canceled and DeadlineExceeded, both before Started and while waiting for completion, with two processes sharing the stream and stdin left open. It checks that both calls return the transport failure while their caller context is still live. Separate completed-process cases verify that Exit and Done followed by stream termination still produce successful waits. Each case also checks that Release still returns nil for EOF and cancellation and the transport error otherwise.

Validation:

  • All eight in-flight failure cases time out against unmodified master.
  • go test -race ./frontend/gateway/grpcclient -count=100 -timeout=120s passes, including the four completed-process cases.
  • golangci-lint run ./frontend/gateway/grpcclient/...

@Zaczero
Zaczero marked this pull request as ready for review October 19, 2026 11:49
@Zaczero

Zaczero commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

ci failure appears unrelated

@crazy-max

crazy-max commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

ci failure appears unrelated

yes maybe a startup race in containerd FIFO handling. Created #7301

Comment thread frontend/gateway/grpcclient/container_test.go Outdated
@crazy-max crazy-max added this to the v0.35.0 milestone Oct 19, 2026
@Zaczero
Zaczero force-pushed the fix-exec-stream-failure branch from fb3e1c9 to afeb757 Compare October 20, 2026 00:18
Process receivers only observe their caller context and deregistration,
so Start and Wait can hang after the shared ExecProcess receiver exits.
Transport errors cancel the forwarder context without notifying them;
EOF and cancellation leave even that context live.

Cancel the forwarder on every receive error and let process receivers
return its cause. Preserve normal deregistration and Release semantics,
and use the existing output-handler cleanup to unblock pending stdin.

Exercise EOF and three gRPC status errors over a real stream with two
processes starting or waiting, plus successful completion before stream
termination. Check that Release still reports only failures other than
EOF and cancellation.

Signed-off-by: Kamil Monicz <kamil@monicz.dev>
@Zaczero
Zaczero force-pushed the fix-exec-stream-failure branch from afeb757 to 3c23c1a Compare October 20, 2026 00:24
Keep sending stdin EOF during cleanup so the gateway can finish its Done
handshake, but ignore send failures when the input copy was interrupted by
closing the pipe. Preserve send errors when the input reaches EOF naturally.

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
@crazy-max

Copy link
Copy Markdown
Member

While checking the completed-process cases with stdin left open, I reproduced a case where the cleanup EOF send replaces a successful process result with io.EOF:

--- FAIL: TestProcessStdinEOF/successful_process_with_open_stdin
    Error: Received unexpected error:
           EOF

A loopback gRPC test also reproduced this when the server sent Exit and Done, then ended the stream while stdin remained open (with a stub server, not full BuildKit daemon)

The additiional commit keeps sending EOF because the gateway needs it before sending Done, but ignores send failures when cleanup closed the input pipe. Errors from sending a natural input EOF still propagate.

I also added coverage for stream termination between Exit and Done:

$ CGO_ENABLED=1 go test -mod=vendor -race ./frontend/gateway/grpcclient -count=100 -timeout=120s
ok  github.com/moby/buildkit/frontend/gateway/grpcclient  15.551s

@crazy-max
crazy-max requested a review from tonistiigi October 23, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants