Failed to stream build logs in CI when build queue is non-empty (zero-logs failure mode not covered by #743)

[frankjrobinson]

Summary

railway up (foreground) fails with Failed to stream build logs: Failed to retrieve build log from CI runners when Railway's build queue is non-empty. PR #743 (v4.44.0) fixed the case where the stream errored after receiving log lines (treats it as success). But there is a second failure mode — stream dies before any log line is received — which #743's if received_any_logs { return Ok(()) } doesn't cover.

Environment

• CLI: @railway/cli@4.44.0 (latest), confirmed against the v4.44.0 source at src/controllers/deployment.rs:158
• GitLab CI shared runners → AWS NAT → Railway (Google Cloud)
• Foreground railway up (no --detach)
• Backend service with a longer Dockerfile (40+ COPY layers, multi-stage Python builder)

Reproduction

Run railway up from CI during peak load (EU business hours seems to correlate). Same project off-peak runs cleanly.

Observed timing comparison

Same project, same Dockerfile, same CLI version (4.44.0):

Off-peak success (UTC 22:13)

22:13:45  Build Logs: <url>
22:14:22  exporting to docker image (build complete = 37s)
22:15:08  Job succeeded

Peak-hour failure (UTC 14:19)

14:19:36  Build Logs: <url>
14:20:43  Failed to stream build logs: Failed to retrieve build log  (67s of total silence)

The 67s window contained zero log lines — no individual build step output was received before the WebSocket subscription died. This indicates the build hadn't begun emitting logs yet (still queued / picking up an executor) and the idle WebSocket got reaped — most likely by AWS NAT idle timeout (60–90s typical for non-keepalive TCP).

Why PR #743 doesn't cover this

#743 added two things:

1. tokio::time::sleep(Duration::from_millis(500)).await; — a 500ms initial delay before subscribing
2. if received_any_logs { return Ok(()) } — treat stream closure as success if data already came through

Neither fixes the queue-wait case. The 500ms delay is finite; the queue can be 30s+. And received_any_logs is false when no logs ever arrive, so the early-return doesn't fire and the loop just retries 12× into the same NAT-killed connection.

Suggested fixes (any one would be sufficient)

1. Client-side WebSocket keepalive ping — send a ping every ~30s while waiting for the first log line. Keeps NAT entries alive across queue waits. Smallest change in scope.
2. Server-side keepalive frames on the GraphQL build-logs subscription, even before the build starts producing real logs.
3. Treat "queued, no logs yet, connection dropped" specifically — if the deployment status is still BUILDING / QUEUED when the WebSocket dies, retry the subscription instead of failing. Caller exit code stays 0 since the deploy is genuinely in flight.

Workaround

railway up --detach skips the subscription entirely. The deploy proceeds normally; only the build-log tail is missed. We've shipped a wrapper in our CI that tries streaming first and falls back to acknowledging the in-flight deploy if the stream dies after the upload landed (i.e., the Build Logs: URL was already printed in the output). Happy to share if helpful.

Related

• Failed to stream build logs #543 (original report)
• Add retry logic with exponential backoff for build log streaming #662 (initial retry logic)
• Failed to stream build logs #696 (reopened — same shape)
• fix: Fix flaky "Failed to stream build logs" error in railway up #743 (partial fix in v4.44.0)

[frankjrobinson]

Confirmation from Railway status page

While this issue was being filed, Railway posted an active incident on their status page confirming the diagnosis:

Build Machines (Metal) / Image Registry (Metal) — Investigating

We are aware of an issue causing builds to queue and take longer than usual to progress. Users may experience builds remaining in a queued state for an extended period.

This is the exact mechanism described above: extended queue waits → idle WebSocket subscription → connection reaped → CLI exits non-zero with . So even though the underlying queue issue is on the Railway-platform side, the CLIs lack of WebSocket keepalive during queue waits turns a transient backend slow-down into a hard CI deploy failure for downstream users.

With keepalive (option 1 from the suggestions above), even an extended queue wait would just produce a slow but successful streaming session, instead of a false failure.

[frankjrobinson]

Unrelated FYI: deprecation warnings during install

Not related to the streaming flake, but surfaced by the same CI runs — flagging here so it doesn't pollute the main thread:

Fresh npm install -g @railway/cli (v4.44.0) emits two deprecation warnings:

npm warn deprecated node-domexception@1.0.0: Use your platform's native DOMException instead
npm warn deprecated tar@6.2.1: Old versions of tar are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version.

Both are transitive (node-domexception via fetch / formdata-node, tar via node-gyp / npm). Native DOMException has been available since Node 17; tar 7.x has the security fixes. Just a heads-up for whoever does the next round of dependency bumps — happy to spin out as a separate issue if it's easier to track that way.