feat(stream): carry HID input over a WebRTC data channel - #118
Closed
szdziedzic wants to merge 1 commit into
Closed
szdziedzic wants to merge 1 commit into
szdziedzic wants to merge 1 commit into
Conversation
On the WebRTC transport, viewers now open one ordered, reliable data channel labeled "input" alongside the video. It carries the exact [tag][JSON] frames the /ws socket carries, and the server funnels both into the same DeviceSession HID dispatch. Input therefore rides the negotiated media path (UDP, direct or TURN) instead of the tunneled TCP WebSocket, whose relay hops and head-of-line blocking made gestures freeze on hosted deployments. The WebSocket stays open in every mode: it still carries screen-config pushes and remains the input fallback. A gesture-boundary router pins every begin/move/end of one gesture to the transport it began on, which closes the cross-channel ordering hazard that motivated removing data-channel input previously. The server closes data channels with any other label, and libwebrtc's delegate thread only enqueues messages onto the existing threadsafe-function queue. Verified end to end: werift-based tests negotiate a real peer, send a tap over the channel, and assert it lands in the event log; a live browser session against the built CLI opened Settings and scrolled via taps and drags delivered over the channel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On the WebRTC transport, viewers open an ordered, reliable data channel labeled
inputalongside the video. It carries the same[tag][JSON]HID frames as the/wssocket, and the server funnels both into the sameDeviceSessiondispatch. The WebSocket stays open for screen-config pushes and as the input fallback.Why
Hosted deployments reach the helper WebSocket through the ngrok HTTP tunnel: every touch move crosses extra relay hops on ordered TCP, where one lost segment stalls the whole gesture stream for a retransmit round trip. Production users saw gestures freeze. The WebRTC media path had already negotiated a direct (or TURN) UDP route between the same two machines, so input now uses it.
How
input, retains it until it closes, and hands each message toCaptureEngine's input handler;SimCapture.subscribeInputmarshals frames onto the JS thread through the existingNodeAsyncQueuewithout blocking a libwebrtc thread. Channels with any other label are closed.DeviceSessionsubscribes to native input when the transport is WebRTC and feeds it intohandleHidMessage, the same path/wsuses.useWebRtcStreamcreates the channel before the offer so SCTP negotiates with the video.HidTransportRouterpins every begin/move/end of one gesture to the transport it began on and only re-evaluates at gesture boundaries, which closes the cross-channel ordering hazard that motivated removing data-channel input earlier. A channel that dies mid-gesture falls back to the socket immediately.Part of the tap-to-photon stack (bottom → top): latency probe → send frames on arrival → 240 Hz seed poll → bitrate floor → playout window → data-channel input → lossy moves lane. Everything is on by default, so a package built from any layer works with the unchanged EAS launch flags (
--transport webrtc --webrtc-codec vp8 --max-dimension 960 --video-bitrate 6000000 --video-fps 60).Measured with
scripts/latency-probe(M5 Pro, loopback, iPhone 17 / iOS 26.5, 25 taps): 0.1.51 p50 32–37 ms / p95 ≈ 43 ms → top of stack p50 27 ms / p95 38 ms, no outliers above 40 ms.🤖 Generated with Claude Code