Skip to content

The stream on the iPad's external monitor, with the iPad as its trackpad - #301

Open
addhumal wants to merge 1 commit into
Saffsanity:mainfrom
addhumal:external-display
Open

addhumal wants to merge 1 commit into
Saffsanity:mainfrom
addhumal:external-display

Conversation

@addhumal

@addhumal addhumal commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

With a monitor on the iPad, the Mac's picture now goes to the monitor, at the monitor's size, and the iPad becomes its trackpad.

What changed

  • The monitor gets its own window. iPadOS gives the app a scene for an external display: a USB-C monitor with Stage Manager off, or AirPlay. The stream's picture moves to that window and comes back to the iPad when the monitor goes.
  • The iPad becomes the controller. Sideways, the trackpad takes the picture's place and says "The Mac is on the external display." The hardware keyboard keeps working. Upright, the picture pane says the same above the trackpad it already has.
  • The Mac sizes the window to the monitor. The viewport is the monitor's size in points, sent again as a monitor comes or goes.
  • Unchanged: no wire change, and nothing changes with Stage Manager on. There the monitor extends the iPad's screen and iPadOS gives no such scene.
  • DEBUG: -SillExternalDisplay 1920x1080 stands in for a monitor's size without its scene, for photos.

Problems found, and how they were fixed

  • No window at all. My first version handed SwiftUI's main scene its own configuration back from configurationForConnecting, and the app came up as a white screen. It returns an unnamed configuration, SwiftUI's own, now. Found on the simulator before anything else.
  • The wrong "in front". The session decides whether it's in front by its display view's window. On the monitor, that window is the monitor's. It now uses the iPad's own scene while the picture is away.
  • Two windows. With two app windows, the window that gave the monitor up never redrew to take its picture back. A published counter makes it redraw.

Checks

  • The three iOS builds: Debug and Release for the simulator, Debug for a device.
  • On a private iPad Pro 13" simulator (iOS 27.0, deleted after) against a synthetic loopback host with -SillExternalDisplay 1920x1080: the host got a 1920×1080 viewport, and the trackpad and the line show upright and sideways.

Not checked: a real external scene. This Xcode has no Simulator.app to attach a display, so the picture actually moving to a monitor, and back, is untested. AirPlay. VoiceOver on the trackpad.

@Saffsanity: this one needs a monitor on the iPad. With Stage Manager off, plug in USB-C: the Mac on the monitor at its size, the iPad a trackpad. Unplug: the picture back on the iPad.

…ackpad

- A monitor's scene (UISceneSession role windowExternalDisplayNonInteractive:
  a USB-C monitor with Stage Manager off, or AirPlay) gets its own window
  (ExternalDisplaySceneDelegate); the picture of the stream screen last shown
  moves there (ExternalDisplay, in SillAppDelegate.swift), and back as the
  monitor goes. Every other scene stays SwiftUI's.
- The iPad: sideways, the trackpad in the picture's place over the input
  view (the hardware keyboard stays), "The Mac is on the external display.";
  upright, the picture pane says so above the trackpad it already has.
- The viewport is the monitor's size in points, so the Mac sizes the
  window to it; sent again as a monitor comes or goes.
- The session judges "in front" by its own window's scene (homeScene) while
  its display view is in the monitor's window.
- DEBUG: -SillExternalDisplay WxH, a monitor's size without its scene.

Found on the way: handing back the main scene's own configuration from
configurationForConnecting left the app with no window at all (a white
screen on a private iPad simulator); it returns an unnamed one, SwiftUI's.

Checks: the three iOS builds; on a private iPad Pro 13" simulator (iOS 27.0,
deleted after) against a synthetic loopback host with -SillExternalDisplay
1920x1080: viewport 1920×1080 sent, the trackpad upright and sideways. Not
run: a real external scene (this Xcode has no Simulator.app to attach one).
@kolyan94

kolyan94 commented Nov 5, 2026

Copy link
Copy Markdown

Hey @addhumal thanks for this one. I tried it on an iPhone 17 Pro (iOS 27.2) with XREAL One Pro glasses.

iOS 27 only connects the external display scene through a registered scene accessory, so I added that. While I was there, I also kept the phone awake while streaming to the glasses, turned off overscan, and made the black pane work as a trackpad, plus a couple of small fixes.

With those changes the picture moves to the glasses and the phone works as the controller. I haven't tried it on an iPad.

Commits are here if you want them: https://github.com/kolyan94/sill/commits/xr-glasses/

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants