Repository navigation
Background windows paused while hidden and at 1×; stream rate capped by size (builds on #72 and #71) - #73
Open
ashleigh-hopkins wants to merge 141 commits into
Open
Background windows paused while hidden and at 1×; stream rate capped by size (builds on #72 and #71)#73ashleigh-hopkins wants to merge 141 commits into
ashleigh-hopkins wants to merge 141 commits into
Conversation
…s its own The Mac client (Sources/MacClient, product SillClient, Scripts/make-client-app.sh): the iOS session code compiled unchanged from iOSClient through Shared/ symlinks, with ClientPlatform and HEVCSampleFeed shared by both apps. Each host window opens as a borderless window of its own, streamed at once (side streams, kinds 29–31), with a handle for this Mac's controls, the remote app's menus in the menu bar, a Dock icon per remote app, the clipboard both ways (kind 32), sound (kind 33) and files dropped in and dragged out (kinds 34–37). Input: pointer and all three buttons, trackpad scrolling with momentum, keys, ⌘Tab and ⌘Space when captured, gestures as the host's shortcuts. The host keeps those windows on one shared hidden virtual display (DeskStages), sized to the viewer's largest screen, taps the shown apps' sound and mutes it there (AppAudio), and takes and gives files (FileDrops, DragOut). Every addition goes only to a device that asks or a host that says it takes it; the iOS app is unchanged on the wire. The iPhone build from Xcode 27.0 can leave the 27.1 hinge API out (SILL_NO_DUO_SDK). New checks: mac-keys and main-router; protocol, compatibility, menus and pointer-control know kinds 29–37.
…wer window Two virtual displays with one vendor and product ID are one monitor to macOS, which takes the older offline when the newer comes; that, not a hardware limit, is why windows on displays of their own fell back to the host's screen. Each window's display now has its own identity from a pool (product 0x100 + slot, a serial kept per slot, so macOS remembers how to show each), and the shared display is gone: a window going full screen takes only its own display, never the other streamed windows. While a window is full screen, its display takes the size of the device's window live (VirtualDisplay.resize keeps the display and its ID), so the full-screen Space fills the viewer window and nothing more; leaving full screen gives the display back its size.
A display fitted to a full-screen window is smaller than it was made, and the next placement read that as too small and made it again, which threw the window out of full screen and round again (black, the sound skipping). A display is reused up to the size it was made at, and a window is fitted from wherever it streams: to the device's window as the main stream, to the size it had before full screen beside it. Each window's display is 2× again unless the window itself is larger than a 2× display can be: the device's widest screen made every display 1× before, soft on a Retina screen.
…de once, back to 2× A host file dragged out of one Sill window and dropped on another is dropped on the host from where it is (FileDrop's new `offer` and `window`): a real drag of the real file, so Finder moves it as it would there, and nothing goes to this Mac and back. Every Sill window takes drops: files from this Mac on a window beside the streamed one land in that window. A window's display was made twice: its first size was the device's screen, the placement then asked for that plus room for the menu bar and made it again. It is made once, at the placement's size. A window larger than a 2× display gets a 1× one, and once it is 5 % under the 2× size again it goes back to 2×.
… windows A full-screen window is its display's size, and that size was taken as the window's: the display was made bigger, which threw the window out of full screen and round again. A window full screen (by Accessibility's word, from the first placement) is sized by the size it had before, and fitted to the device's window only by a size its display can take. The viewer opens no app's untitled strips by itself (a browser's full-screen bar). A drag out of a window counts as leaving it when another window is under the pointer, not only outside its rectangle: a drop on a Sill window lying over the first one went back into the first.
A window full screen on its desk display cannot be moved home, and when the display went its Space went to another of Sill's displays, out of sight: a browser looked closed and would not come back after a quit. Leaving the desk now takes it out of full screen first (1.5 s at most), at Quit too.
Only windows on screen were listed, so a window in another Space (a full-screen one, another desktop) could not be reached from a device: a browser stuck full screen on a display out of sight looked gone. Titled windows of regular apps out of sight are listed as well, from the full list the catalog already fetched for its staged windows (the log line for those is gone with it).
…background The streamed desk window going into or out of full screen is seen by a 0.25 s look at it, where the catalog's 2 s poll left the full-screen window at the wrong size for seconds. A full-screen window shown beside the streamed one keeps the size its display was fitted to: it fell back to its size before full screen, a smaller picture, as soon as another window was clicked. The viewer logs each step of a drop on one of its windows.
A right button whose up never reached the host stayed down there: every move was its drag and left clicks read as right clicks until the next right click. A button press now lets go of any other button the host still holds first, and says so in the log.
docs/mac-client-plan.md is the design and justification as the branch stands (the wire, Sill's desk, sound, files, security notes, what was verified, what is open); CLAUDE.md's entry and Layout and DEVELOPMENT.md's section follow it. The protocol check covers every new payload (round-trips and refusals, 209 checks, 5 new mutants, 28 of 28 caught). A drag offer serves only the device it was made to, and a file's parts are taken only from the device that started it; both are dropped when that device leaves (an open file handle and the drag watch's timer outlived it). The viewer's menu loses Virtual Display, and a Mac viewer's windows always go to Sill's desk rather than the iPad's staging. The desk display's functions lose the "shared" names left from the shared-display phase. One compiler warning fixed.
A folder dropped on a Sill window, or dragged out of one, goes as one zip (FolderZip: ditto, the archive Finder makes, with resource forks, extended attributes and symbolic links) and is unpacked where it lands: on the host beside the dropped files, here where the promise is fulfilled. A folder dragged out onto the Trash goes to the host's Trash whole. The protocol check zips and unpacks a folder with a nested file and a link (215 checks).
…Sill windows Between two windows of one size the router tells the new stream apart only by the host's word that it switched, which reached it on the main thread, after the new stream's first sets could: the new window's picture went into the old one and stayed while nothing there redrew (two Finder windows swapping). The window list's source now marks the switch on the network queue, in the stream's own order (MainRouter.listed; main-router 21 checks, 12 of 12 mutants). A drag out allowed no operation inside Sill Viewer, so another Sill window was never offered the drop (no +): it allows copy and move there now, and the other Mac drops its own files.
…o longer takes them) NSImageView registers for file and image drags and, not editable, refuses them; as the deepest view under the pointer it took every drag meant for the window's DropContainer, so another Sill window never offered a drop.
A drop used to reach the other Mac only where it was let go: its app saw no drag, so it could not show where the files would land (a folder, a web page's drop zone, the address bar), and this Mac's modifier keys did nothing. Now, to a host whose list says `liveDrag`: - The host's own files, dragged out of one of its windows: its drag stays under way, the button held. Over any Sill window its pointer follows onto that window (each on its own desk display); let go there, it lands as a drag of its own (Finder moves, ⌥ copies, ⌥⌘ makes an alias). Let go anywhere else (this Mac's Finder, the Trash), the host gives its drag up with Esc. - This Mac's files over a Sill window: sent at once, the host starts its own drag of them under the pointer as they arrive and follows it; dragged away for 0.35 s, given up and the copies removed. Host: LiveDrags (through the coordinator's deliver: the injector's held buttons and keys, the modifier keys pressed as the device holds them), FileDrops' panel and received files split out. Viewer: LiveDrop, the stream view's and DropContainer's drag hooks, the upload split from the drop. A host without it gets kind 35 at the drop, as before. Checks: protocol (LiveDrag round-trips, the list's flag), compatibility, menus and pointer-control (38 taken, 39 unknown; their mutants that renumbered a kind onto 38 use 39).
…n the Dock - The remote app's icon carries Sill Viewer's in its corner, so it never reads as this Mac's app; written again when the icon arrives after the Dock icon started (it used to stay plain). - The Dock menu marks the windows as macOS marks an app's own: ✓ the app's front window, ◆ one minimized; it follows focus and minimizing. - Kept in the Dock and clicked with nothing open: the copy knows its app (SillRemoteApp in its Info.plist), opens Sill Viewer if need be and asks for the app, whose windows on the other Mac open here, or which is launched there when it has none. Docs: the plan and CLAUDE.md name kind 38 and the live drag, its verification and what is open.
Beside the streamed window no fit is asked; the fallback to the size before full screen was taken before the check that keeps a fitted display, so the display shrank and the video filled a quarter of its window, pinned to a corner, whenever another Sill window took focus.
…ts windows, the icon kept - Kind 39 (DockMenu, `dockMenu` in the list): the host reads an app's own Dock menu (Finder's New Finder Window…, a chat app's New Chat) through Accessibility, which has it only while the Dock shows it, so it shows for about half a second there; read without the app's windows and the Dock's own items, and pressed by its place and title. The Dock icon here adds it under its windows, read again at most every 20 s as its menu opens. - Quit from a Dock icon closes that app's windows here (the app runs on there), where it left them open. - The badged icon is also the copy's own (AppIcon.icns, signed again), so one kept in the Dock keeps its icon while not running instead of a blank one. - A full-screen window in the background is captured from its display: an app reports its full-screen window far wider than its Space, and captured alone the video was a quarter of the picture, in a corner. Checks: protocol (DockMenu round-trips), compatibility, menus and pointer-control (39 taken, 40 unknown).
… not a count - A device asks for the sound compressed (SideStreamRequest.audioCodec "aac-eld"): the host encodes the tapped apps' sound as AAC-ELD, about 96 kbit/s at 48 kHz stereo instead of plain PCM's 1.5 Mbit/s, for about 15 ms more delay; a compressed kind 33 carries the decoder's cookie and is never read as plain samples (the channels byte's high bit). Sill Viewer asks for it unless `SillClient.plainAudio` is set; a host that cannot encode, or a device that asks for none, gets PCM. - Sill's desk takes a new window while its displays leave room for one more within a quarter of this Mac's memory (each display holds its own framebuffer, about 130 MB at the largest Retina size); display identities are taken as needed, no longer a pool of 16. Checks: protocol (the compressed packet, the codec asked for).
… the background; full-screen windows reachable after a restart - A button let go where the stream view never hears of it (it moved to another window while the button was held, as the host switched streams) went nowhere, and the other Mac kept it down: every later click read as that button's until it was pressed again (a left click acting as a right click). Every up in the app is looked at first, and one the view would miss is sent from there; a button the view thinks down that this Mac has let go of is let go of at the next move. - A press on a window in the background brings it forward and, still held once the other Mac streams it, goes on there: a drag out of a window in the background starts with one press. - A full-screen window leaves full screen before it goes home and only then is its display removed (an app reports leaving at once, while its window is still the display's size: removed then, the Space outlived the display on the Mac's own, where Accessibility cannot reach it, and after a restart it streamed black and took no input). A window found in such a Space is reached by bringing its app forward, taken out of full screen, and moved to the desk.
…ached only for a pick, Dock menus only of apps - The windows the viewer asked the host to keep live were read from AppKit's list while a window closed, the closing one still in it: the host kept that window on its desk and captured it, the sharing indicator on, and it came back at the next launch. Only windows still open are asked for and kept. - A window in another Space is reached by bringing its app forward only for the window a device picked, at most once a minute (it fired for a side stream's window in another Space and brought its app forward on the host by itself). - An app's Dock menu is read from its own Dock item only (AXApplicationDockItem): a folder or stack of the same name gave the Dock's own menu (Sort by, Display as).
…n helper windows left out; Dock names - A minimized window is taken out of the Dock before it moves to the desk (moved while minimized it never showed there: "the window vanished while being staged"), and minimized again going home. - A picked window in another Space (another desktop) has its app brought forward first, which shows that Space, so it can be moved from it; full screen there, it leaves full screen first. Once a minute a window at most; never for a minimized window or a side stream's. - A window that could not come to the desk and is not on screen is captured on its own, never cropped from its display "with its app": that showed whatever lay there (another app's window) instead. - Windows above every display (an app's hidden helper at y -360) are no longer listed. - A Dock icon is named after its app: the window's app name, else the installed app's, never its bundle ID while either is known.
…t showed the bundle ID, the copy's file name)
…an app's menu; the ◆ it ignored)
…pp with no window gets one - The Desktop source took the first display ScreenCaptureKit listed, which could be one of Sill's hidden desk displays (the Desktop showed a window staged there, not the Mac's screen). It is one of the Mac's own displays now (never VirtualDisplay's vendor), the main one unless a device picked another: the window list names them (`displays`, `desktopDisplay`) and kind 40 (PickDisplay) picks one, the Desktop restarting on it. Sill Viewer lists them under Desktop in its menu bar icon. WindowCatalog's preferMainDisplay, which kept the iPad's Virtual Display off the Desktop, is gone: the rule covers it. - Opening an app that is running with no window (Finder with none open) did nothing: its Dock icon is clicked now, as a person would, which asks it for a window, and the new window streams. Checks: protocol (the displays and PickDisplay), compatibility, menus and pointer-control (40 taken, 41 unknown).
… in a Space - Sill Viewer shows each of the host's own displays as a Desktop window of its own (the menu bar icon lists them). The focused one streams as the Desktop on its display (PickDisplay, then the Desktop); the others stream beside it as side streams, a display's ID with the top bit set where a window ID goes (DesktopWindow), which the host captures whole. Drops and live drags on a Desktop window land on its display. A host that lists no displays keeps the one Desktop window. - A window off screen is listed when it belongs to a Space (another desktop, a full-screen Space, minimized from one), any app's, a menu bar app's too (its minimized window): an app's own hidden windows belong to none (a chat app's hidden helper, which no position test told apart: it moved onto the screen). SkyLight's answer, read-only, looked up at run time.
…haring on the host lets the devices go
- Sill Viewer never sends on input this Mac's own Sill host posted (the event's posting process):
with a viewer on each Mac showing the other's Desktop, a move went to the other Mac, landed on its
viewer, came back as that host's posted move, and neither pointer could be moved.
- A Desktop stream leaves out Sill Viewer's windows as it leaves out Sill's own: they show other
Macs.
- The person at the host stopping the sharing (the menu bar's screen-sharing item) stops every
stream and lets every device go with a goodbye ("stopped", the Mac's words, no reconnect), where
the main stream only went to nothing and the side streams were tried again.
…ng it forward there
- Sill's menu on the host lists the windows shared now ("Shared Windows"), each with Stop Sharing,
and Stop Sharing All: the window goes home, stops streaming, and the devices are told (kind 41,
TakenBack) and close their windows of it; they stay connected and may open it again.
- An app brought forward by the person at the host (its Dock icon, ⌘Tab, a click) whose windows a
device shows has them taken back the same way, unless a device acted in the last 2 s (its own
clicks, picks and menus bring apps forward too); a skipped one is logged with the device's last
message.
- The system's Stop Sharing takes every window back the same way, the devices staying connected,
where it let every device go.
- The Virtual Display item says what it is for: iPhone and iPad; a Mac viewer always moves windows.
Checks: protocol (TakenBack), compatibility, menus and pointer-control (41 taken, 42 unknown).
Owner
|
From a pass over all the open pull requests (nothing is decided yet): this builds on #72 (and so on #66) and carries #71's three commits unchanged, so it can only merge after them, and its copy of #71 should match whatever version of #71 lands. The host-wide StreamRate cap changes every iPhone and iPad stream, not only Mac viewers, so it would be easier to review as a PR of its own. The overlaps with main's clipboard (#78), #94, #93 and #107 come from #66 and would be settled there. |
This was referenced Oct 15, 2026
grub-basket
added a commit
to grub-basket/sill
that referenced
this pull request
Oct 15, 2026
Review of Saffsanity#111 (the repository owner): Saffsanity#66, and Saffsanity#72 and Saffsanity#73 built on it, add windows from other Spaces and minimized windows to the catalog, so a window can leave the list and come back. The first version of this pull request counted it as new each time: the host dropped a window from its first-seen order when it left and put it first again when it came back, and the device compared each list only with the one before. Every switch of Space or every minimize would have moved windows to the front, on every device, and on Saffsanity#66's Mac viewer too (which compiles StreamClient.swift). - WindowOrder (Sources/StreamProtocol, pure, shared by both ends): seen() is the host's first-seen order: windows never listed before first, in the order listed, then every window seen before, gone ones included, so one that comes back takes its old place; past a cap the windows gone the longest are forgotten, never one listed now. arranged() is a device's arrangement after a list: windows this session has never seen, and the arrangement does not hold, go first; nothing on an unarranged bar or on the session's first list. - WindowCatalog keeps every window it has listed (seenCap 1024) and orders by WindowOrder.seen. - StreamClient keeps every window id this session's lists named (seenWindowIDs; cleared at tearDown and when the Mac changes) and takes WindowOrder.arranged's result, saving it. Tests/checks/window-order: 14 checks, 10 of 10 mutants caught; README and CI's mutants matrix. Tested: swift build -c release; the check and its mutants; the iOS app type-checks in a scratch copy with DuoPostureReader's iOS 27.1 calls stubbed (this Mac has the iOS 27.0 SDK only). Not run on a device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the clipboard's kind 32 negotiation and the display teardown work Conflicts kept both sides: the pill's comment has the bolt and the refresh that restarts captures; the protocol mutants list both sides' cases; CLAUDE.md's wire list has the Dock bounces and full-speed flags beside the clipboard's subscription; the plan keeps this branch's notifications line and mac-client's merge notes.
…anity#78, the clipboard's kind 32 negotiation and the display teardown work Side streams keep this branch's capture plan (StreamRate, the crop's pill after the encoder) with mac-client's rule while display streams deliver nothing: no full-screen display capture and no crop with its app, the window alone. CLAUDE.md's wire list has the paused and background requests beside the clipboard's subscription; the checks list and README take both sides (protocol 249 + 8, 41 mutants).
Author
|
Good idea. The StreamRate cap is now its own PR, #122, from main, applying to the main stream only. This branch keeps its side-stream use of the same |
This was referenced Oct 19, 2026
Open
hewigovens
added a commit
to hewigovens/sill
that referenced
this pull request
Oct 25, 2026
…left quarter A window captured where it is was encoded at its points × captureScale, whatever the scale of the display it is on. ScreenCaptureKit, without scalesToFit, draws a window smaller than its output into the output's top-left corner, so a 1× window at Retina (a non-Retina display, or a Mac with no display once a Screen Sharing session ends) streamed as a picture twice its size with the window in the top-left quarter and the rest black. The host's own log showed it: "Capture started: 3840×1928 … filter (1920.0, 964.0) scale 1.0". windowCaptureScale (pure) takes the stream's scale but never more than the window's own (SCContentFilter.pointPixelScale), at the pick, at the fallback after the virtual display fails, and in the poll's size check, which now also restarts a window whose scale changed. A staged window's display is made at the stream's scale, as before. Standard is unchanged. Tests/checks/window-capture-scale: 1,041 checks, 6 of 6 mutants; in CI's mutants matrix. On this branch (PR Saffsanity#73's Mac viewer) also: a window cropped with its app (AppCrop) and the side streams' windows in place or cropped take the same scale; Sill's desk keeps its own (desk.scale).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #72 (which builds on #66) and on #71 (the slow-state fix, merged in here as it is, same commits). Until those merge, GitHub shows their commits here too; this PR's own are:
Why
The host has one hardware video encoder, and every window open in Sill Viewer had a stream of its own on it: the focused one at full rate, every other one at 15 fps at Retina size, minimised ones included. With four Retina windows open, the background streams took a large share of the encoder, and the focused window got about 46 of the 90 frames a second its app drew; the rest were dropped before encoding.
What changes
Windows no one can see are paused. The viewer lists the windows it shows that are minimised (into their app's Dock icon or the usual way) or wholly covered or off screen (
NSWindow.occlusionState) in a new optionalSideStreamRequest.paused. The host keeps them on its desk, so they come back at once, but captures and sends nothing for them until they can be seen again. The focused window is never paused. A host from before ignores the key and streams them as now.The focused window too, while the viewer is in the background. Once Sill Viewer has been out of front for 3 s, it says so in a new optional
SideStreamRequest.background, and while every connected device has, the host runs the main stream at 1× and 15 fps, unless that window is marked full speed. Coming back to the viewer brings it back at once; a switch to another app and back within 3 s restarts nothing. Seen between two Macs: 2758×1998 at 90 fps went to 1378×998 at 15 fps three seconds after leaving the viewer, and back on returning.Background windows are encoded at 1×. A quarter of a Retina window's pixels at 15 fps and the same bitrate. The focused window stays at the stream's scale, and so does a background window marked full speed with its bolt. A host-side change: no wire.
A stream's rate is capped at what the encoder keeps up with at its size (
StreamRate): the rate a device asks for is kept while its pixel rate fits 520 MP/s, else 90, 60, 48 or 30. A 3930×2188 window asked for at 120 fps runs at 60: a fresh hardware session there returned 65 fps at best, so at 120 half the frames were captured only to be dropped before the encoder. The devices' rate stays what every restart rule compares, so the cap restarts nothing. Background windows at full speed are capped the same way.The encoder's slow-state swaps stop when they don't help (Encoder: stop replacing a slow session when new ones keep going slow again #71, merged in here): a stream whose new sessions go slow again within 30 s twice in a row keeps its session. Today a stream swapped every 10–13 s, 222 times, each a keyframe and a hitch.
A full-screen window in the background is no longer stretched: a background stream's encoder is made at the size actually captured, after the window is placed (it was made at the window's reported size, taller than the band captured of a full-screen window).
Viewer in front while one of its Dock icons is (that icon holds the menu bar while its menus are open), so browsing a remote menu does not drop the window to the background quality; at the 3 s mark the viewer looks again before saying it is in the background.
The encoder settings stay as they are
These were tried behind hidden settings and are removed again, as none beat the defaults.
The same self-scrolling page (10 s scrolling, 2 s still, repeating), focused on the viewing Mac, 5 minutes each on this branch at the 60 fps cap:
The defaults win; no experiment is better, and the peak-rate key keeps the session in its slow state.
Checks
protocol: 245 checks (the new fields' round trips and an older device's request), 38 of 38 mutants caught.Tests/checks/run-all.sh: all pass.stream-rate(new): 15 checks, 7 of 7 mutants.encoder-slowstate: 1,211 checks, 33 of 33 mutants (four new, for relapses).swift build -c release; the viewer built.Verified by hand, between two Macs