Repository navigation
The Mac's sound on the device: Send Audio, kind 43, AAC-ELD (#83) - #94
Merged
Merged
Conversation
Noah's request of 2026-09-25 ("have Sill send audio back with the
video"), planned against origin/main at 8b0d418 and the remote-pacing
branch (3126821). Design only; audio stays out of v1 (BRIEF.md).
- Capture: an audio-only ScreenCaptureKit stream of its own, per app
(the window's app, every app but Sill for the Desktop), so rotation,
Aa and settings restarts never cut the sound. Apple (WWDC22 10155)
filters audio per app, and a single-window filter does carry its
app's sound. Screen Recording covers it; no new permission.
- Codec: AAC-ELD at 128 kbps, 480 or 512 frames a packet, chosen from
an in-memory AudioToolbox probe on macOS 27 and the iOS 27 simulator
runtime: 15.0 ms end to end against AAC-LC's 74.7 ms and Opus's
16.5 ms, at half Opus's encode CPU.
- Wire: kind 28 (23 hello, 24/25/27 menu bar, 26 pointer), a JSON
format then binary packets stamped on the video's clock; the hello
lists the codecs a device plays; Send Audio in kinds 16/17.
- Pacing: never in the home delta-drop count; in the remote byte
backlog, never dropped by it, so video drops first.
- Device: a pure playout model (a floor from the packets' own stamps,
a jitter need, a guard one frame behind the picture, drift by single
frames, jumps over 40 ms) and AVAudioPlayerNode playback (no Swift
render block), .playback mixing with others, a Sound button to mute.
- Send Audio off by default; CLI --audio; stdout byte for byte without.
- Gates H0-H12, S1-S3, P1-P16; 15 open questions with defaults;
about 8-9 days.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked against Apple's documentation (SCStreamConfiguration's audio properties, SCContentFilter, WWDC22 10155 and 10156, AVAudioSession, AVAudioPlayerNode and AVAudioNode, the engine's configuration change, kAudioFormatOpus), main at 150f781, every branch and worktree, and an in-memory AAC-ELD probe (a reset, a late join). Fixed in place: - Kind 29: trackpad-gestures-plan.md reserves 28 (Tier 2). pbxproj A401-A403, clear of remote-pacing's A020 and home-pairing's renumbering. - The need adds the output latency (AirPods got no sound at all at home: every packet late, the 150 ms cap kept it so); "late" and the held packet's deadline count the IO buffer, read back after activation. - Only the Mac's segment flag starts a decoder clean: a packet decoded on a reset decoder mid-stream is noise (-5 dB; the fourth is exact); a segment's first packet is placed by T + P / rate. - No zero fill and no flush at a gap: packets made late raised the need as if the link had jittered. A hand-over resets nothing. - The budget recomputed (the capture chunk and the IO buffer were counted twice, the keyframe's cover was low): 60-155 ms, 20-115 ms behind the picture. "Never leads" bounded: the frames' stamp is taken at encode-out, so the guard cannot see the picture's time on the Mac. - Remote wait behind a keyframe 1-4 s at 4 Mbps; 100 small messages cost about 25 KB/s on a still link; AirPlay, activation failure, the configuration change, a growing pool, recovery at the next select. - Tests: silent simulator gates against the harness, the tone stamped by sample count, SILL_TEST_AUDIO_PAUSE, the model in audiocheck, new invariants and mutants; step 8 updates the README, the App Store text and the privacy policy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Noah moved the sound into scope on 2026-09-27 ("Audio: the plan is done
and parked as a v2 feature by your earlier decision", first in what he
wants worked on): Q13 is answered, now, and every other open question
takes its default. Read again against main (the pointer Saffsanity#31, the menus
Saffsanity#36, remote pacing Saffsanity#34, the iPhone's portrait layout Saffsanity#30, the tour Saffsanity#35)
and the open Saffsanity#37 (pairing at home), Saffsanity#38 (gestures, kind 28) and Saffsanity#39
(away from home), and every branch and worktree:
- Kind 29 is still free; the checks that name the free kinds flip with
it and the mutants that renumber onto 29 move to 250-254.
- The send path on main's StreamServer: the sound counts in remote
pacing's pendingBytes (budget, slack, hold, idle mark) and is never
dropped by it; a send of its own like the tick and kind 26; lastSentAt
set, so TLS clients (remote now, at home with Saffsanity#37) skip their ticks.
- The sound follows the source from active's didSet, where the pointer
and the menus already do, not from select's defer.
- The Sound button on every layout: after Desktop in the bars, Menus
kept first where only one fits; on a phone upright at the end of the
thumbnails' row, row 1's approved five untouched; and a Sound switch
in the panel's group for this device. The tour needs nothing.
- pbxproj AA01-AA03 (A401-A402 went to the tour, A801 to away).
- Test rules: loopback and software-encoder hosts, no-device.sh, the
simulator's engine in manual rendering so nothing reaches a speaker.
The new last section, "Refresh (2026-09-27)", lists every change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The wire of docs/audio-plan.md §3, additive only (Compatibility floor):
kind 29 (host → device) carries a format (type 1, JSON AudioFormat:
codec, rate, channels, the packet size, the priming, the bitrate, the
epoch, the cookie, what it is of) and packets (type 2, binary: epoch,
seq, the segment flag, 1–255 length-prefixed packets). A reader skips
a type it does not know, and AudioMessage.parse never traps: nil for
an empty payload, an unknown type, a count of 0, a length past the
end, bytes left over, a format over 4 KB or packets over 16 KB.
The hello gains `audio`, the codecs a device plays best first, and
AudioCodec.choose is the host's pick: the first of the device's list
it makes ("aac-eld" only). A device without the list (every one before
today) is never sent kind 29. Kinds 16 and 17 gain `sendAudio`, the
state `audioNote`, ClientStats `audioBehindMs` and `audioLate`, every
one optional. StreamSettings takes sendAudio with no default, as it
takes directWireless: the host reports nil until its sound exists (the
next commits), the DEBUG mock and the ledger check nil too.
Checks: compatibility 112 (the sound's JSON, the older types reading
the newer JSON, the pick) and 25 of 25 mutants; protocol 218 (the
plan's example bytes, every malformed payload, 2,000 random and
damaged ones) and 30 of 30; the kind checks say 29 is audio and 28 and
30 unknown, and the mutants that renumber a kind move to 250-254, which
no plan will take (menus 43 of 43, pointer-control 156 and 33 of 33).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AudioPacketizer (pure, Foundation only; docs/audio-plan.md §4.3): the chunks a source delivers become blocks of one packet's frames, 512 when the first chunk is a whole number of 512-frame packets, else 480. Each block is stamped with the host time of the input frame its packet decodes to first, n·F − P, from the anchor of the chunk it falls in, so time stamp jitter never moves a sample. A chunk more than a quarter of a chunk from where the one before ended, later or earlier, ends the segment: what was short of a block is dropped, never flushed, padded or filled with zeros, and the next block starts a segment (the encoder is reset before it). A chunk of another format is another epoch. Beside it: AudioSourceRule (the same app running kept, one that ended started again at most every 10 s, anything else a new stream, none a stop), PCMLayout (a buffer's layout read from its description, never assumed, made interleaved stereo: planar, mono, 16- and 32-bit integers) and AudioBufferTally (the first minute's line P2 records). Tests/checks/audio-packetizer: 89 checks (the stamps ±1 µs at 480, 1024, 441 and 960 frames, a jitter oracle, the tolerance at ±0.1 ms of a quarter chunk both ways, gaps, the rule's table, the conversions, the tally) and 27 of 27 mutants. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AudioEncoder (AudioToolbox only; docs/audio-plan.md §4.4): interleaved stereo Float32 at 48 kHz to AAC-ELD at 128 kbps, 480 or 512 frames a packet, one packet per block from the first; the packet size, the priming (240, 256), the cookie and the bitrate read back; reset() for a new segment. AudioDecoder (iOSClient, AudioToolbox only, not yet in the Xcode project): one packet at a time to a buffer per channel, the decoder's reset for a segment's first packet. TestTone: the synthetic host's 440 Hz at -30 dBFS with a 4 ms 2 kHz click at each whole second of the wall clock, by frame index. Tests/checks/audio-codec (H4): the tone through the packetizer, the encoder and the decoder in memory, every decoded sample placed by its packet's stamp: each click at its whole second, 0 frames off, at 480 and 512 frames a packet and across gaps; a segment's first packet within -28 dB of the tone (both ends reset); a late join's first packet noise and fourth exact; 5 s of a busy signal at 130.2 kbps; nothing but AudioToolbox linked. 21 checks, 9 of 9 mutants. (Without its cookie the ELD decoder reads the frame length from the format, and with it from the cookie: removing either alone is an equivalent mutant.) H0 again on this Mac (macOS 27.0, Xcode 27.0): ELD 480 and 512 give priming 240 and 256, the same after AudioConverterReset, 15.0 and 25.3 ms end to end with 480-frame chunks, and a late join's first packet at -5 to -6 dB, its fourth exact. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/audio-plan.md §4–5. Send Audio (HostConfig.sendAudio, off in `standard`, so in the CLI and a fresh Sill.app) goes through kinds 16 and 17 from either door (DeviceSettings), and is not in restartNeeded: the sound has its own stream and never restarts the picture. - AudioCapture: a second, audio-only SCStream per app (the Desktop: every app but Sill's), 48 kHz stereo, Sill's own sound excluded, its 2×2 picture thrown away, start and stop bounded to 2 s, each buffer's layout read (PCMLayout), the first buffer's line and the first minute's (P2). Not run by any test here: it would capture sound. - SyntheticAudio: a synthetic host's test tone, chunked by sample count, SILL_TEST_AUDIO_CHUNK=1024 and SILL_TEST_AUDIO_PAUSE. - AudioPipeline: follows the source from `active`'s didSet (the same app keeps its stream through the picture's restarts; another source is a new epoch; one that ended is tried again after 10 s), one transition at a time; on sill.audio the packetizer, the encoder and the stamps (host time to the wall clock as each packet goes); the format before each epoch's first packet; failures set a note and never touch the picture. - StreamServer: each device's codec from its hello; kind 29 only to those that play aac-eld, the epoch's format first, on a send path of its own: never inflight, the drain eviction or the keyframes' bookkeeping; away its bytes count in pendingBytes (paceRemote still drops only frames) and lastSentAt skips the TLS ticks. A cap skips a packet only past a second untaken at home, or the drain backstop's 15 s away (3 s, the plan's, cut a fifth of the sound at 4 Mbit/s with 1.5 MB keyframes: the picture itself was 5 s behind). - StreamCoordinator: the key (an app by the staged placement's pid or the catalog's owner, the Desktop, a synthetic host's tone, never a synthetic host's real window, never Sill), the sources' filters, the status (HostStatus.audio) and kind 16's audioNote; "Audio stopped: Sill is quitting" at quit. - The CLI's --audio (one more startup line; without it nothing new); Sill.app's `sendAudio` default (off), for the menu in the next commit. - sillclient.py --audio (a hello with the codec, kind 29 counted, an AUDIO line) and sendAudio in --set and --expect; the encoder-free harness, Scripts/audio (the real StreamServer and sound files with the tone and fake frames, both doors on 127.0.0.1) and its device, Scripts/audiocheck.swift (every packet decoded, each click found at its stamp); the pacing harness's --audio and --sound (H6). Verified before main's merge: the harness at home, 100.03 packets a second, no seq gap, 60 of 60 clicks within 1 ms of their second (0.0 to 0.3 ms), the picture's net.sent, net.dropped and frame age as without sound; 1024-frame chunks 93.84 a second; the pauses and a source change on two devices (segments at their stamps, each epoch's format first, nothing over 40 ms late); away through 8 Mbit/s with 1.5 MB keyframes no packet skipped while frames were, packets no later than the frames; the dip late together and back together. The rest of the gates run on the merge with main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/audio-plan.md §6: the status menu's Send Audio after Resolution
(checked, "The Mac keeps playing it too", setting the value it showed),
a Send audio section in Settings › Streaming after encoding speed with
its footer and, while the sound fails, the orange line ("Sill couldn’t
capture the sound of Safari: …. The picture is unaffected."), the
Permissions pane's Screen Recording words (and their sound when Send
Audio is on), " · sound" ending the card's source row while a device
gets it, -SillSetAfter sendAudio=1, and a `sound` preview sample. The
key, its registered default (off) and its save came with the host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Kind 28 is the gesture's, 29 the sound's, 30 and up unknown. Conflicts: StreamMessage.swift and Compatibility.swift (both kinds, in order); the kind checks (protocol, menus, pointer-control and compatibility say 28 gesture, 29 audio, 30 unknown; compatibility keeps both sides' cases, its "29 not this build's" now "29 the Mac's sound"); the mutants that renumbered a kind onto 29 or 30 stay on 250-254, and main's "kind 28 as 29" is "kind 28 as 250"; Tests/checks/README.md's open-branch paragraph now the sound's. StreamCoordinator (the gestures' handler beside the sound's calls), sillclient.py, ci.yml and MockCatalog merged by themselves. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A source that could not start said so with Swift's description of the error, which for ScreenCaptureKit's NSErrors is a multi-line dump of the domain, code and user info. AudioCapture's own failures (its bounded start and stop) now give their words (AudioSourceFailure), any other error its localized description with its domain and code (-3801 for a missing permission, -3818 for sound that failed to start), in the log's "Audio: could not capture …" line and in the note devices show. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/audio-plan.md gains its Results for steps 1 to 5: the commits, where the build departs from the plan and why (the away cap at 15 s, the dip judged against the picture's own recovery, the sound's own time in "no later than the frames", both tones pausing, 48 kHz stereo only, no real window's sound on a test host, the log lines' order, the decoder's shape, and H12 not met, with open question 16), and every gate's numbers. Noah's decision (2026-09-27: "Audio: the plan is done and parked as a v2 feature by your earlier decision") in BRIEF.md and CLAUDE.md's decisions: the sound is in scope. CLAUDE.md: the current step, the floor (kind 29 and its fields if the sound ships in the first public builds), the Layout (Audio.swift, the host's sound files, AudioDecoder outside the project for now, Scripts/audio, the two checks) and Build and run (--audio, the harness, sendAudio among the keys). DEVELOPMENT.md: --audio, "The Mac's sound", the sound harness. Tests/checks/README.md: audio-packetizer and audio-codec, the counts the sound changed, the kinds paragraph. ci.yml: both checks in the mutants matrix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The plan's §4.10: the host's `client …` line gains ", sound 42 ms behind" when the device's report carries audioBehindMs of 0 or more, and ", 3 late" for the packets too late to play. The late packets are summed over the reports since the line before, as the worst frame age and rtt are, so a report the 1.5 s rate limit skips still counts. Nothing changes for a device that sends neither: every device today. Checked with a stand-in device sending the fields: "client statsclient: 60 fps, frame age 9 ms, rtt 7 ms, sound 42 ms behind, 3 late". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scripts/audio/run.sh's cases as their gates judge them (the segments' pauses in both tones; the dip late together and back together), its header within its width. docs/audio-plan.md's Results name the commits after the five steps. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
iOSClient/AudioPlayout.swift (docs/audio-plan.md §7.2), pure: which of the Mac's sound packets are decoded and played, and where each goes on the player's timeline. The floor, the need and its cover (40 ms at home and 150 away until 2 s of packets have come; home 20–150, away 60–600), the picture's lag and the guard a frame behind it, placement by time and packet after packet, late packets, duplicates, gaps and joins on a reset decoder, drift by single frames, jumps over 40 ms, steps, fades and a frame gained or lost at a packet's quietest point. Not yet in the Xcode project (the next commit). Where it departs from the plan, each found by the check: - Only a packet's last 5 ms (its tail, all a fade needs) waits for the next packet; the body goes to the player at once. Held whole, a packet at the window's largest excess raced its predecessor's deadline with no room at all, and a link sitting there (a path a few ms slower, a step's first second, repeated keyframes) faded every packet out and in. Now the next one has 5 ms more, for no latency. - A packet's (arrival − stamp) is taken when the playout takes it, so the hop between the two queues is part of what the cover covers. - The output latency moves the need and the ear alike, so it moves no packet; a route change still owes a jump for the IO buffer's sake. - "A second of packets" above the floor is an unbroken run lasting a second (buckets of the wall's seconds saw a step up to 2 s late), and a step starts the picture's window again with the floor's: frames from both sides of a clock step put the guard a second off. - A jump behind drops packets until one is due within half a packet of what is scheduled; the rest is left to single frames. Tests/checks/audio-playout: a simulated Mac (a drifting, stepping clock, pauses, skipped packets, a new epoch), link (bursts, a stall, away, a move's two connections), decoder (its state after each reset) and player (a drifting output clock read in IO-sized renders): 205 checks with 5,000 random runs holding every invariant, and 36 of 36 mutants caught (the plan's 14 and 22 more). In CI's mutants matrix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The device plays what kind 29 carries (docs/audio-plan.md §7.3–7.5). Its hello lists "aac-eld", so a Mac with Send Audio on sends it the sound. iOSClient/AudioOutput.swift, on its own queue (sill.audio): each format makes the epoch's decoder, and each packet goes through the playout model, then the decoder (its priming dropped, or dropped whole after a reset), then fades and a frame more or less, then an AVAudioPlayerNode → the main mixer → the output. Sill writes no render block. The session is .playback mixing with other audio, 48 kHz and 5 ms asked. It is active only while the engine runs: a format has come, not muted, in the foreground, a packet within 10 s. Mute turns the player down and stops 20 ms later. Interruptions, route changes (headphones out mute and say so), a media services reset and the engine's configuration change are handled. The output latency the model takes is the session's, or the player's presentation latency if more: that one already includes the output's own latency, and adding the two counted the device twice. In the simulator, in every build, the engine renders in manual (offline) mode: no output device is opened, so nothing can reach a speaker. The stand-ins are an IO buffer of 5 ms and an output latency of 10 ms. DEBUG -SillSoundSink manual|device overrides it, and -SillSoundVolume 0 turns the player down besides. In manual rendering a tap on the player (before its volume) finds each click of a synthetic host's test tone and prints where it was heard. StreamClient (+Audio): kind 29 handed to the sound's queue as it comes; each frame's stamp for the picture's lag; the mute (local, remembered as Sill.soundMuted); the session's start (at home or away) and end; the stream's rate and the rtt; the second's numbers into ClientStats (audioBehindMs, audioLate) and the HUD's " · sound 62". A move's hand-over resets nothing. The four new files are in the Xcode project as AA01–AA04, a block no branch uses. The decoder imports CoreAudio on iOS for its buffer list, which AudioToolbox brings with it on macOS only. Checked: iOS Debug for the simulator builds, with only the known StreamClient warning. Live, against SillHost --synthetic --audio on loopback and the software encoder, in a private simulator: every click was heard 0.0–0.2 ms from where it was placed, and within 0.1 ms of its due time once settled. The host's line reads "sound 53 ms behind". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Mac's sound on every layout (docs/audio-plan.md §7.6, §7.7), only while the Mac sends it (its settings on this connection say sendAudio): - Sideways and in the halves, the Sound button comes after Desktop, before Settings, wherever the bar still holds a whole thumbnail with it. The check is MacMenuButton.fits with one more button, after Menus where Menus shows, so where only one of them fits, Menus keeps its place. The Aa ruler fades it with the buttons after Aa. - On a phone upright (and the Duo's outer display), row 1 keeps its approved five. Sound ends row 2 under Settings; while the Mac's menus show too, Menus moves one place left, under Desktop, and the strip ends 8 pt before the first of them (PhonePortraitLayout's sound, menusBesideSound, stripBesideBoth). - The button is a speaker over "Sound", crossed out when muted, never in the accent colour. VoiceOver reads "Sound from ‹Mac›", On or Muted, as a toggle. On the phone its symbol sits in row 1's box and shows large on a long press. - The panel's This iPad group starts with "Sound from ‹Mac›", the same mute as a switch: the only way to it where no bar holds the button (a Slide Over). - Send Audio is a row after the stream rows' closing footer, before Direct Wireless. It restarts nothing, it carries the Mac's note when sound fails, and its footers are the plan's. The ledger gains the field, and rule 9 keeps it from a Mac that never reported it. The mock: -SillSound on|muted; -SillSettingsCase sound, soundoff and soundnote; -SillSettingsScroll sound. Without them the mock Mac has no sound, so every other photo is as it was. ContentView's contract names them and the device's sound arguments. Checks: - phone-portrait: 156, with every width from 300 to 599 pt for row 2 with Sound and Menus, and 35 of 35 mutants (four new). - tour: 43,786. No step lights the Sound button: a phone at every width, and ten sizes sideways and in the halves. - ledger: 99, with Send Audio's cases and random runs against hosts without sound (rule 9). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…left them The workflow was stopped to save usage (2026-09-27); the review and finalize steps never ran. Not reviewed; see docs/audio-plan.md for what remains. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… release) into audio Both sides kept whole, not rebased. Where they meet: - Kind 16 is per connection on main (Saffsanity#39): every connection's state carries sendAudio (streamSettings(away:)) and the Mac's audioNote beside its own away and link. A device away sets Send Audio through applyingAway. - HostConfig's away pair and requirePairing beside sendAudio, which stays out of restartNeeded. - StreamServer: the sound's codecs are taken from the hello on every door, the TLS ones (Saffsanity#37's Door) included. The sound's bytes now count in every client's pendingBytes and takenThisSecond, as main counts every message for the link judge (Saffsanity#39); it still never counts in inflight, is never paced and never touches the keyframes' bookkeeping. - The sound harness (Scripts/audio) builds against main's StreamServer: the Door's files and LinkJudge in its host set, serve(_:route:hello:). - Tests: away-wire's StreamSettings gains sendAudio; the kinds read 28 gesture, 29 audio, 30 unknown; the mutants matrix is both sides' lists. - The Duo's book pose (Saffsanity#47): at 951 pt the bar has no room for Sound beside the fold, so Sound shows there only in the panel's switch. - CLAUDE.md: the sound's entry first in Current step, both sides' Layout and Build and run; docs/audio-plan.md gains "Merged with main at 985a733". Verified: swift build -c release (only the CaptureProbe warning); Tests/checks/run-all.sh, all 43; the iOS app with Xcode 27.0 compiles everything but main's iOS 27.1 calls in DuoPostureReader.swift (a scratch copy with those stubbed builds Debug and Release for the simulator); this Mac has no Xcode 27.1, so CI builds the real one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The move home (PR Saffsanity#39) gives the playout home's jitter bounds and resets nothing else (docs/audio-plan.md §7.2, rules 2 and 7): AudioOutput.setAway was never called. And the Sound button and switch no longer vanish for up to 2 s at each hand-over while the new connection's first kind 16 is on its way: soundAvailable keeps what the last connection said until then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oded while silent AVAudioEngineConfigurationChange was never observed, though docs/audio-plan.md §7.4 asks for it: an output at another rate (AirPods, Bluetooth's call profile, another app changing the session's rate) stops and uninitializes the engine, while `engine` stayed set and packets kept the idle stop away, so the sound stayed silent until a mute and unmute. Now the change stops it on `sill.audio` (never in the handler), and the next packet builds an engine in the output's new format and reads its latencies again. While the sound cannot be heard (muted, in the background, the session refused during a call) the model is muted (§7.5): each packet counts for the floor and the cover and none is decoded. Before, only mute did that, so the background decoded every packet into the model's waiting list, and back in front they were placed late and reported to the Mac as late. Coming back from any of the three resets the playback, as unmuting did. Also: a start's `.playback` reset now discards what waited for the last timeline (it threw the discards away, keeping their samples until the next stop); unmuting tries a session refused during a call again (the person's tap is the resume signal), where only the app coming back to the foreground did; and the DEBUG click finder no longer holds its engine in a closure the engine keeps (a leak of every stopped engine, devices' Debug builds too), and is set only in manual rendering. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd at a session's end too docs/audio-plan.md §7.4: the engine stops after a fade on mute, on didEnterBackground and at the end of the session. A mute turned the player to volume 0 and stopped the engine 20 ms later; the background and the end of a session stopped it at once, a cut. Measured in manual rendering (the engine's offline mode: no output device, nothing heard): the main mixer ramps a change of the player's volume linearly over 1,200 of its frames, 25 ms at 48 kHz, whatever the render's size (240 or 1,024 frames), so the stop at 20 ms cut the sound at a fifth of its level. The fade is that ramp now, and the engine stops 50 ms after it starts, on mute, in the background and at a session's end (a reconnect's start too). An interruption and a configuration change find the engine stopped already, and the idle stop follows rule 10's faded tail. The plan's 5 ms fade is the mixer's 25 ms: the player node offers nothing finer without a render block of Sill's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md §7.8 has the line "audio: jump +52 ms (clock step)"; the jumps were only counted in the once-a-second line. AudioPlayout now keeps why a jump was owed (rule 13's step, rule 12's error over 40 ms or a sound ahead of the picture, a route's new latency) while it counts, and the next placement by time measures it: where that packet goes against where it would have followed (positive, placed later with silence before it; negative, packets dropped). `takeJump` hands it over once, and AudioOutput prints it as the outputs are applied (DEBUG). The audio-playout check takes every report and holds the clock steps to them: each jump reported once; the Mac's clock a second forward, one clock step of nothing; a second back, the second dropped, then the clock step. M33's pattern follows `owe(.step)`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`adopt` started every remote session's sound away (docs/audio-plan.md §7.2, rule 2: a 150 ms cover to start, 60–600 ms bounds), including one whose way in is `.address`: a LAN address through the remote door, which the Mac's AwayPolicy counts as home. Such a session started 110 ms later than it needed to and kept away's floor of 60 ms. Now only a route through a VPN or over the internet is away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md §7.6: Sound goes where the bar holds it after Menus, so where only one fits (an iPhone SE sideways, 639 pt of row; iPad windows under 532 pt upright) Menus keeps its place and Sound is the Settings panel's switch alone. The bars counted Menus only while the Mac's menus showed, so on those bars Sound showed until the menus' top level came and then went (a reflow after each connect, and with every app without menus). A Mac that sends its sound takes menus, so Menus is counted always, as the book pose's bar already counts it (BarMetrics.clearing). In the book pose the buttons past the fold hold Menus and four more: Sound is the panel's switch alone there, whatever the menus. The tour check's Sound case gains the iPhone Duo's sizes: its inner display both ways and its cover sideways (the cover upright is the phone grid's). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndom runs' line audio-playout: the move home (PR Saffsanity#39) was a scenario that only changed the bounds; now also as it happens, a hand-over with both connections carrying the stream for a moment, away (80 ms) on the old one and home (5 ms) on the new: every copy dropped as a duplicate and counted nowhere, no join, at most one jump, nothing late, home's bounds after. And the random runs' summary said "every invariant held" whenever any run happened; it now passes only when none of them failed. phone-portrait: the grid holds row 2 with Sound and Menus to two whole thumbnails at every width from 375 pt (§11 H3; only the SE was checked). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mes H12's known failure audiocheck exited 0 when its connection never became ready (a refused or failing connection printed a summary of zeros), so run.sh's count of runs that could not run missed it, and a case gated on nothing passed silently. It exits 3 now, and run.py fails the run when a device did. run.sh's --help printed its header but the last case's description. The header names H12's CPU gate as the known failure it is until Noah answers open question 16 (+2.1 to +3.1 points with codec quality Medium, docs/audio-plan.md's Results), so a full run's nonzero status is not read as a regression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S2's live run found every click of the test tone until the first mute, and none after: a new engine's timeline starts at sample 0, while the finder kept the last engine's rendered samples and their start, so it looked for the clicks at the old timeline's places. It is made anew at each engine's start, as at a session's. DEBUG only (manual rendering). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md's "Results: the device": the device's steps (6878c11, aefe477, 5f7b1ac, 9288aa6), the review against the plan and its fixes (012c5de to c2a1802), where the build departs from the plan and why (rule 10's held tail, the output latency's max, packets stamped on `sill.audio`, AA04, the harness's arguments, the mixer's 25 ms fade, a configuration change rebuilding the engine, Sound counting Menus and the book pose's panel switch, the simulator's offline rendering, S2 and S3 against the synthetic host), and the gates: H1 from a scratch copy with the iOS 27.1 names stubbed, H3, H5–H8 with H12 still failing (open question 16), H6 with sound on the merge and the pacing gates against main, H2, H10, H11, S1 (273 photos, every photo without the sound pixel for pixel main's), S2 and S3. The merge's §7.6 note says what the review changed. CLAUDE.md: the entry's state, the device's design in brief, the results and the harness's sound arguments; docs/DEVELOPMENT.md: the device's side of the Mac's sound; Tests/checks/README.md: audio-playout's 214 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ledger had no mutants since its old ones (whole copies of an older ledger) were dropped, so nothing held its check to the rules it states, the sound's seventh field among them. Twelve one-line mutants of HostSettingsLedger.swift and of the merge it uses in HostSettings.swift: rule 9 for Send Audio and for Direct Wireless, the field list, the single field and the merge without Send Audio, rules 2, 4, 5, 6 (the timeout itself) and 7, and isEmpty and applied(to:) without Send Audio. All 12 caught. run.sh --mutants runs them, CI's mutants matrix lists ledger, and the README's table says 12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de Send Audio Main gained Saffsanity#136 (the Apps list finds apps in folders of their own, at any depth: InstalledApps and its check). Two conflicts, both lists: - ci.yml's mutants matrix: both sides' checks, installed-apps among them, with the sound's audio-codec, audio-packetizer, audio-playout and ledger. - Tests/checks/README.md's history: main's installed-apps sentence, then the audio branch's, which now also says what the beta changed in phone-portrait, tour and away-quality. CLAUDE.md merged by itself (InstalledApps in the Layout and the checks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HunterAhlquist
pushed a commit
to HunterAhlquist/sill
that referenced
this pull request
Oct 19, 2026
Kind 29 was free on 985a733, where the pen was built; since, main's comment and checks keep 29 to 31 free (29 the audio plan's, 30 and 31 Saffsanity#66's side streams), 32 is the clipboard, and open pull requests hold 33 to 44 (Saffsanity#66's 33 to 42, Saffsanity#94's 43, Saffsanity#93's keyframeRequest 44, checked against every open head today). So `case pen = 45`, and everything that names the number follows: SessionLink.isInput (8 or 45), the comments of Pen.swift, Switcher, PenTablet, InputInjector and StreamServer (which match the enum, not the number), sillclient.py's --raw45 and its sends, and the docs (DEVELOPMENT.md, CLAUDE.md's entry, Layout and Build and run, the plan's design and gates, with a note that its results before the rebase say 29). The compatibility floor's kinds are 0–28, 32 and 45 again with 29 to 31 free as main has them; kind 26's `seen` counts kinds 8 and 45. The checks: compatibility names kind 45 and 46 unknown (157 checks; its mutant "kind 45 as 46"), protocol keeps main's two kind checks and has the pen at 45 with 46 unknown (193 + 8), pointer-control's table and sweep say 45 (158), fence's input and its two C10 mutants say 45; menus' line and mutant and pointer-control's two renumbering mutants are main's again, 29 being free once more. clipboard passes unchanged. Run here: compatibility 157, protocol 193 and the 8 cross-checks, pointer-control 158, menus 309, clipboard 259, pen 197 and 5,000 random sessions, all passing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The beta's three dated lines (CLAUDE.md's decision, BRIEF.md, the plan's header) said "Noah on PR Saffsanity#94", which reads as a comment on the pull request; his "merge as beta support" was an instruction about it, and the pull request has no such comment from him. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ffsanity#88 beside Send Audio Main gained Saffsanity#110 (Shift+Return and Shift+Tab from an iPad's keyboard reach the Mac with the shift: InputOverlay, KeyChords and the key-strokes check), Saffsanity#87 (VirtualDisplay's BOOL return read as "B" or "c", with its new virtual-display-api check) and Saffsanity#88 (Sill.app universal with SILL_UNIVERSAL=1, release.sh checking every slice). Two conflicts, both lists, both sides kept: - ci.yml's mutants matrix: the sorted union, 49 checks, which are exactly the folders with a mutants.py (the sound's audio-codec, audio-packetizer, audio-playout and ledger, and main's virtual-display-api). - Tests/checks/README.md: the opening list names both the sound's checks and the virtual display's encodings; the history keeps main's virtual-display-api sentence, then the audio branch's. CLAUDE.md and docs/DEVELOPMENT.md merged by themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as a beta "Plays sound from your Mac only as a beta" read as a limit on where the sound plays; the line now says what it does and that it is a beta, off by default, in README's own words for the two ends (your Mac, your device). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2e65f06 stopped handing a frame to the sound's queue until the session's first kind 43 format, so a session without sound pays no hop and no sorted insert per frame. But the guard (rule 3: the sound never ahead of the picture) then had no frame when the first packets were placed, so with a picture that trails by more than the need (away, a slow link) the first packets could play ahead of it until frames came, and the delay then jumped. The output now keeps the stamps of the last 32 frames (about half a second at 60 fps) under the same lock until the first format, which hands them to the model before itself: the guard holds from the sound's first packet, as when every frame went to the model, and a session without sound still hands its queue nothing (a lock and an append per frame). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…finish) Work in progress, paused at the maintainer's request. The plan's header pointed at a last section, "The beta", that did not exist, and CLAUDE.md's audio entry at a last bullet it did not have. Both now say what the beta changed (one line a commit), the defaults it took (Q8 the Mac's alone, Q10 no Sound in Larger Picture's row 1, Q4 and Q5 kept, Q16 with H12 unmet at +4.0 points and accepted while Send Audio is off by default, the background's packets still in the cover) and what was verified on 9104023 with main at 4d990e9: Xcode 27.1's three iOS builds, run-all.sh's 49, mutants of away-quality, audio-codec, audio-packetizer, ledger and away-wire, H2, H7 as the beta makes it, the sound harness, H10's previews, and S1 to S3 on a private iPad simulator. CLAUDE.md's Layout line for AudioOutput says it keeps the last 32 frames before the first format. What remains: the mutants of audio-playout (21 of 38 caught when stopped) and tour (14 of 84), none missed so far; run both to the end, put their counts in place of the "stopped unfinished" words in both files, then push to the pull request's branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 20, 2026
…f 84)
The beta's record in docs/audio-plan.md ("The beta", Checks) and CLAUDE.md
said the mutants of audio-playout and tour had stopped unfinished at a
pause (21 of 38 and 14 of 84 caught, none missed). Both ran to the end on
2026-10-26 from a fresh worktree of the same sources (2f5c1cf: the commit
since 9104023 changes only these two documents): audio-playout 38 of 38
(no run timed out), tour 84 of 84, after both checks passed unmutated
(218 and 111,191) and `swift build -c release` had only the CaptureProbe
warning. The placeholders give way to the counts; "the same morning" for
phone-portrait's run now names its date, 2026-10-20.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Send Audio Main moved while the beta's last mutant runs ran: PR Saffsanity#151 (a sleeping display wakes for a device, issue Saffsanity#134). One merge commit, no rebase. Conflicts, each settled keeping both sides whole: - ci.yml's mutants matrix: the union (display-wake beside the three audio checks and ledger), 50 checks, exactly those with a mutants.py. - CLAUDE.md: both Current step entries, the sound's first, then Saffsanity#151's; the Layout's two paragraphs one after the other; the checks' list with both sides' names. - Tests/checks/README.md: the opening paragraph names both the sound's checks and the display wake's; the history paragraph both. - docs/release-checklist.md part 1 §3: the Clipboard, Display wake and Sound markers in one sentence, and the three perl lines. StreamCoordinator.swift, DEVELOPMENT.md and site/support.html merged by themselves. Where the two meet: display-wake's check counts every message kind and fails until a new one is decided ("every kind there is decided"). Kind 43, the Mac's sound, is the 32nd and only a host sends it, so DisplayWake's `default: return nil` already decides it as no person's: the check now counts 32 kinds and lists 43 among the kinds that never declare (193 checks), with a mutant that makes it count (M61, 61 in all). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md's "The beta" and CLAUDE.md's beta bullet name the merge with 7b4c313 (Saffsanity#151, a sleeping display woken for a device): what met (ci.yml's matrix, CLAUDE.md, the checks README, the release checklist's Clipboard, Display wake and Sound markers) and the one place the two features touch, display-wake's check now counting kind 43 among the kinds that never declare (193 checks, M61). Verified on the merge (2026-10-26, a fresh worktree): `swift build -c release` from `git archive`, only the CaptureProbe warning; run-all.sh, all 50; display-wake's mutants, 61 of 61; H2 against main's CLI at 7b4c313, the host's output identical masked and sorted idle, with a Desktop client and with one whose hello plays sound; H7 (a device's Send Audio refused on and off; the CLI's --audio tone, 979 packets with no seq gap, and its stop; a second client's off refused). The iOS app's sources and StreamProtocol are 9104023's and the sound harness compiles none of the merged files, so the earlier iOS builds, S1-S3 and harness numbers stand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity#93's kind 44 (keyframeRequest: a device's ask for a keyframe after its picture was interrupted) and its Close timeout, and Saffsanity#227's SILL_TEST_NO_SCREEN, beside the sound's kind 43. Saffsanity#151 has been in since 7b4c313, the merge base; its display-wake check is where Saffsanity#93's 44 meets the sound's 43. Nine files conflicted, each resolved by keeping both sides: - .github/workflows/ci.yml: the mutants matrix, kept sorted, keyframe-recovery added beside the sound's audio-codec, audio-packetizer, audio-playout and ledger (51 checks). - CLAUDE.md, five hunks: the Current step's entries (the sound's first, then Saffsanity#93's, then Saffsanity#151's); the Layout's StreamProtocol paragraph (Audio.swift, and kind 44 with no type of its own), its Scripts paragraph (the sound harness, and the restore probe's close step) and its checks list (keyframe-recovery and ledger); the compatibility floor's kinds (0–28, 32 and 44 kept, 43 the sound's below, 29 to 31 and 33 to 42 free). - Sources/StreamProtocol/StreamMessage.swift: 43 audio, then 44 keyframeRequest. - Tests/checks/README.md, four hunks: the opening list (the keyframe ask and the sound's packets), display-wake's row (43 and 44 never declare; 196 checks, 62 mutants), keyframe-recovery's row beside ledger's (the sound's 99 and 12), phone-portrait's row as the sound's (495 and 80), and pointer-control's with 43 and 44 (159). - Tests/checks/display-wake/main.swift and mutants.py: 43 and 44 both among the kinds that never declare, and both mutants, M61 (the Mac's sound counts) and M21b (a keyframe request counts). - Tests/checks/pointer-control/main.swift: the kind table with 32, 43 and 44, every other kind (29–31, 33…42, 45…254) unknown. - docs/release-checklist.md: the paragraph on marked words written once, for the Clipboard, Keyframe, Display wake and Sound blocks, where main's copy repeated itself (Saffsanity#93's with Saffsanity#151's), with the four perl lines. - iOSClient/StreamClient.swift: init ends with sound.setMuted, onRouteMuted, then observePictureActivation. Four checks git merged without a conflict failed on the result, fixed here: compatibility (main.swift:258), menus (main.swift:26) and protocol (main.swift:370) asserted that kind 44 is unknown, and it is .keyframeRequest now; display-wake (main.swift:104) counted 32 kinds, and there are 33. Now display-wake has 196 checks and 62 mutants, pointer-control 159, and Tests/checks/run-all.sh passes all 51. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he row-1 Sound button - CLAUDE.md, the Layout's PhonePortraitLayout: with Larger Picture, Menus in row 1 and no Sound button, the Settings panel's switch alone (6effec7 took the row-1 Sound and `row1Beside` out): `sound` and `stripBesideBoth` are the strip's empty place, `menusBesideSound` is `menus`. - docs/audio-plan.md: "Since the beta" notes where the body still had the design before it, as §4.1 and §7.7 have: Controls' Send Audio (Q8: the Mac's alone, a device's change refused and named in the log, the panel's row read-only) and its Sound button (Q10: none in Larger Picture's row 1, the panel's switch alone there); §5's "A device's change", which there is none of now; §13's hard rule (only Menus joins Larger Picture's row 1); and "Merged with main at 5c323e6", which left the row-1 Sound to Noah, says the beta took it out again. - docs/app-store-metadata.md: "Sill doesn’t play sound from your Mac." goes from any version whose build carries the sound, with or without the sound's bullet, which waits for P1–P16: a device plays the sound once the Mac turns on Send Audio, so a version that shipped the beta before the device tests passed would have kept a false line (Before you submit, the description's lead-in, the next version's note). The counts gain the block without that line and before the bullet: 3,003 characters. - Tests/checks/README.md: since the merge with main at 36e7212 (Saffsanity#93), the audio bullet's kind checks expect 44 to be the keyframe request, and 45 and up unknown; it still said 44 unknown. Docs only: no check reads these files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity#159's Reverse Scrolling (issue Saffsanity#143) beside the sound: the device's own switch (Sill.reverseScroll, never sent) turns a finger's drag over the picture and two fingers on the on-screen trackpad around, coast included (StreamProtocol's ScrollDirection, with protocol's 13 cases and 8 mutants), the touch rig and InputScript's drag and padscroll, and the host's TEST ONLY scroll log line with its deltas and phase (InputInjector.swift, which this branch never touched). No wire change, and no host file both sides changed: the sound's StreamServer, StreamCoordinator and pipeline are as they were. Six files conflicted, each resolved by keeping both sides: - iOSClient/HostSettingsPanel.swift, four hunks. The This iPad group merged by itself in Saffsanity#159's order (Sound from ‹Mac› first, Larger Picture, Share Clipboard, Reverse Scrolling, Three-Finger Gestures). Saffsanity#159's -SillSettingsScroll reverse becomes a case of this branch's startsAt switch (reverseScrollRowID) in place of its startsAtReverseScroll and second scrollTo, whose other line named startsAtGestures, which this branch had folded into startsAt; one onAppear scrolls to whichever the argument names. The type's and the group's doc comments name both sides' rows. - iOSClient/ContentView.swift: the harness contract names -SillSettingsScroll sound and reverse. - CLAUDE.md: the Layout's StreamProtocol paragraph has Audio.swift, then ScrollDirection.swift, then kind 44's sentence once. Saffsanity#159's Layout and Build and run lines and the Current step merged by themselves (Saffsanity#159 has no Current step entry). - Tests/checks/README.md, three hunks: the opening list names Reverse Scrolling and the sound's packets; protocol's row both descriptions, 234 + 8 checks and 42 mutants (the sound's 221 + 8 and 34 with Reverse Scrolling's 13 and 8, from the merged check's run); the history paragraph both clauses, with a sentence on this merge's counts. - docs/release-checklist.md: the paragraph on marked words names each marker once (Clipboard, Keyframe, Display wake, Sound, Reverse Scrolling) with each one's grep, where both sides had rewritten main's repeated paragraph; every perl line kept, Sound's, then Reverse Scrolling's. - docs/app-store-metadata.md: both bullets, the sound's first; the next What's New's draft with both paragraphs is 1,322 characters. site/privacy.html and site/support.html merged by themselves, each pull request's words inside its own markers; with all five blocks taken out as a republish before the releases would, the checklist's greps print what it says (one clipboard line, nothing else). README.md, docs/DEVELOPMENT.md, iOSClient/StreamClient.swift and protocol's main.swift and mutants.py merged by themselves, each side's lines whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 27, 2026
Main gained Send Audio as a beta (Saffsanity#94, the audio branch: kind 43, the host's capture, encoder and packetizer, the device's decoder, playout and output, the Send Audio row, the audio-codec, audio-packetizer and audio-playout checks and ledger's mutants). Where it met the connect screen's tiles and groups (Saffsanity#117): - .github/workflows/ci.yml: the mutants matrix is the sorted union, connection-list beside main's audio-codec, audio-packetizer, audio-playout and ledger (52 checks). - CLAUDE.md: Layout's iOSClient paragraph keeps the tiles' sentence after the tour's line, then main's AudioDecoder, AudioPlayout, AudioOutput and StreamClient+Audio lines; the checks list keeps main's re-wrapped audio and away names, then connection-list before device-gate. Main's Current step entry for the sound and its Build and run lines merged by themselves (this branch has no Current step entry). - Tests/checks/README.md: main's compatibility row (the sound's JSON, 140 cases and 34 mutants), then the connection-list row; the history keeps "connection-list with Saffsanity#117 (Saffsanity#116)" in its first sentence, then main's audio sentences whole. Main's lede phrase for the sound merged by itself. - docs/release-checklist.md: the marked-words paragraph is main's wording (a republish before those releases deletes the blocks of each release not out yet) naming Clipboard, Keyframe, Display wake, Sound (PR Saffsanity#94), Tiles (PR Saffsanity#117) and Reverse Scrolling once each, with both greps ('sound|send audio', 'tiles|group name'); the perl lines keep Sound's and Tiles', six in all. On a copy of the merged site/, the six lines leave no marker and the greps print what the paragraph says. Merged by themselves, each differing from the other parent by exactly that side's hunks: iOSClient/ContentView.swift (main's scene phase for the sound and its harness lines, the tiles' connect screen and harness lines), MockCatalog.swift (main's sound settings cases, the tiles connect case), StreamClient.swift (main's sound in init, the moves, the reader and tearDown; the tiles' groups, previews and tearDown's picture, taken before the sound stops), the project file (AA01-AA04 and FA01-FA04 beside AF01/FF01, each defined once, plutil OK), docs/DEVELOPMENT.md, and site/privacy.html (the Sound items inside their markers, the Tiles item inside its own). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 27, 2026
…5bb5 4bb5bb5 is the merge of main at 74d099c (Saffsanity#94, Send Audio as a beta) into the pull request's head 46684c1, with its resolutions (ci.yml's matrix, CLAUDE.md, the checks README, DEVELOPMENT.md, the release checklist, the project file, privacy.html and support.html). Merged here into the two commits on 46684c1 that take the tile's picture from the stream's last keyframe. One conflict: StreamClient.handle's frame case keeps both, this branch's keyframe store (and Disconnect's wait for it) and main's frameForSound before the display layer's frame. The tree is 4bb5bb5's plus exactly those two commits' changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to scripsweave/sill
that referenced
this pull request
Oct 27, 2026
…affsanity#93, Saffsanity#151) into fix/remote-desktop-qr-code Main brought Send Audio as a beta (Saffsanity#94), Reverse Scrolling (Saffsanity#159), SILL_TEST_NO_SCREEN (Saffsanity#227), the Close timeout and the interrupted picture (Saffsanity#93) and a sleeping display woken for a device (Saffsanity#151). Six files conflicted, each resolved keeping both sides: - .github/workflows/ci.yml: the mutants matrix is main's 51 checks with pairing-sharing, sorted: 52, exactly the folders of Tests/checks with a mutants.py. - Sources/SillMenuBar/HostSettings.swift: Saffsanity#94's sendAudio beside showPairingWindowToScreenSharing in Key and in the registered defaults (sendAudio first, a HostConfig knob next to shareClipboard), as Noah's comment of 2026-10-19 settled the overlap: whichever lands second keeps both. HostConfig's sendAudio, read and saved, and this switch's property merged by themselves. - Sources/SillMenuBar/DebugHooks.swift: -SillSetAfter's list names both keys (sendAudio, and showPairingWindowToScreenSharing=0|1; shareClipboard, which both sides had added, once); both cases of apply merged by themselves. - Tests/checks/README.md: main's rewrapped introduction with this PR's "whether screen sharing shows the pairing window"; the pairing-sharing row beside main's phone-portrait and pointer-control rows; the history's list of checks and their pull requests ends with pairing-sharing and Saffsanity#157. - docs/release-checklist.md: the marker paragraph names every marker once (Clipboard, Keyframe, Display wake, Sound, Reverse Scrolling, Pairing window), the screen sharing grep among those that print nothing, and all six perl lines. - CLAUDE.md: both Current step entries, this PR's first, then main's three; Build and run's defaults list with sendAudio and showPairingWindowToScreenSharing; this PR's entry names this merge, and its verification is this merge's (Tests/checks/run-all.sh, all 52; the previews against main's 74d099c, the Devices pane's eight differ, two new, the other 136 byte for byte). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to cinminori/sill
that referenced
this pull request
Oct 27, 2026
…input Both sides kept where they met: CLAUDE.md's Current step has Saffsanity#147's entry, then the sound's (its header now says main at 74d099c); the checks README names what an input method or dictation sends beside the sound's checks, and keeps text-composition's row with main's tour row; CI's mutants matrix is the union, sorted (52 checks: the three audio checks and ledger from main, text-composition from here); docs/release-checklist.md's markers paragraph names every marker once (Other languages and Sound among them), with every grep and every perl line; the next What's New has both paragraphs, the languages' before the sound's, 1,571 characters, and the metadata's bullet says so. site/support.html and privacy.html merged by themselves, each block inside its own markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 27, 2026
…e-sleep Three files conflicted, each kept whole on both sides: - CLAUDE.md: both Current step entries, this branch's first, then main's "The Mac's sound on the device". - Sources/SillHost/StreamServer.swift: Client's fields, `told` beside Saffsanity#94's audioCodec, audioEpochSent and audioUnsent. - Tests/checks/README.md: this branch's policy row (507, 201) beside main's protocol row (234 + 8, 42). The sound's own send path (broadcastAudio(packets:), sendAudio) does not check `told` yet; the next commit makes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 27, 2026
Saffsanity#94's sound has a send path of its own (broadcastAudio(packets:) and sendAudio, a direct connection.send) that checked no `told`, so after the merge a connection told why it closes (the Mac's sleep, a Quit, Remove) could still be sent kind 43 behind its goodbye, against `told`'s rule and, after the sleep's goodbye, behind its .finalMessage. The packets' loop now skips a told client, and sendAudio guards as `send` does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
that referenced
this pull request
Oct 27, 2026
Noah's decision of 2026-10-27 ("Sound should be a toggle in the settings
menu") in the words:
- docs/audio-plan.md: the header, Controls, §4.1, §3.5, §5, §6, §7.6,
§7.7, §7.8, H7, S2, Q5, Q8 and Q10 marked where they changed, and a
last section, "Two switches in the device's Settings panel", with the
changes, the checks and what was verified.
- CLAUDE.md: a bullet in #94's entry, its superseded lines marked, the
Layout's PhonePortraitLayout and StreamClient+Audio lines, Build and
run's --audio and harness lines, the Decisions' audio line.
- README, site/support.html and privacy.html inside their Sound markers
(a device's settings panel turns it on; Mute silences one device;
nothing new collected; privacy.html's date October 27, 2026),
docs/app-store-metadata.md (the bullet, What's New and their counts;
the review notes as they were), BRIEF.md and DEVELOPMENT.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 27, 2026
One conflict, PortraitStreamScreen's windowBar: main's Sound button (Saffsanity#94) beside Saffsanity#220's Keyboard button. Both kept: while the software keyboard is up the Keyboard button counts in the row, so Menus fits with six buttons and Sound with seven, as each counted one fewer before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #83.
This finishes Noah's
audiobranch, which stopped at 9288aa6 partway through the device step. Main at 985a733 is merged in with one merge commit, not a rebase. Everything followsdocs/audio-plan.md. Its Results sections record each gate, each departure from the plan and the review.What it does
The device plays the sound of what it streams: the picked window's app, or every app for the Desktop. The sound stays in step with the picture and never runs ahead of it, and nothing in the picture waits for it.
audio(host to device; it was 29 until Sill Viewer: a Mac controlling another Mac #66's 29–42 turned up, see 6b0911c) carries a JSONAudioFormatand binaryAudioPackets. It goes only to a device whose hello listsaac-eld. Kinds 16 and 17 gainsendAudio, kind 16 gainsaudioNote, and ClientStats gainsaudioBehindMsandaudioLate.SillProtocolstays 1. Older devices and older Macs see nothing new.HostConfig.sendAudio) is off by default and isn't inrestartNeeded, so turning it on never restarts the picture.excludesCurrentProcessAudio.--syntheticuses a test tone instead.inflightand never paced. Its bytes count inpendingBytesandtakenThisSecond, so the link judge and remote pacing see it.--audio.AudioPlayout, a pure model: the floor, the jitter cover, the picture's lag and the guard, dedupe, placement, drift corrected by single frames, and jumps over 40 ms.AudioDecoder(AudioToolbox).AudioOutput: AVAudioEngine with anAVAudioPlayerNode, session.playback+.mixWithOthers, interruptions, route changes, configuration changes, and an idle stop after 10 s.Sill.soundMuted). Sound continues across every hand-over and the move home.Defaults taken (the plan's open questions; each is one place to change)
.playbackwith.mixWithOthers: the device's own music is never stopped.Verified here
Nothing ran on a device, and nothing captured or played this Mac's sound. Every host ran on loopback with the software encoder (
SILL_TEST_LOOPBACK=1 SILL_TEST_SOFTWARE_ENCODER=1 SILL_TEST_NO_ROUTER=1), and every simulator run used-SillSoundSink manual -SillSoundVolume 0.swift build -c release: only the CaptureProbe warning. iOS Debug and Release for the simulator and Debug for a generic device: only the knownStreamClientcapture warning. These were built with Xcode 27.0 against a scratch copy that stubs main's 27.1 Duo APIs; see Not verified.Tests/checks/run-all.sh: all 43 pass, including the newaudio-playout(218),audio-packetizerandaudio-codec.audio-playout38 of 38).ledgerhas mutants again (12; it had none).Scripts/audio(no encoder in the picture's path).Scripts/pacing --sound. real24, kf25m32, restartkf, stillend and the link verdicts pass. over8 reads 4.8 against 4.9 fps: the sound is 1.9 % of that overfilled 8 Mbit/s link, and drops are unchanged.aud.key, no kind 43.menu.txt, the Streaming and Permissions panes, and the two new sound cards.SillHost --synthetic --audio. The click finder placed clicks within 0.2 ms, with no late packets. Mute, the panel switch, background and return, and Send Audio turned off from a second client all behave as specified.Not verified, for Noah
sill.audiofor seconds on a loaded Mac, giving a burst of late packets at the start.docs/app-store-metadata.mdandsite/privacy.html.