Repository navigation
Conversation
From the roadmap (item 5, smoothing). A spike showed the system's video layer ignores a layer's scaling filter (linear and nearest gave the same pixels, byte for byte), so every level but Automatic now draws with Sill's own renderer: - Settings › This iPad › Picture Smoothing: Automatic, None, Low, Medium, High (Windows App's interpolation levels), this device's own, never sent (Sill.pictureSmoothing). Automatic is the system's video layer, as before, untouched. - MetalStreamRenderer: the stream decoded with VideoToolbox to BGRA and each frame drawn with Metal as it decodes, in the picture's aspect-fit rectangle under the pointer sprite, in two passes (across into an rgba16Float texture, then down, clamped): None the nearest pixel, Low bilinear, Medium Catmull-Rom, High Lanczos-3, each widened by the shrink when the picture is smaller than the stream (the usual case: a Retina stream on an iPad), so fine detail averages out instead of aliasing. The layer takes the stream's colour space. No Metal or no shader: the system's layer draws, with one line. The stream's queue never waits on a drawable. - A change of who draws picks the source again for a keyframe; a change of filter only draws the last frame again. - StreamShader: the Metal source, compiled when the renderer starts. PictureSmoothing: the levels, the kernels and a CPU reference resampler. pbxproj B21–B23. - DEBUG -SillSmoothingAt '<s> <level>[; …]' changes the level as the picker does. Checks: Tests/checks/picture-smoothing, 45 checks of the model and reference and 65 of the shader on this Mac's GPU against the reference, pixel for pixel (every filter, across and down, growing, shrinking and the same size; it passes, saying so, without a Metal device), 16 of 16 mutants of both files; in CI's mutants matrix; run-all.sh all passed. On an iPad Air simulator (iOS 27.0) against SillHost --synthetic with a still picture of 8-pixel stripes: in-between greys Automatic 7.8 %, None 0.0 %, Low 7.8 %, Medium and High 7.0 %; switching Automatic, None, High, Automatic mid-stream, the picture back after each switch. Not measured: latency against the system layer, power, and a device.
Found in the review before this push: every frame that failed to decode printed a line, which for a stream the decoder cannot read is 60 lines a second in the log. Now once per stream (reset with each new format).
…, clear to black Review findings 1–5 on PR Saffsanity#107: 1. iOS invalidates a hardware VTDecompressionSession in the background (-12903 / -12911). MetalStreamRenderer now stores the format description, recreates the session when broken (PictureSmoothing.sessionBroken), and asks the Mac for a keyframe at most once per 2 s while the app is active (PictureSmoothing.DecoderRecovery). 2. setFormat and clear invalidate the old session before clearing `last`, and clear the CAMetalLayer with a Metal clear pass (`drawBlack`) so the old window does not stay frozen on screen and `layer.contents` is never overwritten. 3. Synchronous decode on the stream's serial queue (`flags: [._1xRealTimePlayback]`) so frames draw in strict arrival order; layout and filter redraws hop to that same queue so `nextDrawable()` never blocks the main thread and `drawableSize` is updated in order, with `layer.colorspace` set on main. 4. `HEVCDisplayView.didMoveToWindow()` marks layout dirty so a saved smoothing level on a 3× iPhone updates from the initial 2× fallback scale. 5. Removed the stray blank line in `PictureSmoothing.swift`. Checks: picture-smoothing 52 + 65 checks and 22 of 22 mutants (6 new for sessionBroken and DecoderRecovery); live simulator switch test (Automatic → None → High → Automatic) verified.
# Conflicts: # .github/workflows/ci.yml # iOSClient/StreamClient.swift
# Conflicts: # .github/workflows/ci.yml # Tests/checks/README.md # iOSClient/HostSettingsPanel.swift
This was referenced Oct 15, 2026
Owner
|
From a pass over all the open pull requests (nothing is decided yet): #93 also adds a keyframe re-pick for a decoder interrupted in the background ( |
This was referenced Oct 15, 2026
# Conflicts: # .github/workflows/ci.yml
…affsanity#93; framebufferOnly false for snapshots
Contributor
Author
|
Thanks. Pushed 7c7651e (and merged main in 63c7c28): |
This was referenced Oct 19, 2026
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 19, 2026
…ked again Review of this PR (the triage of 2026-10-16 and 2026-10-19): the recovery picked the source again every 2 s, which restarts every device's stream and raises the window in regular mode, for ever if a layer kept failing. Now the device asks for one keyframe (kind 44), once per interruption. HEVCDisplayView.checkLayer, on the network queue, flushes a layer that failed (`status`, `error`), could not decode a frame (its failedToDecodeNotification) or needs a flush to decode again (requiresFlushToResumeDecoding, and its change notification): at each frame let through, at each notification (a still window sends no frame that would show it) and when the picture's window becomes active. It says so (`onNeedsKeyframe`, the hook Saffsanity#107's renderer calls too), and a keyframe that ends the wait on a healthy layer says the picture is back (`onPictureBack`). A decode failure posted before the picture last came back is about an earlier frame and is ignored, or it would throw the new picture away. The Mac sends its parameter sets before every keyframe and each sample carries the format, so nothing else is sent again. KeyframeRecovery (pure, main thread) decides when StreamClient sends the ask: one per interruption, owed again after a trip away (a keyframe that comes meanwhile is lost to it); only while the picture's window scene is foreground active (each Sill window has its own client) and something streams; at least 2 s after the last ask, an ask held back going when it is due; a picture that fails again within 10 s of its keyframe, Sill active throughout, waits 4, 8, 16, then 32 s, so a decoder that cannot keep up never costs every device a keyframe every 2 s; a trip away or 10 s back ends that. Tests/checks/keyframe-recovery: 51 checks and 20,000 random sessions against a model of the event history, 28 of 28 mutants. It found a real fault on the way: an ask held back until `last + spacing` was held again at that very time, (last + s) - last rounding below s; it compares with the due time now. DEBUG: the console's "display: …" and "keyframe: …" lines, and -SillLayerFail <s>[,<s>…], which enqueues a frame that cannot decode at those seconds after the first picture (ContentView's harness contract). Live, on a private iPad simulator (iOS 27.0) against a loopback synthetic host on the software encoder whose Desktop is still from 3 s: -SillLayerFail 6 gave the layer's own -11821, one ask, one "Keyframe for …" on the host, the picture back 75 to 108 ms after, and a single "Synthetic capture" line (nothing picked again); -SillLayerFail 6,6.6,12 asked at once, then 4.2 and 8.1 s after the ask before, three host lines. A trip to the background does not stop the simulator's layer: nothing was asked and the picture was as it was (a device has to show that case). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity
added a commit
to addhumal/sill
that referenced
this pull request
Oct 19, 2026
CLAUDE.md: the Current step entry for this pull request (what was kept, what replaced the re-pick, kind 44, the rule, the verification, the note for Saffsanity#107 and Saffsanity#66, and what is left for a device); the Layout (kind 44, the coordinator's answer, WindowSizer's CommandOutcome, KeyframeRecovery, HEVCDisplayView's checkLayer, SILL_TEST_STILL_AFTER, the fixture's Slow Close window and the close step, sillclient's --keyframe@T, the check); Build and run (the still host, --keyframe@T, -SillLayerFail); and kind 44 among the compatibility floor's kinds, empty and skipped by a host from before. docs/DEVELOPMENT.md: the picture's recovery under "If the picture freezes" (the host's line, the rule, an older Mac), the lights and Close's timeout after "Windows sized for a device", the menus' test app's Slow Close window and the close step (and that menu-check needs an unlocked screen), -SillLayerFail in the debug harness, the still host among the CLI's test variables, and --keyframe@T beside the test client's flags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml
# Conflicts: # .github/workflows/ci.yml
…unify with KeyframeRecovery)
…ur of KeyframeRecovery
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #106.
What it does
Adds Settings › This iPad / This iPhone › Picture Smoothing with five levels (
Automatic,None,Low,Medium,High), stored per device (Sill.pictureSmoothing) and never sent over the wire.AVSampleBufferDisplayLayeras today, untouched.MetalStreamRenderer(pbxprojAB21–AB23/FB21–FB23) decodes HEVC frames withVTDecompressionSessionto BGRA and draws them in the aspect-fit video rectangle under the pointer sprite using a two-pass separable Metal shader (StreamShader: horizontal pass into anrgba16Floatintermediate texture, vertical pass clamped into the drawable). When downscaling (such as a Retina stream on an iPad), the filter support widens by the scale factor so fine detail averages out instead of aliasing.Switching between
Automaticand the Metal renderer re-picks the active source once for a fresh keyframe; switching betweenNone,Low,MediumandHighredraws the last decoded frame in place. If iOS invalidates the hardware decompression session in the background,MetalStreamRendererrecreates it and asks for a keyframe throughHEVCDisplayView.onNeedsKeyframe(PictureSmoothing.DecoderRecovery).layer.framebufferOnlyisfalsesoUIView.drawHierarchy(#96, #117) can read the drawable.Why
AVSampleBufferDisplayLayerignoresCALayer'smagnificationFilterandminificationFilter(verified on the simulator:linearandnearestproduced byte-identical pixels). Decoding withVideoToolboxand scaling in a two-pass Metal shader gives control over the resampling kernel at every scale factor:AutomaticNoneLowMediumHighChecks
Tests/checks/picture-smoothing: 52 model/kernel/resampler/recovery checks + 65 GPU checks runningStreamShaderon Metal against the CPU reference across all four filters, both axes, upscaling, downscaling and 1:1. 22 of 22 mutants caught. Added to CI's mutants list and the checks README.Tests/checks/run-all.shandswift build -c releasepass with only the known warnings; the iOS app builds for the simulator.SillHost --syntheticwith an 8-pixel stripe test pattern (Automatic7.8 % intermediate greys,None0.0 %,Low7.8 %,Medium/High7.0 %) and mid-stream switching via-SillSmoothingAt.