Skip to content

Apple Pencil pressure and tilt - #175

Open
HunterAhlquist wants to merge 20 commits into
Saffsanity:mainfrom
HunterAhlquist:pencil-pressure
Open

HunterAhlquist wants to merge 20 commits into
Saffsanity:mainfrom
HunterAhlquist:pencil-pressure

Conversation

@HunterAhlquist

@HunterAhlquist HunterAhlquist commented Oct 19, 2026 •

Copy link
Copy Markdown

Apple Pencil pressure and tilt, as discussed in #81. Built on pencil-pressure-rebased, @Saffsanity's merge of my branch into main, with one more commit of mine on top and main at 125be9d merged in.

A Pencil on the stream now draws as a tablet pen in Mac apps that read pressure and tilt, and stays the mouse everywhere else. Double tap and barrel roll will come in a second pull request.

What's in it

  • 593cc12: the stale hover that drew a straight line back to each stroke's start. It also fixes the Pencil-as-mouse path everyone has today, so it is first and can land on its own.
  • My five commits (38fc2f6 … a85fba7), unchanged, merged by @Saffsanity:
    • wire kind 45 pen;
    • PenTablet and PenEvents on the Mac;
    • PenStrokes on the iPad, with palm rejection;
    • the This iPad › Pencil Pressure switch;
    • the Sidecar-matched edge cases.
  • @Saffsanity's fixes after the merge: kind 45, PenStrokes' project IDs, the workspace file out, the docs.
  • 5065289, the hover cursor (new since the review): while a Pencil hovers as a pen, the iPad shows the Mac's cursor only when it is an app's own, such as a drawing app's brush outline. It never shows a standard arrow, I-beam or hand, and it hides as the tip lands.
    • New wire kind 46 cursorKind (CursorKind {standard}), host → device, sent just before each kind 14. Older devices skip it, and a device that gets none treats the cursor as standard.
    • The Mac's CursorClass decides by comparing the cursor's pixels with AppKit's standard cursors. The arrow and I-beam are also recognised by the window server's 1×–10× images, because the CLI has no NSApplication to compare against.
    • The kind was 30 on my branch. Sill Viewer: a Mac controlling another Mac #66 holds 30, and 46 is in no open pull request's head (checked 2026-10-19).
  • Merges of main at 68e880e (iPad: Shift+Return and Shift+Tab reach the Mac with the shift #110) and 125be9d (VirtualDisplay: the BOOL return's encoding follows the architecture #87): CI's mutants matrix keeps both new checks, pen and virtual-display-api; nothing else conflicted.

Checked

  • swift build -c release and the iOS app for the simulator and for a device: only the known warnings.
  • Tests/checks/run-all.sh: all pass.
  • Mutants of the checks this pull request touches, every one caught: pen 97, pointer-presence 48, compatibility 43, protocol 24, pointer-control 33, clipboard 72. One clipboard mutant no longer applied once WindowList's init gained pencil: after clipboard:; its pattern is fixed in the last commit.
  • On my iPad Pro (M4, iPadOS 27.0.1) with a Pencil:
    • Pressure and tilt: pressure in Affinity. Pressure and tilt in PenProbe --canvas, run through Sill on this build: every stroke arrived as tablet points, pressure finer than 8 bits, and tilt from −0.67 to +0.59, the same sign as Sidecar's for the same grip.
    • Hover: on this build, Affinity's brush outline shows under the hovering Pencil, and nothing shows over standard cursors (the toolbar, text, the canvas app's arrow). While building the rule, the arrow, I-beam, pointing hand and window-resize cursors all matched AppKit's; Affinity's outline and another app's tool cursor matched none.
    • The fix: the line back to the start is gone.
  • Sidecar's own behaviour was recorded and is followed for hover, a held button and mouse input during a stroke (docs/pencil-pressure-plan.md §13).

Still open

The plan's device tests P1–P10 beyond those above:

  • other drawing apps;
  • a resting palm;
  • a stroke cut by a rotation, a pick or Wi-Fi;
  • mixed builds.

Known gaps:

  • a palm resting before the Pencil's first touch gives a right-click;
  • with a custom pointer size or colour in Accessibility, standard cursors may read as an app's own and show under a hovering Pencil.

Disclosure, as in #81: Claude Code wrote most of the code; I designed the behaviour with it and tested it on my devices.

🤖 Generated with Claude Code

HunterAhlquist and others added 18 commits October 13, 2026 19:33
It came in with the pen proof of concept; main never tracked it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As the Pencil lifts, iPadOS 27.0.1 on an iPad Pro M4 sends one more hover
update about 1 ms before touchesEnded, carrying where the Pencil hovered
before it landed. handleHover sent it as a move while the Mac's button was
still down, so every stroke in a drawing app ended with a straight line back
to where it began (six strokes of six, main's build too). Hunter Ahlquist
found it and fixed it in his pencil-pressure branch (issue Saffsanity#81), where the
rule is PenStrokes.hover's and covers pen mode too; this is the same rule on
today's Pencil-as-mouse path alone, so it can land ahead of the rest: a
Pencil touch sets pencilTouching from touchesBegan to touchesEnded or
touchesCancelled (and leaving the window clears it), and a hover update
meanwhile sends nothing. The iPad's own pointer is untouched: only a Pencil
touch sets the flag.

Co-authored-by: Hunter Ahlquist <36090375+HunterAhlquist@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… as a pen

Hunter Ahlquist's five commits (issue Saffsanity#81), from main at 985a733, merged as
they are onto main at d1fa97e and the stale-hover commit before this one.
Eighteen files conflicted; every resolution keeps both sides:

- StreamMessage: main's clipboard (32), then the pen (still 29 here).
- Switcher: WindowList keeps `clipboard` and gains `pencil`; one init with
  both parameters, and the coordinator's list sends both.
- InputOverlay: main moved the finger pan onto the scroll path it shares
  with the iPad's trackpad and a wheel (Saffsanity#80); the palm guard stays in
  handlePan, before scroll(gesture, .finger), so it silences fingers only.
  The view's leave as it goes sits beside main's keyboard proxy. The
  stale-hover flag of the commit before gives way to PenStrokes.hover, the
  same rule in both modes (gone() resets it as the view leaves the window).
- StreamClient, MockCatalog, HostSettingsPanel, ContentView, sillclient.py:
  the clipboard's lines and the Pencil's side by side (the This iPad group
  has Larger Picture, Share Clipboard, the gestures, then Pencil Pressure;
  -SillInputScript has Saffsanity#80's steps and the pen's; --copy and --pen).
- The checks: compatibility runs main's clipboard cases and the pen's, its
  mutants follow the new init; protocol and pointer-control list both; the
  README's rows and ci.yml's mutants matrix gain `pen`.
- docs/DEVELOPMENT.md: main's hardware keyboard and clipboard sections, then
  Apple Pencil; site/privacy.html and CLAUDE.md likewise.

Left for the commits after this one: the pen's kind 29 is a number main's
clipboard, compatibility, protocol and pointer-control checks keep free
(they fail on it here), PenStrokes.swift's project IDs AC01/FC01 are
HardwareKeyboardPolicy.swift's since Saffsanity#80 (the iOS app does not build here),
and the workspace file comes back. swift build -c release passes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kind 29 was free on 985a733, where the pen was built; since, main's
comment and checks keep 29 to 31 free (29 the audio plan's, 30 and 31
Saffsanity#66's side streams), 32 is the clipboard, and open pull requests hold 33
to 44 (Saffsanity#66's 33 to 42, Saffsanity#94's 43, Saffsanity#93's keyframeRequest 44, checked against
every open head today). So `case pen = 45`, and everything that names the
number follows: SessionLink.isInput (8 or 45), the comments of Pen.swift,
Switcher, PenTablet, InputInjector and StreamServer (which match the enum,
not the number), sillclient.py's --raw45 and its sends, and the docs
(DEVELOPMENT.md, CLAUDE.md's entry, Layout and Build and run, the plan's
design and gates, with a note that its results before the rebase say 29).

The compatibility floor's kinds are 0–28, 32 and 45 again with 29 to 31
free as main has them; kind 26's `seen` counts kinds 8 and 45.

The checks: compatibility names kind 45 and 46 unknown (157 checks; its
mutant "kind 45 as 46"), protocol keeps main's two kind checks and has the
pen at 45 with 46 unknown (193 + 8), pointer-control's table and sweep say
45 (158), fence's input and its two C10 mutants say 45; menus' line and
mutant and pointer-control's two renumbering mutants are main's again, 29
being free once more. clipboard passes unchanged. Run here: compatibility
157, protocol 193 and the 8 cross-checks, pointer-control 158, menus 309,
clipboard 259, pen 197 and 5,000 random sessions, all passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fork gave PenStrokes.swift A1000001000000000000AC01 and …FC01, free on
985a733; Saffsanity#80 has since given the same pair to HardwareKeyboardPolicy.swift
on main. The project file merges without a conflict but then holds each ID
twice, xcodebuild keeps one ("Skipping duplicate build file"), and the iOS
build fails on HardwareKeyboardPolicy. A045/F045 (after the pen's kind) are
in no project file on main or in any open pull request's head, checked
against all of them today. CLAUDE.md's Layout line names the new pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
be8abf4 untracked iOSClient/Sill.xcodeproj/project.xcworkspace/
contents.xcworkspacedata, which Xcode writes when it opens the project and
main has never tracked; a85fba7 brought it back with the edge cases. Out
again, so this branch's project folder is main's plus PenStrokes.swift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
privacy.html is on main ahead of every release, and a republish before
the Pencil ships (another fix's words) would tell readers the app sends
their Pencil's pressure and tilt and keeps a Pencil Pressure switch, which
the builds out do not. As the clipboard's words are (PR Saffsanity#78), each Pencil
item now stands alone between `<!-- Pencil: …` and `<!-- /Pencil -->`:
"Your Apple Pencil's strokes, with how hard you press and how you tilt it."
among what the device sends, and "On an iPad, whether Pencil Pressure is
on." among what it keeps. docs/release-checklist.md part 1 §3 gains the
perl line that cuts them before that release, and says which Pencil lines
stay after the cut (index.html's intro, terms.html's trademarks). Checked
on a copy: after the Pencil's cut site/ is main's byte for byte, and with
the clipboard's too `grep -i pencil` prints only those two lines. The page's
date is left for the release to set, as the plan says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a pen

So a test can tell a host that takes kind 45 from one that skips it, as it
already tells the clipboard's: against this branch's synthetic host the
line reads "clipboard=1 pencil=1", against main's "clipboard=1" alone (the
device then sends its Pencil as kind 8). The help says so beside --pen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s people

Tests/checks/pen/run.sh still listed "mouse input from any device" and "a
second without a message with the tip up" among the pen's leaves; since
a85fba7 (Sidecar's behaviour) mouse input never takes the pen out of
range and the idle leave comes a second after an up with no hover since,
as PenTablet, its check and the checks' README say. Now the header says so.

docs/pencil-pressure-plan.md was written in Hunter's fork, where "the
owner" meant him; in Sill's repository it reads as the repository's owner.
Who ran, saw, drew or asked is now Hunter by name, and the calls the plan
leaves open (its decisions' defaults, Q1) are Noah's, Sill's maintainer.
Nothing else in the plan changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pencil-pressure-rebased

Only ci.yml's mutants matrix conflicted: it keeps both new checks,
`installed-apps` (Saffsanity#136) and `pen`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s rebase

Hunter's branch carried a 56-line Current step entry written for his fork
(its "the owner" meant him, and its numbers were the fork's); every other
open pull request edits the top of that section too, so the entry is now a
short one in the section's usual form: what the branch is, the wire, the
host, the device, what the maintainer's commits after the merge did, what
was verified, and what waits for a device. The details stay where they were
already: the plan, the Layout lines and Build and run. PenProbe's line in
Build and run says a person runs --post-to-this-mac by hand at the Mac,
never an agent (it said "the owner's to run").

docs/pencil-pressure-plan.md gains "Rebased onto main (2026-10-20)": the
order of the branch (the stale hover on main's mouse path first, Hunter's
five commits merged as they are, the maintainer's commits, main again), the
merge's resolutions (WindowList's one init with both `clipboard` and
`pencil`, the palm guard in handlePan before Saffsanity#80's shared
scroll(gesture, .finger), the stale-hover flag giving way to
PenStrokes.hover), kind 45 and the checks it touched, the project IDs, the
workspace file, the site's markers, and what was verified with its numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While a Pencil hovers as a pen, the device's arrow shows in the Mac cursor's shape only when that
cursor is an app's own (a drawing app's brush outline), never over a standard arrow, I-beam or hand.
Kind 46 `cursorKind` (CursorKind), host → device, just before each kind 14; the host's CursorClass
decides. Re-applied on pencil-pressure-rebased: its kind moved from 30, which Saffsanity#66 holds, to 46, the
first number no open pull request holds after the pen's 45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to pencil-pressure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ding) into pencil-pressure

CI's mutants matrix keeps both new checks, pen and virtual-display-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HunterAhlquist
HunterAhlquist marked this pull request as ready for review October 19, 2026 20:31
… parameter after clipboard

The mutant "clipboard 1 by default" matched the init's line ending at clipboard, which now goes on
to pencil, so it no longer applied (72 of 72 with it applying again).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anurag1212

Copy link
Copy Markdown

<3

Saffsanity added a commit that referenced this pull request Oct 20, 2026
…is run's checks

Work in progress, paused for the night. The plan's "Rebased onto main" and the CLAUDE.md entry
say what ran on f85aa49 (main at 4d990e9 merged as d8984a6, Hunter's clipboard-mutant fix
5c39211 cherry-picked from #175): swift build -c release from git archive and the three iOS
builds from an empty derived-data folder (known warnings only), SillHost --pen-selftest 37 of
37, run-all.sh all 47, H4 (idle identical; client runs identical but for their last stats
second), H5 and S1 on loopback (the pen's enter, tilt, pressure, drags, up and leave against
this host; kind 8 alone against main's).

What remains: clipboard/run.sh --mutants to the end (9 of 72 caught before the pause), and a
decision on #175, which carries Hunter's hover cursor (kind 46, CursorClass) that this branch
does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main took kinds 43 (sound) and 44 (keyframe request); 45 and 46 stay the
Pencil's and the cursor's kind. The pen now goes through main's new rules
as kind 8 does: a revoked device and a locked Mac pass on only what lets
its pen go (PenTablet.mayLetGo, stillGoes), the lock gate withholds kind
46 with kind 14, DisplayWake counts kind 45 as a person, and the pen's
button carries its owner like main's buttons. The palm guard runs before
main's picture zoom in handlePan.

Checks: run-all.sh all passed but display-wake, now taught kinds 45 and
46 (200 checks, 64 of 64 mutants); lock-policy knows kind 46.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants