fix(client/ios): a click wins the pointer back after Escape drops it #34

Merged

enricobuehler merged 1 commits from worktree-ipad-click-relock into main 2026-08-04 16:20:50 +00:00

Conversation 0 Commits 1 Files Changed 1

+35

enricobuehler commented 2026-08-04 16:07:45 +00:00

Owner

Copy Link

Copy Source

The report

iPad client, KDE host. Pressing Escape in game releases the mouse as expected, but clicking back into the stream doesn't restore aiming: clicks still land in the right place, and camera movement is dead. A second user reports touch or mouse input stopping outright, also on Apple.

What's actually happening

The giveaway is "clicks still register in the correct place." That means absolute positions are still forwarding — the host cursor tracks the iPad cursor, so a click lands where you aimed. The capture is simply pinned to the absolute pointer path instead of the locked relative one, and games read relative deltas for camera look. Hence: clicks fine, camera dead.

Why it never recovers:

1. iPadOS drops the pointer lock by itself on Escape, and a bare Escape deliberately never clears captured — it's a game key (StreamViewIOS.swift:182).
2. syncPointerLock() sets useGCMouse = captured && locked → false, so we fall back to absolute (:769-776).
3. The re-lock burst added in #19 fires — 3 attempts over ~0.6 s (:791-858) — and gives up if refused.
4. The user then clicks, and that click cannot ask for the lock back. captured is still true, so onPointerButton takes the already-captured branch and just forwards the click (:453-458).

There are exactly four places that ever request the lock: viewDidAppear (:332), didMove(toParent:) (:337), setCaptured (:732-733), and the burst (:837-846). A click while captured == true hits none of them. So once the burst's budget is spent, nothing re-asks for the remainder of the capture.

And the burst is asking at the worst possible moment — the 0.6 s right after the platform's own "let me out" gesture. The web Pointer Lock API that iPadOS's behaviour mirrors enforces a cooldown after an Escape unlock precisely to stop an app grabbing the pointer straight back.

The fix

Make the click the second stage of the recovery. A click into the video while captured-but-unlocked re-anchors the lock chain and re-asks — which is the request the platform actually wants: a genuine user gesture, not an app re-grabbing on its own initiative.

Three deliberate guards:

• On the button UP, so the click has fully forwarded on one transport first. Asking on the DOWN can flip gcMouseForwarding mid-click and strand the release on the GCMouse path.
• Gated on pointerLockWasEngaged, exactly as the drop path is, so a scene that never qualifies (Stage Manager, Split View) is never bursted at.
• Gated on no burst already in flight — a pending burst mutes absolute motion, so re-arming one on every click would freeze the cursor between clicks of a menu the user is still aiming around.

Worst case is now today's behaviour rather than a permanent one: a refused burst settles, and the next click tries again.

Verification

• Typechecked for arm64-apple-ios17.0 — PunktfunkKit builds clean. (Needs a hand-assembled iOS xcframework slice; build-xcframework.sh still dies on the macOS-floor guard.)
• ⏳ NOT verified on glass. Worth being blunt: #19 shipped in v0.24.0 on a typecheck alone, and this report is that fix coming back failed. The premise here — that a click-driven re-request is honoured — is exactly the class of assumption that went wrong last time. This wants a real iPad before it's trusted.

Not in scope

The second report ("touch or mouse input stops working") is a different failure and is left alone deliberately. Two structural suspects found while triaging, both worth their own change:

• useGCMouse = captured && locked (:769) never checks a GCMouse is actually attached. Locked with no enumerated mouse ⇒ the UIKit path is gated off (:435, :457) and the mouse goes completely dead.
• There is no GCMouseDidDisconnect observer — mice[] (InputCapture.swift:59) is append-only, never pruned.
• Related: the suppressedButton latch (InputCapture.swift:449-456) can strand left-click if a grant lands between the down and the up.

Refs #19.

## The report iPad client, KDE host. Pressing Escape in game releases the mouse as expected, but **clicking back into the stream doesn't restore aiming**: clicks still land in the right place, and camera movement is dead. A second user reports touch or mouse input stopping outright, also on Apple. ## What's actually happening The giveaway is *"clicks still register in the correct place."* That means absolute positions **are** still forwarding — the host cursor tracks the iPad cursor, so a click lands where you aimed. The capture is simply pinned to the **absolute** pointer path instead of the locked relative one, and games read relative deltas for camera look. Hence: clicks fine, camera dead. Why it never recovers: 1. iPadOS drops the pointer lock by itself on Escape, and a bare Escape deliberately never clears `captured` — it's a game key (`StreamViewIOS.swift:182`). 2. `syncPointerLock()` sets `useGCMouse = captured && locked` → false, so we fall back to absolute (`:769-776`). 3. The re-lock burst added in #19 fires — 3 attempts over ~0.6 s (`:791-858`) — and gives up if refused. 4. **The user then clicks, and that click cannot ask for the lock back.** `captured` is still `true`, so `onPointerButton` takes the already-captured branch and just forwards the click (`:453-458`). There are exactly four places that ever request the lock: `viewDidAppear` (`:332`), `didMove(toParent:)` (`:337`), `setCaptured` (`:732-733`), and the burst (`:837-846`). **A click while `captured == true` hits none of them.** So once the burst's budget is spent, nothing re-asks for the remainder of the capture. And the burst is asking at the worst possible moment — the 0.6 s right after the platform's own "let me out" gesture. The web Pointer Lock API that iPadOS's behaviour mirrors enforces a cooldown after an Escape unlock precisely to stop an app grabbing the pointer straight back. ## The fix Make the click the **second stage** of the recovery. A click into the video while captured-but-unlocked re-anchors the lock chain and re-asks — which is the request the platform actually wants: a genuine user gesture, not an app re-grabbing on its own initiative. Three deliberate guards: - **On the button UP**, so the click has fully forwarded on one transport first. Asking on the DOWN can flip `gcMouseForwarding` mid-click and strand the release on the GCMouse path. - **Gated on `pointerLockWasEngaged`**, exactly as the drop path is, so a scene that never qualifies (Stage Manager, Split View) is never bursted at. - **Gated on no burst already in flight** — a pending burst mutes absolute motion, so re-arming one on every click would freeze the cursor between clicks of a menu the user is still aiming around. Worst case is now today's behaviour rather than a permanent one: a refused burst settles, and the next click tries again. ## Verification - Typechecked for `arm64-apple-ios17.0` — `PunktfunkKit` builds clean. (Needs a hand-assembled iOS xcframework slice; `build-xcframework.sh` still dies on the macOS-floor guard.) - ⏳ **NOT verified on glass.** Worth being blunt: #19 shipped in v0.24.0 on a typecheck alone, and this report is that fix coming back failed. The premise here — that a click-driven re-request *is* honoured — is exactly the class of assumption that went wrong last time. This wants a real iPad before it's trusted. ## Not in scope The second report (*"touch or mouse input stops working"*) is a different failure and is left alone deliberately. Two structural suspects found while triaging, both worth their own change: - `useGCMouse = captured && locked` (`:769`) never checks a `GCMouse` is actually attached. Locked with no enumerated mouse ⇒ the UIKit path is gated off (`:435`, `:457`) and the mouse goes completely dead. - There is no `GCMouseDidDisconnect` observer — `mice[]` (`InputCapture.swift:59`) is append-only, never pruned. - Related: the `suppressedButton` latch (`InputCapture.swift:449-456`) can strand left-click if a grant lands between the down and the up. Refs #19.

enricobuehler added 1 commit 2026-08-04 16:07:46 +00:00

fix(client/ios): a click wins the pointer back after Escape drops it ... 6f54fcdd2d

Pressing Escape mid-stream on an iPad leaves the capture in a state it
could never leave: iPadOS releases the pointer lock by itself, a bare
Escape deliberately never clears `captured` (it is a game key), and the
re-lock burst added with the Escape-drop fix is the only thing that ever
asks for the lock back. That burst fires in the 0.6 s immediately after
the platform's own "let me out" gesture — precisely when it is least
likely to be granted — and once its budget is spent nothing re-asks:
`setCaptured` is the only other requester, and `captured` never went
false. The capture then spends the rest of its life on the absolute
pointer path, which is why the field report reads the way it does —
clicks still land exactly where you aim, because absolute positions keep
forwarding, but the game receives no relative deltas and camera look is
dead for the rest of the session.

Make the click the second stage of the recovery. A click into the video
while captured-but-unlocked now re-anchors the lock chain and re-asks,
which is the request the platform actually wants: a genuine user
gesture rather than an app grabbing the pointer straight back.

Asked on the button UP, so the click has fully forwarded on one
transport first — asking on the DOWN can flip `gcMouseForwarding`
mid-click and strand the release on the GCMouse path. Gated on
`pointerLockWasEngaged`, exactly as the drop path is, so a scene that
never qualifies (Stage Manager, Split View) is never bursted at, and on
no burst already being in flight, since a pending burst mutes absolute
motion and re-arming one per click would freeze the cursor between
clicks of a menu the user is still aiming around.

Worst case is now today's behaviour rather than a permanent one: a
refused burst settles, and the next click tries again.

Typechecked for arm64-apple-ios17.0 (PunktfunkKit builds clean). NOT yet
verified on glass — the premise that a click-driven re-request is
honoured is exactly what the previous fix got wrong.

enricobuehler merged commit dbc12dedcc into main 2026-08-04 16:20:50 +00:00

enricobuehler deleted branch worktree-ipad-click-relock 2026-08-04 16:20:58 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX enricobuehler (Enrico Bühler) kszlim nannku neoney sassycrown

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#34