fix(client/ios): Escape keeps the pointer captured instead of handing it back to iPadOS #19

2026-08-02T10:39:27Z

enricobuehler commented

2026-08-02 10:39:27 +00:00

Reported from the field: streaming to iPad with keyboard & mouse, pressing Escape swapped the captured cursor for the iPadOS system cursor, and only a mouse click brought the capture back.

Root cause — not our code

iPadOS releases the scene's pointer lock by itself on Escape: the platform's built-in "let me out", mirroring the web Pointer Lock API's default unlock gesture. A bare Esc never touches captured here, and it still forwards to the host as the game key it is. But the lock going away flips the mouse onto the absolute UIKit path and un-hides the local cursor.

The old syncPointerLock() merely observed the drop (gcMouseForwarding = false) and never re-requested, so the state only re-resolved on the next setCaptured / appearance / parent change — or on the click that re-granted it.

⚠ Apple documents this Escape behaviour nowhere — not on prefersPointerLocked, not on UIPointerLockState, not in WWDC20-10617. The session only confirms setNeedsUpdateOfPrefersPointerLocked() is the dynamic re-request hook.

The fix

syncPointerLock() now arms a bounded re-lock burst when the lock is wanted, was previously held, and is now gone — 3 attempts over ~0.6 s, no restart inside 2 s. Attempt 1 re-asserts prefersPointerLocked; later ones force a real false→true transition (via a pointerLockForcedOff flag held 50 ms) and re-anchor the PointerLockChain.

While a re-lock is in flight the local cursor stays hidden and absolute pointer motion is muted, so the couple of frames read as "Esc did nothing to my mouse" rather than a cursor blinking in and out plus a host cursor teleporting to the pointer's absolute position. Buttons still forward — they carry no position, so a click mid-relock isn't swallowed.

Two guards worth preserving in any refactor:

"previously held" — without it, a scene that never qualifies (Stage Manager, Split View) would hide the cursor for the whole burst chasing a lock that isn't coming. A first grant is still driven by the chain engage in setCaptured/viewDidAppear, exactly as before.
settle only through syncPointerLock() — pointerRelockPending hides the cursor, so every give-up/settle path must funnel through the one place that calls pointerInteraction.invalidate(). The burst clears itself on give-up, so the cursor can never stay hidden on a lock the system won't grant.

Every deliberate release (⌘⎋, ⌃⌥⇧Q, the Stream menu, resigning active) clears captured first, so wantsPointerLock is already false when its drop is observed — none of them are fought.

Verification

iOS build green: swift build --triple arm64-apple-ios17.0 completes clean.
All edits sit inside #if os(iOS) blocks — tvOS and macOS untouched (verified against the file's preprocessor nesting).

⏳ Not yet verified on glass

The premise that iPadOS honours an immediate re-request is untested on a real iPad. The false→true escalation exists precisely as the hedge if a plain re-assert is ignored; if it still misbehaves on device, that's the knob to look at. Worth an on-glass pass before this ships in a release.

Unrelated snag hit on the way

scripts/build-xcframework.sh currently dies on this Mac at its own macOS-floor guard — libpunktfunk_core.a contains objects built for macOS 26.0 (> 14.0) — before ever reaching the iOS slices. The script blames a stale cache, but a fully cold worktree-local target/ reproduces it, so it's the toolchain ignoring MACOSX_DEPLOYMENT_TARGET=14.0; rm -rf target/{aarch64,x86_64}-apple-darwin will not fix it. Worked around here by assembling an iOS-only bundle by hand. Anything needing the macOS slice will hit this.

🤖 Generated with Claude Code

Reported from the field: streaming to iPad with keyboard & mouse, pressing **Escape** swapped the captured cursor for the iPadOS system cursor, and only a mouse **click** brought the capture back. ## Root cause — not our code iPadOS releases the scene's pointer lock **by itself** on Escape: the platform's built-in "let me out", mirroring the web Pointer Lock API's default unlock gesture. A bare Esc never touches `captured` here, and it still forwards to the host as the game key it is. But the lock going away flips the mouse onto the absolute UIKit path and un-hides the local cursor. The old `syncPointerLock()` merely *observed* the drop (`gcMouseForwarding = false`) and never re-requested, so the state only re-resolved on the next `setCaptured` / appearance / parent change — or on the click that re-granted it. > ⚠ Apple documents this Escape behaviour **nowhere** — not on `prefersPointerLocked`, not on `UIPointerLockState`, not in WWDC20-10617. The session only confirms `setNeedsUpdateOfPrefersPointerLocked()` is the dynamic re-request hook. ## The fix `syncPointerLock()` now arms a **bounded re-lock burst** when the lock is wanted, was previously **held**, and is now gone — 3 attempts over ~0.6 s, no restart inside 2 s. Attempt 1 re-asserts `prefersPointerLocked`; later ones force a real false→true transition (via a `pointerLockForcedOff` flag held 50 ms) and re-anchor the `PointerLockChain`. While a re-lock is in flight the local cursor stays hidden and absolute pointer **motion** is muted, so the couple of frames read as "Esc did nothing to my mouse" rather than a cursor blinking in and out plus a host cursor teleporting to the pointer's absolute position. **Buttons still forward** — they carry no position, so a click mid-relock isn't swallowed. Two guards worth preserving in any refactor: - **"previously held"** — without it, a scene that never qualifies (Stage Manager, Split View) would hide the cursor for the whole burst chasing a lock that isn't coming. A first grant is still driven by the chain engage in `setCaptured`/`viewDidAppear`, exactly as before. - **settle only through `syncPointerLock()`** — `pointerRelockPending` hides the cursor, so every give-up/settle path must funnel through the one place that calls `pointerInteraction.invalidate()`. The burst clears itself on give-up, so the cursor can never stay hidden on a lock the system won't grant. Every deliberate release (⌘⎋, ⌃⌥⇧Q, the Stream menu, resigning active) clears `captured` first, so `wantsPointerLock` is already false when its drop is observed — none of them are fought. ## Verification - iOS build green: `swift build --triple arm64-apple-ios17.0` completes clean. - All edits sit inside `#if os(iOS)` blocks — tvOS and macOS untouched (verified against the file's preprocessor nesting). ## ⏳ Not yet verified on glass The premise that iPadOS honours an immediate re-request is **untested on a real iPad**. The false→true escalation exists precisely as the hedge if a plain re-assert is ignored; if it still misbehaves on device, that's the knob to look at. Worth an on-glass pass before this ships in a release. ## Unrelated snag hit on the way `scripts/build-xcframework.sh` currently **dies on this Mac** at its own macOS-floor guard — `libpunktfunk_core.a contains objects built for macOS 26.0 (> 14.0)` — before ever reaching the iOS slices. The script blames a stale cache, but a fully cold worktree-local `target/` reproduces it, so it's the toolchain ignoring `MACOSX_DEPLOYMENT_TARGET=14.0`; `rm -rf target/{aarch64,x86_64}-apple-darwin` will not fix it. Worked around here by assembling an iOS-only bundle by hand. Anything needing the macOS slice will hit this. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

enricobuehler added 1 commit 2026-08-02 10:39:28 +00:00

fix(client/ios): Escape keeps the pointer captured instead of handing it back to iPadOS

ci / web (pull_request) Successful in 1m2s

Details

apple / swift (pull_request) Successful in 1m18s

Details

apple / screenshots (pull_request) Skipped

Details

ci / rust-arm64 (pull_request) Successful in 1m26s

Details

ci / docs-site (pull_request) Successful in 3m46s

Details

ci / rust (pull_request) Successful in 4m8s

Details

d63e913f52

iPadOS releases the scene's pointer lock by itself when Escape is pressed — the platform's
built-in "let me out", mirroring the web Pointer Lock API's default unlock gesture. Nothing in
our code does it: a bare Esc never touches `captured`, and it keeps forwarding to the host as
the game key it is. But the lock going away flips the mouse onto the absolute UIKit path and
un-hides the iPadOS cursor, so pressing Esc for an in-game menu silently cost the capture until
the user clicked into the video to win it back.

Esc is a GAME key in a stream, not a request to hand the pointer back to iPadOS, so an unwanted
drop is now re-requested. `syncPointerLock` arms a short, bounded burst (3 attempts over ~0.6 s,
no restart inside 2 s) whenever the lock is wanted, was previously HELD, and is now gone; the
first attempt re-asserts `prefersPointerLocked`, later ones present a real false→true transition
and re-anchor the PointerLockChain. Every deliberate release (⌘⎋, ⌃⌥⇧Q, the Stream menu,
resigning active) clears `captured` first, so `wantsPointerLock` is already false when their drop
is observed and none of them are fought.

The "previously held" half of the condition keeps a scene that never qualifies (Stage Manager,
Split View) from paying for a lock that isn't coming — there, a first grant is still driven by
the chain engage in setCaptured/viewDidAppear exactly as before.

While a re-lock is in flight the local cursor stays hidden and absolute pointer MOTION stays
muted, so the couple of frames it takes read as "Esc did nothing to my mouse" rather than a
cursor that blinks in and out and a host cursor that teleports to the pointer's absolute
position. Buttons still forward (they carry no position), so a click mid-relock isn't swallowed.
The burst clears itself on give-up, so the cursor can never stay hidden on a lock the system
won't grant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

enricobuehler merged commit 29473d6280 into main

2026-08-02 17:27:57 +00:00

enricobuehler deleted branch worktree-ipad-esc-pointer-relock

2026-08-02 17:28:00 +00:00

enricobuehler referenced this pull request

2026-08-02 18:17:14 +00:00

WIP: fix(0.23.0): the slice wire starves the reassembler, and three presenter defects behind it #21

enricobuehler referenced this pull request

2026-08-04 16:07:45 +00:00

fix(client/ios): a click wins the pointer back after Escape drops it #34

Sign in to join this conversation.