feat(apple,android): three-way touch input — trackpad cursor (default), direct pointer, real multi-touch passthrough
android / android (push) Has been cancelled
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
release / apple (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled

The two touch clients had exactly complementary gaps: iOS forwarded fingers
ONLY as raw wire touches (no way to drive the host cursor from the touch
screen), Android had the two mouse modes but no passthrough. Both now share
one three-way "Touch input" setting: Trackpad (default) / Direct pointer /
Touch passthrough.

iOS/iPadOS: Input/TouchMouse.swift ports the Android gesture engine 1:1
(same px-based acceleration curve; tap=click, two-finger tap=right-click,
two-finger drag=scroll, tap-then-drag=held drag, three-finger tap=stats
HUD via the shared hudEnabled default); direct-pointer mode maps through
the aspect-fit letterbox; the previous always-on behavior lives on as the
passthrough option. The mode latches per gesture (a Settings change never
splits one gesture across models), touchesCancelled releases held state
without synthesizing a click, and session stop flushes a mid-drag button.
Settings picker on iPhone + iPad next to the iPad-only pointer-capture
toggle. Deliberate default change: trackpad, not passthrough.

Android: new nativeSendTouch JNI shim → wire TouchDown/Move/Up (the host
already injects real touch on every backend — libei touchscreen, wlroots,
KWin fake-input, SendInput); streamTouchPassthrough forwards every finger
with stable ids and lifts still-held contacts on teardown; the trackpadMode
Boolean becomes the TouchMode enum (old pref migrated on load, never
rewritten) with a Settings dropdown.

Verified: macOS swift build + full suite (incl. new TouchMouseTests), iOS
Simulator Swift compile, cargo check/fmt/clippy on the native crate, Kotlin
app+kit compile + unit tests. On-glass feel of the iOS ballistics and
Android passthrough against a touch-aware app still pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-03 00:02:04 +02:00
parent e8196b33b8
commit d707ee4d4e
14 changed files with 570 additions and 44 deletions
@@ -339,6 +339,9 @@ public final class StreamViewController: UIViewController {
setCaptured(false)
inputCapture?.stop()
inputCapture = nil
// Release anything the touch-driven mouse still holds (a mid-drag session end) while
// onTouchEvent can still deliver the button-up.
streamView.resetTouchInput()
streamView.onTouchEvent = nil
streamView.onPointerMoveAbs = nil
streamView.onPointerButton = nil
@@ -454,7 +457,8 @@ final class StreamLayerUIView: UIView {
/// Reads the LIVE negotiated mode in pixels (the touch/pointer coordinate space).
var currentHostMode: (() -> CGSize)?
/// Direct fingers / Pencil wire touch events.
/// Direct fingers / Pencil wire events: real touches in passthrough mode, or the
/// touch-driven mouse events (`TouchMouse`) in the trackpad/pointer modes.
var onTouchEvent: ((PunktfunkInputEvent) -> Void)?
/// Indirect pointer (mouse/trackpad with no lock) absolute cursor moves.
var onPointerMoveAbs: ((HostPoint) -> Void)?
@@ -468,6 +472,22 @@ final class StreamLayerUIView: UIView {
/// GameStream button held per active indirect-pointer touch (one click/drag session);
/// released when that touch ends.
private var pointerButtons: [ObjectIdentifier: UInt32] = [:]
/// Touch-driven mouse for the trackpad/pointer `TouchInputMode`s (see TouchMouse.swift).
private lazy var touchMouse: TouchMouse = {
let mouse = TouchMouse()
mouse.send = { [weak self] event in self?.onTouchEvent?(event) }
mouse.hostPoint = { [weak self] point in self?.hostPoint(from: point) }
return mouse
}()
/// The finger route latched at gesture start a Settings change mid-gesture applies to
/// the NEXT touch, so one gesture never splits across input models.
private var fingerRoute: TouchInputMode?
/// Release anything the touch-driven mouse holds and forget gesture state session stop.
func resetTouchInput() {
touchMouse.reset()
fingerRoute = nil
}
#endif
override init(frame: CGRect) {
@@ -504,10 +524,10 @@ final class StreamLayerUIView: UIView {
route(touches, event: event, kind: .up)
}
override func touchesCancelled(_ touches: Set<UITouch>, with event: UIEvent?) {
route(touches, event: event, kind: .up)
route(touches, event: event, kind: .cancel)
}
private enum TouchKind { case down, move, up }
private enum TouchKind { case down, move, up, cancel }
/// Split a touch batch by kind: an INDIRECT POINTER (mouse/trackpad with no lock) drives
/// the host cursor as an absolute mouse; everything else (direct finger, Pencil) is a host
@@ -521,7 +541,28 @@ final class StreamLayerUIView: UIView {
fingers.insert(touch)
}
}
if !fingers.isEmpty { forwardTouches(fingers, kind: kind) }
if !fingers.isEmpty { forwardFingers(fingers, kind: kind) }
}
/// Route direct fingers by the touch-input model, latched for the whole gesture:
/// passthrough real wire touches; trackpad/pointer the TouchMouse gesture engine.
private func forwardFingers(_ touches: Set<UITouch>, kind: TouchKind) {
let mode = fingerRoute ?? TouchInputMode.current
fingerRoute = mode
switch mode {
case .touch:
// A cancellation lifts the wire touch like a normal up the host just sees the
// contact end.
forwardTouches(touches, kind: kind == .cancel ? .up : kind)
case .trackpad, .pointer:
switch kind {
case .down: touchMouse.began(touches, in: self, trackpad: mode == .trackpad)
case .move: touchMouse.moved(touches, in: self)
case .up: touchMouse.ended(touches, in: self)
case .cancel: touchMouse.cancelled(touches)
}
}
if touchIDs.isEmpty, touchMouse.isIdle { fingerRoute = nil }
}
/// An indirect-pointer touch is a button-held click/drag session: forward its position as
@@ -537,7 +578,7 @@ final class StreamLayerUIView: UIView {
onPointerButton?(button, true)
case .move:
if let host { onPointerMoveAbs?(host) }
case .up:
case .up, .cancel:
if let host { onPointerMoveAbs?(host) }
if let button = pointerButtons.removeValue(forKey: key) {
onPointerButton?(button, false)
@@ -554,7 +595,7 @@ final class StreamLayerUIView: UIView {
case .down:
id = nextFreeID()
touchIDs[key] = id
case .move, .up:
case .move, .up, .cancel:
guard let known = touchIDs[key] else { continue }
id = known
}