feat(core+host+android): committed-text input plane (IME path) — InputKind::TextInput

The VK key-event vocabulary cannot express text an input method COMMITS (autocorrect, gesture typing, non-Latin scripts, emoji). Add a first-class text event and negotiate it: - punktfunk-core: InputKind::TextInput (= 15) carries one Unicode scalar per event in `code`; HOST_CAP_TEXT_INPUT (0x04) in Welcome::host_caps. - Host advertises the cap only where the session's inject backend can type text: Windows SendInput (KEYEVENTF_UNICODE, surrogate-pair aware) and the Linux wlroots backend — a dedicated second zwp_virtual_keyboard whose xkb keymap grows Unicode keysyms on demand (the wtype model), so keymap re-uploads never disturb the main device's layout/modifier state. The KWin-fake-input/libei/gamescope backends can only press layout keycodes, so those sessions don't set the bit and clients keep the VK fallback. - GameStream plane: Moonlight's UTF-8 text packet (MAGIC_UTF8, previously recognized-and-dropped) now decodes to the same TextInput events. - Android: KeyCaptureView picks a real editable InputConnection when the host has the cap — the IME runs its full machinery, mirrored to the host live via common-prefix diffs of the composition (backspaces + new suffix), with setComposingRegion adopting committed text so autocorrect-revert flows diff instead of retyping; newline→Enter, deleteSurroundingText→Backspace/Delete. Older hosts keep the TYPE_NULL raw-key path unchanged. - keymap: media VKs (0xB0-0xB3) → evdev so the Android media keys land on Linux hosts too. Verified: punktfunk-core + host gamestream + pf-inject tests green on Linux (Ubuntu box), clippy clean; Android app+native builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 19:19:11 +02:00
parent ddb72edcba
commit abe6228b42
14 changed files with 460 additions and 20 deletions
@@ -450,6 +450,9 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
            factory = { ctx ->
                KeyCaptureView(ctx).also { v ->
                    keyCapture = v
+                    // Real IME text path when the host types committed text (see KeyCaptureView).
+                    v.textHandle =
+                        if (NativeBridge.nativeTextInputSupported(handle)) handle else 0L
                    v.setOnCapturedPointerListener { _, ev ->
                        (ctx as? MainActivity)?.mouseForwarder?.onCapturedPointer(ev) ?: false
                    }
@@ -513,12 +516,13 @@ private fun RemotePointerHint(modifier: Modifier = Modifier) {

 /**
 * Invisible focus anchor for typing on the host: the three-finger swipe summons the device IME
- * onto this view. `TYPE_NULL` puts the IME in "dumb keyboard" mode — it delivers raw [KeyEvent]s
- * (no composing text, no autocorrect), which flow through `MainActivity.dispatchKeyEvent` →
- * `Keymap.toVk` → the host, the exact path a hardware keyboard takes. Text an IME insists on
- * committing instead still arrives: the non-editable [BaseInputConnection] synthesizes KeyEvents
- * for it via `KeyCharacterMap` (with Shift carried as meta state — see the IME-shift wrap in
- * `MainActivity.dispatchKeyEvent`).
+ * onto this view. Two IME models, picked by the host's capabilities:
+ *  * **Text path** ([textHandle] set — the host advertised `HOST_CAP_TEXT_INPUT`): a real
+ *    editable [HostTextConnection], so the IME gives autocorrect, gesture typing, non-Latin
+ *    composition and emoji, all mirrored to the host as committed text + diffs.
+ *  * **Fallback** (older host): `TYPE_NULL` puts the IME in "dumb keyboard" mode — raw
+ *    [KeyEvent]s flow through `MainActivity.dispatchKeyEvent` → `Keymap.toVk` → the host, the
+ *    exact path a hardware keyboard takes (with the IME-shift wrap documented there).
 *
 * Doubles as the pointer-capture grab target: a grab needs a focusable view, and captured-pointer
 * events are delivered to it (routed to [MouseForwarder.onCapturedPointer] via the listener the
@@ -530,6 +534,9 @@ private class KeyCaptureView(context: Context) : View(context) {
        isFocusableInTouchMode = true
    }

+    /** The session handle when the host types committed text; `0` = VK-only fallback. */
+    var textHandle: Long = 0L
+
    /** Whether [setImeVisible] last showed the IME — for toggle-style callers (remote pointer). */
    var imeShown = false
        private set
@@ -537,9 +544,16 @@ private class KeyCaptureView(context: Context) : View(context) {
    override fun onCheckIsTextEditor(): Boolean = true

    override fun onCreateInputConnection(outAttrs: EditorInfo): InputConnection {
-        outAttrs.inputType = InputType.TYPE_NULL
-        outAttrs.imeOptions = EditorInfo.IME_FLAG_NO_EXTRACT_UI or EditorInfo.IME_FLAG_NO_FULLSCREEN
-        return BaseInputConnection(this, false)
+        outAttrs.imeOptions = EditorInfo.IME_FLAG_NO_EXTRACT_UI or
+            EditorInfo.IME_FLAG_NO_FULLSCREEN or EditorInfo.IME_FLAG_NO_ENTER_ACTION
+        return if (textHandle != 0L) {
+            outAttrs.inputType = InputType.TYPE_CLASS_TEXT or
+                InputType.TYPE_TEXT_FLAG_AUTO_CORRECT or InputType.TYPE_TEXT_FLAG_MULTI_LINE
+            HostTextConnection(this, textHandle)
+        } else {
+            outAttrs.inputType = InputType.TYPE_NULL
+            BaseInputConnection(this, false)
+        }
    }

    fun setImeVisible(show: Boolean) {
@@ -554,3 +568,113 @@ private class KeyCaptureView(context: Context) : View(context) {
        }
    }
 }
+
+/**
+ * IME → host text bridge (the `HOST_CAP_TEXT_INPUT` path): a real **editable** connection, so
+ * the IME runs its full machinery (autocorrect, gesture typing, non-Latin composition), mirrored
+ * to the host as it happens. The one piece of host-side state tracked is *what the host currently
+ * shows of the active composition* ([sentComposition]): composing updates send a common-prefix
+ * diff (backspaces + the new suffix) so corrections materialize live on the host; a commit
+ * settles it. [setComposingRegion] adopts already-committed text as the active composition
+ * (autocorrect-revert / backspace-into-word flows), so the next update diffs against it instead
+ * of retyping. Newlines become Enter taps; [deleteSurroundingText] becomes Backspace/Delete taps.
+ *
+ * Known approximation: diff lengths are counted in Unicode scalars, assuming one host Backspace
+ * deletes one scalar — true for the composition text IMEs actually produce (emoji and other
+ * multi-unit graphemes commit directly rather than composing).
+ */
+private class HostTextConnection(
+    view: KeyCaptureView,
+    private val handle: Long,
+) : BaseInputConnection(view, true) {
+    /** What the host currently shows of the active composition ("" = none). */
+    private var sentComposition = ""
+
+    override fun commitText(text: CharSequence, newCursorPosition: Int): Boolean {
+        retype(text.toString())
+        sentComposition = ""
+        val ok = super.commitText(text, newCursorPosition)
+        trimEditable()
+        return ok
+    }
+
+    override fun setComposingText(text: CharSequence, newCursorPosition: Int): Boolean {
+        retype(text.toString())
+        return super.setComposingText(text, newCursorPosition)
+    }
+
+    override fun finishComposingText(): Boolean {
+        // The composition text stands as committed — the host already shows it verbatim.
+        sentComposition = ""
+        return super.finishComposingText()
+    }
+
+    override fun setComposingRegion(start: Int, end: Int): Boolean {
+        val e = editable
+        if (e != null) {
+            val a = start.coerceIn(0, e.length)
+            val b = end.coerceIn(0, e.length)
+            sentComposition = e.subSequence(minOf(a, b), maxOf(a, b)).toString()
+        }
+        return super.setComposingRegion(start, end)
+    }
+
+    override fun deleteSurroundingText(beforeLength: Int, afterLength: Int): Boolean {
+        repeat(beforeLength.coerceIn(0, MAX_TAPS)) { tapVk(VK_BACK) }
+        repeat(afterLength.coerceIn(0, MAX_TAPS)) { tapVk(VK_DELETE) }
+        return super.deleteSurroundingText(beforeLength, afterLength)
+    }
+
+    override fun performEditorAction(actionCode: Int): Boolean {
+        tapVk(VK_RETURN)
+        return true
+    }
+
+    /** Replace the host's view of the composition with [text] via a common-prefix diff. */
+    private fun retype(text: String) {
+        var common = sentComposition.commonPrefixWith(text)
+        // Never split a surrogate pair mid-diff — back off to the pair boundary.
+        if (common.isNotEmpty() && common.last().isHighSurrogate()) {
+            common = common.dropLast(1)
+        }
+        val stale = sentComposition.substring(common.length)
+        repeat(stale.codePointCount(0, stale.length).coerceAtMost(MAX_TAPS)) { tapVk(VK_BACK) }
+        sendText(text.substring(common.length))
+        sentComposition = text
+    }
+
+    /** Forward literal text, turning newlines into Enter taps (control chars never ride text). */
+    private fun sendText(s: String) {
+        var chunk = StringBuilder()
+        for (ch in s) {
+            if (ch == '\n') {
+                if (chunk.isNotEmpty()) {
+                    NativeBridge.nativeSendText(handle, chunk.toString())
+                    chunk = StringBuilder()
+                }
+                tapVk(VK_RETURN)
+            } else {
+                chunk.append(ch)
+            }
+        }
+        if (chunk.isNotEmpty()) NativeBridge.nativeSendText(handle, chunk.toString())
+    }
+
+    private fun tapVk(vk: Int) {
+        NativeBridge.nativeSendKey(handle, vk, true, 0)
+        NativeBridge.nativeSendKey(handle, vk, false, 0)
+    }
+
+    /** Bound the mirror buffer: once nothing is composing, old text serves no purpose. */
+    private fun trimEditable() {
+        val e = editable ?: return
+        if (getComposingSpanStart(e) == -1 && e.length > 4000) e.clear()
+    }
+
+    private companion object {
+        const val VK_BACK = 0x08
+        const val VK_RETURN = 0x0D
+        const val VK_DELETE = 0x2E
+        const val MAX_TAPS = 256
+    }
+}
@@ -287,6 +287,20 @@ object NativeBridge {
    /** One key transition. vk: Windows VK (0 = dropped by Rust). mods: VK modifier mask (0 for now). */
    external fun nativeSendKey(handle: Long, vk: Int, down: Boolean, mods: Int)

+    /**
+     * Whether the host advertised committed-text injection (`HOST_CAP_TEXT_INPUT`) — its inject
+     * backend can type Unicode text directly. Picks the real IME `InputConnection` (autocorrect,
+     * gesture typing, non-Latin scripts) over the TYPE_NULL raw-key fallback. False on `0`.
+     */
+    external fun nativeTextInputSupported(handle: Long): Boolean
+
+    /**
+     * Committed IME text → one `TextInput` wire event per Unicode scalar, in order. Control
+     * characters are skipped natively (Enter/Backspace ride [nativeSendKey]). Only meaningful
+     * when [nativeTextInputSupported] returned true — older hosts ignore the events.
+     */
+    external fun nativeSendText(handle: Long, text: String)
+
    // ---- Gamepad: each controller forwarded on its own wire pad index (0..15, low byte of flags) ----
    // The pad index is assigned per Android device by GamepadRouter; a single controller lands on 0,
    // so its wire is byte-identical to the old single-pad path. The core folds the per-transition
@@ -6,11 +6,11 @@
 //! conventions: buttons 1=left/2=middle/3=right/4=X1/5=X2; scroll axis 0=vertical/1=horizontal,
 //! signed 120-unit delta, +=up/right; keys are Windows VK (mapped from KEYCODE_* on the Kotlin side).

-use jni::objects::{JByteBuffer, JObject};
+use jni::objects::{JByteBuffer, JObject, JString};
 use jni::sys::{jboolean, jint, jlong};
 use jni::JNIEnv;
 use punktfunk_core::input::{InputEvent, InputKind};
-use punktfunk_core::quic::{RichInput, HID_REPORT_MAX};
+use punktfunk_core::quic::{RichInput, HID_REPORT_MAX, HOST_CAP_TEXT_INPUT};

 use super::SessionHandle;

@@ -145,6 +145,45 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendKey(
    send_event(handle, kind, vk as u32, 0, 0, mods as u32);
 }

+/// `NativeBridge.nativeTextInputSupported(handle)` — whether the host advertised
+/// `HOST_CAP_TEXT_INPUT` (its inject backend types committed text), so the Kotlin side can pick
+/// the real IME `InputConnection` over the TYPE_NULL raw-key fallback. `0` handle → false.
+#[no_mangle]
+pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeTextInputSupported(
+    _env: JNIEnv,
+    _this: JObject,
+    handle: jlong,
+) -> jboolean {
+    if handle == 0 {
+        return 0;
+    }
+    // SAFETY: live handle per the nativeConnect/nativeClose contract; host_caps is &self.
+    let h = unsafe { &*(handle as *const SessionHandle) };
+    u8::from(h.client.host_caps() & HOST_CAP_TEXT_INPUT != 0)
+}
+
+/// `NativeBridge.nativeSendText(handle, text)` — committed IME text, one `TextInput` event per
+/// Unicode scalar (`code` = the scalar; multi-char commits are consecutive events in order).
+/// Control characters are skipped — Enter/Backspace/Tab ride the VK key path. Call only when
+/// [`Java_io_unom_punktfunk_kit_NativeBridge_nativeTextInputSupported`] returned true.
+#[no_mangle]
+pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendText(
+    mut env: JNIEnv,
+    _this: JObject,
+    handle: jlong,
+    text: JString,
+) {
+    if handle == 0 {
+        return;
+    }
+    let Ok(s) = env.get_string(&text) else {
+        return;
+    };
+    for ch in String::from(s).chars().filter(|c| !c.is_control()) {
+        send_event(handle, InputKind::TextInput, ch as u32, 0, 0, 0);
+    }
+}
+
 // ---- Gamepad: Kotlin captures (KeyEvent/MotionEvent) → NativeClient::send_input ---------------
 // Multi-pad model: each physical controller is forwarded on its own wire pad index (0..15), carried
 // in the low byte of `flags` on every per-pad event — the Kotlin side (`GamepadRouter`) assigns a