feat(core+host+android): committed-text input plane (IME path) — InputKind::TextInput

The VK key-event vocabulary cannot express text an input method COMMITS
(autocorrect, gesture typing, non-Latin scripts, emoji). Add a first-class
text event and negotiate it:

- punktfunk-core: InputKind::TextInput (= 15) carries one Unicode scalar per
  event in `code`; HOST_CAP_TEXT_INPUT (0x04) in Welcome::host_caps.
- Host advertises the cap only where the session's inject backend can type
  text: Windows SendInput (KEYEVENTF_UNICODE, surrogate-pair aware) and the
  Linux wlroots backend — a dedicated second zwp_virtual_keyboard whose xkb
  keymap grows Unicode keysyms on demand (the wtype model), so keymap
  re-uploads never disturb the main device's layout/modifier state. The
  KWin-fake-input/libei/gamescope backends can only press layout keycodes, so
  those sessions don't set the bit and clients keep the VK fallback.
- GameStream plane: Moonlight's UTF-8 text packet (MAGIC_UTF8, previously
  recognized-and-dropped) now decodes to the same TextInput events.
- Android: KeyCaptureView picks a real editable InputConnection when the host
  has the cap — the IME runs its full machinery, mirrored to the host live via
  common-prefix diffs of the composition (backspaces + new suffix), with
  setComposingRegion adopting committed text so autocorrect-revert flows diff
  instead of retyping; newline→Enter, deleteSurroundingText→Backspace/Delete.
  Older hosts keep the TYPE_NULL raw-key path unchanged.
- keymap: media VKs (0xB0-0xB3) → evdev so the Android media keys land on
  Linux hosts too.

Verified: punktfunk-core + host gamestream + pf-inject tests green on Linux
(Ubuntu box), clippy clean; Android app+native builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-22 19:19:11 +02:00
co-authored by Claude Fable 5
parent ddb72edcba
commit abe6228b42
14 changed files with 460 additions and 20 deletions
+41 -2
View File
@@ -6,11 +6,11 @@
//! conventions: buttons 1=left/2=middle/3=right/4=X1/5=X2; scroll axis 0=vertical/1=horizontal,
//! signed 120-unit delta, +=up/right; keys are Windows VK (mapped from KEYCODE_* on the Kotlin side).
use jni::objects::{JByteBuffer, JObject};
use jni::objects::{JByteBuffer, JObject, JString};
use jni::sys::{jboolean, jint, jlong};
use jni::JNIEnv;
use punktfunk_core::input::{InputEvent, InputKind};
use punktfunk_core::quic::{RichInput, HID_REPORT_MAX};
use punktfunk_core::quic::{RichInput, HID_REPORT_MAX, HOST_CAP_TEXT_INPUT};
use super::SessionHandle;
@@ -145,6 +145,45 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendKey(
send_event(handle, kind, vk as u32, 0, 0, mods as u32);
}
/// `NativeBridge.nativeTextInputSupported(handle)` — whether the host advertised
/// `HOST_CAP_TEXT_INPUT` (its inject backend types committed text), so the Kotlin side can pick
/// the real IME `InputConnection` over the TYPE_NULL raw-key fallback. `0` handle → false.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeTextInputSupported(
_env: JNIEnv,
_this: JObject,
handle: jlong,
) -> jboolean {
if handle == 0 {
return 0;
}
// SAFETY: live handle per the nativeConnect/nativeClose contract; host_caps is &self.
let h = unsafe { &*(handle as *const SessionHandle) };
u8::from(h.client.host_caps() & HOST_CAP_TEXT_INPUT != 0)
}
/// `NativeBridge.nativeSendText(handle, text)` — committed IME text, one `TextInput` event per
/// Unicode scalar (`code` = the scalar; multi-char commits are consecutive events in order).
/// Control characters are skipped — Enter/Backspace/Tab ride the VK key path. Call only when
/// [`Java_io_unom_punktfunk_kit_NativeBridge_nativeTextInputSupported`] returned true.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendText(
mut env: JNIEnv,
_this: JObject,
handle: jlong,
text: JString,
) {
if handle == 0 {
return;
}
let Ok(s) = env.get_string(&text) else {
return;
};
for ch in String::from(s).chars().filter(|c| !c.is_control()) {
send_event(handle, InputKind::TextInput, ch as u32, 0, 0, 0);
}
}
// ---- Gamepad: Kotlin captures (KeyEvent/MotionEvent) → NativeClient::send_input ---------------
// Multi-pad model: each physical controller is forwarded on its own wire pad index (0..15), carried
// in the low byte of `flags` on every per-pad event — the Kotlin side (`GamepadRouter`) assigns a