feat(core+host+android): committed-text input plane (IME path) — InputKind::TextInput
The VK key-event vocabulary cannot express text an input method COMMITS (autocorrect, gesture typing, non-Latin scripts, emoji). Add a first-class text event and negotiate it: - punktfunk-core: InputKind::TextInput (= 15) carries one Unicode scalar per event in `code`; HOST_CAP_TEXT_INPUT (0x04) in Welcome::host_caps. - Host advertises the cap only where the session's inject backend can type text: Windows SendInput (KEYEVENTF_UNICODE, surrogate-pair aware) and the Linux wlroots backend — a dedicated second zwp_virtual_keyboard whose xkb keymap grows Unicode keysyms on demand (the wtype model), so keymap re-uploads never disturb the main device's layout/modifier state. The KWin-fake-input/libei/gamescope backends can only press layout keycodes, so those sessions don't set the bit and clients keep the VK fallback. - GameStream plane: Moonlight's UTF-8 text packet (MAGIC_UTF8, previously recognized-and-dropped) now decodes to the same TextInput events. - Android: KeyCaptureView picks a real editable InputConnection when the host has the cap — the IME runs its full machinery, mirrored to the host live via common-prefix diffs of the composition (backspaces + new suffix), with setComposingRegion adopting committed text so autocorrect-revert flows diff instead of retyping; newline→Enter, deleteSurroundingText→Backspace/Delete. Older hosts keep the TYPE_NULL raw-key path unchanged. - keymap: media VKs (0xB0-0xB3) → evdev so the Android media keys land on Linux hosts too. Verified: punktfunk-core + host gamestream + pf-inject tests green on Linux (Ubuntu box), clippy clean; Android app+native builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -27,10 +27,10 @@ use windows::Win32::System::StationsAndDesktops::{
|
||||
use windows::Win32::UI::Input::KeyboardAndMouse::{
|
||||
GetKeyboardLayout, MapVirtualKeyExW, SendInput, HKL, INPUT, INPUT_0, INPUT_KEYBOARD,
|
||||
INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY, KEYEVENTF_KEYUP, KEYEVENTF_SCANCODE,
|
||||
MAPVK_VK_TO_VSC_EX, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_HWHEEL, MOUSEEVENTF_LEFTDOWN,
|
||||
MOUSEEVENTF_LEFTUP, MOUSEEVENTF_MIDDLEDOWN, MOUSEEVENTF_MIDDLEUP, MOUSEEVENTF_MOVE,
|
||||
MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_VIRTUALDESK, MOUSEEVENTF_WHEEL,
|
||||
MOUSEEVENTF_XDOWN, MOUSEEVENTF_XUP, MOUSEINPUT, VIRTUAL_KEY,
|
||||
KEYEVENTF_UNICODE, MAPVK_VK_TO_VSC_EX, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_HWHEEL,
|
||||
MOUSEEVENTF_LEFTDOWN, MOUSEEVENTF_LEFTUP, MOUSEEVENTF_MIDDLEDOWN, MOUSEEVENTF_MIDDLEUP,
|
||||
MOUSEEVENTF_MOVE, MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_VIRTUALDESK,
|
||||
MOUSEEVENTF_WHEEL, MOUSEEVENTF_XDOWN, MOUSEEVENTF_XUP, MOUSEINPUT, VIRTUAL_KEY,
|
||||
};
|
||||
use windows::Win32::UI::WindowsAndMessaging::{
|
||||
GetForegroundWindow, GetSystemMetrics, GetWindowThreadProcessId, SM_CXVIRTUALSCREEN,
|
||||
@@ -297,6 +297,33 @@ impl InputInjector for SendInputInjector {
|
||||
};
|
||||
self.send(&[key(ki)])
|
||||
}
|
||||
InputKind::TextInput => {
|
||||
// Committed IME text: one Unicode scalar per event, injected as
|
||||
// `KEYEVENTF_UNICODE` packets (wScan = UTF-16 unit, no scancode/layout involved
|
||||
// — the receiving app gets the character verbatim via WM_CHAR). An astral-plane
|
||||
// scalar (emoji) is its surrogate pair, each unit down+up in order — exactly how
|
||||
// Windows expects supplementary characters from unicode injection.
|
||||
let Some(ch) = char::from_u32(event.code) else {
|
||||
return Ok(()); // lone surrogate / out of range — drop
|
||||
};
|
||||
if ch.is_control() {
|
||||
return Ok(()); // control chars ride the VK path (Enter/Backspace/Tab)
|
||||
}
|
||||
let mut units = [0u16; 2];
|
||||
let mut inputs: Vec<INPUT> = Vec::with_capacity(4);
|
||||
for &unit in ch.encode_utf16(&mut units).iter() {
|
||||
for flags in [KEYEVENTF_UNICODE, KEYEVENTF_UNICODE | KEYEVENTF_KEYUP] {
|
||||
inputs.push(key(KEYBDINPUT {
|
||||
wVk: VIRTUAL_KEY(0),
|
||||
wScan: unit,
|
||||
dwFlags: flags,
|
||||
time: 0,
|
||||
dwExtraInfo: 0,
|
||||
}));
|
||||
}
|
||||
}
|
||||
self.send(&inputs)
|
||||
}
|
||||
// Gamepad goes through the XUSB backend. Touch: no SendInput equivalent -> no-op.
|
||||
InputKind::GamepadButton
|
||||
| InputKind::GamepadAxis
|
||||
|
||||
Reference in New Issue
Block a user