feat(core+host+android): committed-text input plane (IME path) — InputKind::TextInput
The VK key-event vocabulary cannot express text an input method COMMITS (autocorrect, gesture typing, non-Latin scripts, emoji). Add a first-class text event and negotiate it: - punktfunk-core: InputKind::TextInput (= 15) carries one Unicode scalar per event in `code`; HOST_CAP_TEXT_INPUT (0x04) in Welcome::host_caps. - Host advertises the cap only where the session's inject backend can type text: Windows SendInput (KEYEVENTF_UNICODE, surrogate-pair aware) and the Linux wlroots backend — a dedicated second zwp_virtual_keyboard whose xkb keymap grows Unicode keysyms on demand (the wtype model), so keymap re-uploads never disturb the main device's layout/modifier state. The KWin-fake-input/libei/gamescope backends can only press layout keycodes, so those sessions don't set the bit and clients keep the VK fallback. - GameStream plane: Moonlight's UTF-8 text packet (MAGIC_UTF8, previously recognized-and-dropped) now decodes to the same TextInput events. - Android: KeyCaptureView picks a real editable InputConnection when the host has the cap — the IME runs its full machinery, mirrored to the host live via common-prefix diffs of the composition (backspaces + new suffix), with setComposingRegion adopting committed text so autocorrect-revert flows diff instead of retyping; newline→Enter, deleteSurroundingText→Backspace/Delete. Older hosts keep the TYPE_NULL raw-key path unchanged. - keymap: media VKs (0xB0-0xB3) → evdev so the Android media keys land on Linux hosts too. Verified: punktfunk-core + host gamestream + pf-inject tests green on Linux (Ubuntu box), clippy clean; Android app+native builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,7 +35,24 @@ pub fn decode(plaintext: &[u8]) -> Vec<InputEvent> {
|
||||
if plaintext.len() < 4 || u16::from_le_bytes([plaintext[0], plaintext[1]]) != INPUT_DATA_TYPE {
|
||||
return Vec::new();
|
||||
}
|
||||
decode_input_packet(&plaintext[4..]).into_iter().collect()
|
||||
let p = &plaintext[4..];
|
||||
// UTF-8 text (Moonlight's client-side keyboard commit) expands to one `TextInput` event per
|
||||
// Unicode scalar — the only magic yielding more than one event, so it's handled before the
|
||||
// single-event dispatch. Injected the same way as the native plane's IME text.
|
||||
if p.len() >= 8 && u32::from_le_bytes([p[4], p[5], p[6], p[7]]) == MAGIC_UTF8 {
|
||||
// NV_INPUT_HEADER.size (BE, excludes itself) counts magic + body.
|
||||
let size = u32::from_be_bytes([p[0], p[1], p[2], p[3]]) as usize;
|
||||
let body_len = size.saturating_sub(4).min(p.len() - 8);
|
||||
return match std::str::from_utf8(&p[8..8 + body_len]) {
|
||||
Ok(s) => s
|
||||
.chars()
|
||||
.filter(|c| !c.is_control())
|
||||
.map(|c| ev(InputKind::TextInput, c as u32, 0, 0, 0))
|
||||
.collect(),
|
||||
Err(_) => Vec::new(),
|
||||
};
|
||||
}
|
||||
decode_input_packet(p).into_iter().collect()
|
||||
}
|
||||
|
||||
fn decode_input_packet(p: &[u8]) -> Option<InputEvent> {
|
||||
@@ -89,7 +106,7 @@ fn decode_input_packet(p: &[u8]) -> Option<InputEvent> {
|
||||
modifiers | crate::inject::KEY_FLAG_SEMANTIC_VK,
|
||||
)
|
||||
}
|
||||
// UTF-8 text, gamepad, pen, touch, haptics — not yet injected.
|
||||
// Gamepad, pen, touch, haptics — not yet injected. (UTF-8 text is handled in `decode`.)
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
@@ -144,6 +161,21 @@ mod tests {
|
||||
assert_eq!(ev[0].flags, 0x04 | crate::inject::KEY_FLAG_SEMANTIC_VK);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decodes_utf8_text_per_scalar() {
|
||||
// "aß😀" — ASCII, Latin-1, and an astral scalar; one TextInput event per scalar.
|
||||
let pt = wrap(MAGIC_UTF8, "aß😀".as_bytes());
|
||||
let ev = decode(&pt);
|
||||
assert_eq!(ev.len(), 3);
|
||||
assert!(ev.iter().all(|e| e.kind == InputKind::TextInput));
|
||||
assert_eq!(ev[0].code, 'a' as u32);
|
||||
assert_eq!(ev[1].code, 'ß' as u32);
|
||||
assert_eq!(ev[2].code, 0x1F600);
|
||||
// Truncated / invalid UTF-8 decodes to nothing rather than mojibake.
|
||||
let bad = wrap(MAGIC_UTF8, &[0xff, 0xfe]);
|
||||
assert!(decode(&bad).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ignores_non_input_type() {
|
||||
let mut pt = vec![0x00, 0x02]; // type 0x0200 (keepalive)
|
||||
|
||||
@@ -444,6 +444,15 @@ pub(super) async fn negotiate(
|
||||
punktfunk_core::quic::HOST_CAP_CLIPBOARD
|
||||
} else {
|
||||
0
|
||||
}
|
||||
// Committed-text injection (InputKind::TextInput): only where the session's inject
|
||||
// backend can actually type text — Windows SendInput (KEYEVENTF_UNICODE) and the
|
||||
// Linux wlroots virtual keyboard (dynamic Unicode keymap). Clients without the bit
|
||||
// keep their VK-synthesis fallback for IME text.
|
||||
| if crate::inject::text_input_supported() {
|
||||
punktfunk_core::quic::HOST_CAP_TEXT_INPUT
|
||||
} else {
|
||||
0
|
||||
},
|
||||
// The negotiated session AEAD (resolved above) + its 32-byte key toward a ChaCha
|
||||
// client; toward everyone else cipher 0 keeps the Welcome byte-identical to the
|
||||
|
||||
Reference in New Issue
Block a user