Compare commits

..

26 Commits

Author SHA1 Message Date
enricobuehler 6bd8c18b4d feat(clipboard): Linux + Windows host clipboard backends (Phase 1 host + Phase 3)
Host-side real-session-clipboard integration, reached through the portable
ClipCoordCmd seam so punktfunk1.rs stays platform-agnostic (design
clipboard-and-file-transfer.md §4). Advertised + started only when the operator
policy (PUNKTFUNK_CLIPBOARD, default off) allows it and the session mirrors a
live compositor.

clipboard/ module (gated cfg(any(linux, windows))):
- mod.rs — the unified HostClipboard enum + §3.5 wire<->platform normalization
  and the shared ClipEvent / PasteResponder.
- wayland.rs — ext-data-control-v1 (KWin / wlroots / Sway / Hyprland), the
  preferred Linux backend. On-glass verified on Hyprland.
- mutter.rs — GNOME. Mutter ships no wl/ext data-control at any version, so this
  talks to its DIRECT org.gnome.Mutter.RemoteDesktop clipboard (the xdg portal
  needs an interactive grant a headless host can't answer). On-glass on GNOME.
- windows.rs — the Win32 clipboard on a dedicated message-loop window:
  AddClipboardFormatListener -> WM_CLIPBOARDUPDATE for host copies, OLE delayed
  rendering (WM_RENDERFORMAT) for host pastes, per-window state via GWLP_USERDATA.
- winfmt.rs — pure Win32<->wire byte conversions (CF_HTML offset math, UTF-16
  text, RTF NUL trim), unit-tested on any host (cfg(any(windows, test))).
- session.rs — the backend-agnostic per-session coordinator owning all four data
  paths (host copy->client, client fetch, client copy->host, host paste).

Cargo.toml: windows crate gains Win32_System_DataExchange + Win32_System_Ole.

Verified: Linux builds + clippy + 282 host tests + fmt clean; the full host crate
compiles native on x86_64-pc-windows-msvc (checked on the RTX box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:08:36 +02:00
enricobuehler af3a7d8cd5 feat(clipboard): wire protocol + client-core task for shared clipboard (Phase 0)
The portable shared-clipboard plane in punktfunk-core, all behind the `quic`
feature (design/clipboard-and-file-transfer.md §3):

- Control messages 0x40–0x44 (ClipControl / ClipOffer / ClipFetch...) and the
  HOST_CAP_CLIPBOARD capability bit, negotiated in the Welcome caps.
- Per-transfer QUIC bi-streams ("PKFs" magic) for lazy fetch of offered content,
  with ClipFetchHdr status/size framing (quic::clipstream).
- The §3.5 portable wire-MIME vocabulary (text/plain;utf-8, text/html, text/rtf,
  image/png) shared by both ends.
- Client-side clipboard task (client.rs) + C ABI surface bumped to v6 (abi.rs,
  regenerated include/punktfunk_core.h).
- Loopback transport tests (quic::tests).

No OS clipboard integration yet — that is the host backends (Phase 1/3) and the
macOS client (Phase 1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:08:21 +02:00
enricobuehler 76594f27c1 feat(nvenc): default-on the Linux direct-SDK NVENC path
ci / web (push) Successful in 45s
ci / docs-site (push) Successful in 1m5s
decky / build-publish (push) Successful in 20s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
apple / swift (push) Successful in 4m49s
ci / bench (push) Successful in 6m13s
docker / deploy-docs (push) Successful in 20s
windows-host / package (push) Successful in 14m19s
arch / build-publish (push) Successful in 20m25s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 18m28s
android / android (push) Successful in 21m8s
deb / build-publish (push) Successful in 20m53s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 20m25s
ci / rust (push) Successful in 24m14s
apple / screenshots (push) Successful in 21m0s
On-glass validated 2026-07-12 on an RTX 5070 Ti host with a real Steam Deck
client (NV12 4:2:0, 1280x800@90): real nvEncInvalidateRefFrames landed 73/73
as clean P-frame recovery anchors (never IDR), losses consistently ~2 frames
deep — well inside the 5-frame DPB. That is the loss recovery the libav
hevc_nvenc path cannot express, so make the direct path the default on NVIDIA.

PUNKTFUNK_NVENC_DIRECT=0 (also false/no/off) is now the libav escape hatch.
Still gated on a CUDA capture payload — the `cuda` check in open_nvenc_probed
keeps AMD/Intel on VAAPI regardless, and the NVENC/CUDA entry points stay
dlopen'd at runtime (no new DT_NEEDED), so non-NVIDIA hosts are unaffected.
Packaging comments updated to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 13:34:53 +02:00
enricobuehler e89b2f60eb build(linux): enable --features nvenc in the arch/deb/rpm host builds
ci / web (push) Successful in 53s
ci / docs-site (push) Successful in 1m3s
decky / build-publish (push) Successful in 20s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 6s
ci / bench (push) Successful in 6m36s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m7s
docker / deploy-docs (push) Successful in 20s
deb / build-publish (push) Successful in 12m12s
apple / swift (push) Successful in 5m5s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m55s
arch / build-publish (push) Successful in 15m6s
android / android (push) Successful in 20m8s
ci / rust (push) Successful in 23m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 18m36s
apple / screenshots (push) Successful in 20m26s
The Phase 5.2 direct-SDK NVENC Linux backend (encode/linux/nvenc_cuda.rs) is
gated `#[cfg(all(target_os = "linux", feature = "nvenc"))]`, but no Linux
packager passed `--features nvenc` (it was historically a Windows-only feature
for the D3D11 NVENC path). So the module was compiled OUT of every arch/deb/rpm
canary regardless of commit — PUNKTFUNK_NVENC_DIRECT was a silent no-op on the
shipped Linux host. Add `--features punktfunk-host/nvenc` to all three package
builds so the code actually ships.

AMD/Intel-SAFE (verified): this is NOT the old Windows link-import crash. The
NVENC/CUDA entry points are dlopen'd at RUNTIME (libloading) — `objdump -p` shows
the feature build's DT_NEEDED is byte-identical to a plain build (no libcuda /
libnvidia-encode), so the binary starts fine driver-less. We use only the crate's
`sys::nvEncodeAPI` types (no code-running safe wrapper / lazy statics), cudarc
stays on `ci-check`/dynamic-loading (no CUDA toolkit at build), and the encoder is
only constructed for a CUDA capture frame + PUNKTFUNK_NVENC_DIRECT — never on a
VAAPI (AMD/Intel) host. The sysext images repackage these outputs, so they inherit
it; no other Linux host build compiles the binary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 12:17:19 +02:00
enricobuehler 63bc2bb10f docs(web-console): fix stale http:// URLs — the console serves HTTPS on :47992
ci / docs-site (push) Successful in 48s
ci / web (push) Successful in 53s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 10s
decky / build-publish (push) Successful in 17s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 14s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
arch / build-publish (push) Has been cancelled
ci / bench (push) Has been cancelled
ci / rust (push) Has been cancelled
deb / build-publish (push) Has been cancelled
android / android (push) Has been cancelled
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
windows-host / package (push) Has been cancelled
The punktfunk-web unit serves HTTP/1.1 over TLS with the host's own
self-signed identity cert, but several guides still told users to open
http://<host>:47992, which fails. Correct the scheme everywhere and note
the one-time browser cert warning in the canonical + SteamOS docs.

The RPM %post web hint was doubly wrong (http://<host-ip>:3000): wrong
scheme and wrong port — the service listens on :47992, not the :3000 dev
default. Also fixes scripts/web-init.sh, so the URL the SteamOS/Linux
installer prints at the end of setup is correct too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 12:14:52 +02:00
enricobuehler ad532b08a0 style(encode): rustfmt the direct-SDK NVENC Linux backend
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 52s
decky / build-publish (push) Successful in 17s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 6s
ci / bench (push) Successful in 6m1s
docker / deploy-docs (push) Successful in 20s
arch / build-publish (push) Successful in 12m23s
android / android (push) Successful in 13m26s
deb / build-publish (push) Successful in 14m27s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m22s
ci / rust (push) Successful in 18m25s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m14s
windows-host / package (push) Successful in 14m15s
apple / swift (push) Successful in 4m49s
apple / screenshots (push) Successful in 20m11s
`cargo fmt` was not run on the Phase 5.2 additions (nvenc_cuda.rs + the encode.rs
dispatcher fork), failing the ci.yml rust fmt job. Whitespace/wrapping only — no
behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 11:35:07 +02:00
enricobuehler 93093f3cf9 feat(encode): direct-SDK NVENC on Linux (CUDA input) with real RFI
ci / rust (push) Failing after 1m1s
ci / web (push) Successful in 1m3s
ci / docs-site (push) Successful in 1m9s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 10s
decky / build-publish (push) Successful in 17s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 57s
apple / swift (push) Successful in 4m42s
ci / bench (push) Successful in 7m37s
docker / deploy-docs (push) Successful in 22s
flatpak / build-publish (push) Successful in 6m51s
windows-host / package (push) Successful in 14m26s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m59s
arch / build-publish (push) Successful in 16m48s
deb / build-publish (push) Successful in 17m14s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m49s
android / android (push) Successful in 18m24s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 18m28s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m36s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
release / apple (push) Successful in 22m56s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m40s
apple / screenshots (push) Successful in 18m53s
Phase 5.2 of design/encoder-recovery-hardening.md (design/linux-direct-nvenc.md).
The Linux NVIDIA host encodes through libavcodec `hevc_nvenc`, which structurally
cannot express `nvEncInvalidateRefFrames` — so every FEC-unrecoverable loss is a
full IDR and, since the client freezes-until-reanchor, a per-loss freeze for
RTT+IDR-encode. This ports the Windows raw-NVENC backend to
NV_ENC_DEVICE_TYPE_CUDA over the shared CUcontext so Linux NVIDIA gets the same
real RFI + F2 recovery-anchor + reset() stall lever + HDR-SEI/Main10 plumbing.

New `encode/linux/nvenc_cuda.rs` (`NvencCudaEncoder`):
- runtime-loaded entry table via `dlopen libnvidia-encode.so.1` (never link-time,
  mirroring the zerocopy::cuda libcuda loader) — one binary still starts on
  AMD/Intel Linux boxes and falls through to VAAPI/software;
- session on the shared CUcontext (zerocopy::cuda::context());
- an encoder-owned ring of registered CUDADEVICEPTR input surfaces
  (zerocopy::cuda::InputSurface + a contiguous-NV12 allocator), each captured
  DeviceBuffer device→device copied in via the existing copy_* helpers — mirrors
  the libav recycled-hwframe-pool copy, so zero regression vs today;
- config/RFI/anchor/reset ported from the Windows backend; sync-only (NVENC async
  is Windows-only, so that whole subsystem is dropped);
- Main10/HDR-SEI wired but inert until a Linux P010 capture path (Phase 5.1).

Wired behind PUNKTFUNK_NVENC_DIRECT (default OFF) in open_nvenc_probed; the Windows
path is untouched (no shared extraction in v1). Two on-hardware `#[ignore]` smokes
added.

Validated on .21 (RTX 5070 Ti, driver 610.43.03): builds on Linux under ci-check,
clippy-clean, full host suite 272/0, NV12 smoke (8 AUs, real invalidate_ref_frames
+ recovery_anchor on a P-frame) and YUV444 FREXT smoke (6 AUs, chroma_444) green;
Windows compile unaffected. Owed: the client-in-the-loop matrix (RFI-survives-ABR,
reset() heal, A/B vs libav) and the default flip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 11:18:21 +02:00
enricobuehler fdda7144ed fix(encode): harden loss-recovery correctness across host encoders (F1–F7)
Phases 1–4 of design/encoder-recovery-hardening.md — make the shipped RFI/
freeze-until-reanchor recovery honest and rebuild-safe across every backend.

F1 — frame-index domain desync: the encode loop now owns a session-lifetime
`au_seq`; `Encoder::submit_indexed(au_seq + inflight)` pins NVENC inputTimeStamp
and AMF LTR slots to the WIRE frame index, so `invalidate_ref_frames` compares
client frame numbers in the same domain and survives adaptive-bitrate rebuilds
(an internal counter desynced on the first rebuild → RFI silently dead / an AMF
force-ref onto a never-decoded frame). `FrameMsg.frame_index` →
`Session::seal_frame_at`; GameStream gets the same via `VideoPacketizer::
packetize(.., Some(idx))`.

F2 — Windows NVENC left the client frozen ~1s per loss: NVENC RFI was
transparent (no anchor tag) while the session glue armed the 750ms IDR cooldown,
so the freeze only lifted on the ~1s keyframe re-ask. NVENC now mirrors AMF —
`pending_anchor` tags the first post-invalidate AU (the clean re-anchor
P-frame) `recovery_anchor`, incl. the covering-range dedupe re-arm; the client
lifts at ~RTT.

F3 — speed-test probe filler burned video frame indexes: moved to its own index
space (`Packetizer::alloc_probe_index` + `Session::submit_probe_frame`) with a
second client reassembly window routed on FLAG_PROBE, gated on the new
VIDEO_CAP_PROBE_SEQ Hello bit (mid-session probes declined for older clients).

F4 — RFI range sanity cap: forward gaps wider than `packet::RFI_MAX_RANGE` (256)
resync via keyframe instead of an out-of-range RFI, host- and client-side
(client huge-gap → keyframe in `RfiRecovery::observe` + the pf-client-core pump).

F5 — reset() parity: Windows NVENC (teardown + lazy re-init), Linux VAAPI
(drop-inner), Linux NVENC (reopen from stored OpenArgs) now give the stall
watchdog a heal lever instead of ending the session.

F6 — sw.rs `pending: VecDeque` (was `Option`), killing the silent AU drop at
capturer pipeline depth > 1. F7 — doc sweep on the RFI/anchor comments.

Verified: punktfunk-core lib tests (macOS + Linux), full punktfunk-host suite on
Linux (RTX 5070 Ti), Windows compile. Owed: the on-glass client matrix (F2
freeze A/B, AMF LTR spike across a bitrate rebuild).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 11:17:19 +02:00
enricobuehler 0dc414f197 docs: guide for embedding the C ABI (webOS, Xbox, Tizen examples)
ci / web (push) Successful in 1m4s
ci / docs-site (push) Successful in 1m19s
decky / build-publish (push) Successful in 33s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 40s
apple / swift (push) Successful in 3m38s
ci / bench (push) Successful in 7m5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9m21s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8m25s
docker / deploy-docs (push) Successful in 35s
android / android (push) Successful in 12m34s
arch / build-publish (push) Successful in 15m3s
deb / build-publish (push) Successful in 16m18s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m7s
apple / screenshots (push) Successful in 19m20s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m17s
ci / rust (push) Successful in 23m12s
Developer integration guide for building a punktfunk client on any platform
by linking punktfunk-core through its stable C ABI: what the core does vs.
what the embedder supplies, build/link/cross-compile, the full client
lifecycle (identity/pairing, connect ladder, video+recovery loop, audio,
input, feedback planes, teardown), plus worked blueprints for webOS, Xbox
(GDK), and Tizen.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 10:49:13 +02:00
enricobuehler a95b518ef3 fix(windows): show app version on About screen + capitalize Punktfunk on licenses
ci / docs-site (push) Successful in 51s
ci / web (push) Successful in 57s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
decky / build-publish (push) Successful in 19s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 38s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 2m1s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 2m3s
apple / swift (push) Successful in 5m1s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m4s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m25s
ci / bench (push) Successful in 8m10s
docker / deploy-docs (push) Successful in 20s
arch / build-publish (push) Successful in 11m11s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m47s
android / android (push) Successful in 16m10s
deb / build-publish (push) Successful in 16m57s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 17m47s
ci / rust (push) Successful in 23m26s
apple / screenshots (push) Successful in 19m57s
The About settings card had no version at all — add an identity block (app name +
"Version <CARGO_PKG_VERSION>", the workspace version) at the top, the WinUI
Settings convention and matching the Apple client's "Version X" wording. Also
capitalize the brand name on the licenses screen (was lowercase "punktfunk").

Verified against the pinned windows-reactor source + cargo fmt --check; full
Windows link left to CI (Windows-only crate).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 02:46:45 +02:00
enricobuehler f77fdee3e9 style(host): fix clippy doc_lazy_continuation in reconfig_allowed
ci / web (push) Successful in 49s
ci / docs-site (push) Successful in 55s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
decky / build-publish (push) Successful in 17s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
ci / bench (push) Successful in 5m57s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Successful in 8m52s
android / android (push) Successful in 12m58s
arch / build-publish (push) Successful in 13m33s
deb / build-publish (push) Successful in 16m27s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 16m10s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 16m26s
apple / swift (push) Successful in 5m21s
ci / rust (push) Successful in 19m47s
apple / screenshots (push) Successful in 19m47s
New clippy (rust-1.96) flags the summary line after the 'Gated OFF for' bullet
list as a lazy list-continuation under -D warnings. Add a blank /// line so it
reads as its own paragraph (clippy's suggested fix), matching intent. Comment
only; no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 02:05:25 +02:00
enricobuehler a85be8e467 feat(displays): clearer virtual-display preset names + descriptions
ci / web (push) Successful in 1m0s
ci / docs-site (push) Successful in 1m15s
decky / build-publish (push) Successful in 18s
apple / swift (push) Successful in 4m55s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 31s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 1m2s
ci / rust (push) Failing after 7m8s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 6m21s
docker / deploy-docs (push) Successful in 24s
windows-host / package (push) Failing after 8m23s
ci / bench (push) Successful in 8m44s
arch / build-publish (push) Successful in 12m26s
android / android (push) Successful in 13m37s
deb / build-publish (push) Successful in 14m23s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m34s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 13m17s
release / apple (push) Successful in 22m53s
apple / screenshots (push) Successful in 20m47s
"Gaming rig" actually meant a dedicated/headless box you only ever stream from,
which confused users — rename it to "Headless box" and rewrite all five preset
summaries to be scenario-first and shorter (the console cards already show the
mechanics as badges). Updated across the host API summaries (mgmt.rs), the web
console labels (en/de), and the docs table + prose. The internal preset id
`gaming-rig` is unchanged (stable API / stored-policy / test contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 01:29:18 +02:00
enricobuehler 1dfcb0b2f6 feat(android): default-UI connect/wake modal (Material dialog)
Mirror the Apple client: the connect/wake overlay was showing the full-screen
aurora takeover in the default touch UI too. Make ConnectOverlay mode-aware —
gamepad/console keeps the aurora ConnectTakeover, the default UI now renders a
Material 3 AlertDialog over the host grid (inert scrim; Back/buttons cancel),
matching the app's other touch dialogs. Extract a shared connectCopy() so both
presentations read identically; ConnectTakeover is now console-only.

Screenshot scenes updated (touch phases -> modal over the host grid via
shootScreen; console stays a root capture); record-mode tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 01:29:18 +02:00
enricobuehler e87fd42cee feat(apple): default-UI connect/wake modal, auto-wake toggle, pixel-exact windowed streaming
Three client-UX changes that share the connect/present path (and settings files):

- Connect/wake overlay is now mode-aware: the console/gamepad UI keeps the
  full-screen aurora takeover, while the default (touch/desktop) UI shows a
  Liquid Glass modal over the host grid — the takeover looked out of place there.
- Add an auto-wake toggle (DefaultsKey.autoWake, default on) across macOS/iOS/tvOS
  Settings + the gamepad settings view; gate startSession/prepareWake and the
  gamepad "Wake & Connect" label on it. MAC-address learning stays always-on.
- Windowed sessions now stream at the window's native pixels (Match-window
  default-on) so the picture is 1:1 pixel-exact instead of the presenter
  resampling a fixed-mode frame; fullscreen reports full-display px, also 1:1.
  Also lands the mid-resize aspect-fit tracking (decoded contentSize) that keeps
  the picture undistorted after a resize.

swift build + swift test (121 tests) green; screenshot scenes verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 01:29:12 +02:00
enricobuehler dd02e1f402 feat(clients): unified full-screen connect/wake takeover + iOS/tvOS Wake-on-LAN
ci / docs-site (push) Successful in 1m8s
ci / web (push) Successful in 54s
decky / build-publish (push) Successful in 20s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 11s
android / android (push) Successful in 12m23s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
ci / bench (push) Successful in 6m3s
arch / build-publish (push) Successful in 12m57s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9m40s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7m4s
docker / deploy-docs (push) Successful in 11s
flatpak / build-publish (push) Successful in 5m57s
deb / build-publish (push) Successful in 13m26s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m50s
apple / swift (push) Successful in 4m31s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m8s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m44s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m7s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m9s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m45s
release / apple (push) Successful in 21m27s
ci / rust (push) Failing after 6m47s
apple / screenshots (push) Successful in 20m18s
Give instant feedback the moment a host is picked, and make the wake wait a
full-screen takeover instead of a modal card — unified into one ConnectOverlay
across every client:

- android: new ConnectOverlay (aurora backdrop; Connecting / Waking / timed-out
  phases) replaces the tiny inline "Connecting…" row and the WakeOverlay card.
  The dial phase is now cancelable and hands off to the wake wait in one frame.
- console (pf-console-ui): the connect/wake overlays become a full-screen aurora
  takeover (draw_takeover) instead of a centered card over a dim scrim; the
  Waking → Connecting handoff no longer blinks.
- apple: new ConnectOverlay mirrors it (macOS / iOS / tvOS), replacing the
  per-tile connecting spinner + the WakeOverlay card; instant "Connecting…" from
  model.phase, and the carousel is gated inactive during the dial.

Also enable Wake-on-LAN on iOS/tvOS now that the multicast entitlement is
approved: enable com.apple.developer.networking.multicast and flip
wakeOnLANAvailable to true on every platform (MACs were already learned from
mDNS, so wake works immediately).

Verified: Android compileDebugKotlin + screenshot renders; console clippy +
36 tests + rendered phases on Linux; Apple swift build + 121 tests + rendered
phases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 23:29:35 +02:00
enricobuehler 2271f67202 style: rustfmt the recovery + resize changes (Windows CI fmt check)
ci / web (push) Successful in 59s
ci / docs-site (push) Successful in 1m1s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 7s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 59s
apple / swift (push) Successful in 4m26s
ci / bench (push) Successful in 5m53s
ci / rust (push) Failing after 7m35s
windows-host / package (push) Failing after 8m56s
flatpak / build-publish (push) Failing after 8m12s
arch / build-publish (push) Successful in 11m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m54s
android / android (push) Successful in 13m28s
deb / build-publish (push) Successful in 13m42s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 13m16s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m42s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m33s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m0s
docker / deploy-docs (push) Successful in 20s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m48s
release / apple (push) Successful in 25m4s
apple / screenshots (push) Successful in 19m32s
The `cargo fmt --check` step on the x86_64-pc-windows-msvc job was
failing: the mid-stream loss-recovery and resize-overlay commits landed
with unformatted wraps across pf-presenter, pf-client-core, punktfunk-core,
pf-console-ui, and a few host files.

Applied `cargo fmt`, and hand-relocated two trailing comments in
session.rs (a decoded-frame note and the wrap-counter note) to their own
lines so rustfmt no longer column-aligns the following comment block to
a deep indent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 23:25:06 +02:00
enricobuehler 89aa6767f9 feat(resize): scrim+spinner resize overlay in the shared presenter (Windows + GTK4)
ci / docs-site (push) Failing after 29s
ci / web (push) Successful in 43s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
decky / build-publish (push) Successful in 33s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
docker / deploy-docs (push) Successful in 20s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m52s
apple / swift (push) Successful in 4m55s
ci / bench (push) Successful in 6m37s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m2s
arch / build-publish (push) Successful in 12m42s
android / android (push) Successful in 12m51s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m7s
deb / build-publish (push) Successful in 13m4s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m24s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 13m56s
apple / screenshots (push) Successful in 19m31s
ci / rust (push) Failing after 39s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 1m4s
flatpak / build-publish (push) Failing after 8m4s
The mid-stream Match-window trigger + Resolution tri-state already shipped on BOTH
desktop clients via the session-always punktfunk-session binary (pf-presenter D1/D2,
C1). This ports the last Apple-parity piece — the resize-in-progress indicator
(clients/apple ResizeIndicator/ResizeIndicatorView) — into the SHARED pf-presenter
overlay, so one implementation covers both the Windows and GTK4 session windows.

- ResizeIndicator (run.rs): the Apple state machine in Rust — `steering` (a switch was
  requested) shows it, `decoded` (a frame reached the target size) clears it, `tick`
  times it out after 2.5 s for a switch the host rejected/capped. The live drag stays
  sharp; only the host's 0.3-2 s virtual-display + encoder rebuild gap is covered. A
  present-while-resizing path keeps the spinner animating through that frame-less gap.
- DecodedImage::dimensions() (pf-client-core): the END signal — a decoded frame at the
  target size means the sharp new-mode picture is on glass (the accept ack alone lands
  ahead of the host's rebuild). Mirrors is_keyframe()'s cfg arms.
- FrameCtx.resizing (pf-presenter/overlay.rs) + Skia draw (pf-console-ui): a full-screen
  55% scrim + the shared rotating theme::spinner + "Resizing…" label. The overlay
  composites its own RGBA quad and can't sample the video to blur it as SwiftUI does, so
  a scrim stands in for the blur — same intent, one draw. resizing_since clocks the
  spinner; Drawn.resize_step defeats the damage gate so it redraws each frame.

Verified on Linux: pf-presenter/pf-console-ui/session/linux-client build + clippy
-D warnings clean; 8 pf-presenter tests green incl. 2 new ResizeIndicator tests.
Windows session-binary compile (cfg-symmetric) + live on-glass both pending.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:40:03 +02:00
enricobuehler 7cea893db5 feat(recovery): wire LTR-RFI loss recovery into every client
ci / web (push) Successful in 48s
ci / rust (push) Failing after 48s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 7s
ci / docs-site (push) Successful in 1m0s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 7s
decky / build-publish (push) Successful in 17s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 11s
apple / swift (push) Successful in 4m51s
ci / bench (push) Successful in 6m13s
docker / deploy-docs (push) Successful in 20s
windows-host / package (push) Failing after 8m22s
flatpak / build-publish (push) Failing after 8m4s
arch / build-publish (push) Successful in 11m53s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m45s
android / android (push) Successful in 12m53s
deb / build-publish (push) Successful in 13m4s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m16s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 13m29s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m42s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m2s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m29s
release / apple (push) Successful in 25m48s
apple / screenshots (push) Successful in 19m31s
Centralize the client-side loss-range detector in punktfunk-core so every
embedder shares one implementation instead of re-deriving the wrapping
frame-index arithmetic:

- NativeClient::note_frame_index(frame_index) folds each received AU (in
  receive order) through RfiRecovery::observe, firing a throttled RFI request
  for the exact lost span [first_missing, frame_index-1] on a forward gap. A
  host that can RFI (AMD LTR / NVENC) re-references a known-good frame instead
  of paying a 20-40x IDR spike; the frames_dropped-driven keyframe path stays
  the backstop for when the recovery frame itself is lost.
- Export request_rfi + note_frame_index over the C ABI (Apple client).
- Call it from the Android (hw+sw pumps), Apple (StreamPump + Stage2Pipeline
  via PunktfunkConnection.noteFrameIndex), and Windows in-process pumps.
  Linux/Deck inherit it through pf-client-core's session pump.
- Split the decision into a pure RfiRecovery::observe(frame_index, now) and add
  8 unit tests: arming, contiguous runs, exact lost-range, single-frame drop,
  the 100ms throttle (burst-suppress then re-open), reorder stragglers, and
  u32 wraparound (contiguous + gap-range).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:11:01 +02:00
enricobuehler e55ff1bb28 feat(recovery): clean mid-stream loss recovery — freeze-until-reanchor + AMD LTR-RFI
ci / rust (push) Failing after 53s
ci / docs-site (push) Successful in 54s
ci / web (push) Successful in 58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 18s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
apple / swift (push) Successful in 4m57s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m4s
docker / deploy-docs (push) Successful in 21s
windows-host / package (push) Failing after 8m42s
flatpak / build-publish (push) Failing after 8m6s
android / android (push) Successful in 11m59s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
arch / build-publish (push) Successful in 13m20s
deb / build-publish (push) Successful in 14m36s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m44s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m21s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m28s
release / apple (push) Successful in 25m54s
apple / screenshots (push) Successful in 19m35s
Removes the "gray frames with motion" artifact on Vulkan-Video clients and lets
AMD/NVENC hosts re-anchor after loss WITHOUT a 20-40x IDR spike.

Client (pf-client-core): after a reference loss the hardware decoder conceals the
missing-reference deltas (on RADV, a gray plate with new motion painted over) and
returns Ok. The pump now freezes on the last good picture until a clean re-anchor
instead of showing the concealment — lifting on a real IDR, an intra-refresh
recovery mark (2nd wave boundary), or an LTR-RFI recovery anchor (1st). The
frame_index gap is the early, precise loss signal and drives an RFI request.

Host recovery signals (inert unless the backend supports them):
- USER_FLAG_RECOVERY_POINT — intra-refresh wave boundary (NVENC constrained GDR).
- USER_FLAG_RECOVERY_ANCHOR — AMD LTR reference-frame-invalidation recovery frame.

AMD LTR-RFI (encode/windows/amf.rs) — the AMD twin of NVENC RFI. AMF's AVC/HEVC API
has no constrained-intra property (intra-refresh cannot heal; PSNR-proven), so the
only clean-recovery lever is user LTR: mark frames as long-term references, and on
loss force the next frame to re-reference the newest known-good one — a clean
P-frame, not an IDR. Two rotating LTR slots, ~0.5s mark cadence, on by default for
AVC/HEVC (PUNKTFUNK_NO_AMF_LTR disables). invalidate_ref_frames picks the newest LTR
before the loss; a range older than the live slots falls back to a keyframe.

Protocol (punktfunk-core): RfiRequest control message + NativeClient::request_rfi().
Host: RfiRequest dispatch -> invalidate_ref_frames (IDR fallback); an RFI success
anchors the keyframe cooldown so the client's frames_dropped echo of the same loss
is coalesced away rather than emitting a redundant IDR.

Spike: synthetic NV12 GPU source for headless AMF encoder testing.

Validated: core rfi_request_roundtrip; pf-client-core 31 unit tests
(incl. an_rfi_anchor_lifts_immediately); punktfunk-host builds + 271 tests on Linux;
punktfunk-host builds clean on Windows; real AMD iGPU spike (invalidate at frame 90
forced re-reference to LTR frame 60 — 180 frames, keyframes=1, no recovery IDR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 17:31:17 +02:00
enricobuehler 890c7531d8 Merge branch 'midstream-resize': mid-stream resolution resize
apple / swift (push) Successful in 4m19s
ci / rust (push) Failing after 46s
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 1m2s
decky / build-publish (push) Successful in 20s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
ci / bench (push) Successful in 5m17s
arch / build-publish (push) Successful in 10m44s
docker / deploy-docs (push) Successful in 11s
android / android (push) Successful in 16m43s
windows-host / package (push) Failing after 8m29s
flatpak / build-publish (push) Failing after 8m13s
deb / build-publish (push) Successful in 11m24s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m51s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m1s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m24s
release / apple (push) Successful in 24m35s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m24s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m2s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m25s
apple / screenshots (push) Successful in 19m55s
Lands the mid-stream resolution resize feature (client-driven Reconfigure so
the host's virtual display + encoder follow a resized client window without a
reconnect), all paths default OFF:

- host hardening H1-H5 + session-binary Match window (C1)
- Apple macOS/iPadOS Match-window trigger + settings (C3) and the resize
  overlay (blur + spinner) client UX
- Windows on-glass fixes: corrective-ack actual resolution + pf-vdisplay
  monitor re-arrival for out-of-list mid-stream modes
- Linux backend matrix + the live-reconfigure gate unit tests

Validated on-glass: Windows IDD-push (.173), Linux Mutter + KWin. Android
(C4) deferred; Apple full build pending on a Mac.
2026-07-11 15:59:07 +02:00
enricobuehler e6fbcecdb9 fix(clients/windows): GUI text inputs read the live value, not a stale render snapshot
ci / web (push) Successful in 50s
ci / docs-site (push) Successful in 1m5s
decky / build-publish (push) Successful in 23s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 16s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 14s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 14s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 1m0s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 2m33s
apple / swift (push) Successful in 4m24s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 2m18s
ci / bench (push) Successful in 7m10s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m17s
arch / build-publish (push) Successful in 11m38s
deb / build-publish (push) Successful in 12m6s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m9s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m46s
android / android (push) Successful in 17m7s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m48s
docker / deploy-docs (push) Successful in 22s
ci / rust (push) Successful in 26m33s
apple / screenshots (push) Successful in 19m29s
A component() page re-renders reliably only when its props change: root wraps
every screen in a stable animated border, so once the entrance tween settles the
reconciler skips that unchanged-props subtree and a page's own use_state writes
never force a re-render. Three text fields read their value at click time from
that stranded local state:

- PIN pairing sent an empty PIN, so pairing always failed with "wrong PIN, or not
  armed?" — the reported bug. The CLI --pair path bypasses the reactor and worked.
- "Add host" Connect captured the empty mount-time address and silently did
  nothing (you open the modal precisely when the host isn't being discovered, so
  no discovery tick re-renders the page while you type).
- Rename round-tripped the draft through an always-deferred AsyncSetState into a
  controlled text box, fighting the caret on fast typing and dropping the last
  character when Save was clicked before the write landed.

Fix: hold each field's live value in a use_ref cell written on every keystroke
and read at commit time (uncontrolled input), instead of a render-time snapshot.
Rename is seeded when its target changes and no longer re-renders the whole page
per keystroke. Reviewed the rest of the app (settings, speed test, library,
stream, connect/request-access/waking, forget) — all driven by root-state props
and wired correctly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:58:25 +02:00
enricobuehler f01c5e210c feat(resize/apple): resize overlay — blur + spinner during mid-stream resize
Make a Match-window resize deliberate instead of a stutter: blur the live
stream and show a spinner while the host rebuilds its virtual display +
encoder and VideoToolbox re-inits on the new-mode IDR. No new protocol —
driven entirely by existing client signals.

- ResizeIndicator (pure core, unit-tested): START = follower steering,
  END = a decoded frame at the target size, TIMEOUT = 2.5s safety net for a
  rejected/capped switch that never yields a new-size frame; re-arms only on
  a CHANGED target, not a repeated same-size drag.
- MatchWindowFollower.onResizeTarget fires the instant the window differs
  from the live mode (deduped via lastSteered); a new onDecodedSize callback
  threads each new-mode IDR's coded dims through StreamPump/Stage2Pipeline →
  SessionPresenter → both stream views.
- SessionModel gains @Published resizing (+ resizeTargeted/resizeDecoded, a
  tick on the 1 Hz stats timer, reset on disconnect); ContentView blurs the
  stream 16px and overlays ResizeIndicatorView while resizing (the 32px
  trust-prompt blur is unchanged and takes precedence).

tvOS declares the props but never fires the follower (it drives modes via
AVDisplayManager), so the overlay stays dormant there. Pure core verified on
the Linux toolchain; full AppKit/UIKit build pending on a Mac.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:58:14 +02:00
enricobuehler d294b3923c test(resize): extract + unit-test the live-reconfigure gate (H1 gamescope / H5 per-client-mode)
The Linux §6 on-glass matrix can validate the gamescope must-REJECT behavior only on
native-gamescope hardware (the NVIDIA dev box fails headless GBM allocation — a nested
Hyprland/sway/gamescope output comes up 0×0), so pin the gate down deterministically
instead: extract the inline `live_reconfig_ok` decision into a pure
`reconfig_allowed(compositor, per_client_mode)` and test it — gamescope rejects in every
identity mode, a per-client-mode policy rejects on every backend, and all other
compositors (plus the synthetic protocol-test source) with the default identity accept.

Also fmt-normalizes the re_add block from the prior commit (whitespace only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:56:59 +02:00
enricobuehler 4c18bb80ca feat(resize/win): mid-stream resize on-glass fixes — corrective-ack actual res + monitor re-arrival
On-glass validation on the .173 Windows IDD-push host confirmed the Reconfigure
protocol + host rebuild work end-to-end and genuinely change pixels for an
advertised mode (1920x1080 -> 1280x720: two SPS/IDR sets, ffprobe both res). It
also surfaced two gaps for out-of-EDID-list target modes, both fixed here.

Fix 2 (corrective ack carries the ACTUAL resolution): the H2/H3 corrective ack
recovered only the achieved REFRESH (interval_hz), taking width/height straight
from the request — so when a backend delivered a different RESOLUTION (Windows
pf-vdisplay falling back to its advertised mode) the client was told it got a
size it never received, and by the D2 discipline never re-asked. New
`delivered_mode(frame.{w,h}, interval)` derives the ack from the captured frame's
real dims (what the encoder opened at / the client decodes) in both the success
and rollback branches. Unit-tested.

Fix 1 (reach arbitrary mid-stream modes via monitor RE-ARRIVAL): the pf-vdisplay
driver freezes a monitor's advertised mode list at IOCTL_ADD, and IddCx exposes
no live update-modes DDI, so an in-place ChangeDisplaySettingsExW to a mode not
advertised at arrival returns DISP_CHANGE_BADMODE. The manager's mid-stream
reconfigure now REMOVEs + re-ADDs the driver monitor at the exact new mode,
reusing the slot's stable per-client id (EDID serial / ContainerId) so the OS
keeps identity + saved DPI. The rebuilt Monitor PRESERVES gen (lease/refcount
continuity) and the group restore snapshot; reisolate_after_swap re-isolates the
new target without recapturing it. Host-only — no driver change. One monitor
hotplug per switch (the design's accepted "re-arrival for everything").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:34:22 +02:00
enricobuehler 89ff326ebf feat(resize/apple): Match-window mid-stream resize trigger + settings (C3)
design/midstream-resolution-resize.md Phase 2, Apple client. The stream mode
follows the session window/scene: a windowed macOS window resize or an iPad
Stage Manager / Split View scene change renegotiates the host's virtual
display + encoder via the existing PunktfunkConnection.requestMode, so a
windowed session streams native-resolution pixels instead of scaling.
Decode/present need nothing — VideoToolbox recreates its session on the
keyframe-derived format-description change (§1 table).

- MatchWindowFollower (new): the shared D2 trigger discipline — physical
  pixels even-floored + clamped ≥320×200, debounce to resize-end, ≥1 s
  between requests, skip a size equal to the live mode, request each distinct
  size at most once (stops re-asking a rejected size / looping on a host
  rollback). Pure normalize/request core is unit-tested (MatchWindowTests).
- macOS StreamLayerView: fed from layoutPresenter() (bounds → convertToBacking),
  guarded to once-in-a-window.
- iOS StreamViewController: fed from viewDidLayoutSubviews (bounds × render
  scale); iOS-only (iPhone fullscreen no-ops, tvOS uses AVDisplayManager).
- Settings: "Match window" toggle in the Stream mode section (iOS + macOS),
  DefaultsKey.matchWindow, read per session by the follower.

Verified on a Linux Swift 6.1.2 toolchain (the app target needs AppKit/UIKit,
unavailable here): the real MatchWindowFollower.swift type-checks in Swift 5
mode against a connection stub, and the pure discipline + the follower's
decision path pass a standalone harness (drag-settle + grow → exactly two
switches, refresh preserved, no re-request loop). A full build + on-device run
(macOS window, iPad Stage Manager) remains for a Mac.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:36:03 +02:00
enricobuehler d0d9bd5bfb feat(resize): mid-stream resolution resize — host hardening (H1–H5) + session-binary Match window (C1)
design/midstream-resolution-resize.md Phase 0 + Phase 1.

Host (Phase 0):
- H1/H5: per-backend Reconfigure acceptance gate — reject for gamescope
  (all sub-modes; a resize must never relaunch the title) and under the
  per-client-mode identity policy (a resize would resolve a different
  display slot). Synthetic stays reconfigurable on purpose (the protocol
  test source; the C-ABI roundtrip test rides it). Plus a 500 ms host-side
  min-interval backstop against Reconfigure spam.
- H2: rollback/corrective ack — the data plane reports the mode actually
  live after a failed rebuild (or a refresh the backend capped) through a
  reconfig_result channel; the control task forwards it as a second
  accepted Reconfigured so the client's mode slot self-corrects.
- H3: live stats mode — SendStats reads a packed AtomicU64 (w|h|hz)
  updated on every switch instead of latching the session-start mode.
- H4: registry::retire(gen) — a mode-switch rebuild force-releases the
  superseded Linux display, so linger/forever keep-alive policies don't
  accumulate kept monitors at stale modes. VirtualOutput carries pool_gen
  (fresh AND reused) and the Pipeline tuple threads it to the switch arm.

Client (Phase 1, default off):
- Settings: match_window policy + persisted last window size; exposed as
  the Resolution tri-state (Native / Match window / explicit) in the Skia
  console, GTK and WinUI settings pages.
- pf-presenter: window opens at the persisted size; Hello mode follows the
  window's pixel size; D2 trigger discipline (400 ms debounce to
  resize-end, ≥1 s spacing, even-floor + ≥320×200 clamp, each distinct
  size requested at most once — covers rejects and host rollbacks) as a
  pure, unit-tested decision; HUD line + title refresh on a switch.
- Session binary wires both --connect and --browse paths; the WinUI shell
  is session-always, so this covers Windows too.

Verified: workspace tests + clippy green; synthetic --remode end-to-end;
live session-binary run (window at persisted 1000×600 → Hello 1000×600@60).
On-glass per-backend matrix (Mutter/KWin/gamescope-reject, keep-alive
accumulation) still pending before any default flip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 12:53:47 +02:00
103 changed files with 12407 additions and 809 deletions
+5 -1
View File
@@ -90,7 +90,11 @@ jobs:
git config --global --add safe.directory "$PWD"
# punktfunk-client-session is the Vulkan/Skia streamer the shell execs for a connect —
# both client binaries must ship (build-client-deb.sh installs both).
cargo build --release --locked \
# --features punktfunk-host/nvenc: the direct-SDK NVENC path (real RFI + recovery anchor on
# Linux NVIDIA; design/linux-direct-nvenc.md). AMD/Intel-safe — NVENC/CUDA is dlopen'd at
# runtime (no link-time dep; identical DT_NEEDED to a plain build), and the encoder is only
# constructed for a CUDA capture frame + PUNKTFUNK_NVENC_DIRECT, never on VAAPI hosts.
cargo build --release --locked --features punktfunk-host/nvenc \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session
- name: Build + smoke-boot web console (bun preset)
@@ -0,0 +1,277 @@
package io.unom.punktfunk
import androidx.activity.compose.BackHandler
import androidx.compose.animation.core.LinearEasing
import androidx.compose.animation.core.RepeatMode
import androidx.compose.animation.core.animateFloat
import androidx.compose.animation.core.infiniteRepeatable
import androidx.compose.animation.core.rememberInfiniteTransition
import androidx.compose.animation.core.tween
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.clickable
import androidx.compose.foundation.interaction.MutableInteractionSource
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.widthIn
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Bedtime
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.Icon
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.drawscope.Stroke
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import androidx.compose.ui.window.DialogProperties
/**
* Which phase of the connect flow to draw — the pure view model [ConnectOverlay] resolves from the
* live dial/wake state, so [ConnectTakeover] / [ConnectModal] can render (and be screenshot-tested)
* statelessly.
*/
internal sealed interface ConnectPhase {
val hostName: String
/** The dial is in flight (shown the instant a host is picked). */
data class Connecting(override val hostName: String) : ConnectPhase
/** A sleeping host is being Wake-on-LAN'd and we're waiting for it to advertise again. */
data class Waking(override val hostName: String, val seconds: Int, val connectsAfter: Boolean) : ConnectPhase
/** The wake wait ran out — offer retry / cancel. */
data class WakeTimedOut(override val hostName: String) : ConnectPhase
}
/** Per-phase copy, shared by the console takeover and the touch modal so both read identically. */
private data class ConnectCopy(
val title: String,
val subtitle: String,
/** Monospace the subtitle so a ticking seconds counter doesn't jitter its width. */
val monoSubtitle: Boolean,
val cancelLabel: String,
)
private fun connectCopy(phase: ConnectPhase): ConnectCopy = when (phase) {
is ConnectPhase.Connecting -> ConnectCopy(
"Connecting to ${phase.hostName}", "Establishing a secure connection…", false, "Cancel",
)
is ConnectPhase.Waking -> ConnectCopy(
"Waking ${phase.hostName}",
"Waiting for it to come online · ${phase.seconds}s",
true,
// A wake-only wait (no dial after) says "Stop Waiting"; a wake that will connect says "Cancel".
if (!phase.connectsAfter) "Stop Waiting" else "Cancel",
)
is ConnectPhase.WakeTimedOut -> ConnectCopy(
"${phase.hostName} didn't wake",
"It may still be booting, or it's powered off / off this network.",
false,
"Cancel",
)
}
/**
* The unified "getting you connected" feedback — one flow for BOTH phases of reaching a host, so the
* user gets feedback the instant they pick one and it flows seamlessly into a wake if the host turns
* out to be asleep:
*
* - **Connecting** ([connectingHostName] non-null): the dial is in flight. Shown immediately on tap,
* so a host that takes a beat to answer no longer looks like nothing happened.
* - **Waking** ([WakeController.waking] non-null): the dial failed on a sleeping host, so we're firing
* Wake-on-LAN and waiting for it to advertise again, escalating to a retry/cancel prompt on timeout.
*
* Presentation is mode-aware (mirrors the Apple client): in the **console / gamepad** UI it's a
* full-screen aurora [ConnectTakeover] — the same signature backdrop the console home uses, driven by
* the pad (B cancels, A retries once timed out) with a hint bar. In the **default touch** UI it's a
* Material [ConnectModal] over the host grid, matching the app's other dialogs — the aurora takeover
* looked out of place there.
*
* The two phases hand off within a single Compose frame (see ConnectScreen's `doConnectDirect` →
* `waker.start` → redial), so nothing blinks between them.
*/
@Composable
fun ConnectOverlay(
connectingHostName: String?,
waker: WakeController,
gamepadUi: Boolean,
onCancelConnect: () -> Unit,
) {
val waking = waker.waking
// Waking takes precedence (it only exists after a dial has failed) so a stray overlap can't strand
// the "Connecting…" phase over a wake in progress.
val phase = when {
waking != null && waking.timedOut -> ConnectPhase.WakeTimedOut(waking.hostName)
waking != null -> ConnectPhase.Waking(waking.hostName, waking.seconds, waking.connectsAfter)
connectingHostName != null -> ConnectPhase.Connecting(connectingHostName)
else -> return
}
// System Back / pad B (remapped) cancels whatever's in flight — a plain dial or the wake wait.
val cancel = { if (waking != null) waker.cancel() else onCancelConnect() }
if (gamepadUi) {
BackHandler { cancel() }
// A retries once a wake has timed out; B falls through to the BackHandler above.
GamepadNavEffect2D(
active = true,
onDirection = {},
onActivate = { if (phase is ConnectPhase.WakeTimedOut) waker.retry() },
)
ConnectTakeover(phase = phase, onCancel = cancel, onRetry = { waker.retry() })
} else {
// The AlertDialog owns its own scrim + system-Back handling (routed to cancel).
ConnectModal(phase = phase, onCancel = cancel, onRetry = { waker.retry() })
}
}
/**
* The default-UI presentation: a Material dialog over the host grid, matching the app's other touch
* dialogs. A spinner (or the sleep glyph once timed out) sits above the title; the scrim is inert so a
* stray tap can't drop a connect in flight — only the buttons or system Back cancel.
*/
@Composable
internal fun ConnectModal(
phase: ConnectPhase,
onCancel: () -> Unit,
onRetry: () -> Unit,
) {
val copy = connectCopy(phase)
val timedOut = phase is ConnectPhase.WakeTimedOut
AlertDialog(
onDismissRequest = onCancel,
properties = DialogProperties(dismissOnClickOutside = false),
icon = {
if (timedOut) {
Icon(Icons.Filled.Bedtime, contentDescription = null)
} else {
CircularProgressIndicator(modifier = Modifier.size(28.dp), strokeWidth = 3.dp)
}
},
title = { Text(copy.title, textAlign = TextAlign.Center) },
text = {
Text(
copy.subtitle,
textAlign = TextAlign.Center,
fontFamily = if (copy.monoSubtitle) FontFamily.Monospace else FontFamily.Default,
)
},
// No confirm action until the wake times out; then "Try Again" is the primary button.
confirmButton = {
if (timedOut) TextButton(onClick = onRetry) { Text("Try Again") }
},
dismissButton = {
TextButton(onClick = onCancel) { Text(copy.cancelLabel) }
},
)
}
/**
* The console / gamepad presentation: an opaque aurora backdrop with a centred spinner/title/subtitle
* for [phase], plus a bottom hint bar spelling out the pad actions (B cancels, A retries once timed
* out) — glyph-driven like every other console screen. onClick keeps the hints tappable too, so a
* user without a working pad can still get out.
*/
@Composable
internal fun ConnectTakeover(
phase: ConnectPhase,
onCancel: () -> Unit,
onRetry: () -> Unit,
) {
val copy = connectCopy(phase)
val timedOut = phase is ConnectPhase.WakeTimedOut
Box(
Modifier
.fillMaxSize()
// Swallow taps so the screen behind can't be touched through the takeover.
.clickable(interactionSource = remember { MutableInteractionSource() }, indication = null) {},
contentAlignment = Alignment.Center,
) {
GamepadAuroraBackground(Modifier.fillMaxSize())
Column(
Modifier.padding(horizontal = 40.dp).widthIn(max = 460.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(18.dp),
) {
if (timedOut) {
Box(Modifier.size(120.dp), contentAlignment = Alignment.Center) {
Icon(
Icons.Filled.Bedtime,
contentDescription = null,
tint = Color.White.copy(alpha = 0.9f),
modifier = Modifier.size(46.dp),
)
}
} else {
PulsingSpinner()
}
Text(
copy.title,
color = Color.White,
fontWeight = FontWeight.Bold,
fontSize = 24.sp,
textAlign = TextAlign.Center,
)
Text(
copy.subtitle,
color = Color.White.copy(alpha = 0.65f),
fontSize = 14.sp,
textAlign = TextAlign.Center,
fontFamily = if (copy.monoSubtitle) FontFamily.Monospace else FontFamily.Default,
)
}
val hints = buildList {
add(PadGlyph.hint('B', copy.cancelLabel, onClick = onCancel))
if (timedOut) add(PadGlyph.hint('A', "Try Again", onClick = onRetry))
}
GamepadHintBar(hints, Modifier.align(Alignment.BottomCenter).padding(bottom = 28.dp))
}
}
/**
* The connecting/waking indicator: a white progress ring inside two brand-violet halo rings that
* expand and fade on a staggered loop — a small sign of life so the takeover reads as working, not
* stalled.
*/
@Composable
private fun PulsingSpinner() {
val transition = rememberInfiniteTransition(label = "connectPulse")
val pulse by transition.animateFloat(
initialValue = 0f,
targetValue = 1f,
animationSpec = infiniteRepeatable(tween(1600, easing = LinearEasing), RepeatMode.Restart),
label = "pulse",
)
Box(Modifier.size(120.dp), contentAlignment = Alignment.Center) {
Canvas(Modifier.fillMaxSize()) {
val maxR = size.minDimension / 2f
for (i in 0..1) {
val p = (pulse + i * 0.5f) % 1f
drawCircle(
color = Color(0xFF8678F5).copy(alpha = (1f - p) * 0.35f),
radius = maxR * (0.42f + p * 0.58f),
style = Stroke(width = 2.dp.toPx()),
)
}
}
CircularProgressIndicator(
color = Color.White,
strokeWidth = 3.dp,
modifier = Modifier.size(54.dp),
)
}
}
@@ -88,6 +88,16 @@ private class RequestAccessState(val target: PendingTrust) {
val cancelled = AtomicBoolean(false)
}
/**
* A plain dial in flight — [hostName] labels the unified [ConnectOverlay]'s "Connecting…" phase, and
* [cancelled] lets its Cancel abort. The native connect is a blocking call with no abort, so Cancel
* returns the UI immediately and a late-arriving handle is torn down silently rather than navigating
* into a session the user already backed out of. Mirrors [RequestAccessState]'s late-result handling.
*/
private class ConnectAttempt(val hostName: String) {
val cancelled = AtomicBoolean(false)
}
@Composable
fun ConnectScreen(
settings: Settings,
@@ -107,6 +117,9 @@ fun ConnectScreen(
var port by remember { mutableStateOf("9777") }
var connecting by remember { mutableStateOf(false) }
var status by remember { mutableStateOf<String?>(null) }
// A plain dial in flight (drives the "Connecting…" phase of the full-screen ConnectOverlay); null
// when idle or when the request-access / wake flows own the screen instead.
var attempt by remember { mutableStateOf<ConnectAttempt?>(null) }
// The host streams at exactly this mode; "Native" settings resolve from the device display.
val (w, h, hz) = settings.effectiveMode(context)
@@ -267,11 +280,20 @@ fun ConnectScreen(
status = "Identity not ready yet — try again in a moment"
return
}
val thisAttempt = ConnectAttempt(name)
attempt = thisAttempt // shows the ConnectOverlay's "Connecting…" phase immediately
connecting = true
status = "Connecting to $targetHost:$targetPort"
status = null
discovery.stop() // free the Wi-Fi radio before the stream session
scope.launch {
val handle = connectNative(id, targetHost, targetPort, pinHex ?: "", CONNECT_TIMEOUT_MS)
// Cancelled mid-dial: the UI's already been returned (and discovery restarted) by
// cancelConnect — drop the just-opened session silently rather than navigating into it.
if (thisAttempt.cancelled.get()) {
if (handle != 0L) withContext(Dispatchers.IO) { NativeBridge.nativeClose(handle) }
return@launch
}
attempt = null
connecting = false
if (handle != 0L) {
if (pinHex == null) { // TOFU: pin what we observed (unpaired)
@@ -284,7 +306,9 @@ fun ConnectScreen(
} else {
discovery.start()
if (onFailure != null) {
status = ""
// Hand off to the wake-and-wait flow — clearing `attempt` above and setting
// `waker.waking` here land in one recompose, so the overlay slides
// Connecting → Waking without a blank frame.
onFailure()
} else {
status = "Connection failed — check host/port, PIN, and logcat"
@@ -293,6 +317,16 @@ fun ConnectScreen(
}
}
// Cancel a plain dial in flight (the overlay's "Connecting…" phase, B / Cancel). The native
// connect can't be aborted, so flag this attempt (a late handle is closed silently in
// doConnectDirect) and return the UI now, resuming the discovery we paused for the dial.
fun cancelConnect() {
attempt?.cancelled?.set(true)
attempt = null
connecting = false
discovery.start()
}
// Wake-aware connect. If auto-wake is on (Settings.autoWakeEnabled) and the target is a saved
// host with a learned MAC that ISN'T currently advertising, fire a wake packet and DIAL
// IMMEDIATELY — mDNS absence does NOT mean unreachable (a host reached over a routed network —
@@ -506,40 +540,21 @@ fun ConnectScreen(
Spacer(Modifier.height(24.dp))
status?.let {
// While connecting it's progress (spinner, neutral); otherwise it's a
// result/error (red). Previously every status showed in error-red, so a
// normal "Connecting…" looked like a failure.
if (connecting) {
Row(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
CircularProgressIndicator(
modifier = Modifier.size(16.dp),
strokeWidth = 2.dp,
)
Text(
it,
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
} else {
// Result/error: a filled error container reads as a real failure banner,
// not just red text lost in the layout.
Surface(
color = MaterialTheme.colorScheme.errorContainer,
shape = MaterialTheme.shapes.medium,
modifier = Modifier.fillMaxWidth(),
) {
Text(
it,
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onErrorContainer,
textAlign = TextAlign.Center,
modifier = Modifier.padding(horizontal = 16.dp, vertical = 12.dp),
)
}
// In-flight progress (connecting / waking) is the full-screen ConnectOverlay's
// job now, so `status` only ever carries a result/error here — a filled error
// container reads as a real failure banner, not just red text lost in the layout.
Surface(
color = MaterialTheme.colorScheme.errorContainer,
shape = MaterialTheme.shapes.medium,
modifier = Modifier.fillMaxWidth(),
) {
Text(
it,
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onErrorContainer,
textAlign = TextAlign.Center,
modifier = Modifier.padding(horizontal = 16.dp, vertical = 12.dp),
)
}
Spacer(Modifier.height(16.dp))
}
@@ -837,8 +852,15 @@ fun ConnectScreen(
}
}
// Topmost: the "Waking…" overlay rides over both the touch grid and the console home.
WakeOverlay(waker, gamepadUi)
// Topmost: the full-screen connect takeover — instant "Connecting…" feedback on any dial, flowing
// seamlessly into the "Waking…" wait if the host turns out to be asleep. Rides over both the touch
// grid and the console home.
ConnectOverlay(
connectingHostName = attempt?.hostName,
waker = waker,
gamepadUi = gamepadUi,
onCancelConnect = { cancelConnect() },
)
}
/**
@@ -26,7 +26,7 @@ import kotlinx.coroutines.launch
* [isOnline]/[onOnline] callbacks all run on the main thread; only the blocking send is off-loaded.
*/
class WakeController(private val scope: CoroutineScope) {
/** null = idle; non-null drives [WakeOverlay]. */
/** null = idle; non-null drives the "Waking…" phase of [ConnectOverlay]. */
data class Waking(
val hostName: String,
/** Whether coming online chains into a connect (Wake & Connect) vs. just stopping. */
@@ -1,124 +0,0 @@
package io.unom.punktfunk
import androidx.activity.compose.BackHandler
import androidx.compose.foundation.background
import androidx.compose.foundation.border
import androidx.compose.foundation.clickable
import androidx.compose.foundation.interaction.MutableInteractionSource
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.widthIn
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Bedtime
import androidx.compose.material3.Button
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.Icon
import androidx.compose.material3.OutlinedButton
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
/**
* The "Waking <host>…" modal shown while [WakeController] brings a sleeping host back — a spinner + a
* live elapsed counter, escalating to a retry/cancel prompt on timeout. The Android mirror of the
* Apple client's `WakeOverlay`. Rendered over BOTH the touch grid and the console home; it swallows
* input to the screen behind it, and in console mode the pad drives it (B cancels, A retries once
* timed out) while the touch buttons work for a pointer.
*/
@Composable
fun WakeOverlay(waker: WakeController, gamepadUi: Boolean) {
val w = waker.waking ?: return
BackHandler { waker.cancel() } // system Back / pad B (remapped) cancels the wait
if (gamepadUi) {
// A retries once timed out; B falls through to the BackHandler above.
GamepadNavEffect2D(
active = true,
onDirection = {},
onActivate = { if (w.timedOut) waker.retry() },
)
}
Box(
Modifier
.fillMaxSize()
.background(Color.Black.copy(alpha = 0.6f))
// Swallow taps so the home behind can't be touched while waking.
.clickable(interactionSource = remember { MutableInteractionSource() }, indication = null) {},
contentAlignment = Alignment.Center,
) {
Column(
Modifier
.padding(40.dp)
.widthIn(max = 380.dp)
.clip(RoundedCornerShape(22.dp))
.background(Color(0xF01A1730))
.border(1.dp, Color.White.copy(alpha = 0.12f), RoundedCornerShape(22.dp))
.padding(28.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(14.dp),
) {
if (w.timedOut) {
Icon(
Icons.Filled.Bedtime,
contentDescription = null,
tint = Color.White.copy(alpha = 0.85f),
modifier = Modifier.size(34.dp),
)
Text(
"${w.hostName} didn't wake",
color = Color.White,
fontWeight = FontWeight.Bold,
fontSize = 19.sp,
textAlign = TextAlign.Center,
)
Text(
"It may still be booting, or it's powered off / off this network.",
color = Color.White.copy(alpha = 0.6f),
fontSize = 13.sp,
textAlign = TextAlign.Center,
)
Row(
horizontalArrangement = Arrangement.spacedBy(12.dp),
modifier = Modifier.padding(top = 6.dp),
) {
OutlinedButton(onClick = { waker.cancel() }) { Text("Cancel") }
Button(onClick = { waker.retry() }) { Text("Try Again") }
}
} else {
CircularProgressIndicator(color = Color.White)
Text(
"Waking ${w.hostName}",
color = Color.White,
fontWeight = FontWeight.Bold,
fontSize = 19.sp,
textAlign = TextAlign.Center,
)
Text(
"Waiting for it to come online · ${w.seconds}s",
color = Color.White.copy(alpha = 0.6f),
fontSize = 13.sp,
fontFamily = FontFamily.Monospace,
)
OutlinedButton(onClick = { waker.cancel() }, modifier = Modifier.padding(top = 6.dp)) {
Text(if (w.connectsAfter) "Cancel" else "Stop Waiting")
}
}
}
}
}
@@ -68,6 +68,29 @@ class ScreenshotTest {
@Config(sdk = [36], qualifiers = "w800dp-h360dp-xxhdpi")
fun streamNormal() = shootRoot("stream-normal") { StreamScene(io.unom.punktfunk.StatsVerbosity.NORMAL) }
// The touch flow is a Material dialog over the host grid (a separate window → shootScreen).
@Test
fun connecting() = shootScreen("connecting") {
HostsScene()
ConnectingScene()
}
@Test
fun waking() = shootScreen("waking") {
HostsScene()
WakingScene()
}
@Test
fun wakeTimedOut() = shootScreen("wake-timed-out") {
HostsScene()
WakeTimedOutScene()
}
// The console flow is the full-screen aurora takeover (a root capture).
@Test
fun connectingConsole() = shootRoot("connecting-console") { ConnectConsoleScene() }
@Test
fun trust() = shootScreen("trust") {
HostsScene()
@@ -26,6 +26,9 @@ import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import io.unom.punktfunk.BrandDark
import io.unom.punktfunk.ConnectModal
import io.unom.punktfunk.ConnectPhase
import io.unom.punktfunk.ConnectTakeover
import io.unom.punktfunk.Settings
import io.unom.punktfunk.TouchMode
import io.unom.punktfunk.SettingsScreen
@@ -215,3 +218,31 @@ internal fun StreamScene(verbosity: StatsVerbosity = StatsVerbosity.DETAILED) {
)
}
}
/**
* The default-UI connect flow (the real [ConnectModal]) in each phase — instant "Connecting…"
* feedback, the "Waking…" wait, and the wake-timed-out prompt. These render as a Material dialog over
* the host grid, so the test composes [HostsScene] behind them and captures the whole screen.
*/
@Composable
internal fun ConnectingScene() =
ConnectModal(ConnectPhase.Connecting("Living Room PC"), onCancel = {}, onRetry = {})
@Composable
internal fun WakingScene() =
ConnectModal(
ConnectPhase.Waking("Living Room PC", seconds = 12, connectsAfter = true),
onCancel = {}, onRetry = {},
)
@Composable
internal fun WakeTimedOutScene() =
ConnectModal(ConnectPhase.WakeTimedOut("Living Room PC"), onCancel = {}, onRetry = {})
/**
* The console / gamepad connect flow (the real full-screen [ConnectTakeover]) — the aurora backdrop
* with a bottom hint bar, the same signature look the console home uses.
*/
@Composable
internal fun ConnectConsoleScene() =
ConnectTakeover(ConnectPhase.Connecting("Living Room PC"), onCancel = {}, onRetry = {})
+9
View File
@@ -243,6 +243,11 @@ fn run_sync(
if pending.is_none() {
match client.next_frame(Duration::from_millis(5)) {
Ok(frame) => {
// Loss recovery (RFI): feed the frame index so a forward gap fires a throttled
// reference-frame-invalidation request — an RFI-capable host (AMD LTR / NVENC)
// recovers with a cheap clean P-frame instead of a full IDR. The frames_dropped
// keyframe path below stays the backstop when the recovery frame itself is lost.
let _ = client.note_frame_index(frame.frame_index);
if fed == 0 {
let p = &frame.data;
log::info!(
@@ -1026,6 +1031,10 @@ fn feeder_loop(
while !shutdown.load(Ordering::Relaxed) {
match client.next_frame(Duration::from_millis(5)) {
Ok(frame) => {
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference-frame-
// invalidation request so an RFI-capable host recovers with a cheap clean P-frame
// instead of a full IDR (the frames_dropped keyframe path is the backstop).
let _ = client.note_frame_index(frame.frame_index);
if stats.enabled() {
let received_ns = now_realtime_ns();
let clock_offset = clock_offset.load(Ordering::Relaxed) as i128;
+4 -12
View File
@@ -14,19 +14,11 @@
<!-- Wake-on-LAN needs to send a UDP broadcast magic packet (a sleeping host has no ARP
entry, so unicast can't wake it). Since iOS 14 / tvOS 14 the OS blocks sending to
broadcast/multicast addresses unless the app carries this managed entitlement — it must
be requested from and approved by Apple for the App ID, then enabled in the provisioning
profile. macOS is not gated by this (its App Sandbox network.client/server cover it).
GATED pending Apple's approval of the request (form filed) — an unauthorized managed
entitlement breaks iOS/tvOS signing, so it's commented out to keep those apps releasable.
ON APPROVAL: (1) uncomment the two lines below, and (2) flip
PunktfunkConnection.wakeOnLANAvailable (PunktfunkConnection.swift) to enable the iOS/tvOS
wake path + UI. Until then iOS/tvOS Wake-on-LAN is a clean no-op — MACs are still learned
from mDNS so it works immediately once ungated. macOS is unaffected (separate entitlements
file, no multicast entitlement needed). -->
<!--
be approved by Apple for the App ID and enabled in the provisioning profile. macOS is not
gated by this (its App Sandbox network.client/server cover it), hence its separate file.
Approved and provisioned, so it's enabled here and PunktfunkConnection.wakeOnLANAvailable
is true on iOS/tvOS too. -->
<key>com.apple.developer.networking.multicast</key>
<true/>
-->
</dict>
</plist>
@@ -60,7 +60,8 @@ struct ContentView: View {
@State private var speedTestTarget: StoredHost?
@State private var libraryTarget: StoredHost?
/// Wakes a sleeping host and waits for it to come back online before connecting (drives the
/// "Waking" overlay). macOS-only in practice WoL is gated off on iOS/tvOS.
/// "Waking" phase of the connect overlay). Available on every platform now that the iOS/tvOS
/// multicast entitlement is granted (see PunktfunkConnection.wakeOnLANAvailable).
@StateObject private var waker = HostWaker()
#if os(macOS)
/// Whether the hosting window is native-fullscreen right now (reported by
@@ -86,6 +87,10 @@ struct ContentView: View {
// with no (extended) controller attached tvOS falls back to HomeView as before.
@ObservedObject private var gamepadManager = GamepadManager.shared
@AppStorage(DefaultsKey.gamepadUIEnabled) private var gamepadUIEnabled = true
/// Auto-wake on connect (Settings General). On (default): a dial to an offline saved host
/// fires Wake-on-LAN up front and falls into the "Waking" wait if the dial fails. Off: connects
/// go straight through with no wake. The explicit "Wake Host" action is unaffected either way.
@AppStorage(DefaultsKey.autoWake) private var autoWakeEnabled = true
private var gamepadUIActive: Bool {
GamepadUIEnvironment.isActive(
gamepadConnected: gamepadManager.active != nil, enabledSetting: gamepadUIEnabled)
@@ -259,9 +264,26 @@ struct ContentView: View {
}
private var home: some View {
// The "Waking" overlay rides over BOTH home UIs (and the pre-connect window is still
// `home`, so it covers the whole wakeonlineconnect sequence).
homeBase.overlay { WakeOverlay(waker: waker) }
// The full-screen connect takeover rides over BOTH home UIs (and the pre-connect window is
// still `home`, so it covers the whole dial wake online connect sequence): instant
// "Connecting" feedback on any dial, flowing seamlessly into the "Waking" wait if the host
// turns out to be asleep.
homeBase.overlay {
ConnectOverlay(
connectingHostName: connectingOverlayName,
waker: waker,
gamepadUI: gamepadUIActive,
onCancelConnect: { model.disconnect() })
}
}
/// The host label for the connect takeover's "Connecting" phase a plain dial in flight. Nil
/// during the delegated-approval wait (that has its own "Waiting for approval" prompt, so the
/// takeover must not stack over it) and, of course, when idle or streaming.
private var connectingOverlayName: String? {
guard awaitingApproval == nil, model.phase == .connecting, let host = model.activeHost
else { return nil }
return host.displayName
}
@ViewBuilder private var homeBase: some View {
@@ -327,12 +349,25 @@ struct ContentView: View {
}()
return ZStack {
stream(captureEnabled: pendingFingerprint == nil)
.blur(radius: pendingFingerprint != nil ? 32 : 0)
// Blur the live stream during the trust prompt (heavy) and during a resize (lighter
// the deliberate "hold on" while the host rebuilds its pipeline and the decoder
// re-inits on the new-mode IDR). Only the resize blur animates; the trust blur snaps
// as before (its own overlay handles the transition).
.blur(radius: pendingFingerprint != nil ? 32 : (model.resizing ? 16 : 0))
.animation(.easeInOut(duration: 0.22), value: model.resizing)
.overlay {
if pendingFingerprint != nil {
Color.black.opacity(0.45)
}
}
// The resize spinner rides over the (blurred) stream; suppressed under the trust
// prompt, which owns the screen. It never hit-tests, so window-drag resizes keep
// steering and the next click still reaches the stream.
.overlay {
if pendingFingerprint == nil {
ResizeIndicatorView(active: model.resizing)
}
}
if let fp = pendingFingerprint {
TrustCardView(
fingerprint: fp,
@@ -410,6 +445,16 @@ struct ContentView: View {
onSessionEnd: { [weak model] in
Task { @MainActor in model?.sessionEnded() }
},
// Resize overlay START the follower is main-actor, so this drives the blur
// + spinner synchronously the instant the window differs from the live mode.
onResizeTarget: { [weak model] w, h in
model?.resizeTargeted(width: w, height: h)
},
// Resize overlay END the coded dims of each new-mode IDR, reported from the
// decode pump thread; hop to the main actor to clear the overlay.
onDecodedSize: { [weak model] w, h in
Task { @MainActor in model?.resizeDecoded(width: w, height: h) }
},
endToEndMeter: model.endToEnd,
decodeMeter: model.decodeStage,
displayMeter: model.displayStage
@@ -544,7 +589,8 @@ struct ContentView: View {
// packet up front, so a genuinely-asleep host is waking while the connect times out; only
// when that dial FAILS do we fall into the visible "Waking" wait a cold box takes far
// longer to boot than a connect will sit and redial once it's back on mDNS.
if PunktfunkConnection.wakeOnLANAvailable, !host.wakeMacs.isEmpty, !discovery.advertises(host) {
if autoWakeEnabled, PunktfunkConnection.wakeOnLANAvailable,
!host.wakeMacs.isEmpty, !discovery.advertises(host) {
discovery.start() // so the wake-wait can observe it reappear
startSessionDirect(
host, launchID: launchID, allowTofu: allowTofu,
@@ -601,7 +647,9 @@ struct ContentView: View {
private func prepareWake(for host: StoredHost) {
if let live = discovery.hosts.first(where: { host.matches($0) }) {
store.updateMacs(host.id, macs: live.macAddresses) // learn on every platform
} else if PunktfunkConnection.wakeOnLANAvailable, !host.wakeMacs.isEmpty {
} else if autoWakeEnabled, PunktfunkConnection.wakeOnLANAvailable, !host.wakeMacs.isEmpty {
// Auto-wake only: fire the up-front packet so a genuinely-asleep host is booting while the
// dial times out. With auto-wake off, connects go straight through (no packet).
let macs = host.wakeMacs
let ip = host.address
DispatchQueue.global(qos: .userInitiated).async {
@@ -0,0 +1,145 @@
// The unified "getting you connected" overlay one look for BOTH phases of reaching a host, so the
// user gets feedback the instant they pick one and it flows seamlessly into a wake if the host turns
// out to be asleep. The Apple mirror of the Android client's `ConnectOverlay` and the shared console
// UI's connect/wake takeover; it replaces the old centered-card `WakeOverlay`.
//
// - Connecting (`connectingHostName` non-nil): the dial is in flight. Shown immediately on activate
// so a host that takes a beat to answer no longer looks like nothing happened.
// - Waking (`waker.waking` non-nil): the dial failed on a sleeping host, so we're firing
// Wake-on-LAN and waiting for it to advertise again, escalating to a retry/cancel prompt on
// timeout.
//
// Presentation is mode-aware: the gamepad ("console") UI gets a full-screen aurora takeover the
// same living backdrop the console home wears, so it reads as a deliberate 10-foot moment; the
// default touch/desktop UI gets a Liquid Glass modal over a dim scrim, which sits right at home among
// the app's other floating surfaces (the trust card, the HUD) instead of a full-screen aurora that
// looked out of place there.
//
// The two phases hand off within a single view update (HostWaker clears `waking` and starts the
// connect in the same MainActor step), so the overlay never blinks between them. It swallows input to
// the screen behind it, and on iOS/macOS the pad drives it (B cancels, A retries once timed out).
import PunktfunkKit
import SwiftUI
struct ConnectOverlay: View {
/// Non-nil while a plain dial is in flight (the delegated-approval wait has its own prompt, so it
/// passes nil here). Drives the "Connecting" phase.
let connectingHostName: String?
@ObservedObject var waker: HostWaker
/// The console launcher is up full-screen aurora takeover; otherwise the default UI's Liquid
/// Glass modal.
var gamepadUI: Bool
/// Cancel a dial in flight tears down the (uncancelable) connect and returns the UI; the late
/// result is discarded by SessionModel's connect guard.
var onCancelConnect: () -> Void
private enum Phase {
case connecting(name: String)
case waking(HostWaker.Waking)
}
/// Waking takes precedence it only ever exists after a dial has already failed, so a stray
/// overlap can't strand the "Connecting" phase over a wake in progress.
private var phase: Phase? {
if let w = waker.waking { return .waking(w) }
if let name = connectingHostName { return .connecting(name: name) }
return nil
}
var body: some View {
if let phase {
ZStack {
if gamepadUI {
// Console: an opaque, living aurora over everything.
Color.black.ignoresSafeArea()
GamepadScreenBackground().ignoresSafeArea()
Color.clear.contentShape(Rectangle()).onTapGesture {}
content(phase).padding(40).frame(maxWidth: 460)
} else {
// Default UI: a Liquid Glass modal over a dim scrim.
Rectangle().fill(.black.opacity(0.5)).ignoresSafeArea()
.contentShape(Rectangle()).onTapGesture {}
content(phase)
.padding(28)
.frame(maxWidth: 380)
.glassBackground(RoundedRectangle(cornerRadius: 26, style: .continuous))
.overlay(
RoundedRectangle(cornerRadius: 26, style: .continuous)
.strokeBorder(.white.opacity(0.12), lineWidth: 1))
.padding(40)
}
}
.environment(\.colorScheme, .dark)
.transition(.opacity)
#if os(iOS) || os(macOS)
.background { ConnectControllerInput(waker: waker, onCancelConnect: onCancelConnect) }
#endif
}
}
@ViewBuilder private func content(_ phase: Phase) -> some View {
// The takeover carries larger type than the compact modal.
let titleSize: CGFloat = gamepadUI ? 24 : 19
let bodySize: CGFloat = gamepadUI ? 14 : 13
VStack(spacing: gamepadUI ? 16 : 14) {
switch phase {
case .connecting(let name):
ProgressView().controlSize(.large).tint(.white)
Text("Connecting to \(name)")
.font(.geist(titleSize, .bold, relativeTo: .title3)).foregroundStyle(.white)
.multilineTextAlignment(.center)
Text("Establishing a secure connection…")
.font(.geist(bodySize, relativeTo: .caption)).foregroundStyle(.white.opacity(0.6))
Button("Cancel") { onCancelConnect() }.buttonStyle(.bordered).padding(.top, 6)
case .waking(let w) where w.timedOut:
Image(systemName: "moon.zzz.fill")
.font(.system(size: gamepadUI ? 40 : 34)).foregroundStyle(.white.opacity(0.9))
Text("\(w.hostName) didn't wake")
.font(.geist(titleSize, .bold, relativeTo: .title3)).foregroundStyle(.white)
.multilineTextAlignment(.center)
Text("It may still be booting, or it's powered off / off this network.")
.font(.geist(bodySize, relativeTo: .caption)).foregroundStyle(.white.opacity(0.6))
.multilineTextAlignment(.center)
HStack(spacing: 12) {
Button("Cancel") { waker.cancel() }.buttonStyle(.bordered)
Button("Try Again") { waker.retry() }.glassProminentButtonStyle()
}
.padding(.top, 6)
case .waking(let w):
ProgressView().controlSize(.large).tint(.white)
Text("Waking \(w.hostName)")
.font(.geist(titleSize, .bold, relativeTo: .title3)).foregroundStyle(.white)
.multilineTextAlignment(.center)
Text("Waiting for it to come online · \(w.seconds)s")
.font(.geistFixed(bodySize)).foregroundStyle(.white.opacity(0.6)).monospacedDigit()
// A wake-only wait (no dial after) offers "Stop Waiting"; a wake-&-connect is "Cancel".
Button(w.connectsAfter ? "Cancel" : "Stop Waiting") { waker.cancel() }
.buttonStyle(.bordered).padding(.top, 6)
}
}
}
}
#if os(iOS) || os(macOS)
/// Controller binding for the overlay: B cancels whatever's in flight (a dial or the wake wait); A
/// retries once a wake has timed out. The closures read the live state on each press, so they stay
/// correct across the Connecting Waking handoff without the view re-mounting. A zero-size backing
/// view owning a `GamepadMenuInput` for the overlay's lifetime (the home is gated inactive while the
/// overlay is up, so nothing else is consuming the pad).
private struct ConnectControllerInput: View {
@ObservedObject var waker: HostWaker
var onCancelConnect: () -> Void
@State private var input = GamepadMenuInput(manager: .shared)
var body: some View {
Color.clear
.onAppear {
input.onBack = { if waker.waking != nil { waker.cancel() } else { onCancelConnect() } }
input.onConfirm = { if waker.waking?.timedOut == true { waker.retry() } }
input.start()
}
.onDisappear { input.stop() }
}
}
#endif
@@ -65,6 +65,9 @@ struct GamepadHomeView: View {
/// Same gate the touch grid's "Browse Library" context-menu item uses (default ON; the
/// Settings "Game library" toggle opts out).
@AppStorage(DefaultsKey.libraryEnabled) private var libraryEnabled = true
/// Auto-wake on connect (default ON) when off, activating an offline host just dials (no wake),
/// so the tile drops its "Wake & Connect" affordance for a plain "Connect".
@AppStorage(DefaultsKey.autoWake) private var autoWakeEnabled = true
#if os(iOS)
/// `.compact` in a landscape phone window drives tighter chrome so everything still fits.
@Environment(\.verticalSizeClass) private var vSizeClass
@@ -192,9 +195,12 @@ struct GamepadHomeView: View {
onActivate: { $0.activate() },
onSecondary: { openLibraryForSelected() },
onTertiary: { showSettings = true },
// Stop consuming the controller while another screen (or the wake overlay) is on top
// otherwise the launcher navigates behind it (invisibly on iPhone, visibly on iPad).
isActive: libraryTarget == nil && !showSettings && !showAddHost && waker.waking == nil
// Stop consuming the controller while another screen (or the connect/wake takeover) is on
// top otherwise the launcher navigates behind it (invisibly on iPhone, visibly on iPad),
// and a second A during a dial would launch a concurrent connect. `.connecting` covers the
// takeover's Connecting phase; `waker.waking` covers its Waking phase.
isActive: libraryTarget == nil && !showSettings && !showAddHost
&& waker.waking == nil && model.phase != .connecting
) { tile in
hostCard(tile, size: CGSize(width: cardWidth, height: cardHeight))
}
@@ -256,7 +262,7 @@ struct GamepadHomeView: View {
isConnecting: model.phase == .connecting && model.activeHost?.id == host.id,
filled: true,
hasLibrary: true,
canWake: PunktfunkConnection.wakeOnLANAvailable
canWake: autoWakeEnabled && PunktfunkConnection.wakeOnLANAvailable
&& !discovery.advertises(host) && !store.probedOnline.contains(host.id)
&& !host.wakeMacs.isEmpty,
activate: { connect(host) })
@@ -1,84 +0,0 @@
// The "Waking <host>" modal shown while HostWaker brings a sleeping host back a spinner + a
// live elapsed counter, escalating to a retry/cancel prompt on timeout. Presented over BOTH the
// touch and gamepad home (a wake only ever starts on macOS today, where WoL is ungated), and it
// drives from either a pointer (the buttons) or a controller (B cancels, A retries once timed out).
import PunktfunkKit
import SwiftUI
struct WakeOverlay: View {
@ObservedObject var waker: HostWaker
var body: some View {
if let w = waker.waking {
ZStack {
// Dim + swallow input to the home behind it.
Rectangle().fill(.black.opacity(0.6)).ignoresSafeArea()
.contentShape(Rectangle())
.onTapGesture {}
card(w)
.frame(maxWidth: 380)
.padding(28)
.consoleGlass(RoundedRectangle(cornerRadius: 22, style: .continuous))
.overlay(
RoundedRectangle(cornerRadius: 22, style: .continuous)
.strokeBorder(.white.opacity(0.12), lineWidth: 1))
.padding(40)
}
.environment(\.colorScheme, .dark)
.transition(.opacity)
#if os(iOS) || os(macOS)
.background { WakeControllerInput(waker: waker) }
#endif
}
}
@ViewBuilder private func card(_ w: HostWaker.Waking) -> some View {
VStack(spacing: 14) {
if w.timedOut {
Image(systemName: "moon.zzz.fill")
.font(.system(size: 34)).foregroundStyle(.white.opacity(0.85))
Text("\(w.hostName) didn't wake")
.font(.geist(19, .bold, relativeTo: .title3)).foregroundStyle(.white)
Text("It may still be booting, or it's powered off / off this network.")
.font(.geist(13, relativeTo: .caption)).foregroundStyle(.white.opacity(0.6))
.multilineTextAlignment(.center)
HStack(spacing: 12) {
Button("Cancel") { waker.cancel() }.buttonStyle(.bordered)
Button("Try Again") { waker.retry() }.glassProminentButtonStyle()
}
.padding(.top, 6)
} else {
ProgressView().controlSize(.large).tint(.white)
Text("Waking \(w.hostName)")
.font(.geist(19, .bold, relativeTo: .title3)).foregroundStyle(.white)
Text("Waiting for it to come online · \(w.seconds)s")
.font(.geistFixed(13)).foregroundStyle(.white.opacity(0.6))
.monospacedDigit()
Button(w.connectsAfter ? "Cancel" : "Stop Waiting") { waker.cancel() }
.buttonStyle(.bordered)
.padding(.top, 6)
}
}
}
}
#if os(iOS) || os(macOS)
/// Controller binding for the overlay: B cancels; A retries once it has timed out. A zero-size
/// backing view owning a `GamepadMenuInput` for the overlay's lifetime (the home carousel/list is
/// gated inactive while a wake is up, so nothing else is consuming the pad).
private struct WakeControllerInput: View {
@ObservedObject var waker: HostWaker
@State private var input = GamepadMenuInput(manager: .shared)
var body: some View {
Color.clear
.onAppear {
input.onBack = { waker.cancel() }
input.onConfirm = { if waker.waking?.timedOut == true { waker.retry() } }
input.start()
}
.onDisappear { input.stop() }
}
}
#endif
@@ -49,8 +49,24 @@ enum ShotScenes {
ShotScene(name: "08-gamepad-addhost", orientation: .natural, colorScheme: .dark) {
AnyView(ShotGamepadAddHost())
},
ShotScene(name: "09-waking", orientation: .natural, colorScheme: .dark) {
AnyView(ShotWaking())
ShotScene(name: "09-connecting", orientation: .natural, colorScheme: .dark) {
AnyView(ShotConnect(kind: .connecting))
},
ShotScene(name: "09b-waking", orientation: .natural, colorScheme: .dark) {
AnyView(ShotConnect(kind: .waking))
},
ShotScene(name: "09c-wake-timed-out", orientation: .natural, colorScheme: .dark) {
AnyView(ShotConnect(kind: .timedOut))
},
// The default-UI presentation (Liquid Glass modal over the touch grid) of the same phases.
ShotScene(name: "09d-connecting-modal", orientation: .natural, colorScheme: .dark) {
AnyView(ShotConnect(kind: .connecting, gamepadUI: false))
},
ShotScene(name: "09e-waking-modal", orientation: .natural, colorScheme: .dark) {
AnyView(ShotConnect(kind: .waking, gamepadUI: false))
},
ShotScene(name: "09f-wake-timed-out-modal", orientation: .natural, colorScheme: .dark) {
AnyView(ShotConnect(kind: .timedOut, gamepadUI: false))
},
]
#endif
@@ -137,23 +153,53 @@ private struct ShotGamepadAddHost: View {
var body: some View { GamepadAddHostView(onAdd: { _ in }) }
}
private struct ShotWaking: View {
/// The unified connect overlay (the real `ConnectOverlay`) in each phase instant "Connecting"
/// feedback, the "Waking" wait, and the wake-timed-out prompt. `gamepadUI` picks the presentation:
/// the console's full-screen aurora takeover over the gamepad home, or the default UI's Liquid Glass
/// modal over the touch host grid.
private struct ShotConnect: View {
enum Kind { case connecting, waking, timedOut }
let kind: Kind
var gamepadUI = true
@StateObject private var store = ShotMock.hostStore()
@StateObject private var model = SessionModel()
@StateObject private var discovery = HostDiscovery()
@StateObject private var waker = HostWaker()
var body: some View {
GamepadHomeView(
store: store, model: model, discovery: discovery,
libraryTarget: .constant(nil), waker: waker,
connect: { _ in }, connectDiscovered: { _ in }
)
.overlay { WakeOverlay(waker: waker) }
.onAppear {
waker.debugSet(.init(
hostID: store.hosts.first?.id ?? UUID(),
hostName: "Battlestation", connectsAfter: true, seconds: 14))
backdrop
.overlay {
ConnectOverlay(
connectingHostName: kind == .connecting ? "Battlestation" : nil,
waker: waker,
gamepadUI: gamepadUI,
onCancelConnect: {})
}
.onAppear {
switch kind {
case .connecting:
break
case .waking:
waker.debugSet(.init(
hostID: store.hosts.first?.id ?? UUID(),
hostName: "Battlestation", connectsAfter: true, seconds: 14))
case .timedOut:
waker.debugSet(.init(
hostID: store.hosts.first?.id ?? UUID(),
hostName: "Battlestation", connectsAfter: true, seconds: 90, timedOut: true))
}
}
}
@ViewBuilder private var backdrop: some View {
if gamepadUI {
GamepadHomeView(
store: store, model: model, discovery: discovery,
libraryTarget: .constant(nil), waker: waker,
connect: { _ in }, connectDiscovered: { _ in })
} else {
ShotHome()
}
}
}
@@ -22,7 +22,7 @@ final class HostWaker: ObservableObject {
var timedOut = false
}
/// nil = idle; non-nil drives `WakeOverlay`.
/// nil = idle; non-nil drives the "Waking" phase of `ConnectOverlay`.
@Published private(set) var waking: Waking?
/// How long to wait for the host to reappear before giving up. Generous a cold boot + service
@@ -0,0 +1,41 @@
// The resize overlay (design/midstream-resolution-resize.md client resize UX). A Match-window
// resize renegotiates the host's virtual display + encoder and re-inits the local VideoToolbox
// decoder on the first new-mode IDR an unavoidable sub-second gap where the last frame lingers,
// briefly freezes, or the picture pops to the new geometry. Rather than let that read as a stutter,
// we make it DELIBERATE: the caller blurs the live stream and this centered spinner + caption
// acknowledges the transition. It clears the instant a frame at the requested size decodes (the
// `onDecodedSize` END signal) or on the follower's safety timeout see `SessionModel.resizing`.
//
// Floating overlay, never a hit-test target: input keeps flowing to the stream underneath so a
// resize the user triggers by dragging the window never swallows their next click.
import PunktfunkKit
import SwiftUI
struct ResizeIndicatorView: View {
/// Mirrors `SessionModel.resizing`; the fade in/out is driven off this.
let active: Bool
var body: some View {
ZStack {
if active {
VStack(spacing: 12) {
ProgressView().controlSize(.large).tint(.white)
Text("Resizing…")
.font(.geist(15, .medium, relativeTo: .callout))
.foregroundStyle(.white.opacity(0.85))
}
.padding(.horizontal, 30)
.padding(.vertical, 24)
.glassBackground(RoundedRectangle(cornerRadius: 20, style: .continuous))
.overlay(
RoundedRectangle(cornerRadius: 20, style: .continuous)
.strokeBorder(.white.opacity(0.12), lineWidth: 1))
.transition(.opacity.combined(with: .scale(scale: 0.92)))
}
}
.environment(\.colorScheme, .dark) // the spinner + glass read over any frame
.animation(.easeInOut(duration: 0.22), value: active)
.allowsHitTesting(false) // the stream keeps receiving input the whole time
}
}
@@ -109,6 +109,16 @@ final class SessionModel: ObservableObject {
/// Mirrors StreamView's capture state (it owns the input capture; this drives the
/// HUD's "click to capture" / " releases" hint).
@Published var mouseCaptured = false
/// Resize overlay (design/midstream-resolution-resize.md client resize UX): true from the
/// instant a Match-window resize starts steering toward a new size until a frame at that size
/// decodes (or a safety timeout). Drives the blur+spinner so the unavoidable host-rebuild delay
/// reads as a deliberate, acknowledged transition instead of a stutter. Pure state lives in
/// `ResizeIndicator`; this mirrors its `active` for SwiftUI.
@Published private(set) var resizing = false
/// START = follower steering (main actor), END = a new-mode IDR's coded dims (decode pump,
/// hopped to main), TIMEOUT = safety net for a rejected/capped switch that never yields a
/// differently-sized frame. Ticked from the 1 Hz stats timer.
private var resizeIndicator = ResizeIndicator()
let meter = FrameMeter()
/// Capturereceived (the host+network stage), fed per AU at receipt by the stream view's
@@ -364,6 +374,8 @@ final class SessionModel: ObservableObject {
lostFrames = 0
lostPct = 0
mouseCaptured = false
resizing = false
resizeIndicator = ResizeIndicator() // no stale target/timer into the next session
}
/// Called (via the main actor) when the pump hits end-of-session.
@@ -374,6 +386,23 @@ final class SessionModel: ObservableObject {
errorMessage = "Session ended by \(name)."
}
/// Resize overlay START (main actor from the Match-window follower's `onResizeTarget`): the
/// window began differing from the live mode, so a `Reconfigure` toward `(width, height)` is
/// imminent. Show the blur+spinner immediately, before the debounced request even leaves.
func resizeTargeted(width: UInt32, height: UInt32) {
resizeIndicator.steering(
width: width, height: height, now: Date().timeIntervalSinceReferenceDate)
resizing = resizeIndicator.active
}
/// Resize overlay END (main actor hopped from the decode pump's `onDecodedSize`): a new-mode
/// IDR decoded at `(width, height)`. Clears the overlay only when that matches the size we're
/// steering to (a same-size loss-recovery IDR, or the initial connect IDR, is a no-op).
func resizeDecoded(width: Int, height: Int) {
resizeIndicator.decoded(width: UInt32(max(width, 0)), height: UInt32(max(height, 0)))
resizing = resizeIndicator.active
}
private func beginStreaming() {
guard let conn = connection else { return }
// Input capture itself is owned by StreamView (engaged by the captureEnabled
@@ -417,6 +446,11 @@ final class SessionModel: ObservableObject {
let timer = Timer(timeInterval: 1.0, repeats: true) { [weak self] _ in
guard let self else { return }
Task { @MainActor in
// Resize-overlay safety net: clear a stuck overlay when a targeted size never
// decodes (a rejected/capped switch). The decoded-frame END clears it promptly on
// success; this only fires after the timeout.
self.resizeIndicator.tick(now: Date().timeIntervalSinceReferenceDate)
self.resizing = self.resizeIndicator.active
let (frames, bytes, total) = self.meter.drain()
self.fps = frames
self.mbps = Double(bytes) * 8 / 1_000_000
@@ -36,6 +36,7 @@ struct GamepadSettingsView: View {
@AppStorage(DefaultsKey.hudPlacement) private var hudPlacement = HUDPlacement.topTrailing.rawValue
@AppStorage(DefaultsKey.libraryEnabled) private var libraryEnabled = true
@AppStorage(DefaultsKey.gamepadUIEnabled) private var gamepadUIEnabled = true
@AppStorage(DefaultsKey.autoWake) private var autoWakeEnabled = true
@AppStorage(DefaultsKey.presenter) private var presenter = SettingsOptions.presenterDefault
@ObservedObject private var gamepads = GamepadManager.shared
@@ -258,6 +259,11 @@ struct GamepadSettingsView: View {
+ "available on the host.",
options: SettingsOptions.compositors, current: compositor
) { compositor = $0 },
toggleRow(
id: "autoWake", icon: "power", label: "Auto-wake on connect",
detail: "Send Wake-on-LAN to a sleeping saved host and wait for it before "
+ "streaming. Off connects straight through.",
value: $autoWakeEnabled),
choiceRow(
id: "codec", header: "Video", icon: "film", label: "Video codec",
@@ -13,6 +13,11 @@ extension SettingsView {
// failed exactly one slice: the iOS archive (macOS/tvOS never compile that branch).
@ViewBuilder var streamModeSection: some View {
Section {
#if os(iOS) || os(macOS)
// Match-window (design/midstream-resolution-resize.md D1): follow the session
// window/scene, renegotiating the host mode on a resize. Off the explicit mode below.
Toggle("Match window", isOn: $matchWindow)
#endif
#if os(iOS)
iosResolutionWheel
iosRefreshRows
@@ -35,8 +40,12 @@ extension SettingsView {
} header: {
Text("Stream mode")
} footer: {
Text("The host creates a virtual output at exactly this mode — "
+ "native resolution, no scaling. \(Self.bitrateFooter)")
Text(matchWindow
? "The stream follows this window — the host resizes its virtual output to match "
+ "as you resize, so the picture stays pixel-exact (1:1) with no scaling. "
+ "\(Self.bitrateFooter)"
: "The host creates a virtual output at exactly this mode — native resolution, but "
+ "a window that isn't this size is scaled to fit. \(Self.bitrateFooter)")
.font(.geist(12, relativeTo: .caption))
.foregroundStyle(.secondary)
}
@@ -286,6 +295,24 @@ extension SettingsView {
}
}
/// Auto-wake on connect fire Wake-on-LAN + wait for a sleeping saved host to come back before
/// giving up. Now available on every platform (the iOS/tvOS multicast entitlement is granted).
@ViewBuilder var wakeSection: some View {
Section {
Toggle("Auto-wake on connect", isOn: $autoWakeEnabled)
} header: {
Text("Wake-on-LAN")
} footer: {
Text("Connecting to a saved host that isn't on the network yet sends a Wake-on-LAN "
+ "packet and waits for it to come back before streaming. Turn off if a host that's "
+ "already on just isn't visible here (e.g. over a VPN), so connects go straight "
+ "through instead of waiting out the wake. A host's “Wake” action still works either "
+ "way.")
.font(.geist(12, relativeTo: .caption))
.foregroundStyle(.secondary)
}
}
@ViewBuilder var windowSection: some View {
#if os(macOS)
Section {
@@ -21,6 +21,10 @@ struct SettingsView: View {
@AppStorage(DefaultsKey.streamWidth) var width = 1920
@AppStorage(DefaultsKey.streamHeight) var height = 1080
@AppStorage(DefaultsKey.streamHz) var hz = 60
// Default ON: a windowed session streams at the window's native pixels (1:1, no scaling) so it
// stays pixel-exact instead of the presenter resampling a fixed-mode frame into the window.
// Off falls back to the explicit mode below (fixed output, scaled to non-matching windows).
@AppStorage(DefaultsKey.matchWindow) var matchWindow = true
@AppStorage(DefaultsKey.compositor) var compositor = 0
@AppStorage(DefaultsKey.gamepadType) var gamepadType = 0
@AppStorage(DefaultsKey.bitrateKbps) var bitrateKbps = 0
@@ -44,6 +48,7 @@ struct SettingsView: View {
@AppStorage(DefaultsKey.hudPlacement) var hudPlacement = HUDPlacement.topTrailing.rawValue
@ObservedObject var gamepads = GamepadManager.shared
@AppStorage(DefaultsKey.gamepadUIEnabled) var gamepadUIEnabled = true
@AppStorage(DefaultsKey.autoWake) var autoWakeEnabled = true
#if DEBUG && !os(tvOS)
@State var showControllerTest = false
#endif
@@ -105,6 +110,7 @@ struct SettingsView: View {
Form {
streamModeSection
compositorSection
wakeSection
}
.formStyle(.grouped)
.tabItem { Label("General", systemImage: "gearshape") }
@@ -234,6 +240,7 @@ struct SettingsView: View {
streamModeSection
pointerSection
compositorSection
wakeSection
}
.formStyle(.grouped)
.navigationTitle("General")
@@ -304,6 +311,10 @@ struct SettingsView: View {
Binding(get: { gamepadUIEnabled ? "on" : "off" }, set: { gamepadUIEnabled = $0 == "on" })
}
private var autoWakeEnabledTag: Binding<String> {
Binding(get: { autoWakeEnabled ? "on" : "off" }, set: { autoWakeEnabled = $0 == "on" })
}
private var tvBody: some View {
let currentTag = "\(width)x\(height)x\(hz)"
let bounds = UIScreen.main.nativeBounds
@@ -343,9 +354,13 @@ struct SettingsView: View {
TVSelectionRow(
title: "10-bit HDR",
options: [("On", "on"), ("Off", "off")], selection: hdrEnabledTag)
TVSelectionRow(
title: "Auto-wake on connect",
options: [("On", "on"), ("Off", "off")], selection: autoWakeEnabledTag)
Text("The host creates a virtual output at exactly this mode — native "
+ "resolution, no scaling. \(Self.bitrateFooter) A specific compositor "
+ "is honored only if available on the host.")
+ "is honored only if available on the host. Auto-wake sends Wake-on-LAN to a "
+ "sleeping saved host and waits for it before streaming.")
.font(.geist(20, relativeTo: .caption))
.foregroundStyle(.secondary)
.multilineTextAlignment(.center)
@@ -70,19 +70,10 @@ func withOptionalCString<R>(_ s: String?, _ body: (UnsafePointer<CChar>?) -> R)
public extension PunktfunkConnection {
/// Whether the Wake-on-LAN broadcast path is usable on this platform/build. macOS can always
/// broadcast (its App Sandbox network entitlements cover it). iOS/tvOS need the managed
/// `com.apple.developer.networking.multicast` entitlement, which is GATED pending Apple's
/// approval (see `Config/Punktfunk.entitlements`) until it's granted, sending a broadcast is
/// blocked by the OS, so the wake path + its UI are gated off there to avoid a dead action.
/// The MAC-learning path stays active on every platform, so flipping this on once the
/// entitlement lands makes wake work immediately. ON APPROVAL: change `#if os(macOS)` below to
/// `true` for iOS/tvOS too (and uncomment the entitlement).
static var wakeOnLANAvailable: Bool {
#if os(macOS)
return true
#else
return false
#endif
}
/// `com.apple.developer.networking.multicast` entitlement now approved and enabled (see
/// `Config/Punktfunk.entitlements`), so wake is available on every platform. Kept as the single
/// switch every call site gates on, should a future build ever need to disable it.
static var wakeOnLANAvailable: Bool { true }
/// Send a Wake-on-LAN magic packet to wake a sleeping host. `macs` are the host's NIC MAC(s)
/// (`aa:bb:cc:dd:ee:ff`, learned from its mDNS `mac` TXT while awake); malformed entries are
@@ -445,6 +436,20 @@ public final class PunktfunkConnection {
_ = punktfunk_connection_request_keyframe(h)
}
/// Feed each received AU's `frameIndex` (in receive order) so the client recovers from loss with a
/// cheap reference-frame invalidation instead of always paying for a full IDR. On a forward gap
/// a `frameIndex` jump means the intervening frames were lost and the following AUs reference a
/// picture that never arrived the core fires a THROTTLED RFI request for the lost range, and an
/// RFI-capable host (AMD LTR / NVENC) recovers with a clean P-frame rather than a 20-40× IDR
/// spike. Call it for every received AU; the `framesDropped`-driven `requestKeyframe()` path stays
/// the backstop for when the recovery frame itself is lost. Cheap; silently dropped after close.
public func noteFrameIndex(_ frameIndex: UInt32) {
abiLock.lock()
defer { abiLock.unlock() }
guard let h = handle, !closeRequested else { return }
_ = punktfunk_connection_note_frame_index(h, frameIndex, nil)
}
/// Cumulative access units the hostclient reassembler dropped as unrecoverable (FEC couldn't
/// rebuild them). The video pump polls this and calls `requestKeyframe()` when it climbs the
/// correct loss trigger under the host's infinite GOP, where unrecoverable loss yields
@@ -11,6 +11,15 @@ public enum DefaultsKey {
public static let streamWidth = "punktfunk.width"
public static let streamHeight = "punktfunk.height"
public static let streamHz = "punktfunk.hz"
/// Match-window resolution policy (design/midstream-resolution-resize.md D1/D2): when on, the
/// stream mode FOLLOWS the session view the connect asks for the view's pixel size and a
/// mid-session resize (a windowed macOS window, an iPad Stage Manager / Split View scene)
/// renegotiates the host's virtual display + encoder (`PunktfunkConnection.requestMode`), so a
/// windowed session streams native-resolution pixels instead of scaling. Off (default): the
/// explicit `streamWidth`/`streamHeight` are used and never auto-resized (a fullscreen session
/// is native either way, so this degenerates to Auto-native there). Read per session by the
/// stream views' `MatchWindowFollower`.
public static let matchWindow = "punktfunk.matchWindow"
public static let compositor = "punktfunk.compositor"
public static let gamepadType = "punktfunk.gamepadType"
public static let gamepadID = "punktfunk.gamepadID"
@@ -88,6 +97,12 @@ public enum DefaultsKey {
/// layout (the console launcher, gamepad-navigable settings, a coverflow-style library)
/// whenever a gamepad is connected. On by default; see `GamepadUIEnvironment.isActive`.
public static let gamepadUIEnabled = "punktfunk.gamepadUIEnabled"
/// Auto-wake on connect: when connecting to a saved host that isn't advertising on mDNS, fire
/// Wake-on-LAN and, if the dial fails, wait for it to come back before retrying (the "Waking"
/// overlay). On by default. Turn off if a host that's already on just isn't seen on mDNS (a
/// routed/VPN host), so connects go straight through instead of waiting out the wake timeout.
/// The explicit "Wake Host" action stays available regardless. Read by ContentView.startSession.
public static let autoWake = "punktfunk.autoWake"
}
extension Notification.Name {
@@ -0,0 +1,153 @@
// Match-window resize follower (design/midstream-resolution-resize.md D1/D2, client C3).
//
// The presenting view feeds this its PHYSICAL-PIXEL size on every layout; it debounces to
// resize-end, spaces requests 1 s apart, and asks the connection to switch the host's virtual
// display + encoder to match (`PunktfunkConnection.requestMode`) so a windowed macOS session or
// an iPad Stage Manager / Split View scene streams native-resolution pixels instead of scaling.
// The decode/present side needs nothing: VideoToolbox recreates its session on the keyframe-derived
// format-description change (the first new-mode AU is an IDR with fresh parameter sets).
//
// The trigger discipline is the shared cross-client one (mirrors the session binary's
// `resize_decision`): physical pixels rounded DOWN to even (the host rejects odd dimensions) and
// clamped 320×200; debounce to resize-end; 1 s between requests; skip a size equal to the live
// mode; and request each distinct size at most once which both stops re-asking a rejected size
// and keeps a host-side rollback (accepted, rebuild failed, corrective ack restored the old mode)
// from looping request rollback request.
import Foundation
/// The pure, side-effect-free core of the Match-window trigger so the normalize/skip discipline
/// is unit-tested without a live connection or a UI (`MatchWindowTests`).
public enum MatchWindow {
/// Even-floor + clamp a physical-pixel size to a host-valid mode dimension: the host's
/// `validate_dimensions` rejects odd sizes, and we never ask below 320×200.
public static func normalize(widthPx: Int, heightPx: Int) -> (width: UInt32, height: UInt32) {
let evenClamp: (Int, UInt32) -> UInt32 = { px, minimum in
let even = UInt32(max(px, 0)) / 2 * 2
return max(even, minimum)
}
return (evenClamp(widthPx, 320), evenClamp(heightPx, 200))
}
/// Whether to request `target` now (the debounce has already settled; spacing is the caller's
/// timer): `nil` to skip equal to the live mode, or already requested once (a rejected size /
/// a host rollback must not loop). `target` is expected already-[normalize]d.
public static func request(
target: (width: UInt32, height: UInt32),
current: (width: UInt32, height: UInt32),
lastRequested: (width: UInt32, height: UInt32)?
) -> (width: UInt32, height: UInt32)? {
if target.width == current.width, target.height == current.height { return nil }
if let lr = lastRequested, lr.width == target.width, lr.height == target.height { return nil }
return target
}
}
/// Owns the debounce timer + serialization state and drives `PunktfunkConnection.requestMode` from
/// the stream view's layout callbacks. Main-actor: the views feed it on the main thread and it reads
/// the connection's live mode there. Enabled per session from the `matchWindow` setting.
@MainActor
public final class MatchWindowFollower {
private weak var connection: PunktfunkConnection?
private let debounce: TimeInterval
private let minSpacing: TimeInterval
private var enabled: Bool
private var work: DispatchWorkItem?
private var pendingSize: (width: Int, height: Int)?
private var lastRequested: (width: UInt32, height: UInt32)?
private var lastRequestAt: Date?
/// The last size we reported via [`onResizeTarget`] dedups the per-layout stream of a drag so
/// the UI is notified once per distinct target, and reset to `nil` when the window is back in
/// sync with the live mode (so a later resize re-reports).
private var lastSteered: (width: UInt32, height: UInt32)?
/// Fired (on the main actor) the instant the window starts differing from the live mode i.e.
/// a resize is under way and a `Reconfigure` for `(width, height)` is imminent. Drives the
/// resize overlay's INSTANT feedback (blur + spinner) BEFORE the debounced request leaves; the
/// overlay clears when a decoded frame reaches this size (or on a timeout). Deduped per target.
public var onResizeTarget: ((_ width: UInt32, _ height: UInt32) -> Void)?
/// `debounce` = quiet time after the last size event before requesting (Win32 gets
/// `WM_EXITSIZEMOVE` for free; we debounce). `minSpacing` = floor between accepted requests
/// (a full host pipeline rebuild each). Defaults match the other clients.
public init(
connection: PunktfunkConnection,
enabled: Bool,
debounce: TimeInterval = 0.4,
minSpacing: TimeInterval = 1.0
) {
self.connection = connection
self.enabled = enabled
self.debounce = debounce
self.minSpacing = minSpacing
}
/// Turn following on/off live (a mid-session settings change; off cancels a pending request).
public func setEnabled(_ on: Bool) {
enabled = on
if !on {
work?.cancel()
work = nil
pendingSize = nil
lastSteered = nil
}
}
/// Feed the presenting view's current PHYSICAL-PIXEL size (its `bounds` × the backing/display
/// scale). Called from every layout pass; coalesced by the debounce so a drag-resize sends one
/// request at its end, never one per frame.
public func noteSize(widthPx: Int, heightPx: Int) {
guard enabled else { return }
pendingSize = (widthPx, heightPx)
schedule()
reportSteering(widthPx: widthPx, heightPx: heightPx)
}
/// Report the resize overlay's START signal (deduped): the moment the normalized window size
/// differs from the live mode we're steering toward a new size. No connection / no negotiated
/// mode yet nothing to compare against, skip.
private func reportSteering(widthPx: Int, heightPx: Int) {
guard let connection else { return }
let target = MatchWindow.normalize(widthPx: widthPx, heightPx: heightPx)
let mode = connection.currentMode()
guard mode.width > 0, mode.height > 0 else { return }
if target.width == mode.width, target.height == mode.height {
lastSteered = nil // back in sync a later change re-reports
return
}
if lastSteered?.width == target.width, lastSteered?.height == target.height { return }
lastSteered = target
onResizeTarget?(target.width, target.height)
}
private func schedule() {
work?.cancel()
let item = DispatchWorkItem { [weak self] in self?.fire() }
work = item
DispatchQueue.main.asyncAfter(deadline: .now() + debounce, execute: item)
}
private func fire() {
guard enabled, let connection, let size = pendingSize else { return }
// 1 s spacing: a request went out recently re-arm the debounce and retry later rather
// than fire early (keeps at most ~one request outstanding the accept ack round-trips in
// milliseconds, ahead of the host's rebuild).
if let last = lastRequestAt, Date().timeIntervalSince(last) < minSpacing {
schedule()
return
}
let target = MatchWindow.normalize(widthPx: size.width, heightPx: size.height)
let mode = connection.currentMode()
pendingSize = nil
guard let req = MatchWindow.request(
target: target,
current: (mode.width, mode.height),
lastRequested: lastRequested
) else { return }
// Keep the current refresh Match-window follows SIZE, not rate.
connection.requestMode(width: req.width, height: req.height, refreshHz: mode.refreshHz)
lastRequested = req
lastRequestAt = Date()
}
}
@@ -0,0 +1,63 @@
// Resize-in-progress indicator state (design/midstream-resolution-resize.md client UX).
//
// A mid-stream resize takes the host 0.32 s to rebuild its virtual display + encoder, and the
// first new-mode frame is an IDR that the decoder re-inits on. Rather than let the stream scale
// (stretch/blur) to the changing window during that gap, the client EMBRACES the delay: it shows a
// deliberate blur + spinner the instant a resize starts and clears it the instant the sharp
// new-resolution frame is on screen so the wait reads as intentional, not as lag.
//
// This is driven ENTIRELY by signals the client already has (no new protocol):
// * START the Match-window follower reports the size it is steering toward (instant, on the
// first resize layout, before the debounced request even leaves).
// * END the decode pipeline reports each new-mode IDR's dimensions; when they reach the target
// the new picture is here.
// * TIMEOUT the safety net for a switch that never delivers the exact target: the host rejected
// it (gamescope), capped it to an advertised mode, or a corrective ack landed a different size.
//
// Pure + side-effect-free so the transition logic is unit-tested without a live session or UI
// (`ResizeIndicatorTests`); `SessionModel` owns an instance and mirrors `active` into a @Published.
import Foundation
/// The pure state of the resize overlay. `now` is a monotonic time in seconds (the caller passes
/// `ProcessInfo.processInfo.systemUptime` or a test clock).
public struct ResizeIndicator {
/// Whether the blur + spinner should be shown.
public private(set) var active = false
/// The size the follower is steering toward cleared once a decoded frame reaches it.
private var target: (width: UInt32, height: UInt32)?
/// When the current `active` span began the timeout is measured from here.
private var since: TimeInterval?
/// How long to keep the overlay up if the target frame never arrives (rejected / capped switch).
public var timeout: TimeInterval
public init(timeout: TimeInterval = 2.5) { self.timeout = timeout }
/// The follower is steering toward `width`×`height` a resize is under way. Show the overlay now
/// (instant feedback). Called only for a genuine change (the follower skips a target equal to the
/// live mode), possibly many times as a drag moves through sizes; the timeout re-arms whenever the
/// target actually changes so a slow drag never trips it mid-gesture.
public mutating func steering(width: UInt32, height: UInt32, now: TimeInterval) {
if !active || target?.width != width || target?.height != height {
since = now
}
target = (width, height)
active = true
}
/// A decoded frame arrived at `width`×`height` (a new-mode IDR). Clears the overlay once it
/// matches the steered target the sharp new-resolution picture is on glass.
public mutating func decoded(width: UInt32, height: UInt32) {
guard active, let t = target, t.width == width, t.height == height else { return }
active = false
since = nil
}
/// Timeout safety net: stop showing the overlay once `timeout` has elapsed with no matching frame
/// (a rejected or host-capped switch never delivers the exact target).
public mutating func tick(now: TimeInterval) {
guard active, let s = since, now - s >= timeout else { return }
active = false
since = nil
}
}
@@ -70,6 +70,15 @@ final class SessionPresenter {
private var stage2Link: CADisplayLink?
private var metalLayer: CAMetalLayer?
private var connection: PunktfunkConnection?
/// The decoded frame's REAL pixel dimensions (ground truth, pushed by the view from the pump's
/// `onDecodedSize` new-mode-IDR callback). Used for the aspect-fit in `layout` in preference to
/// `connection.currentMode()`, which (a) lags a mid-stream resize it only updates on the
/// `Reconfigured` ack, and a resize-END produces no bounds change to re-run `layout` afterward
/// and (b) can disagree with what the host actually DELIVERED (Windows corrective-ack falls back
/// to an advertised mode). The pixels we're drawing are the only correct aspect source; a wrong
/// one here is the "black bars + stretched" resize artifact. nil until the first frame `layout`
/// falls back to `currentMode()`. Main-thread only.
private var contentSize: CGSize?
/// Start the presenter for `connection`. `baseLayer` is the view's AVSampleBufferDisplayLayer:
/// stage-1 enqueues into it; stage-2 leaves it idle and composites an opaque CAMetalLayer
@@ -85,7 +94,8 @@ final class SessionPresenter {
displayMeter: LatencyMeter? = nil,
makeDisplayLink: (AnyObject, Selector) -> CADisplayLink,
onFrame: (@Sendable (AccessUnit) -> Void)?,
onSessionEnd: (@Sendable () -> Void)?
onSessionEnd: (@Sendable () -> Void)?,
onDecodedSize: (@Sendable (Int, Int) -> Void)? = nil
) {
stop()
self.connection = connection
@@ -128,12 +138,14 @@ final class SessionPresenter {
link.add(to: .main, forMode: .common)
stage2Link = link
syncFrameRate(hz: connection.currentMode().refreshHz)
pipeline.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
pipeline.start(
connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd,
onDecodedSize: onDecodedSize)
} else {
let pump = StreamPump()
pump.start(
connection: connection, layer: baseLayer,
onFrame: onFrame, onSessionEnd: onSessionEnd)
onFrame: onFrame, onSessionEnd: onSessionEnd, onDecodedSize: onDecodedSize)
self.pump = pump
}
}
@@ -181,11 +193,18 @@ final class SessionPresenter {
guard let metalLayer, let connection else { return }
let mode = connection.currentMode()
syncFrameRate(hz: mode.refreshHz) // track a mid-session Reconfigure's new refresh
let fit: CGRect = (mode.width > 0 && mode.height > 0)
? AVMakeRect(
aspectRatio: CGSize(width: Int(mode.width), height: Int(mode.height)),
insideRect: bounds)
: bounds
// Aspect source: the ACTUAL decoded dims when known (survives a lagging `currentMode()` and a
// host that delivered a different size than requested), else the negotiated mode. The shader
// stretches the frame across the WHOLE drawable, so this rect's aspect is the only thing that
// keeps the picture undistorted a stale aspect here is the post-resize black-bars+stretch.
let aspect: CGSize? = {
if let c = contentSize, c.width > 0, c.height > 0 { return c }
if mode.width > 0, mode.height > 0 {
return CGSize(width: Int(mode.width), height: Int(mode.height))
}
return nil
}()
let fit: CGRect = aspect.map { AVMakeRect(aspectRatio: $0, insideRect: bounds) } ?? bounds
// No implicit resize animation; contentsScale tracks the view's backing/display scale.
CATransaction.begin()
CATransaction.setDisableActions(true)
@@ -206,10 +225,20 @@ final class SessionPresenter {
#endif
}
/// Record the decoded frame's real dimensions (the view hops the pump's `onDecodedSize` to main
/// and calls this) so `layout` aspect-fits to what's actually on screen instead of the possibly-
/// stale `currentMode()`. Only stores the caller re-runs `layout` right after, because a
/// resize-END produces no bounds change to trigger one. No-op on a zero/unchanged size.
func setContentSize(_ size: CGSize) {
guard size.width > 0, size.height > 0, size != contentSize else { return }
contentSize = size
}
/// Stop the active pump/pipeline ( one poll timeout; stage-2 joins its pump) and detach the
/// stage-2 layer + link. Does not close the connection that stays with whoever owns it.
/// Idempotent.
func stop() {
contentSize = nil // a new session re-derives it from its first frame
pump?.stop()
pump = nil
stage2Link?.invalidate()
@@ -329,7 +329,8 @@ public final class Stage2Pipeline {
public func start(
connection: PunktfunkConnection,
onFrame: (@Sendable (AccessUnit) -> Void)?,
onSessionEnd: (@Sendable () -> Void)?
onSessionEnd: (@Sendable () -> Void)?,
onDecodedSize: (@Sendable (Int, Int) -> Void)? = nil
) {
offsetNs = connection.clockOffsetNs
recovery.bind(connection) // arm host-keyframe recovery for this session
@@ -350,6 +351,9 @@ public final class Stage2Pipeline {
let thread = Thread {
defer { pumpStopped.signal() } // let stop() join the pump (bounded) before decoder.reset()
var format: CMVideoFormatDescription?
// Report coded dims to the resize overlay only on a CHANGE (new-mode IDR), not per
// loss-recovery IDR at the same size (see StreamPump).
var lastDecodedDims: CMVideoDimensions?
var lastFramesDropped = connection.framesDropped()
// Persistent recovery WANT, not a one-shot edge (see StreamPump for the full rationale):
// keep asking until an IDR lands so a request swallowed by the throttle is re-sent.
@@ -384,9 +388,19 @@ public final class Stage2Pipeline {
presenter.setHdrMeta(meta)
}
guard let au = try connection.nextAU(timeoutMs: 100) else { return true }
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference-
// frame-invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers
// with a cheap clean P-frame instead of a full IDR. The framesDropped-driven
// recovery below stays the backstop for when the recovery frame itself is lost.
connection.noteFrameIndex(au.frameIndex)
onFrame?(au)
if let f = connection.videoCodec.formatDescription(fromKeyframe: au.data) {
format = f // refreshed on every IDR (mode changes included)
let dims = CMVideoFormatDescriptionGetDimensions(f)
if lastDecodedDims?.width != dims.width || lastDecodedDims?.height != dims.height {
lastDecodedDims = dims
onDecodedSize?(Int(dims.width), Int(dims.height))
}
awaitingIDR = false // a fresh IDR re-anchored decode recovery complete
}
guard let f = format, !token.isStopped else { return true }
@@ -21,7 +21,8 @@ final class StreamPump {
connection: PunktfunkConnection,
layer: AVSampleBufferDisplayLayer,
onFrame: (@Sendable (AccessUnit) -> Void)?,
onSessionEnd: (@Sendable () -> Void)?
onSessionEnd: (@Sendable () -> Void)?,
onDecodedSize: (@Sendable (Int, Int) -> Void)? = nil
) {
let token = token
// Coalesced host keyframe requests (100 ms throttle see KeyframeRecovery).
@@ -35,6 +36,9 @@ final class StreamPump {
let thread = Thread {
var format: CMVideoFormatDescription?
// Report the coded dims to the resize overlay only when they CHANGE (a new-mode IDR),
// not on every loss-recovery IDR at the same size so it fires once per real switch.
var lastDecodedDims: CMVideoDimensions?
var lastFramesDropped = connection.framesDropped()
// Recovery is a persistent WANT, not a one-shot edge: set it on detected loss (or a
// decoder reset), retry the throttled request EVERY iteration, and clear it only when a
@@ -75,10 +79,20 @@ final class StreamPump {
if awaitingIDR { recovery.request() }
guard let au = try connection.nextAU(timeoutMs: 100) else { return true }
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference-
// frame-invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers
// with a cheap clean P-frame instead of a full IDR. The framesDropped-driven
// recovery above stays the backstop for when the recovery frame itself is lost.
connection.noteFrameIndex(au.frameIndex)
onFrame?(au)
let idrFormat = connection.videoCodec.formatDescription(fromKeyframe: au.data)
if let f = idrFormat {
format = f // refreshed on every IDR (mode changes included)
let dims = CMVideoFormatDescriptionGetDimensions(f)
if lastDecodedDims?.width != dims.width || lastDecodedDims?.height != dims.height {
lastDecodedDims = dims
onDecodedSize?(Int(dims.width), Int(dims.height))
}
if awaitingIDR {
let ms = Int(Date().timeIntervalSince(awaitingSince) * 1000)
pumpLog.notice("video: recovery IDR received — resumed after \(ms, privacy: .public) ms")
@@ -87,6 +87,8 @@ public struct StreamView: NSViewRepresentable {
private let onDisconnectRequest: (() -> Void)?
private let onFrame: (@Sendable (AccessUnit) -> Void)?
private let onSessionEnd: (@Sendable () -> Void)?
private let onResizeTarget: ((UInt32, UInt32) -> Void)?
private let onDecodedSize: (@Sendable (Int, Int) -> Void)?
private let endToEndMeter: LatencyMeter?
private let decodeMeter: LatencyMeter?
private let displayMeter: LatencyMeter?
@@ -108,6 +110,8 @@ public struct StreamView: NSViewRepresentable {
onDisconnectRequest: (() -> Void)? = nil,
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
onSessionEnd: (@Sendable () -> Void)? = nil,
onResizeTarget: ((UInt32, UInt32) -> Void)? = nil,
onDecodedSize: (@Sendable (Int, Int) -> Void)? = nil,
endToEndMeter: LatencyMeter? = nil,
decodeMeter: LatencyMeter? = nil,
displayMeter: LatencyMeter? = nil
@@ -118,6 +122,8 @@ public struct StreamView: NSViewRepresentable {
self.onDisconnectRequest = onDisconnectRequest
self.onFrame = onFrame
self.onSessionEnd = onSessionEnd
self.onResizeTarget = onResizeTarget
self.onDecodedSize = onDecodedSize
self.endToEndMeter = endToEndMeter
self.decodeMeter = decodeMeter
self.displayMeter = displayMeter
@@ -131,6 +137,8 @@ public struct StreamView: NSViewRepresentable {
view.endToEndMeter = endToEndMeter
view.decodeMeter = decodeMeter
view.displayMeter = displayMeter
view.onResizeTarget = onResizeTarget
view.onDecodedSize = onDecodedSize
view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
return view
}
@@ -142,6 +150,8 @@ public struct StreamView: NSViewRepresentable {
view.endToEndMeter = endToEndMeter
view.decodeMeter = decodeMeter
view.displayMeter = displayMeter
view.onResizeTarget = onResizeTarget
view.onDecodedSize = onDecodedSize
// SwiftUI reuses the NSView across state changes repoint the pump only when the
// connection identity actually changed.
if view.connection !== connection {
@@ -165,6 +175,15 @@ public final class StreamLayerView: NSView {
/// stage-1 StreamPump displayLayer path as the Metal-unavailable / DEBUG fallback.
private let presenter = SessionPresenter()
public private(set) var connection: PunktfunkConnection?
/// Match-window resize follower (C3) non-nil while a session is active AND the `matchWindow`
/// setting is on (DEFAULT on, for pixel-exact windowed streaming); fed the view's physical-pixel
/// size on every relayout so the host mode tracks the window (1:1, no presenter resample).
private var matchFollower: MatchWindowFollower?
/// Last decoded frame size fed into the presenter's aspect-fit. A new-mode IDR after a resize
/// re-fits the metal sublayer to the REAL content aspect here `layout()` only re-runs on a
/// bounds change and a resize-END has none, so without this the layer keeps its pre-resize aspect
/// and the shader stretches the new frame into it (black bars + squish). Main-thread only.
private var lastDecodedContentSize: CGSize?
private let cursorCapture = CursorCapture()
private var inputCapture: InputCapture?
private var appObservers: [NSObjectProtocol] = []
@@ -201,6 +220,13 @@ public final class StreamLayerView: NSView {
/// view can't do that itself (the connection's owner disconnects).
public var onDisconnectRequest: (() -> Void)?
/// Resize overlay signals (design/midstream-resolution-resize.md client UX): `onResizeTarget`
/// (main thread, via the follower) fires the instant the window starts steering toward a new
/// size; `onDecodedSize` (PUMP thread) fires when a new-mode IDR's dims land. The owner drives
/// the blur+spinner from these set before `start()`.
public var onResizeTarget: ((UInt32, UInt32) -> Void)?
public var onDecodedSize: (@Sendable (Int, Int) -> Void)?
/// Main-thread only. False = input capture disabled outright (UI layered over the
/// stream); flipping to true auto-engages once.
public var captureEnabled = true {
@@ -618,6 +644,10 @@ public final class StreamLayerView: NSView {
// (explicit VTDecompressionSession decode + a CAMetalLayer/display-link present) by
// default, the stage-1 pump as the Metal-missing / DEBUG fallback. The link comes from
// NSView.displayLink so it tracks the display this view is on.
// Intercept the pump's coded-dims callback: re-fit the metal sublayer to the real content
// aspect (main thread) BEFORE forwarding to the owner's overlay END-signal. Fires only on a
// size CHANGE (first frame + each resolved resize), so this is rare, not per-frame.
let overlayDecodedSize = onDecodedSize
presenter.start(
connection: connection,
baseLayer: displayLayer,
@@ -626,15 +656,38 @@ public final class StreamLayerView: NSView {
displayMeter: displayMeter,
makeDisplayLink: { displayLink(target: $0, selector: $1) },
onFrame: onFrame,
onSessionEnd: onSessionEnd)
onSessionEnd: onSessionEnd,
onDecodedSize: { [weak self] w, h in // resize overlay END signal (new-mode IDR dims)
DispatchQueue.main.async { self?.noteDecodedContentSize(width: w, height: h) }
overlayDecodedSize?(w, h)
})
// Match-window (C3): follow the window's pixel size DEFAULT ON, so a windowed session
// streams 1:1 (pixel-exact) instead of the presenter resampling a fixed-mode frame into a
// non-matching window. The first real `layout()` feeds the initial size, so the stream
// converges to the window even though the connect used the explicit/display mode; entering
// fullscreen reports the full-display px, restoring a native-res 1:1 present there too.
// `?? true` so an unset default matches the Settings toggle (which also defaults on).
let follower = MatchWindowFollower(
connection: connection,
enabled: UserDefaults.standard.object(forKey: DefaultsKey.matchWindow) as? Bool ?? true)
follower.onResizeTarget = onResizeTarget // resize overlay START signal (instant, on the follower)
matchFollower = follower
layoutPresenter()
requestAutoCapture() // entering a session is the deliberate "capture me" moment
}
/// Aspect-fit the stage-2 metal sublayer to the view; refresh contentsScale on a
/// retinanon-retina move (see SessionPresenter.layout).
/// retinanon-retina move (see SessionPresenter.layout). Also feeds the Match-window follower
/// the view's physical-pixel size (bounds backing), so a window resize / retina move follows.
private func layoutPresenter() {
presenter.layout(in: bounds, contentsScale: window?.backingScaleFactor ?? 1)
// Feed the follower only once in a window (backing scale is real then) and with real
// bounds a pre-window layout would report point-sized dimensions.
if window != nil, bounds.width > 0, bounds.height > 0 {
let px = convertToBacking(bounds).size
matchFollower?.noteSize(
widthPx: Int(px.width.rounded()), heightPx: Int(px.height.rounded()))
}
}
public override func viewDidChangeBackingProperties() {
@@ -642,6 +695,18 @@ public final class StreamLayerView: NSView {
layoutPresenter() // backing scale changed (e.g. moved to a non-retina display)
}
/// A new decoded size landed (a new-mode IDR after a resize, or the session's first frame): push
/// it to the presenter's aspect-fit and re-layout NOW. A resize-END triggers no `layout()`, so
/// this is what makes the metal sublayer track the new content aspect instead of stretching the
/// new frame into the pre-resize box. Deduped so a same-size repeat is a no-op. Main thread.
private func noteDecodedContentSize(width: Int, height: Int) {
let size = CGSize(width: width, height: height)
guard size.width > 0, size.height > 0, size != lastDecodedContentSize else { return }
lastDecodedContentSize = size
presenter.setContentSize(size)
layoutPresenter()
}
/// Stop pumping ( one poll timeout). Does not close the connection that stays with
/// whoever owns it (PunktfunkConnection.close() is safe alongside a draining pump).
public func stop() {
@@ -650,6 +715,8 @@ public final class StreamLayerView: NSView {
inputCapture?.stop()
inputCapture = nil
presenter.stop()
matchFollower = nil
lastDecodedContentSize = nil // the next session re-derives it from its first frame
connection = nil
}
@@ -53,6 +53,8 @@ public struct StreamView: UIViewControllerRepresentable {
private let onCaptureChange: ((Bool) -> Void)?
private let onFrame: (@Sendable (AccessUnit) -> Void)?
private let onSessionEnd: (@Sendable () -> Void)?
private let onResizeTarget: ((UInt32, UInt32) -> Void)?
private let onDecodedSize: (@Sendable (Int, Int) -> Void)?
private let endToEndMeter: LatencyMeter?
private let decodeMeter: LatencyMeter?
private let displayMeter: LatencyMeter?
@@ -68,6 +70,8 @@ public struct StreamView: UIViewControllerRepresentable {
onDisconnectRequest: (() -> Void)? = nil,
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
onSessionEnd: (@Sendable () -> Void)? = nil,
onResizeTarget: ((UInt32, UInt32) -> Void)? = nil,
onDecodedSize: (@Sendable (Int, Int) -> Void)? = nil,
endToEndMeter: LatencyMeter? = nil,
decodeMeter: LatencyMeter? = nil,
displayMeter: LatencyMeter? = nil
@@ -77,6 +81,8 @@ public struct StreamView: UIViewControllerRepresentable {
self.onCaptureChange = onCaptureChange
self.onFrame = onFrame
self.onSessionEnd = onSessionEnd
self.onResizeTarget = onResizeTarget
self.onDecodedSize = onDecodedSize
self.endToEndMeter = endToEndMeter
self.decodeMeter = decodeMeter
self.displayMeter = displayMeter
@@ -89,6 +95,8 @@ public struct StreamView: UIViewControllerRepresentable {
controller.endToEndMeter = endToEndMeter
controller.decodeMeter = decodeMeter
controller.displayMeter = displayMeter
controller.onResizeTarget = onResizeTarget
controller.onDecodedSize = onDecodedSize
controller.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
return controller
}
@@ -99,6 +107,8 @@ public struct StreamView: UIViewControllerRepresentable {
controller.endToEndMeter = endToEndMeter
controller.decodeMeter = decodeMeter
controller.displayMeter = displayMeter
controller.onResizeTarget = onResizeTarget
controller.onDecodedSize = onDecodedSize
if controller.connection !== connection {
controller.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
}
@@ -147,6 +157,12 @@ public final class StreamViewController: StreamViewControllerBase {
/// Capture state at the last resign, restored on the next foreground otherwise the
/// mouse/keyboard stay released after navigating out and nothing re-grabs them.
private var wasCapturedOnResign = false
/// Match-window resize follower (C3) non-nil while a session is active AND the `matchWindow`
/// setting is on (DEFAULT on, for pixel-exact scene streaming); fed the view's physical-pixel
/// size from `viewDidLayoutSubviews` so an iPad Stage Manager / Split View scene resize
/// renegotiates the host mode (1:1, no presenter resample). iOS only (iPhone naturally no-ops
/// its fixed full-screen scene; tvOS drives display modes via AVDisplayManager instead).
private var matchFollower: MatchWindowFollower?
#endif
/// Reads whether the scene's pointer is actually locked right now; nil = state
@@ -161,6 +177,18 @@ public final class StreamViewController: StreamViewControllerBase {
}
var onCaptureChange: ((Bool) -> Void)?
/// Resize-overlay START: forwarded to the Match-window follower so a scene resize drives the
/// blur+spinner the instant the window differs from the live mode (iOS only tvOS has no
/// follower). See `MatchWindowFollower.onResizeTarget`.
var onResizeTarget: ((UInt32, UInt32) -> Void)?
/// Resize-overlay END: the presenter reports the coded dims of each new-mode IDR here, so the
/// overlay clears when a frame at the requested size actually decodes.
var onDecodedSize: (@Sendable (Int, Int) -> Void)?
/// Last decoded size fed into the presenter's aspect-fit. A new-mode IDR (an iPad scene resize,
/// or a tvOS AVDisplayManager mode switch) re-fits the metal sublayer to the REAL content aspect
/// here `viewDidLayoutSubviews` only re-runs on a bounds change, which a resize-END lacks, so
/// without this the layer keeps its pre-resize aspect and stretches the new frame into it. Main.
private var lastDecodedContentSize: CGSize?
var captureEnabled = true {
didSet {
@@ -327,11 +355,25 @@ public final class StreamViewController: StreamViewControllerBase {
}
capture.start()
inputCapture = capture
// Match-window (C3): follow the scene's pixel size DEFAULT ON, so a resizable iPad scene
// streams 1:1 (pixel-exact) instead of the presenter resampling a fixed-mode frame into it.
// `viewDidLayoutSubviews` feeds it covers Stage Manager / Split View resizes and rotation.
// iPhone is a fixed full-screen scene, so this naturally no-ops (reports the device mode).
// `?? true` so an unset default matches the Settings toggle (which also defaults on).
let follower = MatchWindowFollower(
connection: connection,
enabled: UserDefaults.standard.object(forKey: DefaultsKey.matchWindow) as? Bool ?? true)
follower.onResizeTarget = onResizeTarget
matchFollower = follower
#endif
// Presenter choice + lifecycle live in SessionPresenter (shared with macOS): stage-2
// (explicit VTDecompressionSession decode + a CAMetalLayer/display-link present) by
// default, the stage-1 pump as the Metal-missing / DEBUG fallback.
// Intercept the pump's coded-dims callback: re-fit the metal sublayer to the real content
// aspect (main thread) BEFORE forwarding to the owner's overlay END-signal. Fires only on a
// size CHANGE (first frame + each resolved resize), so this is rare, not per-frame.
let overlayDecodedSize = onDecodedSize
presenter.start(
connection: connection,
baseLayer: streamView.displayLayer,
@@ -340,7 +382,11 @@ public final class StreamViewController: StreamViewControllerBase {
displayMeter: displayMeter,
makeDisplayLink: { CADisplayLink(target: $0, selector: $1) },
onFrame: onFrame,
onSessionEnd: onSessionEnd)
onSessionEnd: onSessionEnd,
onDecodedSize: { [weak self] w, h in
DispatchQueue.main.async { self?.noteDecodedContentSize(width: w, height: h) }
overlayDecodedSize?(w, h)
})
layoutMetalLayer()
#if os(iOS)
@@ -411,6 +457,7 @@ public final class StreamViewController: StreamViewControllerBase {
streamView.onPointerButton = nil
streamView.onScroll = nil
streamView.currentHostMode = nil
matchFollower = nil
#endif
#if os(tvOS)
// Return the TV to the user's preferred mode the home screen must not stay in the
@@ -419,12 +466,23 @@ public final class StreamViewController: StreamViewControllerBase {
sessionDisplayManager = nil
#endif
presenter.stop()
lastDecodedContentSize = nil // the next session re-derives it from its first frame
connection = nil
}
public override func viewDidLayoutSubviews() {
super.viewDidLayoutSubviews()
layoutMetalLayer()
#if os(iOS)
// Match-window (C3): feed the follower the view's physical-pixel size (points × scale).
let b = streamView.bounds
if b.width > 0, b.height > 0 {
let scale = renderScale
matchFollower?.noteSize(
widthPx: Int((b.width * scale).rounded()),
heightPx: Int((b.height * scale).rounded()))
}
#endif
#if os(tvOS)
applyDisplayCriteriaIfNeeded()
#endif
@@ -485,6 +543,18 @@ public final class StreamViewController: StreamViewControllerBase {
presenter.layout(in: streamView.bounds, contentsScale: renderScale)
}
/// A new decoded size landed (a scene/mode resize's new IDR, or the first frame): push it to the
/// presenter's aspect-fit and re-layout NOW. A resize-END triggers no `viewDidLayoutSubviews`, so
/// this is what makes the metal sublayer track the new content aspect instead of stretching the
/// new frame into the pre-resize box. Deduped so a same-size repeat is a no-op. Main thread.
private func noteDecodedContentSize(width: Int, height: Int) {
let size = CGSize(width: width, height: height)
guard size.width > 0, size.height > 0, size != lastDecodedContentSize else { return }
lastDecodedContentSize = size
presenter.setContentSize(size)
layoutMetalLayer()
}
#if os(iOS)
private func setCaptured(_ on: Bool) {
if on {
@@ -0,0 +1,43 @@
// The Match-window trigger discipline (design/midstream-resolution-resize.md D2), as pure
// functions the same rules the session binary's `resize_decision` unit-tests: physical pixels
// even-floored and clamped 320×200, skip a size equal to the live mode, and request each
// distinct size at most once (so a rejected size / a host rollback can't loop).
import XCTest
@testable import PunktfunkKit
final class MatchWindowTests: XCTestCase {
func testNormalizeEvenFloorsAndClamps() {
// Odd pixels floor to even (the host rejects odd dimensions).
let a = MatchWindow.normalize(widthPx: 1001, heightPx: 601)
XCTAssertEqual(a.width, 1000)
XCTAssertEqual(a.height, 600)
// Already-even sizes pass through.
let b = MatchWindow.normalize(widthPx: 2560, heightPx: 1440)
XCTAssertEqual(b.width, 2560)
XCTAssertEqual(b.height, 1440)
// Tiny / zero clamp to the host floor.
let c = MatchWindow.normalize(widthPx: 100, heightPx: 80)
XCTAssertEqual(c.width, 320)
XCTAssertEqual(c.height, 200)
let z = MatchWindow.normalize(widthPx: 0, heightPx: -4)
XCTAssertEqual(z.width, 320)
XCTAssertEqual(z.height, 200)
}
func testRequestSkipsEqualAndAlreadyRequested() {
// A new size (different from the live mode, not yet requested) request it.
let r = MatchWindow.request(
target: (1000, 600), current: (1280, 720), lastRequested: (800, 500))
XCTAssertEqual(r?.width, 1000)
XCTAssertEqual(r?.height, 600)
// Equal to the live mode nothing to do.
XCTAssertNil(MatchWindow.request(
target: (1280, 720), current: (1280, 720), lastRequested: nil))
// Already requested once don't re-ask (covers a rejected size AND a host rollback:
// accepted rebuild failed corrective ack restored the old mode must not loop).
XCTAssertNil(MatchWindow.request(
target: (1000, 600), current: (1280, 720), lastRequested: (1000, 600)))
}
}
@@ -0,0 +1,52 @@
import XCTest
@testable import PunktfunkKit
final class ResizeIndicatorTests: XCTestCase {
func testInactiveUntilSteered() {
var r = ResizeIndicator()
XCTAssertFalse(r.active)
// A decoded frame with nothing pending is a no-op (session start / steady state).
r.decoded(width: 1920, height: 1080)
XCTAssertFalse(r.active)
}
func testSteeringActivatesAndDecodedTargetClears() {
var r = ResizeIndicator()
r.steering(width: 2560, height: 1440, now: 0)
XCTAssertTrue(r.active)
// A frame at a DIFFERENT size (the old mode still draining) doesn't clear it.
r.decoded(width: 1920, height: 1080)
XCTAssertTrue(r.active)
// The target frame lands clear.
r.decoded(width: 2560, height: 1440)
XCTAssertFalse(r.active)
}
func testTimeoutClearsWhenTargetNeverArrives() {
var r = ResizeIndicator(timeout: 2.5)
r.steering(width: 2560, height: 1440, now: 10)
r.tick(now: 12) // 2 s < timeout still up
XCTAssertTrue(r.active)
r.tick(now: 12.6) // 2.6 s timeout a rejected/capped switch clears
XCTAssertFalse(r.active)
}
func testDragReArmsTimeoutOnEachNewTarget() {
var r = ResizeIndicator(timeout: 2.5)
r.steering(width: 2000, height: 1200, now: 0)
r.steering(width: 2200, height: 1200, now: 2) // target changed since re-armed to 2
r.tick(now: 4) // only 2 s since the last change still up (drag isn't a timeout)
XCTAssertTrue(r.active)
r.tick(now: 4.6) // 2.6 s since the last change clears
XCTAssertFalse(r.active)
}
func testSteadyDragDoesNotResetTimeout() {
var r = ResizeIndicator(timeout: 2.5)
r.steering(width: 2560, height: 1440, now: 0)
r.steering(width: 2560, height: 1440, now: 1) // SAME target since stays 0
r.tick(now: 2.6) // 2.6 s since the ORIGINAL steer clears (not reset by the repeat)
XCTAssertFalse(r.active)
}
}
+29 -16
View File
@@ -264,21 +264,23 @@ pub fn show(
let page = adw::PreferencesPage::new();
let stream = adw::PreferencesGroup::builder().title("Stream").build();
let res_names: Vec<String> = RESOLUTIONS
.iter()
.map(|&(w, h)| {
if w == 0 {
"Native display".to_string()
} else {
format!("{w} × {h}")
}
})
// The D1 tri-state: Native, Match window (a virtual index 1, stored as the
// `match_window` flag), then the explicit sizes.
let res_names: Vec<String> = std::iter::once("Native display".to_string())
.chain(std::iter::once("Match window".to_string()))
.chain(
RESOLUTIONS
.iter()
.skip(1)
.map(|&(w, h)| format!("{w} × {h}")),
)
.collect();
let res_row = ChoiceRow::new(
&dialog,
inline,
"Resolution",
"The host creates a virtual output at exactly this size",
"The host creates a virtual output at exactly this size — Match window follows \
the stream window, including mid-stream resizes",
&res_names.iter().map(String::as_str).collect::<Vec<_>>(),
);
let hz_names: Vec<String> = REFRESH
@@ -470,10 +472,15 @@ pub fn show(
// Seed from the current settings.
{
let s = settings.borrow();
let res_i = RESOLUTIONS
.iter()
.position(|&(w, h)| w == s.width && h == s.height)
.unwrap_or(0);
let res_i = if s.match_window {
1
} else {
RESOLUTIONS
.iter()
.position(|&(w, h)| w == s.width && h == s.height)
.map(|i| if i == 0 { 0 } else { i + 1 })
.unwrap_or(0)
};
res_row.set_selected(res_i as u32);
let hz_i = REFRESH.iter().position(|&r| r == s.refresh_hz).unwrap_or(0);
hz_row.set_selected(hz_i as u32);
@@ -508,8 +515,14 @@ pub fn show(
dialog.add(&page);
dialog.connect_closed(move |_| {
let mut s = settings.borrow_mut();
let (w, h) = RESOLUTIONS[(res_row.selected() as usize).min(RESOLUTIONS.len() - 1)];
(s.width, s.height) = (w, h);
// Index 1 is the virtual "Match window" option; 0 = Native, 2.. = explicit.
let res_i = (res_row.selected() as usize).min(RESOLUTIONS.len());
s.match_window = res_i == 1;
(s.width, s.height) = if res_i <= 1 {
(0, 0)
} else {
RESOLUTIONS[res_i - 1]
};
s.refresh_hz = REFRESH[(hz_row.selected() as usize).min(REFRESH.len() - 1)];
s.bitrate_kbps = (bitrate_row.value() * 1000.0) as u32;
s.gamepad = GAMEPADS[(pad_row.selected() as usize).min(GAMEPADS.len() - 1)].to_string();
+4
View File
@@ -145,6 +145,10 @@ pub fn run(target: Option<&str>) -> u8 {
trust::touch_last_used(&trust::hex(&fingerprint));
})),
overlay: Some(Box::new(overlay)),
window_size: crate::session_main::window_size(&settings_at_start),
// Latched at console start (like the stats tier above): toggling Match window in
// the console's settings screen applies from the next console launch.
match_window: crate::session_main::match_window(&settings_at_start),
};
let result =
+28
View File
@@ -164,6 +164,32 @@ mod session_main {
}
}
/// The window's starting size under Match-window: the persisted last size, so the
/// first connect's mode already matches the glass; `None` (policy off / never
/// stored) = the 1280×720 default.
pub(crate) fn window_size(settings: &trust::Settings) -> Option<(u32, u32)> {
(settings.match_window && settings.last_window_w > 0 && settings.last_window_h > 0)
.then_some((settings.last_window_w, settings.last_window_h))
}
/// The Match-window policy hook for the presenter loop
/// (design/midstream-resolution-resize.md D1/D2): `Some(persist)` turns the
/// debounced resize→`Reconfigure` machinery on; the callback stores each resize-end's
/// logical window size (load-modify-save, like the console settings screen) so the
/// next launch opens at it.
pub(crate) fn match_window(settings: &trust::Settings) -> Option<Box<dyn FnMut(u32, u32)>> {
settings.match_window.then(|| {
Box::new(|w: u32, h: u32| {
let mut s = trust::Settings::load();
if (s.last_window_w, s.last_window_h) != (w, h) {
s.last_window_w = w;
s.last_window_h = h;
s.save();
}
}) as Box<dyn FnMut(u32, u32)>
})
}
/// One JSON status line on stdout (the shell parses these; strings hand-escaped via
/// the minimal rules a reason string can need). `pub(crate)`: browse mode emits its
/// failure through the same contract when spawned with `--json-status`.
@@ -343,6 +369,8 @@ mod session_main {
overlay: Some(Box::new(pf_console_ui::SkiaOverlay::new())),
#[cfg(not(feature = "ui"))]
overlay: None,
window_size: window_size(&settings),
match_window: match_window(&settings),
};
let outcome =
+49 -10
View File
@@ -189,14 +189,21 @@ fn status_row(online: Option<bool>, badge: &str, kind: Pill) -> Element {
/// The in-tile rename editor (ContentDialog can't hold a text field): name box + save/cancel.
/// No tap-to-connect while editing — a click into the box would bubble `Tapped` to the region.
/// `initial` seeds the text box's displayed value and is CONSTANT for the life of the edit — the
/// field is uncontrolled, its live value kept in `live` (read at Save). Driving a *controlled* box
/// from an always-deferred `AsyncSetState` round-trip fights the caret on fast typing and can drop
/// the last char if Save is clicked before the write lands; an uncontrolled box + a ref sidesteps
/// both (and skips a full-page re-render per keystroke). See the seed block in `hosts_page`.
fn rename_editor(
draft: &str,
initial: &str,
fp: String,
live: HookRef<String>,
set_rename: AsyncSetState<Option<(String, String)>>,
) -> Element {
let commit = {
let (fp, draft, sr) = (fp.clone(), draft.to_string(), set_rename.clone());
let (fp, live, sr) = (fp.clone(), live.clone(), set_rename.clone());
move || {
let draft = live.borrow();
let name = draft.trim();
if !name.is_empty() {
let mut known = KnownHosts::load();
@@ -209,12 +216,12 @@ fn rename_editor(
}
};
let on_changed = {
let sr = set_rename.clone();
move |s: String| sr.call(Some((fp.clone(), s)))
let live = live.clone();
move |s: String| live.set(s)
};
card(
vstack((
text_box(draft)
text_box(initial)
.placeholder_text("Host name")
.on_text_changed(on_changed),
hstack((
@@ -240,6 +247,14 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
let set_screen = &props.svc.set_screen;
let set_status = &props.svc.set_status;
let (manual, set_manual) = cx.use_state(String::new());
// The Add-host field's live value, read by Connect at click time. This page's `use_state` is
// unreliable as the click's source of truth: while the modal is open the page usually has no
// reason to re-render (you open it precisely because the host ISN'T being discovered, so no
// discovery tick fires), and the top-down reconcile skips this unchanged-props subtree — so a
// sync `set_manual` write never re-renders the Connect button to re-capture the address, and it
// would connect to the empty mount-time value. Mirror every keystroke into this stable ref (the
// pair-screen PIN pattern). `manual` still drives the text box's displayed value.
let manual_live = cx.use_ref(String::new());
// "Add host" modal open state lives in ROOT (see `HostsProps`).
let show_add = props.show_add;
let set_show_add = &props.set_show_add;
@@ -249,6 +264,18 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
let rename = props.rename.clone();
let set_forget = &props.set_forget;
let set_rename = &props.set_rename;
// The live rename draft, read at Save time (see `rename_editor`). Root `rename` carries only the
// INITIAL name, so it no longer round-trips per keystroke. Seed the draft each time the rename
// TARGET changes (start, cancel, or a switch to another host).
let rename_draft = cx.use_ref(String::new());
let rename_seed = cx.use_ref(Option::<String>::None);
{
let active = rename.as_ref().map(|(fp, _)| fp.clone());
if *rename_seed.borrow() != active {
rename_draft.set(rename.as_ref().map(|(_, n)| n.clone()).unwrap_or_default());
rename_seed.set(active);
}
}
let hover = Hover {
current: props.hover.clone(),
set: props.set_hover.clone(),
@@ -393,8 +420,13 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
for k in &known.hosts {
// Rust 2021 (no let-chains): match the "this tile is being renamed" case explicitly.
if matches!(&rename, Some((fp, _)) if fp == &k.fp_hex) {
let (fp, draft) = rename.clone().unwrap();
tiles.push(rename_editor(&draft, fp, set_rename.clone()));
let (fp, initial) = rename.clone().unwrap();
tiles.push(rename_editor(
&initial,
fp,
rename_draft.clone(),
set_rename.clone(),
));
continue;
}
let target = Target {
@@ -595,14 +627,15 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
// field). The scrim border fills the cell and is hit-testable, so it blocks the page behind;
// it closes only via Cancel/Connect (a scrim tap would bubble `Tapped` up from the card too).
let connect_manual = {
let (ctx2, ss, st, text, sa) = (
let (ctx2, ss, st, live, sa) = (
ctx.clone(),
set_screen.clone(),
set_status.clone(),
manual.clone(),
manual_live.clone(),
set_show_add.clone(),
);
move || {
let text = live.borrow();
let text = text.trim();
if text.is_empty() {
return;
@@ -640,7 +673,13 @@ pub(crate) fn hosts_page(props: &HostsProps, cx: &mut RenderCx) -> Element {
text_box(manual)
.header("Address")
.placeholder_text("192.168.1.20 or my-pc.local")
.on_text_changed(move |s| set_manual.call(s))
.on_text_changed({
let live = manual_live.clone();
move |s: String| {
live.set(s.clone());
set_manual.call(s);
}
})
.margin(edges(0.0, 6.0, 0.0, 0.0)),
hstack((
button("Connect")
+1 -1
View File
@@ -23,7 +23,7 @@ pub(crate) fn licenses_page(set_screen: &AsyncSetState<Screen>) -> Element {
let app_card = card(
vstack((
text_block("punktfunk").font_size(15.0).semibold(),
text_block("Punktfunk").font_size(15.0).semibold(),
text_block("Licensed under MIT OR Apache-2.0, at your option.")
.font_size(12.0)
.wrap()
+20 -4
View File
@@ -14,21 +14,28 @@ pub(crate) fn pair_page(props: &Svc, cx: &mut RenderCx) -> Element {
let set_screen = &props.set_screen;
let set_status = &props.set_status;
let (code, set_code) = cx.use_state(String::new());
// The PIN's live value, read directly by the click handler. This page's props (`Svc`) never
// change, and root wraps every screen in an animated `border` that compares equal once the
// entrance tween settles — so the top-down reconcile `can_skip_update`s this subtree and never
// re-renders the pair component off its *local* `use_state`. A button rebuilt only at mount
// would forever capture the empty mount-time PIN (pairing then fails as a "wrong PIN"). Mirror
// every keystroke into this stable ref instead, so the click reads exactly what was typed.
let live_pin = cx.use_ref(String::new());
let target = ctx.shared.target.lock().unwrap().clone();
let pair_btn = {
let (ctx2, ss, st, code2, target2) = (
let (ctx2, ss, st, live, target2) = (
ctx.clone(),
set_screen.clone(),
set_status.clone(),
code.clone(),
live_pin.clone(),
target.clone(),
);
button("Pair & Connect")
.accent()
.icon(Symbol::Accept)
.on_click(move || {
let pin = code2.trim().to_string();
let pin = live.borrow().trim().to_string();
let (ctx3, ss, st, target3) =
(ctx2.clone(), ss.clone(), st.clone(), target2.clone());
std::thread::spawn(move || {
@@ -109,7 +116,16 @@ pub(crate) fn pair_page(props: &Svc, cx: &mut RenderCx) -> Element {
text_box(code)
.placeholder_text("PIN")
.font_size(28.0)
.on_text_changed(move |s| set_code.call(s)),
.on_text_changed({
let live = live_pin.clone();
move |s: String| {
// Record the live value for the click handler (the source of truth for the
// PIN), and mirror it into `code` so the field stays correct if anything ever
// does re-render this page (theme/DPI change).
live.set(s.clone());
set_code.call(s);
}
}),
hstack((pair_btn, cancel_btn)).spacing(8.0),
text_block(
"Don\u{2019}t have a PIN? Request access instead and approve this device on the host \
+38 -16
View File
@@ -136,29 +136,37 @@ pub(crate) fn settings_page(
let s = ctx.settings.lock().unwrap().clone();
// --- Display ---------------------------------------------------------------------------
// The D1 tri-state: Native, Match window (a virtual index 1, stored as the
// `match_window` flag), then the explicit sizes.
let (res_names, res_i) = {
let names: Vec<String> = RESOLUTIONS
.iter()
.map(|&(w, h)| {
if w == 0 {
"Native display".into()
} else {
format!("{w} \u{00D7} {h}")
}
})
let names: Vec<String> = std::iter::once("Native display".to_string())
.chain(std::iter::once("Match window".to_string()))
.chain(
RESOLUTIONS
.iter()
.skip(1)
.map(|&(w, h)| format!("{w} \u{00D7} {h}")),
)
.collect();
let i = RESOLUTIONS
.iter()
.position(|&(w, h)| w == s.width && h == s.height)
.unwrap_or(0);
let i = if s.match_window {
1
} else {
RESOLUTIONS
.iter()
.position(|&(w, h)| w == s.width && h == s.height)
.map(|i| if i == 0 { 0 } else { i + 1 })
.unwrap_or(0)
};
(names, i)
};
let res_combo = setting_combo(ctx, "Resolution", res_names, res_i, |s, i| {
(s.width, s.height) = RESOLUTIONS[i];
s.match_window = i == 1;
(s.width, s.height) = if i <= 1 { (0, 0) } else { RESOLUTIONS[i - 1] };
})
.tooltip(
"The host creates a virtual display at exactly this size. \u{201C}Native display\u{201D} \
resolves to the monitor this window is on at connect.",
resolves to the monitor this window is on at connect; \u{201C}Match window\u{201D} \
follows the stream window, including mid-stream resizes.",
);
let (hz_names, hz_i) = {
let names: Vec<String> = REFRESH
@@ -361,6 +369,16 @@ pub(crate) fn settings_page(
"Adds \u{201C}Browse library\u{2026}\u{201D} to paired hosts \u{2014} pick a game and it \
launches in the stream. Mirrors the Apple client's toggle.",
);
// App identity + version at the top of the About card (the WinUI Settings convention; the About
// screen previously showed no version at all). CARGO_PKG_VERSION is the workspace version, baked
// in at compile time.
let about_identity = vstack((
text_block("Punktfunk").font_size(20.0).semibold(),
text_block(concat!("Version ", env!("CARGO_PKG_VERSION")))
.font_size(12.0)
.foreground(ThemeRef::SecondaryText),
))
.spacing(2.0);
// The selected section's content — per-control guidance lives on hover tooltips, so the
// card is just the controls.
@@ -395,7 +413,11 @@ pub(crate) fn settings_page(
),
"about" => (
"About",
settings_card(vec![library_toggle.into(), licenses_button.into()]),
settings_card(vec![
about_identity.into(),
library_toggle.into(),
licenses_button.into(),
]),
),
_ => (
"Display",
+5
View File
@@ -380,6 +380,11 @@ fn pump(
Ok(frame) => {
// The `received` point: AU fully reassembled, handed to us, before decode.
let received_ns = now_ns();
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference-frame-
// invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers with a cheap
// clean P-frame instead of a full IDR. The frames_dropped keyframe path below is the
// backstop for when the recovery frame itself is lost.
let _ = connector.note_frame_index(frame.frame_index);
// fps = AUs received per second, Mb/s = received goodput (spec: counted at the
// received point, not the decoded one).
frames_n += 1;
+336 -6
View File
@@ -104,6 +104,81 @@ pub struct Stats {
/// IDR (or a mid-GOP join) unfreezes almost immediately instead of never.
const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3;
/// Longest the pump holds the last good frame waiting for a post-loss re-anchor keyframe before it
/// gives up and resumes display. After a reference loss the hardware decoder does not error — it
/// conceals the reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as
/// a gray plate with the new frame's motion painted over it) and returns Ok, so displaying them is
/// the "gray frames mid-stream" artifact. We instead freeze on the last good picture until a fresh
/// IDR re-anchors decode — the behaviour NVIDIA already shows (its DISTINCT output image + different
/// concealment reads as a brief freeze, not gray). This cap only bounds the freeze when recovery
/// genuinely stalls (host ignores the request, or an RFI recovery that never emits a keyframe), so a
/// glitch can never become a permanent freeze. A recovery IDR round-trips well under this on any
/// live link.
const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500);
/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
/// latest frame gap before the pump lifts its freeze on an IDR-free stream. TWO, not one: with a
/// continuous rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a
/// loss is only partially healed — stripes swept BEFORE the loss still reference the lost frame — and
/// lifting there would flash a partially-stale picture. The SECOND boundary guarantees a full wave
/// swept entirely after the loss, so the picture is clean. This stays correct under repeated loss
/// because every new gap resets the count. The cost is up to ~2 wave periods of holding the last good
/// frame — the deliberate "hold longer, never show garbage" trade.
///
/// [`USER_FLAG_RECOVERY_POINT`]: punktfunk_core::packet::USER_FLAG_RECOVERY_POINT
const REANCHOR_MARKS_TO_LIFT: u32 = 2;
/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark
/// pushes the freeze deadline out by this much, so a live mark stream (the host actively healing via
/// its wave) keeps the client patiently holding the last good frame instead of tripping the IDR
/// floor mid-heal. Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the
/// marks STOP (heal stalled, or the host isn't running intra-refresh) the deadline lapses and the
/// normal recovery-IDR floor fires, so a real stall still recovers.
const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500);
/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is
/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters
/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small
/// positive delta is a forward gap (missing frames whose dependents will decode against absent
/// references); a delta in the top half is an index behind us.
fn index_gap(expected: u32, got: u32) -> Option<u32> {
let ahead = got.wrapping_sub(expected);
(ahead != 0 && ahead < u32::MAX / 2).then_some(ahead)
}
/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze.
///
/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried
/// [`USER_FLAG_RECOVERY_ANCHOR`](punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR), the host's
/// definitive single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a
/// known-good reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait.
/// `has_mark` — this AU carried [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// a host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks
/// seen since the latest gap.
///
/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0
/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested
/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only
/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and
/// lifts immediately.
fn reanchor_after_frame(
is_keyframe: bool,
has_anchor: bool,
has_mark: bool,
marks: u32,
) -> (bool, u32) {
let marks = if has_mark {
marks.saturating_add(1)
} else {
marks
};
if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT {
(true, 0)
} else {
(false, marks)
}
}
/// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs).
/// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old
/// host (no 0xCF at all) this just caps the dead-weight ring.
@@ -319,6 +394,20 @@ fn pump(
// never drops, so the drop-count trigger below stays silent and the stream freezes
// on the last good frame. A short streak forces a fresh IDR to re-anchor.
let mut no_output_streak = 0u32;
// Freeze-until-reanchor: armed the moment we request a recovery keyframe (loss, decode error, or
// a no-output streak), it withholds the decoder's concealed frames from the presenter — which
// then redraws the last good picture — until a fresh keyframe re-anchors decode. See
// [`REANCHOR_FREEZE_MAX`] for why this exists and its backstop deadline.
let mut awaiting_reanchor = false;
let mut reanchor_deadline: Option<Instant> = None;
// Host intra-refresh recovery marks seen since the latest gap (see [`REANCHOR_MARKS_TO_LIFT`]).
// Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks.
let mut recovery_marks: u32 = 0;
// The frame_index we expect next (the host numbers frames consecutively). A jump means a frame
// went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms
// ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of
// the loss window, by which point the concealment already reached the screen).
let mut next_expected_index: Option<u32> = None;
let end: Option<String> = loop {
if stop.load(Ordering::SeqCst) {
@@ -334,9 +423,103 @@ fn pump(
// fps / goodput count every received AU (spec), decoded or not.
frames_n += 1;
bytes_n += frame.data.len() as u64;
// Reference-continuity gate: the host numbers frames consecutively, so a jump in
// frame_index means a frame is missing (lost, or an out-of-order straggler the
// reassembler emitted a newer frame ahead of) and this AU references a picture we
// never decoded. On RADV the decoder conceals that as a gray plate with the new
// motion on top — the reported artifact, and it shows most on high-motion frames (a
// full-screen pan bursts far more packets than a static desktop or a UFO-test's small
// moving sprite, so it is the frame that loses shards). Arm the freeze at the FIRST
// such frame — ~120 ms before `frames_dropped` would — so the gray never reaches the
// screen; recovery IDRs stay on the existing throttled path (see the arm below).
match next_expected_index {
Some(exp) if frame.frame_index == exp => {
next_expected_index = Some(exp.wrapping_add(1)); // contiguous
}
// A forward gap: hold the last good frame — but DO NOT ask for a keyframe here.
// Hiding the concealment is free (the presenter redraws the last picture); an IDR
// is not — at 4K120 it is a multi-megabyte frame and a visible stutter, and it can
// re-trigger the very burst loss that caused this. The existing loss recovery below
// (`frames_dropped`, host-coalesced + throttled) still requests it at exactly the
// cadence it did before this change, so we add zero IDR pressure per pan. A
// straggler behind us (`index_gap` → None) leaves the expectation put so the real
// gap still trips.
Some(exp) => {
if let Some(gap) = index_gap(exp, frame.frame_index) {
let now = Instant::now();
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
next_expected_index = Some(frame.frame_index.wrapping_add(1));
// The gap carries the PRECISE lost range — [first missing, newest
// received - 1] — so this is the one recovery signal that can drive true
// reference-frame invalidation. Prefer an RFI request over a keyframe: an
// RFI-capable host (AMD LTR / NVENC) re-references a known-good picture and
// emits a clean P-frame tagged USER_FLAG_RECOVERY_ANCHOR (the freeze lifts
// on ONE frame, no 20-40× IDR spike); an incapable/old host forces a
// host-coalesced IDR instead, or ignores it (then the frames_dropped /
// overdue keyframe paths below are the backstop). Throttled with those
// paths (one recovery ask per 100 ms) so a burst of gaps — a full-screen
// pan shedding shards — can't storm the control stream. This fires ~120 ms
// before frames_dropped would, so recovery also starts sooner.
//
// A gap wider than RFI_MAX_RANGE is beyond any encoder's reference
// history (a seconds-long outage — or a phantom index jump, e.g. the
// first real AU after an old host's speed-test burst consumed video
// indexes): RFI is hopeless there, so ask for the IDR resync directly.
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now);
if gap > punktfunk_core::packet::RFI_MAX_RANGE {
let _ = connector.request_keyframe();
} else {
let _ = connector
.request_rfi(exp, frame.frame_index.wrapping_sub(1));
}
}
tracing::trace!(
gap,
"frame gap — RFI recovery, holding last frame until re-anchor"
);
}
}
None => next_expected_index = Some(frame.frame_index.wrapping_add(1)),
}
match decoder.decode(&frame.data) {
Ok(Some(image)) => {
no_output_streak = 0; // a decoded frame — the anchor holds
// A decoded frame — the anchor holds.
no_output_streak = 0;
// Host-signalled intra-refresh recovery mark: on an IDR-free intra-refresh
// stream this wave-boundary flag is the only clean point the client can honor
// (the decoder never flags the re-anchor — the coded frame stays `P`). A live
// mark stream also means the host is actively healing, so push the backstop out
// rather than trip a mid-heal IDR (see `RECOVERY_MARK_PATIENCE`).
let has_mark =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_POINT != 0;
// The host's definitive single-frame re-anchor: an LTR-RFI recovery frame (a
// clean P-frame off a known-good reference), the AMD twin of an IDR re-anchor
// but without the spike. It lifts on the FIRST occurrence.
let has_anchor =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR != 0;
if has_mark && awaiting_reanchor {
reanchor_deadline = Some(Instant::now() + RECOVERY_MARK_PATIENCE);
}
// A fresh clean re-anchor lifts the freeze and shows this frame: a real intra
// keyframe (IDR, always clean), an LTR-RFI recovery anchor (also whole), OR the
// second recovery mark since the gap (the first wave boundary is only
// half-healed — see `reanchor_after_frame`).
let (lift, marks) = reanchor_after_frame(
image.is_keyframe(),
has_anchor,
has_mark,
recovery_marks,
);
recovery_marks = marks;
if lift {
awaiting_reanchor = false;
reanchor_deadline = None;
}
total_frames += 1;
dec_path = match &image {
DecodedImage::Cpu(_) => "software",
@@ -391,11 +574,20 @@ fn pump(
DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)),
_ => None,
};
let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns,
decoded_ns,
image,
});
if awaiting_reanchor {
// Post-loss concealment: withhold this frame (it references a lost/gray
// reference) so the presenter keeps redrawing the last good picture
// rather than flashing the decoder's gray plate. Dropped here — the
// hw-decode stat below still samples via `hw_fence` (raw handle + value,
// valid past the guard). Cleared by the next keyframe or the backstop.
tracing::trace!("holding last frame — awaiting post-loss re-anchor");
} else {
let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns,
decoded_ns,
image,
});
}
// `decode` stage: received→decode COMPLETE, single clock.
match hw_fence {
Some((sem, value)) => {
@@ -424,6 +616,12 @@ fn pump(
// trip before asking again instead of flooding.
if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK {
let now = Instant::now();
// Wedged on missing references: hold the last good frame until re-anchor
// (armed even when the IDR request itself is throttled — the stream is broken
// regardless of whether we ask again this iteration).
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
@@ -451,6 +649,9 @@ fn pump(
// through the same throttle as loss recovery below.
if decoder.take_keyframe_request() {
let now = Instant::now();
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
@@ -487,12 +688,33 @@ fn pump(
if dropped > last_dropped {
last_dropped = dropped;
let now = Instant::now();
// A dropped AU means the frames after it reference a picture we never decoded — the
// decoder will conceal them (gray on RADV). Freeze on the last good frame until a fresh
// IDR re-anchors, so the concealment never reaches the screen.
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!(dropped, "requested keyframe (loss recovery)");
}
}
// Re-anchor overdue: the freeze has held the whole window with no keyframe — a lost recovery
// IDR, or a benign reorder that produced no `frames_dropped` and so requested none. Do NOT
// resume to gray (the one thing worse than a freeze): keep holding the last good frame and
// (re-)request a keyframe, throttled + host-coalesced, so a CLEAN re-anchor is what un-freezes
// us. A genuinely dead stream — host gone, link collapsed — is caught by the QUIC idle-timeout
// watchdog (returns to the menu), never by painting the decoder's concealment.
if awaiting_reanchor && reanchor_deadline.is_some_and(|d| Instant::now() >= d) {
let now = Instant::now();
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!("re-anchor overdue — still holding, re-requesting keyframe");
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32();
@@ -614,3 +836,111 @@ fn spawn_audio(
.map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled"))
.ok()
}
#[cfg(test)]
mod tests {
use super::{index_gap, reanchor_after_frame, REANCHOR_MARKS_TO_LIFT};
// Simulate the pump's re-anchor state across a sequence of decoded frames: each `(is_keyframe,
// has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based)
// at which the freeze first lifts, or `None` if it never does. `gap_before` reset points model a
// fresh loss re-arming the freeze (the pump zeroes the count at every gap/arm site).
fn lift_at(frames: &[(bool, bool)]) -> Option<usize> {
let mut marks = 0u32;
for (i, &(is_kf, has_mark)) in frames.iter().enumerate() {
// The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised
// by `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here.
let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks);
marks = m;
if lift {
return Some(i);
}
}
None
}
#[test]
fn a_single_recovery_mark_does_not_lift() {
// The first wave boundary after a loss is only half-healed — one mark must hold the freeze.
assert_eq!(REANCHOR_MARKS_TO_LIFT, 2);
assert_eq!(lift_at(&[(false, true)]), None);
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false)]),
None
);
}
#[test]
fn the_second_recovery_mark_lifts() {
// Two marks = a full wave swept after the loss → clean re-anchor.
assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1));
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false), (false, true)]),
Some(3)
);
}
#[test]
fn a_real_keyframe_lifts_immediately() {
// An IDR is always a clean anchor — no marks needed.
assert_eq!(lift_at(&[(true, false)]), Some(0));
assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1));
}
#[test]
fn a_fresh_gap_resets_the_mark_count() {
// The pump zeroes `recovery_marks` at each arm site, so one mark before a new gap plus one
// after must NOT lift — the model resets the running count to imitate that.
let mut marks = 0u32;
let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap)
marks = m;
assert_eq!(marks, 1);
marks = 0; // a new gap re-arms the freeze → count reset
let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave
assert!(!lift, "a single post-gap mark must not lift");
assert_eq!(m, 1);
}
#[test]
fn an_rfi_anchor_lifts_immediately() {
// An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good
// reference), so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait.
let (lift, marks) = reanchor_after_frame(false, true, false, 0);
assert!(lift, "an RFI anchor must lift the freeze immediately");
assert_eq!(marks, 0, "a lift resets the running mark count");
// Even with zero prior marks and no keyframe, the anchor alone is sufficient.
let (lift, _) = reanchor_after_frame(false, true, true, 1);
assert!(lift, "an anchor lifts regardless of the pending mark count");
}
#[test]
fn contiguous_indices_are_not_a_gap() {
assert_eq!(index_gap(5, 5), None);
assert_eq!(index_gap(0, 0), None);
}
#[test]
fn a_forward_jump_reports_the_skip_count() {
assert_eq!(index_gap(5, 6), Some(1)); // one frame missing
assert_eq!(index_gap(5, 9), Some(4));
}
#[test]
fn a_straggler_behind_us_is_not_a_gap() {
// The reassembler emitted a newer frame first; the late one must not re-arm.
assert_eq!(index_gap(9, 5), None);
assert_eq!(index_gap(1, 0), None);
}
#[test]
fn the_index_counter_wraps_cleanly() {
// last frame = u32::MAX, so the next expected wraps to 0.
// Contiguous across the wrap.
assert_eq!(index_gap(0, 0), None);
// waiting on u32::MAX, frame 0 arrived → MAX was skipped.
assert_eq!(index_gap(u32::MAX, 0), Some(1));
assert_eq!(index_gap(u32::MAX, 2), Some(3));
// an old frame arriving just after the wrap is still a straggler.
assert_eq!(index_gap(0, u32::MAX), None);
}
}
+16
View File
@@ -406,6 +406,19 @@ pub struct Settings {
/// Experimental: the game-library browser ("Browse library…" on saved cards) —
/// mirrors the Apple client's "Show game library" toggle, default off.
pub library_enabled: bool,
/// Match-window resolution policy (design/midstream-resolution-resize.md D1): the
/// stream mode follows the session window — the connect asks for the window's pixel
/// size and a mid-session resize renegotiates the host's virtual display + encoder
/// (`Reconfigure`), so windowed sessions stream native-resolution pixels instead of
/// scaling. Overrides `width`/`height` while on; on fullscreen it degenerates to the
/// display's native mode. Default off (Auto-native stays the shipped default until
/// the per-backend validation matrix is green).
pub match_window: bool,
/// The session window's last logical size under `match_window`: the next launch
/// opens its window at this size, so the first connect's mode already matches what
/// the user will be looking at. `0` = never stored → the 1280×720 default.
pub last_window_w: u32,
pub last_window_h: u32,
}
fn default_codec() -> String {
@@ -466,6 +479,9 @@ impl Default for Settings {
stats_verbosity: None,
fullscreen_on_stream: true,
library_enabled: false,
match_window: false,
last_window_w: 0,
last_window_h: 0,
}
}
}
+65
View File
@@ -98,6 +98,10 @@ pub struct VkVideoFrame {
pub width: u32,
pub height: u32,
pub color: ColorDesc,
/// Intra keyframe (IDR/I): the stream's re-anchor point. The pump resumes display on
/// one after suppressing the concealed frames a reference loss leaves in its wake (on
/// RADV a lost reference decodes to a gray plate with the new motion painted on top).
pub keyframe: bool,
/// Keeps the cloned AVFrame (and through it the VkImage + frames context) alive
/// until the presenter's fence proves the GPU reads done — same mechanism as the
/// VAAPI path's DRM guard.
@@ -143,6 +147,58 @@ impl ColorDesc {
}
}
/// True if the decoder tagged this frame as a full IDR keyframe — a guaranteed clean re-anchor
/// after which the picture is loss-free, so the pump can lift a post-loss display freeze here.
///
/// Keys off `AV_FRAME_FLAG_KEY` (with `pict_type == I` as a belt for decoders that fill pict_type
/// but not the flag). NOTE: FFmpeg's H.264/HEVC decode layer sets this flag **only for true IDR
/// frames**, never for an *intra-refresh recovery point*. H.264 flags key only when a picture's
/// `recovery_frame_cnt == 0` (a moving band uses `> 0`); HEVC clears the flag on every non-IRAP
/// frame regardless of the recovery-point SEI. So an intra-refresh host (NVENC/AMF/QSV) heals the
/// picture over N P-frames with no decoded frame ever flagged key — this function cannot detect
/// that clean point, and the pump would freeze until the `REANCHOR_FREEZE_MAX` backstop (in
/// `session.rs`) forces a real IDR. Detecting an intra-refresh re-anchor requires an out-of-band
/// host wire signal on the AU that completes the wave; that is not yet plumbed.
///
/// # Safety
/// `frame` must point to a valid `AVFrame` alive for the duration of the call.
pub unsafe fn frame_is_keyframe(frame: *const ffmpeg::ffi::AVFrame) -> bool {
// SAFETY: caller guarantees a live AVFrame; plain field reads.
unsafe {
((*frame).flags & ffmpeg::ffi::AV_FRAME_FLAG_KEY) != 0
|| (*frame).pict_type == ffmpeg::ffi::AVPictureType::AV_PICTURE_TYPE_I
}
}
impl DecodedImage {
/// Whether the frame is an intra keyframe — see [`frame_is_keyframe`]. The pump uses
/// this as the stream's re-anchor signal after a loss.
pub fn is_keyframe(&self) -> bool {
match self {
DecodedImage::Cpu(f) => f.keyframe,
#[cfg(target_os = "linux")]
DecodedImage::Dmabuf(f) => f.keyframe,
DecodedImage::VkFrame(f) => f.keyframe,
#[cfg(windows)]
DecodedImage::D3d11(f) => f.keyframe,
}
}
/// The decoded image's pixel dimensions. The presenter's resize indicator uses these
/// as the mid-stream-resize END signal: a frame arriving at the target size means the
/// new-mode picture is on glass (the ack alone lands before the host's rebuild does).
pub fn dimensions(&self) -> (u32, u32) {
match self {
DecodedImage::Cpu(f) => (f.width, f.height),
#[cfg(target_os = "linux")]
DecodedImage::Dmabuf(f) => (f.width, f.height),
DecodedImage::VkFrame(f) => (f.width, f.height),
#[cfg(windows)]
DecodedImage::D3d11(f) => (f.width, f.height),
}
}
}
/// The YCbCr→RGB conversion as three vec4 rows for a shader constant buffer / push-constant
/// block: `rgb[i] = dot(r[i].xyz, yuv) + r[i].w` — bit-depth exact. The ONE coefficient
/// implementation every presenter derives its CSC from (Vulkan push constants, the Windows
@@ -205,6 +261,8 @@ pub struct CpuFrame {
/// pixels are full-range RGB), but a PQ/BT.2020 stream keeps its transfer + primaries
/// baked in — the presenter tags the texture so GTK tone-maps it.
pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
pub keyframe: bool,
}
/// A decoded frame still on the GPU: dmabuf fds + plane layout for
@@ -222,6 +280,8 @@ pub struct DmabufFrame {
/// Signaling of the source frame — drives the `GdkDmabufTexture` color state (BT.709
/// narrow for SDR, BT.2020 PQ for an HDR stream).
pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See [`VkVideoFrame`].
pub keyframe: bool,
pub guard: DrmFrameGuard,
}
@@ -644,6 +704,9 @@ impl SoftwareDecoder {
stride: dst_linesize[0] as usize,
rgba,
color,
// `is_key()` reads the same intra flag `frame_is_keyframe` derives from pict_type
// for the hardware paths; ffmpeg-next handles the FFmpeg-version binding split.
keyframe: frame.is_key(),
})
}
}
@@ -844,6 +907,7 @@ impl VaapiDecoder {
// SAFETY: `self.frame` is the live decoded AVFrame (unref'd only after
// this returns); plain CICP field reads.
color: ColorDesc::from_raw(self.frame),
keyframe: frame_is_keyframe(self.frame),
guard,
})
}
@@ -1363,6 +1427,7 @@ impl VulkanDecoder {
width: (*self.frame).width as u32,
height: (*self.frame).height as u32,
color: ColorDesc::from_raw(self.frame),
keyframe: frame_is_keyframe(self.frame),
guard: DrmFrameGuard(clone),
})
}
+5
View File
@@ -99,6 +99,9 @@ pub struct D3d11Frame {
/// BT.709 full-range RGB — regardless of the stream's own CICP (a PQ stream was
/// tone-mapped). The presenter keys SDR/HDR handling off this, so it always reads SDR.
pub color: ColorDesc,
/// Intra keyframe (IDR/I) — the pump's post-loss re-anchor signal. See
/// `crate::video::VkVideoFrame`.
pub keyframe: bool,
/// The ring slot's NT shared handle (`IDXGIResource1::CreateSharedHandle`), stable for the
/// ring's lifetime. Raw `isize` so the frame crosses the pump→presenter channel.
pub handle: isize,
@@ -692,6 +695,8 @@ impl D3d11vaDecoder {
matrix: 0, // identity — RGB
full_range: true,
},
// SAFETY: `self.frame` is the live decoded AVFrame for this call.
keyframe: crate::video::frame_is_keyframe(self.frame),
handle,
generation,
})
+35 -8
View File
@@ -176,7 +176,9 @@ fn row_spec(id: RowId, ctx: &Ctx) -> RowSpec {
RowId::Resolution => (
Some("Stream"),
"Resolution",
if s.width == 0 {
if s.match_window {
"Match window".into()
} else if s.width == 0 {
"Native".into()
} else {
format!("{} × {}", s.width, s.height)
@@ -259,7 +261,8 @@ fn row_spec(id: RowId, ctx: &Ctx) -> RowSpec {
fn detail(id: RowId) -> &'static str {
match id {
RowId::Resolution => {
"The host creates a virtual display at exactly this size — no scaling."
"The host creates a virtual display at exactly this size — no scaling. \
Match window follows this window, including mid-stream resizes."
}
RowId::Refresh => "Native follows the display this window is on.",
RowId::Bitrate => "Automatic uses the host's default (20 Mbps).",
@@ -303,11 +306,20 @@ fn adjust(id: RowId, delta: i32, wrap: bool, ctx: &mut Ctx) -> bool {
let s = &mut *ctx.settings;
match id {
RowId::Resolution => {
let cur = RESOLUTIONS
.iter()
.position(|(w, h)| (*w, *h) == (s.width, s.height));
step_option(cur, RESOLUTIONS.len(), delta, wrap).map(|i| {
(s.width, s.height) = RESOLUTIONS[i];
// The D1 tri-state as one picker: Native, Match window, then the explicit
// sizes (RESOLUTIONS keeps its (0,0) = Native head; Match window is the
// virtual index 1, stored as the `match_window` flag with w/h cleared).
let cur = if s.match_window {
Some(1)
} else {
RESOLUTIONS
.iter()
.position(|(w, h)| (*w, *h) == (s.width, s.height))
.map(|i| if i == 0 { 0 } else { i + 1 })
};
step_option(cur, RESOLUTIONS.len() + 1, delta, wrap).map(|i| {
s.match_window = i == 1;
(s.width, s.height) = if i <= 1 { (0, 0) } else { RESOLUTIONS[i - 1] };
})
}
RowId::Refresh => {
@@ -401,14 +413,29 @@ mod tests {
device_name: "t",
t: 0.0,
};
// Resolution starts at Native (index 0): left refuses, right steps.
// Resolution starts at Native (index 0): left refuses, right steps — first onto
// Match window (the D1 tri-state's middle option), then the explicit sizes.
assert!(!adjust(RowId::Resolution, -1, false, &mut ctx));
assert!(adjust(RowId::Resolution, 1, false, &mut ctx));
assert!(ctx.settings.match_window, "Native → Match window");
assert_eq!((ctx.settings.width, ctx.settings.height), (0, 0));
assert!(adjust(RowId::Resolution, 1, false, &mut ctx));
assert!(
!ctx.settings.match_window,
"explicit size clears the policy"
);
assert_eq!((ctx.settings.width, ctx.settings.height), (1280, 720));
// Stepping back from an explicit size returns to Match window, then Native.
assert!(adjust(RowId::Resolution, -1, false, &mut ctx));
assert!(ctx.settings.match_window);
assert!(adjust(RowId::Resolution, -1, false, &mut ctx));
assert!(!ctx.settings.match_window);
assert_eq!(ctx.settings.width, 0, "back to Native");
// Cycle from the last option wraps to the first.
(ctx.settings.width, ctx.settings.height) = (3840, 2160);
assert!(adjust(RowId::Resolution, 1, true, &mut ctx));
assert_eq!(ctx.settings.width, 0, "wrapped to Native");
assert!(!ctx.settings.match_window);
}
#[test]
+158 -130
View File
@@ -19,7 +19,9 @@ use anyhow::{anyhow, Result};
use pf_client_core::gamepad::{MenuDir, MenuEvent, MenuPulse, PadInfo};
use pf_client_core::trust;
use pf_presenter::overlay::OverlayAction;
use skia_safe::{Canvas, Color4f, Data, Paint, Rect, RuntimeEffect};
use skia_safe::{
gradient_shader, Canvas, Color4f, Data, Paint, Point, Rect, RuntimeEffect, TileMode,
};
use std::collections::VecDeque;
use std::time::Instant;
@@ -240,6 +242,11 @@ impl Shell {
self.wake = None;
if let Some(Some(intent)) = intent {
self.start_connect(intent);
// The wake takeover was already full-screen; skip the connect fade-in so the
// Waking → Connecting handoff is seamless (no flash of the home behind).
if let Some(c) = &mut self.connecting {
c.appear = 1.0;
}
}
}
}
@@ -607,77 +614,68 @@ impl Shell {
t: f64,
fonts: &Fonts,
) {
if let Some(c) = &mut self.connecting {
c.appear = approach(c.appear, 1.0, dt, 0.07);
let a = c.appear;
canvas.draw_rect(
Rect::from_wh(w as f32, h as f32),
&Paint::new(Color4f::new(0.0, 0.0, 0.0, (0.45 * a) as f32), None),
);
let title = if c.canceling {
"Canceling…".to_string()
// Resolve the connect/wake takeover — the two phases of reaching a host — into one
// full-screen shape (spinner, title, one detail line, its own hints). Connecting flows
// straight out of a wake (see `sync`) so they share the same backdrop and never blink
// between them. Mirrors the Android client's unified `ConnectOverlay`.
let takeover: Option<(f64, bool, String, String, Vec<Hint>)> =
if let Some(c) = &mut self.connecting {
c.appear = approach(c.appear, 1.0, dt, 0.07);
if c.canceling {
Some((
c.appear,
true,
"Canceling…".to_string(),
String::new(),
vec![],
))
} else {
Some((
c.appear,
true,
format!("Connecting to {}", c.title),
"Starting the stream in this window.".to_string(),
vec![Hint::new(HintKey::Back, "Cancel")],
))
}
} else if let Some(wk) = &self.wake {
// Service-driven, so it appears settled (no fade-in).
if wk.timed_out {
Some((
1.0,
false,
format!("{} didn't wake", wk.name),
"Check its power settings, or wake it manually and try again.".to_string(),
vec![
Hint::new(HintKey::Confirm, "Try Again"),
Hint::new(HintKey::Back, "Cancel"),
],
))
} else {
Some((
1.0,
true,
format!("Waking {}", wk.name),
format!("Waiting for it to come online · {} s", wk.seconds),
// A wake-only wait (no dial after) offers "Stop Waiting"; a wake-&-connect
// is a plain "Cancel".
vec![Hint::new(
HintKey::Back,
if wk.then_connect {
"Cancel"
} else {
"Stop Waiting"
},
)],
))
}
} else {
format!("Connecting to {}", c.title)
None
};
let hints = if c.canceling {
vec![]
} else {
vec![Hint::new(HintKey::Back, "Cancel")]
};
card(
canvas,
fonts,
w,
h,
k,
a,
t,
self.glyphs,
true,
&title,
"Starting the stream in this window.",
&hints,
if let Some((appear, spinner, title, body, hints)) = takeover {
self.draw_takeover(
canvas, w, h, k, appear, t, fonts, spinner, &title, &body, &hints,
);
} else if let Some(wk) = &self.wake {
let a = 1.0; // the wake card is service-driven; it appears settled
canvas.draw_rect(
Rect::from_wh(w as f32, h as f32),
&Paint::new(Color4f::new(0.0, 0.0, 0.0, 0.45), None),
);
if wk.timed_out {
card(
canvas,
fonts,
w,
h,
k,
a,
t,
self.glyphs,
false,
&format!("{} didn't wake", wk.name),
"Check its power settings, or wake it manually and try again.",
&[
Hint::new(HintKey::Confirm, "Try Again"),
Hint::new(HintKey::Back, "Cancel"),
],
);
} else {
card(
canvas,
fonts,
w,
h,
k,
a,
t,
self.glyphs,
true,
&format!("Waking {}", wk.name),
&format!("Waiting for it to come online · {} s", wk.seconds),
&[Hint::new(HintKey::Back, "Cancel")],
);
}
}
// The toast: a transient pill above the hint bar; slides in, fades out.
@@ -799,70 +797,91 @@ impl LayerEnv<'_> {
}
}
/// A centered modal card: spinner (or not), a title, one detail line, and its own hint
/// row — the connect/wake overlays share this one shape.
#[allow(clippy::too_many_arguments)]
fn card(
canvas: &Canvas,
fonts: &Fonts,
w: f64,
h: f64,
k: f64,
appear: f64,
t: f64,
glyphs: GlyphStyle,
spinner: bool,
title: &str,
body: &str,
hints: &[Hint],
) {
let cw = (440.0 * k).min(w * 0.86);
let ch = 190.0 * k;
let cx = w / 2.0;
let top = h / 2.0 - ch / 2.0 + (1.0 - appear) * 14.0 * k;
canvas.save_layer_alpha_f(None, appear as f32);
let rect = Rect::from_xywh((cx - cw / 2.0) as f32, top as f32, cw as f32, ch as f32);
crate::theme::drop_shadow(canvas, rect, 22.0, k as f32, 0.5);
crate::theme::panel(
canvas,
rect,
22.0,
Some(Color4f::new(0.07, 0.06, 0.12, 0.85)),
PanelStroke::Plain(0.14),
k as f32,
);
let mut y = top + 44.0 * k;
if spinner {
crate::theme::spinner(canvas, cx, y, 14.0 * k, t);
y += 34.0 * k;
} else {
y += 6.0 * k;
}
fonts.centered(canvas, title, W::SemiBold, 19.0 * k, WHITE, cx, y, cw * 0.9);
fonts.centered(
canvas,
body,
W::Regular,
13.0 * k,
DIM,
cx,
y + 30.0 * k,
cw * 0.86,
);
if !hints.is_empty() {
// Centered inside the card's bottom band.
let probe = hint_bar(canvas, fonts, hints, glyphs, -10_000.0, -10_000.0, k);
hint_bar(
impl Shell {
/// A full-screen connect/wake takeover: a fresh aurora over everything (so the carousel and
/// chrome fall away), a centered spinner (or none, when a wake has timed out), a title, one
/// detail line, and its own bottom hint row. `appear` fades the whole thing in over the home;
/// a wake that hands off to a connect passes 1.0 so the two never blink between them. The
/// console counterpart of the Android/Apple `ConnectOverlay` — one full-screen shape, not a
/// centered modal card.
#[allow(clippy::too_many_arguments)]
fn draw_takeover(
&self,
canvas: &Canvas,
w: f64,
h: f64,
k: f64,
appear: f64,
t: f64,
fonts: &Fonts,
spinner: bool,
title: &str,
body: &str,
hints: &[Hint],
) {
let cx = w / 2.0;
canvas.save_layer_alpha_f(None, appear as f32);
// Opaque aurora — the same living backdrop the home wears, so the takeover reads as the
// console taking over rather than a card popping up.
self.draw_aurora(canvas, w, h, t);
// A soft pool of shade under the centre seats the white text against a bright aurora.
let mut vignette = Paint::default();
vignette.set_shader(gradient_shader::radial(
Point::new(cx as f32, (h / 2.0) as f32),
(w.max(h) * 0.42) as f32,
gradient_shader::GradientShaderColors::Colors(&[
Color4f::new(0.0, 0.0, 0.0, 0.5).to_color(),
Color4f::new(0.0, 0.0, 0.0, 0.0).to_color(),
]),
None,
TileMode::Clamp,
None,
None,
));
canvas.draw_rect(Rect::from_wh(w as f32, h as f32), &vignette);
// Centre the spinner + title + detail as a group around the middle of the screen.
let title_y = h / 2.0 + if spinner { 14.0 * k } else { 0.0 };
if spinner {
crate::theme::spinner(canvas, cx, title_y - 52.0 * k, 22.0 * k, t);
}
fonts.centered(
canvas,
fonts,
hints,
glyphs,
cx - probe.0 / 2.0,
top + ch - 16.0 * k,
k,
title,
W::SemiBold,
23.0 * k,
WHITE,
cx,
title_y,
w * 0.82,
);
if !body.is_empty() {
fonts.centered(
canvas,
body,
W::Regular,
14.0 * k,
DIM,
cx,
title_y + 32.0 * k,
w * 0.66,
);
}
if !hints.is_empty() {
// Centered near the bottom, where every console screen's legend sits.
let probe = hint_bar(canvas, fonts, hints, self.glyphs, -10_000.0, -10_000.0, k);
hint_bar(
canvas,
fonts,
hints,
self.glyphs,
cx - probe.0 / 2.0,
h - 34.0 * k,
k,
);
}
canvas.restore();
}
canvas.restore();
}
#[cfg(test)]
@@ -1154,6 +1173,15 @@ mod tests {
then_connect: true,
}));
dump(&mut s, 10, 8, "08-waking", true);
console.set_wake(Some(WakeStatus {
key: "bb22".into(),
name: "Office Tower".into(),
seconds: 90,
timed_out: true,
online: false,
then_connect: true,
}));
dump(&mut s, 10, 8, "08b-wake-timed-out", true);
console.set_wake(None);
s.set_connecting(Some("Elden Ring".into()));
dump(&mut s, 10, 8, "09-connecting", true);
+64 -2
View File
@@ -50,6 +50,10 @@ struct Drawn {
hint: Option<String>,
/// The start banner's alpha, quantized — a fade step is a redraw, steady is not.
banner_step: u8,
/// The resize scrim's spinner phase, quantized — a nonzero, ever-changing step while a
/// mid-stream resize is in flight forces the per-frame redraw the spinner needs; `0`
/// when no resize is showing (so a still stream stays damage-free).
resize_step: u16,
}
/// Where the console starts (the session binary's `--browse` forms).
@@ -85,6 +89,9 @@ pub struct SkiaOverlay {
streaming_since: Option<Instant>,
/// The banner's words (set per stream from the active-pad state).
banner_text: Option<String>,
/// When the current mid-stream resize scrim began showing — drives its spinner phase.
/// `None` = no resize in flight (`FrameCtx::resizing` was false last frame).
resizing_since: Option<Instant>,
}
struct Gpu {
@@ -114,6 +121,7 @@ impl SkiaOverlay {
shell: None,
streaming_since: None,
banner_text: None,
resizing_since: None,
}
}
@@ -340,10 +348,15 @@ impl Overlay for SkiaOverlay {
}));
}
// --- Stream chrome: stats OSD + capture hint + the start banner ---------------
// --- Stream chrome: stats OSD + capture hint + start banner + resize scrim -----
let banner_alpha = self.banner_alpha(ctx);
let banner_step = (banner_alpha * 32.0).round() as u8;
if ctx.stats.is_none() && ctx.hint.is_none() && banner_step == 0 {
let resize_phase = self.resize_phase(ctx);
// 120 steps/s: every ~16 ms frame lands on a fresh step, so the spinner keeps
// spinning through the damage gate; `+ 1` keeps an active resize's step nonzero
// even on its first frame (phase 0) so the guard below doesn't skip it.
let resize_step = resize_phase.map_or(0, |p| (p * 120.0) as u16 + 1);
if ctx.stats.is_none() && ctx.hint.is_none() && banner_step == 0 && resize_step == 0 {
self.drawn = Drawn::default(); // forget content so re-show re-renders
return Ok(None);
}
@@ -353,6 +366,7 @@ impl Overlay for SkiaOverlay {
stats: ctx.stats.map(str::to_owned),
hint: ctx.hint.map(str::to_owned),
banner_step,
resize_step,
};
if want == self.drawn {
// Unchanged — hand the presenter the already-rendered image.
@@ -374,6 +388,10 @@ impl Overlay for SkiaOverlay {
let canvas = slot.surface.canvas();
canvas.clear(Color4f::new(0.0, 0.0, 0.0, 0.0));
let font = self.font.as_ref().expect("init ran");
// The resize scrim sits UNDER the OSD/hint so those stay legible over it.
if let Some(phase) = resize_phase {
draw_resize_scrim(canvas, font, ctx.width, ctx.height, phase);
}
if let Some(stats) = &want.stats {
draw_osd_panel(canvas, font, stats, 12.0, 12.0);
}
@@ -445,6 +463,23 @@ impl SkiaOverlay {
((BANNER_S - age) / BANNER_FADE_S).min(1.0)
}
/// The mid-stream-resize spinner's phase (elapsed seconds since the scrim came up), or
/// `None` when no resize is in flight. Latches the start on the first `resizing` frame
/// and clears it the moment the run loop drops the flag (the target frame landed or the
/// switch timed out), so the next resize starts its spinner from zero.
fn resize_phase(&mut self, ctx: &FrameCtx) -> Option<f64> {
if !ctx.resizing {
self.resizing_since = None;
return None;
}
Some(
self.resizing_since
.get_or_insert_with(Instant::now)
.elapsed()
.as_secs_f64(),
)
}
/// Make `slots[i]` a render target of exactly `width`×`height` (rebuilt on resize).
fn ensure_slot(&mut self, i: usize, width: u32, height: u32) -> Result<()> {
if self.slots[i]
@@ -540,6 +575,33 @@ fn draw_osd_panel(canvas: &Canvas, font: &Font, text: &str, x: f32, y: f32) {
}
}
/// The mid-stream-resize cover: a full-screen dark scrim, the shared rotating spinner, and
/// a "Resizing…" label centered over it — so the host's 0.32 s virtual-display + encoder
/// rebuild reads as a deliberate pause rather than the stream stretching to the changed
/// window. This is the presenter's analog of the Apple client's blur overlay: the overlay
/// composites its own RGBA quad and cannot sample the video to blur it, so an opaque scrim
/// hides the stretched in-between frame instead (same intent, one draw).
fn draw_resize_scrim(canvas: &Canvas, font: &Font, width: u32, height: u32, phase: f64) {
let (wf, hf) = (width as f32, height as f32);
canvas.draw_rect(
Rect::from_wh(wf, hf),
&Paint::new(Color4f::new(0.0, 0.0, 0.0, 0.55), None),
);
// Spinner slightly above center; the label sits below it.
let (cx, cy) = (f64::from(width) / 2.0, f64::from(height) / 2.0);
let r = (f64::from(width.min(height)) * 0.045).clamp(16.0, 44.0);
crate::theme::spinner(canvas, cx, cy - r, r, phase);
let (_, metrics) = font.metrics();
let label = "Resizing\u{2026}";
let tw = font.measure_str(label, None).0;
canvas.draw_str(
label,
Point::new((wf - tw) / 2.0, (cy + r * 0.9) as f32 - metrics.ascent),
font,
&Paint::new(Color4f::new(1.0, 1.0, 1.0, 0.9), None),
);
}
/// The capture hint / start banner: a centered pill near the bottom edge.
fn draw_hint_pill(canvas: &Canvas, font: &Font, text: &str, width: u32, height: u32, alpha: f32) {
let (_, metrics) = font.metrics();
+5
View File
@@ -37,6 +37,11 @@ pub struct FrameCtx<'a> {
pub stats: Option<&'a str>,
/// The capture hint (bottom-center pill, "click to capture…"); `None` = hidden.
pub hint: Option<&'a str>,
/// A mid-stream Match-window resize is in flight (design/midstream-resolution-resize.md,
/// client UX): draw a full-screen scrim + spinner so the host's 0.32 s virtual-display
/// and encoder rebuild reads as an intentional pause rather than the stream stretching to
/// the changed window. Cleared the instant the sharp new-resolution frame is on glass.
pub resizing: bool,
/// The active gamepad's name (the console library's controller chip).
pub pad: Option<&'a str>,
/// The active pad's resolved kind — drives the console UI's button glyphs
+449 -9
View File
@@ -52,6 +52,18 @@ pub struct SessionOpts {
/// stay stdout-only). An overlay whose `init` fails degrades to `None` with a
/// warning rather than killing the session. Browse mode requires one.
pub overlay: Option<Box<dyn Overlay>>,
/// The window's starting logical size; `None` = the 1280×720 default. The binary
/// passes the persisted last-window size under the Match-window policy so the first
/// connect's mode already matches what the user will be looking at.
pub window_size: Option<(u32, u32)>,
/// Match-window resolution policy (design/midstream-resolution-resize.md D1/D2):
/// `Some` = the stream mode follows the window. At session start the params' mode
/// w/h are replaced by the window's physical pixel size; a mid-session resize sends
/// a debounced `Reconfigure` so the host's virtual display + encoder follow. The
/// callback receives the window's logical size at each resize-end — the binary
/// persists it for the next launch. `None` = never auto-resize (Auto-native /
/// Explicit keep today's behavior).
pub match_window: Option<Box<dyn FnMut(u32, u32)>>,
}
pub enum Outcome {
@@ -173,6 +185,25 @@ struct StreamState {
/// The last pump window, kept so a Ctrl+Alt+Shift+S tier cycle can re-render the
/// OSD immediately instead of waiting up to 1 s for the next Stats event.
last_stats: Option<Stats>,
/// Match-window (D2) debounce state: the last resize event's stamp. `Some` = a
/// resize is pending; the tick fires the request once ~400 ms pass with no further
/// size events (never per drag-frame — each accepted switch is a full host rebuild).
resize_pending: Option<Instant>,
/// When the last `Reconfigure` was sent — the ≥ 1 s spacing between requests (D2).
/// The accept ack round-trips in milliseconds (it precedes the host's rebuild), so
/// this spacing also serializes: at most ~one request is ever outstanding.
resize_sent_at: Option<Instant>,
/// The last size actually requested. Each distinct size is requested at most once:
/// this both implements "don't re-request a rejected size until it changes" (D2) and
/// keeps a host-side rollback (accept ack, rebuild failed, corrective ack restored
/// the old mode) from looping request → rollback → request forever.
resize_requested: Option<(u32, u32)>,
/// The connector mode last shown in the HUD/title — a change (an accepted switch's
/// ack, or a corrective rollback) refreshes both.
shown_mode: Option<Mode>,
/// Resize-in-progress overlay (scrim + spinner) — armed by [`resize_tick`] when it
/// requests a switch, cleared when a decoded frame reaches the target (or on timeout).
resize_overlay: ResizeIndicator,
}
impl StreamState {
@@ -217,6 +248,11 @@ impl StreamState {
hw_fails: 0,
osd_text: String::new(),
last_stats: None,
resize_pending: None,
resize_sent_at: None,
resize_requested: None,
shown_mode: None,
resize_overlay: ResizeIndicator::default(),
}
}
@@ -270,7 +306,10 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
.register_custom_event::<FrameWake>()
.map_err(|e| anyhow::anyhow!("register FrameWake event: {e}"))?;
let mut window = {
let mut b = video.window(&opts.window_title, 1280, 720);
// Match-window (D1): open at the persisted last size, so the first connect's
// mode already matches the glass. 1280×720 stays the fallback/default.
let (ww, wh) = opts.window_size.unwrap_or((1280, 720));
let mut b = video.window(&opts.window_title, ww.max(320), wh.max(200));
match opts.window_pos {
Some((x, y)) => b.position(x, y),
None => b.position_centered(),
@@ -340,12 +379,15 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
let mut stream: Option<StreamState> = match &mut mode {
ModeCtl::Single(build) => {
let force_software = Arc::new(AtomicBool::new(false));
let params = build(
let mut params = build(
&gamepad,
native,
force_software.clone(),
presenter.vulkan_decode(),
);
if opts.match_window.is_some() {
apply_match_window(&mut params, &window);
}
Some(StreamState::new(
params,
force_software,
@@ -423,6 +465,14 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
WindowEvent::PixelSizeChanged(..) | WindowEvent::Resized(..) => {
presenter.recreate_swapchain(&window)?;
presenter.present(&window, FrameInput::Redraw, overlay_frame.as_ref())?;
// Match-window (D2): (re)stamp the debounce — the request fires
// once ~400 ms pass with no further size events, never per
// drag-frame (each accepted switch is a full host rebuild).
if opts.match_window.is_some() {
if let Some(st) = stream.as_mut() {
st.resize_pending = Some(Instant::now());
}
}
}
WindowEvent::Exposed => {
presenter.present(&window, FrameInput::Redraw, overlay_frame.as_ref())?;
@@ -616,7 +666,10 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
presenter.vulkan_decode(),
) {
ActionOutcome::Handled => {}
ActionOutcome::Start(params) => {
ActionOutcome::Start(mut params) => {
if opts.match_window.is_some() {
apply_match_window(&mut params, &window);
}
stream = Some(StreamState::new(
*params,
force_software,
@@ -753,6 +806,18 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
}
}
// --- Match-window (D2): debounced mode-follow + HUD/title refresh on a switch ----
if let Some(persist) = opts.match_window.as_mut() {
if let Some(st) = stream.as_mut() {
resize_tick(st, &mut window, &opts.window_title, persist.as_mut());
}
}
// Resize overlay timeout: a switch the host rejected/capped never delivers the exact
// target frame — drop the scrim so it can't linger. A no-op unless one is showing.
if let Some(st) = stream.as_mut() {
st.resize_overlay.tick(Instant::now());
}
// --- Console UI: damage-driven overlay re-render for this iteration --------------
if let Some(o) = overlay.as_mut() {
let (pw, ph) = window.size_in_pixels();
@@ -776,11 +841,15 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
};
let pad = gamepad.active();
let pads = gamepad.pads();
let resizing = stream
.as_ref()
.is_some_and(|st| st.connector.is_some() && st.resize_overlay.active());
let ctx = FrameCtx {
width: pw,
height: ph,
stats,
hint,
resizing,
pad: pad.as_ref().map(|p| p.name.as_str()),
pad_pref: pad.as_ref().map(|p| p.pref),
pads: &pads,
@@ -814,6 +883,10 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
newest = Some(f);
}
if let Some(f) = newest {
// Resize END: a frame at the steered target size means the sharp new-mode
// picture is here — lift the scrim. A no-op unless a switch is in flight.
let (fw, fh) = f.image.dimensions();
st.resize_overlay.decoded(fw, fh);
let DecodedFrame {
pts_ns,
decoded_ns,
@@ -991,12 +1064,15 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
}
}
// Browse with no video driving presents (library / connecting): composite the
// overlay every iteration — FIFO vsync-throttles this to the display rate.
if matches!(mode, ModeCtl::Browse(_))
&& !presented_video
&& stream.as_ref().is_none_or(|s| s.connector.is_none())
{
// Composite the overlay every iteration when no video frame drove a present but
// something on-screen still animates: browse-idle (library / connecting), OR a
// mid-stream resize scrim + spinner (the host's virtual-display + encoder rebuild
// leaves a gap with no frames — without this the spinner would freeze). FIFO
// vsync-throttles this to the display rate; the 15 ms wait keeps it smooth.
let resize_scrim = stream.as_ref().is_some_and(|s| s.resize_overlay.active());
let browse_idle = matches!(mode, ModeCtl::Browse(_))
&& stream.as_ref().is_none_or(|s| s.connector.is_none());
if !presented_video && (resize_scrim || browse_idle) {
presenter.present(&window, FrameInput::Redraw, overlay_frame.as_ref())?;
}
};
@@ -1013,6 +1089,195 @@ fn run_inner(mut opts: SessionOpts, mut mode: ModeCtl) -> Result<Option<Outcome>
Ok(outcome)
}
/// Match-window (D1): replace the params' requested w/h with the window's physical pixel
/// size — even-floored (the host's `validate_dimensions` rejects odd) and clamped to a
/// sane minimum — keeping the resolved refresh. Under `--fullscreen` the window IS the
/// display, so this degenerates to the display's native mode.
fn apply_match_window(params: &mut SessionParams, window: &sdl3::video::Window) {
let (pw, ph) = window.size_in_pixels();
params.mode.width = (pw & !1).max(320);
params.mode.height = (ph & !1).max(200);
tracing::info!(
w = params.mode.width,
h = params.mode.height,
"match-window: requesting the window's pixel size"
);
}
/// Match-window (D2) per-iteration tick: refresh the HUD line + window title when the
/// live mode moves (an accepted switch's ack, or a corrective rollback), then fire the
/// debounced `Reconfigure` once ~400 ms pass with no further resize events. The shared
/// trigger discipline:
/// * physical pixels, even-floored, clamped ≥ 320×200; the current refresh is kept;
/// * ≥ 1 s between requests (the accept ack round-trips in milliseconds — it precedes
/// the host's rebuild — so the spacing also keeps at most ~one request outstanding);
/// * each distinct size is requested at most ONCE (`resize_requested`): a rejected
/// size isn't re-asked until the window changes, and a host-side rollback (accepted,
/// rebuild failed, corrective ack restored the old mode) can't loop.
fn resize_tick(
st: &mut StreamState,
window: &mut sdl3::video::Window,
title_base: &str,
persist: &mut dyn FnMut(u32, u32),
) {
let Some(c) = &st.connector else {
return; // not connected yet — the pending stamp survives until we are
};
// HUD/title follow the live mode slot (updated by any accepted ack).
let m = c.mode();
if st.shown_mode.is_some_and(|prev| prev != m) {
st.mode_line = format!("{}×{}@{}", m.width, m.height, m.refresh_hz);
tracing::info!(mode = %st.mode_line, "stream mode switched");
let _ = window.set_title(&format!("{title_base} · {}", st.mode_line));
}
st.shown_mode = Some(m);
match resize_decision(
Instant::now(),
&mut st.resize_pending,
st.resize_sent_at,
st.resize_requested,
(m.width, m.height),
window.size_in_pixels(),
) {
ResizeAction::Wait => {}
ResizeAction::Settled(target) => {
// The debounce settled: persist the window's LOGICAL size for the next
// launch (its window is created in logical units) even when no request goes
// out (e.g. resized back to the streamed size).
let (lw, lh) = window.size();
persist(lw, lh);
let Some((w, h)) = target else { return };
tracing::info!(w, h, "window resized — requesting mode switch");
if c.request_mode(Mode {
width: w,
height: h,
refresh_hz: m.refresh_hz,
})
.is_err()
{
tracing::warn!("mode-switch request dropped — control channel closed");
}
st.resize_requested = Some((w, h));
st.resize_sent_at = Some(Instant::now());
// Show the scrim + spinner until a frame at this size lands (or the timeout):
// the live drag itself stays sharp; only the host's rebuild gap is covered.
st.resize_overlay.steering(w, h, Instant::now());
}
}
}
/// What one [`resize_decision`] tick decided.
#[derive(Debug, PartialEq, Eq)]
enum ResizeAction {
/// Nothing to do yet (no resize pending, still debouncing, or spacing defers — the
/// pending stamp is kept so a later tick retries).
Wait,
/// The debounce settled (pending cleared, the caller persists the window size), with
/// the mode to request — `None` when the size needs no switch (equal to the streamed
/// mode, or this exact size was already requested once).
Settled(Option<(u32, u32)>),
}
/// The D2 trigger discipline as a pure decision (unit-tested — CI can't open windows):
/// debounce to resize-end, ≥ 1 s between requests, physical pixels even-floored and
/// clamped ≥ 320×200, skip when equal to the streamed mode, and each distinct size
/// requested at most once (covers rejected sizes AND host-side rollbacks).
fn resize_decision(
now: Instant,
pending: &mut Option<Instant>,
sent_at: Option<Instant>,
requested: Option<(u32, u32)>,
current: (u32, u32),
pixel_size: (u32, u32),
) -> ResizeAction {
const DEBOUNCE: Duration = Duration::from_millis(400);
const SPACING: Duration = Duration::from_secs(1);
let Some(since) = *pending else {
return ResizeAction::Wait;
};
if now.duration_since(since) < DEBOUNCE {
return ResizeAction::Wait;
}
if sent_at.is_some_and(|at| now.duration_since(at) < SPACING) {
return ResizeAction::Wait; // keep the pending stamp — a later tick retries
}
*pending = None;
let target = ((pixel_size.0 & !1).max(320), (pixel_size.1 & !1).max(200));
if current == target || requested == Some(target) {
return ResizeAction::Settled(None);
}
ResizeAction::Settled(Some(target))
}
/// Resize-in-progress overlay state (design/midstream-resolution-resize.md — client UX),
/// ported from the Apple client's `ResizeIndicator`. A mid-stream Match-window switch takes
/// the host 0.32 s to rebuild its virtual display + encoder, and the first new-mode frame
/// is an IDR the decoder re-inits on. Rather than let the stream stretch to the changed
/// window during that gap, the presenter EMBRACES the delay: a deliberate scrim + spinner
/// the instant a switch is requested, cleared the instant the sharp new-resolution frame is
/// on screen — so the wait reads as intentional, not as lag.
///
/// Driven entirely by signals the presenter already has (no new protocol):
/// * START — [`resize_tick`] reports the size it just requested (`steering`).
/// * END — the decode pipeline reports each frame's dimensions; when they reach the
/// target the new picture is here (`decoded`). The accepted-switch ack alone can't
/// end it: the ack round-trips in milliseconds, ahead of the host's rebuild.
/// * TIMEOUT — the safety net for a switch that never delivers the exact target (a
/// gamescope reject, an advertised-mode cap, or a corrective ack landing a different
/// size); `tick` clears it after [`ResizeIndicator::TIMEOUT`].
///
/// Pure + clock-injected so the transition logic is unit-tested without a live session.
#[derive(Default)]
struct ResizeIndicator {
/// The size the follower is steering toward — cleared once a decoded frame reaches it.
/// `Some` ⇔ the scrim + spinner should be shown.
target: Option<(u32, u32)>,
/// When the current active span began — the timeout is measured from here.
since: Option<Instant>,
}
impl ResizeIndicator {
/// How long to keep the overlay up if the target frame never arrives.
const TIMEOUT: Duration = Duration::from_millis(2500);
/// Whether the scrim + spinner should be shown.
fn active(&self) -> bool {
self.target.is_some()
}
/// A switch to `w`×`h` was just requested — show the overlay now. The timeout re-arms
/// only when the target actually changes, so a drag that walks through several sizes
/// (each its own request) never trips the timeout mid-gesture.
fn steering(&mut self, w: u32, h: u32, now: Instant) {
if self.target != Some((w, h)) {
self.since = Some(now);
}
self.target = Some((w, h));
}
/// A decoded frame arrived at `w`×`h`. Clears the overlay once it matches the steered
/// target — the sharp new-resolution picture is on glass.
fn decoded(&mut self, w: u32, h: u32) {
if self.target == Some((w, h)) {
self.target = None;
self.since = None;
}
}
/// Timeout safety net: stop showing once [`TIMEOUT`](Self::TIMEOUT) has elapsed with no
/// matching frame (a rejected or host-capped switch never delivers the exact target).
fn tick(&mut self, now: Instant) {
if self
.since
.is_some_and(|s| now.duration_since(s) >= Self::TIMEOUT)
{
self.target = None;
self.since = None;
}
}
}
/// Apply the capture state to the window: pointer lock (relative mouse + hidden cursor)
/// and — on Windows — a keyboard grab, so system chords (Alt+Tab, the Windows key) reach
/// the host while captured instead of the local shell. SDL implements the grab there
@@ -1115,6 +1380,181 @@ fn stats_text(
mod tests {
use super::*;
#[test]
fn resize_decision_follows_the_d2_discipline() {
let t0 = Instant::now();
let ms = Duration::from_millis;
// No resize pending → nothing to do.
let mut pending = None;
assert_eq!(
resize_decision(t0, &mut pending, None, None, (1280, 720), (1000, 600)),
ResizeAction::Wait
);
// Still debouncing (a drag in progress) → wait, pending kept.
let mut pending = Some(t0);
assert_eq!(
resize_decision(
t0 + ms(399),
&mut pending,
None,
None,
(1280, 720),
(1000, 600)
),
ResizeAction::Wait
);
assert!(pending.is_some(), "pending survives the wait");
// Debounce settled → request the even-floored, clamped pixel size.
assert_eq!(
resize_decision(
t0 + ms(400),
&mut pending,
None,
None,
(1280, 720),
(1001, 601)
),
ResizeAction::Settled(Some((1000, 600))),
"odd pixels floor to even"
);
assert!(pending.is_none(), "pending consumed");
// Spacing: a request went out < 1 s ago → wait WITHOUT dropping the pending
// stamp, so a later tick retries.
let mut pending = Some(t0);
assert_eq!(
resize_decision(
t0 + ms(900),
&mut pending,
Some(t0),
Some((1000, 600)),
(1280, 720),
(800, 500)
),
ResizeAction::Wait
);
assert!(pending.is_some());
assert_eq!(
resize_decision(
t0 + ms(1000),
&mut pending,
Some(t0),
Some((1000, 600)),
(1280, 720),
(800, 500)
),
ResizeAction::Settled(Some((800, 500)))
);
// Equal to the streamed mode → settle (persist) but no request.
let mut pending = Some(t0);
assert_eq!(
resize_decision(
t0 + ms(400),
&mut pending,
None,
None,
(1280, 720),
(1280, 720)
),
ResizeAction::Settled(None)
);
// A size already requested once (rejected, or rolled back host-side) is never
// re-asked — no request → rollback → request loop.
let mut pending = Some(t0);
assert_eq!(
resize_decision(
t0 + ms(400),
&mut pending,
None,
Some((1000, 600)),
(1280, 720),
(1000, 600)
),
ResizeAction::Settled(None)
);
// Tiny windows clamp to the host's floor.
let mut pending = Some(t0);
assert_eq!(
resize_decision(
t0 + ms(400),
&mut pending,
None,
None,
(1280, 720),
(100, 80)
),
ResizeAction::Settled(Some((320, 200)))
);
}
#[test]
fn resize_indicator_shows_until_the_target_frame_or_timeout() {
let t0 = Instant::now();
let ms = Duration::from_millis;
// Idle at rest.
let mut ind = ResizeIndicator::default();
assert!(!ind.active());
// A requested switch shows the overlay immediately.
ind.steering(1000, 600, t0);
assert!(ind.active());
// A frame at a DIFFERENT size (a stale old-mode frame still draining) doesn't lift it.
ind.decoded(1280, 720);
assert!(ind.active(), "an off-target frame keeps the scrim up");
// The sharp new-resolution frame arrives → cleared.
ind.decoded(1000, 600);
assert!(!ind.active(), "the target frame lifts the scrim");
ind.tick(t0 + ms(10_000)); // a late tick after clearing is inert
assert!(!ind.active());
// A switch whose target frame never arrives (rejected / host-capped) times out.
let mut ind = ResizeIndicator::default();
ind.steering(1000, 600, t0);
ind.tick(t0 + ResizeIndicator::TIMEOUT - ms(1));
assert!(ind.active(), "still within the timeout window");
ind.tick(t0 + ResizeIndicator::TIMEOUT);
assert!(!ind.active(), "timeout lifts a switch that never delivered");
}
#[test]
fn resize_indicator_retargets_and_rearms_the_timeout_mid_drag() {
let t0 = Instant::now();
let ms = Duration::from_millis;
// A drag that walks through sizes (each a fresh request) re-arms the timeout, so a
// slow gesture never trips it: at t0 steer A, then near-timeout steer B, then a B
// frame lands well after A's timeout would have fired.
let mut ind = ResizeIndicator::default();
ind.steering(1000, 600, t0);
let near = t0 + ResizeIndicator::TIMEOUT - ms(1);
ind.steering(1200, 700, near); // new target → timeout re-armed from `near`
ind.tick(t0 + ResizeIndicator::TIMEOUT + ms(1)); // past A's window, within B's
assert!(
ind.active(),
"retarget re-armed the timeout — no mid-drag flicker"
);
// Re-steering the SAME size does NOT re-arm (so a repeated identical request can't
// hold the scrim open forever).
let mut ind = ResizeIndicator::default();
ind.steering(1000, 600, t0);
ind.steering(1000, 600, t0 + ms(500)); // same target, later — `since` unchanged
ind.tick(t0 + ResizeIndicator::TIMEOUT);
assert!(
!ind.active(),
"an unchanged target keeps the original timeout"
);
}
fn sample() -> (Stats, PresentedWindow) {
(
Stats {
+470
View File
@@ -526,6 +526,10 @@ pub struct PunktfunkConnection {
/// `inner.audio_channels`; `pcm` is a fixed-capacity reusable buffer the returned pointer
/// borrows until the next PCM call (same contract as `last_audio`).
audio_pcm: std::sync::Mutex<AudioPcmState>,
/// Backs the `data`/`len` pointer of the last `punktfunk_connection_next_clipboard` event
/// (a fetched payload, an offer's format list, or a fetch-request's MIME) —
/// borrow-until-next-call, same contract as `last`.
last_clip: std::sync::Mutex<Option<Vec<u8>>>,
}
/// Lazily-initialized in-core Opus decode state. A coupled-1-stream multistream decoder is
@@ -922,6 +926,14 @@ pub const PUNKTFUNK_CODEC_HEVC: u8 = 0x02;
/// Codec bit: AV1. (Mirrors `quic::CODEC_AV1`.)
pub const PUNKTFUNK_CODEC_AV1: u8 = 0x04;
/// Host-capability bit in [`punktfunk_connection_host_caps`]: the host applies gamepad-state
/// snapshots (a capable client sends full-state snapshots instead of per-transition events).
/// (Mirrors `quic::HOST_CAP_GAMEPAD_STATE`.)
pub const PUNKTFUNK_HOST_CAP_GAMEPAD_STATE: u8 = 0x01;
/// Host-capability bit in [`punktfunk_connection_host_caps`]: the host supports the shared
/// clipboard, so a client may offer the toggle. (Mirrors `quic::HOST_CAP_CLIPBOARD`.)
pub const PUNKTFUNK_HOST_CAP_CLIPBOARD: u8 = 0x02;
// Keep the ABI cap bits in lockstep with the wire constants (compile-time guard against drift).
#[cfg(feature = "quic")]
const _: () = {
@@ -931,6 +943,8 @@ const _: () = {
assert!(PUNKTFUNK_CODEC_H264 == crate::quic::CODEC_H264);
assert!(PUNKTFUNK_CODEC_HEVC == crate::quic::CODEC_HEVC);
assert!(PUNKTFUNK_CODEC_AV1 == crate::quic::CODEC_AV1);
assert!(PUNKTFUNK_HOST_CAP_GAMEPAD_STATE == crate::quic::HOST_CAP_GAMEPAD_STATE);
assert!(PUNKTFUNK_HOST_CAP_CLIPBOARD == crate::quic::HOST_CAP_CLIPBOARD);
};
// Keep the ABI gamepad constants in lockstep with the wire enum (compile-time guard against drift).
@@ -1395,6 +1409,7 @@ pub unsafe extern "C" fn punktfunk_connect_ex7(
last: std::sync::Mutex::new(None),
last_audio: std::sync::Mutex::new(None),
audio_pcm: std::sync::Mutex::new(AudioPcmState::default()),
last_clip: std::sync::Mutex::new(None),
}))
}
Err(_) => std::ptr::null_mut(),
@@ -2261,6 +2276,397 @@ pub unsafe extern "C" fn punktfunk_connection_gamepad(
})
}
// ============================================================================================
// Shared clipboard (design/clipboard-and-file-transfer.md §5.1). Additive, ABI v6. All poll/serve
// bytes ride the mTLS-pinned QUIC session; nothing here opens a new listener or port.
// ============================================================================================
/// [`PunktfunkClipEvent::kind`] — the host announced clipboard content is available
/// (`transfer_id` = the offer `seq`; `data`/`len` = a `\n`-separated `"<mime>\t<size_hint>"`
/// format list). Fetch it lazily (only on a local paste) via
/// [`punktfunk_connection_clipboard_fetch`].
pub const PUNKTFUNK_CLIP_REMOTE_OFFER: u8 = 1;
/// [`PunktfunkClipEvent::kind`] — host ack / policy / backend update (`enabled`/`policy`/`reason`
/// valid). Reflect it in the toggle UI.
pub const PUNKTFUNK_CLIP_STATE: u8 = 2;
/// [`PunktfunkClipEvent::kind`] — the host is pasting our offered data: answer with
/// [`punktfunk_connection_clipboard_serve`] (`transfer_id` = `req_id`; `seq`/`file_index` valid;
/// `data`/`len` = the requested MIME).
pub const PUNKTFUNK_CLIP_FETCH_REQUEST: u8 = 3;
/// [`PunktfunkClipEvent::kind`] — bytes for a fetch we started (`transfer_id` = `xfer_id`;
/// `data`/`len` = the payload, borrowed until the next `next_clipboard`; `last` = final chunk).
pub const PUNKTFUNK_CLIP_DATA: u8 = 4;
/// [`PunktfunkClipEvent::kind`] — a transfer was cancelled (`transfer_id` = the id).
pub const PUNKTFUNK_CLIP_CANCELLED: u8 = 5;
/// [`PunktfunkClipEvent::kind`] — a transfer failed (`transfer_id` = the id; `status` = a
/// `PunktfunkStatus` code).
pub const PUNKTFUNK_CLIP_ERROR: u8 = 6;
/// One advertised clipboard format passed to [`punktfunk_connection_clipboard_offer`].
#[cfg(feature = "quic")]
#[repr(C)]
pub struct PunktfunkClipKind {
/// NUL-terminated UTF-8 wire MIME (e.g. `text/plain;charset=utf-8`). ≤ 128 bytes on the wire.
pub mime: *const std::os::raw::c_char,
/// Best-effort size in bytes; `0` = unknown.
pub size_hint: u64,
}
/// A shared-clipboard event, filled by [`punktfunk_connection_next_clipboard`]. A flat tagged
/// struct (like `PunktfunkHidOutput`): read the fields named in the `kind`'s doc; the rest are 0.
#[cfg(feature = "quic")]
#[repr(C)]
#[derive(Clone, Copy)]
pub struct PunktfunkClipEvent {
/// One of `PUNKTFUNK_CLIP_*`.
pub kind: u8,
/// `State`: 1 = enabled, 0 = disabled.
pub enabled: u8,
/// `State`: bitfield of `quic::CLIP_POLICY_*` — what the host currently permits.
pub policy: u8,
/// `State`: one of `quic::CLIP_REASON_*`.
pub reason: u8,
/// `Data`: 1 = final chunk of this transfer.
pub last: u8,
/// Per-transfer id: the offer `seq` (RemoteOffer), the `req_id` (FetchRequest), or the
/// `xfer_id` (Data/Cancelled/Error).
pub transfer_id: u32,
/// `FetchRequest`: the offer `seq` the request is against.
pub seq: u32,
/// `FetchRequest`: file index, or `quic::CLIP_FILE_INDEX_NONE`.
pub file_index: u32,
/// `Error`: a `PunktfunkStatus` code (negative); 0 otherwise.
pub status: i32,
/// RemoteOffer/FetchRequest/Data: a pointer into a per-connection slot, valid until the next
/// `next_clipboard` call; NULL for the other kinds.
pub data: *const u8,
/// Byte length of `data` (0 when `data` is NULL).
pub len: usize,
}
/// Fill a [`PunktfunkClipEvent`] from a core event, parking any variable-length bytes in `slot`
/// (borrow-until-next-call) and pointing `data`/`len` at them.
#[cfg(feature = "quic")]
fn build_clip_event(
ev: crate::clipboard::ClipEventCore,
slot: &mut Option<Vec<u8>>,
) -> PunktfunkClipEvent {
use crate::clipboard::ClipEventCore as E;
let mut out = PunktfunkClipEvent {
kind: 0,
enabled: 0,
policy: 0,
reason: 0,
last: 0,
transfer_id: 0,
seq: 0,
file_index: 0,
status: 0,
data: std::ptr::null(),
len: 0,
};
*slot = None;
match ev {
E::RemoteOffer { seq, kinds } => {
out.kind = PUNKTFUNK_CLIP_REMOTE_OFFER;
out.transfer_id = seq;
let mut blob = String::new();
for k in &kinds {
blob.push_str(&k.mime);
blob.push('\t');
blob.push_str(&k.size_hint.to_string());
blob.push('\n');
}
*slot = Some(blob.into_bytes());
}
E::State {
enabled,
policy,
reason,
} => {
out.kind = PUNKTFUNK_CLIP_STATE;
out.enabled = enabled as u8;
out.policy = policy;
out.reason = reason;
}
E::FetchRequest {
req_id,
seq,
file_index,
mime,
} => {
out.kind = PUNKTFUNK_CLIP_FETCH_REQUEST;
out.transfer_id = req_id;
out.seq = seq;
out.file_index = file_index;
*slot = Some(mime.into_bytes());
}
E::Data {
xfer_id,
bytes,
last,
} => {
out.kind = PUNKTFUNK_CLIP_DATA;
out.transfer_id = xfer_id;
out.last = last as u8;
*slot = Some(bytes);
}
E::Cancelled { id } => {
out.kind = PUNKTFUNK_CLIP_CANCELLED;
out.transfer_id = id;
}
E::Error { id, code } => {
out.kind = PUNKTFUNK_CLIP_ERROR;
out.transfer_id = id;
out.status = code;
}
}
if let Some(v) = slot.as_ref() {
out.data = v.as_ptr();
out.len = v.len();
}
out
}
/// The host capability bitfield the session's `Welcome` carried — a bitfield of
/// `PUNKTFUNK_HOST_CAP_GAMEPAD_STATE` / `PUNKTFUNK_HOST_CAP_CLIPBOARD`. A client tests
/// `caps & PUNKTFUNK_HOST_CAP_CLIPBOARD` to decide whether to offer the shared-clipboard toggle.
/// Safe any time after connect.
///
/// # Safety
/// `c` is a valid connection handle; `caps` is writable (NULL is skipped).
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_host_caps(
c: *const PunktfunkConnection,
caps: *mut u8,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
unsafe {
if !caps.is_null() {
*caps = c.inner.host_caps();
}
}
PunktfunkStatus::Ok
})
}
/// Enable or disable the shared clipboard for this session (`design` §3.1). Opt-in: nothing is
/// announced or served until this is called with `enabled = true`. `flags` carries
/// `quic::CLIP_FLAG_FILES` (allow file transfer). The host replies with a `State` event.
///
/// # Safety
/// `c` is a valid connection handle.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_clipboard_control(
c: *const PunktfunkConnection,
enabled: bool,
flags: u8,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
match c.inner.clip_control(enabled, flags) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// Announce that the local clipboard changed — the lazy format-list offer. `seq` is a monotonic
/// per-sender counter (newest wins); `kinds`/`n` is the advertised formats (≤ 16). The bytes cross
/// only if the host later fetches.
///
/// # Safety
/// `c` is a valid connection handle; `kinds` points to `n` `PunktfunkClipKind`s (NULL allowed only
/// when `n == 0`), each `mime` a valid NUL-terminated UTF-8 string.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_clipboard_offer(
c: *const PunktfunkConnection,
seq: u32,
kinds: *const PunktfunkClipKind,
n: usize,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if kinds.is_null() && n != 0 {
return PunktfunkStatus::NullPointer;
}
let mut out = Vec::with_capacity(n);
if n != 0 {
let slice = unsafe { std::slice::from_raw_parts(kinds, n) };
for k in slice {
let mime = if k.mime.is_null() {
String::new()
} else {
match unsafe { std::ffi::CStr::from_ptr(k.mime) }.to_str() {
Ok(s) => s.to_string(),
Err(_) => return PunktfunkStatus::InvalidArg,
}
};
out.push(crate::quic::ClipKind {
mime,
size_hint: k.size_hint,
});
}
}
match c.inner.clip_offer(seq, out) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// Start pulling one format (`mime`) of the host's current offer `seq` — lazily, on a local paste.
/// `file_index` selects a file for a file transfer, or `quic::CLIP_FILE_INDEX_NONE` for a non-file
/// format. Writes the transfer id (echoed on the resulting `Data`/`Error`/`Cancelled` event) to
/// `xfer_id_out`.
///
/// # Safety
/// `c` is a valid connection handle; `mime` is a valid NUL-terminated UTF-8 string; `xfer_id_out`
/// is writable (NULL is skipped).
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_clipboard_fetch(
c: *const PunktfunkConnection,
seq: u32,
mime: *const std::os::raw::c_char,
file_index: u32,
xfer_id_out: *mut u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if mime.is_null() {
return PunktfunkStatus::NullPointer;
}
let mime = match unsafe { std::ffi::CStr::from_ptr(mime) }.to_str() {
Ok(s) => s.to_string(),
Err(_) => return PunktfunkStatus::InvalidArg,
};
match c.inner.clip_fetch(seq, mime, file_index) {
Ok(xfer_id) => {
unsafe {
if !xfer_id_out.is_null() {
*xfer_id_out = xfer_id;
}
}
PunktfunkStatus::Ok
}
Err(e) => e.status(),
}
})
}
/// Provide bytes answering a `FetchRequest` event (the host is pasting our offered data). Call
/// repeatedly to stream a large payload; `last = true` completes it. `data` may be NULL only when
/// `len == 0` (e.g. a final empty chunk). `punktfunk_connection_clipboard_cancel(req_id)` aborts.
///
/// # Safety
/// `c` is a valid connection handle; `data` points to `len` bytes (NULL allowed only when
/// `len == 0`).
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_clipboard_serve(
c: *const PunktfunkConnection,
req_id: u32,
data: *const u8,
len: usize,
last: bool,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if data.is_null() && len != 0 {
return PunktfunkStatus::NullPointer;
}
let bytes = if len == 0 {
Vec::new()
} else {
unsafe { std::slice::from_raw_parts(data, len) }.to_vec()
};
match c.inner.clip_serve(req_id, bytes, last) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// Cancel a clipboard transfer by id — either an outbound fetch (`xfer_id` from
/// [`punktfunk_connection_clipboard_fetch`]) or an inbound serve (`req_id` from a `FetchRequest`).
///
/// # Safety
/// `c` is a valid connection handle.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_clipboard_cancel(
c: *const PunktfunkConnection,
id: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
match c.inner.clip_cancel(id) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// Pull the next shared-clipboard event into `*out`. [`PunktfunkStatus::NoFrame`] on timeout,
/// [`PunktfunkStatus::Closed`] once the session ended. A native client drains this on its own
/// thread and drives the OS pasteboard from it. The `data`/`len` pointer (when non-NULL) borrows a
/// per-connection buffer valid until the next `next_clipboard` call on this handle.
///
/// # Safety
/// `c` is a valid connection handle; `out` is writable for one `PunktfunkClipEvent`.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_next_clipboard(
c: *mut PunktfunkConnection,
out: *mut PunktfunkClipEvent,
timeout_ms: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if out.is_null() {
return PunktfunkStatus::NullPointer;
}
match c
.inner
.next_clip(std::time::Duration::from_millis(timeout_ms as u64))
{
Ok(ev) => {
let mut slot = c.last_clip.lock().unwrap();
let out_ev = build_clip_event(ev, &mut slot);
unsafe { *out = out_ev };
PunktfunkStatus::Ok
}
Err(e) => e.status(),
}
})
}
/// The compositor backend the host actually resolved for this session (one of the
/// `PUNKTFUNK_COMPOSITOR_*` values; the `Welcome`'s echo of the [`punktfunk_connect_ex`]
/// preference). `PUNKTFUNK_COMPOSITOR_AUTO` = an older host that didn't say. Clients use it for
@@ -2405,6 +2811,70 @@ pub unsafe extern "C" fn punktfunk_connection_request_keyframe(
})
}
/// Ask the host to recover from loss by **reference-frame invalidation** rather than a full IDR:
/// report the range `[first_frame, last_frame]` of access units the client can no longer trust
/// (the first missing `frame_index` through the newest received). An RFI-capable host (AMD LTR /
/// NVENC) re-references a known-good picture before `first_frame` and emits a clean P-frame tagged
/// `USER_FLAG_RECOVERY_ANCHOR` — no 20-40x IDR spike; a host that can't RFI forces an IDR instead
/// (same effect as [`punktfunk_connection_request_keyframe`]). Non-blocking, fire-and-forget; the
/// recovered frame is the only ack, so THROTTLE it exactly like the keyframe request. Prefer this
/// over the keyframe request on loss so AMD/RFI hosts avoid the spike; keep the keyframe request as
/// the backstop for when the recovery frame itself is lost.
///
/// # Safety
/// `c` is a valid connection handle.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_request_rfi(
c: *const PunktfunkConnection,
first_frame: u32,
last_frame: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
match c.inner.request_rfi(first_frame, last_frame) {
Ok(()) => PunktfunkStatus::Ok,
Err(e) => e.status(),
}
})
}
/// Feed each received frame's `frame_index` (the [`PunktfunkFrame::frame_index`] field, in receive
/// order) so the client recovers from loss with a cheap reference-frame invalidation instead of a
/// full IDR. On a forward gap (a `frame_index` jump = the intervening frames were lost and the
/// following AUs reference a picture that never arrived) this fires a THROTTLED
/// [`punktfunk_connection_request_rfi`] for the lost range; an RFI-capable host (AMD LTR / NVENC)
/// then recovers with a clean P-frame instead of a 20-40x IDR spike. Call it for every received
/// frame — it is cheap and idempotent, and the [`punktfunk_connection_frames_dropped`]-driven
/// keyframe request stays the backstop. Writes whether a forward gap was detected this call to
/// `gap_out` (nullable — a client with a post-loss display freeze can use it to re-arm; most
/// clients pass NULL and ignore it).
///
/// # Safety
/// `c` is a valid connection handle; `gap_out` is writable or NULL.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_note_frame_index(
c: *const PunktfunkConnection,
frame_index: u32,
gap_out: *mut bool,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
let gap = c.inner.note_frame_index(frame_index);
if !gap_out.is_null() {
unsafe { *gap_out = gap };
}
PunktfunkStatus::Ok
})
}
/// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't
/// rebuild them). A video loop polls this and calls [`punktfunk_connection_request_keyframe`]
/// when it climbs — the correct loss trigger under the host's infinite GOP, where unrecoverable
+433 -8
View File
@@ -12,14 +12,16 @@
//! channel. All methods are safe to call from any single embedder thread.
use crate::abr::BitrateController;
use crate::clipboard::{ClipCommand, ClipEventCore};
use crate::config::{CompositorPref, GamepadPref, Mode, Role};
use crate::error::{PunktfunkError, Result};
use crate::input::InputEvent;
use crate::packet::FLAG_PROBE;
use crate::quic::{
accept_resync, endpoint, io, wall_clock_ns, window_loss_ppm, BitrateChanged, ClockEcho,
ClockResync, ColorInfo, HdrMeta, Hello, HidOutput, LossReport, ProbeRequest, ProbeResult,
Reconfigure, Reconfigured, RequestKeyframe, ResyncStep, RichInput, SetBitrate, Start, Welcome,
accept_resync, endpoint, io, wall_clock_ns, window_loss_ppm, BitrateChanged, ClipControl,
ClipKind, ClipOffer, ClipState, ClockEcho, ClockResync, ColorInfo, HdrMeta, Hello, HidOutput,
LossReport, ProbeRequest, ProbeResult, Reconfigure, Reconfigured, RequestKeyframe, ResyncStep,
RfiRequest, RichInput, SetBitrate, Start, Welcome,
};
use crate::session::{Frame, Session};
use crate::transport::UdpTransport;
@@ -49,6 +51,10 @@ enum CtrlRequest {
Mode(Mode),
Probe(ProbeRequest),
Keyframe,
/// Reference-frame-invalidation recovery: the client saw a `frame_index` gap and reports the
/// invalidation range so an RFI-capable host re-references a known-good picture instead of
/// forcing a full IDR. See [`RfiRequest`].
Rfi(RfiRequest),
Loss(LossReport),
/// Adaptive bitrate: ask the host to re-target its encoder (kbps). Sent by the pump's
/// [`BitrateController`] when the user's bitrate setting is Automatic.
@@ -57,6 +63,12 @@ enum CtrlRequest {
/// its report tick after the first no-op clock flush — the "the clock stepped under me"
/// signal; the control task also self-triggers one every [`CLOCK_RESYNC_INTERVAL`].
ClockResync,
/// Shared-clipboard enable/disable for this session (`design/clipboard-and-file-transfer.md`
/// §3.1). Idempotent; carries the file-permission flag.
ClipControl(ClipControl),
/// Announce that the local clipboard changed — the lazy format-list offer (bytes cross later on
/// a fetch stream). Symmetric message; the host may send one too.
ClipOffer(ClipOffer),
}
/// What the worker reports to [`NativeClient::connect`] once the handshake lands: the
@@ -88,6 +100,10 @@ struct Negotiated {
audio_channels: u8,
/// The single codec the host will emit (`quic::CODEC_*`).
codec: u8,
/// The host capability bitfield ([`Welcome::host_caps`]): [`crate::quic::HOST_CAP_GAMEPAD_STATE`],
/// [`crate::quic::HOST_CAP_CLIPBOARD`]. Exposed to the embedder via
/// [`NativeClient::host_caps`] so a native client greys out unsupported toggles.
host_caps: u8,
}
/// Accumulated state of an in-flight / finished speed test. The data-plane pump mirrors the
@@ -346,6 +362,12 @@ const HDR_META_QUEUE: usize = 8;
/// harmless, it's per-frame observability, not state.
const HOST_TIMING_QUEUE: usize = 512;
/// Clipboard event plane depth (offers, host acks, fetch-requests, fetched payloads). Clipboard
/// activity is human-paced and sparse; a small ring is ample. Overflow drops the newest event
/// (try_send), same discipline as the other planes — a dropped offer heals on the next copy, and
/// a dropped fetch-request makes the serving stream time out and reset cleanly.
const CLIP_EVENT_QUEUE: usize = 32;
/// One Opus packet from the host's audio datagram stream (48 kHz stereo, 5 ms frames).
#[derive(Clone, Debug)]
pub struct AudioPacket {
@@ -355,6 +377,83 @@ pub struct AudioPacket {
pub data: Vec<u8>,
}
/// At most one client→host RFI request per this window, so a burst of frame-index gaps (a
/// full-screen pan shedding shards) can't storm the control stream. Matches the shared Vulkan pump's
/// recovery-request throttle; the host coalesces further.
const RFI_THROTTLE: Duration = Duration::from_millis(100);
/// State for [`NativeClient::note_frame_index`] — the client-side loss-range detector shared by every
/// embedder (Android, the C-ABI Apple client, the Windows shell pump) so none re-derives the wrapping
/// frame-index arithmetic. `next_expected` is the `frame_index` expected next in receive order;
/// `last_req` throttles the RFI requests a gap fires.
#[derive(Default)]
struct RfiRecovery {
next_expected: Option<u32>,
last_req: Option<Instant>,
}
/// What a forward gap should ask the host for: a precise RFI for a recoverable range, a plain
/// keyframe for a range wider than any encoder's reference history
/// ([`crate::packet::RFI_MAX_RANGE`] — a seconds-long outage, or a phantom index jump such as an
/// old host's speed-test burst consuming video indexes), or nothing (contiguous / straggler /
/// throttled).
#[derive(Debug, PartialEq, Eq)]
enum RecoveryAsk {
None,
Rfi(u32, u32),
Keyframe,
}
impl RfiRecovery {
/// Pure decision behind [`NativeClient::note_frame_index`]: fold one received `frame_index` (in
/// receive order) observed at `now`, advancing the expectation and returning `(gap, ask)`.
/// `gap` is whether this frame revealed a forward gap (the embedder arms its post-loss display
/// freeze on it); `ask` is the (throttled) recovery request to fire — an RFI naming the exact
/// lost span, or a keyframe when the span exceeds [`crate::packet::RFI_MAX_RANGE`] (RFI is
/// hopeless there: no encoder holds references that old, and a huge jump is more likely a
/// resync — e.g. the first real AU after an old host's speed test — than a real loss). Split
/// out from the connection so the wrapping arithmetic + [`RFI_THROTTLE`] are unit-testable
/// without a live session (see the tests below).
fn observe(&mut self, frame_index: u32, now: Instant) -> (bool, RecoveryAsk) {
match self.next_expected {
Some(exp) => {
// Wrapping split at the half-space: a small positive delta is a forward gap
// (missing frames); a delta in the top half is a straggler behind us.
let ahead = frame_index.wrapping_sub(exp);
if ahead == 0 {
self.next_expected = Some(frame_index.wrapping_add(1)); // contiguous
(false, RecoveryAsk::None)
} else if ahead < u32::MAX / 2 {
// Forward gap: [exp, frame_index-1] lost. Advance past this frame so the same
// gap isn't re-detected, then fire a throttled recovery ask for the lost range.
self.next_expected = Some(frame_index.wrapping_add(1));
let send = self
.last_req
.is_none_or(|t| now.duration_since(t) >= RFI_THROTTLE);
if send {
self.last_req = Some(now);
}
let ask = if !send {
RecoveryAsk::None
} else if ahead > crate::packet::RFI_MAX_RANGE {
RecoveryAsk::Keyframe
} else {
RecoveryAsk::Rfi(exp, frame_index.wrapping_sub(1))
};
(true, ask)
} else {
// Straggler behind the delivery point — leave the expectation.
(false, RecoveryAsk::None)
}
}
None => {
self.next_expected = Some(frame_index.wrapping_add(1));
(false, RecoveryAsk::None)
}
}
}
}
pub struct NativeClient {
// Each plane's receiver sits behind its own mutex so `NativeClient` is `Sync` and Rust
// embedders can share one `Arc<NativeClient>` across their plane threads (the same
@@ -381,6 +480,19 @@ pub struct NativeClient {
/// Bounded ([`CTRL_QUEUE`]) — the requests are sparse; a full queue means the control task
/// is wedged/dead, and callers treat it like a closed session.
ctrl_tx: tokio::sync::mpsc::Sender<CtrlRequest>,
/// Inbound shared-clipboard events (remote offers, host acks, fetch-requests, fetched
/// payloads), drained by [`NativeClient::next_clip`] → the C ABI poll. Fed by the control task
/// (metadata) and the clipboard task (fetch data).
clip: Mutex<Receiver<ClipEventCore>>,
/// Outbound clipboard fetch/serve/cancel commands → the worker's clipboard task
/// ([`crate::clipboard::run`]). Unbounded like `input_tx`; the commands are sparse and each
/// carries at most one paste's bytes.
clip_cmd_tx: tokio::sync::mpsc::UnboundedSender<ClipCommand>,
/// Monotonic id for outbound fetches ([`NativeClient::clip_fetch`]); stays below
/// [`crate::clipboard::INBOUND_REQ_FLAG`] so it never collides with an inbound serve `req_id`.
next_xfer_id: AtomicU32,
/// The host capability bitfield ([`Welcome::host_caps`]) — see [`NativeClient::host_caps`].
pub host_caps: u8,
/// Speed-test accumulator, shared with the data-plane pump + control task.
probe: Arc<Mutex<ProbeState>>,
shutdown: Arc<AtomicBool>,
@@ -398,6 +510,10 @@ pub struct NativeClient {
/// for the client stats HUDs (the unified spec's per-window `FEC` counter — proof FEC is
/// earning its keep); readers window it by diffing successive reads.
fec_recovered: Arc<AtomicU64>,
/// Client-side RFI-on-loss detector state for [`note_frame_index`](Self::note_frame_index): the
/// next `frame_index` expected in receive order + the last RFI-request time (throttle). Lets every
/// embedder share one loss-range detector instead of re-deriving the wrapping frame arithmetic.
rfi: Mutex<RfiRecovery>,
/// Kernel ids of the client's latency-critical native threads: the internal data-plane pump
/// (UDP receive + FEC reassembly) plus any embedder plane threads registered via
/// [`NativeClient::register_hot_thread`]. The Android client feeds these to an ADPF hint session
@@ -580,6 +696,9 @@ impl NativeClient {
let (mic_tx, mic_rx) = tokio::sync::mpsc::channel::<(u32, u64, Vec<u8>)>(MIC_QUEUE);
let (rich_input_tx, rich_input_rx) = tokio::sync::mpsc::unbounded_channel::<RichInput>();
let (ctrl_tx, ctrl_rx) = tokio::sync::mpsc::channel::<CtrlRequest>(CTRL_QUEUE);
let (clip_event_tx, clip_event_rx) =
std::sync::mpsc::sync_channel::<ClipEventCore>(CLIP_EVENT_QUEUE);
let (clip_cmd_tx, clip_cmd_rx) = tokio::sync::mpsc::unbounded_channel::<ClipCommand>();
let (ready_tx, ready_rx) = std::sync::mpsc::channel::<Result<Negotiated>>();
let shutdown = Arc::new(AtomicBool::new(false));
let quit = Arc::new(AtomicBool::new(false));
@@ -645,6 +764,8 @@ impl NativeClient {
rich_input_rx,
ctrl_rx,
ctrl_tx: ctrl_tx_pump,
clip_event_tx,
clip_cmd_rx,
ready_tx,
shutdown: shutdown_w,
quit: quit_w,
@@ -678,12 +799,17 @@ impl NativeClient {
mic_tx,
rich_input_tx,
ctrl_tx,
clip: Mutex::new(clip_event_rx),
clip_cmd_tx,
next_xfer_id: AtomicU32::new(1),
host_caps: negotiated.host_caps,
probe,
shutdown,
quit,
worker: Some(worker),
frames_dropped,
fec_recovered,
rfi: Mutex::new(RfiRecovery::default()),
hot_tids,
clock_offset,
mode: mode_slot,
@@ -868,6 +994,62 @@ impl NativeClient {
.map_err(|_| PunktfunkError::Closed)
}
/// Ask the host to recover from loss by **reference-frame invalidation** rather than a full IDR:
/// the client reports the range `[first_frame, last_frame]` of access units it can no longer trust
/// (from the first missing `frame_index` through the newest received). An RFI-capable host
/// re-references a known-good picture before `first_frame` (AMD LTR / NVENC RFI) and emits a clean
/// P-frame tagged [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`]; a host that can't RFI forces an IDR
/// instead (same as [`request_keyframe`](Self::request_keyframe)). Non-blocking, fire-and-forget —
/// the recovered frame is the only ack; throttle it like the keyframe request. Prefer this over
/// `request_keyframe` on loss so AMD/RFI hosts avoid the IDR spike; the keyframe request remains
/// the backstop when the recovery frame itself is lost.
pub fn request_rfi(&self, first_frame: u32, last_frame: u32) -> Result<()> {
self.ctrl_tx
.try_send(CtrlRequest::Rfi(RfiRequest {
first_frame,
last_frame,
}))
.map_err(|_| PunktfunkError::Closed)
}
/// Feed each received AU's `frame_index` (in receive order) so the client recovers from loss with
/// a cheap reference-frame invalidation instead of always paying for a full IDR. On a **forward
/// gap** — a `frame_index` jump means the intervening frames were lost and the following AUs
/// reference a picture the decoder never got — this fires a **throttled**
/// [`request_rfi`](Self::request_rfi) for the lost range `[first_missing, frame_index-1]`. An
/// RFI-capable host (AMD LTR / NVENC) then re-references a known-good frame (a clean P-frame, no
/// 20-40x IDR spike); a host that can't RFI forces an IDR, same as the keyframe path.
///
/// Call it for EVERY received frame; it is cheap and idempotent, and the
/// [`frames_dropped`](Self::frames_dropped)-driven [`request_keyframe`](Self::request_keyframe)
/// loop stays the backstop for when the recovery frame itself is lost. Returns `true` when a
/// forward gap was detected on this call (whether or not the RFI was throttled), so a client with
/// a post-loss display freeze can (re-)arm it on the same signal.
///
/// This centralizes the loss-range detection so every embedder gets identical behavior. (The
/// in-process Vulkan session pump keeps its own copy because it gates a display freeze on the same
/// signal and shares one throttle across RFI + keyframe requests.)
pub fn note_frame_index(&self, frame_index: u32) -> bool {
// Decide (and update state) under the lock; fire the request after releasing it.
let (gap, ask) = self
.rfi
.lock()
.unwrap()
.observe(frame_index, Instant::now());
match ask {
RecoveryAsk::Rfi(first, last) => {
let _ = self.request_rfi(first, last);
}
// A gap wider than any encoder's reference history (RFI_MAX_RANGE) — a seconds-long
// outage or a phantom index jump: RFI can't repair it, resync on a keyframe instead.
RecoveryAsk::Keyframe => {
let _ = self.request_keyframe();
}
RecoveryAsk::None => {}
}
gap
}
/// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't
/// rebuild them). A video loop polls this and calls [`request_keyframe`](Self::request_keyframe)
/// when it increases — the correct loss trigger under infinite GOP, where unrecoverable loss
@@ -1097,6 +1279,83 @@ impl NativeClient {
self.input_tx.send(*ev).map_err(|_| PunktfunkError::Closed)
}
/// The host capability bitfield the [`Welcome`] carried ([`crate::quic::HOST_CAP_GAMEPAD_STATE`],
/// [`crate::quic::HOST_CAP_CLIPBOARD`]). A native client tests
/// `host_caps() & HOST_CAP_CLIPBOARD` to decide whether to offer the shared-clipboard toggle.
pub fn host_caps(&self) -> u8 {
self.host_caps
}
/// Enable or disable the shared clipboard for this session (`design/clipboard-and-file-transfer.md`
/// §3.1). Opt-in: nothing is announced or served until this crosses with `enabled = true`.
/// `flags` carries [`crate::quic::CLIP_FLAG_FILES`]. Non-blocking; the host replies with a
/// `State` event ([`NativeClient::next_clip`]).
pub fn clip_control(&self, enabled: bool, flags: u8) -> Result<()> {
self.ctrl_tx
.try_send(CtrlRequest::ClipControl(ClipControl { enabled, flags }))
.map_err(|_| PunktfunkError::Closed)
}
/// Announce that the local clipboard changed — the lazy format-list offer. `seq` is a
/// monotonic per-sender counter (newest wins); `kinds` is the advertised formats (≤
/// [`crate::quic::CLIP_MAX_KINDS`]). The bytes cross only if the host later fetches.
pub fn clip_offer(&self, seq: u32, kinds: Vec<ClipKind>) -> Result<()> {
self.ctrl_tx
.try_send(CtrlRequest::ClipOffer(ClipOffer { seq, kinds }))
.map_err(|_| PunktfunkError::Closed)
}
/// Start pulling one format (`mime`) of the host's current offer `seq` — lazily, when a local
/// app pastes. `file_index` selects a file for a file transfer, or
/// [`crate::quic::CLIP_FILE_INDEX_NONE`] for a non-file format. Returns the `xfer_id` echoed on
/// the resulting `Data` / `Error` / `Cancelled` event.
pub fn clip_fetch(&self, seq: u32, mime: String, file_index: u32) -> Result<u32> {
let xfer_id = self.next_xfer_id.fetch_add(1, Ordering::Relaxed);
// Stay in the low id space (inbound serve ids carry the high bit); wrap defensively.
let xfer_id = xfer_id & !crate::clipboard::INBOUND_REQ_FLAG;
self.clip_cmd_tx
.send(ClipCommand::Fetch {
xfer_id,
seq,
file_index,
mime,
})
.map_err(|_| PunktfunkError::Closed)?;
Ok(xfer_id)
}
/// Provide bytes answering a `FetchRequest` event (the host is pasting our offered data). Call
/// repeatedly to stream a large payload; `last = true` completes it. `clip_cancel(req_id)`
/// aborts instead.
pub fn clip_serve(&self, req_id: u32, bytes: Vec<u8>, last: bool) -> Result<()> {
self.clip_cmd_tx
.send(ClipCommand::Serve {
req_id,
bytes,
last,
})
.map_err(|_| PunktfunkError::Closed)
}
/// Cancel a clipboard transfer by id — either an outbound fetch (`xfer_id` from
/// [`NativeClient::clip_fetch`]) or an inbound serve (`req_id` from a `FetchRequest` event).
pub fn clip_cancel(&self, id: u32) -> Result<()> {
self.clip_cmd_tx
.send(ClipCommand::Cancel { id })
.map_err(|_| PunktfunkError::Closed)
}
/// Pull the next shared-clipboard event (remote offer, host ack/state, fetch-request, fetched
/// data, cancel, error); same timeout/closed semantics as [`NativeClient::next_hidout`]. A
/// native client drains this on its own thread and drives the OS pasteboard from it.
pub fn next_clip(&self, timeout: Duration) -> Result<ClipEventCore> {
match self.clip.lock().unwrap().recv_timeout(timeout) {
Ok(e) => Ok(e),
Err(RecvTimeoutError::Timeout) => Err(PunktfunkError::NoFrame),
Err(RecvTimeoutError::Disconnected) => Err(PunktfunkError::Closed),
}
}
/// Queue one Opus mic frame for delivery as a 0xCB uplink datagram (the inverse of
/// [`next_audio`](Self::next_audio)). `seq`/`pts_ns` are the caller's own counters (the host
/// uses them only for diagnostics). The host decodes it into a virtual microphone source.
@@ -1196,6 +1455,8 @@ struct WorkerArgs {
rich_input_rx: tokio::sync::mpsc::UnboundedReceiver<RichInput>,
ctrl_rx: tokio::sync::mpsc::Receiver<CtrlRequest>,
ctrl_tx: tokio::sync::mpsc::Sender<CtrlRequest>,
clip_event_tx: SyncSender<ClipEventCore>,
clip_cmd_rx: tokio::sync::mpsc::UnboundedReceiver<ClipCommand>,
ready_tx: std::sync::mpsc::Sender<Result<Negotiated>>,
shutdown: Arc<AtomicBool>,
/// Deliberate-quit flag (see [`NativeClient::quit`]): the worker closes with the quit code if set.
@@ -1239,6 +1500,8 @@ async fn worker_main(args: WorkerArgs) {
mut rich_input_rx,
mut ctrl_rx,
ctrl_tx,
clip_event_tx,
clip_cmd_rx,
ready_tx,
shutdown,
quit,
@@ -1297,7 +1560,12 @@ async fn worker_main(args: WorkerArgs) {
// when the matching bit is set, so `0` stays an 8-bit BT.709 stream. HOST_TIMING is
// OR'd in unconditionally: every NativeClient build demuxes the 0xCF plane, and the
// bit only asks the host for observability datagrams (never changes the encode).
video_caps: video_caps | crate::quic::VIDEO_CAP_HOST_TIMING,
// PROBE_SEQ likewise: the shared reassembler keeps probe filler in its own window
// (every embedder inherits it), so the host may burst speed tests without consuming
// video frame indexes.
video_caps: video_caps
| crate::quic::VIDEO_CAP_HOST_TIMING
| crate::quic::VIDEO_CAP_PROBE_SEQ,
// Requested surround channel count; the host echoes the resolved value in Welcome.
audio_channels,
// The codecs this client can decode + its soft preference (0 = auto). The host
@@ -1384,22 +1652,22 @@ async fn worker_main(args: WorkerArgs) {
chroma_format: welcome.chroma_format,
audio_channels: welcome.audio_channels,
codec: welcome.codec,
host_caps: welcome.host_caps,
},
welcome.host_caps,
))
};
let (conn, mut session, mut ctrl_send, mut ctrl_recv, negotiated, host_caps) = match setup.await
{
let (conn, mut session, mut ctrl_send, mut ctrl_recv, negotiated) = match setup.await {
Ok(t) => t,
Err(e) => {
let _ = ready_tx.send(Err(e));
return;
}
};
// Copies the pump needs after `negotiated` is handed over to `connect`.
// Copies the worker needs after `negotiated` is handed over to `connect`.
let clock_rtt_ns = negotiated.clock_rtt_ns;
let resolved_bitrate_kbps = negotiated.bitrate_kbps;
let host_caps = negotiated.host_caps;
// Seed the live offset with the connect-time estimate BEFORE the embedder can observe the
// client (ready_tx): clock_offset_now_ns() never reads a pre-handshake 0 on a skewed pair.
clock_offset.store(negotiated.clock_offset_ns, Ordering::Relaxed);
@@ -1491,6 +1759,9 @@ async fn worker_main(args: WorkerArgs) {
let bitrate_ack = bitrate_ack.clone();
let clock_offset = clock_offset.clone();
let clock_gen = clock_gen.clone();
// The control task feeds clipboard metadata events (ClipState/ClipOffer) onto the same event
// plane the clipboard task uses for fetch data; the original tx goes to that task below.
let clip_event_tx = clip_event_tx.clone();
tokio::spawn(async move {
// Mid-stream clock re-sync (see [`ClockResync`]): a batch runs every
// CLOCK_RESYNC_INTERVAL and whenever the pump asks (CtrlRequest::ClockResync after
@@ -1511,6 +1782,7 @@ async fn worker_main(args: WorkerArgs) {
CtrlRequest::Mode(m) => Reconfigure { mode: m }.encode(),
CtrlRequest::Probe(p) => p.encode(),
CtrlRequest::Keyframe => RequestKeyframe.encode(),
CtrlRequest::Rfi(r) => r.encode(),
CtrlRequest::Loss(r) => r.encode(),
CtrlRequest::SetBitrate(k) => SetBitrate { bitrate_kbps: k }.encode(),
CtrlRequest::ClockResync => {
@@ -1519,6 +1791,8 @@ async fn worker_main(args: WorkerArgs) {
}
resync.begin(wall_clock_ns()).encode()
}
CtrlRequest::ClipControl(c) => c.encode(),
CtrlRequest::ClipOffer(o) => o.encode(),
};
if io::write_msg(&mut ctrl_send, &bytes).await.is_err() {
break;
@@ -1600,6 +1874,21 @@ async fn worker_main(args: WorkerArgs) {
}
ResyncStep::Idle => {}
}
} else if let Ok(state) = ClipState::decode(&msg) {
// Host ack / policy / backend update for the toggle UI (try_send: a
// lagging embedder drops the newest — a stale toggle heals on the next).
let _ = clip_event_tx.try_send(ClipEventCore::State {
enabled: state.enabled,
policy: state.policy,
reason: state.reason,
});
} else if let Ok(offer) = ClipOffer::decode(&msg) {
// The host copied something: surface the lazy format list; the embedder
// fetches only if a local app pastes.
let _ = clip_event_tx.try_send(ClipEventCore::RemoteOffer {
seq: offer.seq,
kinds: offer.kinds,
});
} else {
tracing::warn!("unknown control message — ignoring");
}
@@ -1676,6 +1965,17 @@ async fn worker_main(args: WorkerArgs) {
}
});
// Clipboard task: the fetch-stream accept loop (host pulls what we offered) + outbound fetches
// (we pull what the host offered). Metadata (enable/offer/state) rides the control task above;
// only bulk bytes flow here. Dies with the connection (accept_bi errors) or when the embedder
// drops the command sender. Always spawned — a host without HOST_CAP_CLIPBOARD simply never
// opens a clip stream, and our control-plane offers hit its "unknown message" arm harmlessly.
tokio::spawn(crate::clipboard::run(
conn.clone(),
clip_event_tx,
clip_cmd_rx,
));
// Watch for connection close → stop the pump.
{
let shutdown = shutdown.clone();
@@ -1952,6 +2252,131 @@ mod host_port_tests {
}
}
#[cfg(test)]
mod rfi_recovery_tests {
//! The client-side loss-range detector shared by every embedder (Android, the C-ABI Apple
//! client, the Windows shell pump). `observe` is pure over `(frame_index, now)`, so the wrapping
//! frame arithmetic and the RFI throttle are exercised here without a live session.
use super::{RecoveryAsk, RfiRecovery, RFI_THROTTLE};
use std::time::{Duration, Instant};
// A fixed base instant; offsets model the throttle window deterministically (no sleeping).
fn base() -> Instant {
Instant::now()
}
#[test]
fn first_frame_arms_without_a_gap() {
let mut r = RfiRecovery::default();
// The opening frame only seeds the expectation — there is no prior frame to be missing.
assert_eq!(r.observe(100, base()), (false, RecoveryAsk::None));
assert_eq!(r.next_expected, Some(101));
}
#[test]
fn contiguous_frames_never_gap() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(100, t);
assert_eq!(r.observe(101, t), (false, RecoveryAsk::None));
assert_eq!(r.observe(102, t), (false, RecoveryAsk::None));
assert_eq!(r.observe(103, t), (false, RecoveryAsk::None));
assert_eq!(r.next_expected, Some(104));
}
#[test]
fn forward_gap_reports_the_exact_lost_range() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(100, t); // expecting 101 next
// 101..=104 were lost; 105 arrived. The RFI must name exactly the missing span.
assert_eq!(r.observe(105, t), (true, RecoveryAsk::Rfi(101, 104)));
// The expectation advances past the delivered frame so the same gap can't re-fire.
assert_eq!(r.next_expected, Some(106));
}
#[test]
fn single_frame_drop_names_a_unit_range() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(100, t);
// Exactly one frame (101) lost → range is the single index [101, 101].
assert_eq!(r.observe(102, t), (true, RecoveryAsk::Rfi(101, 101)));
}
#[test]
fn throttle_suppresses_bursts_then_re_opens() {
let mut r = RfiRecovery::default();
let t0 = base();
r.observe(100, t0);
// First gap fires the request and stamps the throttle.
assert_eq!(r.observe(105, t0), (true, RecoveryAsk::Rfi(101, 104)));
// A second gap 50 ms later is still a gap, but the request is throttled away.
assert_eq!(
r.observe(110, t0 + Duration::from_millis(50)),
(true, RecoveryAsk::None)
);
// Past the window, the request re-opens for the still-accurate lost span.
assert_eq!(
r.observe(120, t0 + RFI_THROTTLE + Duration::from_millis(1)),
(true, RecoveryAsk::Rfi(111, 119))
);
}
#[test]
fn stragglers_behind_the_delivery_point_are_ignored() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(100, t);
r.observe(105, t); // expecting 106 next
// A reordered late arrival (103, well behind 106) is neither a gap nor a request, and it
// must not rewind the expectation — otherwise the next in-order frame would false-gap.
assert_eq!(r.observe(103, t), (false, RecoveryAsk::None));
assert_eq!(r.next_expected, Some(106));
}
#[test]
fn wraparound_is_contiguous_across_u32_max() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(u32::MAX - 1, t); // expecting u32::MAX next
assert_eq!(r.observe(u32::MAX, t), (false, RecoveryAsk::None)); // contiguous, wraps to 0
assert_eq!(r.next_expected, Some(0));
assert_eq!(r.observe(0, t), (false, RecoveryAsk::None)); // still contiguous across the wrap
assert_eq!(r.next_expected, Some(1));
}
#[test]
fn gap_range_wraps_across_u32_max() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(u32::MAX - 1, t); // expecting u32::MAX next
// u32::MAX was lost and 1 arrived → the lost span wraps: [u32::MAX, 0].
assert_eq!(r.observe(1, t), (true, RecoveryAsk::Rfi(u32::MAX, 0)));
assert_eq!(r.next_expected, Some(2));
}
#[test]
fn huge_gap_resyncs_via_keyframe_not_rfi() {
let mut r = RfiRecovery::default();
let t = base();
r.observe(100, t); // expecting 101 next
// A jump wider than any encoder's reference history (RFI_MAX_RANGE): no valid
// reference exists for an RFI, and the jump may be a phantom (an old host's
// speed-test burst consuming video indexes) — ask for the IDR resync instead.
let jump = 100 + crate::packet::RFI_MAX_RANGE + 2;
assert_eq!(r.observe(jump, t), (true, RecoveryAsk::Keyframe));
// The expectation still advances past the delivered frame (no re-fire on the next one).
assert_eq!(r.next_expected, Some(jump + 1));
assert_eq!(r.observe(jump + 1, t), (false, RecoveryAsk::None));
// A huge gap consumes the shared throttle too — an immediate follow-up gap stays quiet.
assert_eq!(
r.observe(jump + 10, t + Duration::from_millis(1)),
(true, RecoveryAsk::None)
);
}
}
#[cfg(test)]
mod frame_channel_tests {
use super::{FrameChannel, FramePop, FRAME_QUEUE_HARD_CAP};
+304
View File
@@ -0,0 +1,304 @@
//! Client-side shared-clipboard transport (`design/clipboard-and-file-transfer.md` §5.1).
//!
//! This is the client-core half of Phase 0: the per-session async task that runs the fetch-stream
//! **accept loop** (so the host can pull data the client offered) and drives **outbound fetches**
//! (so the client can pull what the host offered), surfacing everything to the embedder as
//! poll-style [`ClipEventCore`] events. The metadata plane — enabling sync and announcing offers —
//! rides the control stream as ordinary [`ClipControl`]/[`ClipOffer`] control messages
//! ([`crate::client`] routes those); only the bulk bytes flow through here, over the
//! [`crate::quic::clipstream`] fetch bi-streams.
//!
//! There is intentionally **no OS pasteboard code here** — that lives in the native client (macOS
//! first). The event/command seam is what the C ABI ([`crate::abi`]) exposes so a native client
//! polls offers/fetch-requests and answers with bytes.
use std::collections::HashMap;
use std::sync::mpsc::SyncSender;
use std::sync::{Arc, Mutex};
use tokio::sync::mpsc::UnboundedReceiver;
use tokio::sync::oneshot;
use crate::error::PunktfunkStatus;
use crate::quic::{
clipstream, ClipFetch, ClipFetchHdr, ClipKind, CLIP_FETCH_DENIED, CLIP_FETCH_OK,
CLIP_FETCH_STALE, CLIP_FETCH_UNAVAILABLE,
};
/// Per-fetch requester-side size cap (bytes). A holder that streams more than this is treated as a
/// cap breach and the fetch fails rather than buffering unboundedly (§7). Phase 0 uses one fixed
/// value; a future host-policy `PUNKTFUNK_CLIP_MAX_MB` tightens it per session.
pub const CLIP_FETCH_CAP: usize = 64 << 20;
/// Inbound-serve `req_id`s carry this high bit so they never collide with the client-assigned
/// outbound-fetch `xfer_id`s (which count up from 1). A single [`ClipCommand::Cancel`] `id` can
/// then be routed to the right table.
pub const INBOUND_REQ_FLAG: u32 = 0x8000_0000;
/// Overall stall bound for one outbound fetch (`design/clipboard-and-file-transfer.md` §3.4): a
/// holder that never answers fails the transfer instead of hanging it.
const FETCH_STALL_SECS: u64 = 60;
/// A clipboard event surfaced to the embedder via `NativeClient::next_clip` (→ the C ABI poll
/// `punktfunk_connection_next_clipboard`).
#[derive(Clone, Debug)]
pub enum ClipEventCore {
/// The host announced new clipboard content (the host copied). The embedder decides whether to
/// fetch it — lazily, only when a local app actually pastes.
RemoteOffer { seq: u32, kinds: Vec<ClipKind> },
/// Host ack / unsolicited policy or backend update, for the toggle UI.
State {
enabled: bool,
policy: u8,
reason: u8,
},
/// The host is pasting content the client offered: it opened a fetch stream for
/// `(mime, file_index)`. The embedder must answer with `clip_serve(req_id, …)` (or
/// `clip_cancel(req_id)`).
FetchRequest {
req_id: u32,
seq: u32,
file_index: u32,
mime: String,
},
/// Bytes for a fetch the embedder started (`xfer_id` from `clip_fetch`). Phase 0 delivers the
/// whole payload in one event (`last = true`).
Data {
xfer_id: u32,
bytes: Vec<u8>,
last: bool,
},
/// A transfer was cancelled (by either side).
Cancelled { id: u32 },
/// A transfer failed; `code` is a [`PunktfunkStatus`] value (negative).
Error { id: u32, code: i32 },
}
/// A command from the embedder (via the C ABI) into the clipboard task. `ClipControl`/`ClipOffer`
/// are *not* here — they ride the control stream as ordinary control messages.
pub enum ClipCommand {
/// Open a fetch of the remote's offered content; `xfer_id` is client-assigned and echoed back
/// on the resulting [`ClipEventCore::Data`]/`Error`/`Cancelled`.
Fetch {
xfer_id: u32,
seq: u32,
file_index: u32,
mime: String,
},
/// Provide bytes answering an inbound [`ClipEventCore::FetchRequest`] (`req_id`). Chunks
/// accumulate; `last` completes the transfer.
Serve {
req_id: u32,
bytes: Vec<u8>,
last: bool,
},
/// Cancel a transfer by id — either an outbound fetch (`xfer_id`) or an inbound serve
/// (`req_id`, high bit set).
Cancel { id: u32 },
}
type ServeWaiters = Arc<Mutex<HashMap<u32, oneshot::Sender<Option<Vec<u8>>>>>>;
/// Map a non-OK [`ClipFetchHdr::status`] to the [`PunktfunkStatus`] surfaced on an
/// [`ClipEventCore::Error`].
fn fetch_status_to_code(status: u8) -> i32 {
let s = match status {
CLIP_FETCH_STALE => PunktfunkStatus::NoFrame, // stale offer → "nothing to insert"
CLIP_FETCH_UNAVAILABLE => PunktfunkStatus::Unsupported,
CLIP_FETCH_DENIED => PunktfunkStatus::InvalidArg,
_ => PunktfunkStatus::BadPacket,
};
s as i32
}
/// The per-session clipboard task. Runs until the connection closes or the embedder drops the
/// command sender. It owns no clipboard *content* — the embedder supplies bytes on demand.
pub async fn run(
conn: quinn::Connection,
events: SyncSender<ClipEventCore>,
mut cmd_rx: UnboundedReceiver<ClipCommand>,
) {
// Inbound fetch-serve waiters: req_id -> a oneshot the serving stream task parks on until the
// embedder answers with bytes (`Some`) or denies/cancels (`None`).
let serve_waiters: ServeWaiters = Arc::new(Mutex::new(HashMap::new()));
// Accumulation buffers for chunked `clip_serve` (req_id -> bytes so far).
let mut serve_bufs: HashMap<u32, Vec<u8>> = HashMap::new();
// Outbound fetch cancel triggers: xfer_id -> a oneshot that aborts the fetch task.
let mut fetch_cancels: HashMap<u32, oneshot::Sender<()>> = HashMap::new();
let mut next_req_id: u32 = 1;
loop {
tokio::select! {
// The host opened a fetch bi-stream toward us (it is pasting our offered data).
accepted = conn.accept_bi() => {
let Ok((send, recv)) = accepted else { break }; // connection gone
let req_id = INBOUND_REQ_FLAG | next_req_id;
next_req_id = next_req_id.wrapping_add(1);
if next_req_id == 0 {
next_req_id = 1;
}
let events = events.clone();
let waiters = serve_waiters.clone();
tokio::spawn(serve_inbound(send, recv, req_id, events, waiters));
}
cmd = cmd_rx.recv() => {
let Some(cmd) = cmd else { break }; // NativeClient dropped
match cmd {
ClipCommand::Fetch { xfer_id, seq, file_index, mime } => {
let (cancel_tx, cancel_rx) = oneshot::channel();
fetch_cancels.insert(xfer_id, cancel_tx);
let conn = conn.clone();
let events = events.clone();
let req = ClipFetch { seq, file_index, mime };
tokio::spawn(run_outbound_fetch(conn, xfer_id, req, events, cancel_rx));
}
ClipCommand::Serve { req_id, bytes, last } => {
serve_bufs.entry(req_id).or_default().extend_from_slice(&bytes);
if last {
let full = serve_bufs.remove(&req_id).unwrap_or_default();
if let Some(tx) = serve_waiters.lock().unwrap().remove(&req_id) {
let _ = tx.send(Some(full));
}
}
}
ClipCommand::Cancel { id } => {
// Route to whichever table owns the id (they are disjoint by the high bit).
if let Some(tx) = fetch_cancels.remove(&id) {
let _ = tx.send(());
}
serve_bufs.remove(&id);
if let Some(tx) = serve_waiters.lock().unwrap().remove(&id) {
let _ = tx.send(None); // deny — the serving task writes UNAVAILABLE
}
}
}
}
}
}
}
/// Serve one inbound fetch stream: validate the header + request, emit a [`ClipEventCore::FetchRequest`],
/// then park until the embedder supplies bytes (or denies), and stream them back.
async fn serve_inbound(
mut send: quinn::SendStream,
mut recv: quinn::RecvStream,
req_id: u32,
events: SyncSender<ClipEventCore>,
waiters: ServeWaiters,
) {
let _ = send.set_priority(-1);
let kind = match clipstream::read_stream_header(&mut recv).await {
Ok(k) => k,
Err(_) => return,
};
if kind != clipstream::CLIP_STREAM_KIND_FETCH {
let _ = send.reset(clipstream::cancelled_code());
return;
}
let req = match clipstream::read_fetch(&mut recv).await {
Ok(r) => r,
Err(_) => return,
};
// Register the waiter before emitting the event, so an immediate `clip_serve` can't race ahead
// of the insert.
let (tx, rx) = oneshot::channel();
waiters.lock().unwrap().insert(req_id, tx);
let ev = ClipEventCore::FetchRequest {
req_id,
seq: req.seq,
file_index: req.file_index,
mime: req.mime,
};
if events.try_send(ev).is_err() {
// The embedder isn't draining events (or the session is ending): refuse cleanly.
waiters.lock().unwrap().remove(&req_id);
let _ = clipstream::write_fetch_hdr(
&mut send,
&ClipFetchHdr {
status: CLIP_FETCH_UNAVAILABLE,
total_size: 0,
},
)
.await;
return;
}
match rx.await {
Ok(Some(bytes)) => {
if clipstream::write_fetch_hdr(
&mut send,
&ClipFetchHdr {
status: CLIP_FETCH_OK,
total_size: bytes.len() as u64,
},
)
.await
.is_ok()
{
let _ = clipstream::write_data(&mut send, &bytes).await;
}
}
// Denied, cancelled, or the waiter was dropped (task ending): tell the peer it's gone.
_ => {
let _ = clipstream::write_fetch_hdr(
&mut send,
&ClipFetchHdr {
status: CLIP_FETCH_UNAVAILABLE,
total_size: 0,
},
)
.await;
}
}
}
/// Drive one outbound fetch: open the stream, read the header + data (bounded), and emit a
/// [`ClipEventCore::Data`] / `Error` / `Cancelled`.
async fn run_outbound_fetch(
conn: quinn::Connection,
xfer_id: u32,
req: ClipFetch,
events: SyncSender<ClipEventCore>,
cancel_rx: oneshot::Receiver<()>,
) {
let transfer = async {
let (send, mut recv) = clipstream::open_fetch(&conn, &req)
.await
.map_err(|_| PunktfunkStatus::Io as i32)?;
let hdr = clipstream::read_fetch_hdr(&mut recv)
.await
.map_err(|_| PunktfunkStatus::Io as i32)?;
if hdr.status != CLIP_FETCH_OK {
return Err(fetch_status_to_code(hdr.status));
}
let data = clipstream::read_data(&mut recv, CLIP_FETCH_CAP)
.await
.map_err(|_| PunktfunkStatus::Io as i32)?;
drop(send); // done — dropping the send half is a clean FIN-less close on our side
Ok(data)
};
tokio::select! {
r = transfer => match r {
Ok(data) => {
let _ = events.try_send(ClipEventCore::Data { xfer_id, bytes: data, last: true });
}
Err(code) => {
let _ = events.try_send(ClipEventCore::Error { id: xfer_id, code });
}
},
_ = cancel_rx => {
// The `transfer` future is dropped here; its streams reset on drop.
let _ = events.try_send(ClipEventCore::Cancelled { id: xfer_id });
}
// Overall stall bound (design/clipboard-and-file-transfer.md §3.4): a holder that never
// answers must not hang the transfer. Dropping `transfer` resets the streams.
_ = tokio::time::sleep(std::time::Duration::from_secs(FETCH_STALL_SECS)) => {
let _ = events.try_send(ClipEventCore::Error {
id: xfer_id,
code: PunktfunkStatus::Timeout as i32,
});
}
}
}
+10 -1
View File
@@ -30,6 +30,11 @@ mod abr;
pub mod audio;
#[cfg(feature = "quic")]
pub mod client;
/// Client-side shared-clipboard transport: the per-session task that runs the fetch-stream accept
/// loop, drives outbound fetches, and serves inbound ones — surfaced to the embedder as poll
/// events. Wire codecs live in [`quic`]; the OS pasteboard integration lives in the native client.
#[cfg(feature = "quic")]
pub mod clipboard;
pub mod config;
pub mod crypto;
pub mod error;
@@ -61,7 +66,11 @@ pub use stats::Stats;
/// TTL of a v2 envelope; `punktfunk_connection_next_rumble` is unchanged and drops it). Additive —
/// the wire is backward-compatible (the envelope is a length-tolerant tail on 0xCA), so
/// [`WIRE_VERSION`] is unchanged.
pub const ABI_VERSION: u32 = 5;
/// v6: added the shared-clipboard client surface — `punktfunk_connection_host_caps` and
/// `punktfunk_connection_clipboard_{control,offer,fetch,serve,cancel}` +
/// `punktfunk_connection_next_clipboard`. Additive; the wire grows only backward-compatible control
/// messages (0x40-0x44) and a new `Welcome::host_caps` bit, so [`WIRE_VERSION`] is unchanged.
pub const ABI_VERSION: u32 = 6;
/// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
/// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
+240 -33
View File
@@ -35,6 +35,34 @@ pub const FLAG_SOF: u8 = 0x4;
/// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it).
pub const FLAG_PROBE: u8 = 0x8;
/// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client
/// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that
/// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is
/// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its
/// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible
/// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host
/// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four.
pub const USER_FLAG_RECOVERY_POINT: u32 = 0x10;
/// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike
/// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss
/// is only half-healed so the client waits for the second), this marks an access unit the host coded
/// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation**
/// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client
/// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts
/// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets
/// `AV_FRAME_FLAG_KEY` — this host flag is the only signal.
pub const USER_FLAG_RECOVERY_ANCHOR: u32 = 0x20;
/// Widest lost-frame range (frames, wrapping `last - first`) a reference-frame-invalidation
/// recovery may be asked to repair; anything wider goes straight to the keyframe path on BOTH
/// ends. RFI can only re-reference history the encoder still holds — NVENC keeps a 5-frame DPB,
/// AMD LTR ~1 s of marks — and a genuine loss this wide (>1 s even at 240 fps) has no valid
/// reference anywhere, so an RFI request for it is either hopeless or (worse) a phantom range
/// from a desynced counter. Shared by the host's RFI dispatch (range → keyframe fallback) and the
/// client-side gap detectors (huge gap → resync + keyframe request, no RFI).
pub const RFI_MAX_RANGE: u32 = 256;
/// Crypto framing overhead [`Session`](crate::session::Session) adds when encrypting:
/// an 8-byte sequence prefix plus the GCM tag.
pub const CRYPTO_OVERHEAD: usize = 8 + crate::crypto::TAG_LEN;
@@ -98,8 +126,18 @@ const _: () = assert!(HEADER_LEN == 40, "PacketHeader must be 40 bytes / unpadde
// ---------------------------------------------------------------------------
/// Splits encoded access units into FEC-protected shard packets. Host-side only.
///
/// Frame numbering: a caller can pass an **explicit** `frame_index` to
/// [`packetize_each`](Self::packetize_each) (the punktfunk/1 encode loop owns the video numbering
/// so the encoder's reference-frame-invalidation bookkeeping stays 1:1 with the wire across
/// encoder rebuilds/resets), or pass `None` to draw from the internal counter (the legacy path —
/// synthetic/spike/ABI sessions where no encoder cares). Speed-test probe filler draws from a
/// **separate** index space ([`alloc_probe_index`](Self::alloc_probe_index)) so a burst never
/// consumes video indexes — see [`crate::quic::VIDEO_CAP_PROBE_SEQ`].
pub struct Packetizer {
next_frame_index: u32,
/// Probe-space frame counter (see [`alloc_probe_index`](Self::alloc_probe_index)).
next_probe_index: u32,
next_seq: u32,
shard_payload: usize,
fec: crate::config::FecConfig,
@@ -115,6 +153,7 @@ impl Packetizer {
pub fn new(config: &Config) -> Self {
Packetizer {
next_frame_index: 0,
next_probe_index: 0,
next_seq: 0,
shard_payload: config.shard_payload,
fec: config.fec,
@@ -123,6 +162,17 @@ impl Packetizer {
}
}
/// Allocate the next **probe-space** frame index (speed-test filler). A separate counter from
/// the video `frame_index`es so a multi-thousand-AU probe burst never advances the video
/// numbering — the client routes [`FLAG_PROBE`]-flagged shards into its own reassembly window
/// (see [`Reassembler`]), so the two spaces never collide. Only used against clients that
/// advertise [`crate::quic::VIDEO_CAP_PROBE_SEQ`].
pub fn alloc_probe_index(&mut self) -> u32 {
let i = self.next_probe_index;
self.next_probe_index = i.wrapping_add(1);
i
}
/// Live-adjust the FEC recovery percentage (adaptive FEC). Takes effect on the next
/// [`packetize`](Self::packetize); the wire is self-describing (each packet carries its block's
/// data/recovery counts), so the receiver needs no notification. Clamped to ≤ 90.
@@ -146,7 +196,7 @@ impl Packetizer {
coder: &dyn ErasureCoder,
) -> Result<Vec<Vec<u8>>> {
let mut packets = Vec::new();
self.packetize_each(frame, pts_ns, user_flags, coder, |hdr, body| {
self.packetize_each(frame, pts_ns, user_flags, None, coder, |hdr, body| {
let mut pkt = Vec::with_capacity(HEADER_LEN + body.len());
pkt.extend_from_slice(hdr.as_bytes());
pkt.extend_from_slice(body);
@@ -162,17 +212,27 @@ impl Packetizer {
/// shard straight into a pooled wire buffer and seal in place
/// ([`Session::seal_frame`](crate::session::Session::seal_frame)). An `emit` error aborts
/// the frame mid-way (packet numbering has already advanced — callers treat it as fatal).
///
/// `frame_index`: `Some(i)` stamps the AU with the caller's index — the punktfunk/1 encode
/// loop numbers video AUs itself so the encoder's RFI bookkeeping (LTR marks, DPB timestamps)
/// is 1:1 with what the client sees, surviving encoder rebuilds/resets that restart internal
/// counters. `None` draws from the internal counter (the legacy/self-numbering path). A
/// session must not mix the two styles for the same index space.
pub fn packetize_each(
&mut self,
frame: &[u8],
pts_ns: u64,
user_flags: u32,
frame_index: Option<u32>,
coder: &dyn ErasureCoder,
mut emit: impl FnMut(&PacketHeader, &[u8]) -> Result<()>,
) -> Result<()> {
let payload = self.shard_payload;
let frame_index = self.next_frame_index;
self.next_frame_index = self.next_frame_index.wrapping_add(1);
let frame_index = frame_index.unwrap_or_else(|| {
let i = self.next_frame_index;
self.next_frame_index = i.wrapping_add(1);
i
});
// At least one (zero-padded) data shard even for an empty frame.
let total_data = frame.len().div_ceil(payload).max(1);
@@ -324,10 +384,13 @@ impl ReassemblerLimits {
}
}
/// Buffers incoming shards, recovers lost ones via FEC, and emits whole access units.
/// Client-side only.
pub struct Reassembler {
limits: ReassemblerLimits,
/// One frame-index space's reassembly state: the in-flight frames, the recently-emitted memory,
/// and the loss-window anchor. The [`Reassembler`] keeps two — video and speed-test probe filler —
/// because the two ride **separate index counters** on a [`VIDEO_CAP_PROBE_SEQ`]-aware host
/// (a probe burst must neither advance the video loss window nor be dropped as "stale" against
/// it). [`VIDEO_CAP_PROBE_SEQ`]: crate::quic::VIDEO_CAP_PROBE_SEQ
#[derive(Default)]
struct ReassemblyWindow {
frames: HashMap<u32, FrameBuf>,
/// Recently-emitted frames, so stray/late shards can't resurrect them. Pruned to
/// the reorder window alongside `frames`.
@@ -338,13 +401,27 @@ pub struct Reassembler {
newest_frame: Option<(u32, u64)>,
}
/// Buffers incoming shards, recovers lost ones via FEC, and emits whole access units.
/// Client-side only.
pub struct Reassembler {
limits: ReassemblerLimits,
/// The video stream's window — its aged-out incomplete frames count into `frames_dropped`
/// (the client's loss-recovery trigger).
video: ReassemblyWindow,
/// Speed-test probe filler ([`FLAG_PROBE`] in `user_flags`). Routed by the flag, so it also
/// captures an OLD host's probe frames (which still carry video-space indexes — they complete
/// fine here, and keeping them out of the video window means a burst can no longer advance the
/// video loss anchor). Aged-out probe frames are NOT `frames_dropped` — probe loss is measured
/// bytes-wise by the probe accumulator and must not fire video recovery.
probe: ReassemblyWindow,
}
impl Reassembler {
pub fn new(limits: ReassemblerLimits) -> Self {
Reassembler {
limits,
frames: HashMap::new(),
completed: HashSet::new(),
newest_frame: None,
video: ReassemblyWindow::default(),
probe: ReassemblyWindow::default(),
}
}
@@ -405,18 +482,28 @@ impl Reassembler {
}
let payload = pkt[HEADER_LEN..HEADER_LEN + shard_bytes].to_vec();
self.advance_window(hdr.frame_index, hdr.pts_ns, stats);
// Route by index space: speed-test probe filler (FLAG_PROBE in user_flags) reassembles in
// its own window so its indexes never interact with the video loss window — a probe burst
// can neither advance the video anchor nor be dropped as stale against it (and its aged-out
// frames never count as `frames_dropped`, which would fire video loss recovery).
let is_probe = hdr.user_flags & (FLAG_PROBE as u32) != 0;
let win = if is_probe {
&mut self.probe
} else {
&mut self.video
};
win.advance_window(hdr.frame_index, hdr.pts_ns, stats, !is_probe);
// Drop shards for frames we've already emitted (e.g. the recovery shards of a
// frame that completed early via the all-originals-present fast path) or that
// have fallen out of the loss window.
if self.completed.contains(&hdr.frame_index) || self.is_stale(hdr.frame_index, hdr.pts_ns) {
if win.completed.contains(&hdr.frame_index) || win.is_stale(hdr.frame_index, hdr.pts_ns) {
drop(stats);
return Ok(None);
}
// First packet of a frame establishes its geometry; later packets must agree.
let frame = self
let frame = win
.frames
.entry(hdr.frame_index)
.or_insert_with(|| FrameBuf {
@@ -502,8 +589,8 @@ impl Reassembler {
// Whole frame ready?
if frame.block_data.len() == frame.block_count {
let frame = self.frames.remove(&hdr.frame_index).unwrap();
self.completed.insert(hdr.frame_index);
let frame = win.frames.remove(&hdr.frame_index).unwrap();
win.completed.insert(hdr.frame_index);
// Reserve based on the bytes we actually hold, not the (already-bounded but
// still caller-supplied) frame_bytes, so a small frame can't over-reserve.
let actual: usize = frame.block_data.values().map(|b| b.len()).sum();
@@ -522,11 +609,30 @@ impl Reassembler {
Ok(None)
}
/// Drop all in-flight state — every partially-assembled frame and the completed/abandoned
/// index memory, in both index spaces — as if the session just started. Used by the client's
/// backlog flush ([`Session::flush_backlog`](crate::session::Session::flush_backlog)): after
/// the socket backlog is discarded wholesale, the partial frames here can never complete
/// (their remaining shards were just thrown away) and the window anchors (`newest_frame`)
/// point into the discarded past.
pub fn reset(&mut self) {
self.video = ReassemblyWindow::default();
self.probe = ReassemblyWindow::default();
}
}
impl ReassemblyWindow {
/// Track the newest frame, declare incomplete frames that fell out of the loss window
/// ([`LOSS_WINDOW_NS`] behind the newest pts, or [`HARD_LOSS_WINDOW`] indices) lost — counting
/// them dropped, which is what drives the client's recovery-keyframe request — and prune the
/// completed-index memory to [`REORDER_WINDOW`].
fn advance_window(&mut self, frame_index: u32, pts_ns: u64, stats: &StatsCounters) {
/// ([`LOSS_WINDOW_NS`] behind the newest pts, or [`HARD_LOSS_WINDOW`] indices) lost — for the
/// video window (`count_drops`) counting them dropped, which is what drives the client's
/// recovery-keyframe request — and prune the completed-index memory to [`REORDER_WINDOW`].
fn advance_window(
&mut self,
frame_index: u32,
pts_ns: u64,
stats: &StatsCounters,
count_drops: bool,
) {
let (newest, newest_pts) = match self.newest_frame {
// `frame_index` is newer iff it's within the forward half of the index space.
Some((n, p)) if frame_index.wrapping_sub(n) > u32::MAX / 2 => (n, p),
@@ -548,29 +654,17 @@ impl Reassembler {
keep
});
let pruned = before - self.frames.len();
if pruned > 0 {
if pruned > 0 && count_drops {
StatsCounters::add(&stats.frames_dropped, pruned as u64);
}
self.completed
.retain(|&idx| newest.wrapping_sub(idx) <= REORDER_WINDOW);
}
/// Drop all in-flight state — every partially-assembled frame and the completed/abandoned
/// index memory — as if the session just started. Used by the client's backlog flush
/// ([`Session::flush_backlog`](crate::session::Session::flush_backlog)): after the socket
/// backlog is discarded wholesale, the partial frames here can never complete (their remaining
/// shards were just thrown away) and the window anchor (`newest_frame`) points into the
/// discarded past.
pub fn reset(&mut self) {
self.frames.clear();
self.completed.clear();
self.newest_frame = None;
}
/// True if this packet's frame lies outside the loss window (behind the newest frame by more
/// than [`LOSS_WINDOW_NS`] of capture time or [`HARD_LOSS_WINDOW`] indices) — its shards
/// arrive too late to be useful, and accepting one would only create a frame buffer the next
/// [`advance_window`] immediately declares lost.
/// [`advance_window`](Self::advance_window) immediately declares lost.
fn is_stale(&self, frame_index: u32, pts_ns: u64) -> bool {
match self.newest_frame {
Some((n, newest_pts)) => {
@@ -750,6 +844,119 @@ mod tests {
assert_eq!(stats.snapshot().frames_dropped, 1, "no double-count");
}
/// The explicit-index path stamps the caller's `frame_index` and leaves the internal video
/// counter untouched — the punktfunk/1 encode loop owns the numbering, and mixing must not
/// perturb the legacy self-numbering path (tests/ABI/synthetic).
#[test]
fn explicit_frame_index_is_stamped_and_internal_counter_untouched() {
use crate::config::{FecConfig, FecScheme, ProtocolPhase, Role};
let cfg = Config {
role: Role::Host,
phase: ProtocolPhase::P2Punktfunk,
fec: FecConfig {
scheme: FecScheme::Gf16,
fec_percent: 0,
max_data_per_block: 8,
},
shard_payload: 16,
max_frame_bytes: 4096,
encrypt: false,
key: [0u8; 16],
salt: [0u8; 4],
loopback_drop_period: 0,
};
let coder = coder_for(FecScheme::Gf16);
let mut pk = Packetizer::new(&cfg);
let mut seen = Vec::new();
pk.packetize_each(&[1u8; 16], 0, 0, Some(4242), coder.as_ref(), |hdr, _| {
seen.push(hdr.frame_index);
Ok(())
})
.unwrap();
assert_eq!(seen, vec![4242]);
// The legacy wrapper still numbers from the untouched internal counter.
let pkts = pk.packetize(&[1u8; 16], 0, 0, coder.as_ref()).unwrap();
let hdr = PacketHeader::read_from_bytes(&pkts[0][..HEADER_LEN]).unwrap();
assert_eq!(hdr.frame_index, 0);
// The probe space is a third, independent counter.
assert_eq!(pk.alloc_probe_index(), 0);
assert_eq!(pk.alloc_probe_index(), 1);
}
/// Probe filler (FLAG_PROBE in user_flags) reassembles in its OWN window: a probe frame whose
/// index is far behind the video stream's completes anyway (an old client's single window
/// would drop it as stale), and video frames complete undisturbed around it.
#[test]
fn probe_frames_reassemble_in_their_own_window() {
let mut r = Reassembler::new(limits());
let coder = coder_for(FecScheme::Gf8);
let stats = StatsCounters::default();
// Establish a video stream far into its index space.
let mut v = base_header();
v.frame_index = 100_000;
v.pts_ns = 1_000_000_000;
assert!(r
.push(&packet(v), coder.as_ref(), &stats)
.unwrap()
.is_some());
// A probe frame at index 0 — 100k "behind" the video window — must still complete.
let mut p = base_header();
p.frame_index = 0;
p.pts_ns = 1_000_000_100;
p.user_flags = FLAG_PROBE as u32;
let got = r.push(&packet(p), coder.as_ref(), &stats).unwrap();
assert!(got.is_some(), "probe frame must complete in its own window");
assert_eq!(got.unwrap().flags & FLAG_PROBE as u32, FLAG_PROBE as u32);
// The probe burst must not have advanced the VIDEO window: the next video frame is
// contiguous and completes, with nothing counted dropped.
let mut v2 = base_header();
v2.frame_index = 100_001;
v2.pts_ns = 1_000_000_200;
assert!(r
.push(&packet(v2), coder.as_ref(), &stats)
.unwrap()
.is_some());
assert_eq!(stats.snapshot().frames_dropped, 0);
}
/// An incomplete probe frame aging out of the probe window is NOT a video `frames_dropped`
/// (which would fire the client's loss recovery) — probe loss is measured bytes-wise by the
/// probe accumulator.
#[test]
fn aged_out_probe_frames_do_not_count_as_dropped() {
let mut r = Reassembler::new(limits());
let coder = coder_for(FecScheme::Gf8);
let stats = StatsCounters::default();
// Probe frame 0: one of two shards — incomplete.
let mut p = base_header();
p.user_flags = FLAG_PROBE as u32;
p.data_shards = 2;
p.frame_bytes = 32;
assert!(r
.push(&packet(p), coder.as_ref(), &stats)
.unwrap()
.is_none());
// A much newer probe frame ages it out of the probe window.
let mut p2 = base_header();
p2.user_flags = FLAG_PROBE as u32;
p2.frame_index = 1;
p2.pts_ns = LOSS_WINDOW_NS + 1;
assert!(r
.push(&packet(p2), coder.as_ref(), &stats)
.unwrap()
.is_some());
assert_eq!(
stats.snapshot().frames_dropped,
0,
"probe-window drops must not fire video loss recovery"
);
}
#[test]
fn rejects_wrong_shard_bytes_and_oversized_frame() {
let coder = coder_for(FecScheme::Gf8);
@@ -0,0 +1,115 @@
//! Per-transfer clipboard fetch streams (`design/clipboard-and-file-transfer.md` §3.3).
//!
//! Bulk clipboard / file bytes never ride the control stream (u16-capped) or datagrams (lossy,
//! single-packet). The **requester opens a fresh QUIC bi-stream** toward the data holder, writes a
//! small stream header + a [`ClipFetch`]; the holder replies with a [`ClipFetchHdr`] then raw data
//! chunks until FIN. One transfer per stream ⇒ natural flow control, clean cancelation
//! (`RESET_STREAM`), and no head-of-line blocking against control or other transfers.
//!
//! These helpers are the transport half only; they hold no clipboard state, so the host and the
//! client-core reuse the exact same open/accept/serve wire dance (the accept-loop that dispatches
//! by stream kind lives on each side, since the two sides own their connections differently).
use super::{io, ClipFetch, ClipFetchHdr};
/// First bytes an opener writes on a freshly-opened clipboard bi-stream: a magic keeping this
/// stream namespace disjoint from any future stream kind, plus a 1-byte kind discriminator. A
/// distinct magic means a stream opened for some other future purpose can never be misrouted here.
pub const STREAM_MAGIC: &[u8; 4] = b"PKFs";
/// Stream-kind byte: a clipboard fetch (request/response of one format). Future stream kinds
/// (e.g. a bulk file-content push) mux under the same [`STREAM_MAGIC`] with a different byte.
pub const CLIP_STREAM_KIND_FETCH: u8 = 0x01;
/// QUIC application error code used to `reset`/`stop` a clipboard fetch stream on cancel — sync
/// disabled mid-transfer, paste timed out, size cap exceeded, teardown. Distinct from the
/// connection close codes ([`super::QUIT_CLOSE_CODE`] `0x51` / [`super::APP_EXITED_CLOSE_CODE`]
/// `0x52`) and the connection reject code `0x42`.
pub const CLIP_CANCELLED_CODE: u32 = 0x60;
/// Chunk size for streaming fetch data (64 KiB writes — matches the control-frame bound).
pub const CLIP_CHUNK: usize = 64 * 1024;
/// The `VarInt` form of [`CLIP_CANCELLED_CODE`], for `SendStream::reset` / `RecvStream::stop`.
pub fn cancelled_code() -> quinn::VarInt {
quinn::VarInt::from_u32(CLIP_CANCELLED_CODE)
}
/// Requester side: open a fresh bi-stream toward the holder, deprioritize it under the control
/// stream, write the stream header + the [`ClipFetch`], and hand back both halves. The send half
/// is returned so the caller can `reset`/`finish` for cancelation; the recv half is positioned to
/// read the [`ClipFetchHdr`] next (see [`read_fetch_hdr`]).
pub async fn open_fetch(
conn: &quinn::Connection,
req: &ClipFetch,
) -> std::io::Result<(quinn::SendStream, quinn::RecvStream)> {
let (mut send, recv) = conn.open_bi().await.map_err(std::io::Error::other)?;
// Yield to the control stream (default priority 0) so a large paste never head-of-line-blocks
// the input/audio/control traffic sharing this connection.
let _ = send.set_priority(-1);
// The opener MUST write before the peer's `accept_bi()` can return (quinn contract), so send
// the whole request eagerly.
let mut hdr = Vec::with_capacity(5);
hdr.extend_from_slice(STREAM_MAGIC);
hdr.push(CLIP_STREAM_KIND_FETCH);
send.write_all(&hdr).await.map_err(std::io::Error::other)?;
io::write_msg(&mut send, &req.encode()).await?;
Ok((send, recv))
}
/// Holder side, step 1: after `accept_bi()`, read and validate the 5-byte stream header. Returns
/// the kind byte (e.g. [`CLIP_STREAM_KIND_FETCH`]); an unknown magic is an error and the caller
/// should `stop` the stream.
pub async fn read_stream_header(recv: &mut quinn::RecvStream) -> std::io::Result<u8> {
let mut hdr = [0u8; 5];
recv.read_exact(&mut hdr)
.await
.map_err(std::io::Error::other)?;
if &hdr[0..4] != STREAM_MAGIC {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"bad clip stream magic",
));
}
Ok(hdr[4])
}
/// Holder side, step 2: read the [`ClipFetch`] request that follows the header.
pub async fn read_fetch(recv: &mut quinn::RecvStream) -> std::io::Result<ClipFetch> {
let raw = io::read_msg(recv).await?;
ClipFetch::decode(&raw)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "bad ClipFetch"))
}
/// Holder side, step 3: send the response header (before any data chunks).
pub async fn write_fetch_hdr(
send: &mut quinn::SendStream,
hdr: &ClipFetchHdr,
) -> std::io::Result<()> {
io::write_msg(send, &hdr.encode()).await
}
/// Holder side, step 4 (only when the header was [`super::CLIP_FETCH_OK`]): stream `data` as
/// 64 KiB chunks then FIN so the requester's [`read_data`] terminates.
pub async fn write_data(send: &mut quinn::SendStream, data: &[u8]) -> std::io::Result<()> {
for chunk in data.chunks(CLIP_CHUNK) {
send.write_all(chunk).await.map_err(std::io::Error::other)?;
}
send.finish().map_err(std::io::Error::other)?;
Ok(())
}
/// Requester side: read the [`ClipFetchHdr`] the holder sends before any data chunks.
pub async fn read_fetch_hdr(recv: &mut quinn::RecvStream) -> std::io::Result<ClipFetchHdr> {
let raw = io::read_msg(recv).await?;
ClipFetchHdr::decode(&raw)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "bad ClipFetchHdr"))
}
/// Requester side: after an OK [`ClipFetchHdr`], drain the data chunks to a `Vec`, bounded by
/// `max_bytes` (the requester's size cap — a breach errors, and the caller resets the stream).
pub async fn read_data(recv: &mut quinn::RecvStream, max_bytes: usize) -> std::io::Result<Vec<u8>> {
recv.read_to_end(max_bytes)
.await
.map_err(std::io::Error::other)
}
+4
View File
@@ -49,6 +49,10 @@ pub mod endpoint;
/// Async framed-message IO over a quinn stream (`u16 LE length || payload`).
pub mod io;
/// Per-transfer clipboard fetch bi-streams (`PKFs` magic + kind byte, then request/response). The
/// transport half of the shared clipboard; wire codecs are in [`msgs`], state lives per side.
pub mod clipstream;
/// SPAKE2 over Ed25519 for the pairing ceremony. The two roles use the asymmetric flow so
/// the identities are ordered; each side binds **both** certificate fingerprints as the
/// SPAKE2 identities, so the derived key only matches when client and host agree on the PIN
+371
View File
@@ -107,6 +107,18 @@ pub const VIDEO_CAP_444: u8 = 0x04;
/// host ignores it and simply never sends any); a client that doesn't set it keeps the combined
/// stage. Purely observability — never changes what the host encodes.
pub const VIDEO_CAP_HOST_TIMING: u8 = 0x08;
/// [`Hello::video_caps`] bit: the client's reassembler keeps **speed-test probe filler in its own
/// frame-index space** (a second reassembly window keyed on the [`crate::packet::FLAG_PROBE`]
/// user-flag), so probe bursts no longer consume video `frame_index`es. Without this, a mid-session
/// speed test burns thousands of video indexes that are invisible to every client-side gap detector
/// (probe frames are filtered before the pump sees them) — the first real AU afterwards reads as a
/// phantom multi-thousand-frame loss (spurious freeze + a nonsense RFI). It also lets the host's
/// encode loop own the video numbering outright (the wire-index contract
/// [`crate::packet::Packetizer::packetize_each`] documents), which reference-frame invalidation
/// depends on. The host runs mid-session probe bursts ONLY against clients that set this bit — an
/// older client gets a declined (zeroed) [`ProbeResult`] instead of a measurement its single-window
/// reassembler would silently drop as stale.
pub const VIDEO_CAP_PROBE_SEQ: u8 = 0x10;
/// QUIC application error code a punktfunk/1 client closes the control connection with on a
/// **deliberate quit** (a user "stop", not a network drop). The host reads it off the connection's
@@ -130,6 +142,14 @@ pub const APP_EXITED_CLOSE_CODE: u32 = 0x52;
/// button/axis events; toward a host that doesn't set the bit it keeps the legacy events.
pub const HOST_CAP_GAMEPAD_STATE: u8 = 0x01;
/// [`Welcome::host_caps`] bit: the host has a shared-clipboard service (a working OS backend)
/// **and** its operator policy does not hard-disable it, so the client may offer the clipboard
/// toggle. Absent (an older host, or `PUNKTFUNK_CLIPBOARD` off) ⇒ the client greys the toggle
/// out. Purely additive: nothing clipboard-related happens until a [`ClipControl`]`{ enabled:
/// true }` crosses (see `design/clipboard-and-file-transfer.md` §3.1). Packs into the existing
/// trailing `host_caps` byte — no wire-layout change.
pub const HOST_CAP_CLIPBOARD: u8 = 0x02;
/// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software**
/// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST
/// advertise this.
@@ -355,6 +375,24 @@ pub struct Reconfigured {
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct RequestKeyframe;
/// `client → host`: reference-frame-invalidation recovery — the loss-aware sibling of
/// [`RequestKeyframe`]. The client detected a `frame_index` gap and reports the range `[first_frame,
/// last_frame]` of access units it can no longer trust (from the first missing index through the
/// newest received). Instead of a full IDR (a 20-40× spike that deepens the loss it recovers), a host
/// whose encoder supports RFI re-references a known-good picture *before* `first_frame` — an AMD LTR
/// force-reference or an NVENC `nvEncInvalidateRefFrames` — emitting a single clean P-frame it tags
/// [`crate::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its freeze on it. A host that
/// can't RFI (no valid reference / libavcodec backend) forces an IDR instead, exactly as for a bare
/// [`RequestKeyframe`]; a host that predates this ignores the unknown message and the client's
/// keyframe backstop still recovers. Fire-and-forget — the recovered frame is the only ack.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct RfiRequest {
/// First access-unit `frame_index` the client can no longer trust (the gap start).
pub first_frame: u32,
/// Newest received `frame_index` at the time of the report (the invalidation range end).
pub last_frame: u32,
}
/// `client → host`, periodic: the client's observed data-plane loss, so the host can size FEC to
/// the link instead of a flat percentage (adaptive FEC). `loss_ppm` is parts-per-million of shards
/// that arrived missing-but-recovered (plus a bump when frames went unrecoverable) over the report
@@ -467,6 +505,8 @@ pub const MSG_LOSS_REPORT: u8 = 0x04;
pub const MSG_SET_BITRATE: u8 = 0x05;
/// Type byte of [`BitrateChanged`].
pub const MSG_BITRATE_CHANGED: u8 = 0x06;
/// Type byte of [`RfiRequest`].
pub const MSG_RFI_REQUEST: u8 = 0x07;
/// Type byte of [`ProbeRequest`].
pub const MSG_PROBE_REQUEST: u8 = 0x20;
/// Type byte of [`ProbeResult`].
@@ -476,6 +516,147 @@ pub const MSG_CLOCK_PROBE: u8 = 0x30;
/// Type byte of [`ClockEcho`].
pub const MSG_CLOCK_ECHO: u8 = 0x31;
// ---------------------------------------------------------------------------------------------
// Shared clipboard & file transfer (design/clipboard-and-file-transfer.md §3). The small
// metadata messages ride the control stream (0x40-0x42); the two fetch-stream messages
// (0x43-0x44) travel on a per-transfer bi-stream (see the [`super::clipstream`] helpers), never
// the control stream, so they are never dispatched by the control loops. All are typed
// (`CTL_MAGIC` + type byte), so an older peer hits its "unknown control message" arm and drops
// any it doesn't know — the whole feature is forward-safe.
// ---------------------------------------------------------------------------------------------
/// Type byte of [`ClipControl`] (client → host): enable/disable the shared clipboard for this
/// session. Idempotent; opt-in is enforced here, not just in UI.
pub const MSG_CLIP_CONTROL: u8 = 0x40;
/// Type byte of [`ClipState`] (host → client): ack + unsolicited policy/backend updates.
pub const MSG_CLIP_STATE: u8 = 0x41;
/// Type byte of [`ClipOffer`] (symmetric): the lazy announcement — format list only, no bytes.
pub const MSG_CLIP_OFFER: u8 = 0x42;
/// Type byte of [`ClipFetch`] (requester → holder, **fetch stream only**): pull one format of the
/// current offer.
pub const MSG_CLIP_FETCH: u8 = 0x43;
/// Type byte of [`ClipFetchHdr`] (holder → requester, **fetch stream only**): the fetch response
/// header that precedes the data chunks.
pub const MSG_CLIP_FETCH_HDR: u8 = 0x44;
/// [`ClipControl::flags`] bit: the client permits file kinds to be offered/fetched this session.
/// Absent ⇒ files are filtered out of offers in both directions (text/rich/image only).
pub const CLIP_FLAG_FILES: u8 = 0x01;
/// [`ClipState::policy`] bit: the host permits non-file formats (text/RTF/HTML/image). Always set
/// while enabled unless a future direction limit clears it.
pub const CLIP_POLICY_TEXT: u8 = 0x01;
/// [`ClipState::policy`] bit: the host permits file formats. Cleared by the operator `no-files`
/// / `text-only` policy so the client can grey out "Include files".
pub const CLIP_POLICY_FILES: u8 = 0x02;
/// [`ClipState::reason`]: normal ack, nothing exceptional.
pub const CLIP_REASON_OK: u8 = 0;
/// [`ClipState::reason`]: this session type has no working clipboard backend (e.g. a gamescope
/// session with no data-control global) — the client shows "not supported in this session type".
pub const CLIP_REASON_BACKEND_UNAVAILABLE: u8 = 1;
/// [`ClipState::reason`]: another client took over the single per-desktop clipboard binding; this
/// one was disabled (last `ClipControl{enabled}` wins).
pub const CLIP_REASON_TAKEN_OVER: u8 = 2;
/// [`ClipState::reason`]: the host operator policy (`PUNKTFUNK_CLIPBOARD=off`) disables clipboard.
pub const CLIP_REASON_POLICY_DISABLED: u8 = 3;
/// [`ClipState::reason`]: enabled, but the host policy forbids file transfer (`no-files` /
/// `text-only`) — surfaced so the client greys "Include files" with a footnote.
pub const CLIP_REASON_NO_FILES: u8 = 4;
/// [`ClipFetchHdr::status`]: the requested format is being served; data chunks follow until FIN.
pub const CLIP_FETCH_OK: u8 = 0;
/// [`ClipFetchHdr::status`]: the fetch named a `seq` that is no longer the holder's current offer;
/// the requester degrades the paste to "nothing inserted" rather than wrong data. No chunks follow.
pub const CLIP_FETCH_STALE: u8 = 1;
/// [`ClipFetchHdr::status`]: the format/index is not available (no backend, or it vanished). No
/// chunks follow.
pub const CLIP_FETCH_UNAVAILABLE: u8 = 2;
/// [`ClipFetchHdr::status`]: policy/cap denies this fetch (e.g. a file fetch under `no-files`). No
/// chunks follow.
pub const CLIP_FETCH_DENIED: u8 = 3;
/// Maximum number of [`ClipKind`] entries in one [`ClipOffer`] (resource cap, §7).
pub const CLIP_MAX_KINDS: usize = 16;
/// Maximum length in bytes of a [`ClipKind::mime`] string (resource cap, §7).
pub const CLIP_MAX_MIME: usize = 128;
/// [`ClipFetch::file_index`] sentinel meaning "not a file fetch" (a whole non-file format, or the
/// file *manifest* itself). Real file fetches use `0..n`.
pub const CLIP_FILE_INDEX_NONE: u32 = u32::MAX;
/// One advertised clipboard format inside a [`ClipOffer`] — a portable MIME name plus a size hint.
/// The bytes never ride here; they cross lazily on a fetch stream only when the destination pastes.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClipKind {
/// Portable wire MIME, e.g. `text/plain;charset=utf-8`, `text/html`, `image/png`,
/// `application/x-punktfunk-files`. Each end maps it to a platform type at fetch time. ≤
/// [`CLIP_MAX_MIME`] bytes; a longer one is rejected on decode.
pub mime: String,
/// Best-effort total size of this format in bytes; `0` = unknown (a streaming provider).
pub size_hint: u64,
}
/// `client → host` ([`MSG_CLIP_CONTROL`]): flip the shared clipboard on/off for this session.
/// Sent when the user toggles the per-host pref and once at session start if it is on. **Nothing
/// clipboard-related happens on either side until an `enabled: true` arrives** — opt-in at the
/// protocol layer.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ClipControl {
pub enabled: bool,
/// Bitfield of [`CLIP_FLAG_FILES`] (+ reserved bits for future direction limits).
pub flags: u8,
}
/// `host → client` ([`MSG_CLIP_STATE`]): acknowledge a [`ClipControl`] and push unsolicited
/// updates (policy changed, backend lost). The client surfaces `reason`/`policy` in the toggle UI
/// instead of failing silently.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ClipState {
pub enabled: bool,
/// Bitfield of [`CLIP_POLICY_TEXT`] / [`CLIP_POLICY_FILES`] — what the host currently permits.
pub policy: u8,
/// One of the `CLIP_REASON_*` values explaining `enabled`/`policy`.
pub reason: u8,
}
/// Symmetric ([`MSG_CLIP_OFFER`], either direction): the lazy announcement. Sent when the local
/// clipboard changes; carries the **format list only** (comfortably inside the 64 KiB control
/// frame). A new offer replaces the sender's previous one; `seq` lets the holder reject stale
/// fetches (§3.4). Files are announced as one `application/x-punktfunk-files` kind — the file
/// list itself is fetched lazily, never inlined here.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClipOffer {
/// Monotonic per sender; newest wins.
pub seq: u32,
/// ≤ [`CLIP_MAX_KINDS`] entries.
pub kinds: Vec<ClipKind>,
}
/// `requester → holder` ([`MSG_CLIP_FETCH`], **fetch stream only**): the first message on a
/// per-transfer bi-stream, naming which format (and, for files, which entry) of `seq` to pull.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClipFetch {
/// The offer `seq` this fetch is against; the holder answers [`CLIP_FETCH_STALE`] if it is no
/// longer current.
pub seq: u32,
/// File index for a file transfer, or [`CLIP_FILE_INDEX_NONE`] for a non-file format / the
/// file manifest.
pub file_index: u32,
/// The requested wire MIME (≤ [`CLIP_MAX_MIME`] bytes).
pub mime: String,
}
/// `holder → requester` ([`MSG_CLIP_FETCH_HDR`], **fetch stream only**): the response header that
/// precedes the raw data chunks (which run until the stream's FIN). When `status` is anything
/// other than [`CLIP_FETCH_OK`] no chunks follow.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ClipFetchHdr {
/// One of the `CLIP_FETCH_*` values.
pub status: u8,
/// Total byte count that will follow; `0` = unknown (a streaming provider — FIN ends it).
pub total_size: u64,
}
// ---------------------------------------------------------------------------------------------
// Pairing ceremony (typed control messages): instead of a session Hello, a client may open
// the control stream with PairRequest. The host shows a short PIN out-of-band (log/UI); the
@@ -1032,6 +1213,28 @@ impl RequestKeyframe {
}
}
impl RfiRequest {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] first_frame[5..9] last_frame[9..13]
let mut b = Vec::with_capacity(13);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_RFI_REQUEST);
b.extend_from_slice(&self.first_frame.to_le_bytes());
b.extend_from_slice(&self.last_frame.to_le_bytes());
b
}
pub fn decode(b: &[u8]) -> Result<RfiRequest> {
if b.len() != 13 || &b[0..4] != CTL_MAGIC || b[4] != MSG_RFI_REQUEST {
return Err(PunktfunkError::InvalidArg("bad RfiRequest"));
}
Ok(RfiRequest {
first_frame: u32::from_le_bytes(b[5..9].try_into().unwrap()),
last_frame: u32::from_le_bytes(b[9..13].try_into().unwrap()),
})
}
}
impl LossReport {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] loss_ppm[5..9]
@@ -1213,6 +1416,174 @@ impl ClockEcho {
}
}
/// Append one [`ClipKind`] to `b`: `mime_len u8 || mime bytes || size_hint u64 LE`.
fn put_clip_kind(b: &mut Vec<u8>, k: &ClipKind) {
let mime = k.mime.as_bytes();
let n = mime.len().min(CLIP_MAX_MIME);
b.push(n as u8);
b.extend_from_slice(&mime[..n]);
b.extend_from_slice(&k.size_hint.to_le_bytes());
}
/// Read one [`ClipKind`] at `off`, returning it and the next offset.
fn get_clip_kind(b: &[u8], off: usize) -> Result<(ClipKind, usize)> {
if off >= b.len() {
return Err(PunktfunkError::InvalidArg("truncated ClipKind"));
}
let n = b[off] as usize;
if n > CLIP_MAX_MIME {
return Err(PunktfunkError::InvalidArg("ClipKind mime too long"));
}
let mime_start = off + 1;
let size_start = mime_start + n;
if size_start + 8 > b.len() {
return Err(PunktfunkError::InvalidArg("ClipKind overruns message"));
}
let mime = String::from_utf8_lossy(&b[mime_start..size_start]).into_owned();
let size_hint = u64::from_le_bytes(b[size_start..size_start + 8].try_into().unwrap());
Ok((ClipKind { mime, size_hint }, size_start + 8))
}
impl ClipControl {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] enabled[5] flags[6]
let mut b = Vec::with_capacity(7);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CLIP_CONTROL);
b.push(self.enabled as u8);
b.push(self.flags);
b
}
pub fn decode(b: &[u8]) -> Result<ClipControl> {
if b.len() != 7 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CLIP_CONTROL {
return Err(PunktfunkError::InvalidArg("bad ClipControl"));
}
Ok(ClipControl {
enabled: b[5] != 0,
flags: b[6],
})
}
}
impl ClipState {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] enabled[5] policy[6] reason[7]
let mut b = Vec::with_capacity(8);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CLIP_STATE);
b.push(self.enabled as u8);
b.push(self.policy);
b.push(self.reason);
b
}
pub fn decode(b: &[u8]) -> Result<ClipState> {
if b.len() != 8 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CLIP_STATE {
return Err(PunktfunkError::InvalidArg("bad ClipState"));
}
Ok(ClipState {
enabled: b[5] != 0,
policy: b[6],
reason: b[7],
})
}
}
impl ClipOffer {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] seq[5..9] count[9] then `count` ClipKinds
let mut b = Vec::with_capacity(10 + self.kinds.len() * 16);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CLIP_OFFER);
b.extend_from_slice(&self.seq.to_le_bytes());
let count = self.kinds.len().min(CLIP_MAX_KINDS);
b.push(count as u8);
for k in &self.kinds[..count] {
put_clip_kind(&mut b, k);
}
b
}
pub fn decode(b: &[u8]) -> Result<ClipOffer> {
if b.len() < 10 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CLIP_OFFER {
return Err(PunktfunkError::InvalidArg("bad ClipOffer"));
}
let seq = u32::from_le_bytes(b[5..9].try_into().unwrap());
let count = b[9] as usize;
if count > CLIP_MAX_KINDS {
return Err(PunktfunkError::InvalidArg("ClipOffer too many kinds"));
}
let mut kinds = Vec::with_capacity(count);
let mut off = 10;
for _ in 0..count {
let (k, next) = get_clip_kind(b, off)?;
kinds.push(k);
off = next;
}
if off != b.len() {
return Err(PunktfunkError::InvalidArg("trailing bytes"));
}
Ok(ClipOffer { seq, kinds })
}
}
impl ClipFetch {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] seq[5..9] file_index[9..13] mime(len u8 || bytes)[13..]
let mime = self.mime.as_bytes();
let n = mime.len().min(CLIP_MAX_MIME);
let mut b = Vec::with_capacity(14 + n);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CLIP_FETCH);
b.extend_from_slice(&self.seq.to_le_bytes());
b.extend_from_slice(&self.file_index.to_le_bytes());
b.push(n as u8);
b.extend_from_slice(&mime[..n]);
b
}
pub fn decode(b: &[u8]) -> Result<ClipFetch> {
if b.len() < 14 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CLIP_FETCH {
return Err(PunktfunkError::InvalidArg("bad ClipFetch"));
}
let seq = u32::from_le_bytes(b[5..9].try_into().unwrap());
let file_index = u32::from_le_bytes(b[9..13].try_into().unwrap());
let n = b[13] as usize;
if n > CLIP_MAX_MIME || b.len() != 14 + n {
return Err(PunktfunkError::InvalidArg("bad ClipFetch mime"));
}
let mime = String::from_utf8_lossy(&b[14..14 + n]).into_owned();
Ok(ClipFetch {
seq,
file_index,
mime,
})
}
}
impl ClipFetchHdr {
pub fn encode(&self) -> Vec<u8> {
// magic[0..4] type[4] status[5] total_size[6..14]
let mut b = Vec::with_capacity(14);
b.extend_from_slice(CTL_MAGIC);
b.push(MSG_CLIP_FETCH_HDR);
b.push(self.status);
b.extend_from_slice(&self.total_size.to_le_bytes());
b
}
pub fn decode(b: &[u8]) -> Result<ClipFetchHdr> {
if b.len() != 14 || &b[0..4] != CTL_MAGIC || b[4] != MSG_CLIP_FETCH_HDR {
return Err(PunktfunkError::InvalidArg("bad ClipFetchHdr"));
}
Ok(ClipFetchHdr {
status: b[5],
total_size: u64::from_le_bytes(b[6..14].try_into().unwrap()),
})
}
}
/// Frame a message for the control stream: `u16 LE length || payload`.
pub fn frame(payload: &[u8]) -> Vec<u8> {
let mut b = Vec::with_capacity(2 + payload.len());
+416
View File
@@ -633,6 +633,35 @@ fn request_keyframe_roundtrip() {
assert!(RequestKeyframe::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
}
#[test]
fn rfi_request_roundtrip() {
for (first_frame, last_frame) in [(0u32, 0u32), (40, 47), (5, 5), (1_000_000, u32::MAX)] {
let r = RfiRequest {
first_frame,
last_frame,
};
assert_eq!(RfiRequest::decode(&r.encode()).unwrap(), r);
}
// Disjoint from the bare keyframe request (its loss-unaware sibling) and others: type byte + length.
assert!(RfiRequest::decode(&RequestKeyframe.encode()).is_err());
assert!(RequestKeyframe::decode(
&RfiRequest {
first_frame: 1,
last_frame: 2
}
.encode()
)
.is_err());
// Exact length — no trailing bytes.
let bytes = RfiRequest {
first_frame: 3,
last_frame: 9,
}
.encode();
assert!(RfiRequest::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
assert!(RfiRequest::decode(&bytes[..bytes.len() - 1]).is_err());
}
#[test]
fn loss_report_roundtrip() {
for loss_ppm in [0u32, 1, 12_345, 50_000, 1_000_000] {
@@ -1130,3 +1159,390 @@ fn fingerprint_is_sha256_of_der() {
assert_eq!(a, endpoint::cert_fingerprint(b"cert-a"));
assert_ne!(a, endpoint::cert_fingerprint(b"cert-b"));
}
// ---- Shared clipboard control + fetch-stream message codecs (0x40-0x44) -----------------------
#[test]
fn clip_control_roundtrip() {
for (enabled, flags) in [
(true, 0u8),
(false, 0),
(true, CLIP_FLAG_FILES),
(false, 0xFF),
] {
let m = ClipControl { enabled, flags };
assert_eq!(ClipControl::decode(&m.encode()).unwrap(), m);
}
// Disjoint from its host→client sibling (type byte + length) and exact length.
assert!(ClipControl::decode(
&ClipState {
enabled: true,
policy: 0,
reason: 0
}
.encode()
)
.is_err());
let bytes = ClipControl {
enabled: true,
flags: 0,
}
.encode();
assert!(ClipControl::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
assert!(ClipControl::decode(&bytes[..bytes.len() - 1]).is_err());
}
#[test]
fn clip_state_roundtrip() {
let cases = [
ClipState {
enabled: true,
policy: CLIP_POLICY_TEXT | CLIP_POLICY_FILES,
reason: CLIP_REASON_OK,
},
ClipState {
enabled: false,
policy: 0,
reason: CLIP_REASON_BACKEND_UNAVAILABLE,
},
ClipState {
enabled: true,
policy: CLIP_POLICY_TEXT,
reason: CLIP_REASON_NO_FILES,
},
];
for m in cases {
assert_eq!(ClipState::decode(&m.encode()).unwrap(), m);
}
// A ClipControl must not decode as a ClipState (type byte).
assert!(ClipState::decode(
&ClipControl {
enabled: true,
flags: 0
}
.encode()
)
.is_err());
let bytes = cases[0].encode();
assert!(ClipState::decode(&bytes[..bytes.len() - 1]).is_err());
}
#[test]
fn clip_offer_roundtrip() {
// Empty offer, one kind, and a full multi-format offer (text/rich/image/files).
let cases = [
ClipOffer {
seq: 0,
kinds: vec![],
},
ClipOffer {
seq: 1,
kinds: vec![ClipKind {
mime: "text/plain;charset=utf-8".into(),
size_hint: 12,
}],
},
ClipOffer {
seq: u32::MAX,
kinds: vec![
ClipKind {
mime: "text/plain;charset=utf-8".into(),
size_hint: 0,
},
ClipKind {
mime: "text/html".into(),
size_hint: 4096,
},
ClipKind {
mime: "image/png".into(),
size_hint: 1 << 30,
},
ClipKind {
mime: "application/x-punktfunk-files".into(),
size_hint: 5_000_000_000,
},
],
},
];
for m in &cases {
assert_eq!(&ClipOffer::decode(&m.encode()).unwrap(), m);
}
// Trailing bytes are rejected (get_clip_kind consumes exactly to the end).
let mut padded = cases[1].encode();
padded.push(0);
assert!(ClipOffer::decode(&padded).is_err());
// A count byte over the cap is rejected before allocating.
let mut over = cases[0].encode();
over[9] = (CLIP_MAX_KINDS + 1) as u8;
assert!(ClipOffer::decode(&over).is_err());
// Disjoint from a same-family control message.
assert!(ClipOffer::decode(
&ClipControl {
enabled: true,
flags: 0
}
.encode()
)
.is_err());
}
#[test]
fn clip_fetch_roundtrip() {
let cases = [
ClipFetch {
seq: 1,
file_index: CLIP_FILE_INDEX_NONE,
mime: "text/plain;charset=utf-8".into(),
},
ClipFetch {
seq: 7,
file_index: 0,
mime: "application/x-punktfunk-files".into(),
},
ClipFetch {
seq: u32::MAX,
file_index: 41,
mime: String::new(),
},
];
for m in &cases {
assert_eq!(&ClipFetch::decode(&m.encode()).unwrap(), m);
}
// Trailing + truncation both rejected (exact-length mime check).
let bytes = cases[0].encode();
assert!(ClipFetch::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
assert!(ClipFetch::decode(&bytes[..bytes.len() - 1]).is_err());
// A fetch-stream message must not decode as a control-stream offer, and vice-versa.
assert!(ClipOffer::decode(&cases[0].encode()).is_err());
assert!(ClipFetch::decode(
&ClipOffer {
seq: 1,
kinds: vec![]
}
.encode()
)
.is_err());
}
#[test]
fn clip_fetch_hdr_roundtrip() {
for (status, total_size) in [
(CLIP_FETCH_OK, 15u64),
(CLIP_FETCH_STALE, 0),
(CLIP_FETCH_UNAVAILABLE, 0),
(CLIP_FETCH_DENIED, 0),
(CLIP_FETCH_OK, u64::MAX),
] {
let m = ClipFetchHdr { status, total_size };
assert_eq!(ClipFetchHdr::decode(&m.encode()).unwrap(), m);
}
let bytes = ClipFetchHdr {
status: CLIP_FETCH_OK,
total_size: 1,
}
.encode();
assert!(ClipFetchHdr::decode(&[bytes.as_slice(), &[0]].concat()).is_err());
assert!(ClipFetchHdr::decode(&bytes[..bytes.len() - 1]).is_err());
}
#[test]
fn host_cap_clipboard_bit_is_distinct_and_survives_welcome() {
// The new cap packs into the existing trailing host_caps byte with no layout change.
assert_ne!(HOST_CAP_CLIPBOARD, HOST_CAP_GAMEPAD_STATE);
let mut w = Welcome {
abi_version: 1,
udp_port: 1,
mode: Mode {
width: 1920,
height: 1080,
refresh_hz: 60,
},
fec: FecConfig {
scheme: FecScheme::Gf16,
fec_percent: 0,
max_data_per_block: 1024,
},
shard_payload: 1024,
encrypt: false,
key: [0; 16],
salt: [0; 4],
frames: 0,
compositor: CompositorPref::Auto,
gamepad: GamepadPref::Auto,
bitrate_kbps: 0,
bit_depth: 8,
color: ColorInfo::SDR_BT709,
chroma_format: CHROMA_IDC_420,
audio_channels: 2,
codec: CODEC_HEVC,
host_caps: HOST_CAP_GAMEPAD_STATE | HOST_CAP_CLIPBOARD,
};
let got = Welcome::decode(&w.encode()).unwrap();
assert_eq!(got.host_caps & HOST_CAP_CLIPBOARD, HOST_CAP_CLIPBOARD);
assert_eq!(
got.host_caps & HOST_CAP_GAMEPAD_STATE,
HOST_CAP_GAMEPAD_STATE
);
// Clipboard-off host: the bit is clear, gamepad bit still set.
w.host_caps = HOST_CAP_GAMEPAD_STATE;
assert_eq!(
Welcome::decode(&w.encode()).unwrap().host_caps & HOST_CAP_CLIPBOARD,
0
);
}
// ---- In-process QUIC loopback: the real clipstream fetch transport, both success and cancel ----
mod clip_loopback {
use super::*;
use crate::quic::clipstream;
/// Stand up two loopback quinn endpoints, connect, and return
/// `(server_ep, client_ep, host_conn, client_conn)`. Both endpoints are returned so the caller
/// keeps them in scope — dropping a `quinn::Endpoint` tears down its connections.
async fn connect_pair() -> (
quinn::Endpoint,
quinn::Endpoint,
quinn::Connection,
quinn::Connection,
) {
let server = endpoint::server("127.0.0.1:0".parse().unwrap()).unwrap();
let addr = server.local_addr().unwrap();
let client = endpoint::client_insecure().unwrap();
let accept = tokio::spawn(async move {
let incoming = server.accept().await.expect("incoming connection");
let conn = incoming.await.expect("host side connects");
(server, conn)
});
let client_conn = client
.connect(addr, "punktfunk")
.unwrap()
.await
.expect("client side connects");
let (server, host_conn) = accept.await.unwrap();
(server, client, host_conn, client_conn)
}
#[tokio::test]
async fn fetch_text_transfers_then_cancel_resets() {
let (_server_ep, _client_ep, host_conn, client_conn) = connect_pair().await;
let payload = b"hello clipboard \xf0\x9f\x93\x8b".to_vec(); // text + a 4-byte emoji
let holder_payload = payload.clone();
// Holder = the host side: accept two fetch streams. Serve the first; cancel the second.
let holder = tokio::spawn(async move {
// Fetch #1 — serve the payload.
let (mut send, mut recv) = host_conn.accept_bi().await.expect("accept fetch #1");
let kind = clipstream::read_stream_header(&mut recv)
.await
.expect("stream header #1");
assert_eq!(kind, clipstream::CLIP_STREAM_KIND_FETCH);
let req = clipstream::read_fetch(&mut recv)
.await
.expect("fetch req #1");
assert_eq!(req.seq, 1);
assert_eq!(req.file_index, CLIP_FILE_INDEX_NONE);
assert_eq!(req.mime, "text/plain;charset=utf-8");
clipstream::write_fetch_hdr(
&mut send,
&ClipFetchHdr {
status: CLIP_FETCH_OK,
total_size: holder_payload.len() as u64,
},
)
.await
.expect("write hdr #1");
clipstream::write_data(&mut send, &holder_payload)
.await
.expect("write data #1");
// Fetch #2 — read the request, then cancel mid-transfer with RESET_STREAM.
let (mut send2, mut recv2) = host_conn.accept_bi().await.expect("accept fetch #2");
clipstream::read_stream_header(&mut recv2)
.await
.expect("stream header #2");
let _ = clipstream::read_fetch(&mut recv2)
.await
.expect("fetch req #2");
send2.reset(clipstream::cancelled_code()).unwrap();
host_conn // keep alive until the requester side is done
});
// Requester = the client side.
// #1: full lazy fetch of the text payload.
let req = ClipFetch {
seq: 1,
file_index: CLIP_FILE_INDEX_NONE,
mime: "text/plain;charset=utf-8".into(),
};
let (_send, mut recv) = clipstream::open_fetch(&client_conn, &req)
.await
.expect("open fetch #1");
let hdr = clipstream::read_fetch_hdr(&mut recv)
.await
.expect("read hdr #1");
assert_eq!(hdr.status, CLIP_FETCH_OK);
assert_eq!(hdr.total_size as usize, payload.len());
let got = clipstream::read_data(&mut recv, 8 << 20)
.await
.expect("read data #1");
assert_eq!(got, payload);
// #2: the holder resets the stream — the requester surfaces an error rather than hanging.
let req2 = ClipFetch {
seq: 2,
file_index: CLIP_FILE_INDEX_NONE,
mime: "text/plain;charset=utf-8".into(),
};
let (_send2, mut recv2) = clipstream::open_fetch(&client_conn, &req2)
.await
.expect("open fetch #2");
assert!(
clipstream::read_fetch_hdr(&mut recv2).await.is_err(),
"a cancelled fetch must surface as an error, not a hang"
);
let _host_conn = holder.await.unwrap();
}
#[tokio::test]
async fn read_data_enforces_size_cap() {
let (_server_ep, _client_ep, host_conn, client_conn) = connect_pair().await;
let big = vec![0xABu8; 200_000]; // > the 64 KiB chunk, and > the cap we set below
let holder_payload = big.clone();
let holder = tokio::spawn(async move {
let (mut send, mut recv) = host_conn.accept_bi().await.expect("accept");
clipstream::read_stream_header(&mut recv).await.unwrap();
let _ = clipstream::read_fetch(&mut recv).await.unwrap();
clipstream::write_fetch_hdr(
&mut send,
&ClipFetchHdr {
status: CLIP_FETCH_OK,
total_size: holder_payload.len() as u64,
},
)
.await
.unwrap();
let _ = clipstream::write_data(&mut send, &holder_payload).await;
host_conn
});
let req = ClipFetch {
seq: 1,
file_index: CLIP_FILE_INDEX_NONE,
mime: "application/octet-stream".into(),
};
let (_send, mut recv) = clipstream::open_fetch(&client_conn, &req).await.unwrap();
assert_eq!(
clipstream::read_fetch_hdr(&mut recv).await.unwrap().status,
CLIP_FETCH_OK
);
// Cap below the payload size ⇒ read_data errors instead of buffering unboundedly.
assert!(clipstream::read_data(&mut recv, 64 * 1024).await.is_err());
let _host_conn = holder.await.unwrap();
}
}
+69 -26
View File
@@ -161,6 +161,31 @@ impl Session {
data: &[u8],
pts_ns: u64,
user_flags: u32,
) -> Result<Vec<Vec<u8>>> {
self.seal_frame_inner(data, pts_ns, user_flags, None)
}
/// [`seal_frame`](Self::seal_frame) with the caller's **explicit** `frame_index` instead of the
/// packetizer's internal counter. The punktfunk/1 encode loop owns the video numbering (one
/// session-lifetime counter, stamped per AU) so the encoder's reference-frame-invalidation
/// bookkeeping stays 1:1 with the wire across encoder rebuilds/resets — see
/// [`Packetizer::packetize_each`]. A session must use ONE numbering style per index space.
pub fn seal_frame_at(
&mut self,
data: &[u8],
pts_ns: u64,
user_flags: u32,
frame_index: u32,
) -> Result<Vec<Vec<u8>>> {
self.seal_frame_inner(data, pts_ns, user_flags, Some(frame_index))
}
fn seal_frame_inner(
&mut self,
data: &[u8],
pts_ns: u64,
user_flags: u32,
frame_index: Option<u32>,
) -> Result<Vec<Vec<u8>>> {
if self.config.role != Role::Host {
return Err(PunktfunkError::InvalidArg(
@@ -184,35 +209,36 @@ impl Session {
} = self;
let mut wires = std::mem::take(wire_pool);
let mut used = 0usize;
let result = packetizer.packetize_each(data, pts_ns, user_flags, coder.as_ref(), {
let wires = &mut wires;
let used = &mut used;
move |hdr, body| {
if *used == wires.len() {
wires.push(Vec::new());
}
let wire = &mut wires[*used];
*used += 1;
let seq = *next_seq;
*next_seq = next_seq.wrapping_add(1);
wire.clear();
match crypto {
Some(c) => {
// seq(8) ‖ header(40) ‖ shard ‖ tag scratch(16), sealed over [8..].
wire.extend_from_slice(&seq.to_be_bytes());
wire.extend_from_slice(hdr.as_bytes());
wire.extend_from_slice(body);
wire.resize(wire.len() + crate::crypto::TAG_LEN, 0);
c.seal_in_place(seq, &mut wire[8..])?;
let result =
packetizer.packetize_each(data, pts_ns, user_flags, frame_index, coder.as_ref(), {
let wires = &mut wires;
let used = &mut used;
move |hdr, body| {
if *used == wires.len() {
wires.push(Vec::new());
}
None => {
wire.extend_from_slice(hdr.as_bytes());
wire.extend_from_slice(body);
let wire = &mut wires[*used];
*used += 1;
let seq = *next_seq;
*next_seq = next_seq.wrapping_add(1);
wire.clear();
match crypto {
Some(c) => {
// seq(8) ‖ header(40) ‖ shard ‖ tag scratch(16), sealed over [8..].
wire.extend_from_slice(&seq.to_be_bytes());
wire.extend_from_slice(hdr.as_bytes());
wire.extend_from_slice(body);
wire.resize(wire.len() + crate::crypto::TAG_LEN, 0);
c.seal_in_place(seq, &mut wire[8..])?;
}
None => {
wire.extend_from_slice(hdr.as_bytes());
wire.extend_from_slice(body);
}
}
Ok(())
}
Ok(())
}
});
});
result?;
// A smaller frame uses fewer buffers than the pool holds: drop the unused tail, same
// as the previous `resize_with(packets.len(), ..)` did.
@@ -258,6 +284,23 @@ impl Session {
r.map(|_| ())
}
/// Host: seal + send one **speed-test probe filler** access unit in the probe index space
/// (its own frame counter + the [`crate::packet::FLAG_PROBE`] user-flag) so a burst never
/// consumes video `frame_index`es — the client reassembles probe frames in a separate window
/// and its gap detectors never see them. Only call this against a client that advertised
/// [`crate::quic::VIDEO_CAP_PROBE_SEQ`]; an older client's single-window reassembler would
/// drop probe-space indexes as stale against the video stream.
pub fn submit_probe_frame(&mut self, data: &[u8], pts_ns: u64) -> Result<()> {
let idx = self.packetizer.alloc_probe_index();
let wires =
self.seal_frame_inner(data, pts_ns, crate::packet::FLAG_PROBE as u32, Some(idx))?;
let refs: Vec<&[u8]> = wires.iter().map(|w| w.as_slice()).collect();
let r = self.send_sealed(&refs);
drop(refs);
self.reclaim_wires(wires);
r.map(|_| ())
}
/// Host: live-adjust the FEC recovery percentage (adaptive FEC). Affects the next
/// [`submit_frame`](Self::submit_frame)/[`seal_frame`](Self::seal_frame); the receiver needs no
/// notification (each packet's header carries its block's data/recovery shard counts).
+19 -4
View File
@@ -88,12 +88,22 @@ openh264 = "0.9"
[target.'cfg(target_os = "linux")'.dependencies]
# `screencast` gates the ScreenCast portal module; `remote_desktop` adds the RemoteDesktop
# portal we use for libei input on KWin/GNOME; `tokio` is the default runtime.
# `open_pipe_wire_remote` is unconditional, so ashpd's own `pipewire` feature is not
# needed — we drive PipeWire with the `pipewire` crate below.
# portal we use for libei input on KWin/GNOME; `tokio` is the default runtime. `open_pipe_wire_remote`
# is unconditional, so ashpd's own `pipewire` feature is not needed — we drive PipeWire with the
# `pipewire` crate below. (The GNOME shared-clipboard backend uses Mutter's *direct*
# `org.gnome.Mutter.RemoteDesktop.Session` clipboard via raw `ashpd::zbus` — NOT the xdg
# `org.freedesktop.portal.Clipboard`, which needs interactive approval — so no `clipboard` feature.)
ashpd = { version = "0.13", features = ["screencast", "remote_desktop"] }
ffmpeg-next = "8"
libc = "0.2"
# Direct-SDK NVENC on Linux (design/linux-direct-nvenc.md): the RAW `sys::nvEncodeAPI` types only —
# the entry points are resolved at RUNTIME from the driver's `libnvidia-encode.so.1`
# (encode/linux/nvenc_cuda.rs), NOT link-imported, so the same binary starts fine on AMD/Intel
# Linux boxes (no NVIDIA driver) and falls through to VAAPI/software. `ci-check` = vendored
# bindings + cudarc `dynamic-loading` (no CUDA toolkit/headers at build); we never call the crate's
# cudarc — CUDA is driven through the existing `zerocopy::cuda` dlopen table. Same crate + feature
# as the Windows target dep (Cargo.toml, windows target section) so the `sys` structs never drift.
nvidia-video-codec-sdk = { version = "0.4", features = ["ci-check"], optional = true }
# Must match the pipewire crate ashpd 0.13 links (libspa/pipewire-sys `links` key is
# unique per build), i.e. 0.9 — NOT the 0.10 the setup doc mentions.
pipewire = "0.9"
@@ -107,7 +117,7 @@ wayland-protocols-wlr = { version = "0.3", features = ["client"] }
wayland-protocols-misc = { version = "0.3", features = ["client"] }
# `xdg-output` (zxdg_output_v1): the per-output *logical* geometry (post-scale size + global
# position), used by the KWin fake_input backend to map absolute coordinates under display scaling.
wayland-protocols = { version = "0.32", features = ["client"] }
wayland-protocols = { version = "0.32", features = ["client", "staging"] }
# Codegen for KDE's `zkde_screencast_unstable_v1` (vendored in `protocols/`): create a KWin
# virtual output sized to the client's resolution and get its PipeWire node (KRdp's path).
# `wayland-backend` is referenced by the generated interface tables.
@@ -192,6 +202,11 @@ windows = { version = "0.62", features = [
# See capture/windows/dxgi.rs `install_gpu_pref_hook`. No trampoline (we fully replace the fn) → no detour
# crate / no C length-disassembler dep; a 12-byte absolute-jmp prologue patch suffices.
"Win32_System_Memory",
# Shared-clipboard host backend (src/clipboard/windows.rs): the Win32 clipboard API
# (Open/Get/SetClipboardData, AddClipboardFormatListener, delayed rendering) plus the
# CLIPBOARD_FORMAT / CF_UNICODETEXT newtype from Ole (design/clipboard-and-file-transfer.md §3).
"Win32_System_DataExchange",
"Win32_System_Ole",
# Per-monitor-v2 DPI awareness — IDXGIOutput5::DuplicateOutput1 (the modern capture path Apollo
# uses; FP16/format-list, robust to overlay/format churn) requires the process to be DPI-aware.
"Win32_UI_HiDpi",
+3
View File
@@ -468,3 +468,6 @@ pub mod dxgi;
pub mod idd_push;
#[cfg(target_os = "linux")]
mod linux;
#[cfg(target_os = "windows")]
#[path = "capture/windows/synthetic_nv12.rs"]
pub mod synthetic_nv12;
@@ -0,0 +1,182 @@
//! A headless synthetic **NV12 D3D11** capture source for exercising the GPU encoders on Windows
//! without a real capture session.
//!
//! The native AMF path (and the D3D11 zero-copy NVENC/QSV paths) require an NV12 texture that lives
//! on the GPU — the CPU-Bgrx [`SyntheticCapturer`](crate::capture::SyntheticCapturer) can't provide
//! one, and DXGI Desktop Duplication can't create one under an ssh session-0 (E_ACCESSDENIED). This
//! source builds an NV12 texture on the selected render adapter and fills it with a **moving** luma
//! ramp each frame, so the encoder sees genuine motion (P-frame residuals + the intra-refresh wave
//! under content change) — exactly what an intra-refresh recovery validation needs. Driven by
//! `spike --source synthetic-nv12`.
use crate::capture::dxgi::{make_device, D3d11Frame};
use crate::capture::{CapturedFrame, Capturer, FramePayload, PixelFormat};
use anyhow::{Context, Result};
use windows::Win32::Graphics::Direct3D11::{
ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D, D3D11_BIND_SHADER_RESOURCE,
D3D11_CPU_ACCESS_WRITE, D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_WRITE, D3D11_TEXTURE2D_DESC,
D3D11_USAGE, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING,
};
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_NV12, DXGI_SAMPLE_DESC};
use windows::Win32::Graphics::Dxgi::{CreateDXGIFactory1, IDXGIAdapter1, IDXGIFactory4};
/// Synthetic NV12 frames on the GPU. Owns its own D3D11 device + immediate context and two NV12
/// textures: a CPU-writable STAGING scratch it fills each frame, and a DEFAULT texture it copies
/// into and hands to the encoder. The encoder copies out of the DEFAULT texture synchronously
/// (spike drives capture→submit→poll on one thread), so reusing one DEFAULT texture is sound.
pub struct SyntheticNv12Capturer {
device: ID3D11Device,
context: ID3D11DeviceContext,
default_tex: ID3D11Texture2D,
staging: ID3D11Texture2D,
width: u32,
height: u32,
fps: u32,
frame_idx: u64,
}
// SAFETY: mirrors `D3d11Frame`'s reasoning — the device is created free-threaded (`make_device`
// passes no `SINGLETHREADED` flag) and D3D11 uses interlocked COM refcounting, so moving the whole
// capturer (device + immediate context + textures) to its owning thread and using it only there is
// sound. The value is moved, never aliased (no `Sync`), so the single-threaded immediate context is
// never touched concurrently.
unsafe impl Send for SyntheticNv12Capturer {}
impl SyntheticNv12Capturer {
pub fn new(width: u32, height: u32, fps: u32) -> Result<Self> {
// NV12 is 4:2:0 — both dimensions must be even (the chroma plane is width/2 × height/2).
let width = (width & !1).max(2);
let height = (height & !1).max(2);
// SAFETY: a self-contained builder owning every handle it creates; each COM call is checked
// and the returned owners drop with their wrappers.
unsafe {
let adapter =
resolve_render_adapter().context("resolve render adapter for NV12 source")?;
let (device, context) = make_device(&adapter).context("create D3D11 device")?;
let default_tex = create_nv12(
&device,
width,
height,
D3D11_USAGE_DEFAULT,
0,
D3D11_BIND_SHADER_RESOURCE.0 as u32,
)
.context("create NV12 default texture")?;
let staging = create_nv12(
&device,
width,
height,
D3D11_USAGE_STAGING,
D3D11_CPU_ACCESS_WRITE.0 as u32,
0,
)
.context("create NV12 staging texture")?;
Ok(SyntheticNv12Capturer {
device,
context,
default_tex,
staging,
width,
height,
fps,
frame_idx: 0,
})
}
}
}
impl Capturer for SyntheticNv12Capturer {
fn next_frame(&mut self) -> Result<CapturedFrame> {
let pts_ns = self.frame_idx * 1_000_000_000 / self.fps.max(1) as u64;
// SAFETY: Map/Unmap/CopyResource on this capturer's own single-threaded immediate context;
// all writes stay within the mapped NV12 surface (Y: H rows of RowPitch; UV: H/2 rows of
// RowPitch beginning at RowPitch*H — the standard NV12 plane layout).
unsafe {
let mut map = D3D11_MAPPED_SUBRESOURCE::default();
self.context
.Map(&self.staging, 0, D3D11_MAP_WRITE, 0, Some(&mut map))
.context("Map(NV12 staging)")?;
let pitch = map.RowPitch as usize;
let base = map.pData as *mut u8;
// A diagonal luma ramp that shifts 4 codes/frame — strong, deterministic motion.
let shift = (self.frame_idx as u32).wrapping_mul(4);
for y in 0..self.height {
let row = base.add(y as usize * pitch);
for x in 0..self.width {
*row.add(x as usize) = x.wrapping_add(y).wrapping_add(shift) as u8;
}
}
// UV plane (neutral gray = 128) at offset RowPitch*H: H/2 rows, `width` bytes each
// (width/2 interleaved Cb,Cr pairs).
let uv = base.add(pitch * self.height as usize);
for r in 0..(self.height / 2) {
let row = uv.add(r as usize * pitch);
for c in 0..self.width {
*row.add(c as usize) = 128;
}
}
self.context.Unmap(&self.staging, 0);
self.context.CopyResource(&self.default_tex, &self.staging);
}
self.frame_idx += 1;
Ok(CapturedFrame {
width: self.width,
height: self.height,
pts_ns,
format: PixelFormat::Nv12,
payload: FramePayload::D3d11(D3d11Frame {
texture: self.default_tex.clone(),
device: self.device.clone(),
}),
})
}
}
/// Resolve the same render adapter the encoder will pick (`PUNKTFUNK_RENDER_ADAPTER` / preference /
/// max-VRAM LUID), falling back to adapter 0.
///
/// # Safety
/// Calls DXGI factory/adapter enumeration; returns owned COM objects or an error.
unsafe fn resolve_render_adapter() -> Result<IDXGIAdapter1> {
let factory: IDXGIFactory4 = CreateDXGIFactory1().context("CreateDXGIFactory1")?;
if let Some(luid) = crate::win_adapter::resolve_render_adapter_luid() {
if let Ok(a) = factory.EnumAdapterByLuid::<IDXGIAdapter1>(luid) {
return Ok(a);
}
}
factory.EnumAdapters1(0).context("EnumAdapters1(0)")
}
/// Create an NV12 `Texture2D` with the given usage/CPU-access/bind flags.
///
/// # Safety
/// `device` must be a live D3D11 device; the returned texture is owned by the caller.
unsafe fn create_nv12(
device: &ID3D11Device,
width: u32,
height: u32,
usage: D3D11_USAGE,
cpu_access: u32,
bind: u32,
) -> Result<ID3D11Texture2D> {
let desc = D3D11_TEXTURE2D_DESC {
Width: width,
Height: height,
MipLevels: 1,
ArraySize: 1,
Format: DXGI_FORMAT_NV12,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: usage,
BindFlags: bind,
CPUAccessFlags: cpu_access,
..Default::default()
};
let mut tex: Option<ID3D11Texture2D> = None;
device
.CreateTexture2D(&desc, None, Some(&mut tex))
.context("CreateTexture2D(NV12)")?;
tex.context("CreateTexture2D returned a null NV12 texture")
}
+409
View File
@@ -0,0 +1,409 @@
//! Host-side shared-clipboard backend.
//!
//! The wire protocol and the client half live in `punktfunk-core`
//! (`punktfunk_core::quic` + `punktfunk_core::clipboard`); this module drives the **host's** real
//! session clipboard so it can offer what a host app copied and paste what the remote client
//! offered (`design/clipboard-and-file-transfer.md` §4).
//!
//! Concrete backends, selected at session start ([`HostClipboard::open`]) and presented as one
//! [`HostClipboard`] to the [`session`] coordinator:
//! * [`wayland`] (Linux) — `ext-data-control-v1` (KWin, wlroots / Sway, Hyprland). Preferred when present.
//! * [`mutter`] (Linux) — GNOME. Mutter implements **no** wlr/ext data-control, but its *direct*
//! `org.gnome.Mutter.RemoteDesktop.Session` D-Bus API carries the same clipboard operations (the
//! xdg `org.freedesktop.portal.Clipboard` would need an interactive grant a headless host can't
//! answer — so we skip it and talk to Mutter directly, as the input injector already does).
//! * [`windows`] — the Win32 clipboard: a hidden message-only window watches `WM_CLIPBOARDUPDATE`
//! and serves client content via OLE delayed rendering (`WM_RENDERFORMAT`).
//!
//! The `zwlr-data-control-unstable-v1` fallback (older wlroots/KWin) is a follow-up. The module
//! compiles on Linux and Windows; the [`session`] coordinator is backend-agnostic.
#[cfg(target_os = "linux")]
mod mutter;
#[cfg(target_os = "linux")]
mod wayland;
#[cfg(target_os = "windows")]
mod windows;
/// Pure Win32-clipboard ↔ wire byte conversions (CF_HTML offset math, UTF-16 text, RTF NUL
/// trimming). Free of any Win32 dependency, so it compiles — and its unit tests run — on any host
/// (`cfg(test)`); the Windows backend is the only production consumer.
#[cfg(any(target_os = "windows", test))]
mod winfmt;
pub mod session;
#[cfg(target_os = "linux")]
use std::io::Write as _;
#[cfg(target_os = "linux")]
use std::os::fd::OwnedFd;
use std::sync::Arc;
/// A clipboard event surfaced by a host backend to the [`session`] coordinator. Both the
/// data-control and Mutter backends emit this identical shape.
pub enum ClipEvent {
/// The host selection changed (a host app copied). `mimes` are the **wire** MIMEs offered (empty
/// = the clipboard was cleared). The coordinator forwards these as a `ClipOffer` to the client;
/// bytes cross only if the client later fetches.
Selection { mimes: Vec<String> },
/// A host app is pasting content the client offered. The coordinator fetches the wire-`mime`
/// bytes from the client and hands them to `responder`.
Paste {
mime: String,
responder: PasteResponder,
},
/// The backend ended (compositor / session gone).
Closed,
}
/// How a backend receives the bytes answering a [`ClipEvent::Paste`]. The two host clipboard
/// mechanisms complete a paste differently, so the coordinator stays agnostic by handing bytes to
/// whichever responder the backend attached.
pub enum PasteResponder {
/// data-control: the compositor handed us the destination pipe on the `send` event — write the
/// bytes and close it (EOF completes the paste).
#[cfg(target_os = "linux")]
Fd(OwnedFd),
/// Mutter: hand the bytes back to the backend actor, which owns the `SelectionWrite` fd and the
/// trailing `SelectionWriteDone` call that Mutter's transfer requires.
#[cfg(target_os = "linux")]
Channel(tokio::sync::oneshot::Sender<Vec<u8>>),
/// Windows: hand the bytes to the `WM_RENDERFORMAT` handler blocking the clipboard message-loop
/// thread, which then `SetClipboardData`s them for the pasting app (`std::sync::mpsc`, since that
/// thread waits synchronously — see [`windows`]).
#[cfg(target_os = "windows")]
Sync(std::sync::mpsc::Sender<Vec<u8>>),
}
impl PasteResponder {
/// Deliver the fetched bytes (empty on a failed fetch → an empty paste, never a hang).
pub async fn respond(self, bytes: Vec<u8>) {
match self {
#[cfg(target_os = "linux")]
PasteResponder::Fd(fd) => {
let _ = tokio::task::spawn_blocking(move || fulfill_paste(fd, &bytes)).await;
}
#[cfg(target_os = "linux")]
PasteResponder::Channel(tx) => {
let _ = tx.send(bytes);
}
#[cfg(target_os = "windows")]
PasteResponder::Sync(tx) => {
let _ = tx.send(bytes);
}
}
}
}
/// Write `bytes` into a paste pipe `fd` and close it (EOF signals the reader). Blocking — run off the
/// reactor for large payloads.
#[cfg(target_os = "linux")]
fn fulfill_paste(fd: OwnedFd, bytes: &[u8]) -> std::io::Result<()> {
let mut file = std::fs::File::from(fd);
file.write_all(bytes)?;
Ok(())
}
/// The active host clipboard backend, chosen per session: `ext-data-control`
/// (KWin/wlroots/Hyprland/Sway) or Mutter's direct RemoteDesktop clipboard (GNOME) on Linux, or the
/// Win32 clipboard on Windows. Presented as one type so the [`session`] coordinator is
/// backend-agnostic.
pub enum HostClipboard {
#[cfg(target_os = "linux")]
DataControl(wayland::ClipboardBackend),
#[cfg(target_os = "linux")]
Mutter(mutter::MutterClipboard),
#[cfg(target_os = "windows")]
Windows(windows::WindowsClipboard),
}
impl HostClipboard {
/// Open whichever backend this session supports. Linux tries data-control first
/// (KWin/wlroots/Hyprland/Sway) then Mutter's direct clipboard (GNOME); Windows opens the Win32
/// clipboard. Errors when none is available (gamescope, no live compositor) — the caller then
/// reports `BACKEND_UNAVAILABLE`.
pub async fn open() -> anyhow::Result<(
HostClipboard,
tokio::sync::mpsc::UnboundedReceiver<ClipEvent>,
)> {
#[cfg(target_os = "linux")]
{
// data-control's bind does blocking Wayland roundtrips — keep them off the reactor.
let dc = tokio::task::spawn_blocking(wayland::ClipboardBackend::open)
.await
.map_err(|e| anyhow::anyhow!("data-control open join: {e}"))?;
match dc {
Ok((b, rx)) => return Ok((HostClipboard::DataControl(b), rx)),
Err(e) => tracing::debug!(
error = format!("{e:#}"),
"no ext-data-control — trying Mutter direct clipboard"
),
}
let (m, rx) = mutter::MutterClipboard::open().await.map_err(|e| {
e.context("no clipboard backend (neither ext-data-control nor Mutter)")
})?;
Ok((HostClipboard::Mutter(m), rx))
}
#[cfg(target_os = "windows")]
{
let (b, rx) = windows::WindowsClipboard::open().await?;
Ok((HostClipboard::Windows(b), rx))
}
}
/// The current host selection's wire MIMEs (empty = nothing to offer).
pub fn current_wire_mimes(&self) -> Vec<String> {
match self {
#[cfg(target_os = "linux")]
HostClipboard::DataControl(b) => b.current_wire_mimes(),
#[cfg(target_os = "linux")]
HostClipboard::Mutter(m) => m.current_wire_mimes(),
#[cfg(target_os = "windows")]
HostClipboard::Windows(w) => w.current_wire_mimes(),
}
}
/// Install a client's offered formats as the host selection.
pub fn set_offer(&self, wire_mimes: &[String]) -> anyhow::Result<()> {
match self {
#[cfg(target_os = "linux")]
HostClipboard::DataControl(b) => b.set_offer(wire_mimes),
#[cfg(target_os = "linux")]
HostClipboard::Mutter(m) => {
m.set_offer(wire_mimes);
Ok(())
}
#[cfg(target_os = "windows")]
HostClipboard::Windows(w) => {
w.set_offer(wire_mimes);
Ok(())
}
}
}
/// Drop the host selection we own.
pub fn clear_offer(&self) -> anyhow::Result<()> {
match self {
#[cfg(target_os = "linux")]
HostClipboard::DataControl(b) => b.clear_offer(),
#[cfg(target_os = "linux")]
HostClipboard::Mutter(m) => {
m.clear_offer();
Ok(())
}
#[cfg(target_os = "windows")]
HostClipboard::Windows(w) => {
w.clear_offer();
Ok(())
}
}
}
/// Read one wire format of the current host selection (a client's fetch). Async: data-control
/// blocks on a pipe (offloaded), Mutter round-trips D-Bus + reads a pipe, Windows reads the
/// clipboard on a blocking thread.
pub async fn read_current(self: &Arc<Self>, wire_mime: &str) -> anyhow::Result<Vec<u8>> {
match &**self {
#[cfg(target_os = "linux")]
HostClipboard::DataControl(_) => {
let me = Arc::clone(self);
let wire = wire_mime.to_string();
tokio::task::spawn_blocking(move || match &*me {
HostClipboard::DataControl(b) => b.read_current(&wire),
_ => unreachable!("variant checked above"),
})
.await
.map_err(|e| anyhow::anyhow!("data-control read join: {e}"))?
}
#[cfg(target_os = "linux")]
HostClipboard::Mutter(m) => m.read_current(wire_mime).await,
#[cfg(target_os = "windows")]
HostClipboard::Windows(w) => w.read_current(wire_mime).await,
}
}
}
// ---- Format normalization (design/clipboard-and-file-transfer.md §3.5) ------------------------
//
// One portable vocabulary crosses the wire; each end maps to platform types at fetch time. Phase 1
// covers text / RTF / HTML / PNG (files are Phase 2). The wire MIMEs match the core's table.
/// Wire MIME for UTF-8 plain text.
pub const WIRE_TEXT: &str = "text/plain;charset=utf-8";
/// Wire MIME for HTML.
pub const WIRE_HTML: &str = "text/html";
/// Wire MIME for rich text.
pub const WIRE_RTF: &str = "text/rtf";
/// Wire MIME for a PNG image.
pub const WIRE_PNG: &str = "image/png";
/// Map a Wayland selection MIME to its canonical wire MIME, or `None` to drop it (internal targets
/// like `TARGETS`/`TIMESTAMP`/`SAVE_TARGETS`, and formats we don't sync in Phase 1). Aliases
/// collapse onto one canonical wire name so the offered list dedups cleanly.
#[cfg(target_os = "linux")]
pub fn wayland_to_wire(wl: &str) -> Option<&'static str> {
// Strip any parameter noise for the plain-text aliases (some apps send `text/plain;charset=...`
// with odd charsets, or bare `text/plain`).
let base = wl.split(';').next().unwrap_or(wl).trim();
match wl {
"text/html" => Some(WIRE_HTML),
"text/rtf" | "application/rtf" | "text/richtext" => Some(WIRE_RTF),
"image/png" => Some(WIRE_PNG),
_ => match base {
"text/plain" | "UTF8_STRING" | "STRING" | "TEXT" => Some(WIRE_TEXT),
_ => None,
},
}
}
/// The Wayland MIME candidates to request, in preference order, when a client fetches `wire` from
/// the host clipboard. The first one present in the current offer is used.
#[cfg(target_os = "linux")]
pub fn wayland_candidates(wire: &str) -> &'static [&'static str] {
match wire {
WIRE_TEXT => &[
"text/plain;charset=utf-8",
"text/plain",
"UTF8_STRING",
"STRING",
"TEXT",
],
WIRE_HTML => &["text/html"],
WIRE_RTF => &["text/rtf", "application/rtf", "text/richtext"],
WIRE_PNG => &["image/png"],
_ => &[],
}
}
/// Pick the Wayland MIME to `receive()` for a wire fetch: the first [`wayland_candidates`] entry the
/// current selection actually advertises.
#[cfg(target_os = "linux")]
pub fn pick_wayland_mime(wire: &str, available: &[String]) -> Option<String> {
wayland_candidates(wire)
.iter()
.find(|c| available.iter().any(|a| a == *c))
.map(|c| c.to_string())
}
/// Normalize a raw Wayland offer's MIME list into the deduplicated wire MIME list announced to the
/// client (drops internal targets; collapses aliases; preserves a stable order).
#[cfg(target_os = "linux")]
pub fn offer_wire_mimes(raw: &[String]) -> Vec<&'static str> {
let mut out: Vec<&'static str> = Vec::new();
for m in raw {
if let Some(wire) = wayland_to_wire(m) {
if !out.contains(&wire) {
out.push(wire);
}
}
}
out
}
/// The Wayland MIMEs to advertise when installing a source for a client's offer. Each wire MIME
/// expands to its canonical Wayland name(s); a rich-text-only offer also advertises `text/plain`
/// so plain-text targets always paste (§3.5 synthesis — destination-side, one direction only).
#[cfg(target_os = "linux")]
pub fn wayland_offers_for(wire_mimes: &[String]) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
let mut push = |s: &str| {
if !out.iter().any(|o| o == s) {
out.push(s.to_string());
}
};
let mut has_plain = false;
let mut has_rich = false;
for w in wire_mimes {
match w.as_str() {
WIRE_TEXT => {
has_plain = true;
push("text/plain;charset=utf-8");
push("text/plain");
push("UTF8_STRING");
push("STRING");
}
WIRE_HTML => {
has_rich = true;
push("text/html");
}
WIRE_RTF => {
has_rich = true;
push("text/rtf");
}
WIRE_PNG => push("image/png"),
other => push(other),
}
}
// Synthesis: rich text without plain text → also advertise plain (the source derives it lazily).
if has_rich && !has_plain {
push("text/plain;charset=utf-8");
push("text/plain");
push("UTF8_STRING");
push("STRING");
}
out
}
#[cfg(all(test, target_os = "linux"))]
mod tests {
use super::*;
#[test]
fn wayland_to_wire_canonicalizes_and_drops_targets() {
assert_eq!(wayland_to_wire("text/plain"), Some(WIRE_TEXT));
assert_eq!(wayland_to_wire("UTF8_STRING"), Some(WIRE_TEXT));
assert_eq!(wayland_to_wire("text/plain;charset=utf-8"), Some(WIRE_TEXT));
assert_eq!(wayland_to_wire("text/html"), Some(WIRE_HTML));
assert_eq!(wayland_to_wire("application/rtf"), Some(WIRE_RTF));
assert_eq!(wayland_to_wire("image/png"), Some(WIRE_PNG));
// Internal targets and unsupported formats are dropped.
assert_eq!(wayland_to_wire("TARGETS"), None);
assert_eq!(wayland_to_wire("TIMESTAMP"), None);
assert_eq!(wayland_to_wire("image/jpeg"), None);
}
#[test]
fn offer_wire_mimes_dedups_aliases() {
let raw = vec![
"TARGETS".to_string(),
"UTF8_STRING".to_string(),
"text/plain;charset=utf-8".to_string(),
"text/plain".to_string(),
"text/html".to_string(),
];
// text aliases collapse to one WIRE_TEXT; TARGETS dropped; html kept.
assert_eq!(offer_wire_mimes(&raw), vec![WIRE_TEXT, WIRE_HTML]);
}
#[test]
fn pick_wayland_mime_prefers_canonical() {
let avail = vec!["text/plain".to_string(), "UTF8_STRING".to_string()];
// Canonical charset form isn't present, so it falls to the next candidate.
assert_eq!(
pick_wayland_mime(WIRE_TEXT, &avail),
Some("text/plain".to_string())
);
let avail2 = vec![
"text/plain;charset=utf-8".to_string(),
"text/plain".to_string(),
];
assert_eq!(
pick_wayland_mime(WIRE_TEXT, &avail2),
Some("text/plain;charset=utf-8".to_string())
);
assert_eq!(pick_wayland_mime(WIRE_PNG, &avail2), None);
}
#[test]
fn wayland_offers_synthesizes_plain_for_rich_only() {
let offers = wayland_offers_for(&[WIRE_HTML.to_string()]);
assert!(offers.iter().any(|m| m == "text/html"));
assert!(
offers.iter().any(|m| m == "text/plain;charset=utf-8"),
"rich-only offer must synthesize plain text: {offers:?}"
);
// Plain already present → no duplicate synthesis, and text aliases included.
let offers2 = wayland_offers_for(&[WIRE_TEXT.to_string()]);
assert!(offers2.iter().any(|m| m == "UTF8_STRING"));
assert_eq!(offers2.iter().filter(|m| *m == "text/plain").count(), 1);
}
}
@@ -0,0 +1,524 @@
//! GNOME clipboard backend via Mutter's **direct** `org.gnome.Mutter.RemoteDesktop.Session` D-Bus
//! API (`design/clipboard-and-file-transfer.md` §4.1).
//!
//! Mutter implements no wlr/ext `data-control` (a deliberate privacy stance), so [`super::wayland`]
//! can't bind on GNOME. But Mutter's own RemoteDesktop session — the same interface the input
//! injector drives directly to dodge the xdg-portal approval dialog (`inject/linux/libei.rs`
//! `connect_mutter`) — carries the full clipboard surface: `EnableClipboard`, `SetSelection`,
//! `SelectionRead`/`SelectionWrite`/`SelectionWriteDone`, and the `SelectionOwnerChanged` /
//! `SelectionTransfer` signals. We open our **own** standalone session for it (it coexists with the
//! injector's input session; validated on GNOME 50), so this backend is self-contained just like the
//! data-control one.
//!
//! One actor task owns the zbus connection + session and multiplexes the two signals with a command
//! channel; the fds Mutter hands out are **non-blocking**, so reads/writes flip them to blocking and
//! run on a blocking thread. Option/signal dict keys are hyphenated: `mime-types`, `session-is-owner`.
use std::collections::HashMap;
use std::io::{Read as _, Write as _};
use std::os::fd::{AsRawFd, OwnedFd};
use std::sync::{Arc, Mutex};
use anyhow::{anyhow, Context, Result};
use ashpd::zbus::{
self,
zvariant::{OwnedObjectPath, OwnedValue, Value},
};
use futures_util::StreamExt;
use tokio::sync::{mpsc, oneshot};
use super::{ClipEvent, PasteResponder};
const RD_BUS: &str = "org.gnome.Mutter.RemoteDesktop";
const RD_PATH: &str = "/org/gnome/Mutter/RemoteDesktop";
const RD_IFACE: &str = "org.gnome.Mutter.RemoteDesktop";
const SESSION_IFACE: &str = "org.gnome.Mutter.RemoteDesktop.Session";
/// Upper bound on one `SelectionRead` (matches the wire clipboard cap, §7).
const CLIP_READ_CAP: u64 = 64 << 20;
/// Handle to the Mutter clipboard actor, held (inside a [`super::HostClipboard`]) by the session
/// coordinator.
pub struct MutterClipboard {
cmd_tx: mpsc::UnboundedSender<Cmd>,
/// Raw MIMEs the current host selection advertises (empty when we own it, or nothing is copied).
/// Written by the actor on `SelectionOwnerChanged`; read for `current_wire_mimes` / fetches.
current_raw: Arc<Mutex<Vec<String>>>,
}
enum Cmd {
SetOffer(Vec<String>),
ClearOffer,
ReadCurrent {
wire: String,
reply: oneshot::Sender<Result<Vec<u8>>>,
},
}
impl MutterClipboard {
/// Create a standalone Mutter RemoteDesktop session, `Start` + `EnableClipboard` it, and spawn
/// the actor. Errors when Mutter isn't running (not GNOME) — the caller falls through to
/// `BACKEND_UNAVAILABLE`.
pub async fn open() -> Result<(MutterClipboard, mpsc::UnboundedReceiver<ClipEvent>)> {
let conn = zbus::Connection::session()
.await
.context("session D-Bus (Mutter clipboard)")?;
let rd = zbus::Proxy::new(&conn, RD_BUS, RD_PATH, RD_IFACE)
.await
.context("Mutter RemoteDesktop proxy (is gnome-shell running?)")?;
let session_path: OwnedObjectPath = rd
.call("CreateSession", &())
.await
.context("Mutter RemoteDesktop.CreateSession (clipboard)")?;
let session = zbus::Proxy::new(&conn, RD_BUS, session_path, SESSION_IFACE)
.await
.context("Mutter RemoteDesktop.Session proxy")?;
session
.call_method("Start", &())
.await
.context("Mutter RemoteDesktop.Session.Start (clipboard)")?;
let empty: HashMap<&str, Value> = HashMap::new();
session
.call_method("EnableClipboard", &(empty,))
.await
.context("Mutter EnableClipboard")?;
let (event_tx, event_rx) = mpsc::unbounded_channel();
let (cmd_tx, cmd_rx) = mpsc::unbounded_channel();
let current_raw = Arc::new(Mutex::new(Vec::new()));
tokio::spawn(actor(conn, session, cmd_rx, event_tx, current_raw.clone()));
tracing::info!("clipboard backend bound (Mutter RemoteDesktop direct)");
Ok((
MutterClipboard {
cmd_tx,
current_raw,
},
event_rx,
))
}
/// Install a client's offered formats as the host selection (fire-and-forget on the actor).
pub fn set_offer(&self, wire_mimes: &[String]) {
let _ = self.cmd_tx.send(Cmd::SetOffer(wire_mimes.to_vec()));
}
/// Relinquish the selection we own.
pub fn clear_offer(&self) {
let _ = self.cmd_tx.send(Cmd::ClearOffer);
}
/// The current host selection's wire MIMEs (empty = nothing / we own it).
pub fn current_wire_mimes(&self) -> Vec<String> {
super::offer_wire_mimes(&self.current_raw.lock().unwrap())
.into_iter()
.map(str::to_string)
.collect()
}
/// Read one wire format of the current host selection (a client's fetch). Round-trips the actor
/// (SelectionRead + a blocking fd read).
pub async fn read_current(&self, wire: &str) -> Result<Vec<u8>> {
let (reply, rx) = oneshot::channel();
self.cmd_tx
.send(Cmd::ReadCurrent {
wire: wire.to_string(),
reply,
})
.map_err(|_| anyhow!("Mutter clipboard actor gone"))?;
rx.await
.map_err(|_| anyhow!("Mutter clipboard read dropped"))?
}
}
/// The actor: owns the connection + session, subscribes to the two clipboard signals, and serves
/// commands. Exits when the command channel closes (session ending) or a signal stream ends.
async fn actor(
conn: zbus::Connection,
session: zbus::Proxy<'static>,
mut cmd_rx: mpsc::UnboundedReceiver<Cmd>,
event_tx: mpsc::UnboundedSender<ClipEvent>,
current_raw: Arc<Mutex<Vec<String>>>,
) {
let (mut owner, mut transfer) = match (
session.receive_signal("SelectionOwnerChanged").await,
session.receive_signal("SelectionTransfer").await,
) {
(Ok(o), Ok(t)) => (o, t),
_ => {
tracing::warn!("Mutter clipboard: could not subscribe to selection signals");
let _ = event_tx.send(ClipEvent::Closed);
return;
}
};
loop {
tokio::select! {
sig = owner.next() => {
let Some(msg) = sig else { break };
let Ok((opts,)) = msg.body().deserialize::<(HashMap<String, OwnedValue>,)>() else {
continue;
};
let is_owner = dict_bool(&opts, "session-is-owner").unwrap_or(false);
let raw = dict_mimes(&opts, "mime-types");
if is_owner {
// Our own offer (the client's content) — not host clipboard; don't announce it,
// and clear `current_raw` so a fetch never reads our own source back.
current_raw.lock().unwrap().clear();
} else {
*current_raw.lock().unwrap() = raw.clone();
let wire = super::offer_wire_mimes(&raw)
.into_iter()
.map(str::to_string)
.collect();
if event_tx.send(ClipEvent::Selection { mimes: wire }).is_err() {
break;
}
}
}
sig = transfer.next() => {
let Some(msg) = sig else { break };
let Ok((mime, serial)) = msg.body().deserialize::<(String, u32)>() else {
continue;
};
match super::wayland_to_wire(&mime) {
Some(wire) => {
// A host app pastes our offer: hand the fetch to the coordinator, then serve
// the returned bytes into the SelectionWrite fd off-task. NB Mutter issues
// *two* transfers per read (a size probe + the real read), so the coordinator
// fetches from the client twice per paste — correct, just not deduplicated.
let (tx, rx) = oneshot::channel();
if event_tx
.send(ClipEvent::Paste {
mime: wire.to_string(),
responder: PasteResponder::Channel(tx),
})
.is_err()
{
break;
}
let session = session.clone();
tokio::spawn(async move {
let bytes = rx.await.unwrap_or_default();
serve_write(&session, serial, bytes).await;
});
}
// Format we don't sync — fail the transfer cleanly.
None => serve_write(&session, serial, Vec::new()).await,
}
}
cmd = cmd_rx.recv() => {
let Some(cmd) = cmd else { break }; // coordinator gone → session ending
match cmd {
Cmd::SetOffer(wire) => {
let wl = super::wayland_offers_for(&wire);
if let Err(e) = set_selection(&session, &wl).await {
tracing::debug!(error = %e, "Mutter SetSelection failed");
}
}
Cmd::ClearOffer => {
if let Err(e) = set_selection(&session, &[]).await {
tracing::debug!(error = %e, "Mutter clear selection failed");
}
}
Cmd::ReadCurrent { wire, reply } => {
let raw = current_raw.lock().unwrap().clone();
let _ = reply.send(read_selection(&session, &wire, &raw).await);
}
}
}
}
}
// Keep the connection owned for the actor's whole life (Mutter ties the session to it).
drop(conn);
let _ = event_tx.send(ClipEvent::Closed);
}
/// Offer `wl_mimes` as the host selection; an empty list relinquishes ownership.
async fn set_selection(session: &zbus::Proxy<'_>, wl_mimes: &[String]) -> Result<()> {
let mut opts: HashMap<&str, Value> = HashMap::new();
if !wl_mimes.is_empty() {
let refs: Vec<&str> = wl_mimes.iter().map(String::as_str).collect();
opts.insert("mime-types", Value::from(refs));
}
session
.call_method("SetSelection", &(opts,))
.await
.context("Mutter SetSelection")?;
Ok(())
}
/// Read the current selection's `wire` format: pick a concrete offered MIME, `SelectionRead` it, and
/// read the (non-blocking) fd to EOF on a blocking thread.
async fn read_selection(session: &zbus::Proxy<'_>, wire: &str, raw: &[String]) -> Result<Vec<u8>> {
let mime =
super::pick_wayland_mime(wire, raw).context("format not offered by the host clipboard")?;
let fd: zbus::zvariant::OwnedFd = session
.call("SelectionRead", &(mime.as_str(),))
.await
.context("Mutter SelectionRead")?;
let fd = OwnedFd::from(fd);
tokio::task::spawn_blocking(move || read_fd_to_end(fd))
.await
.map_err(|e| anyhow!("SelectionRead join: {e}"))?
}
/// Serve one `SelectionTransfer`: `SelectionWrite` → write `bytes` → `SelectionWriteDone`. Any write
/// failure still reports done (success=false) so Mutter completes the transfer.
async fn serve_write(session: &zbus::Proxy<'_>, serial: u32, bytes: Vec<u8>) {
let ok = match write_selection(session, serial, bytes).await {
Ok(()) => true,
Err(e) => {
tracing::debug!(error = %e, "Mutter SelectionWrite failed");
false
}
};
let _ = session
.call_method("SelectionWriteDone", &(serial, ok))
.await;
}
async fn write_selection(session: &zbus::Proxy<'_>, serial: u32, bytes: Vec<u8>) -> Result<()> {
let fd: zbus::zvariant::OwnedFd = session
.call("SelectionWrite", &(serial,))
.await
.context("Mutter SelectionWrite")?;
let fd = OwnedFd::from(fd);
tokio::task::spawn_blocking(move || write_fd(fd, &bytes))
.await
.map_err(|e| anyhow!("SelectionWrite join: {e}"))?
}
/// Read a Mutter clipboard fd to EOF (capped). The fd is `O_NONBLOCK`; flip it to blocking first.
fn read_fd_to_end(fd: OwnedFd) -> Result<Vec<u8>> {
set_blocking(&fd)?;
let file = std::fs::File::from(fd);
let mut buf = Vec::new();
file.take(CLIP_READ_CAP)
.read_to_end(&mut buf)
.context("read SelectionRead fd")?;
Ok(buf)
}
/// Write `bytes` into a Mutter clipboard fd and close it (EOF). Flip the `O_NONBLOCK` fd to blocking.
fn write_fd(fd: OwnedFd, bytes: &[u8]) -> Result<()> {
set_blocking(&fd)?;
let mut file = std::fs::File::from(fd);
file.write_all(bytes).context("write SelectionWrite fd")?;
Ok(())
}
/// Peel any `Value::Value` (variant) wrappers to the concrete value. The `a{sv}` dict values Mutter
/// sends arrive as variants, so a plain `TryFrom<OwnedValue>` (which matches the concrete type) never
/// sees through them — this strips the layer first.
fn peel<'a>(v: &'a Value<'a>) -> &'a Value<'a> {
let mut cur = v;
while let Value::Value(inner) = cur {
cur = inner;
}
cur
}
/// Extract a boolean dict entry (e.g. `session-is-owner`), unwrapping the variant.
fn dict_bool(opts: &HashMap<String, OwnedValue>, key: &str) -> Option<bool> {
match peel(opts.get(key)?) {
Value::Bool(b) => Some(*b),
_ => None,
}
}
/// Extract a string-array dict entry (e.g. `mime-types`), unwrapping the variant. Mutter wraps the
/// array in a single-field struct (`(as)`, seen on `SelectionOwnerChanged`), so unwrap that too.
fn dict_mimes(opts: &HashMap<String, OwnedValue>, key: &str) -> Vec<String> {
let Some(v) = opts.get(key) else {
return Vec::new();
};
let mut val = peel(v);
if let Value::Structure(s) = val {
match s.fields().first() {
Some(first) => val = peel(first),
None => return Vec::new(),
}
}
let Value::Array(arr) = val else {
return Vec::new();
};
arr.inner()
.iter()
.filter_map(|e| match peel(e) {
Value::Str(s) => Some(s.to_string()),
_ => None,
})
.collect()
}
/// Clear `O_NONBLOCK` on a Mutter clipboard fd so a blocking `spawn_blocking` read/write works.
fn set_blocking(fd: &OwnedFd) -> Result<()> {
let raw = fd.as_raw_fd();
// SAFETY: `raw` is a valid fd owned by `fd` for the duration of these fcntl calls.
let flags = unsafe { libc::fcntl(raw, libc::F_GETFL) };
if flags < 0 {
return Err(anyhow!(
"fcntl F_GETFL: {}",
std::io::Error::last_os_error()
));
}
// SAFETY: as above; clearing O_NONBLOCK on our own fd.
let rc = unsafe { libc::fcntl(raw, libc::F_SETFL, flags & !libc::O_NONBLOCK) };
if rc < 0 {
return Err(anyhow!(
"fcntl F_SETFL: {}",
std::io::Error::last_os_error()
));
}
Ok(())
}
/// On-glass tests against a **live GNOME/Mutter** session (`WAYLAND_DISPLAY=wayland-0`). `#[ignore]`d
/// — run explicitly under GNOME (Mutter has no `wl-clipboard`, so a second Mutter session plays the
/// "host app"):
///
/// ```text
/// cargo test -p punktfunk-host --bin punktfunk-host -- --ignored --nocapture clipboard::mutter::live
/// ```
///
/// Skips (does not fail) when Mutter isn't running, so `--ignored` off-GNOME is a clean no-op.
#[cfg(test)]
mod live {
use super::*;
use std::time::Duration;
/// A second Mutter session standing in for a host clipboard app.
struct Helper {
session: zbus::Proxy<'static>,
_conn: zbus::Connection,
}
impl Helper {
async fn open() -> Result<Helper> {
let conn = zbus::Connection::session().await?;
let rd = zbus::Proxy::new(&conn, RD_BUS, RD_PATH, RD_IFACE).await?;
let path: OwnedObjectPath = rd.call("CreateSession", &()).await?;
let session = zbus::Proxy::new(&conn, RD_BUS, path, SESSION_IFACE).await?;
session.call_method("Start", &()).await?;
let empty: HashMap<&str, Value> = HashMap::new();
session.call_method("EnableClipboard", &(empty,)).await?;
Ok(Helper {
session,
_conn: conn,
})
}
/// Own the selection offering plain text, serving `payload` on every transfer request.
async fn offer_text(&self, payload: &'static [u8]) {
let mut transfer = self
.session
.receive_signal("SelectionTransfer")
.await
.unwrap();
let session = self.session.clone();
tokio::spawn(async move {
while let Some(msg) = transfer.next().await {
if let Ok((_mime, serial)) = msg.body().deserialize::<(String, u32)>() {
serve_write(&session, serial, payload.to_vec()).await;
}
}
});
set_selection(
&self.session,
&[
"text/plain;charset=utf-8".to_string(),
"text/plain".to_string(),
],
)
.await
.unwrap();
}
/// Paste the current selection's plain text.
async fn read_text(&self) -> Vec<u8> {
let fd: zbus::zvariant::OwnedFd = self
.session
.call("SelectionRead", &("text/plain;charset=utf-8",))
.await
.unwrap();
let fd = OwnedFd::from(fd);
tokio::task::spawn_blocking(move || read_fd_to_end(fd))
.await
.unwrap()
.unwrap()
}
}
async fn next_selection(
rx: &mut mpsc::UnboundedReceiver<ClipEvent>,
timeout: Duration,
) -> Option<Vec<String>> {
tokio::time::timeout(timeout, async {
loop {
match rx.recv().await {
Some(ClipEvent::Selection { mimes }) if !mimes.is_empty() => {
return Some(mimes)
}
Some(_) => continue,
None => return None,
}
}
})
.await
.ok()
.flatten()
}
#[test]
#[ignore = "needs a live GNOME/Mutter session (WAYLAND_DISPLAY=wayland-0)"]
fn live_mutter_roundtrip() {
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.unwrap();
rt.block_on(async {
let (backend, mut rx) = match MutterClipboard::open().await {
Ok(v) => v,
Err(e) => {
eprintln!("SKIP: no Mutter clipboard (not GNOME?): {e:#}");
return;
}
};
let helper = Helper::open().await.expect("helper Mutter session");
// --- host copy → backend observes Selection + read_current returns the bytes ---
helper.offer_text(b"gnome-host-copied").await;
let mimes = next_selection(&mut rx, Duration::from_secs(3))
.await
.expect("Selection after the helper offered text");
assert!(
mimes.iter().any(|m| m == super::super::WIRE_TEXT),
"offer carries wire text: {mimes:?}"
);
let got = backend
.read_current(super::super::WIRE_TEXT)
.await
.expect("read_current text");
assert_eq!(got, b"gnome-host-copied");
// --- backend offers client content → the host app (helper) pastes it ---
backend.set_offer(&[super::super::WIRE_TEXT.to_string()]);
tokio::time::sleep(Duration::from_millis(500)).await; // let SetSelection take effect
// Answer every Paste request the host app (helper) triggers, until the read completes.
let paste_side = async {
while let Some(ev) = rx.recv().await {
if let ClipEvent::Paste { responder, .. } = ev {
responder.respond(b"punktfunk-served".to_vec()).await;
}
}
};
let read = tokio::select! {
r = helper.read_text() => r,
_ = paste_side => Vec::new(),
};
assert_eq!(read, b"punktfunk-served");
});
}
}
@@ -0,0 +1,254 @@
//! Host clipboard coordinator (`design/clipboard-and-file-transfer.md` §4.2).
//!
//! One async task per streaming session that bridges the real session clipboard (whichever
//! [`super::HostClipboard`] backend this platform opened) to the QUIC clipboard plane. It owns all
//! four data paths:
//!
//! * **host copy → client** — a backend [`ClipEvent::Selection`] becomes a [`ClipOffer`] pushed to the
//! control loop (`offer_tx`), which forwards it to the client.
//! * **client fetch of the host clipboard** — the fetch-stream `accept_bi` loop lives here; each
//! stream is answered by reading the current host selection ([`ClipboardBackend::read_current`]).
//! * **client copy → host** — a [`ClipCoordCmd::RemoteOffer`] installs the client's formats as a lazy
//! host selection ([`HostClipboard::set_offer`]).
//! * **host paste of client content** — a backend [`ClipEvent::Paste`] triggers an *outbound* fetch to
//! the client, whose bytes are handed to the backend's [`PasteResponder`].
//!
//! The coordinator is backend-agnostic (Linux data-control / Mutter, Windows Win32); the control loop
//! reaches it through the portable [`ClipCoordCmd`] channel so `punktfunk1` compiles on every host
//! platform.
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::mpsc::{UnboundedReceiver, UnboundedSender};
use punktfunk_core::clipboard::CLIP_FETCH_CAP;
use punktfunk_core::quic::{
clipstream, ClipFetch, ClipFetchHdr, ClipKind, ClipOffer, CLIP_FETCH_OK, CLIP_FETCH_STALE,
CLIP_FETCH_UNAVAILABLE, CLIP_FILE_INDEX_NONE,
};
use super::{ClipEvent, HostClipboard, PasteResponder};
use crate::punktfunk1::ClipCoordCmd;
/// Upper bound on one outbound fetch (host pasting client content). A client that never answers must
/// not hang the pasting host app's pipe read (§3.4) — the paste falls back to empty instead.
const FETCH_TIMEOUT: Duration = Duration::from_secs(60);
/// Open whichever host clipboard backend this session supports (data-control, else Mutter direct) and
/// spawn the coordinator. Returns `true` when a live backend was bound (the caller's control loop
/// then serves real clipboard data); `false` when none is available (gamescope, no live compositor),
/// in which case the channels are dropped so the control loop reports `CLIP_REASON_BACKEND_UNAVAILABLE`
/// and declines fetches defensively.
pub async fn start(
conn: quinn::Connection,
clip_enabled: Arc<AtomicBool>,
cmd_rx: UnboundedReceiver<ClipCoordCmd>,
offer_tx: UnboundedSender<ClipOffer>,
) -> bool {
match HostClipboard::open().await {
Ok((backend, clip_rx)) => {
tokio::spawn(run(
conn,
Arc::new(backend),
clip_rx,
clip_enabled,
cmd_rx,
offer_tx,
));
true
}
Err(e) => {
tracing::info!(error = %format!("{e:#}"), "clipboard backend unavailable — fetches will be declined");
false
}
}
}
/// The coordinator loop. Multiplexes control-loop commands, backend clipboard events, and inbound
/// fetch streams; exits when any of the three peers goes away (session ending).
async fn run(
conn: quinn::Connection,
backend: Arc<HostClipboard>,
mut clip_rx: UnboundedReceiver<ClipEvent>,
clip_enabled: Arc<AtomicBool>,
mut cmd_rx: UnboundedReceiver<ClipCoordCmd>,
offer_tx: UnboundedSender<ClipOffer>,
) {
// Seq of the offer the host most recently announced; a client fetch naming a different seq is
// stale (the host clipboard moved on) and is declined.
let host_seq = Arc::new(AtomicU32::new(0));
let mut next_seq: u32 = 1;
// Seq of the client's most recent offer, echoed on the outbound fetch we open when a host app
// pastes client content (informational for the client's serve side).
let mut client_seq: u32 = 0;
loop {
tokio::select! {
cmd = cmd_rx.recv() => {
let Some(cmd) = cmd else { break }; // control loop gone → session ending
match cmd {
ClipCoordCmd::SetEnabled(true) => {
// A just-enabled client should see whatever the host already has copied.
let mimes = backend.current_wire_mimes();
if !mimes.is_empty() {
let _ = offer_tx.send(build_offer(&mut next_seq, &host_seq, mimes));
}
}
ClipCoordCmd::SetEnabled(false) => {
if let Err(e) = backend.clear_offer() {
tracing::debug!(error = %e, "clipboard clear_offer failed");
}
}
ClipCoordCmd::RemoteOffer { seq, mimes } => {
client_seq = seq;
let res = if mimes.is_empty() {
backend.clear_offer()
} else {
backend.set_offer(&mimes)
};
if let Err(e) = res {
tracing::debug!(error = %e, "clipboard apply remote offer failed");
}
}
}
}
ev = clip_rx.recv() => {
let Some(ev) = ev else { break }; // backend dispatch thread ended
match ev {
ClipEvent::Selection { mimes } => {
// Forward host copies (empty `mimes` = the clipboard was cleared) only while
// the client has sync on — the offer is metadata; bytes still cross lazily.
if clip_enabled.load(Ordering::SeqCst) {
let _ = offer_tx.send(build_offer(&mut next_seq, &host_seq, mimes));
}
}
ClipEvent::Paste { mime, responder } => {
// A host app is pasting the client's offered content: pull that format from
// the client and hand it to the backend's responder. Off-task so the loop
// keeps serving.
tokio::spawn(fetch_into_pipe(conn.clone(), client_seq, mime, responder));
}
ClipEvent::Closed => break,
}
}
accepted = conn.accept_bi() => {
let Ok((send, recv)) = accepted else { break }; // connection gone
// The control stream is already accepted at the handshake, so every stream here is a
// clipboard fetch. Serve it off-task (the read blocks on the source app's pipe).
tokio::spawn(serve_fetch(
send,
recv,
Arc::clone(&backend),
Arc::clone(&host_seq),
clip_enabled.load(Ordering::SeqCst),
));
}
}
}
// Session ending: don't leave our lazy source as the compositor's active selection.
let _ = backend.clear_offer();
}
/// Mint a [`ClipOffer`] for `mimes`, advancing the host offer seq (skipping 0, the "never offered"
/// sentinel) and publishing it as the current one for staleness checks.
fn build_offer(next_seq: &mut u32, host_seq: &AtomicU32, mimes: Vec<String>) -> ClipOffer {
let seq = *next_seq;
*next_seq = next_seq.wrapping_add(1);
if *next_seq == 0 {
*next_seq = 1;
}
host_seq.store(seq, Ordering::SeqCst);
let kinds = mimes
.into_iter()
.map(|mime| ClipKind { mime, size_hint: 0 })
.collect();
ClipOffer { seq, kinds }
}
/// Serve one inbound fetch stream (a client pulling the host clipboard): validate the header +
/// request, then answer with the current host selection's bytes for the requested wire MIME.
async fn serve_fetch(
mut send: quinn::SendStream,
mut recv: quinn::RecvStream,
backend: Arc<HostClipboard>,
host_seq: Arc<AtomicU32>,
enabled: bool,
) {
let _ = send.set_priority(-1);
match clipstream::read_stream_header(&mut recv).await {
Ok(k) if k == clipstream::CLIP_STREAM_KIND_FETCH => {}
_ => {
let _ = send.reset(clipstream::cancelled_code());
return;
}
}
let req = match clipstream::read_fetch(&mut recv).await {
Ok(r) => r,
Err(_) => return,
};
let decline = |status: u8| ClipFetchHdr {
status,
total_size: 0,
};
if !enabled {
let _ = clipstream::write_fetch_hdr(&mut send, &decline(CLIP_FETCH_UNAVAILABLE)).await;
return;
}
if req.seq != host_seq.load(Ordering::SeqCst) {
let _ = clipstream::write_fetch_hdr(&mut send, &decline(CLIP_FETCH_STALE)).await;
return;
}
// `read_current` reads the host selection (a blocking pipe read, offloaded by the backend).
match backend.read_current(&req.mime).await {
Ok(data) => {
let hdr = ClipFetchHdr {
status: CLIP_FETCH_OK,
total_size: data.len() as u64,
};
if clipstream::write_fetch_hdr(&mut send, &hdr).await.is_ok() {
let _ = clipstream::write_data(&mut send, &data).await;
}
}
// The format vanished (clipboard changed mid-fetch) or the read failed → nothing to send.
Err(_) => {
let _ = clipstream::write_fetch_hdr(&mut send, &decline(CLIP_FETCH_UNAVAILABLE)).await;
}
}
}
/// Pull `mime` of the client's current offer (`seq`) over an outbound fetch stream and hand the bytes
/// to the backend's paste `responder`. Any failure (timeout, decline, I/O) responds with empty bytes
/// so the pasting host app gets an empty paste instead of hanging.
async fn fetch_into_pipe(
conn: quinn::Connection,
seq: u32,
mime: String,
responder: PasteResponder,
) {
let req = ClipFetch {
seq,
file_index: CLIP_FILE_INDEX_NONE,
mime,
};
let fetched = tokio::time::timeout(FETCH_TIMEOUT, async {
let (send, mut recv) = clipstream::open_fetch(&conn, &req).await.ok()?;
let hdr = clipstream::read_fetch_hdr(&mut recv).await.ok()?;
if hdr.status != CLIP_FETCH_OK {
return None;
}
let data = clipstream::read_data(&mut recv, CLIP_FETCH_CAP)
.await
.ok()?;
drop(send); // clean close of our half
Some(data)
})
.await
.ok()
.flatten();
responder.respond(fetched.unwrap_or_default()).await;
}
@@ -0,0 +1,599 @@
//! `ext-data-control-v1` clipboard backend (`design/clipboard-and-file-transfer.md` §4.1).
//!
//! A dedicated thread owns the `wayland-client` [`EventQueue`] and runs a poll loop that dispatches
//! selection + paste events, emitting them over a channel. Everything else — installing a lazy
//! source (a client's offer) and `receive()`-ing the host selection (a client's fetch) — is issued
//! from the session thread on the shared, `Send + Sync` proxy handles; only *dispatch* is
//! single-threaded (per the wayland-client contract). Templated on `inject/linux/wlr.rs`.
//!
//! The `zwlr-data-control-unstable-v1` fallback for older wlroots/KWin is a mechanical parallel of
//! this file (the protocols are 1:1) — a follow-up.
use std::collections::HashMap;
use std::io::Read;
use std::os::fd::{AsFd, AsRawFd, FromRawFd, OwnedFd};
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::{Arc, Mutex};
use anyhow::{anyhow, Context, Result};
use wayland_client::backend::ObjectId;
use wayland_client::protocol::wl_registry;
use wayland_client::protocol::wl_seat::WlSeat;
use wayland_client::{event_created_child, Connection, Dispatch, Proxy, QueueHandle};
use wayland_protocols::ext::data_control::v1::client::{
ext_data_control_device_v1::{self, ExtDataControlDeviceV1},
ext_data_control_manager_v1::ExtDataControlManagerV1,
ext_data_control_offer_v1::{self, ExtDataControlOfferV1},
ext_data_control_source_v1::{self, ExtDataControlSourceV1},
};
use super::{ClipEvent, PasteResponder};
/// Upper bound on bytes read from one `receive()` transfer (matches the wire clipboard cap, §7) so a
/// hostile host app can't stream unboundedly into our buffer.
const CLIP_READ_CAP: u64 = 64 << 20;
/// The current host selection, shared between the dispatch thread (writer) and the session thread
/// (reader, for `receive()`).
struct CurrentSelection {
offer: ExtDataControlOfferV1,
/// Raw Wayland MIMEs the offer advertises (what `receive()` accepts).
mimes: Vec<String>,
}
/// Dispatch-thread state. Also collects the manager + seat during the bind roundtrip.
struct State {
mgr: Option<ExtDataControlManagerV1>,
seat: Option<WlSeat>,
/// Offers accumulating their MIME list before the `selection` event promotes one.
pending: HashMap<ObjectId, Vec<String>>,
current: Arc<Mutex<Option<CurrentSelection>>>,
/// Pending count of our own `set_selection`s whose `selection` echo must be dropped rather than
/// announced back to the client (loop prevention, §3.4). Bumped by the session before each set;
/// each of our sets produces exactly one echo on wlroots/KWin, so one decrement per echo pairs
/// them up — a counter (not a bool) keeps rapid back-to-back offers from leaking a self-echo.
suppress_echoes: Arc<AtomicU32>,
tx: tokio::sync::mpsc::UnboundedSender<ClipEvent>,
}
impl Dispatch<wl_registry::WlRegistry, ()> for State {
fn event(
state: &mut Self,
registry: &wl_registry::WlRegistry,
event: wl_registry::Event,
_: &(),
_: &Connection,
qh: &QueueHandle<Self>,
) {
if let wl_registry::Event::Global {
name,
interface,
version,
} = event
{
match interface.as_str() {
"ext_data_control_manager_v1" => {
state.mgr = Some(registry.bind(name, version.min(1), qh, ()));
}
"wl_seat" => {
state.seat = Some(registry.bind(name, version.min(7), qh, ()));
}
_ => {}
}
}
}
}
// Manager + seat emit nothing we consume.
impl Dispatch<ExtDataControlManagerV1, ()> for State {
fn event(
_: &mut Self,
_: &ExtDataControlManagerV1,
_: <ExtDataControlManagerV1 as Proxy>::Event,
_: &(),
_: &Connection,
_: &QueueHandle<Self>,
) {
}
}
impl Dispatch<WlSeat, ()> for State {
fn event(
_: &mut Self,
_: &WlSeat,
_: <WlSeat as Proxy>::Event,
_: &(),
_: &Connection,
_: &QueueHandle<Self>,
) {
}
}
impl Dispatch<ExtDataControlDeviceV1, ()> for State {
fn event(
state: &mut Self,
_dev: &ExtDataControlDeviceV1,
event: ext_data_control_device_v1::Event,
_: &(),
_: &Connection,
_: &QueueHandle<Self>,
) {
use ext_data_control_device_v1::Event;
match event {
// A new offer is being introduced; its `offer` events follow before `selection`.
Event::DataOffer { id } => {
state.pending.insert(id.id(), Vec::new());
}
// The active selection changed. `Some` = a new clipboard; `None` = cleared.
Event::Selection { id } => {
// Consume one pending self-echo if any (atomic vs. the session thread's bumps; the
// dispatch thread is the only decrementer). `Ok` = there was one → suppress.
let suppressed = state
.suppress_echoes
.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |c| c.checked_sub(1))
.is_ok();
match id {
Some(offer) => {
let mimes = state.pending.remove(&offer.id()).unwrap_or_default();
if suppressed {
// Our own source's echo — don't store it as the host clipboard and
// don't announce it back to the client.
return;
}
let wire = super::offer_wire_mimes(&mimes)
.into_iter()
.map(str::to_string)
.collect::<Vec<_>>();
*state.current.lock().unwrap() = Some(CurrentSelection { offer, mimes });
let _ = state.tx.send(ClipEvent::Selection { mimes: wire });
}
None => {
*state.current.lock().unwrap() = None;
if !suppressed {
let _ = state.tx.send(ClipEvent::Selection { mimes: Vec::new() });
}
}
}
}
Event::Finished => {
let _ = state.tx.send(ClipEvent::Closed);
}
// Primary selection is out of scope for the shared clipboard.
_ => {}
}
}
event_created_child!(State, ExtDataControlDeviceV1, [
ext_data_control_device_v1::EVT_DATA_OFFER_OPCODE => (ExtDataControlOfferV1, ()),
]);
}
impl Dispatch<ExtDataControlOfferV1, ()> for State {
fn event(
state: &mut Self,
offer: &ExtDataControlOfferV1,
event: ext_data_control_offer_v1::Event,
_: &(),
_: &Connection,
_: &QueueHandle<Self>,
) {
if let ext_data_control_offer_v1::Event::Offer { mime_type } = event {
if let Some(list) = state.pending.get_mut(&offer.id()) {
list.push(mime_type);
}
}
}
}
impl Dispatch<ExtDataControlSourceV1, ()> for State {
fn event(
state: &mut Self,
_src: &ExtDataControlSourceV1,
event: ext_data_control_source_v1::Event,
_: &(),
_: &Connection,
_: &QueueHandle<Self>,
) {
use ext_data_control_source_v1::Event;
match event {
// A host app pasted our (the client's) offered data.
Event::Send { mime_type, fd } => match super::wayland_to_wire(&mime_type) {
Some(wire) => {
let _ = state.tx.send(ClipEvent::Paste {
mime: wire.to_string(),
responder: PasteResponder::Fd(fd),
});
}
// We can't satisfy this format — closing the fd yields an empty paste.
None => drop(fd),
},
// Our source was superseded (a host app or another client set a new selection).
Event::Cancelled => {}
_ => {}
}
}
}
/// The host clipboard backend handle used by the session thread.
pub struct ClipboardBackend {
conn: Connection,
mgr: ExtDataControlManagerV1,
device: ExtDataControlDeviceV1,
qh: QueueHandle<State>,
current: Arc<Mutex<Option<CurrentSelection>>>,
suppress_echoes: Arc<AtomicU32>,
active_source: Mutex<Option<ExtDataControlSourceV1>>,
stop: Arc<AtomicBool>,
thread: Option<std::thread::JoinHandle<()>>,
}
impl ClipboardBackend {
/// Connect to the active session's Wayland display (env already applied by
/// `vdisplay::apply_session_env`), bind `ext_data_control`, and start the dispatch thread.
/// Returns the handle plus the event stream. Errors if the compositor lacks the protocol
/// (caller reports `BackendUnavailable`).
pub fn open() -> Result<(
ClipboardBackend,
tokio::sync::mpsc::UnboundedReceiver<ClipEvent>,
)> {
let conn = Connection::connect_to_env()
.context("connect to Wayland for clipboard (WAYLAND_DISPLAY/XDG_RUNTIME_DIR set?)")?;
let mut queue = conn.new_event_queue();
let qh = queue.handle();
let _registry = conn.display().get_registry(&qh, ());
let (tx, rx) = tokio::sync::mpsc::unbounded_channel();
let current = Arc::new(Mutex::new(None));
let suppress_echoes = Arc::new(AtomicU32::new(0));
let mut state = State {
mgr: None,
seat: None,
pending: HashMap::new(),
current: current.clone(),
suppress_echoes: suppress_echoes.clone(),
tx,
};
queue
.roundtrip(&mut state)
.context("Wayland registry roundtrip")?;
let mgr = state
.mgr
.clone()
.context("compositor lacks ext_data_control_manager_v1")?;
let seat = state
.seat
.clone()
.context("compositor advertised no wl_seat")?;
let device = mgr.get_data_device(&seat, &qh, ());
// Second roundtrip: the compositor sends the initial selection for the freshly-bound device
// (the current host clipboard), which the session announces to the client.
queue
.roundtrip(&mut state)
.context("Wayland get_data_device roundtrip")?;
let stop = Arc::new(AtomicBool::new(false));
let thread = {
let conn = conn.clone();
let stop = stop.clone();
std::thread::Builder::new()
.name("punktfunk-clipboard".into())
.spawn(move || dispatch_loop(conn, queue, state, stop))
.context("spawn clipboard dispatch thread")?
};
Ok((
ClipboardBackend {
conn,
mgr,
device,
qh,
current,
suppress_echoes,
active_source: Mutex::new(None),
stop,
thread: Some(thread),
},
rx,
))
}
/// Install a lazy source advertising a client's offered formats (wire MIMEs) as the host
/// selection. A later host-app paste fires a [`ClipEvent::Paste`]. Replaces any previous offer.
pub fn set_offer(&self, wire_mimes: &[String]) -> Result<()> {
let wl_mimes = super::wayland_offers_for(wire_mimes);
if wl_mimes.is_empty() {
return self.clear_offer();
}
let src = self.mgr.create_data_source(&self.qh, ());
for m in &wl_mimes {
src.offer(m.clone());
}
// Suppress the selection echo our own set triggers (loop prevention).
self.suppress_echoes.fetch_add(1, Ordering::SeqCst);
self.device.set_selection(Some(&src));
self.conn.flush().context("flush set_selection")?;
let mut slot = self.active_source.lock().unwrap();
if let Some(old) = slot.take() {
old.destroy();
}
*slot = Some(src);
Ok(())
}
/// Drop the host selection we own (client disabled sync / offered nothing).
pub fn clear_offer(&self) -> Result<()> {
let mut slot = self.active_source.lock().unwrap();
if let Some(old) = slot.take() {
self.suppress_echoes.fetch_add(1, Ordering::SeqCst);
self.device.set_selection(None);
old.destroy();
self.conn.flush().context("flush clear selection")?;
}
Ok(())
}
/// The current host selection's wire MIMEs (what a client offer announcement would carry), or
/// empty if the clipboard is empty. Used to answer an immediate query.
pub fn current_wire_mimes(&self) -> Vec<String> {
match self.current.lock().unwrap().as_ref() {
Some(sel) => super::offer_wire_mimes(&sel.mimes)
.into_iter()
.map(str::to_string)
.collect(),
None => Vec::new(),
}
}
/// Read one format (`wire_mime`) of the current host selection into a byte vector — a client's
/// lazy fetch. BLOCKS on the pipe until the source app finishes, so call from a blocking
/// context (e.g. `spawn_blocking`). Errors if there is no selection or the format isn't offered.
pub fn read_current(&self, wire_mime: &str) -> Result<Vec<u8>> {
let (offer, wl_mime) = {
let cur = self.current.lock().unwrap();
let sel = cur.as_ref().context("no current host selection")?;
let wl = super::pick_wayland_mime(wire_mime, &sel.mimes)
.context("format not offered by the host clipboard")?;
(sel.offer.clone(), wl)
};
let (read_fd, write_fd) = make_pipe()?;
offer.receive(wl_mime, write_fd.as_fd());
self.conn.flush().context("flush receive")?;
// Close our write end so the pipe reaches EOF once the source app closes its dup.
drop(write_fd);
let mut buf = Vec::new();
// `read_fd` is a fresh, uniquely-owned pipe read end; `File` takes sole ownership and closes
// it on drop.
let file = std::fs::File::from(read_fd);
file.take(CLIP_READ_CAP)
.read_to_end(&mut buf)
.context("read clipboard transfer")?;
Ok(buf)
}
}
impl Drop for ClipboardBackend {
fn drop(&mut self) {
self.stop.store(true, Ordering::SeqCst);
if let Some(t) = self.thread.take() {
let _ = t.join();
}
}
}
/// The dispatch thread: poll the Wayland socket with a short timeout so `stop` is honored promptly,
/// dispatching selection/paste events into `state`.
fn dispatch_loop(
conn: Connection,
mut queue: wayland_client::EventQueue<State>,
mut state: State,
stop: Arc<AtomicBool>,
) {
while !stop.load(Ordering::SeqCst) {
if queue.dispatch_pending(&mut state).is_err() {
break;
}
if conn.flush().is_err() {
break;
}
let Some(guard) = conn.prepare_read() else {
// Events are already queued; loop to dispatch them.
continue;
};
let raw_fd = guard.connection_fd().as_raw_fd();
let mut pfd = libc::pollfd {
fd: raw_fd,
events: libc::POLLIN,
revents: 0,
};
// SAFETY: `pfd` is a single valid pollfd; `poll` reads/writes exactly it for 200 ms.
let rc = unsafe { libc::poll(&mut pfd, 1, 200) };
if rc < 0 {
let err = std::io::Error::last_os_error();
drop(guard);
if err.kind() == std::io::ErrorKind::Interrupted {
continue; // EINTR — recheck stop, retry
}
break;
}
if rc == 0 {
drop(guard); // timeout — recheck stop
continue;
}
if pfd.revents & libc::POLLIN != 0 {
if guard.read().is_err() {
break;
}
} else {
drop(guard); // POLLHUP / POLLERR — connection gone
break;
}
}
let _ = state.tx.send(ClipEvent::Closed);
}
/// Create a `pipe2(O_CLOEXEC)`, returning `(read_end, write_end)` as owned fds.
fn make_pipe() -> Result<(OwnedFd, OwnedFd)> {
let mut fds = [0 as libc::c_int; 2];
// SAFETY: `pipe2` fully initializes the 2-element `fds` on success (returns 0); on failure (-1)
// we bail before reading it. Each returned fd is fresh and owned by exactly one `OwnedFd`.
let rc = unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC) };
if rc < 0 {
return Err(anyhow!("pipe2 failed: {}", std::io::Error::last_os_error()));
}
// SAFETY: `fds[0]`/`fds[1]` are the fresh, uniquely-owned pipe ends from the checked `pipe2`.
let read_fd = unsafe { OwnedFd::from_raw_fd(fds[0]) };
// SAFETY: as above for the write end.
let write_fd = unsafe { OwnedFd::from_raw_fd(fds[1]) };
Ok((read_fd, write_fd))
}
/// On-glass tests against a **live** `data-control` compositor (Hyprland / Sway / KWin). `#[ignore]`d
/// — run explicitly under such a session with `wl-clipboard` present:
///
/// ```text
/// WAYLAND_DISPLAY=wayland-1 cargo test -p punktfunk-host --bin punktfunk-host \
/// -- --ignored --nocapture clipboard::wayland::live
/// ```
///
/// Each test skips (does not fail) when `open()` finds no backend — so `--ignored` on GNOME (no
/// data-control) or a headless CI runner is a clean no-op instead of a false failure.
#[cfg(test)]
mod live {
use super::*;
use std::io::Write as _;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
/// Poll the event channel (sync `try_recv`, no runtime) until `pred` matches or `timeout`.
fn wait_event(
rx: &mut tokio::sync::mpsc::UnboundedReceiver<ClipEvent>,
timeout: Duration,
mut pred: impl FnMut(&ClipEvent) -> bool,
) -> Option<ClipEvent> {
let deadline = Instant::now() + timeout;
loop {
match rx.try_recv() {
Ok(ev) if pred(&ev) => return Some(ev),
Ok(_) => {}
Err(tokio::sync::mpsc::error::TryRecvError::Empty) => {
if Instant::now() >= deadline {
return None;
}
std::thread::sleep(Duration::from_millis(20));
}
Err(tokio::sync::mpsc::error::TryRecvError::Disconnected) => return None,
}
}
}
/// Set the compositor selection from a "host app" (`wl-copy`, which forks a server that holds it).
fn wl_copy(bytes: &[u8], mime: &str) {
let mut child = Command::new("wl-copy")
.arg("--type")
.arg(mime)
.stdin(Stdio::piped())
.spawn()
.expect("spawn wl-copy");
child
.stdin
.take()
.unwrap()
.write_all(bytes)
.expect("write to wl-copy");
let _ = child.wait(); // foreground exits; the fork keeps serving
std::thread::sleep(Duration::from_millis(150));
}
fn open_or_skip() -> Option<(
ClipboardBackend,
tokio::sync::mpsc::UnboundedReceiver<ClipEvent>,
)> {
if Command::new("wl-copy").arg("--version").output().is_err() {
eprintln!("SKIP: wl-clipboard not installed");
return None;
}
match ClipboardBackend::open() {
Ok(v) => Some(v),
Err(e) => {
eprintln!("SKIP: no data-control backend on this compositor: {e:#}");
None
}
}
}
/// Host copy → we observe a `Selection` and can `read_current` the exact bytes back — both text
/// and PNG (§3.5 format normalization end to end).
#[test]
#[ignore = "needs a live data-control compositor (WAYLAND_DISPLAY)"]
fn live_host_copy_is_readable() {
let Some((backend, mut rx)) = open_or_skip() else {
return;
};
// Text.
wl_copy(b"hello-from-host-app", "text/plain;charset=utf-8");
let ev = wait_event(&mut rx, Duration::from_secs(3), |e| {
matches!(e, ClipEvent::Selection { mimes } if mimes.iter().any(|m| m == super::super::WIRE_TEXT))
})
.expect("Selection event carrying text after wl-copy");
assert!(matches!(ev, ClipEvent::Selection { .. }));
assert_eq!(
backend.read_current(super::super::WIRE_TEXT).unwrap(),
b"hello-from-host-app"
);
// PNG (arbitrary bytes tagged image/png — data-control is format-agnostic).
let png = b"\x89PNG\r\n\x1a\n-fake-but-tagged-image/png";
wl_copy(png, "image/png");
wait_event(&mut rx, Duration::from_secs(3), |e| {
matches!(e, ClipEvent::Selection { mimes } if mimes.iter().any(|m| m == super::super::WIRE_PNG))
})
.expect("Selection event carrying image/png");
assert_eq!(backend.read_current(super::super::WIRE_PNG).unwrap(), png);
}
/// We install a client's offer as the host selection; a host app (`wl-paste`) pasting it fires a
/// `Paste` event that we fulfill with bytes, and the host app receives exactly those bytes.
#[test]
#[ignore = "needs a live data-control compositor (WAYLAND_DISPLAY)"]
fn live_set_offer_is_pasteable() {
let Some((backend, mut rx)) = open_or_skip() else {
return;
};
backend
.set_offer(&[super::super::WIRE_TEXT.to_string()])
.expect("install offer");
// A host app pastes our offered selection.
let child = Command::new("wl-paste")
.arg("-n")
.stdout(Stdio::piped())
.spawn()
.expect("spawn wl-paste");
let paste = wait_event(&mut rx, Duration::from_secs(3), |e| {
matches!(e, ClipEvent::Paste { .. })
})
.expect("Paste event after wl-paste reads our offer");
match paste {
ClipEvent::Paste { mime, responder } => {
assert_eq!(
mime,
super::super::WIRE_TEXT,
"paste requested the text format"
);
match responder {
PasteResponder::Fd(fd) => {
super::super::fulfill_paste(fd, b"served-by-punktfunk").expect("fulfill");
}
PasteResponder::Channel(_) => panic!("data-control paste must carry an fd"),
}
}
_ => unreachable!(),
}
let out = child.wait_with_output().expect("wl-paste output");
assert_eq!(out.stdout, b"served-by-punktfunk");
}
}
@@ -0,0 +1,664 @@
//! Host-side shared-clipboard backend for Windows (`design/clipboard-and-file-transfer.md` §4, Phase
//! 3). The Win32 clipboard is thread-affine and message-driven, so the whole backend lives on one
//! dedicated **message-loop thread** owning a hidden message-only window:
//!
//! * **host copy → client** — `AddClipboardFormatListener` delivers `WM_CLIPBOARDUPDATE`; we map the
//! available formats to wire MIMEs, cache them, and emit [`ClipEvent::Selection`]. Our own offers
//! are suppressed by the owner-check (we never forward what we ourselves put on the clipboard).
//! * **client fetch of the host clipboard** — a `Cmd::Read` reads the requested format's HGLOBAL and
//! converts it to wire bytes ([`super::winfmt`]).
//! * **client copy → host** — a `Cmd::SetOffer` installs the client's formats via OLE **delayed
//! rendering**: `SetClipboardData(fmt, NULL)`, so no bytes cross until a host app actually pastes.
//! * **host paste of client content** — the paste triggers `WM_RENDERFORMAT`; the message-loop thread
//! blocks (bounded) while the coordinator fetches the bytes from the client, then `SetClipboardData`s
//! them for the pasting app.
//!
//! The async coordinator drives the thread through a [`Cmd`] channel woken by `PostMessage(WM_APP_CMD)`
//! (`PostMessage` is the documented thread-safe way to poke a message loop). Per-window state hangs
//! off `GWLP_USERDATA`, so multiple concurrent sessions each get their own window + state.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
use std::cell::RefCell;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use anyhow::Context as _;
use ::windows::core::{w, PCWSTR};
use ::windows::Win32::Foundation::{
GetLastError, GlobalFree, HANDLE, HGLOBAL, HINSTANCE, HWND, LPARAM, LRESULT, WPARAM,
};
use ::windows::Win32::System::DataExchange::{
AddClipboardFormatListener, CloseClipboard, EmptyClipboard, GetClipboardData,
GetClipboardOwner, IsClipboardFormatAvailable, OpenClipboard, RegisterClipboardFormatW,
SetClipboardData,
};
use ::windows::Win32::System::LibraryLoader::GetModuleHandleW;
use ::windows::Win32::System::Memory::{
GlobalAlloc, GlobalLock, GlobalSize, GlobalUnlock, GMEM_MOVEABLE, GMEM_ZEROINIT,
};
use ::windows::Win32::System::Ole::CF_UNICODETEXT;
use ::windows::Win32::UI::WindowsAndMessaging::{
CreateWindowExW, DefWindowProcW, DestroyWindow, DispatchMessageW, GetMessageW,
GetWindowLongPtrW, PostMessageW, PostQuitMessage, RegisterClassW, SetWindowLongPtrW,
TranslateMessage, GWLP_USERDATA, HWND_MESSAGE, MSG, WINDOW_EX_STYLE, WINDOW_STYLE, WM_APP,
WM_CLIPBOARDUPDATE, WM_DESTROY, WM_RENDERFORMAT, WNDCLASSW,
};
use super::winfmt;
use super::{ClipEvent, PasteResponder, WIRE_HTML, WIRE_PNG, WIRE_RTF, WIRE_TEXT};
/// Custom app message that wakes the pump to drain the [`Cmd`] channel.
const WM_APP_CMD: u32 = WM_APP + 1;
/// Upper bound the message-loop thread waits for the client's bytes during a `WM_RENDERFORMAT` paste.
/// The pasting app is frozen until we answer, so this caps how long a paste can hang; on expiry the
/// format is left unrendered (an empty paste) rather than blocking indefinitely.
const RENDER_TIMEOUT: Duration = Duration::from_secs(10);
/// `OpenClipboard` fails while another process transiently holds the clipboard (clipboard managers do
/// this constantly); retry briefly before giving up.
const OPEN_RETRIES: u32 = 20;
const OPEN_RETRY_DELAY: Duration = Duration::from_millis(5);
/// `RegisterClassW` returns this when the (process-global) class already exists — expected on the 2nd+
/// concurrent session, and not an error (we never unregister the class).
const ERROR_CLASS_ALREADY_EXISTS: u32 = 1410;
type ClipTx = tokio::sync::mpsc::UnboundedSender<ClipEvent>;
/// A command from the async coordinator into the message-loop thread. Delivered over a tokio channel
/// and drained on `WM_APP_CMD`.
enum Cmd {
/// Install the client's wire MIMEs as a delayed-render host selection (empty ⇒ clear).
SetOffer(Vec<String>),
/// Drop the selection we own.
Clear,
/// Read one wire format of the current host selection for a client fetch.
Read {
wire: String,
resp: tokio::sync::oneshot::Sender<anyhow::Result<Vec<u8>>>,
},
/// Tear the window + thread down.
Shutdown,
}
/// Message-loop-thread-owned state, reached from the `WndProc` via `GWLP_USERDATA`. Only the message
/// thread ever dereferences it, so the `RefCell`s are sound (no cross-thread sharing); the fields the
/// async handle also touches (`current_wire`) are behind their own `Arc<Mutex>`.
struct WinClip {
/// Backend → coordinator events.
clip_tx: ClipTx,
/// The current host selection's wire MIMEs, shared with the [`WindowsClipboard`] handle.
current_wire: Arc<Mutex<Vec<String>>>,
/// Coordinator → backend commands, drained on `WM_APP_CMD`.
cmd_rx: RefCell<tokio::sync::mpsc::UnboundedReceiver<Cmd>>,
/// Clipboard format ids we currently promise via delayed rendering (for `WM_RENDERFORMAT`).
offered: RefCell<Vec<u32>>,
fmt_html: u32,
fmt_rtf: u32,
fmt_png: u32,
/// Our own message window — used for the owner-check and clipboard opens.
own_hwnd: HWND,
}
impl WinClip {
/// `WM_CLIPBOARDUPDATE`: a host app copied (or the clipboard was cleared). Suppress our own
/// delayed-render echoes via the owner-check, else announce the new wire MIMEs.
fn on_clipboard_update(&self, hwnd: HWND) {
// SAFETY: GetClipboardOwner has no preconditions and needs no open clipboard.
let owner = unsafe { GetClipboardOwner() }.unwrap_or_default();
if owner.0 == hwnd.0 {
// Our own offer's echo (we own the clipboard) — not a host copy.
return;
}
let mimes = self.available_wire_mimes();
*self.current_wire.lock().unwrap() = mimes.clone();
let _ = self.clip_tx.send(ClipEvent::Selection { mimes });
}
/// The wire MIMEs the current clipboard advertises, in a stable order.
fn available_wire_mimes(&self) -> Vec<String> {
// SAFETY: IsClipboardFormatAvailable has no preconditions and needs no open clipboard.
let avail = |fmt: u32| unsafe { IsClipboardFormatAvailable(fmt) }.is_ok();
let mut out = Vec::new();
if avail(CF_UNICODETEXT.0 as u32) {
out.push(WIRE_TEXT.to_string());
}
if avail(self.fmt_html) {
out.push(WIRE_HTML.to_string());
}
if avail(self.fmt_rtf) {
out.push(WIRE_RTF.to_string());
}
if avail(self.fmt_png) {
out.push(WIRE_PNG.to_string());
}
out
}
/// `WM_APP_CMD`: run every queued coordinator command on this thread. Drained into a `Vec` first so
/// the `cmd_rx` borrow is released before any command runs (defensive against re-entry).
fn drain_commands(&self, hwnd: HWND) {
let mut cmds = Vec::new();
{
let mut rx = self.cmd_rx.borrow_mut();
while let Ok(c) = rx.try_recv() {
cmds.push(c);
}
}
for c in cmds {
match c {
Cmd::SetOffer(wire) => self.apply_offer(hwnd, &wire),
Cmd::Clear => self.clear(hwnd),
Cmd::Read { wire, resp } => {
let _ = resp.send(self.read(&wire));
}
Cmd::Shutdown => {
// Drop our offer first so no WM_RENDERALLFORMATS fires as the window dies (we do
// NOT want the client's content to outlive the session on the host clipboard).
self.clear(hwnd);
// SAFETY: our own live window; triggers WM_DESTROY → PostQuitMessage → pump exit.
unsafe {
let _ = DestroyWindow(hwnd);
}
return;
}
}
}
}
/// Install the client's offer as a delayed-render host selection.
fn apply_offer(&self, hwnd: HWND, wire: &[String]) {
let fmts = self.formats_for_offer(wire);
if fmts.is_empty() {
self.clear(hwnd);
return;
}
if open_clipboard_retry(hwnd).is_err() {
tracing::debug!("clipboard: OpenClipboard for set_offer failed");
return;
}
let _guard = ClipboardGuard;
// SAFETY: the clipboard is open (ClipboardGuard closes it); EmptyClipboard makes us the owner,
// then each SetClipboardData(_, None) registers a delayed-render promise for that format.
unsafe {
let _ = EmptyClipboard();
for &f in &fmts {
let _ = SetClipboardData(f, None);
}
}
*self.offered.borrow_mut() = fmts;
}
/// Drop the selection we own (empty the clipboard iff we're still its owner).
fn clear(&self, hwnd: HWND) {
let had = {
let mut o = self.offered.borrow_mut();
let was = !o.is_empty();
o.clear();
was
};
if !had {
return;
}
// SAFETY: GetClipboardOwner has no preconditions.
let owner = unsafe { GetClipboardOwner() }.unwrap_or_default();
if owner.0 != hwnd.0 {
return; // someone else took the clipboard already
}
if open_clipboard_retry(hwnd).is_err() {
return;
}
let _guard = ClipboardGuard;
// SAFETY: the clipboard is open (ClipboardGuard closes it); empty it to drop our promises.
unsafe {
let _ = EmptyClipboard();
}
}
/// Read one wire format of the current host selection (a client fetch).
fn read(&self, wire: &str) -> anyhow::Result<Vec<u8>> {
let fmt = self
.format_for_wire(wire)
.context("unsupported wire MIME")?;
// If we own the clipboard, its content is our own delayed-render offer (the client's copy),
// not a host selection — declining avoids GetClipboardData re-entering our own WM_RENDERFORMAT.
// SAFETY: GetClipboardOwner has no preconditions.
if unsafe { GetClipboardOwner() }.unwrap_or_default().0 == self.own_hwnd.0 {
anyhow::bail!("clipboard currently held by our own offer");
}
open_clipboard_retry(self.own_hwnd)?;
let _guard = ClipboardGuard;
// SAFETY: the clipboard is open (ClipboardGuard closes it). GetClipboardData hands back a
// clipboard-owned HGLOBAL (we must NOT free it); GlobalLock/Size/Unlock are balanced and we
// copy exactly GlobalSize bytes out before the lock is released.
let raw = unsafe {
let handle = GetClipboardData(fmt).context("GetClipboardData")?;
let hg = HGLOBAL(handle.0);
let p = GlobalLock(hg);
if p.is_null() {
anyhow::bail!("GlobalLock failed");
}
let n = GlobalSize(hg);
let mut buf = vec![0u8; n];
std::ptr::copy_nonoverlapping(p as *const u8, buf.as_mut_ptr(), n);
let _ = GlobalUnlock(hg);
buf
};
Ok(convert_from_win(wire, &raw))
}
/// `WM_RENDERFORMAT`: a host app is pasting a format we promised. Fetch the bytes from the client
/// (blocking this thread, bounded) and `SetClipboardData` them for the paster.
fn on_render_format(&self, fmt: u32) {
let Some(wire) = self.wire_for_format(fmt) else {
return;
};
let (tx, rx) = std::sync::mpsc::channel::<Vec<u8>>();
let ev = ClipEvent::Paste {
mime: wire.to_string(),
responder: PasteResponder::Sync(tx),
};
if self.clip_tx.send(ev).is_err() {
return; // coordinator gone
}
let bytes = match rx.recv_timeout(RENDER_TIMEOUT) {
Ok(b) => b,
Err(_) => return, // timeout / dropped → leave the format unrendered (empty paste)
};
let win_bytes = convert_to_win(wire, &bytes);
let Ok(hg) = alloc_hglobal(&win_bytes) else {
return;
};
// Do NOT OpenClipboard here — the pasting app already holds it open across WM_RENDERFORMAT.
// SAFETY: `hg` is a freshly-filled moveable HGLOBAL. On success the clipboard takes ownership
// (we must not free it); on failure ownership stays with us, so we free it.
unsafe {
if SetClipboardData(fmt, Some(HANDLE(hg.0))).is_err() {
let _ = GlobalFree(Some(hg));
}
}
}
/// The Win32 clipboard format id for a wire MIME (`None` = unsupported).
fn format_for_wire(&self, wire: &str) -> Option<u32> {
match wire {
WIRE_TEXT => Some(CF_UNICODETEXT.0 as u32),
WIRE_HTML => Some(self.fmt_html),
WIRE_RTF => Some(self.fmt_rtf),
WIRE_PNG => Some(self.fmt_png),
_ => None,
}
}
/// The wire MIME for a Win32 clipboard format id (`None` = one we don't offer).
fn wire_for_format(&self, fmt: u32) -> Option<&'static str> {
if fmt == CF_UNICODETEXT.0 as u32 {
Some(WIRE_TEXT)
} else if fmt == self.fmt_html {
Some(WIRE_HTML)
} else if fmt == self.fmt_rtf {
Some(WIRE_RTF)
} else if fmt == self.fmt_png {
Some(WIRE_PNG)
} else {
None
}
}
/// The clipboard format ids to promise for a client offer (dedup, 1:1 with the wire MIMEs — the OS
/// auto-synthesizes CF_TEXT/CF_OEMTEXT from CF_UNICODETEXT, so no manual text fan-out is needed).
fn formats_for_offer(&self, wire: &[String]) -> Vec<u32> {
let mut out = Vec::new();
for w in wire {
if let Some(f) = self.format_for_wire(w) {
if !out.contains(&f) {
out.push(f);
}
}
}
out
}
}
/// The active Windows clipboard backend handle held by [`super::HostClipboard`]. All Win32 work runs
/// on the message-loop thread; this is just the async-side control surface.
pub struct WindowsClipboard {
cmd_tx: tokio::sync::mpsc::UnboundedSender<Cmd>,
/// The message window's `HWND` as an `isize` (so the handle stays `Send`/`Sync`); rebuilt for the
/// `PostMessage` wakeups, which are documented thread-safe.
hwnd: isize,
current_wire: Arc<Mutex<Vec<String>>>,
join: Option<std::thread::JoinHandle<()>>,
}
impl WindowsClipboard {
/// Spin up the message-loop thread + hidden window and return once it has bound (or failed).
pub async fn open() -> anyhow::Result<(
WindowsClipboard,
tokio::sync::mpsc::UnboundedReceiver<ClipEvent>,
)> {
let (clip_tx, clip_rx) = tokio::sync::mpsc::unbounded_channel::<ClipEvent>();
let (cmd_tx, cmd_rx) = tokio::sync::mpsc::unbounded_channel::<Cmd>();
let current_wire = Arc::new(Mutex::new(Vec::new()));
// Register the three custom formats up front — process-global and thread-agnostic, so this is
// fine off the message thread and lets bring-up fail cleanly if the atoms can't be created.
let fmt_html = register_format(w!("HTML Format"))?;
let fmt_rtf = register_format(w!("Rich Text Format"))?;
let fmt_png = register_format(w!("PNG"))?;
let (ready_tx, ready_rx) = tokio::sync::oneshot::channel::<anyhow::Result<isize>>();
let cw = Arc::clone(&current_wire);
let join = std::thread::Builder::new()
.name("punktfunk-clipboard-win".into())
.spawn(move || pump_thread(clip_tx, cmd_rx, cw, fmt_html, fmt_rtf, fmt_png, ready_tx))
.context("spawn windows clipboard thread")?;
let hwnd = match tokio::time::timeout(Duration::from_secs(3), ready_rx).await {
Ok(Ok(Ok(h))) => h,
Ok(Ok(Err(e))) => return Err(e),
Ok(Err(_)) => anyhow::bail!("windows clipboard thread exited during bring-up"),
Err(_) => anyhow::bail!("windows clipboard bring-up timed out"),
};
Ok((
WindowsClipboard {
cmd_tx,
hwnd,
current_wire,
join: Some(join),
},
clip_rx,
))
}
/// The current host selection's wire MIMEs (empty = nothing to offer).
pub fn current_wire_mimes(&self) -> Vec<String> {
self.current_wire.lock().unwrap().clone()
}
/// Install a client's offered formats as the host selection (fire-and-forget onto the thread).
pub fn set_offer(&self, wire_mimes: &[String]) {
let _ = self.cmd_tx.send(Cmd::SetOffer(wire_mimes.to_vec()));
self.wake();
}
/// Drop the host selection we own (fire-and-forget onto the thread).
pub fn clear_offer(&self) {
let _ = self.cmd_tx.send(Cmd::Clear);
self.wake();
}
/// Read one wire format of the current host selection (a client's fetch).
pub async fn read_current(&self, wire_mime: &str) -> anyhow::Result<Vec<u8>> {
let (tx, rx) = tokio::sync::oneshot::channel();
self.cmd_tx
.send(Cmd::Read {
wire: wire_mime.to_string(),
resp: tx,
})
.map_err(|_| anyhow::anyhow!("clipboard thread gone"))?;
self.wake();
rx.await
.map_err(|_| anyhow::anyhow!("clipboard read dropped"))?
}
/// Poke the message loop so it drains the command channel.
fn wake(&self) {
// SAFETY: PostMessageW is documented thread-safe; `hwnd` is our message window (or already
// destroyed, in which case the post harmlessly fails and is ignored).
let _ = unsafe {
PostMessageW(
Some(HWND(self.hwnd as *mut core::ffi::c_void)),
WM_APP_CMD,
WPARAM(0),
LPARAM(0),
)
};
}
}
impl Drop for WindowsClipboard {
fn drop(&mut self) {
let _ = self.cmd_tx.send(Cmd::Shutdown);
self.wake();
if let Some(j) = self.join.take() {
let _ = j.join();
}
}
}
/// RAII `CloseClipboard` guard — pairs with a successful `open_clipboard_retry`, closing on scope exit
/// (including early `?`/`bail!` returns).
struct ClipboardGuard;
impl Drop for ClipboardGuard {
fn drop(&mut self) {
// SAFETY: constructed only after a successful OpenClipboard on this thread.
unsafe {
let _ = CloseClipboard();
}
}
}
/// Register (or resolve the existing id of) a custom clipboard format.
fn register_format(name: PCWSTR) -> anyhow::Result<u32> {
// SAFETY: RegisterClipboardFormatW is thread-agnostic and process-global; `name` is a static
// NUL-terminated wide literal.
let id = unsafe { RegisterClipboardFormatW(name) };
if id == 0 {
anyhow::bail!("RegisterClipboardFormatW failed");
}
Ok(id)
}
/// Allocate a moveable HGLOBAL holding `bytes` (zero-init so an empty payload is still a valid, locked
/// buffer). Ownership is transferred to the clipboard by a following `SetClipboardData`.
fn alloc_hglobal(bytes: &[u8]) -> anyhow::Result<HGLOBAL> {
// SAFETY: allocate at least one byte (GlobalLock of a 0-size block is unreliable), lock it, copy
// the payload in, unlock. Alloc/lock/unlock are balanced; on lock failure we free before erroring.
unsafe {
let hg = GlobalAlloc(GMEM_MOVEABLE | GMEM_ZEROINIT, bytes.len().max(1))
.context("GlobalAlloc")?;
let p = GlobalLock(hg);
if p.is_null() {
let _ = GlobalFree(Some(hg));
anyhow::bail!("GlobalLock failed");
}
std::ptr::copy_nonoverlapping(bytes.as_ptr(), p as *mut u8, bytes.len());
let _ = GlobalUnlock(hg);
Ok(hg)
}
}
/// `OpenClipboard(hwnd)` with a brief retry loop (another process often holds it transiently).
fn open_clipboard_retry(hwnd: HWND) -> anyhow::Result<()> {
for _ in 0..OPEN_RETRIES {
// SAFETY: OpenClipboard with our window as owner; balanced by ClipboardGuard/CloseClipboard.
if unsafe { OpenClipboard(Some(hwnd)) }.is_ok() {
return Ok(());
}
std::thread::sleep(OPEN_RETRY_DELAY);
}
anyhow::bail!("OpenClipboard failed after retries")
}
/// Convert a Win32 clipboard payload to wire bytes.
fn convert_from_win(wire: &str, raw: &[u8]) -> Vec<u8> {
match wire {
WIRE_TEXT => winfmt::text_from_utf16(raw),
WIRE_HTML => winfmt::html_from_cf(raw),
WIRE_RTF => winfmt::rtf_from_cf(raw),
_ => raw.to_vec(), // PNG + anything else: verbatim
}
}
/// Convert wire bytes to a Win32 clipboard payload.
fn convert_to_win(wire: &str, wire_bytes: &[u8]) -> Vec<u8> {
match wire {
WIRE_TEXT => winfmt::text_to_utf16(wire_bytes),
WIRE_HTML => winfmt::html_to_cf(wire_bytes),
_ => wire_bytes.to_vec(), // RTF + PNG + anything else: verbatim
}
}
/// Create the hidden message-only window (registering the class once, process-wide).
fn create_window() -> anyhow::Result<HWND> {
// SAFETY: standard window-class registration + message-only window creation; every argument is a
// valid handle / static literal, and `wndproc` matches the WNDPROC ABI.
unsafe {
let hinstance: HINSTANCE = GetModuleHandleW(PCWSTR::null())
.context("GetModuleHandleW")?
.into();
let class_name = w!("PunktfunkClipboardWindow");
let wc = WNDCLASSW {
lpfnWndProc: Some(wndproc),
hInstance: hinstance,
lpszClassName: class_name,
..Default::default()
};
if RegisterClassW(&wc) == 0 {
let code = GetLastError();
if code.0 != ERROR_CLASS_ALREADY_EXISTS {
anyhow::bail!("RegisterClassW failed: {code:?}");
}
}
let hwnd = CreateWindowExW(
WINDOW_EX_STYLE(0),
class_name,
w!(""),
WINDOW_STYLE(0),
0,
0,
0,
0,
Some(HWND_MESSAGE),
None,
Some(hinstance),
None,
)
.context("CreateWindowExW")?;
Ok(hwnd)
}
}
/// The message-loop thread body: build the window, wire up state, then pump until `WM_QUIT`.
fn pump_thread(
clip_tx: ClipTx,
cmd_rx: tokio::sync::mpsc::UnboundedReceiver<Cmd>,
current_wire: Arc<Mutex<Vec<String>>>,
fmt_html: u32,
fmt_rtf: u32,
fmt_png: u32,
ready_tx: tokio::sync::oneshot::Sender<anyhow::Result<isize>>,
) {
let hwnd = match create_window() {
Ok(h) => h,
Err(e) => {
let _ = ready_tx.send(Err(e));
return;
}
};
// A clone that outlives the boxed state, so we can announce Closed after the pump ends.
let closed_tx = clip_tx.clone();
let state = Box::new(WinClip {
clip_tx,
current_wire,
cmd_rx: RefCell::new(cmd_rx),
offered: RefCell::new(Vec::new()),
fmt_html,
fmt_rtf,
fmt_png,
own_hwnd: hwnd,
});
let ptr = Box::into_raw(state);
// SAFETY: stash the state pointer for the WndProc; the window was created on this thread and the
// pointer stays valid until we reclaim the Box after the pump exits.
unsafe {
SetWindowLongPtrW(hwnd, GWLP_USERDATA, ptr as isize);
}
// Snapshot whatever is already on the host clipboard, so the first client `enable` announces it
// (AddClipboardFormatListener only delivers *subsequent* changes).
{
// SAFETY: `ptr` is the live state we just stored; only this thread dereferences it.
let st = unsafe { &*ptr };
*st.current_wire.lock().unwrap() = st.available_wire_mimes();
}
// SAFETY: `hwnd` is our live window; start receiving WM_CLIPBOARDUPDATE.
if let Err(e) = unsafe { AddClipboardFormatListener(hwnd) } {
// SAFETY: tear down the half-built window and reclaim the leaked state box.
unsafe {
let _ = DestroyWindow(hwnd);
drop(Box::from_raw(ptr));
}
let _ = ready_tx.send(Err(
anyhow::Error::new(e).context("AddClipboardFormatListener")
));
return;
}
let _ = ready_tx.send(Ok(hwnd.0 as isize));
// SAFETY: the standard Win32 message pump. GetMessageW returns >0 for a message, 0 for WM_QUIT,
// and -1 on error — `.0 > 0` exits on both 0 and -1.
unsafe {
let mut msg = MSG::default();
while GetMessageW(&mut msg, None, 0, 0).0 > 0 {
let _ = TranslateMessage(&msg);
DispatchMessageW(&msg);
}
}
// Pump exited (window destroyed): reclaim the leaked state box. No WndProc runs after this point.
// SAFETY: `ptr` came from Box::into_raw above, is dereferenced only on this thread, and the
// message loop has ended so no further access occurs.
unsafe {
drop(Box::from_raw(ptr));
}
let _ = closed_tx.send(ClipEvent::Closed);
}
/// The window procedure. Reaches per-window state through `GWLP_USERDATA`; runs only on the message
/// thread. Registered as the class `WNDPROC` (a safe fn coerces to the `unsafe extern "system"` ABI).
extern "system" fn wndproc(hwnd: HWND, msg: u32, wparam: WPARAM, lparam: LPARAM) -> LRESULT {
// SAFETY: GWLP_USERDATA holds the `*const WinClip` stored right after window creation (0/null for
// the WM_(NC)CREATE messages that fire before that — handled by the null check below).
let ptr = unsafe { GetWindowLongPtrW(hwnd, GWLP_USERDATA) } as *const WinClip;
if ptr.is_null() {
// SAFETY: default processing before our state pointer is attached.
return unsafe { DefWindowProcW(hwnd, msg, wparam, lparam) };
}
// SAFETY: `ptr` is the live Box<WinClip> leaked in pump_thread, owned by this (the only) message
// thread and freed only after the pump exits; the WndProc is not re-entered for this window, so
// `&*ptr` is a valid shared borrow.
let st = unsafe { &*ptr };
match msg {
WM_CLIPBOARDUPDATE => {
st.on_clipboard_update(hwnd);
LRESULT(0)
}
WM_RENDERFORMAT => {
st.on_render_format(wparam.0 as u32);
LRESULT(0)
}
WM_APP_CMD => {
st.drain_commands(hwnd);
LRESULT(0)
}
WM_DESTROY => {
// SAFETY: ends the GetMessageW pump by posting WM_QUIT to this thread's queue.
unsafe {
PostQuitMessage(0);
}
LRESULT(0)
}
// SAFETY: default handling for every other message.
_ => unsafe { DefWindowProcW(hwnd, msg, wparam, lparam) },
}
}
@@ -0,0 +1,257 @@
//! Pure byte conversions between the Win32 clipboard formats and the portable wire MIMEs
//! (`design/clipboard-and-file-transfer.md` §3.5). Kept free of any `windows`-crate dependency so it
//! compiles on every host and its unit tests exercise the fiddly bits (CF_HTML offset math, UTF-16
//! (de)serialization) without a Windows box. The [`super::windows`] backend is the only production
//! consumer; it wraps these with the actual `GetClipboardData`/`SetClipboardData` calls.
//!
//! Format map (Win32 ↔ wire):
//! * `CF_UNICODETEXT` (UTF-16LE + NUL) ↔ `text/plain;charset=utf-8`
//! * `"HTML Format"` (CF_HTML, UTF-8 + ASCII header) ↔ `text/html`
//! * `"Rich Text Format"` (raw RTF) ↔ `text/rtf`
//! * `"PNG"` (raw PNG) ↔ `image/png` — identity, handled inline by the backend.
// ---- CF_UNICODETEXT ↔ text/plain;charset=utf-8 -----------------------------------------------
/// `CF_UNICODETEXT` HGLOBAL bytes → UTF-8 wire bytes. `raw` is the exact `GlobalSize`-length buffer;
/// it holds little-endian UTF-16 code units terminated by a single `0x0000`.
pub fn text_from_utf16(raw: &[u8]) -> Vec<u8> {
// Reinterpret each LE 2-byte pair as a UTF-16 code unit; a stray odd trailing byte (never present
// in valid data) is dropped by `chunks_exact`.
let mut units: Vec<u16> = raw
.chunks_exact(2)
.map(|c| u16::from_le_bytes([c[0], c[1]]))
.collect();
// Strip exactly one trailing NUL terminator if present (guard against eating a real code unit).
if units.last() == Some(&0) {
units.pop();
}
String::from_utf16_lossy(&units).into_bytes()
}
/// UTF-8 wire bytes → `CF_UNICODETEXT` HGLOBAL bytes (UTF-16LE + a required `0x0000` terminator).
pub fn text_to_utf16(wire: &[u8]) -> Vec<u8> {
let s = String::from_utf8_lossy(wire);
let mut out = Vec::with_capacity(wire.len() * 2 + 2);
for u in s.encode_utf16() {
out.extend_from_slice(&u.to_le_bytes());
}
out.extend_from_slice(&0u16.to_le_bytes()); // REQUIRED NUL terminator for CF_UNICODETEXT
out
}
// ---- "HTML Format" (CF_HTML) ↔ text/html -----------------------------------------------------
//
// CF_HTML is UTF-8: an ASCII `Key:Value\r\n` header carrying byte offsets, then the HTML with
// `<!--StartFragment-->`/`<!--EndFragment-->` markers. Offsets are byte counts from buffer start;
// the offsets live *inside* the header, so their digit-width feeds back into the header length. The
// spec-blessed fix (Chromium/Firefox/LibreOffice) is fixed-width 10-digit zero-padded offsets, which
// makes the header a compile-time constant and every offset a one-pass computation.
const CF_HTML_HEADER: &str = "Version:0.9\r\n\
StartHTML:0000000000\r\n\
EndHTML:0000000000\r\n\
StartFragment:0000000000\r\n\
EndFragment:0000000000\r\n";
const CF_HTML_PREFIX: &str = "<html><body>\r\n<!--StartFragment-->";
const CF_HTML_SUFFIX: &str = "<!--EndFragment-->\r\n</body></html>";
/// UTF-8 HTML fragment (wire bytes) → a `CF_HTML` buffer, NUL-terminated. The trailing NUL is the
/// conventional CF_HTML expectation (§4); `EndHTML` still points at content end, before the NUL.
pub fn html_to_cf(wire: &[u8]) -> Vec<u8> {
let fragment = String::from_utf8_lossy(wire);
let start_html = CF_HTML_HEADER.len(); // 105
let start_fragment = start_html + CF_HTML_PREFIX.len(); // 139
let end_fragment = start_fragment + fragment.len(); // byte length — fragment may be multibyte
let end_html = end_fragment + CF_HTML_SUFFIX.len();
let mut buf = Vec::with_capacity(end_html + 1);
buf.extend_from_slice(CF_HTML_HEADER.as_bytes());
buf.extend_from_slice(CF_HTML_PREFIX.as_bytes());
buf.extend_from_slice(fragment.as_bytes());
buf.extend_from_slice(CF_HTML_SUFFIX.as_bytes());
// Overwrite the four zero-padded fields in place, restricting the search to the header region so a
// fragment that happens to contain "StartHTML:" can't fool the patcher.
patch_offset(&mut buf[..start_html], b"StartHTML:", start_html);
patch_offset(&mut buf[..start_html], b"EndHTML:", end_html);
patch_offset(&mut buf[..start_html], b"StartFragment:", start_fragment);
patch_offset(&mut buf[..start_html], b"EndFragment:", end_fragment);
buf.push(0); // conventional NUL terminator
buf
}
/// A `CF_HTML` buffer → the UTF-8 HTML fragment (wire bytes). Uses the `StartFragment`/`EndFragment`
/// offsets; falls back to `StartHTML`/`EndHTML`, then to the whole buffer, if the markers are absent.
pub fn html_from_cf(raw: &[u8]) -> Vec<u8> {
let range = header_range(raw, b"StartFragment:", b"EndFragment:")
.or_else(|| header_range(raw, b"StartHTML:", b"EndHTML:"));
match range {
// Content is UTF-8 per spec; return the exact slice (drop any trailing NUL for cleanliness).
Some((start, end)) => {
let slice = &raw[start..end];
strip_trailing_nul(slice).to_vec()
}
None => strip_trailing_nul(raw).to_vec(),
}
}
/// Resolve `[start_label .. end_label]` into a validated byte range within `raw`.
fn header_range(raw: &[u8], start_label: &[u8], end_label: &[u8]) -> Option<(usize, usize)> {
let start = read_header_offset(raw, start_label)?;
let end = read_header_offset(raw, end_label)?;
if start <= end && end <= raw.len() {
Some((start, end))
} else {
None
}
}
/// Overwrite the 10 ASCII digits following `label` in `header` with `value`, zero-padded.
fn patch_offset(header: &mut [u8], label: &[u8], value: usize) {
if let Some(pos) = find(header, label) {
let at = pos + label.len();
if at + 10 <= header.len() {
let digits = format!("{value:010}");
header[at..at + 10].copy_from_slice(digits.as_bytes());
}
}
}
/// Read the decimal integer following `label:` in the ASCII header. The colon-suffixed labels only
/// match in the header, never the marker comments (`<!--StartFragment-->`) or fragment text.
fn read_header_offset(raw: &[u8], label: &[u8]) -> Option<usize> {
let mut at = find(raw, label)? + label.len();
let mut n: usize = 0;
let mut any = false;
while let Some(&b) = raw.get(at) {
if b.is_ascii_digit() {
n = n.checked_mul(10)?.checked_add((b - b'0') as usize)?;
any = true;
at += 1;
} else {
break; // stops at '\r'
}
}
any.then_some(n)
}
fn find(hay: &[u8], needle: &[u8]) -> Option<usize> {
if needle.is_empty() || needle.len() > hay.len() {
return None;
}
hay.windows(needle.len()).position(|w| w == needle)
}
// ---- "Rich Text Format" ↔ text/rtf -----------------------------------------------------------
/// `"Rich Text Format"` HGLOBAL bytes → RTF wire bytes. RTF is `{ }`-delimited; some producers append
/// a NUL past the final `}`, so strip a single trailing NUL to keep the wire payload byte-clean.
pub fn rtf_from_cf(raw: &[u8]) -> Vec<u8> {
strip_trailing_nul(raw).to_vec()
}
fn strip_trailing_nul(b: &[u8]) -> &[u8] {
match b.last() {
Some(0) => &b[..b.len() - 1],
_ => b,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn text_round_trips_and_handles_terminator() {
// UTF-8 → UTF-16LE+NUL → UTF-8.
let wire = "héllo 🌍".as_bytes();
let cf = text_to_utf16(wire);
// Ends with a single 0x0000 terminator.
assert_eq!(&cf[cf.len() - 2..], &[0, 0]);
assert_eq!(text_from_utf16(&cf), wire);
// A CF buffer *without* a terminator still decodes (no code unit eaten).
let no_term: Vec<u8> = "hi".encode_utf16().flat_map(u16::to_le_bytes).collect();
assert_eq!(text_from_utf16(&no_term), b"hi");
// Empty text → just the terminator → empty wire.
assert_eq!(text_to_utf16(b""), vec![0, 0]);
assert_eq!(text_from_utf16(&[0, 0]), b"");
}
#[test]
fn cf_html_matches_the_spec_offsets() {
// The worked example from the format reference: fragment "Hello".
let cf = html_to_cf(b"Hello");
let s = String::from_utf8(cf.clone()).unwrap();
assert!(s.contains("StartHTML:0000000105"), "{s}");
assert!(s.contains("EndHTML:0000000178"), "{s}");
assert!(s.contains("StartFragment:0000000139"), "{s}");
assert!(s.contains("EndFragment:0000000144"), "{s}");
// The declared fragment range must slice back to exactly "Hello".
let start = read_header_offset(&cf, b"StartFragment:").unwrap();
let end = read_header_offset(&cf, b"EndFragment:").unwrap();
assert_eq!(&cf[start..end], b"Hello");
// Trailing NUL present, and EndHTML points *before* it.
assert_eq!(*cf.last().unwrap(), 0);
assert_eq!(read_header_offset(&cf, b"EndHTML:").unwrap(), cf.len() - 1);
}
#[test]
fn cf_html_round_trips_including_multibyte() {
for frag in [
"Hello",
"<b>bold</b> & <i>ital</i>",
"café ☕ <span>x</span>",
"",
] {
let cf = html_to_cf(frag.as_bytes());
assert_eq!(html_from_cf(&cf), frag.as_bytes(), "fragment {frag:?}");
}
}
#[test]
fn cf_html_extract_tolerates_foreign_producers() {
// A producer that adds SourceURL and uses Version 1.0 — offsets must still drive extraction,
// never a hardcoded 105-byte header.
let fragment = "picked";
let prefix = "<html><body><!--StartFragment-->";
let header_body = format!(
"Version:1.0\r\nStartHTML:{sh:010}\r\nEndHTML:{eh:010}\r\n\
StartFragment:{sf:010}\r\nEndFragment:{ef:010}\r\nSourceURL:https://x/\r\n",
sh = 0,
eh = 0,
sf = 0,
ef = 0,
);
// Compute real offsets against this ad-hoc layout.
let start_html = header_body.len();
let start_fragment = start_html + prefix.len();
let end_fragment = start_fragment + fragment.len();
let end_html = end_fragment + "<!--EndFragment--></body></html>".len();
let full = format!(
"Version:1.0\r\nStartHTML:{start_html:010}\r\nEndHTML:{end_html:010}\r\n\
StartFragment:{start_fragment:010}\r\nEndFragment:{end_fragment:010}\r\nSourceURL:https://x/\r\n\
{prefix}{fragment}<!--EndFragment--></body></html>"
);
assert_eq!(html_from_cf(full.as_bytes()), fragment.as_bytes());
}
#[test]
fn cf_html_extract_falls_back_without_markers() {
// No fragment markers at all → whole buffer (minus any NUL).
let mut b = b"<p>no markers</p>".to_vec();
assert_eq!(html_from_cf(&b), b"<p>no markers</p>");
b.push(0);
assert_eq!(html_from_cf(&b), b"<p>no markers</p>");
}
#[test]
fn rtf_strips_one_trailing_nul() {
assert_eq!(rtf_from_cf(br"{\rtf1 hi}"), br"{\rtf1 hi}");
assert_eq!(rtf_from_cf(b"{\\rtf1 hi}\0"), br"{\rtf1 hi}");
// Only one NUL is stripped.
assert_eq!(rtf_from_cf(b"x\0\0"), b"x\0");
}
}
+104 -14
View File
@@ -19,6 +19,16 @@ pub struct EncodedFrame {
pub pts_ns: u64,
/// True for IDR/keyframes (sets the SOF/keyframe wire flags).
pub keyframe: bool,
/// True when this AU is a **reference-frame-invalidation recovery frame** — a clean P-frame the
/// encoder coded against a known-good reference in response to
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames). The pump tags it
/// [`punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR`] so the client lifts its post-loss
/// freeze on it without an IDR. Set by BOTH RFI backends: native AMF (the LTR force-reference
/// frame) and Windows direct-NVENC (the first frame encoded after `nvEncInvalidateRefFrames` —
/// the invalidation applies at the next `encode_picture`, so that AU is by construction the
/// clean re-anchor). Without it the client's freeze can only lift on an IDR — which the host
/// suppresses after a successful RFI (the cooldown), a ~1 s frozen stall per loss event.
pub recovery_anchor: bool,
}
/// Codec selection negotiated with the client.
@@ -194,8 +204,9 @@ pub struct EncoderCaps {
/// The encoder can perform real reference-frame invalidation — i.e.
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) can return `true`. When `false`
/// the caller skips that always-`false` call and forces a keyframe directly on loss recovery.
/// Only the Windows direct-NVENC path implements RFI; libavcodec (Linux NVENC), VAAPI and
/// AMF/QSV always keyframe.
/// Two backends implement RFI: Windows direct-NVENC (`nvEncInvalidateRefFrames`) and native
/// AMF (user-LTR force-reference, when the driver accepted the LTR slots at open). The
/// libavcodec paths (Linux NVENC, VAAPI, QSV) can't express it and always keyframe.
pub supports_rfi: bool,
/// The encoder emits in-band HDR mastering/CLL SEI from [`set_hdr_meta`](Encoder::set_hdr_meta).
/// When `false`, `set_hdr_meta` is a no-op and no in-band grade reaches the client. Only the
@@ -208,17 +219,49 @@ pub struct EncoderCaps {
/// the encoder's real chroma disagrees with what was negotiated (the in-band SPS is authoritative
/// for the decoder either way).
pub chroma_444: bool,
/// The encoder runs a periodic **intra-refresh wave** (a moving band of intra blocks +
/// recovery-point SEI, no periodic IDR): FEC-unrecoverable loss self-heals within one wave, so
/// the session glue rate-limits client keyframe requests instead of answering each with a full
/// IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC sets it when
/// `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/software never do.
/// The encoder runs a periodic **intra-refresh wave** a moving band of intra blocks that
/// re-codes the whole picture over ~0.5 s, no periodic IDR. FEC-unrecoverable loss self-heals as
/// the band sweeps, so the session glue rate-limits client keyframe requests instead of answering
/// each with a full IDR (the 20-40× frame-size spike that cascades under loss). Linux NVENC / AMF
/// set it when `PUNKTFUNK_INTRA_REFRESH` opened the encoder in that mode; VAAPI/QSV/software never
/// do. NOTE — the wave carries NO decoder-visible clean-point: FFmpeg never sets `AV_FRAME_FLAG_KEY`
/// at a recovery point (H.264 flags key only when `recovery_frame_cnt == 0`; HEVC only on IRAP),
/// and AMF emits no recovery-point SEI at all. So this cap ALONE does not let the client lift its
/// post-loss freeze without an IDR — that needs [`intra_refresh_recovery`](Self::intra_refresh_recovery).
pub intra_refresh: bool,
/// The intra-refresh wave is a *validated constrained GDR* — verified on real hardware to fully
/// heal a lost picture within one wave period with no residual artifacts. Only then does the host
/// tag each wave-boundary AU with [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// so the client can lift its freeze on the second mark (a proven clean re-anchor) instead of
/// waiting out its backstop and forcing a full IDR. Default `false` on every backend until on-glass
/// validation flips it — an un-validated encoder keeps the IDR recovery path, so this is inert and
/// cannot regress. Meaningless unless [`intra_refresh`](Self::intra_refresh) is also set.
pub intra_refresh_recovery: bool,
/// Length of the intra-refresh wave in frames — the boundary period the host marks on (it sets
/// `USER_FLAG_RECOVERY_POINT` on every Nth emitted AU, re-phased at each IDR). 0 when intra-refresh
/// is off. Only consulted when [`intra_refresh_recovery`](Self::intra_refresh_recovery) is set.
pub intra_refresh_period: u32,
}
/// A hardware encoder. One per session; runs on the encode thread.
pub trait Encoder: Send {
fn submit(&mut self, frame: &CapturedFrame) -> Result<()>;
/// [`submit`](Self::submit) with the **wire frame index** this frame's AU will carry — the
/// number the packetizer stamps on it and the client's loss reports/RFI requests name. The
/// session glue predicts it exactly as `AUs sent so far + frames in flight` (AUs are emitted
/// FIFO, one per submission; anything that would break the prediction — an in-place reset, a
/// device-change teardown, an encoder rebuild — forfeits the in-flight frames on BOTH sides
/// and clears the encoder's reference state, so stale predictions die with it). The RFI
/// backends pin their frame numbering (LTR marks, DPB timestamps) to this so
/// [`invalidate_ref_frames`](Self::invalidate_ref_frames) compares client frame numbers
/// against the same domain — an encoder-internal counter desyncs from the wire on the first
/// mid-stream rebuild (adaptive bitrate steps do this under congestion, exactly when losses
/// happen). Default: ignore the index and delegate to `submit` (backends without per-frame
/// reference bookkeeping don't care).
fn submit_indexed(&mut self, frame: &CapturedFrame, wire_index: u32) -> Result<()> {
let _ = wire_index;
self.submit(frame)
}
/// This encoder's static [capabilities](EncoderCaps) (RFI, HDR SEI), so the session glue can
/// route by query rather than rely on the no-op/`false` defaults of
/// [`invalidate_ref_frames`](Self::invalidate_ref_frames) / [`set_hdr_meta`](Self::set_hdr_meta).
@@ -236,13 +279,14 @@ pub trait Encoder: Send {
/// Default: no-op (SDR encoders / libavcodec paths that don't attach it yet). Cheap to call
/// every frame; only the direct-NVENC path consumes it.
fn set_hdr_meta(&mut self, _meta: Option<punktfunk_core::quic::HdrMeta>) {}
/// Invalidate a contiguous range of previously-encoded reference frames (client frame numbers,
/// as reported in a loss-recovery request) so the encoder re-references an older still-valid
/// frame instead of emitting a full IDR. Returns `true` if a real reference invalidation was
/// performed; `false` means the encoder couldn't (range older than the DPB, or the backend has
/// no RFI) and the caller should fall back to [`request_keyframe`](Self::request_keyframe).
/// Default: `false` — only the Windows direct-NVENC path implements true RFI; libavcodec
/// (Linux NVENC) and VAAPI can't express `nvEncInvalidateRefFrames`, so they keyframe.
/// Invalidate a contiguous range of previously-encoded reference frames (client frame numbers
/// — WIRE frame indexes, the domain [`submit_indexed`](Self::submit_indexed) pins the encoder's
/// bookkeeping to) so the encoder re-references an older still-valid frame instead of emitting
/// a full IDR. Returns `true` if a real reference invalidation was performed; `false` means the
/// encoder couldn't (range older than the DPB/LTR history, or the backend has no RFI) and the
/// caller should fall back to [`request_keyframe`](Self::request_keyframe). Default: `false` —
/// the Windows direct-NVENC path (`nvEncInvalidateRefFrames`) and native AMF (LTR
/// force-reference) implement true RFI; the libavcodec paths can't express it, so they keyframe.
fn invalidate_ref_frames(&mut self, _first_frame: i64, _last_frame: i64) -> bool {
false
}
@@ -418,6 +462,9 @@ impl Encoder for TrackedEncoder {
fn submit(&mut self, frame: &CapturedFrame) -> Result<()> {
self.inner.submit(frame)
}
fn submit_indexed(&mut self, frame: &CapturedFrame, wire_index: u32) -> Result<()> {
self.inner.submit_indexed(frame, wire_index)
}
fn caps(&self) -> EncoderCaps {
self.inner.caps()
}
@@ -710,6 +757,29 @@ fn open_nvenc_probed(
bit_depth: u8,
chroma: ChromaFormat,
) -> Result<Box<dyn Encoder>> {
// Direct-SDK NVENC (design/linux-direct-nvenc.md): the DEFAULT on NVIDIA, and only for a CUDA
// capture payload (it registers CUDADEVICEPTR inputs — a CPU/dmabuf frame can't feed it, so those
// keep the libav path; and `cuda` is false on AMD/Intel, so they stay on VAAPI). Set
// PUNKTFUNK_NVENC_DIRECT=0 to fall back to libav. It self-clamps the bitrate internally (its own
// level-ceiling binary search at session open), so it skips the probe-loop stepping below.
#[cfg(feature = "nvenc")]
if cuda && nvenc_direct_enabled() {
tracing::info!(
codec = codec.nvenc_name(),
"Linux direct-SDK NVENC (real RFI + recovery anchor) — set PUNKTFUNK_NVENC_DIRECT=0 for libav"
);
return Ok(Box::new(nvenc_cuda::NvencCudaEncoder::open(
codec,
format,
width,
height,
fps,
bitrate_bps,
cuda,
bit_depth,
chroma,
)?) as Box<dyn Encoder>);
}
const MIN_PROBE_BPS: u64 = 50_000_000;
let mut candidates = vec![bitrate_bps];
let cap = codec.max_bitrate_bps();
@@ -747,6 +817,19 @@ fn open_nvenc_probed(
Err(last.unwrap_or_else(|| anyhow::anyhow!("encoder open failed at every probed bitrate")))
}
/// Whether the direct-SDK NVENC path is active. **Default ON** — on-glass validated 2026-07-12:
/// real RFI landed 73/73 as clean P-frame recovery anchors (never IDR) on an RTX host with a real
/// Steam Deck client (design/linux-direct-nvenc.md §9). `PUNKTFUNK_NVENC_DIRECT=0` (also `false`/
/// `no`/`off`) is the libav escape hatch. Only consulted for a CUDA capture payload on an NVIDIA
/// host — the `cuda` gate in `open_nvenc_probed` keeps AMD/Intel on VAAPI regardless — and only
/// with `--features nvenc`.
#[cfg(all(target_os = "linux", feature = "nvenc"))]
fn nvenc_direct_enabled() -> bool {
std::env::var("PUNKTFUNK_NVENC_DIRECT")
.map(|v| !matches!(v.trim(), "0" | "false" | "no" | "off"))
.unwrap_or(true)
}
/// Cheap, side-effect-free NVIDIA-presence probe for the `auto` backend selector: the NVIDIA
/// kernel driver exposes these device nodes, AMD/Intel boxes have neither. Deliberately does NOT
/// create a CUDA context (that would allocate GPU state on every host that merely *might* be
@@ -1084,9 +1167,16 @@ mod amf;
mod ffmpeg_win;
#[cfg(target_os = "linux")]
mod linux;
// Direct-SDK NVENC on Linux (CUDA input; design/linux-direct-nvenc.md) — real RFI + recovery anchor
// + reset() lever the libavcodec `linux::NvencEncoder` can't express. Opt-in behind
// `PUNKTFUNK_NVENC_DIRECT` until on-glass validated; the `.so` resolves at runtime like the Windows
// path, so `--features nvenc` stays safe on a driver-less/AMD Linux box.
#[cfg(all(target_os = "windows", feature = "nvenc"))]
#[path = "encode/windows/nvenc.rs"]
mod nvenc;
#[cfg(all(target_os = "linux", feature = "nvenc"))]
#[path = "encode/linux/nvenc_cuda.rs"]
mod nvenc_cuda;
// Software (openh264) H.264 encoder — the GPU-less path on BOTH Windows and Linux (a headless /
// GPU-less test box, or a fallback when no hardware encoder is available). Platform-agnostic: it
// consumes CPU RGB `CapturedFrame`s and the statically-bundled openh264 build.
@@ -149,6 +149,22 @@ fn nvenc_input(format: PixelFormat) -> (Pixel, bool) {
}
}
/// The [`NvencEncoder::open`] arguments, kept on the encoder so [`Encoder::reset`] can rebuild it
/// in place with the session's negotiated parameters — the encode-stall watchdog's recovery lever
/// (drop the wedged libavcodec encoder, reopen fresh, forfeit the owed AUs, restart at an IDR).
#[derive(Clone, Copy)]
struct OpenArgs {
codec: Codec,
format: PixelFormat,
width: u32,
height: u32,
fps: u32,
bitrate_bps: u64,
cuda: bool,
bit_depth: u8,
chroma: ChromaFormat,
}
pub struct NvencEncoder {
enc: encoder::video::Encoder,
/// Reusable 4-bpp CPU input frame (CPU path only; `None` for the zero-copy/CUDA path).
@@ -177,6 +193,12 @@ pub struct NvencEncoder {
/// Opened in intra-refresh mode (surfaced via [`caps`](Encoder::caps) so the session glue
/// rate-limits forced IDRs — the wave heals loss without them).
intra_refresh: bool,
/// Resolved wave length in frames when [`intra_refresh`](Self::intra_refresh), else 0. Cached at
/// open so the pump's per-AU `caps()` doesn't re-read `PUNKTFUNK_IR_PERIOD_FRAMES`; the pump marks
/// every Nth AU with `USER_FLAG_RECOVERY_POINT` for the client's clean re-anchor.
intra_refresh_period: u32,
/// The open arguments, for the in-place [`reset`](Encoder::reset) rebuild.
args: OpenArgs,
}
// `CudaHw` holds raw `AVBufferRef`s and `sws_444` a raw `SwsContext`; the encoder lives on a single
@@ -525,6 +547,22 @@ impl NvencEncoder {
frame_idx: 0,
force_kf: false,
intra_refresh,
intra_refresh_period: if intra_refresh {
intra_refresh_period(fps).max(1) as u32
} else {
0
},
args: OpenArgs {
codec,
format,
width,
height,
fps,
bitrate_bps,
cuda,
bit_depth,
chroma,
},
})
}
}
@@ -536,6 +574,12 @@ impl Encoder for NvencEncoder {
// convert. RFI/HDR-SEI stay unsupported on libavcodec NVENC (the trait defaults).
chroma_444: self.want_444,
intra_refresh: self.intra_refresh,
// NVENC intra-refresh is purpose-built GDR loss recovery (moving band + recovery-point
// SEI): the wave heals a lost picture within one period, so mark the boundary AUs and let
// the client re-anchor on them instead of forcing a full IDR. Tied to `intra_refresh`
// (already the `PUNKTFUNK_INTRA_REFRESH` opt-in), unlike AMF/QSV which stay unvalidated.
intra_refresh_recovery: self.intra_refresh,
intra_refresh_period: self.intra_refresh_period,
..super::EncoderCaps::default()
}
}
@@ -567,6 +611,35 @@ impl Encoder for NvencEncoder {
self.force_kf = true;
}
/// Encode-stall recovery: drop the wedged libavcodec encoder and reopen it fresh with the
/// session's negotiated parameters (the stored [`OpenArgs`]) — the drop-and-reopen lever the
/// QSV/VAAPI paths use, so the encode-stall watchdog can heal a wedged NVENC/driver instead of
/// ending the session. Owed AUs are forfeited; the fresh encoder opens on an IDR.
fn reset(&mut self) -> bool {
let a = self.args;
match Self::open(
a.codec,
a.format,
a.width,
a.height,
a.fps,
a.bitrate_bps,
a.cuda,
a.bit_depth,
a.chroma,
) {
Ok(mut fresh) => {
fresh.force_kf = true;
*self = fresh; // drops the wedged encoder (frees its contexts) in the same step
true
}
Err(e) => {
tracing::error!(error = %format!("{e:#}"), "NVENC in-place reopen failed");
false
}
}
}
fn poll(&mut self) -> Result<Option<EncodedFrame>> {
let mut pkt = Packet::empty();
match self.enc.receive_packet(&mut pkt) {
@@ -578,6 +651,7 @@ impl Encoder for NvencEncoder {
data,
pts_ns,
keyframe: pkt.is_key(),
recovery_anchor: false,
}))
}
// No packet ready yet (need another input frame).
File diff suppressed because it is too large Load Diff
@@ -294,6 +294,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<Option<En
data,
pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(),
recovery_anchor: false,
}))
}
Err(ffmpeg::Error::Other { errno })
@@ -1125,6 +1126,19 @@ impl Encoder for VaapiEncoder {
self.force_kf = true;
}
/// Encode-stall recovery: drop the wedged libavcodec encoder (its `Drop` releases the VA
/// surfaces/filter graph/devices) and let the next `submit` rebuild it lazily from the first
/// frame's payload, exactly like first-frame bring-up — the same drop-and-reopen lever the
/// Windows QSV path has. The owed AUs are forfeited (`in_flight` zeroed) and the rebuilt
/// encoder's first frame is forced IDR so the client resyncs immediately. Without this the
/// encode-stall watchdog had no lever on Linux AMD/Intel and a wedged driver ended the session.
fn reset(&mut self) -> bool {
self.inner = None;
self.in_flight = 0;
self.force_kf = true;
true
}
fn poll(&mut self) -> Result<Option<EncodedFrame>> {
// With `async_depth > 1`, `submit` no longer waits for the ASIC — the AU for the frame we
// just sent lands ~one hardware-encode-time later. Wait for it (bounded) so it still ships
+10 -5
View File
@@ -20,6 +20,7 @@ use openh264::encoder::{
};
use openh264::formats::YUVSlices;
use openh264::OpenH264API;
use std::collections::VecDeque;
pub struct OpenH264Encoder {
enc: Oh264,
@@ -34,8 +35,11 @@ pub struct OpenH264Encoder {
v_plane: Vec<u8>,
frame_idx: i64,
force_kf: bool,
/// At most one AU per submit (no lookahead), handed back by the next `poll`.
pending: Option<EncodedFrame>,
/// One AU per submit (no lookahead), handed back FIFO by `poll`. A queue, not an `Option`:
/// the session loop pipelines up to `capturer.pipeline_depth()` submits before polling, and a
/// single-slot pending would silently overwrite (lose) the older AUs — including the opening
/// IDR — and permanently skew the loop's FIFO pts pairing.
pending: VecDeque<EncodedFrame>,
}
// openh264's Encoder holds a raw C handle (not auto-Send); it lives on the single encode thread.
@@ -88,7 +92,7 @@ impl OpenH264Encoder {
v_plane: vec![0; (w / 2) * (h / 2)],
frame_idx: 0,
force_kf: false,
pending: None,
pending: VecDeque::new(),
})
}
@@ -207,10 +211,11 @@ impl Encoder for OpenH264Encoder {
if !data.is_empty() {
let keyframe = matches!(bs.frame_type(), FrameType::IDR | FrameType::I);
let pts_ns = self.frame_idx as u64 * 1_000_000_000 / self.fps.max(1) as u64;
self.pending = Some(EncodedFrame {
self.pending.push_back(EncodedFrame {
data,
pts_ns,
keyframe,
recovery_anchor: false,
});
}
self.frame_idx += 1;
@@ -222,7 +227,7 @@ impl Encoder for OpenH264Encoder {
}
fn poll(&mut self) -> Result<Option<EncodedFrame>> {
Ok(self.pending.take())
Ok(self.pending.pop_front())
}
fn flush(&mut self) -> Result<()> {
+348 -20
View File
@@ -644,6 +644,26 @@ struct CodecProps {
/// HEVC 64-px CTBs. `None` on AV1 (v1.4.36 exposes only a mode enum, no slot-size control —
/// loss recovery stays IDR there).
intra_refresh: Option<(PCWSTR, u32)>,
/// LTR-RFI recovery property names (design: the AMD twin of NVENC intra-refresh recovery).
/// `None` on AV1 — its reference management uses a frame-marking OBU mechanism this path does
/// not drive, so LTR recovery is AVC/HEVC-only.
ltr: Option<LtrProps>,
}
/// The four AMF LTR (long-term-reference) property names, codec-prefixed (AVC bare, HEVC `Hevc*`).
/// Two are static (`max_*`, set once at open); two are per-frame (`mark`/`force`, set on the input
/// surface each `submit`). Together they let a loss re-reference a known-good older frame — a clean
/// P-frame instead of a 2040× IDR spike.
struct LtrProps {
/// `MaxOfLTRFrames` — number of user LTR slots (we request [`NUM_LTR_SLOTS`]).
max_ltr_frames: PCWSTR,
/// `MaxNumRefFrames` — reference-picture budget; must exceed 1 for LTR to engage.
max_num_ref_frames: PCWSTR,
/// `MarkCurrentWithLTRIndex` (per-frame) — tag the current frame as long-term reference slot N.
mark_ltr_index: PCWSTR,
/// `ForceLTRReferenceBitfield` (per-frame) — force the current frame to reference only the LTR
/// slots in the bitfield (`1<<N`), breaking the corrupted short-term chain after a loss.
force_ltr_bitfield: PCWSTR,
}
/// The two payload shapes `lowlatency` takes across codecs.
@@ -689,6 +709,12 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("OutColorPrimaries"),
hdr_metadata: None,
intra_refresh: Some((w!("IntraRefreshMBsNumberPerSlot"), 16)),
ltr: Some(LtrProps {
max_ltr_frames: w!("MaxOfLTRFrames"),
max_num_ref_frames: w!("MaxNumRefFrames"),
mark_ltr_index: w!("MarkCurrentWithLTRIndex"),
force_ltr_bitfield: w!("ForceLTRReferenceBitfield"),
}),
},
Codec::H265 => CodecProps {
component: w!("AMFVideoEncoderHW_HEVC"),
@@ -716,6 +742,12 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("HevcOutColorPrimaries"),
hdr_metadata: Some(w!("HevcInHDRMetadata")),
intra_refresh: Some((w!("HevcIntraRefreshCTBsNumberPerSlot"), 64)),
ltr: Some(LtrProps {
max_ltr_frames: w!("HevcMaxOfLTRFrames"),
max_num_ref_frames: w!("HevcMaxNumRefFrames"),
mark_ltr_index: w!("HevcMarkCurrentWithLTRIndex"),
force_ltr_bitfield: w!("HevcForceLTRReferenceBitfield"),
}),
},
Codec::Av1 => CodecProps {
component: w!("AMFVideoEncoderHW_AV1"),
@@ -743,6 +775,7 @@ fn codec_props(codec: Codec) -> CodecProps {
out_primaries: w!("Av1OutputColorPrimaries"),
hdr_metadata: Some(w!("Av1InHDRMetadata")),
intra_refresh: None,
ltr: None,
},
}
}
@@ -797,6 +830,45 @@ fn intra_refresh_period(fps: u32) -> u32 {
.unwrap_or_else(|| (fps.max(16) / 2).max(2))
}
/// Number of user-controlled LTR slots. AMD exposes up to 2; two rotating slots hold a sliding pair
/// of recent long-term references, so a loss can re-reference the newest one *before* the loss point.
const NUM_LTR_SLOTS: usize = 2;
/// AMD's real clean loss-recovery path (the NVENC-RFI twin): the encoder marks frames as long-term
/// references, and on loss forces a later frame to re-reference a known-good one — a clean P-frame,
/// not a 20-40× IDR spike. On by default when the driver supports it (AMF intra-refresh cannot heal —
/// no constrained-intra-prediction property exists in the API, header-confirmed + PSNR-proven — and
/// LTR is mutually exclusive with it, so LTR wins). `PUNKTFUNK_NO_AMF_LTR=1` forces the old full-IDR
/// recovery for debugging.
fn ltr_disabled() -> bool {
std::env::var("PUNKTFUNK_NO_AMF_LTR")
.map(|v| matches!(v.trim(), "1" | "true" | "yes" | "on"))
.unwrap_or(false)
}
/// Cadence (frames) between LTR marks — a fresh long-term reference roughly every half second by
/// default (`PUNKTFUNK_LTR_INTERVAL_FRAMES` overrides). With [`NUM_LTR_SLOTS`] slots this keeps ~one
/// second of recent references, so a loss up to ~1 s old still has a known-good frame to force; a
/// smaller interval means the forced reference is more recent (a smaller recovery-frame residual).
fn ltr_mark_interval(fps: u32) -> i64 {
std::env::var("PUNKTFUNK_LTR_INTERVAL_FRAMES")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.filter(|v| *v >= 1)
.unwrap_or_else(|| (fps.max(2) / 2).max(1) as i64)
}
/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N` the encoder
/// self-triggers its real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path, so a
/// headless spike run can exercise LTR recovery end-to-end (mark → force → recovery-anchor tag)
/// without a live client sending an [`RfiRequest`](punktfunk_core::quic::RfiRequest). `None` normally.
fn ltr_test_force_at() -> Option<i64> {
std::env::var("PUNKTFUNK_LTR_FORCE_AT")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.filter(|v| *v > 0)
}
// ---------------------------------------------------------------------------------------------
// Owned-pointer guards (release exactly once; Terminate before Release for context/component,
// mirroring amfenc.c's teardown order).
@@ -930,11 +1002,12 @@ struct Inner {
dctx: ID3D11DeviceContext,
ring: Vec<ID3D11Texture2D>,
next: usize,
/// (pts_ns, forced-IDR) per submitted-but-unretrieved frame, FIFO — the AMF encoder emits
/// AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`. Its length is
/// the count of input surfaces AMF still holds, so `submit` bounds it below [`RING`] to keep
/// the input ring from being overwritten under it.
pending: VecDeque<(u64, bool)>,
/// (pts_ns, forced-IDR, recovery-anchor) per submitted-but-unretrieved frame, FIFO — the AMF
/// encoder emits AUs in submit order (B-frames are never enabled), pairing with `QueryOutput`.
/// The third field tags the LTR-RFI re-anchor frame so the AU carries `recovery_anchor` for the
/// client's freeze-lift. Its length is the count of input surfaces AMF still holds, so `submit`
/// bounds it below [`RING`] to keep the input ring from being overwritten under it.
pending: VecDeque<(u64, bool, bool)>,
/// AUs already pulled by `submit`'s backpressure drain, waiting to be handed out by `poll`
/// (FIFO, strictly older than anything still in `pending`). Empty in the steady state — only
/// fills when the encoder falls behind and `submit` drains to free an input slot.
@@ -988,6 +1061,26 @@ pub struct AmfEncoder {
/// gates [`EncoderCaps::intra_refresh`] so keyframe-request rate-limiting only happens when
/// the wave really runs.
ir_active: bool,
// --- Long-Term-Reference reference-frame-invalidation recovery (the AMD RFI path) ---
/// The driver accepted the LTR properties at open — gates [`EncoderCaps::supports_rfi`] and all
/// the per-frame LTR marking/forcing below. When true, intra-refresh is NOT set (mutually
/// exclusive) and loss recovery re-references a known-good LTR instead of forcing a full IDR.
ltr_active: bool,
/// The `frame_idx` currently stored in each of the two LTR slots (`None` = never marked). On loss
/// the newest slot with an index *before* the loss is the known-good reference to force.
ltr_slots: [Option<i64>; NUM_LTR_SLOTS],
/// The slot the next LTR mark writes (round-robins `0,1,0,1,…` so the two slots hold a sliding
/// pair of recent references).
next_ltr_slot: usize,
/// Cadence (frames) between LTR marks — a fresh long-term reference roughly this often.
ltr_mark_interval: i64,
/// Set by [`invalidate_ref_frames`](Encoder::invalidate_ref_frames): the LTR slot the *next*
/// submitted frame must force-reference (`ForceLTRReferenceBitfield`). Consumed on that submit.
pending_force: Option<usize>,
/// Validation hook (`PUNKTFUNK_LTR_FORCE_AT=N`, spike-only): at `frame_idx == N`, self-trigger the
/// real [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) path so a headless spike run can
/// exercise LTR recovery end-to-end without a live client. `None` in normal operation.
ltr_test_force_at: Option<i64>,
/// Consecutive [`reset`](Self::reset)s that have NOT been followed by a produced AU (cleared in
/// `poll` on any output). An in-place `Terminate`+re-`Init` heals a transient component stall,
/// but it re-inits the SAME context — so if the fault is the context / VCN session itself (the
@@ -1084,17 +1177,33 @@ impl AmfEncoder {
force_kf: false,
hdr_meta: None,
ir_active: false,
ltr_active: false,
ltr_slots: [None; NUM_LTR_SLOTS],
next_ltr_slot: 0,
ltr_mark_interval: ltr_mark_interval(fps),
pending_force: None,
ltr_test_force_at: ltr_test_force_at(),
resets_without_output: 0,
})
}
/// Whether this encoder should *attempt* the LTR-RFI recovery path (design: the AMD twin of
/// NVENC intra-refresh recovery). Gated to AVC/HEVC — AMF exposes user LTR only for those two
/// codecs — and defeatable via `PUNKTFUNK_NO_AMF_LTR`. Whether the driver actually *accepts* the
/// properties is a separate question answered by [`apply_static_props`], which sets `ltr_active`.
fn ltr_wanted(&self) -> bool {
!ltr_disabled() && matches!(self.codec, Codec::H264 | Codec::H265)
}
/// Apply the static encoder configuration (design §3.4 — the native mirror of the ffmpeg
/// opts block in `open_win_encoder`). Called before `Init`, and again on a `reset()`
/// re-`Init` (Terminate does not guarantee property retention across every driver).
/// Returns whether the intra-refresh wave was requested AND accepted by this driver — the
/// caller stores it so [`Encoder::caps`] only rate-limits keyframe requests when the wave
/// really runs.
unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<bool> {
/// Returns `(ir_active, ltr_active)`: whether the intra-refresh wave / the LTR-RFI slots were
/// requested AND accepted by this driver. The two are mutually exclusive (LTR wins when both are
/// wanted). The caller stores both — `ir_active` so [`Encoder::caps`] only rate-limits keyframe
/// requests when a wave runs, `ltr_active` so [`Encoder::caps`] advertises `supports_rfi` and the
/// per-frame mark/force logic in `submit` only fires when the slots exist.
unsafe fn apply_static_props(&self, comp: *mut sys::AmfComponent) -> Result<(bool, bool)> {
let p = &self.props;
// Usage first: it "fully configures parameter set" — everything after is an override.
set_prop(
@@ -1145,7 +1254,39 @@ impl AmfEncoder {
// whole picture refreshes every `period` frames — per-slot units = ceil(total blocks /
// period). Optional by VCN generation; the return value gates `caps().intra_refresh`.
let mut ir_active = false;
if let Some((name, block)) = p.intra_refresh {
let mut ltr_active = false;
if let Some(ltr) = p.ltr.as_ref().filter(|_| self.ltr_wanted()) {
// LTR-RFI recovery (design: the AMD twin of NVENC intra-refresh recovery). Request
// NUM_LTR_SLOTS user-controlled long-term references. LTR needs >1 reference frames and
// is MUTUALLY EXCLUSIVE with intra-refresh (AMF disables one if both are set), so the
// intra-refresh block below is skipped whenever LTR engages.
let ref_ok = set_prop(
comp,
ltr.max_num_ref_frames,
AmfVariant::from_i64(NUM_LTR_SLOTS as i64),
false,
)?;
let ltr_ok = set_prop(
comp,
ltr.max_ltr_frames,
AmfVariant::from_i64(NUM_LTR_SLOTS as i64),
false,
)?;
ltr_active = ref_ok && ltr_ok;
if ltr_active {
tracing::info!(
slots = NUM_LTR_SLOTS,
mark_interval = self.ltr_mark_interval,
"AMF LTR-RFI recovery enabled (loss recovery re-references a known-good LTR, not a full IDR)"
);
} else {
tracing::warn!(
ref_ok,
ltr_ok,
"this VCN/driver rejected an LTR property — loss recovery stays full-IDR"
);
}
} else if let Some((name, block)) = p.intra_refresh {
if intra_refresh_requested() {
let period = intra_refresh_period(self.fps);
let blocks = self.width.div_ceil(block) * self.height.div_ceil(block);
@@ -1273,7 +1414,7 @@ impl AmfEncoder {
AmfVariant::from_i64(primaries),
self.ten_bit,
)?;
Ok(ir_active)
Ok((ir_active, ltr_active))
}
/// Build (or rebuild, on a capture-device change) the AMF context + encoder component on the
@@ -1323,7 +1464,7 @@ impl AmfEncoder {
bail!("AMF CreateComponent returned null");
}
let comp = Component(comp);
let ir_active = self.apply_static_props(comp.0)?;
let (ir_active, ltr_active) = self.apply_static_props(comp.0)?;
let fmt = if self.ten_bit {
sys::AMF_SURFACE_P010
} else {
@@ -1334,6 +1475,14 @@ impl AmfEncoder {
"AMF encoder Init",
)?;
self.ir_active = ir_active;
// A rebuilt component starts with fresh (empty) LTR slots — a new context has no
// reference history, so any prior marks are void and the first frame re-IDRs anyway.
self.ltr_active = ltr_active;
if ltr_active {
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
}
// Owned input ring on the capturer's device (design §3.2): RENDER_TARGET |
// SHADER_RESOURCE, the same bind flags the validated ffmpeg zero-copy pool uses.
@@ -1594,7 +1743,7 @@ enum DrainOutcome {
/// single encode thread with no other AMF call to this component in flight.
unsafe fn drain_one_output(
comp: *mut sys::AmfComponent,
pending: &mut VecDeque<(u64, bool)>,
pending: &mut VecDeque<(u64, bool, bool)>,
output_data_type: PCWSTR,
output_key_max: i64,
) -> Result<DrainOutcome> {
@@ -1641,11 +1790,12 @@ unsafe fn drain_one_output(
bail!("AMF output buffer is empty");
}
let au = std::slice::from_raw_parts(native as *const u8, size).to_vec();
let (pts_ns, forced) = pending.pop_front().unwrap_or((0, false));
let (pts_ns, forced, recovery_anchor) = pending.pop_front().unwrap_or((0, false, false));
Ok(DrainOutcome::Frame(EncodedFrame {
data: au,
pts_ns,
keyframe: key_prop || forced,
recovery_anchor,
}))
}
@@ -1689,9 +1839,62 @@ impl Encoder for AmfEncoder {
expected
);
self.ensure_inner(&frame.device)?;
let forced = std::mem::take(&mut self.force_kf) || self.frame_idx == 0;
let cur_idx = self.frame_idx;
// A component's FIRST submission must be a forced IDR (stream-start contract: in-band
// headers + LTR re-anchor). Detected via the fresh ring counter, NOT `frame_idx == 0`:
// `submit_indexed` pins frame_idx to the wire index, which is non-zero when a mid-session
// rebuild (bitrate step / reset escalation) brings a new component up.
let opening = self.inner.as_ref().is_none_or(|i| i.next == 0);
let forced = std::mem::take(&mut self.force_kf) || opening;
let pts_100ns = self.frame_idx * 10_000_000 / self.fps.max(1) as i64;
self.frame_idx += 1;
// --- LTR-RFI per-frame decisions (design: the AMD twin of NVENC intra-refresh recovery) ---
// Decided here, before borrowing `inner`, because the test hook re-enters `&mut self`
// (`invalidate_ref_frames`) and the mark cadence mutates the slot bookkeeping. The two
// per-frame property names are copied out (PCWSTR is Copy) so the unsafe surface block can
// set them without re-borrowing `self.props` under the live `inner` borrow.
let ltr_names = self
.props
.ltr
.as_ref()
.map(|l| (l.mark_ltr_index, l.force_ltr_bitfield));
let mut mark_slot: Option<usize> = None;
let mut force_slot: Option<usize> = None;
let mut recovery_anchor = false;
if self.ltr_active {
if forced {
// An IDR resets the decoder's reference buffers — every prior LTR mark is void.
// Re-anchor from scratch: drop the stale slots (the mark cadence below tags the IDR
// as the first fresh long-term reference) and cancel any force queued against them.
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
} else if self.ltr_test_force_at == Some(cur_idx) {
// Spike-only validation hook: self-trigger the real invalidate path so a headless
// run exercises mark → force → recovery-anchor without a live client's RfiRequest.
let triggered = self.invalidate_ref_frames(cur_idx, cur_idx);
tracing::info!(
frame = cur_idx,
triggered,
"AMF LTR test hook fired invalidate_ref_frames"
);
}
// Apply a queued force (from invalidate_ref_frames / the test hook) to THIS frame: it
// becomes the clean re-anchor P-frame the client lifts its post-loss freeze on.
if let Some(slot) = self.pending_force.take() {
force_slot = Some(slot);
recovery_anchor = true;
}
// Mark cadence: refresh a long-term reference on every IDR and every `ltr_mark_interval`
// frames — but never on the recovery frame itself (marking rotates `next_ltr_slot` and
// could overwrite the very slot being forced; the next cadence mark re-establishes it).
if force_slot.is_none() && (forced || cur_idx % self.ltr_mark_interval == 0) {
let slot = self.next_ltr_slot;
self.ltr_slots[slot] = Some(cur_idx);
self.next_ltr_slot = (self.next_ltr_slot + 1) % NUM_LTR_SLOTS;
mark_slot = Some(slot);
}
}
let inner = self.inner.as_mut().expect("ensure_inner succeeded");
// Push the HDR mastering metadata when it changed (or a rebuilt component lost it) — a
// dynamic property, so mid-stream regrades take effect on the next IDR. Best-effort: a
@@ -1831,6 +2034,47 @@ impl Encoder for AmfEncoder {
Codec::Av1 => {}
}
}
// LTR-RFI per-frame properties (design: the AMD twin of NVENC intra-refresh recovery).
// `mark_slot`/`force_slot` were decided above. Marking tags the current frame as a
// long-term reference; forcing makes it re-reference a known-good LTR — a clean P-frame
// that breaks the corrupted short-term chain after a loss, no 20-40× IDR. Best-effort:
// a rejecting driver just leaves the client on its keyframe-request fallback.
if let Some((mark_name, force_name)) = ltr_names {
if let Some(slot) = mark_slot {
let r = ((*(*surf.0).vtbl).set_property)(
surf.0,
mark_name.0,
AmfVariant::from_i64(slot as i64),
);
if r != sys::AMF_OK {
tracing::warn!(
slot,
result = %format!("{} ({r})", result_name(r)),
"AMF LTR mark rejected"
);
}
}
if let Some(slot) = force_slot {
let r = ((*(*surf.0).vtbl).set_property)(
surf.0,
force_name.0,
AmfVariant::from_i64(1_i64 << slot),
);
if r == sys::AMF_OK {
tracing::info!(
slot,
frame = cur_idx,
"AMF LTR-RFI: re-referencing known-good LTR (clean recovery, no IDR)"
);
} else {
tracing::warn!(
slot,
result = %format!("{} ({r})", result_name(r)),
"AMF LTR force-reference rejected — client stays frozen until its IDR fallback"
);
}
}
}
let mut r = ((*(*inner.comp.0).vtbl).submit_input)(inner.comp.0, surf.0);
// Backstop back-pressure: the in-flight bound above already keeps a slot free, but if
// AMF's own input queue is momentarily full, AMF_INPUT_FULL is "busy, drain me and
@@ -1873,10 +2117,27 @@ impl Encoder for AmfEncoder {
}
}
}
inner.pending.push_back((captured.pts_ns, forced));
inner
.pending
.push_back((captured.pts_ns, forced, recovery_anchor));
Ok(())
}
/// Pin this submission's frame number to the wire frame index its AU will carry (see the
/// trait doc): the LTR slots then store WIRE indexes, so [`invalidate_ref_frames`]'s
/// pre-loss check (`slot < first`, both in client frame numbers) stays correct across every
/// encoder rebuild/reset — an internal counter desyncs on the first adaptive-bitrate rebuild,
/// making the check vacuously true and risking a force-reference to an LTR marked INSIDE the
/// lost range (a corrupted frame shipped as a clean recovery anchor). `frame_idx` also feeds
/// the AMF SetPts; a re-pin only ever moves it backward across a reset (fresh component, so a
/// pts restart is harmless) and forward on a rebuild (monotonic within any one component).
///
/// [`invalidate_ref_frames`]: Encoder::invalidate_ref_frames
fn submit_indexed(&mut self, frame: &CapturedFrame, wire_index: u32) -> Result<()> {
self.frame_idx = wire_index as i64;
self.submit(frame)
}
fn request_keyframe(&mut self) {
self.force_kf = true;
}
@@ -1887,11 +2148,65 @@ impl Encoder for AmfEncoder {
self.hdr_meta = meta;
}
/// LTR-RFI recovery (the AMD twin of the Windows NVENC `nvEncInvalidateRefFrames` path): a loss
/// of client frames `[first, last]` is answered by forcing the *next* submitted frame to
/// re-reference the newest long-term reference marked *before* the loss — a clean P-frame the
/// client can decode against a picture it still holds, instead of a 20-40× IDR spike.
///
/// Returns `true` when a usable pre-loss LTR exists (so the caller must NOT also force an IDR);
/// `false` when the loss predates every live LTR — then the only correct recovery is a keyframe,
/// and the caller falls back to [`request_keyframe`](Self::request_keyframe). Runs on the encode
/// thread (like submit/poll); the force is applied on the next `submit`.
fn invalidate_ref_frames(&mut self, first: i64, last: i64) -> bool {
// No live LTR session (driver declined the slots, or AV1 which has no user-LTR path) or a
// nonsense range → caller forces a full IDR.
if !self.ltr_active || first < 0 || first > last {
return false;
}
// Pick the newest LTR strictly OLDER than the loss: the most recent known-good reference the
// client still holds, so re-referencing it costs the least (smallest recovery-frame residual).
// `ltr_slots` store the WIRE frame index of the marked frame (`submit_indexed` pins
// `frame_idx` to it per submission), so they compare directly against the client's `first`
// — and stay comparable across encoder rebuilds/resets, where an internal counter would
// make this check vacuous and risk force-referencing an LTR marked INSIDE the lost range.
let mut best: Option<(usize, i64)> = None;
for (slot, marked) in self.ltr_slots.iter().enumerate() {
if let Some(idx) = *marked {
if idx < first && best.is_none_or(|(_, b)| idx > b) {
best = Some((slot, idx));
}
}
}
match best {
Some((slot, ltr_frame)) => {
// Queue the force for the next submit; that frame ships tagged `recovery_anchor`.
self.pending_force = Some(slot);
tracing::info!(
first,
last,
slot,
ltr_frame,
"AMF LTR-RFI: forcing the next frame to re-reference a known-good LTR (no IDR)"
);
true
}
None => {
tracing::info!(
first,
last,
"AMF LTR-RFI: no live LTR older than the loss — falling back to IDR recovery"
);
false
}
}
}
fn caps(&self) -> EncoderCaps {
EncoderCaps {
// AMF has no NVENC-style reference invalidation the intra-refresh wave is the
// loss-recovery substitute; without it every unrecoverable loss costs an IDR.
supports_rfi: false,
// LTR-RFI: AMD's reference invalidation is the user long-term-reference path (mark a
// frame, force a later one to re-reference it). True only when the live driver accepted
// the LTR slots at open — otherwise loss recovery falls back to a full IDR.
supports_rfi: self.ltr_active,
// In-band mastering/CLL via `*InHDRMetadata` (HEVC SEI / AV1 metadata OBU); AVC has
// no such property (and no HDR sessions negotiate H.264).
supports_hdr_metadata: self.ten_bit && self.props.hdr_metadata.is_some(),
@@ -1901,6 +2216,11 @@ impl Encoder for AmfEncoder {
// accepted the property (queried per loss event, so the post-first-frame value is
// what the session glue's IDR rate-limiting sees).
intra_refresh: self.ir_active,
// Not yet: the AMD VCN wave heals in principle, but its constrained-GDR
// heal-within-a-period is unvalidated on-glass and AMF emits no recovery-point SEI, so
// the host keeps the IDR recovery path. Flip both once verified on real hardware.
intra_refresh_recovery: false,
intra_refresh_period: 0,
}
}
@@ -1992,6 +2312,7 @@ impl Encoder for AmfEncoder {
self.inner = None;
self.bound_device = 0;
self.ir_active = false;
self.ltr_active = false;
return true;
}
let inner = self
@@ -2016,8 +2337,14 @@ impl Encoder for AmfEncoder {
sys::AMF_SURFACE_NV12
};
match self.apply_static_props(comp) {
Ok(ir) => {
Ok((ir, ltr)) => {
self.ir_active = ir;
// Re-Init voids the reference history: the rebuilt stream restarts at IDR with
// empty LTR slots, so any prior marks are stale and must be dropped.
self.ltr_active = ltr;
self.ltr_slots = [None; NUM_LTR_SLOTS];
self.next_ltr_slot = 0;
self.pending_force = None;
((*(*comp).vtbl).init)(comp, fmt, self.width as i32, self.height as i32)
== sys::AMF_OK
}
@@ -2030,6 +2357,7 @@ impl Encoder for AmfEncoder {
);
} else {
self.ir_active = false;
self.ltr_active = false;
// Full teardown; the next submit reopens context + component on the current device.
tracing::warn!("AMF in-place re-Init failed — full context teardown, reopening lazily");
self.inner = None;
@@ -339,6 +339,7 @@ fn poll_encoder(enc: &mut encoder::video::Encoder, fps: u32) -> Result<PollOutco
data,
pts_ns: pts * 1_000_000_000 / fps as u64,
keyframe: pkt.is_key(),
recovery_anchor: false,
}))
}
Err(ffmpeg::Error::Other { errno })
@@ -429,10 +429,25 @@ pub struct NvencD3d11Encoder {
async_rt: Option<AsyncRetrieve>,
/// `NV_ENC_CAPS_ASYNC_ENCODE_SUPPORT` from the caps probe — gates the async retrieve mode.
async_supported: bool,
/// (bitstream, mapped input resource to unmap after retrieval, pts_ns) per in-flight encode.
pending: VecDeque<(nv::NV_ENC_OUTPUT_PTR, nv::NV_ENC_INPUT_PTR, u64)>,
/// (bitstream, mapped input resource to unmap after retrieval, pts_ns, recovery-anchor) per
/// in-flight encode. The fourth field tags the first frame encoded after a successful
/// [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) — the clean re-anchor P-frame the
/// client lifts its post-loss freeze on (see [`EncodedFrame::recovery_anchor`]).
pending: VecDeque<(nv::NV_ENC_OUTPUT_PTR, nv::NV_ENC_INPUT_PTR, u64, bool)>,
/// The frame number of the NEXT submission (also its `inputTimeStamp`). Pinned per frame by
/// [`Encoder::submit_indexed`] to the WIRE frame index the AU will carry, so the DPB timestamps
/// `invalidate_ref_frames` compares client frame numbers against stay 1:1 with the wire across
/// encoder rebuilds/resets (an internal counter desyncs on the first adaptive-bitrate rebuild —
/// RFI then never matches again). Self-increments as a fallback for un-indexed callers (tests).
frame_idx: i64,
force_kf: bool,
/// A successful [`invalidate_ref_frames`](Encoder::invalidate_ref_frames) arms this; the next
/// `submit` consumes it into `pending` so that AU ships as the recovery anchor. NVENC applies
/// the invalidation at the next `encode_picture`, so that frame is by construction the first
/// one coded against only-valid references — without tagging it the client's freeze can only
/// lift on an IDR, which the session glue suppresses after an RFI success (the cooldown):
/// a ~1 s frozen stall per loss event on NVIDIA hosts.
pending_anchor: bool,
inited: bool,
/// GPU capabilities probed once via `nvEncGetEncodeCaps` before configuring (Apollo's
/// `get_encoder_cap`): gates 10-bit/custom-VBV/RFI on what this card actually supports instead
@@ -507,6 +522,7 @@ impl NvencD3d11Encoder {
pending: VecDeque::new(),
frame_idx: 0,
force_kf: false,
pending_anchor: false,
inited: false,
rfi_supported: false,
custom_vbv: false,
@@ -536,7 +552,7 @@ impl NvencD3d11Encoder {
while rt.done_rx.try_recv().is_ok() {}
}
// Unmap any in-flight inputs, then unregister every cached texture and destroy the bitstreams.
for (_, map, _) in &self.pending {
for (_, map, _, _) in &self.pending {
if !map.is_null() {
let _ = (api().unmap_input_resource)(self.encoder, *map);
}
@@ -569,8 +585,10 @@ impl NvencD3d11Encoder {
self.inited = false;
self.next = 0;
// The new session starts with an empty DPB (its first frame is an IDR), so any prior
// invalidation range is meaningless against it.
// invalidation range is meaningless against it — and the IDR is itself the re-anchor,
// so a pending anchor tag from a pre-teardown RFI is stale too.
self.last_rfi_range = None;
self.pending_anchor = false;
}
/// Query one `NV_ENC_CAPS` value for this codec on an open session; 0 on any error (treat an
@@ -1133,7 +1151,7 @@ impl NvencD3d11Encoder {
/// error surfaces AFTER the unmap (the resource is retired either way) so the session glue's
/// rebuild path starts from clean state.
fn absorb_done(&mut self, done: RetrieveDone) -> Result<()> {
let Some((bs, map, pts_ns)) = self.pending.pop_front() else {
let Some((bs, map, pts_ns, anchor)) = self.pending.pop_front() else {
bail!("NVENC async: completion with no in-flight frame (pairing bug)");
};
if bs as usize != done.bs {
@@ -1157,6 +1175,7 @@ impl NvencD3d11Encoder {
data,
pts_ns,
keyframe,
recovery_anchor: anchor,
});
Ok(())
}
@@ -1248,6 +1267,11 @@ impl Encoder for NvencD3d11Encoder {
self.init_session(&device)?;
self.init_device = dev_raw;
}
// The session's opening frame — NVENC emits it as an IDR regardless of pic flags, so the
// in-band HDR SEI must ride it too. Detected via the still-empty output slot counter
// (`teardown` zeroes it), NOT via `pts == 0`: `submit_indexed` pins pts to the wire frame
// index, which is non-zero on a mid-session encoder rebuild's first frame.
let opening = self.next == 0;
// Async backpressure: never hand NVENC an output bitstream that is still in flight, and
// keep in-flight depth within the capturer's texture ring (see `async_inflight_cap`). At
// the cap, block on the OLDEST completion (the retrieve thread is already waiting on its
@@ -1322,6 +1346,10 @@ impl Encoder for NvencD3d11Encoder {
} else {
0
};
// Recovery anchor (armed by a successful invalidate_ref_frames): THIS frame is the
// first one encoded after the invalidation — the clean re-anchor. A simultaneous
// forced IDR is itself the re-anchor, so the tag is dropped in that case.
let anchor = std::mem::take(&mut self.pending_anchor) && flags == 0;
let mut pic = nv::NV_ENC_PIC_PARAMS {
version: nv::NV_ENC_PIC_PARAMS_VER,
inputWidth: self.width,
@@ -1348,7 +1376,7 @@ impl Encoder for NvencD3d11Encoder {
// built from the source display's metadata. Any decoder — incl. stock Moonlight — then
// tone-maps from the real grade. HEVC/H.264 carry SEI; AV1 uses metadata OBUs (follow-up).
// The scratch buffers must outlive `encode_picture`, so they live in this scope.
let is_idr = flags != 0 || pts == 0;
let is_idr = flags != 0 || opening;
let mastering_sei = self
.hdr_meta
.map(|m| crate::hdr::hevc_mastering_display_sei(&m));
@@ -1390,8 +1418,12 @@ impl Encoder for NvencD3d11Encoder {
(api().encode_picture)(self.encoder, &mut pic)
.nv_ok()
.map_err(|e| anyhow!("encode_picture: {e:?}"))?;
self.pending
.push_back((self.bitstreams[slot], mp.mappedResource, captured.pts_ns));
self.pending.push_back((
self.bitstreams[slot],
mp.mappedResource,
captured.pts_ns,
anchor,
));
// Async: hand the in-flight encode to the retrieve thread (channel capacity = POOL ≥
// in-flight, so this send never blocks). The pending entry above pairs with its
// completion FIFO in `absorb_done`.
@@ -1408,6 +1440,16 @@ impl Encoder for NvencD3d11Encoder {
Ok(())
}
/// Pin this submission's frame number (= its `inputTimeStamp`) to the wire frame index the AU
/// will carry, so the DPB timestamps `invalidate_ref_frames` matches client frame numbers
/// against are the wire's — 1:1 across every rebuild/reset (see the trait doc). Within a
/// session the loop's prediction is nondecreasing; a repeat after a reset lands on a fresh
/// session (teardown cleared the DPB and `last_rfi_range`), so re-pinning is always sound.
fn submit_indexed(&mut self, frame: &CapturedFrame, wire_index: u32) -> Result<()> {
self.frame_idx = wire_index as i64;
self.submit(frame)
}
fn request_keyframe(&mut self) {
self.force_kf = true;
}
@@ -1424,6 +1466,8 @@ impl Encoder for NvencD3d11Encoder {
// The direct-NVENC path recovers via real RFI (or a forced IDR), not the Linux
// libavcodec intra-refresh mode.
intra_refresh: false,
intra_refresh_recovery: false,
intra_refresh_period: 0,
}
}
@@ -1439,9 +1483,13 @@ impl Encoder for NvencD3d11Encoder {
if self.encoder.is_null() || !self.rfi_supported || first < 0 || first > last {
return false;
}
// Already invalidated a covering range for this loss event — nothing more to do, no IDR.
// Already invalidated a covering range for this loss event — no new driver calls needed,
// no IDR. RE-ARM the anchor though: the client re-asking means the previous recovery
// anchor AU may itself have been lost, and the next frame is just as clean a re-anchor
// (it too references only valid frames).
if let Some((pf, pl)) = self.last_rfi_range {
if first >= pf && last <= pl {
self.pending_anchor = true;
return true;
}
}
@@ -1457,9 +1505,11 @@ impl Encoder for NvencD3d11Encoder {
if first > last {
return false;
}
// We tag each input with `inputTimeStamp = frame_idx` (0,1,2,…), which is also the client's
// frame number (the packetizer numbers frames in submit order), so the client's lost-frame
// range maps 1:1 onto the timestamps NVENC invalidates here.
// Each input's `inputTimeStamp` is `frame_idx`, which `submit_indexed` pins to the WIRE
// frame index the AU carries — so the client's lost-frame range maps 1:1 onto the
// timestamps NVENC invalidates here, and stays 1:1 across encoder rebuilds/resets (an
// internal counter would desync on the first adaptive-bitrate rebuild and RFI would then
// clamp every range into first > last, silently degrading to IDR-only forever).
// SAFETY: `invalidate_ref_frames` is a function pointer from the runtime-loaded `EncodeApi` table.
// `self.encoder` was checked non-null at the top of this fn and is the live session; this runs
// on the encode thread (like submit/poll), so there is no concurrent NVENC use. Each `ts` was
@@ -1477,6 +1527,11 @@ impl Encoder for NvencD3d11Encoder {
}
}
self.last_rfi_range = Some((first, last));
// The next submitted frame is the first one encoded after the invalidation — the clean
// re-anchor P-frame. Arm the tag so its AU ships with `recovery_anchor` and the client
// lifts its post-loss freeze on it (instead of waiting ~1 s for the cooldown-suppressed
// IDR fallback).
self.pending_anchor = true;
true
}
@@ -1501,7 +1556,7 @@ impl Encoder for NvencD3d11Encoder {
.ready
.pop_front());
}
let Some((bs, map, pts_ns)) = self.pending.pop_front() else {
let Some((bs, map, pts_ns, anchor)) = self.pending.pop_front() else {
return Ok(None);
};
// SAFETY: a non-empty `pending` implies `submit` ran, so `self.encoder` is the live session
@@ -1542,10 +1597,28 @@ impl Encoder for NvencD3d11Encoder {
data,
pts_ns,
keyframe,
recovery_anchor: anchor,
}))
}
}
/// Encode-stall recovery: tear the whole session down (the same teardown a capture-device
/// change uses) and let the next `submit` rebuild it lazily on the current device — the owed
/// AUs are forfeited and the fresh session opens on an IDR. Gives the encode-stall watchdog a
/// healing lever on NVENC instead of ending the session. Caveat: the SYNC retrieve mode blocks
/// inside `lock_bitstream`, so a driver wedge that hangs the lock never returns to the loop
/// for the watchdog to fire — this lever fully protects the async retrieve mode (5 s event
/// timeouts surface as poll errors) and the submit-side failure paths.
fn reset(&mut self) -> bool {
// SAFETY: `teardown` (an `unsafe fn`) requires the encode thread with no NVENC call in
// flight and a session whose cached resources belong to `self.encoder` — all hold here
// (reset is called from the session loop between submit/poll, like every other method),
// and it early-returns on an already-null session.
unsafe { self.teardown() };
self.force_kf = true;
true
}
fn flush(&mut self) -> Result<()> {
Ok(()) // P1/ULL + frameIntervalP=1: each submit yields its AU; no internal queue to drain.
}
+39 -8
View File
@@ -436,7 +436,10 @@ fn sendmmsg_all(sock: &UdpSocket, pkts: &[Vec<u8>]) -> std::io::Result<()> {
/// behind encode (measured ~3 ms/frame at 4K, which capped GameStream's frame rate well below what
/// the encoder alone can sustain).
struct RawFrame {
aus: Vec<(Vec<u8>, FrameType)>,
/// `(bitstream, type, wire frameIndex)` per AU. The stream loop assigns the index (it owns
/// the numbering — see its `au_seq`), so the encoder's RFI bookkeeping stays 1:1 with what
/// Moonlight sees across mid-stream encoder rebuilds.
aus: Vec<(Vec<u8>, FrameType, u32)>,
ts: u32,
}
@@ -460,8 +463,8 @@ fn spawn_packetizer(
crate::punktfunk1::boost_thread_priority(false);
while let Ok(frame) = rx.recv() {
let mut batch: PacketBatch = Vec::new();
for (au, ft) in frame.aus {
batch.extend(pk.packetize(&au, ft, frame.ts));
for (au, ft, idx) in frame.aus {
batch.extend(pk.packetize(&au, ft, frame.ts, Some(idx)));
}
if batch.is_empty() {
continue;
@@ -660,6 +663,16 @@ fn stream_body(
// routed through the same coalesce gate as client IDR requests so a burst of drops (congestion)
// can't become an IDR storm.
let mut recover_after_drop = false;
// The stream's wire frameIndex numbering, owned HERE (the index of the next AU handed to the
// packetizer thread; a dropped-at-the-queue frame consumes none). A submission's future index
// is `au_seq + enc_inflight` (AUs are emitted FIFO, one per submission); passing it to
// `Encoder::submit_indexed` keeps the encoder's RFI bookkeeping 1:1 with Moonlight's frame
// numbers across the in-place encoder rebuild above (an internal counter would desync there).
// A pipeline-head drop desyncs the prediction by the dropped AU count for the frames already
// in flight — bounded and self-healing: the drop arms `recover_after_drop`, whose forced IDR
// resets the encoder's reference state (stale LTR/DPB bookkeeping dies with it).
let mut au_seq: u32 = 0;
let mut enc_inflight: u32 = 0;
while running.load(Ordering::SeqCst) {
let tick = Instant::now();
@@ -728,6 +741,10 @@ fn stream_body(
enc.request_keyframe();
last_keyframe = Some(Instant::now());
next_frame = Instant::now();
// The old encoder died with its in-flight submissions — their AUs will never
// arrive, so the numbering prediction restarts at `au_seq` (the fresh encoder's
// reference state is empty, so the reused predictions meet no stale bookkeeping).
enc_inflight = 0;
tracing::info!("gamestream: source rebuilt — stream continues");
continue;
}
@@ -742,7 +759,13 @@ fn stream_body(
if let Some((first, last)) = rfi_range.lock().unwrap().take() {
// Prefer reference-frame invalidation when the encoder supports it (no costly IDR
// spike); otherwise — or if the range is too old to invalidate — fall back to a keyframe.
if !(supports_rfi && enc.invalidate_ref_frames(first, last)) {
// Sanity-cap the range first: wider than RFI_MAX_RANGE exceeds any encoder's reference
// history (or is a phantom range from a desynced counter) — keyframe, never a
// force-reference that could ship corruption as a clean frame.
let width = (last as u32).wrapping_sub(first as u32);
if width > punktfunk_core::packet::RFI_MAX_RANGE
|| !(supports_rfi && enc.invalidate_ref_frames(first, last))
{
want_keyframe = true;
}
}
@@ -766,21 +789,27 @@ fn stream_body(
tracing::debug!("video: keyframe request coalesced (IDR still in flight)");
}
}
enc.submit(&frame).context("encoder submit")?;
enc.submit_indexed(&frame, au_seq.wrapping_add(enc_inflight))
.context("encoder submit")?;
enc_inflight = enc_inflight.wrapping_add(1);
let t_enc = tick.elapsed();
// 90 kHz RTP timestamp from wall-clock, so a variable capture rate stays correct.
let ts = (stream_start.elapsed().as_secs_f64() * 90_000.0) as u32;
// Drain the encoder's access units (owned buffers) — FEC/packetization runs on the
// packetizer thread, off this loop, so it never serializes behind encode.
let mut aus: Vec<(Vec<u8>, FrameType)> = Vec::new();
// packetizer thread, off this loop, so it never serializes behind encode. Each AU is
// stamped with its wire frameIndex here (`au_seq + position`); the numbering only
// ADVANCES if the batch is actually enqueued below (a dropped batch consumes none).
let mut aus: Vec<(Vec<u8>, FrameType, u32)> = Vec::new();
while let Some(au) = enc.poll().context("encoder poll")? {
let ft = if au.keyframe {
FrameType::Idr
} else {
FrameType::P
};
aus.push((au.data, ft));
let idx = au_seq.wrapping_add(aus.len() as u32);
aus.push((au.data, ft, idx));
enc_inflight = enc_inflight.saturating_sub(1);
}
let t_pkt = tick.elapsed();
@@ -788,9 +817,11 @@ fn stream_body(
// (packetizer, or the paced sender behind it) is behind — drop this frame (FEC/RFI covers the
// client) and keep encoding, so a downstream stall can never cap the encode rate.
if !aus.is_empty() {
let batch_len = aus.len() as u32;
match raw_tx.try_send(RawFrame { aus, ts }) {
Ok(()) => {
sent_batches += 1;
au_seq = au_seq.wrapping_add(batch_len);
}
Err(std::sync::mpsc::TrySendError::Full(_)) => {
dropped_batches += 1;
+15 -7
View File
@@ -69,14 +69,22 @@ impl VideoPacketizer {
}
/// Packetize one encoded AU into wire datagrams (data shards + Cauchy RS parity shards).
///
/// `frame_index`: `Some(i)` stamps the caller's index (the stream loop owns the numbering so
/// the encoder's RFI bookkeeping stays 1:1 with the wire across mid-stream encoder rebuilds —
/// see `Encoder::submit_indexed`); `None` draws from the internal counter (tests/harnesses).
pub fn packetize(
&mut self,
au: &[u8],
frame_type: FrameType,
timestamp_90k: u32,
frame_index: Option<u32>,
) -> Vec<Vec<u8>> {
let frame_index = self.frame_index;
self.frame_index = self.frame_index.wrapping_add(1);
let frame_index = frame_index.unwrap_or_else(|| {
let i = self.frame_index;
self.frame_index = i.wrapping_add(1);
i
});
let pps = self.payload_per_shard;
let blocksize = SHARD_HEADER + pps; // = packet_size + 16
let pct = self.fec_percentage;
@@ -235,7 +243,7 @@ mod tests {
let mut pk = VideoPacketizer::new(1392, 0, 0); // data-only; pps = 1392+16-32 = 1376
assert_eq!(pk.payload_per_shard, 1376);
let au = vec![0xABu8; 4000]; // 8+4000 = 4008 → ceil(4008/1376) = 3 data shards
let pkts = pk.packetize(&au, FrameType::Idr, 90_000);
let pkts = pk.packetize(&au, FrameType::Idr, 90_000, None);
assert_eq!(pkts.len(), 3);
for p in &pkts {
assert_eq!(p.len(), SHARD_HEADER + 1376);
@@ -266,7 +274,7 @@ mod tests {
for ps in [0usize, 15, 16, 17, 32] {
let mut pk = VideoPacketizer::new(ps, 20, 2);
assert!(pk.payload_per_shard >= 1, "pps must never be 0 (ps={ps})");
let _ = pk.packetize(&[0xCDu8; 200], FrameType::Idr, 0); // must not panic
let _ = pk.packetize(&[0xCDu8; 200], FrameType::Idr, 0, None); // must not panic
}
}
@@ -274,7 +282,7 @@ mod tests {
fn multi_block_split() {
let mut pk = VideoPacketizer::new(1392, 0, 0); // data-only
let au = vec![0u8; 600_000];
let pkts = pk.packetize(&au, FrameType::P, 0);
let pkts = pk.packetize(&au, FrameType::P, 0, None);
let total = (8 + au.len()).div_ceil(1376);
assert_eq!(pkts.len(), total);
let n_blocks = total.div_ceil(255).clamp(1, 4);
@@ -286,7 +294,7 @@ mod tests {
fn emits_parity_shards() {
let mut pk = VideoPacketizer::new(1392, 20, 0); // pps = 1376, 20% FEC
let au = vec![0xABu8; 4000]; // 8+4000 = 4008 → 3 data shards (k=3)
let pkts = pk.packetize(&au, FrameType::Idr, 0);
let pkts = pk.packetize(&au, FrameType::Idr, 0, None);
// m = ceil(3*20/100) = 1 parity shard → 4 packets; wire_pct = 100*1/3 = 33.
assert_eq!(pkts.len(), 4);
for p in &pkts {
@@ -313,7 +321,7 @@ mod tests {
fn parity_recovers_full_datagram_incl_flags() {
let mut pk = VideoPacketizer::new(1392, 50, 0); // high pct → plenty of parity
let au = vec![0x5Au8; 4000]; // k = 3
let pkts = pk.packetize(&au, FrameType::Idr, 0);
let pkts = pk.packetize(&au, FrameType::Idr, 0, None);
let k = 3usize;
let m = pkts.len() - k;
assert!(m >= 1);
@@ -687,6 +687,88 @@ fn alloc_pitched_nv12(
Ok(((y_ptr, y_pitch), (uv_ptr, uv_pitch)))
}
/// Allocate ONE pitched buffer holding a *contiguous* NV12 surface — Y rows `[0, H)` immediately
/// followed by interleaved-chroma rows `[H, 3H/2)`, all at the driver's single pitch. Unlike
/// [`alloc_pitched_nv12`] (two separate allocations, the capture/IPC layout) this is the layout the
/// direct-SDK NVENC encoder registers as a single `CUDADEVICEPTR` input: NVENC reads the UV plane
/// at `ptr + pitch*height`. Used only by [`InputSurface`] (encode side), never the wire.
fn alloc_pitched_nv12_contiguous(width: u32, height: u32) -> Result<(CUdeviceptr, usize)> {
let mut ptr: CUdeviceptr = 0;
let mut pitch: usize = 0;
// Y is `width` bytes/row × H rows; the interleaved chroma plane is W/2 samples × 2 bytes =
// `width` bytes/row × H/2 rows. One allocation of `H + H/2` rows keeps them contiguous under a
// single pitch so NVENC finds UV at `ptr + pitch*H`.
let rows = height as usize + (height as usize / 2).max(1);
// SAFETY: `cuMemAllocPitch_v2` (wrapper → live table) writes the allocation pointer and pitch
// into the two live, distinct stack out-params `&mut ptr`/`&mut pitch`, which outlive the
// synchronous call; width/rows/element-size are by-value ints. No aliasing.
unsafe {
ck(
cuMemAllocPitch_v2(&mut ptr, &mut pitch, width as usize, rows, 16),
"cuMemAllocPitch_v2(NV12 contiguous)",
)?;
}
Ok((ptr, pitch))
}
/// An encoder-owned, contiguous pitched CUDA surface that the direct-SDK NVENC Linux backend
/// (`encode/linux/nvenc_cuda.rs`, design/linux-direct-nvenc.md) registers **once** as a
/// `NV_ENC_INPUT_RESOURCE_TYPE_CUDADEVICEPTR` input and copies each captured frame into (via the
/// `copy_*_to_device` helpers) before `encode_picture`. Distinct from [`DeviceBuffer`]: these are
/// laid out exactly as NVENC's single-pointer register expects — NV12 = Y then interleaved-UV under
/// one pitch, YUV444 = Y|U|V stacked, RGB = packed 4-byte — and are never pooled or sent on the
/// wire. Frees its allocation on drop (context made current first, since drop may run off-thread).
pub struct InputSurface {
/// Base device pointer NVENC registers. For NV12 the chroma plane lives at `ptr + pitch*height`;
/// for YUV444 the U/V planes at `ptr + pitch*height` / `ptr + 2*pitch*height`.
pub ptr: CUdeviceptr,
/// Row stride in bytes (the driver's pitch), shared by every plane of the surface.
pub pitch: usize,
/// Luma height in rows — the plane stride multiplier NVENC / the copy helpers key off.
pub height: u32,
}
impl InputSurface {
/// Contiguous NV12 (8-bit 4:2:0): one allocation, Y then interleaved UV under one pitch.
pub fn alloc_nv12(width: u32, height: u32) -> Result<InputSurface> {
let (ptr, pitch) = alloc_pitched_nv12_contiguous(width, height)?;
Ok(InputSurface { ptr, pitch, height })
}
/// Planar YUV444 (8-bit 4:4:4): one allocation, Y|U|V full-res planes stacked (see
/// [`alloc_pitched_yuv444`]).
pub fn alloc_yuv444(width: u32, height: u32) -> Result<InputSurface> {
let (ptr, pitch) = alloc_pitched_yuv444(width, height)?;
Ok(InputSurface { ptr, pitch, height })
}
/// Packed 4-byte RGB/BGRx: one contiguous pitched allocation (NVENC does the internal CSC when
/// registered as an `ABGR`/`ARGB` input).
pub fn alloc_rgb(width: u32, height: u32) -> Result<InputSurface> {
let (ptr, pitch) = alloc_pitched(width, height)?;
Ok(InputSurface { ptr, pitch, height })
}
}
impl Drop for InputSurface {
fn drop(&mut self) {
if self.ptr == 0 {
return;
}
// SAFETY: this surface exclusively owns `self.ptr` (a single `cuMemAllocPitch_v2` allocation
// from one of the constructors above), freed exactly once here — `drop` runs once and the
// `ptr == 0` guard skips a moved-out/empty surface, so no double-free. The shared context is
// made current first because drop may run on a thread where it isn't, and `cuMemFree_v2`
// needs it. Wrapper → live table; result ignored (best-effort teardown).
unsafe {
if let Some(c) = CONTEXT.get() {
let _ = cuCtxSetCurrent(c.0);
}
let _ = cuMemFree_v2(self.ptr);
}
}
}
/// Free-list of recycled device allocations for one resolution. Shared (via `Arc`) between the
/// capture thread that hands out buffers and the encode thread where a [`DeviceBuffer`] drops and
/// returns its allocation here. Bulk-freed when the last reference drops. For NV12 each free entry
+10 -1
View File
@@ -20,6 +20,11 @@
mod audio;
mod capture;
/// Host-side shared-clipboard backend. The wire protocol + client live in `punktfunk-core`; this
/// drives the host session's real clipboard (`design/clipboard-and-file-transfer.md` §4). Linux uses
/// Wayland data-control / Mutter; Windows uses the Win32 clipboard (delayed rendering).
#[cfg(any(target_os = "linux", target_os = "windows"))]
mod clipboard;
mod config;
mod discovery;
mod wol;
@@ -696,10 +701,14 @@ fn parse_spike(args: &[String]) -> Result<Options> {
"--source" => {
source = match next()?.as_str() {
"synthetic" => Source::Synthetic,
"synthetic-nv12" => Source::SyntheticNv12,
"portal" => Source::Portal,
"kwin-virtual" => Source::KwinVirtual,
other => {
bail!("unknown --source '{other}' (synthetic|portal|kwin-virtual)")
bail!(
"unknown --source '{other}' \
(synthetic|synthetic-nv12|portal|kwin-virtual)"
)
}
}
}
+5 -5
View File
@@ -1008,11 +1008,11 @@ struct DisplaySettingsState {
fn preset_summary(id: &str) -> &'static str {
match id {
"default" => "Today's behavior: a short linger absorbs reconnects, the streamed output is the sole desktop, extra clients get their own view.",
"gaming-rig" => "Dedicated couch/headless box: the game and its display survive disconnects; whoever connects takes the box over.",
"shared-desktop" => "A desktop you also use in person: never blank the real monitors, never keep ghost displays, concurrent viewers each get a view.",
"hotdesk" => "One user at a time with fast reattach; a second user is told the box is busy; each device+resolution keeps its own scaling.",
"workstation" => "Multi-monitor daily driver: your displays come back exactly where you arranged them, per-client identity, exclusive.",
"default" => "Good for most setups. Reconnects resume quickly, the stream is the whole desktop, and extra viewers each get their own screen.",
"gaming-rig" => "For a machine with no monitor that you only stream from. The game keeps running when you disconnect, and whoever connects next takes it over.",
"shared-desktop" => "For a PC you also use in person. Your real monitors are never blanked or left with a leftover display, and extra viewers each get their own screen.",
"hotdesk" => "One person at a time — roam between your own devices with an instant reconnect. Anyone else is told the box is busy.",
"workstation" => "Your multi-monitor daily driver. Displays come back exactly where you arranged them, each client keeps its own settings, and the desktop is yours alone.",
_ => "",
}
}
+4
View File
@@ -26,12 +26,16 @@ pub fn pump_once(
data,
pts_ns,
keyframe,
recovery_anchor,
}) = encoder.poll()?
{
let mut flags = FLAG_PIC as u32;
if keyframe {
flags |= FLAG_SOF as u32;
}
if recovery_anchor {
flags |= punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR;
}
// core does FEC + packetize + pace + send.
session.submit_frame(&data, pts_ns, flags)?;
}
File diff suppressed because it is too large Load Diff
+30
View File
@@ -22,6 +22,11 @@ use std::time::Instant;
pub enum Source {
/// Deterministic moving BGRx test pattern — no capture session required.
Synthetic,
/// Deterministic moving NV12 texture on the GPU (Windows only) — no capture session required.
/// Feeds the native AMF / D3D11 zero-copy encoders, which demand an NV12 GPU texture the CPU
/// `Synthetic` source can't give them. Used to validate GPU-encoder behaviour (e.g. AMF
/// intra-refresh) headlessly.
SyntheticNv12,
/// Live monitor via the xdg ScreenCast portal + PipeWire.
Portal,
/// KWin virtual output created at `width`x`height` (zkde_screencast). Lets us validate
@@ -56,6 +61,31 @@ pub fn run(opts: Options) -> Result<()> {
);
Box::new(SyntheticCapturer::new(opts.width, opts.height, opts.fps))
}
Source::SyntheticNv12 => {
#[cfg(target_os = "windows")]
{
tracing::info!(
width = opts.width,
height = opts.height,
fps = opts.fps,
"spike source: synthetic NV12 GPU texture (moving luma ramp)"
);
Box::new(
capture::synthetic_nv12::SyntheticNv12Capturer::new(
opts.width,
opts.height,
opts.fps,
)
.context("open synthetic NV12 capturer")?,
)
}
#[cfg(not(target_os = "windows"))]
{
anyhow::bail!(
"--source synthetic-nv12 is Windows-only (native AMF / D3D11 encoders)"
);
}
}
Source::Portal => {
tracing::info!("spike source: xdg ScreenCast portal (live monitor)");
capture::open_portal_monitor().context("open portal capturer")?
+9
View File
@@ -79,6 +79,13 @@ pub struct VirtualOutput {
/// keep-alive pool is Linux).
#[cfg(target_os = "linux")]
pub reused_gen: Option<u64>,
/// The registry pool generation of this display (fresh AND reused — unlike `reused_gen`), so a
/// mid-stream mode-switch rebuild can [`registry::retire`](crate::vdisplay::registry::retire) the
/// display it supersedes instead of leaving it to accumulate under a linger/forever keep-alive
/// policy (`design/midstream-resolution-resize.md` H4). `None` for non-poolable outputs.
/// Linux-only (the keep-alive pool is Linux).
#[cfg(target_os = "linux")]
pub pool_gen: Option<u64>,
}
impl VirtualOutput {
@@ -100,6 +107,8 @@ impl VirtualOutput {
ownership: DisplayOwnership::Owned,
#[cfg(target_os = "linux")]
reused_gen: None,
#[cfg(target_os = "linux")]
pool_gen: None,
}
}
}
@@ -241,6 +241,7 @@ impl VirtualDisplay for GamescopeDisplay {
keepalive: Box::new(()),
ownership: DisplayOwnership::External,
reused_gen: None,
pool_gen: None,
});
}
check_gamescope_version(); // diagnostic only — warns on known-deadlock-prone versions
@@ -366,6 +367,7 @@ fn managed_output(node_id: u32, mode: Mode) -> VirtualOutput {
keepalive: Box::new(()),
ownership: DisplayOwnership::SessionManaged,
reused_gen: None,
pool_gen: None,
}
}
@@ -198,6 +198,7 @@ impl VirtualDisplay for HyprlandDisplay {
// `remote_fd.is_some()` — same as wlroots.
ownership: DisplayOwnership::Owned,
reused_gen: None,
pool_gen: None,
})
}
}
@@ -135,6 +135,7 @@ impl VirtualDisplay for WlrootsDisplay {
// `remote_fd.is_some()` (keep-alive stays off for wlroots until fresh-portal re-attach).
ownership: DisplayOwnership::Owned,
reused_gen: None,
pool_gen: None,
})
}
}
+33 -4
View File
@@ -176,6 +176,21 @@ pub fn mark_failed(gen: u64) {
let _ = gen;
}
/// Force-release a **superseded** kept display by its generation stamp
/// (`design/midstream-resolution-resize.md` H4): after a mid-stream mode-switch rebuild, the old
/// display's lease drop is indistinguishable from a disconnect, so under a `linger`/`forever`
/// keep-alive policy every resize would accumulate kept monitors at stale modes. The mode-switch
/// arm calls this once the new pipeline is up and the old capturer is dropped. Only a KEPT
/// (lingering/pinned) entry is released — an Active one is refused, like `/display/release` — and
/// a gen that's already gone (immediate teardown) is a no-op. No-op off Linux (Windows
/// reconfigures the same monitor in place — nothing is superseded).
pub fn retire(gen: u64) {
#[cfg(target_os = "linux")]
linux::retire(gen);
#[cfg(not(target_os = "linux"))]
let _ = gen;
}
/// Invalidate every kept display of `backend` — its compositor instance is gone (a Game↔Desktop switch
/// tore it down), so `/display/state` must stop listing it and its keepalive must be reaped
/// (`design/gamemode-and-dedicated-sessions.md` A4). Called from the session-switch watcher / a
@@ -386,6 +401,9 @@ mod linux {
// A2: tell the pipeline builder this was a REUSED kept display, so a first-frame failure can
// `mark_failed(gen)` (tear the corpse down) rather than re-wedge the retry loop on the same node.
out.reused_gen = reused.then_some(gen);
// H4: every pooled display carries its gen, so a mode-switch rebuild can `retire` the entry
// this output's successor supersedes.
out.pool_gen = Some(gen);
out
}
@@ -819,6 +837,20 @@ mod linux {
}
pub(super) fn force_release(slot: Option<u64>) -> usize {
release_kept(slot, "released (mgmt /display/release)")
}
/// H4 — force-release a display superseded by a mid-stream mode switch. Same machinery as
/// [`force_release`] (kept entries only — an Active entry is refused, and a gen already torn
/// down under `immediate` is a no-op), distinct log line.
pub(super) fn retire(gen: u64) {
release_kept(Some(gen), "retired (superseded by a mode switch)");
}
/// Remove + tear down KEPT (lingering/pinned) entries — all of them, or one by gen — running /
/// handing off group topology restores, with keepalive drops outside the lock. The shared core
/// of [`force_release`] (mgmt) and [`retire`] (mode-switch supersede).
fn release_kept(slot: Option<u64>, why: &'static str) -> usize {
let Some(r) = REG.get() else { return 0 };
let (released, restores) = {
let mut es = r.entries.lock().unwrap();
@@ -847,10 +879,7 @@ mod linux {
restore();
}
for e in released {
tracing::info!(
backend = e.backend,
"virtual display released (mgmt /display/release)"
);
tracing::info!(backend = e.backend, "virtual display {why}");
drop(e);
}
n
@@ -557,31 +557,68 @@ impl VirtualDisplayManager {
// This slot already has a live monitor — join it (refcount++). Covers same-client concurrent
// sessions AND the build-then-drop overlap of a mid-stream Reconfigure (the new lease is taken
// while the old is still held). Reconfigure the shared monitor if the requested mode differs.
if let Some(SlotState::Active { mon, refs }) = inner.slots.get_mut(&slot) {
*refs += 1;
let reconfigured = mon.mode != mode;
if reconfigured {
// SAFETY: `reconfigure` only manipulates the live display topology via the CCD/GDI
// helpers and needs an exclusive `&mut Monitor`. `mon` is the `&mut` into this slot's
// `Active` state, held under the `state` lock, so nothing else reconfigures it
// concurrently.
unsafe { self.reconfigure(mon, mode) };
// while the old is still held).
if matches!(inner.slots.get(&slot), Some(SlotState::Active { .. })) {
// A DIFFERENT mode is a mid-stream resize (Reconfigure). The pf-vdisplay driver freezes its
// advertised mode list at ADD time, so we can't reach an arbitrary new mode in place — RE-
// ARRIVE the monitor at the exact mode instead (Fix 1). Own the slot for the swap: `re_add`
// needs `&mut inner` for the topology re-isolate, which the borrowed `mon` would block.
let cur_mode = match inner.slots.get(&slot) {
Some(SlotState::Active { mon, .. }) => mon.mode,
_ => unreachable!("just matched Active"),
};
if cur_mode != mode {
let Some(SlotState::Active { mon, refs }) = inner.slots.remove(&slot) else {
unreachable!("just matched Active");
};
// SAFETY: `dev` is the handle `ensure_device()` returned above; `re_add` touches the
// live topology under the held `state` lock. `mon` is owned here (removed from the map).
let new_mon =
match unsafe { self.re_add(dev, &mut inner, slot, &mon, mode, client_hdr) } {
Ok(m) => m,
Err(e) => {
// The re-arrival failed — put the OLD monitor back so the session keeps
// streaming its current mode (the control task already acked the switch; the
// rebuild reuses the old target and Fix 2's corrective ack tells the client the
// resolution didn't change). Its `gen`/`refs` are intact, so leases stay valid.
inner.slots.insert(slot, SlotState::Active { mon, refs });
return Err(e).context("mid-stream resize re-arrival");
}
};
// `re_add` preserved `gen`, so both the old session's lease and this new one match on
// release. +1 ref for the new (build-then-drop overlap) lease.
let out = self.output_for(slot, &new_mon, quit);
inner.slots.insert(
slot,
SlotState::Active {
mon: new_mon,
refs: refs + 1,
},
);
// The width changed — re-arrange the group so auto-row siblings don't overlap the
// resized display (no-op for a single member).
self.apply_group_layout(&mut inner);
tracing::info!(
slot,
refs = refs + 1,
backend = self.driver.name(),
"virtual monitor re-arrived for a mid-stream resize"
);
return Ok(out);
}
// Same mode — a plain concurrent-session JOIN (refcount++), no re-arrival.
let Some(SlotState::Active { mon, refs }) = inner.slots.get_mut(&slot) else {
unreachable!("just matched Active");
};
*refs += 1;
tracing::info!(
slot,
refs = *refs,
backend = self.driver.name(),
"virtual monitor reused (concurrent / reconfigure session)"
"virtual monitor reused (concurrent session)"
);
warn_if_pick_moved(mon);
let out = self.output_for(slot, mon, quit);
if reconfigured {
// A mode change alters this member's width — re-arrange the group so auto-row
// siblings don't overlap the resized display (no-op for a single member).
self.apply_group_layout(&mut inner);
}
return Ok(out);
return Ok(self.output_for(slot, mon, quit));
}
// Display budget (Stage W3): a display we can't afford is DECLINED at admission
@@ -761,6 +798,53 @@ impl VirtualDisplayManager {
}
/// Create a fresh monitor at `mode` for `slot` (the client's stable identity slot, `0` = auto):
/// Wait for Windows to auto-activate a freshly-ADDed IDD target into its OWN display path and
/// return its GDI name — the capture target. Shared by the fresh CREATE and the mid-stream
/// re-arrival ([`re_add`](Self::re_add)).
///
/// The IDD comes up EXTENDED alongside any existing/basic display; the caller then promotes it to
/// primary / isolates it. Returns `None` on a GPU-less box (target added but not WDDM-activated) —
/// the capture backend re-resolves once a GPU is present.
///
/// We do NOT force a topology change FIRST: the bare `SDC_TOPOLOGY_EXTEND` preset is ACCESS_DENIED
/// from our Session-0 service context on a headless box and BREAKS this auto-activate (it regressed
/// the headless path — the IDD then never gets its own path → "not an active display path" → black).
/// force-EXTEND is only the FALLBACK, for an integrated-screen box (e.g. a laptop panel) where a
/// fresh IDD is CLONED onto the existing display, sharing its source, so it never gets its own
/// committed path (observed on an Intel-iGPU + NVIDIA-Optimus laptop, commit 8e87e61):
/// `resolve_gdi_name` stays None → the `is_none()` fallback force-EXTENDs to de-clone and the
/// second resolve finds the now-committed path. Headless/extended boxes resolve on the first loop
/// and skip it — which is the point, since force-EXTEND is ACCESS_DENIED from our service context
/// there.
///
/// CAVEAT (unobserved for IddCx, untested across GPU/driver/OS): textbook CCD also lets a clone
/// appear as a *shared-source ACTIVE* path (resolve → Some), which the `is_none()` gate would NOT
/// catch. If that ever shows up, widen the gate to also fire when the IDD target's source is shared
/// with another active path (a `target_is_cloned` helper) — needs on-laptop validation first.
///
/// # Safety
/// Runs the CCD (QueryDisplayConfig / SetDisplayConfig) FFI; call under the `state` lock.
unsafe fn resolve_target_gdi(&self, target_id: u32) -> Option<String> {
for _ in 0..15 {
thread::sleep(Duration::from_millis(200));
// SAFETY: `resolve_gdi_name` is `unsafe` for its CCD FFI; it takes a plain `Copy` `u32`
// target id by value and returns an owned `String`, so no caller memory is borrowed.
if let Some(n) = unsafe { resolve_gdi_name(target_id) } {
return Some(n);
}
}
// SAFETY: `force_extend_topology` only calls `SetDisplayConfig` (CCD) with no borrowed memory.
unsafe { force_extend_topology() };
for _ in 0..15 {
thread::sleep(Duration::from_millis(200));
// SAFETY: as the resolve loop above.
if let Some(n) = unsafe { resolve_gdi_name(target_id) } {
return Some(n);
}
}
None
}
/// ADD via the driver (pinning the discrete render GPU under the usual conditions), ensure the
/// device-level watchdog pinger, resolve the GDI name, force the mode + apply the GROUP topology
/// (first member isolates and captures the restore; a later member re-issues the isolate with
@@ -796,53 +880,10 @@ impl VirtualDisplayManager {
self.ensure_pinger();
// Resolve the capture target — wait for Windows to auto-activate the freshly-ADDed IDD into its
// OWN display path (it comes up EXTENDED alongside any existing/basic display; `set_active_mode`
// below then promotes it to primary and `isolate_displays_ccd` makes it the sole composited
// desktop — the proven flow). May be None on a GPU-less box (target added but not WDDM-activated);
// the capture backend re-resolves once a GPU is present.
//
// We do NOT force a topology change FIRST: the bare `SDC_TOPOLOGY_EXTEND` preset is ACCESS_DENIED
// from our Session-0 service context on a headless box and BREAKS this auto-activate (it regressed
// the headless path — the IDD then never gets its own path → "not an active display path" → black).
// force-EXTEND is only the FALLBACK below, for an integrated-screen box where a fresh IDD is CLONED
// onto the panel (shares its source) instead of getting its own path.
let mut gdi_name = None;
for _ in 0..15 {
thread::sleep(Duration::from_millis(200));
// SAFETY: `resolve_gdi_name` is `unsafe` for its CCD (QueryDisplayConfig) FFI; it takes a
// plain `Copy` `u32` target id by value and returns an owned `String`, so no caller memory
// is borrowed across the call.
if let Some(n) = unsafe { resolve_gdi_name(added.target_id) } {
gdi_name = Some(n);
break;
}
}
// Fallback for an integrated-screen box (e.g. a laptop panel): Windows CLONES a freshly-added
// IDD onto the existing display, sharing its source, so it never gets its own committed path. On
// the IddCx clone behaviour observed live (commit 8e87e61, an Intel-iGPU + NVIDIA-Optimus laptop)
// `resolve_gdi_name` then stays None — so this `is_none()` fallback fires, force-EXTENDs to
// de-clone, and the second resolve finds the now-committed path. Headless/extended boxes already
// resolved above (the IDD auto-activates with its OWN source) and skip this — which is the whole
// point, since force-EXTEND's bare preset is ACCESS_DENIED from our service context there.
//
// CAVEAT (unobserved for IddCx, untested across GPU/driver/OS): textbook CCD also lets a clone
// appear as a *shared-source ACTIVE* path (resolve → Some), which this `is_none()` gate would NOT
// catch. If that ever shows up, widen the gate to also fire when the IDD target's source is shared
// with another active path (a `target_is_cloned` helper) — needs on-laptop validation first.
if gdi_name.is_none() {
// SAFETY: as above — `force_extend_topology` only calls `SetDisplayConfig` (CCD) with no
// borrowed caller memory, under the `state` lock.
unsafe { force_extend_topology() };
for _ in 0..15 {
thread::sleep(Duration::from_millis(200));
// SAFETY: as the resolve loop above.
if let Some(n) = unsafe { resolve_gdi_name(added.target_id) } {
gdi_name = Some(n);
break;
}
}
}
// OWN display path, with the integrated-screen clone fallback (shared by the re-arrival path).
// SAFETY: `resolve_target_gdi` runs the CCD FFI (a `Copy` `u32` target by value, owned return),
// under the `state` lock.
let gdi_name = unsafe { self.resolve_target_gdi(added.target_id) };
match &gdi_name {
Some(n) => {
tracing::info!(backend = self.driver.name(), "target {} -> {n}", added.target_id);
@@ -974,28 +1015,134 @@ impl VirtualDisplayManager {
})
}
/// Re-apply a (possibly new) mode to a reused monitor on reconnect, re-resolving its GDI name.
/// Mid-stream resize by monitor RE-ARRIVAL (`design/midstream-resolution-resize.md` Fix 1).
///
/// The pf-vdisplay driver freezes a monitor's advertised mode list at `IOCTL_ADD` time (the
/// requested mode + `default_modes()`), so a plain `ChangeDisplaySettingsExW` can only reach a
/// mode the monitor advertised on arrival — an out-of-list target (e.g. a session that arrived at
/// 1080p resizing to 1440p) returns `DISP_CHANGE_BADMODE`. IddCx exposes no live "update modes"
/// DDI, so to follow the client to an ARBITRARY new mode we REMOVE the driver monitor and ADD a
/// fresh one at the new mode, reusing the slot's stable per-client id (EDID serial / ConnectorIndex
/// / ContainerId) so the OS keeps the monitor's identity + saved per-monitor DPI. The visible cost
/// is one monitor hotplug per switch (the design's accepted "re-arrival for everything").
///
/// Refcount/lease continuity: the rebuilt `Monitor` PRESERVES the old `gen`, so the outstanding
/// session lease(s) still match on release — the linger/refcount machine is untouched. The group
/// restore snapshot (`group.ccd_saved` / DDC / PnP) is likewise PRESERVED (a mid-session swap, not
/// a first-member create): [`reisolate_after_swap`](Self::reisolate_after_swap) re-isolates the new
/// target without recapturing it. Caller owns the slot's `Monitor` + `refs` across this call.
///
/// # Safety
/// Touches the live display topology via the CCD/GDI helpers.
unsafe fn reconfigure(&self, mon: &mut Monitor, mode: Mode) {
/// `dev` must be the live control handle; touches the live display topology via CCD/GDI.
unsafe fn re_add(
&'static self,
dev: HANDLE,
inner: &mut MgrInner,
slot: u32,
old: &Monitor,
mode: Mode,
client_hdr: Option<punktfunk_core::quic::HdrMeta>,
) -> Result<Monitor> {
tracing::info!(
old = format!(
"{}x{}@{}",
mon.mode.width, mon.mode.height, mon.mode.refresh_hz
),
new = format!("{}x{}@{}", mode.width, mode.height, mode.refresh_hz),
"virtual-display: reconfiguring reused monitor to the new client mode"
slot,
old = %format!("{}x{}@{}", old.mode.width, old.mode.height, old.mode.refresh_hz),
new = %format!("{}x{}@{}", mode.width, mode.height, mode.refresh_hz),
old_target = old.target_id,
"virtual-display: re-arriving monitor for a mid-stream resize (exact mode)"
);
// SAFETY: `resolve_gdi_name` is `unsafe` for its CCD FFI; it takes the `Copy` `u32`
// `mon.target_id` by value and returns an owned `String`, so nothing borrowed crosses the call.
if let Some(n) = unsafe { resolve_gdi_name(mon.target_id) } {
mon.gdi_name = Some(n);
// 1. Depart the OLD driver monitor — a bare REMOVE IOCTL (no topology restore, pinger stays
// up): the surviving/grown-set re-isolate happens after the new ADD. Frees the preferred id
// so the ADD below can reuse the same stable identity. Best-effort — a REMOVE failure still
// lets the ADD proceed (the driver reaps a stale same-id monitor on the next create anyway).
// SAFETY: `dev` is the live control handle (this fn's contract); `&old.key` borrows the
// still-owned `MonitorKey`, alive across the synchronous IOCTL.
if let Err(e) = unsafe { self.driver.remove_monitor(dev, &old.key) } {
tracing::warn!(
old_target = old.target_id,
"re-arrival REMOVE failed (continuing to ADD): {e:#}"
);
}
if let Some(n) = &mon.gdi_name {
set_active_mode(n, mode);
// Let the OS finish the ASYNC monitor departure before the ADD — a back-to-back REMOVE→ADD
// races the teardown and the ADD is rejected under churn (same 400 ms settle as the reconnect
// preempt path).
thread::sleep(Duration::from_millis(400));
// 2. ADD a fresh monitor at the NEW mode, reusing the slot as the preferred (stable) id.
let render_pin = resolve_render_pin();
// SAFETY: `dev` is the live control handle; `render_pin`/`client_hdr` are owned `Copy`/`Option`
// values passed by value — no borrow crosses the call.
let added = unsafe {
self.driver
.add_monitor(dev, mode, render_pin, slot, client_hdr)
.context("re-arrival ADD at the new mode")?
};
self.ensure_pinger();
// 3. Resolve the NEW target's GDI name (target_id changes across a re-arrival).
// SAFETY: CCD FFI over a `Copy` target id, under the `state` lock.
let gdi_name = unsafe { self.resolve_target_gdi(added.target_id) };
match &gdi_name {
Some(n) => {
tracing::info!(backend = self.driver.name(), "re-arrival target {} -> {n}", added.target_id);
// ADD only advertises the mode; force it active so DXGI/IDD captures the new size.
set_active_mode(n, mode);
// 4. Re-isolate the composited set with the NEW target replacing the old — preserving
// the group's first-member restore snapshot.
// SAFETY: CCD FFI over borrowed Copy target ids, under the `state` lock.
unsafe { self.reisolate_after_swap(inner, added.target_id) };
thread::sleep(Duration::from_millis(1500)); // let the topology settle before capture reopens
}
None => tracing::warn!(
"re-arrival target {} not yet an active display path (needs a WDDM GPU to activate)",
added.target_id
),
}
// 5. Rebuild the Monitor from the ADD reply, PRESERVING `gen` (lease/refcount continuity) and
// the group-layout `position`. A fresh `gen` would strand the old session's lease release.
Ok(Monitor {
key: added.key,
target_id: added.target_id,
luid: added.luid,
render_pin,
wudf_pid: added.wudf_pid,
gdi_name,
mode,
resolved_monitor_id: added.resolved_monitor_id,
position: old.position,
gen: old.gen,
})
}
/// Re-isolate the composited display set after a mid-stream monitor re-arrival ([`re_add`]) put a
/// NEW target in place of the old one — WITHOUT recapturing the group restore snapshot (the first
/// member captured it at session start; teardown restores that, not the mid-session state). The
/// old slot has already been removed from the map by the caller, so `inner.target_ids()` is the
/// surviving siblings; the new target joins them.
///
/// # Safety
/// Drives the CCD topology FFI; call under the `state` lock.
unsafe fn reisolate_after_swap(&self, inner: &mut MgrInner, new_target: u32) {
use crate::vdisplay::policy::Topology;
match topology_action() {
Topology::Exclusive => {
// Grown-set semantics: isolate to the surviving siblings + the new target. The returned
// snapshot is DISCARDED — the group keeps the first member's (design §6.1).
let mut keep = inner.target_ids();
keep.push(new_target);
// SAFETY: borrowed slice of Copy target ids, owned return, under the `state` lock.
let _ = unsafe { isolate_displays_ccd(&keep) };
}
Topology::Primary => {
// Make the new target primary again (its predecessor held primary), preserving the
// original restore snapshot: `set_virtual_primary_ccd` recaptures one, so save + restore
// the group's around the call.
let keep_saved = inner.group.ccd_saved.take();
// SAFETY: `Copy` target id by value, owned return, under the `state` lock.
let _ = unsafe { set_virtual_primary_ccd(new_target) };
inner.group.ccd_saved = keep_saved;
}
Topology::Extend | Topology::Auto => {
// The re-ADDed target auto-activates extended — nothing to isolate/promote.
}
}
mon.mode = mode;
}
/// Tear down `mon`, which the caller has ALREADY removed from `inner.slots`: on the LAST member
+1 -1
View File
@@ -85,7 +85,7 @@ fallback without one. More detail — including the CLI `punktfunk-host service
systemctl --user enable --now punktfunk-web
```
Then open `http://<host-ip>:47992`. Reading its [login password](/docs/web-console#login-password)
Then open `https://<host-ip>:47992`. Reading its [login password](/docs/web-console#login-password)
and [arming PIN pairing](/docs/web-console#arm-pairing) are covered in
[The Web Console](/docs/web-console).
+2 -1
View File
@@ -86,7 +86,8 @@ When it finishes it prints the web-console URL and how to pair.
By default the host **requires PIN pairing** (secure). Two ways to pair:
- **Web console** (printed at the end of step 2): open `http://<device-ip>:47992`,
- **Web console** (printed at the end of step 2): open `https://<device-ip>:47992` (self-signed host
cert — your browser warns once; trust it and continue),
[arm pairing](/docs/web-console#arm-pairing), and enter the PIN on your client.
- **From the client directly**: pick this host (it advertises over mDNS as `_punktfunk._udp`) and
enter the PIN the host shows.
+9 -9
View File
@@ -15,8 +15,8 @@ on Windows). A change applies to the **next** connection — a running session k
opened on.
> **You rarely need to touch this.** The default behavior matches how punktfunk has always worked.
> Reach for a preset when you want a specific experience — a dedicated couch/gaming box, a desktop
> you also use in person, or a multi-monitor workstation.
> Reach for a preset when you want a specific experience — a dedicated box you only stream from, a
> desktop you also use in person, or a multi-monitor workstation.
> **What's live today:** **keep-alive** (linger, or **forever**), **topology** (extend / primary /
> exclusive), **conflict handling**, **per-client identity + persistent scaling** (Windows, KDE/KWin
@@ -32,11 +32,11 @@ the individual options documented further down.
| Preset | What it's for |
|---|---|
| **Default** | Today's behavior. A short linger absorbs reconnects, the streamed output becomes the sole desktop, and extra clients each get their own view. |
| **Gaming rig** | A dedicated couch/headless box. The game and its display survive disconnects indefinitely (keep-alive **forever**), and whoever connects takes the box over. Release it from the console when you're done. |
| **Shared desktop** | A desktop you also use in person. punktfunk never blanks your real monitors and never leaves a ghost display behind; concurrent viewers each get a view. |
| **Hot-desk** | One user at a time with fast reattach — roaming between your own devices. A second user is told the box is busy, and each device+resolution keeps its own scaling. |
| **Workstation** | The multi-monitor daily driver. Your displays come back exactly where you arranged them, with per-client identity and an exclusive desktop. |
| **Default** | Good for most setups. Reconnects resume quickly, the streamed output becomes the whole desktop, and extra viewers each get their own screen. |
| **Headless box** | A machine with no monitor that you only ever stream from. The game and its display survive disconnects indefinitely (keep-alive **forever**), and whoever connects next takes the box over. Release it from the console when you're done. |
| **Shared desktop** | A PC you also use in person. punktfunk never blanks your real monitors and never leaves a leftover display behind; extra viewers each get their own screen. |
| **Hot-desk** | One person at a time — roam between your own devices with an instant reconnect. Anyone else is told the box is busy, and each device+resolution keeps its own scaling. |
| **Workstation** | Your multi-monitor daily driver. Displays come back exactly where you arranged them, each client keeps its own settings, and the desktop is yours alone. |
## Save your own preset
@@ -69,7 +69,7 @@ this also keeps the **game itself running** so you can reconnect straight back i
- **A duration** (seconds) — keep it for that long; a reconnect inside the window drops you straight
back in, with no re-negotiation and no desktop reshuffle.
- **Forever** — keep it until you stop the host or **release it** from the console (Host → *Virtual
displays* → *Release*). This is the gaming-rig model.
displays* → *Release*). This is the headless-box model.
Default: **10 seconds**. Windows has always lingered 10 s; the Linux backends previously tore down
immediately — a short linger makes reconnects smoother on both.
@@ -194,6 +194,6 @@ landing on a dead stream — and switching between game mode and the KDE / GNOME
follows the switch. If a launched game **exits**, a dedicated session ends and returns you to your
library; a game mode / desktop session keeps streaming.
**My couch box's TV stayed on the streamed session after I disconnected.** With the **gaming-rig**
**My couch box's TV stayed on the streamed session after I disconnected.** With the **Headless box**
preset (keep alive = *forever*), a managed Steam session is held indefinitely so a reconnect resumes
instantly — return to game mode on the box (or restart the host) to hand the TV back.
+4 -3
View File
@@ -5,7 +5,8 @@ description: Enable the punktfunk browser console, read or change its login pass
The web console is the browser UI for a punktfunk host — live status, paired devices, and the PIN
pairing flow. It ships as the **`punktfunk-web`** systemd user unit on Linux and the **`PunktfunkWeb`**
task on Windows, and serves on **`http://<host-ip>:47992`**. It's the surface you expose on the LAN to
task on Windows, and serves on **`https://<host-ip>:47992`** (HTTPS with the host's own self-signed
identity cert — your browser warns once; trust it and continue). It's the surface you expose on the LAN to
administer the host; the host's own management API (47990) keeps every admin action loopback-only and
off-loopback serves only read-only status + game-library browsing to paired clients.
@@ -19,11 +20,11 @@ off-loopback serves only read-only status + game-library browsing to paired clie
```sh
systemctl --user enable --now punktfunk-web
# then browse to http://<host-ip>:47992
# then browse to https://<host-ip>:47992
```
- **Windows host:** the installer sets up the console, its runtime, and the `PunktfunkWeb` task and
starts it at boot. There is nothing to enable — open `http://<this-PC>:47992`.
starts it at boot. There is nothing to enable — open `https://<this-PC>:47992`.
- **SteamOS host:** the install script builds and starts the console as a user service for you. It
prints the URL when it finishes.
+1 -1
View File
@@ -64,7 +64,7 @@ the Windows specifics follow.
The installer also sets up the **web management console** (status, paired devices, the PIN pairing
flow): it bundles the console plus its own runtime and runs it as the **`PunktfunkWeb`** task on
**`http://<this-PC>:47992`**, starting at boot.
**`https://<this-PC>:47992`**, starting at boot.
#### Console login password
+719
View File
@@ -0,0 +1,719 @@
# Embedding punktfunk-core (the C ABI)
This guide is for developers who want to build their **own** punktfunk client — on a platform we
don't ship an app for — by linking `punktfunk-core` through its stable C ABI. It covers what the
core does (and, importantly, what it doesn't), how to build and link it, the full client lifecycle,
and worked integration blueprints for **webOS**, **Xbox**, and **Tizen**.
The authoritative header is [`include/punktfunk_core.h`](../include/punktfunk_core.h) — it is
generated from Rust by cbindgen and every symbol carries a doc comment. This guide is the narrative;
the header is the contract.
---
## 1. What the core gives you — and what it doesn't
`punktfunk-core` is the *one* implementation of the wire format, shared by the host and every
first-party client. When you embed it you get the entire network side of a client for free:
**The core does:**
- The `punktfunk/1` handshake (Hello/Welcome/Start) over a **QUIC control plane**.
- The **UDP video data plane**: packetization, reassembly, pacing, socket tuning.
- **Forward error correction** (GF(2⁸) ReedSolomon and GF(2¹⁶) Leopard-RS) — loss recovery.
- **AES-128-GCM** session crypto, **cert pinning / TOFU**, and **SPAKE2 PIN pairing**.
- **Clock synchronization** (host↔client offset for glass-to-glass latency math).
- Delivery of every plane: reassembled **video access units**, **Opus audio** (raw or decoded to
PCM in-core), **rumble**, **DualSense HID output**, **HDR metadata**, **host timing**.
- The uplink: **input events**, **mic**, **rich input** (touchpad/motion).
- Loss-recovery signalling: keyframe requests and **reference-frame invalidation** (RFI).
**The core does NOT do (this is your job):**
- **Video decoding.** The core hands you encoded access units (H.264, HEVC, or AV1 NAL units /
OBUs). You feed them to the platform's hardware decoder.
- **Rendering / presentation.** You own the swapchain / surface and put decoded frames on glass.
- **Audio output.** You own the audio sink; the core only delivers Opus (or f32 PCM).
- **Input capture / controller enumeration.** You read the platform's input and translate it into
`PunktfunkInputEvent`s.
- **UI, discovery UX, settings.** (mDNS discovery is not part of the C ABI; see §4.)
Think of it as: **the core is the modem and the codec-agnostic protocol brain; you are the TV and
the remote.**
```
┌─────────────────────────── punktfunk-core (C ABI) ───────────────────────────┐
your platform │ │ punktfunk host
┌───────────┐ input │ punktfunk_connection_send_input / _send_mic / _send_rich_input ───────────────▶
│ remote / │──────────▶│ │
│ gamepad │ │ QUIC control + UDP data + FEC + AES-GCM │
└───────────┘ │ │
┌───────────┐ AUs │ ◀── punktfunk_connection_next_au (H.264 / HEVC / AV1 access units) │◀─── encoder
│ HW decoder│◀──────────│ ◀── punktfunk_connection_next_audio(_pcm)(Opus / f32 PCM) │
│ + display │ │ ◀── punktfunk_connection_next_rumble2 / _next_hidout / _next_hdr_meta │
└───────────┘ └──────────────────────────────────────────────────────────────────────────────┘
```
---
## 2. Building and linking the library
### 2.1 The `quic` feature is mandatory for clients
The entire connection API (`punktfunk_connect*`, `punktfunk_connection_*`, pairing, probe) is
gated, in both Rust and the C header, behind the **`quic`** Cargo feature. Everything under
`#if defined(PUNKTFUNK_FEATURE_QUIC)` in the header only exists when the core was built with it.
> **Two things must line up or nothing links:**
> 1. Build the library with `--features quic`.
> 2. Compile your C/C++ with `-DPUNKTFUNK_FEATURE_QUIC` so the header declares those prototypes.
The non-QUIC surface (`punktfunk_session_new`, `punktfunk_host_submit_frame`,
`punktfunk_client_poll_frame`, the loopback test pair) is the **raw transport** used by the host and
by tests. **Client embedders do not use it** — you use the `punktfunk_connect*` family.
### 2.2 Build outputs
```sh
# Native (host machine) — good for a desktop prototype
cargo build -p punktfunk-core --features quic --release
```
`crate-type = ["lib", "cdylib", "staticlib"]`, so one build produces all three:
| Output | File (per platform) | Use when |
|-------------|----------------------------------------------------------------|----------|
| `cdylib` | `libpunktfunk_core.so` / `.dylib` / `punktfunk_core.dll` | dynamic linking (most TV apps, sandboxed apps) |
| `staticlib` | `libpunktfunk_core.a` / `punktfunk_core.lib` | static embedding into a single binary |
The header lands at `include/punktfunk_core.h` (regenerated on build; also checked in).
### 2.3 Cross-compiling for your device
Every target below is a standard Rust target. Add it and point Cargo at the platform sysroot/linker.
```sh
rustup target add aarch64-unknown-linux-gnu # most ARM Smart TVs (webOS, Tizen)
rustup target add armv7-unknown-linux-gnueabihf # older 32-bit TV SoCs
rustup target add x86_64-pc-windows-msvc # Xbox (GDK consoles are x64)
# Tell Cargo which cross-linker + sysroot to use (example: aarch64 TV NDK)
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
export CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
cargo build -p punktfunk-core --features quic --release \
--target aarch64-unknown-linux-gnu
```
The QUIC tree is deliberately **ring-only** (no aws-lc-rs / no cmake), so cross builds do not need a
C crypto toolchain — this is what keeps the TV cross-compiles simple. `libopus` (for the in-core PCM
decode path) is the only C dependency and builds with the target `CC`.
### 2.4 Compiling and linking your app
```sh
# compile
cc -std=c11 -DPUNKTFUNK_FEATURE_QUIC -I path/to/include -c myclient.c
# link, dynamic
cc myclient.o -L target/aarch64-unknown-linux-gnu/release -lpunktfunk_core -o myclient
# link, static — you also need the native libs rustc pulls in
NATIVE=$(cargo rustc -p punktfunk-core --features quic --release \
--target aarch64-unknown-linux-gnu \
--crate-type staticlib -- --print native-static-libs 2>&1 \
| sed -n 's/.*native-static-libs: //p' | tail -1)
cc myclient.o target/aarch64-unknown-linux-gnu/release/libpunktfunk_core.a $NATIVE -o myclient
```
`--print native-static-libs` is the reliable way to discover the platform libs (`-lpthread`,
`-lm`, WinSock, etc.) a static link needs — the C harness at
[`crates/punktfunk-core/tests/c/run.sh`](../crates/punktfunk-core/tests/c/run.sh) does exactly this
and is your smallest end-to-end reference.
### 2.5 Version check first, always
```c
#include "punktfunk_core.h"
if (punktfunk_abi_version() != ABI_VERSION) {
// The .so you loaded is a different core than this header was generated from. Abort.
}
```
`ABI_VERSION` (the embeddable C surface) is distinct from `WIRE_VERSION` (what the handshake
carries). The ABI can grow — new `connect_ex` variants, new planes — without touching the wire, so a
newer client keeps talking to a deployed host. Only equal `WIRE_VERSION`s interoperate on the wire;
the core enforces that inside the handshake.
---
## 3. Core concepts
### 3.1 The connection handle
`PunktfunkConnection *` is an **opaque handle to one live client session** — QUIC control plane plus
UDP data plane, with all the I/O pumped on threads the core owns internally. You obtain it from a
`punktfunk_connect*` call and release it with `punktfunk_connection_close`.
### 3.2 The threading contract (read this before you design your loops)
The planes are **independent single-consumer queues**. The rule:
> Each plane (video, audio, rumble, HID-out, HDR, host-timing) may be pulled from **its own thread,
> at most one thread per plane**. Different planes may be pulled concurrently. Never pull the *same*
> plane from two threads.
A typical client runs **three threads**:
- **Video thread** — blocks on `punktfunk_connection_next_au`, decodes, presents.
- **Audio thread** — blocks on `punktfunk_connection_next_audio_pcm` (or `_next_audio`).
- **Feedback thread** — pulls rumble / HID-out / HDR-meta (all low-rate; poll or short-timeout).
Input is sent from whatever thread reads your input (send calls are non-blocking enqueues and are
safe to call from any thread).
### 3.3 Borrowed memory
`next_au`, `next_audio`, `next_audio_pcm` return a struct whose `data`/`samples` pointer **borrows
core-owned memory that is valid only until your next pull on that same handle/plane.** Decode or copy
before you call again. Cross-plane pulls do not invalidate each other (the audio pull won't free the
video buffer).
### 3.4 Status codes
Every fallible call returns `PunktfunkStatus`: `PUNKTFUNK_STATUS_OK == 0`, all errors negative
(`rc < 0`). The two you branch on constantly in the pull loops:
- `PUNKTFUNK_STATUS_NO_FRAME` (-5) — nothing ready within `timeout_ms`; loop again.
- `PUNKTFUNK_STATUS_CLOSED` (-10) — the session ended; tear down and leave the loop.
---
## 4. Identity, pairing, discovery, wake
### 4.1 Generate a persistent identity once
```c
char cert[4096], key[4096];
if (punktfunk_generate_identity(cert, sizeof cert, key, sizeof key) != PUNKTFUNK_STATUS_OK) { /* … */ }
// Persist BOTH strings securely (platform secure storage / keychain).
// The certificate's SHA-256 is how a host recognizes this client after pairing.
```
Do this **once per device/install** and store the PEM strings. Pass them to every `pair` and
`connect` call. Anonymous (`NULL`/`NULL`) sessions are rejected by hosts running `--require-pairing`.
### 4.2 Pair (PIN ceremony)
```c
uint8_t host_fp[32];
PunktfunkStatus rc = punktfunk_pair(host, port, cert, key,
user_typed_pin, "Living Room TV",
host_fp, /*timeout_ms=*/10000);
// rc == PUNKTFUNK_STATUS_CRYPTO => wrong PIN.
// rc == PUNKTFUNK_STATUS_OK => persist host_fp; it's the pin for future connects.
```
The host shows a PIN; the user types it into your UI. On success you get the host's verified
fingerprint — store it keyed by host, and pass it as `pin_sha256` on every later connect (that pins
the host and defeats MITM). For a first connect without prior pairing you may pass `NULL` for the pin
and capture `observed_sha256_out` (trust-on-first-use), then pin it thereafter.
### 4.3 Reachability probe and Wake-on-LAN
- `punktfunk_probe(host, port, timeout_ms)` — a bounded, trust-agnostic handshake attempt. Returns
`OK` if the host answered (drives "online" pips), `TIMEOUT` otherwise. Works over routed
networks (Tailscale/VPN) where mDNS never reaches. Call off the UI thread.
- `punktfunk_wake_on_lan(macs, mac_count, last_known_ip)` — sends WoL magic packets. The host's wake
MAC(s) arrive out-of-band via the mDNS `mac` TXT record, so no connection is needed to wake a
sleeping host.
> **Discovery is out of scope for the C ABI.** mDNS/DNS-SD browsing is the embedder's job (use the
> platform's Bonjour/Avahi/`nsd` API to find `_punktfunk._udp` hosts, or let the user type an
> IP/hostname). The core only *connects*.
---
## 5. Connecting
There is a **ladder** of `connect` variants; each adds parameters and defaults the rest to the prior
variant's behavior. Use the highest one you need — they all return `PunktfunkConnection *` (NULL on
failure) and block up to `timeout_ms` for the handshake, so **call off your UI thread.**
| Variant | Adds |
|------------------|------|
| `punktfunk_connect` | base: `width`, `height`, `refresh_hz`, pin, identity |
| `punktfunk_connect_ex` | `compositor` preference (Linux hosts) |
| `punktfunk_connect_ex2` | `gamepad` backend (X-Box 360 / DualSense / …) |
| `punktfunk_connect_ex3` | `bitrate_kbps` |
| `punktfunk_connect_ex4` | `launch_id` (auto-launch a library title) |
| `punktfunk_connect_ex5` | `video_caps` (10-bit / HDR / 4:4:4 / host-timing) |
| `punktfunk_connect_ex6` | `audio_channels` (2 / 6 / 8) |
| **`punktfunk_connect_ex7`** | **`video_codecs` + `preferred_codec`** — the recommended full call |
```c
uint8_t caps = 0; // add PUNKTFUNK_VIDEO_CAP_10BIT | _HDR | _444 as supported
uint8_t codecs = PUNKTFUNK_CODEC_HEVC; // OR-in _H264 (needed for software hosts) / _AV1
uint8_t host_fp[32]; uint8_t observed[32];
PunktfunkConnection *c = punktfunk_connect_ex7(
"192.168.1.50", 9777,
1920, 1080, 60, // mode
PUNKTFUNK_COMPOSITOR_AUTO,
PUNKTFUNK_GAMEPAD_XBOX360,
/*bitrate_kbps=*/20000,
caps,
/*audio_channels=*/2,
codecs, /*preferred_codec=*/PUNKTFUNK_CODEC_HEVC,
/*launch_id=*/NULL,
host_fp, // pin (or NULL for TOFU)
observed, // filled with the observed fp
cert, key, // identity
/*timeout_ms=*/8000);
if (!c) { /* handshake failed */ }
```
> **Advertise only what you can actually decode and present.** The host upgrades to 10-bit / HDR /
> 4:4:4 / AV1 **only** when you set the matching bit *and* it opted in. Setting a cap you can't honor
> gives you a stream you can't render.
### 5.1 Read the resolved session (right after connect)
The host echoes what it actually chose. Build your decoder and present path from **these**, never
from your request:
```c
uint8_t codec, prim, trc, mtx, full, depth, chroma, ch;
punktfunk_connection_codec(c, &codec); // PUNKTFUNK_CODEC_{H264,HEVC,AV1}
punktfunk_connection_color_info(c, &prim, &trc, &mtx, &full, &depth); // CICP + bit depth
punktfunk_connection_chroma_format(c, &chroma); // 1 = 4:2:0, 3 = 4:4:4
punktfunk_connection_audio_channels(c, &ch); // 2 / 6 / 8
int64_t clk = 0; punktfunk_connection_clock_offset_ns(c, &clk); // hostclient ns, for latency math
uint32_t w, h, hz; punktfunk_connection_mode(c, &w, &h, &hz);
```
`trc == 16` (PQ) or `18` (HLG) means an **HDR** session — set up an HDR present path and drain
`punktfunk_connection_next_hdr_meta`. On `PUNKTFUNK_COMPOSITOR_GAMESCOPE`
(`punktfunk_connection_compositor`) draw a client-side cursor by default (that capture carries none).
---
## 6. The video loop
The core delivers **complete, in-order access units** with in-band parameter sets. The first AU is
an IDR — build your decoder from it (and from the resolved codec/color above). The stream is then an
**infinite-GOP** of P-frames: no periodic IDRs, so loss recovery is explicit and is the part you must
get right.
```c
PunktfunkFrame f;
for (;;) {
PunktfunkStatus rc = punktfunk_connection_next_au(c, &f, /*timeout_ms=*/20);
if (rc == PUNKTFUNK_STATUS_NO_FRAME) continue;
if (rc == PUNKTFUNK_STATUS_CLOSED) break;
if (rc != PUNKTFUNK_STATUS_OK) continue;
// (1) Loss recovery — let the core do the hard part. Call this EVERY frame:
bool gap = false;
punktfunk_connection_note_frame_index(c, f.frame_index, &gap);
// On a forward gap it fires a throttled RFI request for you (clean P-frame recovery on
// AMD-LTR/NVENC hosts, no IDR spike). `gap == true` is your cue to freeze on the last good
// picture until re-anchor (see (3)).
// (2) Clean re-anchor points (infinite-GOP has no periodic keyframes):
bool recovery_point = f.flags & USER_FLAG_RECOVERY_POINT; // intra-refresh wave boundary
bool recovery_anchor = f.flags & USER_FLAG_RECOVERY_ANCHOR; // definitive LTR/RFI re-anchor
// (3) Decode + present. f.data/f.len are valid until the next next_au call.
decode_and_present(f.data, f.len, f.pts_ns);
// If you implement freeze-until-reanchor: while frozen, keep redrawing the last good frame
// and lift the freeze on a real keyframe, on the FIRST recovery_anchor, or the SECOND
// recovery_point after the gap.
// (4) Backstop trigger: poll the reassembler's unrecoverable-drop count. Under infinite GOP,
// unrecoverable loss yields reference-missing deltas the decoder *silently conceals*
// (frozen/garbage, no decode error) — so this counter, not a decode error, is the signal.
uint64_t dropped = 0; punktfunk_connection_frames_dropped(c, &dropped);
if (dropped > last_dropped) { // THROTTLE this (≤ ~1/100ms) — see below
punktfunk_connection_request_keyframe(c);
last_dropped = dropped;
}
}
```
**Recovery, in priority order:**
1. `note_frame_index` every frame — the cheapest, earliest signal. It issues a **throttled RFI**
request (`request_rfi`) for the exact lost range so RFI-capable hosts recover with a clean
P-frame (no 2040× IDR bandwidth spike).
2. `frames_dropped` climbing → `request_keyframe` as the backstop when RFI can't help or the recovery
frame itself was lost.
3. A wedged decoder (received AUs but no decoded output for several frames) → `request_keyframe`.
**Throttle** both `request_keyframe` and `request_rfi` — decode stays wedged for several frames until
recovery lands, so one request per ~100 ms is right; requesting per frame floods the control stream.
If you prefer not to hand-roll this, `note_frame_index` + a throttled `frames_dropped`→keyframe check
is a complete, correct recovery policy on its own.
### 6.1 Mid-session resolution / refresh changes
```c
punktfunk_connection_request_mode(c, new_w, new_h, new_hz);
```
Non-blocking. On acceptance the next AU is a fresh IDR with in-band parameter sets — **rebuild your
decoder from it** — and `punktfunk_connection_mode` reflects the switch. Use this when your window
resizes or the display refresh changes.
---
## 7. Audio
Two mutually-exclusive ways to consume audio — pick one, on one dedicated thread:
- **`punktfunk_connection_next_audio_pcm`** — the core decodes to interleaved **f32 PCM** at 48 kHz
in channel order `FL FR FC LFE RL RR SL SR`. Use this if you lack a *multistream*-capable Opus
decoder (e.g. Apple's AudioToolbox is stereo-only; many TV audio stacks are too). Simplest path.
- **`punktfunk_connection_next_audio`** — raw Opus packets (5 ms frames). Use only if you have a
multistream Opus decoder and build it from `punktfunk_connection_audio_channels` (see
`audio::layout_for`). Do **not** mix the two on one connection.
```c
PunktfunkAudioPcm a;
for (;;) {
PunktfunkStatus rc = punktfunk_connection_next_audio_pcm(c, &a, /*timeout_ms=*/100);
if (rc == PUNKTFUNK_STATUS_CLOSED) break;
if (rc != PUNKTFUNK_STATUS_OK) continue;
// a.samples = a.frame_count * a.channels f32s, valid until the next PCM call.
audio_sink_write(a.samples, a.frame_count, a.channels);
}
```
---
## 8. Input (uplink)
Fill a `PunktfunkInputEvent` and send it — non-blocking, from any thread:
```c
PunktfunkInputEvent ev; memset(&ev, 0, sizeof ev);
```
Field meaning depends on `kind` (`PunktfunkInputKind`). The contracts that trip people up:
- **Absolute pointer / touch** (`MOUSE_MOVE_ABS`, `TOUCH_DOWN/MOVE`): `x`/`y` are pixel coordinates
and **`flags` must pack your coordinate space as `(width << 16) | height`** — the host normalizes
against it. A **zero `flags` is dropped**, so always set it.
- **Gamepad button** (`GAMEPAD_BUTTON`): `code` = a `PUNKTFUNK_BTN_*` bit, `x != 0` = pressed,
`flags` = pad index (0..15).
- **Gamepad axis** (`GAMEPAD_AXIS`): `code` = `PUNKTFUNK_AXIS_*`, `flags` = pad index. Sticks are
**i16 (32768..32767), +y = up** (opposite of screen coordinates); triggers are 0..255.
- **Relative mouse** (`MOUSE_MOVE`): `x`/`y` carry `dx`/`dy`. **Scroll** (`MOUSE_SCROLL`): `x` is the
signed delta.
```c
// Example: press A on pad 0
ev.kind = PUNKTFUNK_INPUT_KIND_GAMEPAD_BUTTON;
ev.code = PUNKTFUNK_BTN_A; ev.x = 1; ev.flags = 0;
punktfunk_connection_send_input(c, &ev);
// Example: absolute touch at (640,360) on a 1280x720 surface
ev.kind = PUNKTFUNK_INPUT_KIND_TOUCH_DOWN;
ev.code = 0 /*finger id*/; ev.x = 640; ev.y = 360;
ev.flags = (1280u << 16) | 720u;
punktfunk_connection_send_input(c, &ev);
```
> Gamepad state on the C ABI uses **per-transition** button/axis events (above). The wire also has an
> idempotent full-pad *snapshot* mode (`HOST_CAP_GAMEPAD_STATE`), but it is not exposed as a
> dedicated C entry point — per-transition events are accepted by every host and are the C path.
**Other uplinks (all non-blocking):**
- `punktfunk_connection_send_mic(c, opus, len, seq, pts_ns)` — Opus mic frames (you encode).
- `punktfunk_connection_send_rich_input(c, &rich)` — DualSense touchpad contact / motion sample.
- `punktfunk_connection_send_rich_input2(c, &richEx)` — the forward-compatible superset (Steam
trackpads, signed coords, pressure); set `struct_size = sizeof(PunktfunkRichInputEx)`.
---
## 9. Feedback planes (rumble, HID, HDR)
Pull these on your feedback thread (or poll with `timeout_ms = 0`). Same
`NO_FRAME`/`CLOSED` semantics as everywhere.
- **Rumble**`punktfunk_connection_next_rumble2(c, &pad, &low, &high, &ttl_ms, timeout)`.
Amplitudes 0..0xFFFF; `(0,0)` = stop. `ttl_ms` is a host-supplied self-terminating lease — render
the level for that long unless renewed; `PUNKTFUNK_RUMBLE_NO_TTL` means fall back to your own
staleness timeout. (The v1 `_next_rumble` drops the TTL — prefer v2.)
- **DualSense HID output**`punktfunk_connection_next_hidout(c, &out, timeout)`. `out.kind` selects
lightbar RGB / player LEDs / adaptive-trigger effect / trackpad haptic. Replay on a real DualSense
via the platform's controller API. Only a DualSense-backend session emits these.
- **HDR metadata**`punktfunk_connection_next_hdr_meta(c, &meta, timeout)`. ST.2086 mastering
display + content light level, in HDR10 SEI fixed-point units — ready to hand to DXGI
`DXGI_HDR_METADATA_HDR10`, Apple `CAEDRMetadata`, or Android `KEY_HDR_STATIC_INFO`. Only an HDR
session emits these; apply the latest to your display.
- **Host timing** (optional, if you advertised `VIDEO_CAP_HOST_TIMING`) —
`punktfunk_connection_next_host_timing` gives the host's capture→sent µs per AU, so your stats HUD
can split `host` vs `network` latency.
---
## 10. Speed test and stats
```c
punktfunk_connection_speed_test(c, /*target_kbps=*/50000, /*duration_ms=*/2000); // pauses video briefly
PunktfunkProbeResult r;
do { punktfunk_connection_probe_result(c, &r); } while (!r.done); // poll
// r.throughput_kbps drives a bitrate choice; r.loss_pct vs r.host_drop_pct
// distinguishes a lossy link from a host that can't keep up.
```
---
## 11. Teardown
```c
punktfunk_connection_disconnect_quit(c); // user pressed "stop": host tears down now, no linger
punktfunk_connection_close(c); // joins internal threads, frees the handle
```
Call `disconnect_quit` **only** on a deliberate user quit — it makes the host drop the virtual
display immediately. On a network drop / backgrounding, skip it (a plain `close`) so the host lingers
and a reconnect can resume. After `close`, the handle is dead; stop all your plane threads first.
---
# Platform integration blueprints
Everything above is exact — it's the punktfunk side, which is identical on every platform. The
sections below are **blueprints**: they name the real decode/present/input subsystems on each target
and show where the punktfunk glue plugs in. Treat the platform-SDK calls as illustrative — confirm
signatures against that platform's current SDK. The pattern is always the same:
> **connect → build HW decoder from the resolved codec/color → three threads (video decode+present,
> audio, feedback) → translate native input into `PunktfunkInputEvent`s.**
---
## 12. webOS (LG Smart TV)
**App model.** webOS ships both web apps and **native** apps via the **webOS NDK** (Linux, ARM —
usually `aarch64`, some older panels `armv7`). A low-latency streaming client wants the native path:
you get real threads, a hardware video pipeline, and SDL for window/input. (This mirrors how the
community Moonlight webOS client is built.)
**Cross-compile the core.** Build `libpunktfunk_core.so` with the webOS NDK sysroot:
```sh
rustup target add aarch64-unknown-linux-gnu
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=$WEBOS_NDK/.../aarch64-webos-linux-gnu-gcc
export CC_aarch64_unknown_linux_gnu=$WEBOS_NDK/.../aarch64-webos-linux-gnu-gcc
cargo build -p punktfunk-core --features quic --release --target aarch64-unknown-linux-gnu
```
Bundle the `.so` in your `.ipk` alongside the header-compiled client.
**Video decode + present.** Use the webOS media pipeline for the SoC's hardware decoder — historically
**NDL-DirectMedia** and on newer panels the **Starfish** media pipeline
(`com.webos.service.mediapipeline`). You push the punktfunk AUs (Annex-B H.264/HEVC/AV1) into the
pipeline's `feed`/`push` entry point; it decodes and composits to the TV's video plane, and you draw
your overlay/UI on the graphics plane over it. Present cadence is driven by the pipeline, not you.
**Audio.** Simplest is `punktfunk_connection_next_audio_pcm`**PulseAudio** (webOS's audio server)
via a simple playback stream. Or route the raw Opus into the media pipeline if it accepts a
secondary audio ES.
**Input.** The Magic Remote / standard remote and Bluetooth gamepads surface as **SDL2** events (LG
ships SDL for NDK apps). Map remote keys to `KEY_DOWN/UP`, the pointer to `MOUSE_MOVE_ABS` (set
`flags = (w<<16)|h`), and `SDL_GameController` axes/buttons to `GAMEPAD_AXIS`/`GAMEPAD_BUTTON`.
**Skeleton:**
```c
// 1. connect (HEVC, stereo, SDR to start)
PunktfunkConnection *c = punktfunk_connect_ex7(host, 9777, 1920,1080,60,
PUNKTFUNK_COMPOSITOR_AUTO, PUNKTFUNK_GAMEPAD_XBOX360, 20000,
/*caps=*/0, /*ch=*/2, PUNKTFUNK_CODEC_HEVC, PUNKTFUNK_CODEC_HEVC,
NULL, host_fp, observed, cert, key, 8000);
uint8_t codec; punktfunk_connection_codec(c, &codec);
StarfishPipeline *p = starfish_open(codec /*→ "video/hevc" etc*/, 1920,1080);
// 2. video thread
PunktfunkFrame f;
while (running) {
if (punktfunk_connection_next_au(c, &f, 20) != PUNKTFUNK_STATUS_OK) continue;
bool gap; punktfunk_connection_note_frame_index(c, f.frame_index, &gap);
starfish_feed(p, f.data, f.len, f.pts_ns); // HW decode + present
uint64_t d; punktfunk_connection_frames_dropped(c, &d);
if (d > last_d) { punktfunk_connection_request_keyframe(c); last_d = d; } // throttle
}
// 3. audio thread → next_audio_pcm → pulse_write(...)
// 4. SDL input thread → fill PunktfunkInputEvent → punktfunk_connection_send_input(c, &ev)
```
**Gotchas.** webOS app networking is permitted for UDP/QUIC, but background/suspend policy is
aggressive — pull-loop `CLOSED` on backgrounding and reconnect on resume. Video-plane / graphics-plane
z-order and scaling are set through the pipeline's display-window API, not by drawing pixels yourself.
---
## 13. Xbox (GDK — Series X|S / One)
**App model.** Use the **Microsoft GDK** (Game Development Kit) for consoles — a **C++** title with
**Direct3D 12** and the **GameInput** API. Consoles are **x64**, so the core target is
`x86_64-pc-windows-msvc`. (Requires an ID@Xbox / partner console in Developer Mode; UWP on retail is
possible but GDK is the low-latency path.)
**Build the core.** From an MSVC environment:
```sh
rustup target add x86_64-pc-windows-msvc
cargo build -p punktfunk-core --features quic --release --target x86_64-pc-windows-msvc
```
You get `punktfunk_core.dll` (+ import lib) and `punktfunk_core.lib` (static). Compile your C++ with
`/DPUNKTFUNK_FEATURE_QUIC` and add `include/` to the include path. The header is `extern "C"`-clean
for C++ (`__cplusplus` guards are in place).
**Video decode + present.** Feed the AUs to **Media Foundation** with a **DXVA2 / D3D12
hardware-accelerated decoder** (`IMFTransform` H.264/HEVC/AV1 decoder), or a D3D12 Video decode
(`ID3D12VideoDecoder`) if you want to own the DPB. Output NV12/P010 textures and present with your
D3D12 swapchain. For HDR, set the swapchain to `DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020` and pass
`punktfunk_connection_next_hdr_meta` straight into `IDXGISwapChain4::SetHDRMetaData` — the core's
`PunktfunkHdrMeta` is already in `DXGI_HDR_METADATA_HDR10` units.
**Audio.** `punktfunk_connection_next_audio_pcm` (f32) → **XAudio2** source voice, or **WASAPI**
shared-mode render. Request 6/8 channels at connect for surround.
**Input.** **GameInput** (`IGameInput::GetCurrentReading`) gives you gamepad state; diff it per frame
and emit `GAMEPAD_BUTTON`/`GAMEPAD_AXIS` events. Because a real Xbox pad drives this, connect with
`PUNKTFUNK_GAMEPAD_XBOXONE` for matching glyphs. Rumble comes **back** from the host — feed
`punktfunk_connection_next_rumble2` into `IGameInputDevice::SetRumbleState` (map `low`
low-frequency, `high`→high-frequency motors).
**Skeleton (C++):**
```cpp
auto* c = punktfunk_connect_ex7(host, 9777, 3840,2160,60,
PUNKTFUNK_COMPOSITOR_AUTO, PUNKTFUNK_GAMEPAD_XBOXONE, /*bitrate=*/60000,
PUNKTFUNK_VIDEO_CAP_10BIT | PUNKTFUNK_VIDEO_CAP_HDR, // 4K HDR
/*ch=*/6, PUNKTFUNK_CODEC_HEVC, PUNKTFUNK_CODEC_HEVC,
nullptr, hostFp, observed, cert, key, 8000);
uint8_t trc; punktfunk_connection_color_info(c, nullptr,&trc,nullptr,nullptr,nullptr);
bool hdr = (trc == 16 || trc == 18);
auto decoder = MakeMFHevcDecoder(d3d12Device, hdr /*P010*/);
// video thread
PunktfunkFrame f;
while (running) {
if (punktfunk_connection_next_au(c, &f, 20) != PUNKTFUNK_STATUS_OK) continue;
bool gap; punktfunk_connection_note_frame_index(c, f.frame_index, &gap);
decoder.Decode(f.data, f.len, f.pts_ns); // → NV12/P010 texture → D3D12 present
}
// feedback thread
uint16_t pad, lo, hi; uint32_t ttl;
while (punktfunk_connection_next_rumble2(c,&pad,&lo,&hi,&ttl, 100) == PUNKTFUNK_STATUS_OK)
SetRumble(pad, lo, hi, ttl);
// HDR: PunktfunkHdrMeta hm; next_hdr_meta(...) → swapChain4->SetHDRMetaData(HDR10, &hm-as-DXGI)
```
**Gotchas.** GDK console socket use is allowed but goes through the title's network stack — enable it
in your MicrosoftGame.config and test in the console sandbox. Retail devices need proper cert; keep
the DLL and header ABI versions locked (§2.5) in your CI.
---
## 14. Tizen (Samsung Smart TV)
**App model.** Use **Tizen Native** (C, EFL) — .NET/web apps can't get low-latency HW-decode feed
access cleanly. TVs are ARM (`aarch64` on modern panels, `armv7` on older). Build with the Tizen
Studio / GBS toolchain and its sysroot.
**Cross-compile the core.**
```sh
rustup target add aarch64-unknown-linux-gnu # or armv7-unknown-linux-gnueabihf
export CC_aarch64_unknown_linux_gnu=$TIZEN_TOOLCHAIN/bin/aarch64-tizen-linux-gnu-gcc
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=$CC_aarch64_unknown_linux_gnu
cargo build -p punktfunk-core --features quic --release --target aarch64-unknown-linux-gnu
```
Add the `.so` to your project's `lib/` and link with `-lpunktfunk_core`, compiling with
`-DPUNKTFUNK_FEATURE_QUIC`.
**Video decode + present.** Use **`capi-media-codec`** (`mediacodec_*`) — Tizen's hardware codec
binding over OpenMAX. Configure it from the resolved codec (`MEDIACODEC_H264`/`_HEVC`/AV1), wrap each
AU in a **`media_packet`** (`capi-media-tool`) and `mediacodec_process_input`; on the output callback
render the decoded packet to an **Evas GL** surface (or hand tunpacked frames to the TV video plane).
For HDR, drive the panel's HDR mode from `punktfunk_connection_color_info` + `next_hdr_meta`.
**Audio.** `punktfunk_connection_next_audio_pcm` (f32, 48 kHz) → **`capi-media-audio-io`**
(`audio_out_*`). Convert f32→s16 if you open the sink as PCM S16.
**Input.** TV remote and Bluetooth gamepads arrive as **Ecore** key events (`Ecore_Event_Key`).
Register key grabs (`elm_win_keygrab_set`) for the remote, translate D-pad/OK/back to
`KEY_DOWN/UP` or gamepad buttons, and map any HID gamepad to `GAMEPAD_AXIS`/`GAMEPAD_BUTTON`.
**Skeleton:**
```c
PunktfunkConnection *c = punktfunk_connect_ex7(host, 9777, 1920,1080,60,
PUNKTFUNK_COMPOSITOR_AUTO, PUNKTFUNK_GAMEPAD_XBOX360, 20000,
/*caps=*/0, /*ch=*/2, PUNKTFUNK_CODEC_HEVC | PUNKTFUNK_CODEC_H264,
PUNKTFUNK_CODEC_HEVC, NULL, host_fp, observed, cert, key, 8000);
uint8_t codec; punktfunk_connection_codec(c, &codec);
mediacodec_h mc; mediacodec_create(&mc);
mediacodec_set_codec(mc, codec==PUNKTFUNK_CODEC_HEVC?MEDIACODEC_HEVC:MEDIACODEC_H264,
MEDIACODEC_DECODER | MEDIACODEC_SUPPORT_TYPE_HW);
mediacodec_set_output_buffer_available_cb(mc, on_decoded /*→ Evas GL present*/, NULL);
mediacodec_prepare(mc);
// video thread
PunktfunkFrame f;
while (running) {
if (punktfunk_connection_next_au(c, &f, 20) != PUNKTFUNK_STATUS_OK) continue;
bool gap; punktfunk_connection_note_frame_index(c, f.frame_index, &gap);
media_packet_h pkt = wrap_au(f.data, f.len, f.pts_ns); // capi-media-tool
mediacodec_process_input(mc, pkt, 0);
uint64_t d; punktfunk_connection_frames_dropped(c, &d);
if (d > last_d) { punktfunk_connection_request_keyframe(c); last_d = d; } // throttle
}
// audio thread: next_audio_pcm → audio_out_write(...)
// Ecore key handler: fill PunktfunkInputEvent → punktfunk_connection_send_input(c, &ev)
```
**Gotchas.** Declare the `internet` and `network.get` privileges in `tizen-manifest.xml` or the QUIC
socket won't open. `mediacodec` prefers Annex-B start codes and periodic parameter sets — the
punktfunk IDR carries in-band parameter sets, but if your SoC decoder wants an explicit
codec-config/CSD, extract VPS/SPS/PPS from the first IDR and feed it before the stream. On a
mid-session mode change (`request_mode`), tear down and re-prepare the `mediacodec` from the new IDR.
---
## 15. Checklist for a new port
- [ ] Build the core with `--features quic`; compile your app with `-DPUNKTFUNK_FEATURE_QUIC`.
- [ ] `punktfunk_abi_version() == ABI_VERSION` at startup.
- [ ] Persist a generated identity; implement PIN pairing; pin the host fingerprint.
- [ ] Connect off the UI thread; **build the decoder from the *resolved* codec/color/chroma**, not
your request.
- [ ] One thread per plane; never two on the same plane; decode/copy borrowed buffers before the next
pull.
- [ ] `note_frame_index` every video frame; throttled `frames_dropped``request_keyframe` backstop.
- [ ] Rebuild the decoder on the IDR after any accepted `request_mode`.
- [ ] Set `flags = (w<<16)|h` on absolute pointer/touch events (nonzero!).
- [ ] `disconnect_quit` only on deliberate user quit; always `close` and stop plane threads on
teardown.
## 16. Reference
- **Header (the contract):** [`include/punktfunk_core.h`](../include/punktfunk_core.h)
- **Minimal C link + round-trip proof:** [`crates/punktfunk-core/tests/c/`](../crates/punktfunk-core/tests/c/)
- **Core crate README:** [`crates/punktfunk-core/README.md`](../crates/punktfunk-core/README.md)
- **A full reference client (Rust, same ABI surface):** `crates/pf-client-core/src/session.rs`
+412 -1
View File
@@ -25,7 +25,11 @@
// TTL of a v2 envelope; `punktfunk_connection_next_rumble` is unchanged and drops it). Additive —
// the wire is backward-compatible (the envelope is a length-tolerant tail on 0xCA), so
// [`WIRE_VERSION`] is unchanged.
#define ABI_VERSION 5
// v6: added the shared-clipboard client surface — `punktfunk_connection_host_caps` and
// `punktfunk_connection_clipboard_{control,offer,fetch,serve,cancel}` +
// `punktfunk_connection_next_clipboard`. Additive; the wire grows only backward-compatible control
// messages (0x40-0x44) and a new `Welcome::host_caps` bit, so [`WIRE_VERSION`] is unchanged.
#define ABI_VERSION 6
// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
@@ -159,11 +163,60 @@
// Codec bit: AV1. (Mirrors `quic::CODEC_AV1`.)
#define PUNKTFUNK_CODEC_AV1 4
// Host-capability bit in [`punktfunk_connection_host_caps`]: the host applies gamepad-state
// snapshots (a capable client sends full-state snapshots instead of per-transition events).
// (Mirrors `quic::HOST_CAP_GAMEPAD_STATE`.)
#define PUNKTFUNK_HOST_CAP_GAMEPAD_STATE 1
// Host-capability bit in [`punktfunk_connection_host_caps`]: the host supports the shared
// clipboard, so a client may offer the toggle. (Mirrors `quic::HOST_CAP_CLIPBOARD`.)
#define PUNKTFUNK_HOST_CAP_CLIPBOARD 2
// `*ttl_ms` sentinel written by [`punktfunk_connection_next_rumble2`] for a legacy (v1) rumble
// datagram — an old host that sent no self-termination lease. The client then falls back to its
// own staleness heuristic for that update instead of a host-supplied deadline.
#define PUNKTFUNK_RUMBLE_NO_TTL 4294967295
// [`PunktfunkClipEvent::kind`] — the host announced clipboard content is available
// (`transfer_id` = the offer `seq`; `data`/`len` = a `\n`-separated `"<mime>\t<size_hint>"`
// format list). Fetch it lazily (only on a local paste) via
// [`punktfunk_connection_clipboard_fetch`].
#define PUNKTFUNK_CLIP_REMOTE_OFFER 1
// [`PunktfunkClipEvent::kind`] — host ack / policy / backend update (`enabled`/`policy`/`reason`
// valid). Reflect it in the toggle UI.
#define PUNKTFUNK_CLIP_STATE 2
// [`PunktfunkClipEvent::kind`] — the host is pasting our offered data: answer with
// [`punktfunk_connection_clipboard_serve`] (`transfer_id` = `req_id`; `seq`/`file_index` valid;
// `data`/`len` = the requested MIME).
#define PUNKTFUNK_CLIP_FETCH_REQUEST 3
// [`PunktfunkClipEvent::kind`] — bytes for a fetch we started (`transfer_id` = `xfer_id`;
// `data`/`len` = the payload, borrowed until the next `next_clipboard`; `last` = final chunk).
#define PUNKTFUNK_CLIP_DATA 4
// [`PunktfunkClipEvent::kind`] — a transfer was cancelled (`transfer_id` = the id).
#define PUNKTFUNK_CLIP_CANCELLED 5
// [`PunktfunkClipEvent::kind`] — a transfer failed (`transfer_id` = the id; `status` = a
// `PunktfunkStatus` code).
#define PUNKTFUNK_CLIP_ERROR 6
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Per-fetch requester-side size cap (bytes). A holder that streams more than this is treated as a
// cap breach and the fetch fails rather than buffering unboundedly (§7). Phase 0 uses one fixed
// value; a future host-policy `PUNKTFUNK_CLIP_MAX_MB` tightens it per session.
#define CLIP_FETCH_CAP (64 << 20)
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Inbound-serve `req_id`s carry this high bit so they never collide with the client-assigned
// outbound-fetch `xfer_id`s (which count up from 1). A single [`ClipCommand::Cancel`] `id` can
// then be routed to the right table.
#define INBOUND_REQ_FLAG 2147483648
#endif
// 16-byte AEAD authentication tag appended by GCM.
#define TAG_LEN 16
@@ -255,6 +308,34 @@
// feeding them to the decoder. Punktfunk/1 only (GameStream never sets it).
#define FLAG_PROBE 8
// Application `user_flags` bit (the u32 [`PacketHeader::user_flags`] word, surfaced to the client
// as [`crate::session::Frame::flags`]) — NOT a transport packet flag. Marks the access unit that
// **completes an intra-refresh wave**: the picture is loss-free from here even though the frame is
// a coded `P` (no IDR, so the decoder never sets `AV_FRAME_FLAG_KEY`). The client lifts its
// post-loss display freeze on this bit as well as on a real keyframe — the only bitstream-invisible
// clean point it can honor without forcing a full IDR. Lives above the low nibble because the host
// reuses `FLAG_PIC`/`FLAG_SOF`/`FLAG_PROBE` bit values inside `user_flags`; `0x10` clears all four.
#define USER_FLAG_RECOVERY_POINT 16
// Application `user_flags` bit — a **definitive single-frame clean re-anchor**. Unlike
// [`USER_FLAG_RECOVERY_POINT`] (an intra-refresh wave boundary, where the first boundary after a loss
// is only half-healed so the client waits for the second), this marks an access unit the host coded
// to reference a **known-good** picture on purpose — an AMD **LTR reference-frame-invalidation**
// recovery frame (`ForceLTRReferenceBitfield`): a clean P-frame off a long-term reference the client
// already has, not an IDR. The picture is loss-free the instant this AU decodes, so the client lifts
// its post-loss freeze on the **first** such mark. Coded `P` (no IDR), so the decoder never sets
// `AV_FRAME_FLAG_KEY` — this host flag is the only signal.
#define USER_FLAG_RECOVERY_ANCHOR 32
// Widest lost-frame range (frames, wrapping `last - first`) a reference-frame-invalidation
// recovery may be asked to repair; anything wider goes straight to the keyframe path on BOTH
// ends. RFI can only re-reference history the encoder still holds — NVENC keeps a 5-frame DPB,
// AMD LTR ~1 s of marks — and a genuine loss this wide (>1 s even at 240 fps) has no valid
// reference anywhere, so an RFI request for it is either hopeless or (worse) a phantom range
// from a desynced counter. Shared by the host's RFI dispatch (range → keyframe fallback) and the
// client-side gap detectors (huge gap → resync + keyframe request, no RFI).
#define RFI_MAX_RANGE 256
// Largest UDP datagram the core will send or accept. `Config::validate` bounds
// `shard_payload` so `HEADER_LEN + shard_payload + CRYPTO_OVERHEAD ≤ MAX_DATAGRAM_BYTES`.
#define MAX_DATAGRAM_BYTES 2048
@@ -363,6 +444,21 @@
#define VIDEO_CAP_HOST_TIMING 8
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Hello::video_caps`] bit: the client's reassembler keeps **speed-test probe filler in its own
// frame-index space** (a second reassembly window keyed on the [`crate::packet::FLAG_PROBE`]
// user-flag), so probe bursts no longer consume video `frame_index`es. Without this, a mid-session
// speed test burns thousands of video indexes that are invisible to every client-side gap detector
// (probe frames are filtered before the pump sees them) — the first real AU afterwards reads as a
// phantom multi-thousand-frame loss (spurious freeze + a nonsense RFI). It also lets the host's
// encode loop own the video numbering outright (the wire-index contract
// [`crate::packet::Packetizer::packetize_each`] documents), which reference-frame invalidation
// depends on. The host runs mid-session probe bursts ONLY against clients that set this bit — an
// older client gets a declined (zeroed) [`ProbeResult`] instead of a measurement its single-window
// reassembler would silently drop as stale.
#define VIDEO_CAP_PROBE_SEQ 16
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// QUIC application error code a punktfunk/1 client closes the control connection with on a
// **deliberate quit** (a user "stop", not a network drop). The host reads it off the connection's
@@ -391,6 +487,16 @@
#define HOST_CAP_GAMEPAD_STATE 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Welcome::host_caps`] bit: the host has a shared-clipboard service (a working OS backend)
// **and** its operator policy does not hard-disable it, so the client may offer the clipboard
// toggle. Absent (an older host, or `PUNKTFUNK_CLIPBOARD` off) ⇒ the client greys the toggle
// out. Purely additive: nothing clipboard-related happens until a [`ClipControl`]`{ enabled:
// true }` crosses (see `design/clipboard-and-file-transfer.md` §3.1). Packs into the existing
// trailing `host_caps` byte — no wire-layout change.
#define HOST_CAP_CLIPBOARD 2
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software**
// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST
@@ -462,6 +568,11 @@
#define MSG_BITRATE_CHANGED 6
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`RfiRequest`].
#define MSG_RFI_REQUEST 7
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ProbeRequest`].
#define MSG_PROBE_REQUEST 32
@@ -482,6 +593,119 @@
#define MSG_CLOCK_ECHO 49
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ClipControl`] (client → host): enable/disable the shared clipboard for this
// session. Idempotent; opt-in is enforced here, not just in UI.
#define MSG_CLIP_CONTROL 64
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ClipState`] (host → client): ack + unsolicited policy/backend updates.
#define MSG_CLIP_STATE 65
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ClipOffer`] (symmetric): the lazy announcement — format list only, no bytes.
#define MSG_CLIP_OFFER 66
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ClipFetch`] (requester → holder, **fetch stream only**): pull one format of the
// current offer.
#define MSG_CLIP_FETCH 67
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`ClipFetchHdr`] (holder → requester, **fetch stream only**): the fetch response
// header that precedes the data chunks.
#define MSG_CLIP_FETCH_HDR 68
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipControl::flags`] bit: the client permits file kinds to be offered/fetched this session.
// Absent ⇒ files are filtered out of offers in both directions (text/rich/image only).
#define CLIP_FLAG_FILES 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::policy`] bit: the host permits non-file formats (text/RTF/HTML/image). Always set
// while enabled unless a future direction limit clears it.
#define CLIP_POLICY_TEXT 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::policy`] bit: the host permits file formats. Cleared by the operator `no-files`
// / `text-only` policy so the client can grey out "Include files".
#define CLIP_POLICY_FILES 2
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::reason`]: normal ack, nothing exceptional.
#define CLIP_REASON_OK 0
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::reason`]: this session type has no working clipboard backend (e.g. a gamescope
// session with no data-control global) — the client shows "not supported in this session type".
#define CLIP_REASON_BACKEND_UNAVAILABLE 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::reason`]: another client took over the single per-desktop clipboard binding; this
// one was disabled (last `ClipControl{enabled}` wins).
#define CLIP_REASON_TAKEN_OVER 2
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::reason`]: the host operator policy (`PUNKTFUNK_CLIPBOARD=off`) disables clipboard.
#define CLIP_REASON_POLICY_DISABLED 3
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipState::reason`]: enabled, but the host policy forbids file transfer (`no-files` /
// `text-only`) — surfaced so the client greys "Include files" with a footnote.
#define CLIP_REASON_NO_FILES 4
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipFetchHdr::status`]: the requested format is being served; data chunks follow until FIN.
#define CLIP_FETCH_OK 0
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipFetchHdr::status`]: the fetch named a `seq` that is no longer the holder's current offer;
// the requester degrades the paste to "nothing inserted" rather than wrong data. No chunks follow.
#define CLIP_FETCH_STALE 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipFetchHdr::status`]: the format/index is not available (no backend, or it vanished). No
// chunks follow.
#define CLIP_FETCH_UNAVAILABLE 2
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipFetchHdr::status`]: policy/cap denies this fetch (e.g. a file fetch under `no-files`). No
// chunks follow.
#define CLIP_FETCH_DENIED 3
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Maximum number of [`ClipKind`] entries in one [`ClipOffer`] (resource cap, §7).
#define CLIP_MAX_KINDS 16
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Maximum length in bytes of a [`ClipKind::mime`] string (resource cap, §7).
#define CLIP_MAX_MIME 128
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// [`ClipFetch::file_index`] sentinel meaning "not a file fetch" (a whole non-file format, or the
// file *manifest* itself). Real file fetches use `0..n`.
#define CLIP_FILE_INDEX_NONE UINT32_MAX
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Type byte of [`PairRequest`].
#define MSG_PAIR_REQUEST 16
@@ -538,6 +762,25 @@
#define ColorInfo_MC_BT2020_NCL 9
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Stream-kind byte: a clipboard fetch (request/response of one format). Future stream kinds
// (e.g. a bulk file-content push) mux under the same [`STREAM_MAGIC`] with a different byte.
#define CLIP_STREAM_KIND_FETCH 1
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// QUIC application error code used to `reset`/`stop` a clipboard fetch stream on cancel — sync
// disabled mid-transfer, paste timed out, size cap exceeded, teardown. Distinct from the
// connection close codes ([`super::QUIT_CLOSE_CODE`] `0x51` / [`super::APP_EXITED_CLOSE_CODE`]
// `0x52`) and the connection reject code `0x42`.
#define CLIP_CANCELLED_CODE 96
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Chunk size for streaming fetch data (64 KiB writes — matches the control-frame bound).
#define CLIP_CHUNK (64 * 1024)
#endif
// Stable C ABI status codes. `Ok` is 0; all errors are negative so callers can
// test `rc < 0`. Do not renumber existing variants — only append.
enum PunktfunkStatus
@@ -867,6 +1110,47 @@ typedef struct {
} PunktfunkRichInputEx;
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// One advertised clipboard format passed to [`punktfunk_connection_clipboard_offer`].
typedef struct {
// NUL-terminated UTF-8 wire MIME (e.g. `text/plain;charset=utf-8`). ≤ 128 bytes on the wire.
const char *mime;
// Best-effort size in bytes; `0` = unknown.
uint64_t size_hint;
} PunktfunkClipKind;
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// A shared-clipboard event, filled by [`punktfunk_connection_next_clipboard`]. A flat tagged
// struct (like `PunktfunkHidOutput`): read the fields named in the `kind`'s doc; the rest are 0.
typedef struct {
// One of `PUNKTFUNK_CLIP_*`.
uint8_t kind;
// `State`: 1 = enabled, 0 = disabled.
uint8_t enabled;
// `State`: bitfield of `quic::CLIP_POLICY_*` — what the host currently permits.
uint8_t policy;
// `State`: one of `quic::CLIP_REASON_*`.
uint8_t reason;
// `Data`: 1 = final chunk of this transfer.
uint8_t last;
// Per-transfer id: the offer `seq` (RemoteOffer), the `req_id` (FetchRequest), or the
// `xfer_id` (Data/Cancelled/Error).
uint32_t transfer_id;
// `FetchRequest`: the offer `seq` the request is against.
uint32_t seq;
// `FetchRequest`: file index, or `quic::CLIP_FILE_INDEX_NONE`.
uint32_t file_index;
// `Error`: a `PunktfunkStatus` code (negative); 0 otherwise.
int32_t status;
// RemoteOffer/FetchRequest/Data: a pointer into a per-connection slot, valid until the next
// `next_clipboard` call; NULL for the other kinds.
const uint8_t *data;
// Byte length of `data` (0 when `data` is NULL).
uintptr_t len;
} PunktfunkClipEvent;
#endif
// A speed-test measurement, filled by [`punktfunk_connection_probe_result`]. `done` is 0 until
// the host's end-of-burst report lands, then 1 (the numbers are final). `throughput_kbps` is the
// delivered wire throughput to drive a bitrate choice from; `loss_pct` is the link loss and
@@ -1506,6 +1790,96 @@ PunktfunkStatus punktfunk_connection_mode(const PunktfunkConnection *c,
PunktfunkStatus punktfunk_connection_gamepad(const PunktfunkConnection *c, uint32_t *gamepad);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// The host capability bitfield the session's `Welcome` carried — a bitfield of
// `PUNKTFUNK_HOST_CAP_GAMEPAD_STATE` / `PUNKTFUNK_HOST_CAP_CLIPBOARD`. A client tests
// `caps & PUNKTFUNK_HOST_CAP_CLIPBOARD` to decide whether to offer the shared-clipboard toggle.
// Safe any time after connect.
//
// # Safety
// `c` is a valid connection handle; `caps` is writable (NULL is skipped).
PunktfunkStatus punktfunk_connection_host_caps(const PunktfunkConnection *c, uint8_t *caps);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Enable or disable the shared clipboard for this session (`design` §3.1). Opt-in: nothing is
// announced or served until this is called with `enabled = true`. `flags` carries
// `quic::CLIP_FLAG_FILES` (allow file transfer). The host replies with a `State` event.
//
// # Safety
// `c` is a valid connection handle.
PunktfunkStatus punktfunk_connection_clipboard_control(const PunktfunkConnection *c,
bool enabled,
uint8_t flags);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Announce that the local clipboard changed — the lazy format-list offer. `seq` is a monotonic
// per-sender counter (newest wins); `kinds`/`n` is the advertised formats (≤ 16). The bytes cross
// only if the host later fetches.
//
// # Safety
// `c` is a valid connection handle; `kinds` points to `n` `PunktfunkClipKind`s (NULL allowed only
// when `n == 0`), each `mime` a valid NUL-terminated UTF-8 string.
PunktfunkStatus punktfunk_connection_clipboard_offer(const PunktfunkConnection *c,
uint32_t seq,
const PunktfunkClipKind *kinds,
uintptr_t n);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Start pulling one format (`mime`) of the host's current offer `seq` — lazily, on a local paste.
// `file_index` selects a file for a file transfer, or `quic::CLIP_FILE_INDEX_NONE` for a non-file
// format. Writes the transfer id (echoed on the resulting `Data`/`Error`/`Cancelled` event) to
// `xfer_id_out`.
//
// # Safety
// `c` is a valid connection handle; `mime` is a valid NUL-terminated UTF-8 string; `xfer_id_out`
// is writable (NULL is skipped).
PunktfunkStatus punktfunk_connection_clipboard_fetch(const PunktfunkConnection *c,
uint32_t seq,
const char *mime,
uint32_t file_index,
uint32_t *xfer_id_out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Provide bytes answering a `FetchRequest` event (the host is pasting our offered data). Call
// repeatedly to stream a large payload; `last = true` completes it. `data` may be NULL only when
// `len == 0` (e.g. a final empty chunk). `punktfunk_connection_clipboard_cancel(req_id)` aborts.
//
// # Safety
// `c` is a valid connection handle; `data` points to `len` bytes (NULL allowed only when
// `len == 0`).
PunktfunkStatus punktfunk_connection_clipboard_serve(const PunktfunkConnection *c,
uint32_t req_id,
const uint8_t *data,
uintptr_t len,
bool last);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Cancel a clipboard transfer by id — either an outbound fetch (`xfer_id` from
// [`punktfunk_connection_clipboard_fetch`]) or an inbound serve (`req_id` from a `FetchRequest`).
//
// # Safety
// `c` is a valid connection handle.
PunktfunkStatus punktfunk_connection_clipboard_cancel(const PunktfunkConnection *c, uint32_t id);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Pull the next shared-clipboard event into `*out`. [`PunktfunkStatus::NoFrame`] on timeout,
// [`PunktfunkStatus::Closed`] once the session ended. A native client drains this on its own
// thread and drives the OS pasteboard from it. The `data`/`len` pointer (when non-NULL) borrows a
// per-connection buffer valid until the next `next_clipboard` call on this handle.
//
// # Safety
// `c` is a valid connection handle; `out` is writable for one `PunktfunkClipEvent`.
PunktfunkStatus punktfunk_connection_next_clipboard(PunktfunkConnection *c,
PunktfunkClipEvent *out,
uint32_t timeout_ms);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// The compositor backend the host actually resolved for this session (one of the
// `PUNKTFUNK_COMPOSITOR_*` values; the `Welcome`'s echo of the [`punktfunk_connect_ex`]
@@ -1573,6 +1947,43 @@ PunktfunkStatus punktfunk_connection_request_mode(const PunktfunkConnection *c,
PunktfunkStatus punktfunk_connection_request_keyframe(const PunktfunkConnection *c);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Ask the host to recover from loss by **reference-frame invalidation** rather than a full IDR:
// report the range `[first_frame, last_frame]` of access units the client can no longer trust
// (the first missing `frame_index` through the newest received). An RFI-capable host (AMD LTR /
// NVENC) re-references a known-good picture before `first_frame` and emits a clean P-frame tagged
// `USER_FLAG_RECOVERY_ANCHOR` — no 20-40x IDR spike; a host that can't RFI forces an IDR instead
// (same effect as [`punktfunk_connection_request_keyframe`]). Non-blocking, fire-and-forget; the
// recovered frame is the only ack, so THROTTLE it exactly like the keyframe request. Prefer this
// over the keyframe request on loss so AMD/RFI hosts avoid the spike; keep the keyframe request as
// the backstop for when the recovery frame itself is lost.
//
// # Safety
// `c` is a valid connection handle.
PunktfunkStatus punktfunk_connection_request_rfi(const PunktfunkConnection *c,
uint32_t first_frame,
uint32_t last_frame);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Feed each received frame's `frame_index` (the [`PunktfunkFrame::frame_index`] field, in receive
// order) so the client recovers from loss with a cheap reference-frame invalidation instead of a
// full IDR. On a forward gap (a `frame_index` jump = the intervening frames were lost and the
// following AUs reference a picture that never arrived) this fires a THROTTLED
// [`punktfunk_connection_request_rfi`] for the lost range; an RFI-capable host (AMD LTR / NVENC)
// then recovers with a clean P-frame instead of a 20-40x IDR spike. Call it for every received
// frame — it is cheap and idempotent, and the [`punktfunk_connection_frames_dropped`]-driven
// keyframe request stays the backstop. Writes whether a forward gap was detected this call to
// `gap_out` (nullable — a client with a post-loss display freeze can use it to re-arm; most
// clients pass NULL and ignore it).
//
// # Safety
// `c` is a valid connection handle; `gap_out` is writable or NULL.
PunktfunkStatus punktfunk_connection_note_frame_index(const PunktfunkConnection *c,
uint32_t frame_index,
bool *gap_out);
#endif
#if defined(PUNKTFUNK_FEATURE_QUIC)
// Cumulative access units the host→client reassembler dropped as unrecoverable (FEC couldn't
// rebuild them). A video loop polls this and calls [`punktfunk_connection_request_keyframe`]
+7 -1
View File
@@ -67,7 +67,13 @@ build() {
# caveat the RPM documents): ln -s "$(find / -name libcuda.so -path '*stubs*'|head -1)" /usr/lib/
# punktfunk-client-session is the Vulkan/Skia streamer the shell (punktfunk-client) execs for
# a connect — both binaries must ship or streaming from the desktop client breaks.
cargo build --release --locked \
# `--features punktfunk-host/nvenc` compiles in the direct-SDK NVENC path (real RFI + recovery
# anchor on Linux NVIDIA; design/linux-direct-nvenc.md). AMD/Intel-SAFE: the NVENC/CUDA entry
# points are dlopen'd at RUNTIME (libloading), never link-imported — `objdump -p` shows the same
# DT_NEEDED as a plain build (no libcuda/libnvidia-encode), so the binary starts fine driver-less
# and only touches NVIDIA on a CUDA capture frame (default on NVIDIA; PUNKTFUNK_NVENC_DIRECT=0
# opts back to libav). AMD/Intel never reach it — the `cuda` gate leaves them on VAAPI.
cargo build --release --locked --features punktfunk-host/nvenc \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-tray
# Management web console (opt-in): the Nitro `bun`-preset .output bundle (Bun.serve TLS),
# built AND run with bun.
+1 -1
View File
@@ -85,7 +85,7 @@ ujust add-user-to-input-group # virtual gamepads need /dev/uinput (re-
mkdir -p ~/.config/punktfunk
cp /usr/share/punktfunk/host.env.bazzite ~/.config/punktfunk/host.env # gamescope defaults
systemctl --user enable --now punktfunk-host
# Web console — enable it and read the auto-generated login password (then open http://<host-ip>:47992):
# Web console — enable it and read the auto-generated login password (then open https://<host-ip>:47992):
systemctl --user enable --now punktfunk-web
journalctl --user -u punktfunk-web-init | sed -n 's/.*password generated: //p'
```
+7 -2
View File
@@ -175,7 +175,12 @@ export PUNKTFUNK_BUILD_VERSION="%{version}-%{release}"
# --locked: reproducible from (commit + Cargo.lock), matching the .deb build path.
# punktfunk-client-session is the Vulkan/Skia streamer the shell execs for a connect — both
# client binaries must ship or streaming from the desktop client breaks.
cargo build --release --locked \
# --features punktfunk-host/nvenc: the direct-SDK NVENC path (real RFI + recovery anchor on Linux
# NVIDIA; design/linux-direct-nvenc.md). AMD/Intel-safe — the NVENC/CUDA entry points are dlopen'd
# at runtime (no link-time dep; __requires_exclude already drops libcuda), so the binary starts
# driver-less; the encoder engages only on a CUDA frame (default on NVIDIA; PUNKTFUNK_NVENC_DIRECT=0
# opts back to libav) — the `cuda` gate keeps AMD/Intel on VAAPI regardless.
cargo build --release --locked --features punktfunk-host/nvenc \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-tray
%if %{with web}
@@ -383,7 +388,7 @@ echo "punktfunk-web installed. Enable the console for your user:"
echo " systemctl --user enable --now punktfunk-web"
echo "A login password is generated on first start read it with:"
echo " journalctl --user -u punktfunk-web-init | sed -n 's/.*password generated: //p'"
echo "Then open http://<host-ip>:3000"
echo "Then open https://<host-ip>:47992"
%endif
%changelog
+1 -1
View File
@@ -15,5 +15,5 @@ if [ ! -s "$PWFILE" ]; then
(umask 077; printf 'PUNKTFUNK_UI_PASSWORD=%s\n' "$PW" > "$PWFILE")
chmod 600 "$PWFILE" 2>/dev/null || true
echo "punktfunk web console login password generated: $PW"
echo "(stored in $PWFILE — open http://<host-ip>:47992 and log in)"
echo "(stored in $PWFILE — open https://<host-ip>:47992 and log in)"
fi

Some files were not shown because too many files have changed in this diff Show More