Compare commits

...

19 Commits

Author SHA1 Message Date
enricobuehler a9dc6efe55 fix(windows): drop the orphaned touch_last_used re-export
ci / web (push) Successful in 44s
ci / docs-site (push) Successful in 1m0s
ci / bench (push) Successful in 6m1s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m2s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m29s
ci / rust (push) Successful in 22m28s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 13s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 12s
decky / build-publish (push) Successful in 19s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 11s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
android-screenshots / screenshots (push) Successful in 3m6s
flatpak / build-publish (push) Successful in 7m12s
web-screenshots / screenshots (push) Successful in 2m53s
linux-client-screenshots / screenshots (push) Successful in 10m8s
docker / deploy-docs (push) Successful in 23s
arch / build-publish (push) Successful in 11m54s
deb / build-publish (push) Successful in 12m53s
windows-host / package (push) Successful in 15m26s
android / android (push) Successful in 15m45s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 17m25s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 16m16s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m59s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m59s
apple / swift (push) Successful in 4m56s
release / apple (push) Successful in 24m44s
apple / screenshots (push) Successful in 20m44s
More builtin-removal fallout: trust.rs re-exported pf_client_core::trust::
touch_last_used, whose only consumer was the deleted in-process session pump. In
a binary crate an unused pub-use is a hard -D warnings error (it surfaced only
after the gamepad dead-code errors were cleared, which had suppressed the
unused_imports pass). Drop it; every other re-export still has a user.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 08:37:24 +02:00
enricobuehler 9822fc3b1c fix(windows): drop the orphaned in-process gamepad forwarding hooks
ci / web (push) Successful in 1m7s
ci / docs-site (push) Successful in 1m0s
ci / bench (push) Successful in 5m49s
android-screenshots / screenshots (push) Successful in 3m17s
ci / rust (push) Successful in 17m50s
decky / build-publish (push) Successful in 18s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 59s
apple / swift (push) Successful in 4m25s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m22s
android / android (push) Successful in 14m55s
arch / build-publish (push) Successful in 13m12s
deb / build-publish (push) Successful in 11m29s
web-screenshots / screenshots (push) Successful in 2m42s
docker / deploy-docs (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 10s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 11s
linux-client-screenshots / screenshots (push) Successful in 7m16s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 12m58s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m36s
windows-host / package (push) Successful in 15m17s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m10s
release / apple (push) Successful in 22m45s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m59s
flatpak / build-publish (push) Successful in 5m41s
apple / screenshots (push) Successful in 19m29s
Removing the builtin stream path (ef580825) left GamepadService's attach/detach/
active/auto_pref + the Ctl::Attach/Detach variants with no callers — the spawned
punktfunk-session binary owns pad forwarding now. The client still compiled, but
clippy -D warnings tripped on the dead code. Drop the forwarding hooks + the
active-pad mirror; the service keeps pads() (Settings list) and set_pinned()
(persist the forwarded-pad selection the session child reads).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 07:49:38 +02:00
enricobuehler cdb43f00fe style: rustfmt the freeze-until-reanchor client wiring
ci / web (push) Successful in 57s
ci / docs-site (push) Successful in 1m7s
ci / bench (push) Successful in 5m11s
decky / build-publish (push) Successful in 28s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 40s
deb / build-publish (push) Successful in 11m37s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
arch / build-publish (push) Successful in 15m30s
apple / swift (push) Successful in 4m19s
android / android (push) Successful in 16m52s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8m34s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 10m14s
windows-host / package (push) Successful in 13m57s
flatpak / build-publish (push) Successful in 5m48s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m46s
ci / rust (push) Successful in 22m54s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 2m7s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 5m1s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m27s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m1s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m30s
release / apple (push) Successful in 21m2s
docker / deploy-docs (push) Successful in 21s
apple / screenshots (push) Successful in 19m51s
cargo fmt --all --check flagged the reanchor gate wiring (decode.rs / session.rs /
abi.rs / reanchor.rs): wrapped signatures + comparisons, and two multi-line comments
that followed a trailing-comment line were restructured to their own lines so
rustfmt keeps them at normal indentation instead of deep-aligning them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 07:36:43 +02:00
enricobuehler 644c035a34 feat(encode/amf): accept AMF runtime >=1.4.34 (graceful degrade) + log loaded amfrt64.dll identity
ci / rust (push) Failing after 47s
ci / web (push) Successful in 55s
ci / docs-site (push) Successful in 56s
decky / build-publish (push) Successful in 18s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 7s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
docker / deploy-docs (push) Successful in 11s
ci / bench (push) Successful in 5m8s
android / android (push) Successful in 14m4s
arch / build-publish (push) Successful in 12m44s
deb / build-publish (push) Successful in 13m48s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m4s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 14m19s
windows-host / package (push) Successful in 9m36s
apple / swift (push) Successful in 4m40s
apple / screenshots (push) Successful in 19m47s
The Windows AMF encoder hard-rejected any runtime <1.4.36 — a Jan-2025
(Adrenalin 25.1.1) driver floor. Every AMD host on an older driver failed the
session with "update the AMD driver" after 8 retries, notably Boot Camp Macs
whose bundled amfrt64.dll lags far behind.

Split the single pin:
- AMF_MIN_VERSION (1.4.34): the ABI floor accepted at load. Every vtable slot
  the FFI mirrors is a base-interface slot stable since well before 1.4.34; the
  1.4.35/1.4.36-only features are string-keyed encoder properties already applied
  via set_prop(required=false), which log-and-continue — so an older driver
  degrades those features individually instead of failing.
- AMF_HEADER_VERSION (1.4.36): the header the mirror targets, now passed to
  AMFInit capped at min(header, runtime) so claiming a version newer than the
  runtime can't make AMFInit reject an otherwise-usable older driver.

No functionality removed: a >=1.4.36 runtime behaves exactly as before.

Also logs, once per process, the AMF runtime version AND the loaded amfrt64.dll's
full path + file-version resource (via GetModuleFileNameW + VerQueryValueW). This
surfaces the Boot Camp failure mode where the display driver reads 25.x but the
System32 amfrt64.dll is a stale build reporting an old AMF version; the too-old
decline now names the DLL path/build and points at reboot + DDU reinstall.

Not compile-verified: amf.rs is Windows-only and this Linux box can't cross-build
it (a dependency's C build fails for the msvc target). Needs cargo check/clippy on
the Windows build box / CI. rustfmt-clean; the windows-crate FFI signatures were
verified against the on-disk 0.62.2 bindings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 01:32:51 +02:00
enricobuehler 3c16c1dd30 chore(release): bump workspace version to 0.10.0
audit / bun-audit (push) Successful in 12s
ci / rust (push) Failing after 42s
ci / docs-site (push) Successful in 48s
ci / web (push) Successful in 1m17s
audit / cargo-audit (push) Successful in 2m35s
ci / bench (push) Successful in 5m56s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 5m9s
apple / swift (push) Successful in 4m44s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m31s
apple / screenshots (push) Successful in 19m30s
decky / build-publish (push) Successful in 24s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 27s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 11s
android-screenshots / screenshots (push) Successful in 3m19s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6m58s
windows-host / package (push) Successful in 9m32s
linux-client-screenshots / screenshots (push) Successful in 8m5s
docker / deploy-docs (push) Successful in 22s
web-screenshots / screenshots (push) Successful in 2m44s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 2m36s
arch / build-publish (push) Successful in 12m38s
deb / build-publish (push) Successful in 12m44s
android / android (push) Successful in 13m10s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 4m28s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 13m59s
release / apple (push) Successful in 24m1s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 19m20s
flatpak / build-publish (push) Failing after 8m2s
Bumps [workspace.package] version 0.9.2 -> 0.10.0 (14 workspace crates). Apple
MARKETING_VERSION / Android versionName are set from the release tag by CI, so no
client manifest changes; the nested Windows-driver workspace keeps its independent
0.0.1 version.

Also syncs Cargo.lock: the version bump for the 14 members, plus dropping the
now-orphaned crossbeam-channel entry the Windows builtin removal (ef580825) left
behind (it dropped the dep from the manifest but not the lock), so
`cargo build --locked` (ci.yml / deb.yml) stays green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 01:32:26 +02:00
enricobuehler c0fc2d8ee8 feat(apple): iPad ⌃⌥⇧Q release chord + click-to-recapture, pixel-grid snap, match-window opt-in
ci / docs-site (push) Successful in 49s
ci / rust (push) Failing after 51s
ci / web (push) Successful in 52s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 9s
decky / build-publish (push) Successful in 16s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 9s
deb / build-publish (push) Failing after 4m20s
apple / swift (push) Successful in 4m43s
arch / build-publish (push) Failing after 4m56s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Failing after 4m8s
ci / bench (push) Successful in 5m43s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 6m21s
docker / deploy-docs (push) Successful in 21s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Failing after 3m33s
windows-host / package (push) Successful in 9m2s
flatpak / build-publish (push) Failing after 8m11s
android / android (push) Successful in 12m4s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 3m54s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m39s
windows / build (aarch64-pc-windows-msvc) (push) Failing after 5m3s
release / apple (push) Successful in 21m56s
windows / build (x86_64-pc-windows-msvc) (push) Failing after 5m31s
apple / screenshots (push) Successful in 19m11s
- InputCapture / StreamViewIOS: iPad ⌃⌥⇧Q un-capture chord, recognized from the
  GCKeyboard HID stream (no NSEvent monitor on iOS) for cross-client parity with
  the macOS/Windows/Linux combo; and a click into the video re-engages capture —
  the iPad analogue of macOS mouseDown → engageCapture(fromClick:), with the
  engaging click suppressed toward the host.
- SessionPresenter: snap the aspect-fit sublayer frame to the backing pixel grid.
  AVMakeRect centers the fit rect at fractional points, so the compositor
  resampled the layer — a uniform "everything soft" blur even when the drawable
  was pixel-exact 1:1. Rounding origin + size to device pixels makes the composite
  a true 1:1 blit; idempotent when already aligned.
- MetalVideoPresenter: PUNKTFUNK_BILINEAR_LUMA=1 A/B lever — compiles the shader
  with Catmull-Rom luma off (plain bilinear) to isolate bicubic overshoot from
  upstream fringing.
- SettingsView / StreamView / StreamViewIOS: match-window reverted to opt-in
  (default OFF) — the explicit mode is used and never auto-resized unless enabled.
2026-07-13 01:22:09 +02:00
enricobuehler ef5808254a refactor(windows): remove the legacy in-process builtin stream path
The real Windows client is the spawned punktfunk-session Vulkan binary
(pf-client-core); the in-process builtin GUI stream — reachable only via
PUNKTFUNK_BUILTIN_STREAM=1 — was dead weight kept alive by nothing and a
recurring source of wasted effort. Remove it: delete present/render/input/
audio.rs and the builtin remainder of session/video.rs, rip all the builtin
wiring (app/mod, connect, stream), and make connect always spawn.

Preserve the two shipped keepers that happened to live in those files by
relocating them to a new probe.rs: run_speed_probe (the per-host network speed
test used by the Settings speed page and --headless --speed-test) and
decodable_codecs (the codec-capability advert on the probe connect). Trim gpu.rs
to just the Settings adapter picker (adapter_names + helpers). --headless now
supports only --speed-test — the in-process decode/frame-counter went with the
pump.

Drops the now-orphaned deps opus, wasapi, crossbeam-channel, anyhow; keeps
ffmpeg-next (probe::decodable_codecs still needs it). Net 4432 deletions.
Statically verified (module wiring, imports, orphaned symbols/deps all clean);
the type-level compile runs on the windows-amd64 CI runner, which has the
toolchain this non-Windows host lacks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 01:22:09 +02:00
enricobuehler 8a18e130a2 feat(client): freeze-until-reanchor loss recovery on Android + Apple via shared core gate
After unrecoverable loss the host keeps sending delta frames that reference a
picture the client never received; hardware decoders conceal these as gray/
garbage with a success status. Linux already withheld them and held the last
good frame until a proven clean re-anchor — this brings that behavior to the
Android and Apple clients.

Extract the Linux pump's freeze state machine into a shared `ReanchorGate` in
punktfunk-core (reanchor.rs, 18 tests) exposed over the C ABI (ABI v6, additive —
no wire change) for the Swift clients. Migrate the Linux/Deck pump
(pf-client-core) onto it as the parity proof (no-op refactor). Then wire:

- Android (decode.rs, both sync + async loops): arm on the frame-index gap, a
  pts-keyed flag map carries the wire flags to the output-buffer release, fold
  the gate per drained output, gate.poll replaces the dropped-climb block.
- Apple Stage2Pipeline (default): arm on a gap (new noteFrameIndexGap), withhold
  at the ring-submit seam (CAMetalLayer holds its last drawable), poll
  framesDropped, fold VT decode errors through the no-output streak.
- Apple StreamPump (stage-1): fold at enqueue, withhold via
  kCMSampleAttachmentKey_DoNotDisplay so the layer keeps decoding (reference
  chain intact) but holds the last displayed frame.
- Apple VideoDecoder: thread the AU's wire flags to the async decode callback via
  a retained FrameContext refcon (replaces the receivedNs bit-pattern scalar).

Lifts only on a proven re-anchor (IDR / RFI anchor / 2nd recovery mark) with a
500 ms backstop so a lost re-anchor can never freeze forever. Apple: swift build
clean, 123/123 tests pass (incl. VideoToolboxRoundTripTests). On-glass
loss-injection validation still owed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 01:22:09 +02:00
enricobuehler cd701a9594 fix(vdisplay): preserve FramePublisher across swap-chain reassign (sibling-join freeze)
windows-drivers / probe-and-proto (push) Successful in 28s
ci / web (push) Successful in 54s
ci / docs-site (push) Successful in 1m0s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
decky / build-publish (push) Successful in 19s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 9s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 7s
windows-drivers / driver-build (push) Successful in 1m35s
apple / swift (push) Successful in 4m34s
ci / bench (push) Successful in 5m39s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6m50s
docker / deploy-docs (push) Successful in 33s
windows-host / package (push) Successful in 10m8s
android / android (push) Successful in 12m24s
arch / build-publish (push) Successful in 16m41s
deb / build-publish (push) Successful in 17m40s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 12m13s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 18m21s
ci / rust (push) Successful in 23m52s
apple / screenshots (push) Successful in 20m9s
When a second client got its own virtual display mid-stream, the FIRST
client's IDD-push stream froze (video only; `new_fps=0 repeat_fps=240`
forever). Adding/removing/resizing a sibling display re-commits the CCD
topology, which makes the OS unassign+reassign the first monitor's IddCx
swap chain. `unassign_swap_chain` dropped the SwapChainProcessor, dropping
`run_core`'s local FramePublisher and closing the sealed-channel handles.
The fresh worker then polled the frame-channel stash — but that stash is
consumed once at session open, and the host only re-delivers on a ring
recreate (a descriptor change). The first monitor's descriptor didn't
change and WUDFHost stayed alive, so no watchdog fired: the driver drained
the swap chain without publishing and the host repeated its last frame
indefinitely. Confirmed twice on the .173 box (host.log 21:12 & 21:15).

Preserve the live FramePublisher across the flap instead of dropping it:
the host-owned ring (header/event/textures) it holds stays valid — only
the swap chain died.

- frame_transport.rs: FramePublisher records its render-adapter LUID +
  exposes render_adapter().
- monitor.rs: MonitorObject.preserved_publisher + preserve_publisher()
  (mirrors set_frame_channel) + take_preserved_publisher() (mirrors
  take_frame_channel). Monitor teardown drops the stashed publisher and
  closes its ring handles, so nothing leaks.
- swap_chain_processor.rs run_core: after SetDevice OK, re-adopt a
  preserved publisher ONLY when the new swap chain renders on the same
  LUID (same pooled Direct3DDevice → its context + opened textures are
  valid); on loop exit, stash the publisher back on the monitor.

Safe: the old worker is fully joined (drop-outside-lock discipline)
before the new one runs, so no concurrent context use; a stale re-adopted
publisher is superseded by the existing is_stale() + has_frame_channel()
newest-wins checks at the loop top.

Verified clippy -D warnings clean on rustc 1.96.0 via a faithful mock
crate (the real crate needs the WDK to compile). Needs a driver rebuild +
reinstall on the host to take effect; not yet hardware-validated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 00:19:20 +02:00
enricobuehler 05868ef634 fix(encode): Vulkan-HEVC full-RPS reference retention + AV1 feature gate (RFI review)
ci / web (push) Successful in 56s
ci / docs-site (push) Successful in 1m0s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 14s
decky / build-publish (push) Successful in 30s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 52s
apple / swift (push) Successful in 4m47s
ci / bench (push) Successful in 6m2s
docker / deploy-docs (push) Successful in 24s
windows-host / package (push) Successful in 8m6s
arch / build-publish (push) Successful in 11m4s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 13m19s
android / android (push) Successful in 16m58s
deb / build-publish (push) Successful in 17m45s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 18m38s
ci / rust (push) Successful in 24m8s
apple / screenshots (push) Successful in 20m38s
2026-07-12 review of the host encoders / client decoders / RFI plane.
NVENC (both), AMF-LTR, the session glue, and the client RfiTracker came
out clean; every fix lands in the Vulkan Video backend + dispatch:

1. HEVC: author each P-frame's short-term RPS to retain ALL resident DPB
   pictures (minus the setup slot), not just its one reference. HEVC
   8.3.2 evicts unlisted pictures, and clients keep FEEDING the decoder
   while frozen — so with the old single-pic RPS, a conforming parser
   (FFmpeg = the Linux VAAPI/Vulkan and Windows D3D11VA clients) had
   already discarded the picture an RFI recovery anchor references
   whenever a fed post-loss frame preceded it: generate_missing_ref, and
   the "clean" anchor plus everything chained after it decodes as
   garbage. Pure builder (`build_h265_rps_s0`) + unit tests; AV1 needs
   nothing (slot-based retention). The smoke test now encodes a fed
   post-loss frame between loss@4 and anchor@6 so an ffmpeg decode of
   the dropped dump exercises exactly this (expect ONE POC-4 complaint,
   never POC 3) — revalidate on the AMD box; this NVIDIA dev box fails
   the backend earlier at HEVC header retrieval (pre-existing).

2. AV1: chain PhysicalDeviceVideoEncodeAV1FeaturesKHR (videoEncodeAV1 =
   TRUE, stype 1000513004) into device creation — spec-required for the
   ENCODE_AV1 codec op; RADV tolerated the omission, validation layers
   and stricter drivers do not.

3. RFI decline no longer self-arms force_kf — that bypassed the session
   glue's 750 ms IDR cooldown, turning a storm of hopeless RFI requests
   into one full IDR each. Decline like NVENC/AMF and let the caller's
   coalesced keyframe path own the fallback; add the missing
   first>last guard for parity.

4. open_video_backend now returns the label of the branch that ACTUALLY
   opened, so the mgmt API / web console reports "vulkan" instead of
   "vaapi" for the default-on Vulkan sessions (the old dispatch-mirror
   resolved_backend_label went stale when the backend gained its VAAPI
   fallback; deleted).

Structure: the ~230-line inline HEVC coding block moves to
record_coding_h265 (symmetric with record_coding_av1) and the duplicated
pre-encode barriers dedupe into begin_encode_cmd.

Follow-up plan (separate, punktfunk-planning): bring the post-loss
freeze + RECOVERY_ANCHOR/POINT lift to the Android/Windows/Apple clients
via a shared ReanchorGate (design/client-reanchor-freeze-parity.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 23:12:12 +02:00
enricobuehler 2d37835545 feat(encode): AV1 on the Linux Vulkan Video encoder (real RFI)
ci / web (push) Successful in 1m12s
ci / docs-site (push) Successful in 1m22s
decky / build-publish (push) Successful in 19s
ci / bench (push) Successful in 5m36s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
arch / build-publish (push) Successful in 12m32s
docker / deploy-docs (push) Successful in 21s
android / android (push) Successful in 13m27s
ci / rust (push) Successful in 17m43s
deb / build-publish (push) Successful in 13m40s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 15m19s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 15m36s
windows-host / package (push) Successful in 14m43s
apple / swift (push) Successful in 4m42s
apple / screenshots (push) Successful in 19m20s
Extend the raw Vulkan Video backend to AV1 (`VK_KHR_video_encode_av1`)
alongside HEVC, so AMD/Intel Linux hosts get the same clean-P-frame loss
recovery for AV1 that HEVC already has — no full IDR on packet loss.

ash 0.38.0+1.3.281 predates the AV1-encode extension (finalized in Vulkan
1.3.290) and bumping ash breaks the SDL/Vulkan client (it drops the
lifetime on AllocationCallbacks, which sdl3-sys still generates). So the
AV1-encode structs/flags/enums are vendored host-only in
`encode/linux/vk_av1_encode.rs`, copied verbatim from ash-master's
generated code and chained into ash's generic video-encode calls via raw
p_next — the common StdVideoAV1* types (from AV1 decode) are reused from
ash 1.3.281.

`vulkan_video.rs` gains a parallel AV1 path: AV1 Main profile/caps/session
(+ max-level session-create), a bit-packed sequence-header OBU + per-frame
temporal-delimiter framing (Vulkan AV1 encode, unlike H26x, emits only the
frame OBU), and per-frame StdVideoEncodeAV1PictureInfo with the RFI
reference model — a normal P inherits CDF context from its reference for
compression, while an IDR or recovery anchor sets primary_ref_frame=NONE +
error_resilient_mode so it decodes independent of the (possibly lost)
frames since its reference. HEVC recording is unchanged; the shared CSC /
ring / DPB-barrier pipeline is reused as-is. Codec routing in
`open_video_backend` extends the HEVC arm to HEVC|AV1.

The seq header enables only order-hint (+128px superblocks per caps),
matching FFmpeg's proven Vulkan AV1 config — enabling CDEF/restoration made
VCN emit frame-header sections our seq header didn't match, desyncing every
inter frame.

Headless-validated on real RADV (780M): open + 6-frame encode (I P P P P P)
decodes 0-error on both dav1d and ffmpeg/cbs; the RFI recovery anchor at
frame 4 is a clean P (not IDR), and dropping the "lost" frame 3 still
decodes clean (re-anchored to frame 2). HEVC smoke unchanged (no
regression). `cargo clippy --features vulkan-encode -- -D warnings` and the
no-feature build both green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 22:06:24 +02:00
enricobuehler 9514a8c0e2 fix(client): correct Linux/Windows "Forwarded controller" copy for multi-controller
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 56s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 10s
decky / build-publish (push) Successful in 52s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 8s
apple / swift (push) Successful in 4m39s
ci / bench (push) Successful in 6m27s
windows-host / package (push) Successful in 8m34s
flatpak / build-publish (push) Successful in 7m15s
docker / deploy-docs (push) Successful in 24s
android / android (push) Successful in 12m49s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, --no-default-features, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 4m16s
deb / build-publish (push) Successful in 12m50s
arch / build-publish (push) Successful in 16m32s
windows-msix / package (x64, C:\Users\Public\ffmpeg, , x86_64-pc-windows-msvc, C:\t) (push) Successful in 3m53s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m44s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 5m14s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 17m53s
ci / rust (push) Successful in 25m18s
release / apple (push) Successful in 22m3s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 5m57s
apple / screenshots (push) Successful in 18m56s
The relm4 (Linux) and native (Windows) shells spawn punktfunk-session, which
since the native-plane rework forwards ALL controllers by default — but the
"Forwarded controller" settings dropdowns still described the pre-rework world
("Automatic (most recent)", "Exactly one controller is forwarded to the host").

The dropdown already lists every detected pad and wires set_pinned(None)=all /
set_pinned(key)=single-player; this fixes only the misleading labels, subtitle,
tooltip, and stale leading comments to match: Automatic forwards every real
controller (each its own player); pick one to force single-player.

cargo check -p punktfunk-client-linux green; Windows is windows-gated (pure
string edits, CI windows.yml).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 21:53:07 +02:00
enricobuehler 97c67b2692 feat(apple): multi-controller support
Roll the pf-client-core slot pattern to the Apple client (Swift):

- GamepadManager tracks all connected GCControllers, assigning each a stable
  lowest-free wire pad index + concrete type, emitting GamepadArrival on
  connect and GamepadRemove on disconnect (index freed for reuse on re-plug).
- GamepadCapture binds every controller with per-controller Slot state
  (buttons/axes/fingers/motion), threading the pad index into flags on every
  event; GamepadWire/InputEvents carry the pad + the two new events.
- GamepadFeedback + RumbleRenderer go per-pad (rumbleByPad, slots[pad]),
  routing rumble/HID back to the correct controller by wire index.
- ContentView/Settings surface every forwarded controller.

pad 0 => flags 0, so single-controller wire is byte-identical. Cannot build
on the Linux dev box (no Swift toolchain / Apple frameworks); wire bytes
hand-checked against input.rs and GamepadWireTests extended for multi-pad.
CI apple.yml (swift build/test on macOS) is the compile gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 21:53:07 +02:00
enricobuehler 0ad4e6eff7 feat(android): multi-controller support
Roll the pf-client-core slot pattern to the Android client (Kotlin + JNI):

- New kit/GamepadRouter.kt: the Android analogue of the client-core Slot
  model — a deviceId→Slot map assigning each InputDevice a stable lowest-free
  wire pad index held for its lifetime, GamepadArrival(pref) before a pad's
  first input, GamepadRemove on onInputDeviceRemoved, per-slot AxisMapper +
  held-bitmask so two pads never clobber each other. The isForwardable gate
  (excludes DualSense/DS4 all-zero sensor sibling nodes) is centralized in
  slotFor so no entry point can open a phantom slot.
- native/src/session/input.rs: JNI shims take a pad arg -> flags=pad
  (nativeSendGamepadButton/Axis, plus nativeSendGamepadArrival/Remove).
- native/src/feedback.rs: pad carried in rumble bits 49..52 + a leading
  hidout pad byte; GamepadFeedback.kt routes rumble/lightbar/LED back to the
  originating device by pad via deviceForPad.
- MainActivity.kt routes key/motion events by device; ControllersScreen.kt
  badges every forwarded pad (was hardcoded i==0), reading getControllerNumber.

A lone controller lands on wire index 0, so its per-transition datagrams stay
byte-identical to the old single-pad path. gradle :app:assembleDebug green
(Rust cross-compiled via cargo-ndk); JNI signatures hand-verified 1:1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 21:53:07 +02:00
enricobuehler 76be4c3e12 feat(gamepad): multi-controller support on the native plane
Host was already built for 16 pads; the blocker was every client
hard-coding pad 0. This lands the host-side + reference-client contract:

- input.rs: new wire kinds GamepadArrival=14 (declares a pad's type:
  code=GamepadPref byte, flags=pad) and GamepadRemove=13 (flags=seq<<24|pad,
  shares the snapshot seq space via encode/decode_gamepad_remove).
- pf-client-core/gamepad.rs: reworked from a single `open` pad to a
  slots: Vec<Slot> model — every forwarded controller gets a stable
  lowest-free wire index held for its lifetime, per-slot held/axis/touch/
  rumble state, GamepadArrival on open + GamepadRemove on close, and
  feedback routed back per wire index. Automatic forwards all real pads;
  a pin forces single-player.
- punktfunk1.rs: replaced the single-session PadBackend enum with a Pads
  router — per-pad kinds[]/owner[] arrays, lazily-created per-kind managers,
  pure route_decision keeping a live device in its manager across a kind
  change (no ghost/dup). Input thread seq-gates GamepadRemove (clears the
  pad_mask bit, resets rumble) and applies GamepadArrival kinds.
- inject linux/windows backends: add the two new no-op InputKind arms.

Native/session + default-Windows clients (both spawn punktfunk-session)
inherit this. 57 core + 33 client-core + 272 host tests green; clippy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 21:53:07 +02:00
enricobuehler 84329205eb feat(encode): default-on the Linux Vulkan Video HEVC backend
ci / docs-site (push) Successful in 59s
ci / web (push) Successful in 1m7s
decky / build-publish (push) Successful in 18s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 8s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 8s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 44s
apple / swift (push) Successful in 4m13s
ci / bench (push) Successful in 7m13s
windows-host / package (push) Successful in 9m43s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 9m26s
docker / deploy-docs (push) Successful in 28s
android / android (push) Successful in 12m35s
deb / build-publish (push) Successful in 12m46s
arch / build-publish (push) Successful in 16m31s
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 19m32s
apple / screenshots (push) Successful in 18m49s
ci / rust (push) Successful in 25m10s
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Successful in 18m0s
On-glass validated 2026-07-12 on an AMD RADV 780M with a real Deck-class
client: the pipelined raw-Vulkan HEVC encoder ran a rock-solid 1080p@240
session and healed loss with clean P-frame recovery anchors (real RFI the
libav VAAPI path can't express). Ship it on by default, mirroring the NVENC
default-on.

- vulkan_encode_enabled() defaults ON; PUNKTFUNK_VULKAN_ENCODE=0 is the libav
  VAAPI escape hatch. A failed open still falls back to VAAPI, so a device
  without h265 Vulkan encode (or an untested Intel/ANV that misbehaves at open)
  degrades gracefully instead of breaking the stream.
- Ring depth defaults to 2 (one frame of overlap, lowest added latency — the
  on-glass-validated real-time setting); PUNKTFUNK_VULKAN_INFLIGHT still tunes it.
- Compile --features punktfunk-host/vulkan-encode into the arch/deb/rpm host
  builds (pure-Rust ash, no new lib / no link-time dep), alongside nvenc.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 20:20:31 +02:00
enricobuehler e9d8f2bc04 perf(encode): pipeline the Vulkan Video encoder (frames in flight)
The Linux raw-Vulkan HEVC backend blocked on two GPU fences per frame, so
CPU readback and the next capture could not overlap the GPU encode. Refactor
into a small ring of per-in-flight-frame resources (own command buffers,
CSC descriptor set + Y/UV/NV12 scratch, bitstream buffer, feedback query and
sync objects) so submit() records into a free slot and returns without
waiting, and poll() reads back the oldest slot once its fence signals. The
pump's non-blocking poll then overlaps a frame's CSC+encode with the next
capture — the throughput win — with no capturer/pump change (VAAPI untouched).

- New `Frame` struct + `make_frame`; encoder holds `frames`/`ring`/`in_flight`.
- `record_submit` (non-blocking) + `read_slot` (fence-gated readback) replace
  the synchronous `encode_frame`; `enqueue` applies backpressure by draining
  the oldest slot when the ring is full.
- DPB self-barrier between consecutive encodes: orders frame N's reconstruct
  write before N+1's reference read now that they can be in flight together.
- flush() drains all in-flight slots in order; reset() waits idle + discards.
- Ring depth defaults to 3, overridable via PUNKTFUNK_VULKAN_INFLIGHT (2..=6).
- Smoke test drains via poll-loop + flush (async breaks one-AU-per-submit).

Headless-validated on real RADV 780M: cargo check (feature on/off) + clippy
-D warnings + rustfmt clean; smoke test at ring depth 2/3/6 all ffmpeg-decode
clean (I P P P P P) and drop-heal (I P P P P) with 0 errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 19:51:21 +02:00
enricobuehler bbbb7f5723 fix+perf(encode): clamp Vulkan CSC to source edge + cache dmabuf imports
Two refinements after the initial on-glass validation on RADV (780M):

- Green padding bar at non-16-aligned heights (e.g. 1080 → coded 1088): the CSC
  compute shader read past the edge of the shorter source dmabuf for the 8
  alignment-padding rows, producing undefined/green garbage that showed on a
  client rendering the coded frame. Clamp every source fetch to `textureSize-1`
  so padding rows duplicate the last real row (invisible, and the SPS
  conformance window still crops it for a compliant decoder). BT.709 conversion
  is byte-identical for in-bounds pixels. 5120x1440 (exactly aligned) was never
  affected.

- Per-frame dmabuf import churn: the backend created + imported + destroyed a
  VkImage every frame (allocation jitter → stutter). PipeWire cycles a small
  fixed pool, so import each underlying buffer ONCE (keyed by st_dev/st_ino —
  each frame's fd is a fresh dup of the same buffer) and reuse it, matching the
  CUDA-path VkBridge. First import acquires from the foreign producer; cached
  re-reads keep queue ownership and use a plain visibility barrier. On-glass:
  ~3-6 imports per session then silent (was ~one per frame at 240 Hz), stutter
  gone at resolutions with headroom.

Also adds a PF_SMOKE_W/H override to the headless smoke test to exercise the
conformance-window crop path (ffprobe confirms coded 1088 → displayed 1080).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 19:24:12 +02:00
enricobuehler 5ab6daa694 feat(encode): raw Vulkan Video HEVC backend on Linux (AMD/Intel) with real RFI
Add `VulkanVideoEncoder` (`VK_KHR_video_encode_h265` via ash) — the open-stack
twin of the direct-NVENC RFI path, giving AMD/Intel Linux hosts real
reference-frame invalidation loss recovery: a clean P-frame recovery anchor
that re-references a known-good older frame instead of a full IDR. The app owns
the DPB, so recovery = pointing the P-frame's single L0 reference at a resident
slot strictly older than the loss (never a concealed frame).

The backend owns its own ash instance/device with encode + compute queues,
authors VPS/SPS/PPS (Main, conformance-window crop for non-16-aligned heights
like 1080->1088), runs a DPB-ring reference-slot state machine with monotonic
POC and CBR rate control, and does an on-GPU RGB->NV12 BT.709 compute CSC
(embedded rgb2yuv.spv) since capture delivers packed-RGB dmabufs — importing
each frame's dmabuf (explicit DRM modifier) or uploading a CPU-RGB fallback,
CSC on the compute queue, then encode on the encode queue, ordered by a
semaphore.

Wired into `open_video_backend`: an AMD/Intel HEVC session opens this instead
of libav VAAPI when `PUNKTFUNK_VULKAN_ENCODE=1` (VAAPI fallback on any open
error, so it can only improve recovery, never break a stream); `PUNKTFUNK_
ENCODER=vulkan` forces it. Gated behind the new `vulkan-encode` Cargo feature,
which pulls no new dependency (reuses the `ash` bindings already carried for
the dmabuf zero-copy bridge). Opt-in until on-glass validated, mirroring how
the direct-NVENC path shipped.

Headless-validated on real RADV (RDNA3 780M, Mesa 26): open + multi-frame
encode + `invalidate_ref_frames` all run through the real struct and ffmpeg
decodes the output `I P P P P P` with 0 errors; the recovery frame is a clean
P-frame (not an IDR); and dropping the "lost" AU still decodes cleanly because
the recovery re-anchored to an older frame — the RFI heal, proven on real
hardware. `cargo check`/`clippy -D warnings` green with the feature on and off.

Design: design/linux-vulkan-video-encode.md. Harness: design/vkenc-probe-harness/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 17:08:15 +02:00
81 changed files with 7251 additions and 5931 deletions
+4 -1
View File
@@ -94,7 +94,10 @@ jobs:
# Linux NVIDIA; design/linux-direct-nvenc.md). AMD/Intel-safe — NVENC/CUDA is dlopen'd at # Linux NVIDIA; design/linux-direct-nvenc.md). AMD/Intel-safe — NVENC/CUDA is dlopen'd at
# runtime (no link-time dep; identical DT_NEEDED to a plain build), and the encoder is only # runtime (no link-time dep; identical DT_NEEDED to a plain build), and the encoder is only
# constructed for a CUDA capture frame + PUNKTFUNK_NVENC_DIRECT, never on VAAPI hosts. # constructed for a CUDA capture frame + PUNKTFUNK_NVENC_DIRECT, never on VAAPI hosts.
cargo build --release --locked --features punktfunk-host/nvenc \ # --features punktfunk-host/vulkan-encode: the AMD/Intel twin — raw VK_KHR_video_encode_h265
# with real RFI (design/linux-vulkan-video-encode.md). Pure Rust ash (no new lib / link dep);
# default on for HEVC (PUNKTFUNK_VULKAN_ENCODE=0 → libav VAAPI), failed open falls back to VAAPI.
cargo build --release --locked --features punktfunk-host/nvenc,punktfunk-host/vulkan-encode \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session
- name: Build + smoke-boot web console (bun preset) - name: Build + smoke-boot web console (bun preset)
Generated
+14 -27
View File
@@ -870,15 +870,6 @@ dependencies = [
"itertools 0.10.5", "itertools 0.10.5",
] ]
[[package]]
name = "crossbeam-channel"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2"
dependencies = [
"crossbeam-utils",
]
[[package]] [[package]]
name = "crossbeam-deque" name = "crossbeam-deque"
version = "0.8.6" version = "0.8.6"
@@ -2154,7 +2145,7 @@ dependencies = [
[[package]] [[package]]
name = "latency-probe" name = "latency-probe"
version = "0.9.2" version = "0.10.0"
[[package]] [[package]]
name = "lazy_static" name = "lazy_static"
@@ -2286,7 +2277,7 @@ checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]] [[package]]
name = "loss-harness" name = "loss-harness"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"punktfunk-core", "punktfunk-core",
] ]
@@ -2765,7 +2756,7 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]] [[package]]
name = "pf-client-core" name = "pf-client-core"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-channel", "async-channel",
@@ -2787,7 +2778,7 @@ dependencies = [
[[package]] [[package]]
name = "pf-console-ui" name = "pf-console-ui"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"ash", "ash",
@@ -2808,7 +2799,7 @@ dependencies = [
[[package]] [[package]]
name = "pf-ffvk" name = "pf-ffvk"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"ash", "ash",
"bindgen", "bindgen",
@@ -2817,7 +2808,7 @@ dependencies = [
[[package]] [[package]]
name = "pf-presenter" name = "pf-presenter"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"ash", "ash",
@@ -3001,7 +2992,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-client-android" name = "punktfunk-client-android"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"android_logger", "android_logger",
"jni", "jni",
@@ -3017,7 +3008,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-client-linux" name = "punktfunk-client-linux"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-channel", "async-channel",
@@ -3033,7 +3024,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-client-session" name = "punktfunk-client-session"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"pf-client-core", "pf-client-core",
@@ -3048,14 +3039,11 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-client-windows" name = "punktfunk-client-windows"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow",
"async-channel", "async-channel",
"crossbeam-channel",
"ffmpeg-next", "ffmpeg-next",
"mdns-sd", "mdns-sd",
"opus",
"pf-client-core", "pf-client-core",
"punktfunk-core", "punktfunk-core",
"sdl3", "sdl3",
@@ -3063,7 +3051,6 @@ dependencies = [
"serde_json", "serde_json",
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
"wasapi",
"windows 0.62.2 (git+https://github.com/microsoft/windows-rs?rev=a4f7b2cb7c63c6bb7fc77a2affe57145be1d8c4f)", "windows 0.62.2 (git+https://github.com/microsoft/windows-rs?rev=a4f7b2cb7c63c6bb7fc77a2affe57145be1d8c4f)",
"windows-reactor", "windows-reactor",
"windows-reactor-setup", "windows-reactor-setup",
@@ -3072,7 +3059,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-core" name = "punktfunk-core"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"aes-gcm", "aes-gcm",
"bytes", "bytes",
@@ -3103,7 +3090,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-host" name = "punktfunk-host"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"aes", "aes",
"aes-gcm", "aes-gcm",
@@ -3175,7 +3162,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-probe" name = "punktfunk-probe"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"mdns-sd", "mdns-sd",
@@ -3189,7 +3176,7 @@ dependencies = [
[[package]] [[package]]
name = "punktfunk-tray" name = "punktfunk-tray"
version = "0.9.2" version = "0.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"ksni", "ksni",
+1 -1
View File
@@ -35,7 +35,7 @@ exclude = [
ndk = { path = "clients/android/native/vendor/ndk" } ndk = { path = "clients/android/native/vendor/ndk" }
[workspace.package] [workspace.package]
version = "0.9.2" version = "0.10.0"
edition = "2021" edition = "2021"
rust-version = "1.82" rust-version = "1.82"
license = "MIT OR Apache-2.0" license = "MIT OR Apache-2.0"
@@ -158,8 +158,11 @@ fun ControllersScreen(gamepadSetting: Int, onBack: () -> Unit) {
color = MaterialTheme.colorScheme.onSurfaceVariant, color = MaterialTheme.colorScheme.onSurfaceVariant,
) )
} }
pads.forEachIndexed { i, dev -> // Every real controller is forwarded now (Automatic forwards them all, each on its own
PadRow(dev, forwarded = i == 0, gamepadSetting = gamepadSetting) // wire pad index) — not just the first. A joystick-only device Android doesn't classify as
// a gamepad still can't be forwarded (the host wants a gamepad), so gate the badge on it.
pads.forEach { dev ->
PadRow(dev, forwarded = isForwarded(dev), gamepadSetting = gamepadSetting)
} }
} }
@@ -222,8 +225,12 @@ private fun PadRow(dev: InputDevice, forwarded: Boolean, gamepadSetting: Int) {
Row(modifier = Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically) { Row(modifier = Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically) {
Text(dev.name, style = MaterialTheme.typography.bodyLarge, modifier = Modifier.weight(1f)) Text(dev.name, style = MaterialTheme.typography.bodyLarge, modifier = Modifier.weight(1f))
if (forwarded) { if (forwarded) {
// Android's own controller number (1-based; 0 = unassigned), shown so a multi-pad
// user can tell which physical pad is which. The stream's wire pad index is
// assigned separately (lowest-free per device) once streaming starts.
val number = dev.controllerNumber
Text( Text(
"forwarded to host", if (number > 0) "forwarded · player $number" else "forwarded to host",
style = MaterialTheme.typography.labelSmall, style = MaterialTheme.typography.labelSmall,
color = MaterialTheme.colorScheme.primary, color = MaterialTheme.colorScheme.primary,
) )
@@ -319,6 +326,15 @@ private fun Group(title: String, content: @Composable ColumnScope.() -> Unit) {
} }
} }
/**
* Whether this device is actually forwarded to the host — the same rule the stream's [GamepadRouter]
* applies: a real, non-virtual controller whose source classes include GAMEPAD. A joystick-only node
* (e.g. a DualSense motion-sensor sibling, or an adapter that enumerates as bare joystick) shows in
* the list but isn't forwarded.
*/
private fun isForwarded(dev: InputDevice): Boolean =
!dev.isVirtual && dev.sources and InputDevice.SOURCE_GAMEPAD == InputDevice.SOURCE_GAMEPAD
/** Whether the controller reports a rumble motor — via VibratorManager (API 31+) or the legacy Vibrator. */ /** Whether the controller reports a rumble motor — via VibratorManager (API 31+) or the legacy Vibrator. */
private fun deviceHasVibrator(dev: InputDevice): Boolean = private fun deviceHasVibrator(dev: InputDevice): Boolean =
if (Build.VERSION.SDK_INT >= 31) { if (Build.VERSION.SDK_INT >= 31) {
@@ -16,6 +16,7 @@ import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.setValue import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier import androidx.compose.ui.Modifier
import io.unom.punktfunk.kit.Gamepad import io.unom.punktfunk.kit.Gamepad
import io.unom.punktfunk.kit.GamepadRouter
import io.unom.punktfunk.kit.Keymap import io.unom.punktfunk.kit.Keymap
import io.unom.punktfunk.kit.NativeBridge import io.unom.punktfunk.kit.NativeBridge
@@ -27,8 +28,12 @@ class MainActivity : ComponentActivity() {
*/ */
var streamHandle: Long = 0L var streamHandle: Long = 0L
/** Joystick-axis state mapper for the active session (built/reset by StreamScreen). */ /**
var axisMapper: Gamepad.AxisMapper? = null * Multi-controller router for the active session (built/released by StreamScreen): assigns each
* connected pad a stable wire index, threads it onto every event, declares/removes pads on
* hot-plug, and routes rumble/HID feedback back by pad index. Null while not streaming.
*/
var gamepadRouter: GamepadRouter? = null
/** /**
* Input observers for the Controllers debug screen (set while it is shown, like [streamHandle]). * Input observers for the Controllers debug screen (set while it is shown, like [streamHandle]).
@@ -44,9 +49,6 @@ class MainActivity : ComponentActivity() {
*/ */
var requestStreamExit: (() -> Unit)? = null var requestStreamExit: (() -> Unit)? = null
/** Currently-held forwarded pad buttons (bitmask of `Gamepad.BTN_*`), for chord detection. */
private var heldPadButtons = 0
/** /**
* Whether the last console input came from a real gamepad (face buttons / stick) vs. a TV D-pad * Whether the last console input came from a real gamepad (face buttons / stick) vs. a TV D-pad
* remote (which has no A/B/X/Y). The console UI reads this to show glyphs the user recognises — pad * remote (which has no A/B/X/Y). The console UI reads this to show glyphs the user recognises — pad
@@ -125,23 +127,12 @@ class MainActivity : ComponentActivity() {
if (event.isFromSource(InputDevice.SOURCE_GAMEPAD)) { if (event.isFromSource(InputDevice.SOURCE_GAMEPAD)) {
val bit = Gamepad.buttonBit(event.keyCode) val bit = Gamepad.buttonBit(event.keyCode)
if (bit != 0) { if (bit != 0) {
when (event.action) { // The router forwards the bit on this device's own wire pad index, tracks held
// repeatCount guard: don't re-send a held button as auto-repeat. // state per pad, and reports when the emergency-exit chord (Select + Start + L1 +
KeyEvent.ACTION_DOWN -> { // R1) completed on any one pad (a couch user has no keyboard/Back).
if (event.repeatCount == 0) NativeBridge.nativeSendGamepadButton(handle, bit, true) if (gamepadRouter?.onButton(event, bit) == true) {
heldPadButtons = heldPadButtons or bit
// Emergency exit: Select + Start + L1 + R1 held together leaves the stream
// (a couch user has no keyboard/Back). Fired once per full chord.
if (heldPadButtons and STREAM_EXIT_CHORD == STREAM_EXIT_CHORD) {
heldPadButtons = 0
requestStreamExit?.let { exit -> window.decorView.post { exit() } } requestStreamExit?.let { exit -> window.decorView.post { exit() } }
} }
}
KeyEvent.ACTION_UP -> {
NativeBridge.nativeSendGamepadButton(handle, bit, false)
heldPadButtons = heldPadButtons and bit.inv()
}
}
return true // consumed return true // consumed
} }
} }
@@ -203,7 +194,7 @@ class MainActivity : ComponentActivity() {
override fun dispatchGenericMotionEvent(event: MotionEvent): Boolean { override fun dispatchGenericMotionEvent(event: MotionEvent): Boolean {
if (streamHandle != 0L) { if (streamHandle != 0L) {
if (axisMapper?.onMotion(event) == true) return true if (gamepadRouter?.onMotion(event) == true) return true
return super.dispatchGenericMotionEvent(event) return super.dispatchGenericMotionEvent(event)
} }
// The Controllers debug screen sees pad motion before the stick→D-pad synthesis below. // The Controllers debug screen sees pad motion before the stick→D-pad synthesis below.
@@ -248,9 +239,4 @@ class MainActivity : ComponentActivity() {
-> true -> true
else -> KeyEvent.isGamepadButton(kc) else -> KeyEvent.isGamepadButton(kc)
} }
private companion object {
/** Emergency stream-exit chord: Select + Start + L1 + R1 held together. */
val STREAM_EXIT_CHORD = Gamepad.BTN_BACK or Gamepad.BTN_START or Gamepad.BTN_LB or Gamepad.BTN_RB
}
} }
@@ -32,8 +32,8 @@ import androidx.core.content.ContextCompat
import androidx.core.view.WindowCompat import androidx.core.view.WindowCompat
import androidx.core.view.WindowInsetsCompat import androidx.core.view.WindowInsetsCompat
import androidx.core.view.WindowInsetsControllerCompat import androidx.core.view.WindowInsetsControllerCompat
import io.unom.punktfunk.kit.Gamepad
import io.unom.punktfunk.kit.GamepadFeedback import io.unom.punktfunk.kit.GamepadFeedback
import io.unom.punktfunk.kit.GamepadRouter
import io.unom.punktfunk.kit.NativeBridge import io.unom.punktfunk.kit.NativeBridge
import io.unom.punktfunk.kit.VideoDecoders import io.unom.punktfunk.kit.VideoDecoders
import java.util.concurrent.atomic.AtomicBoolean import java.util.concurrent.atomic.AtomicBoolean
@@ -174,18 +174,24 @@ fun StreamScreen(handle: Long, micEnabled: Boolean, onDisconnect: () -> Unit) {
val priorOrientation = activity?.requestedOrientation val priorOrientation = activity?.requestedOrientation
activity?.requestedOrientation = ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE activity?.requestedOrientation = ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE
activity?.streamHandle = handle // route hardware keys to this session activity?.streamHandle = handle // route hardware keys to this session
activity?.axisMapper = Gamepad.AxisMapper(handle) // route joystick axes // Multi-controller router: a stable wire pad index per connected controller, per-device axis
// state, Arrival/Remove on hot-plug, and feedback routed back by pad index. Forwards every
// controller (Automatic). Built here, released on dispose.
val router = GamepadRouter(context, handle, initialSettings.gamepad)
activity?.gamepadRouter = router
// Select+Start+L1+R1 chord leaves the stream — a deliberate quit (signal it so the host skips // Select+Start+L1+R1 chord leaves the stream — a deliberate quit (signal it so the host skips
// the keep-alive linger), unlike a host-ended / backgrounded drop. // the keep-alive linger), unlike a host-ended / backgrounded drop.
activity?.requestStreamExit = { NativeBridge.nativeDisconnectQuit(handle); onDisconnect() } activity?.requestStreamExit = { NativeBridge.nativeDisconnectQuit(handle); onDisconnect() }
activity?.setConsoleHighRefreshRate(false) // let the decoder's setFrameRate pick the panel rate activity?.setConsoleHighRefreshRate(false) // let the decoder's setFrameRate pick the panel rate
// Host→client feedback (rumble + DualSense lightbar/LEDs); poll threads stopped before close. // Host→client feedback (rumble + DualSense lightbar/LEDs), routed to each controller by pad
val feedback = GamepadFeedback(handle).also { it.start() } // index via the router; poll threads stopped + joined before the router is released and the
// session closed.
val feedback = GamepadFeedback(handle, router).also { it.start() }
onDispose { onDispose {
closed.set(true) // from here the handle gets freed; surfaceDestroyed must not touch it closed.set(true) // from here the handle gets freed; surfaceDestroyed must not touch it
feedback.stop() // stop + join the poll threads BEFORE nativeClose frees the handle feedback.stop() // stop + join the poll threads BEFORE the router is released / handle freed
activity?.axisMapper?.reset() // release-all so nothing sticks on the host router.release() // flush every slot (nothing sticks host-side) + drop the hot-plug listener
activity?.axisMapper = null activity?.gamepadRouter = null
activity?.streamHandle = 0L activity?.streamHandle = 0L
activity?.requestStreamExit = null activity?.requestStreamExit = null
activity?.setConsoleHighRefreshRate(true) // back to the console UI's max refresh activity?.setConsoleHighRefreshRate(true) // back to the console UI's max refresh
@@ -171,47 +171,26 @@ object Gamepad {
} }
/** /**
* Maps joystick MotionEvents to axis (+ HAT→dpad) sends for one session, **on change only**. * Maps one controller's joystick MotionEvents to axis (+ HAT→dpad) sends on wire pad index [pad],
* Holds the previous axis/hat state so an unchanged frame emits nothing. One instance per * **on change only**. Holds the previous axis/hat state so an unchanged frame emits nothing. One
* session; call [reset] on release-all (focus loss / disconnect / session stop) so nothing * instance per forwarded controller (owned by [GamepadRouter], which routes each device's events
* sticks on the host (which has no client-side held-state knowledge). * to its own mapper so a second pad can't clobber the first); call [reset] on that slot closing
* (disconnect / session stop) so nothing sticks on the host (which has no client-side held-state
* knowledge).
* *
* Single-source: only ONE qualifying controller feeds pad 0. Events must come from a device * The router only ever feeds this a qualifying event from the mapper's own device — a real
* whose source classes include GAMEPAD (see [onMotion]) and the mapper pins itself to the * gamepad (its source classes include GAMEPAD), never a controller's joystick-classified sibling
* first such device — a controller's joystick-classified sibling nodes (DualSense/DS4 motion * node (DualSense/DS4 motion sensors), which reports every pad axis as 0. [onMotion] therefore
* sensors) and any second pad report every axis as 0, and folding them into the same state * folds the event straight in without re-qualifying it.
* flapped a held trigger/stick between its value and 0 on every event interleave.
*/ */
class AxisMapper(private val handle: Long) { class AxisMapper(private val handle: Long, private val pad: Int) {
// Sentinel so the first real value (incl. 0) always sends once after attach (Linux parity). // Sentinel so the first real value (incl. 0) always sends once after attach (Linux parity).
private val last = IntArray(6) { Int.MIN_VALUE } private val last = IntArray(6) { Int.MIN_VALUE }
private var hatX = 0 // -1 / 0 / +1 private var hatX = 0 // -1 / 0 / +1
private var hatY = 0 private var hatY = 0
/** deviceId of the controller pad 0 is pinned to; 1 until the first qualifying event. */ /** Fold one joystick ACTION_MOVE from this mapper's controller onto its pad index. */
private var deviceId = -1 fun onMotion(event: MotionEvent) {
/** Returns true if this was a joystick ACTION_MOVE we consumed. */
fun onMotion(event: MotionEvent): Boolean {
if (!event.isFromSource(InputDevice.SOURCE_JOYSTICK)) return false
if (event.actionMasked != MotionEvent.ACTION_MOVE) return false
// Only a true gamepad drives pad 0. A joystick ACTION_MOVE's own source is plain
// JOYSTICK for every sender, so qualify by the DEVICE's source classes: a real pad
// carries the GAMEPAD (button) class too, its sensor/touchpad sibling nodes and
// joystick-class remotes don't — and those report every pad axis as 0 (see the
// class doc for the held-trigger flap this caused).
val dev = event.device ?: return false
if (dev.sources and InputDevice.SOURCE_GAMEPAD != InputDevice.SOURCE_GAMEPAD) return false
// Single-pad model: pin to the first qualifying controller so a second pad (or its
// stick drift) can't fight pad 0; re-adopt only once the pinned device is gone.
if (deviceId != event.deviceId) {
if (deviceId != -1) {
if (InputDevice.getDevice(deviceId) != null) return false
reset() // the pinned pad is gone — lift its held state before adopting
}
deviceId = event.deviceId
}
// Sticks: Android floats 1..1, +y = down → ±32767, negate Y for the wire's +y = up. // Sticks: Android floats 1..1, +y = down → ±32767, negate Y for the wire's +y = up.
sendAxis(AXIS_LS_X, stick(event.getAxisValue(MotionEvent.AXIS_X))) sendAxis(AXIS_LS_X, stick(event.getAxisValue(MotionEvent.AXIS_X)))
sendAxis(AXIS_LS_Y, stick(-event.getAxisValue(MotionEvent.AXIS_Y))) sendAxis(AXIS_LS_Y, stick(-event.getAxisValue(MotionEvent.AXIS_Y)))
@@ -253,10 +232,9 @@ object Gamepad {
if (hy < 0) btn(BTN_DPAD_UP, true) else if (hy > 0) btn(BTN_DPAD_DOWN, true) if (hy < 0) btn(BTN_DPAD_UP, true) else if (hy > 0) btn(BTN_DPAD_DOWN, true)
hatY = hy hatY = hy
} }
return true
} }
/** Release-all: zero every axis and clear the held dpad. */ /** Release-all: zero every axis and clear the held dpad (all on this mapper's pad index). */
fun reset() { fun reset() {
for (id in 0..5) sendAxis(id, 0) for (id in 0..5) sendAxis(id, 0)
if (hatX < 0) btn(BTN_DPAD_LEFT, false) else if (hatX > 0) btn(BTN_DPAD_RIGHT, false) if (hatX < 0) btn(BTN_DPAD_LEFT, false) else if (hatX > 0) btn(BTN_DPAD_RIGHT, false)
@@ -268,10 +246,10 @@ object Gamepad {
private fun sendAxis(id: Int, v: Int) { private fun sendAxis(id: Int, v: Int) {
if (last[id] == v) return if (last[id] == v) return
last[id] = v last[id] = v
NativeBridge.nativeSendGamepadAxis(handle, id, v) NativeBridge.nativeSendGamepadAxis(handle, id, v, pad)
} }
private fun btn(bit: Int, down: Boolean) = NativeBridge.nativeSendGamepadButton(handle, bit, down) private fun btn(bit: Int, down: Boolean) = NativeBridge.nativeSendGamepadButton(handle, bit, down, pad)
// 1..1 float → ±32767 i16 (matches the Apple client's 32767 scale). // 1..1 float → ±32767 i16 (matches the Apple client's 32767 scale).
private fun stick(v: Float): Int = (v.coerceIn(-1f, 1f) * 32767f).toInt() private fun stick(v: Float): Int = (v.coerceIn(-1f, 1f) * 32767f).toInt()
@@ -15,21 +15,26 @@ import android.view.InputDevice
import java.nio.ByteBuffer import java.nio.ByteBuffer
/** /**
* Host→client gamepad feedback for one session (single-pad model — pad 0 only). Two daemon poll * Host→client gamepad feedback for one session, routed per controller by wire pad index. Two daemon
* threads drain the blocking native pulls and render in Kotlin: rumble → the controller's * poll threads drain the blocking native pulls and render in Kotlin: rumble → the addressed
* `VibratorManager` (API 31+) or its single legacy `Vibrator` on API 2830; HID-output → lightbar / * controller's `VibratorManager` (API 31+) or its single legacy `Vibrator` on API 2830; HID-output
* player-LED via `LightsManager` (API 33+); adaptive * → that controller's lightbar / player-LED via `LightsManager` (API 33+); adaptive triggers are
* triggers are parse-validated and logged (Android has no public adaptive-trigger API). * parse-validated and logged (Android has no public adaptive-trigger API).
*
* Each pull carries the wire pad index it is addressed to; [GamepadRouter.deviceForPad] resolves it
* to the physical controller currently holding that index — so a rumble the host aimed at pad 1
* drives pad 1's motors, and an update for an index with no live controller (a pad that just
* unplugged) is dropped. Per-controller rumble/light bindings are built lazily and cached by device
* id (bounded — at most 16 pads).
* *
* Mirrors `nativeStartAudio`'s lifecycle: [start]/[stop] driven by the StreamScreen. [stop] flips a * Mirrors `nativeStartAudio`'s lifecycle: [start]/[stop] driven by the StreamScreen. [stop] flips a
* flag; the ~100 ms native pull timeout lets the threads exit, then they're joined (bounded) — and * flag; the ~100 ms native pull timeout lets the threads exit, then they're joined (bounded) — and
* this MUST run before `nativeClose` frees the session handle. * this MUST run before the router is released and `nativeClose` frees the session handle.
* *
* The active pad is resolved from the connected input devices (first gamepad/joystick). With none * With no controller connected (emulator) rumble/lights become logged no-ops — exactly the
* connected (emulator) rumble/lights become logged no-ops — exactly the verification path; the * verification path; the `Log.i` receipt lines fire regardless of rendering hardware.
* `Log.i` receipt lines fire regardless of rendering hardware.
*/ */
class GamepadFeedback(private val handle: Long) { class GamepadFeedback(private val handle: Long, private val router: GamepadRouter?) {
private companion object { private companion object {
const val TAG = "pf.feedback" const val TAG = "pf.feedback"
const val TAG_LED: Byte = 0x01 const val TAG_LED: Byte = 0x01
@@ -40,42 +45,48 @@ class GamepadFeedback(private val handle: Long) {
const val LEGACY_RUMBLE_MS = 60_000L const val LEGACY_RUMBLE_MS = 60_000L
} }
/** One controller's rumble binding — VibratorManager (API 31+) OR the legacy single Vibrator (API 2830). */
private class RumbleBind(
val vm: VibratorManager?,
val legacy: Vibrator?,
val ids: IntArray,
val amplitudeControlled: Boolean,
)
/** One controller's lights binding (API 33+): its open session + the RGB / player-id lights it exposes. */
private class LightBind(
val session: LightsManager.LightsSession,
val rgb: Light?,
val player: Light?,
)
@Volatile private var running = false @Volatile private var running = false
private var rumbleThread: Thread? = null private var rumbleThread: Thread? = null
private var hidoutThread: Thread? = null private var hidoutThread: Thread? = null
private var vm: VibratorManager? = null // Per-controller bindings, keyed by device id, built lazily. rumbleBinds is touched ONLY by the
// API 2830 fallback: the controller's single legacy Vibrator (no per-motor VibratorManager // rumble thread and lightBinds ONLY by the hidout thread while running; stop() reads both from the
// until API 31). Exactly one of [vm] / [legacy] is bound; rumble degrades to one blended motor. // main thread AFTER joining those threads (join establishes the happens-before), so plain maps are
private var legacy: Vibrator? = null // race-free. A null value caches "this controller has no vibrator / no controllable lights".
private var vibratorIds: IntArray = IntArray(0) private val rumbleBinds = HashMap<Int, RumbleBind?>()
private var amplitudeControlled = false private val lightBinds = HashMap<Int, LightBind?>()
private var lightsSession: LightsManager.LightsSession? = null
private var rgbLight: Light? = null
private var playerLight: Light? = null
fun start() { fun start() {
val dev = resolvePad()
bindRumble(dev)
if (Build.VERSION.SDK_INT >= 33) {
bindLights(dev)
} else {
Log.i(TAG, "lights need API 33 (have ${Build.VERSION.SDK_INT}) — lightbar/playerLed no-op")
}
running = true running = true
rumbleThread = Thread({ rumbleThread = Thread({
while (running) { while (running) {
val ev = NativeBridge.nativeNextRumble(handle) val ev = NativeBridge.nativeNextRumble(handle)
if (ev < 0L) continue // timeout / closed if (ev < 0L) continue // timeout / closed
// ev bit 48 = has a v2 lease; bits 32..47 = ttl_ms; 16..31 = low; 0..15 = high. The // ev bits 49..52 = wire pad index; bit 48 = has a v2 lease; bits 32..47 = ttl_ms;
// lease flag is out-of-band, so any ttl_ms (incl. 0xFFFF) is a real lease — no // 16..31 = low; 0..15 = high. The lease flag is out-of-band, so any ttl_ms (incl.
// in-band sentinel. No lease (legacy host) → the prior long one-shot. // 0xFFFF) is a real lease — no in-band sentinel. No lease (legacy host) → the prior
// long one-shot.
val pad = ((ev ushr 49) and 0xFL).toInt()
val hasLease = ((ev ushr 48) and 0x1L) == 0x1L val hasLease = ((ev ushr 48) and 0x1L) == 0x1L
val ttl = ((ev ushr 32) and 0xFFFF).toInt() val ttl = ((ev ushr 32) and 0xFFFF).toInt()
val durationMs = if (hasLease) ttl.toLong() else LEGACY_RUMBLE_MS val durationMs = if (hasLease) ttl.toLong() else LEGACY_RUMBLE_MS
renderRumble( renderRumble(
pad,
((ev ushr 16) and 0xFFFF).toInt(), ((ev ushr 16) and 0xFFFF).toInt(),
(ev and 0xFFFF).toInt(), (ev and 0xFFFF).toInt(),
durationMs, durationMs,
@@ -93,100 +104,99 @@ class GamepadFeedback(private val handle: Long) {
}, "pf-hidout").apply { isDaemon = true; start() } }, "pf-hidout").apply { isDaemon = true; start() }
} }
/** Idempotent. Stops + joins the poll threads (must complete before the session handle is freed). */ /** Idempotent. Stops + joins the poll threads (must complete before the router is released / handle freed). */
fun stop() { fun stop() {
running = false running = false
rumbleThread?.interrupt() rumbleThread?.interrupt()
hidoutThread?.interrupt() hidoutThread?.interrupt()
runCatching { vm?.cancel() } // drop any held rumble immediately
runCatching { legacy?.cancel() }
// Join WITHOUT a timeout. These poll threads dereference the native session handle on every // Join WITHOUT a timeout. These poll threads dereference the native session handle on every
// pull (nativeNextRumble/nativeNextHidout), so they MUST be dead before StreamScreen's // pull (nativeNextRumble/nativeNextHidout) and read the router, so they MUST be dead before
// onDispose reaches nativeClose, which frees that handle. A *bounded* join that times out // StreamScreen's onDispose reaches router.release() / nativeClose, which free that state. A
// would let a thread survive into the freed handle → use-after-free SIGSEGV (the // *bounded* join that times out would let a thread survive into the freed handle → use-after-
// back-while-streaming crash, on the one path the main-thread `closed` guard can't cover). // free SIGSEGV (the back-while-streaming crash, on the one path the main-thread `closed` guard
// Safe to block unbounded: the native pulls are internally time-bounded (PULL_TIMEOUT ~100 ms) // can't cover). Safe to block unbounded: the native pulls are internally time-bounded
// and rendering is a quick best-effort binder call, so each thread observes running=false and // (PULL_TIMEOUT ~100 ms) and rendering is a quick best-effort binder call, so each thread
// exits within ~one timeout — the join returns promptly (well under any ANR threshold). // observes running=false and exits within ~one timeout — the join returns promptly.
runCatching { rumbleThread?.join() } runCatching { rumbleThread?.join() }
runCatching { hidoutThread?.join() } runCatching { hidoutThread?.join() }
rumbleThread = null rumbleThread = null
hidoutThread = null hidoutThread = null
runCatching { lightsSession?.close() } // Threads are dead — drop any held rumble and close every lights session.
lightsSession = null for (b in rumbleBinds.values) b?.let {
rgbLight = null runCatching { it.vm?.cancel() }
playerLight = null runCatching { it.legacy?.cancel() }
vm = null }
legacy = null for (b in lightBinds.values) b?.let { runCatching { it.session.close() } }
vibratorIds = IntArray(0) rumbleBinds.clear()
lightBinds.clear()
} }
/** First connected gamepad/joystick InputDevice, or null (→ logged no-op on the emulator). */
private fun resolvePad(): InputDevice? = Gamepad.firstPad()
// ---- Rumble ---- // ---- Rumble ----
private fun bindRumble(dev: InputDevice?) { /** The rumble binding for the controller on wire pad [pad], or null (no live pad / no vibrator). Cached by device id. */
if (dev == null) { private fun rumbleBindFor(pad: Int): RumbleBind? {
Log.i(TAG, "rumble: no controller connected — rumble no-op (emulator path)") val dev = router?.deviceForPad(pad) ?: return null
return if (rumbleBinds.containsKey(dev.id)) return rumbleBinds[dev.id]
val bind = bindRumble(dev)
rumbleBinds[dev.id] = bind
return bind
} }
private fun bindRumble(dev: InputDevice): RumbleBind? {
if (Build.VERSION.SDK_INT >= 31) { if (Build.VERSION.SDK_INT >= 31) {
val m = dev.vibratorManager val m = dev.vibratorManager
val ids = m.vibratorIds val ids = m.vibratorIds
if (ids.isEmpty()) { if (ids.isEmpty()) {
Log.i(TAG, "rumble: controller '${dev.name}' has no vibrators — rumble no-op") Log.i(TAG, "rumble: controller '${dev.name}' has no vibrators — rumble no-op")
return return null
}
val amp = ids.all { m.getVibrator(it).hasAmplitudeControl() }
Log.i(TAG, "rumble: bound ${ids.size} vibrators for '${dev.name}' amplitudeControl=$amp")
return RumbleBind(m, null, ids, amp)
} }
vm = m
vibratorIds = ids
amplitudeControlled = ids.all { m.getVibrator(it).hasAmplitudeControl() }
Log.i(TAG, "rumble: bound ${ids.size} vibrators amplitudeControl=$amplitudeControlled")
} else {
// API 2830: no VibratorManager — fall back to the controller's single legacy Vibrator. // API 2830: no VibratorManager — fall back to the controller's single legacy Vibrator.
@Suppress("DEPRECATION") @Suppress("DEPRECATION")
val v = dev.vibrator val v = dev.vibrator
if (!v.hasVibrator()) { if (!v.hasVibrator()) {
Log.i(TAG, "rumble: controller '${dev.name}' has no vibrator — rumble no-op") Log.i(TAG, "rumble: controller '${dev.name}' has no vibrator — rumble no-op")
return return null
}
legacy = v
amplitudeControlled = v.hasAmplitudeControl()
Log.i(TAG, "rumble: bound legacy vibrator amplitudeControl=$amplitudeControlled")
} }
Log.i(TAG, "rumble: bound legacy vibrator for '${dev.name}' amplitudeControl=${v.hasAmplitudeControl()}")
return RumbleBind(null, v, IntArray(0), v.hasAmplitudeControl())
} }
/** /**
* low = heavy/left motor, high = light/right motor; both 0..0xFFFF (the host's u16 amplitudes). * low = heavy/left motor, high = light/right motor; both 0..0xFFFF (the host's u16 amplitudes),
* `durationMs` is the host's v2 envelope TTL — the one-shot self-terminates after it unless the * addressed to wire pad [pad]. `durationMs` is the host's v2 envelope TTL — the one-shot self-
* host renews, so a lost stop (or a dead host) silences at the lease instead of the old fixed * terminates after it unless the host renews, so a lost stop (or a dead host) silences at the
* 60 s. Against a legacy host it is [LEGACY_RUMBLE_MS] (the prior fixed duration). * lease instead of the old fixed 60 s. Against a legacy host it is [LEGACY_RUMBLE_MS].
*/ */
private fun renderRumble(low: Int, high: Int, durationMs: Long) { private fun renderRumble(pad: Int, low: Int, high: Int, durationMs: Long) {
Log.i(TAG, "rumble low=$low high=$high ttlMs=$durationMs") // verification line — BEFORE any no-op return Log.i(TAG, "rumble pad=$pad low=$low high=$high ttlMs=$durationMs") // verification line — BEFORE any no-op return
val bind = rumbleBindFor(pad) ?: return
val lo = toAmplitude(low) val lo = toAmplitude(low)
val hi = toAmplitude(high) val hi = toAmplitude(high)
val m = vm val m = bind.vm
if (m != null) { if (m != null) {
if (lo == 0 && hi == 0) { if (lo == 0 && hi == 0) {
m.cancel() // (0,0) = stop m.cancel() // (0,0) = stop
return return
} }
val combo = CombinedVibration.startParallel() val combo = CombinedVibration.startParallel()
if (amplitudeControlled && vibratorIds.size >= 2) { if (bind.amplitudeControlled && bind.ids.size >= 2) {
// ids[0] = light/right, ids[1] = heavy/left (XInput/Moonlight convention). // ids[0] = light/right, ids[1] = heavy/left (XInput/Moonlight convention).
if (hi != 0) combo.addVibrator(vibratorIds[0], oneShot(hi, durationMs)) if (hi != 0) combo.addVibrator(bind.ids[0], oneShot(hi, durationMs))
if (lo != 0) combo.addVibrator(vibratorIds[1], oneShot(lo, durationMs)) if (lo != 0) combo.addVibrator(bind.ids[1], oneShot(lo, durationMs))
} else { } else {
// Single motor or no amplitude control: blend both into one effect. // Single motor or no amplitude control: blend both into one effect.
val a = (lo * 0.8 + hi * 0.33).toInt().coerceIn(1, 255) val a = (lo * 0.8 + hi * 0.33).toInt().coerceIn(1, 255)
for (id in vibratorIds) combo.addVibrator(id, oneShot(a, durationMs)) for (id in bind.ids) combo.addVibrator(id, oneShot(a, durationMs))
} }
runCatching { m.vibrate(combo.combine()) } runCatching { m.vibrate(combo.combine()) }
return return
} }
// API 2830 legacy single-motor path: blend both motors into one effect. // API 2830 legacy single-motor path: blend both motors into one effect.
val lv = legacy ?: return val lv = bind.legacy ?: return
if (lo == 0 && hi == 0) { if (lo == 0 && hi == 0) {
lv.cancel() // (0,0) = stop lv.cancel() // (0,0) = stop
return return
@@ -194,7 +204,7 @@ class GamepadFeedback(private val handle: Long) {
val a = (lo * 0.8 + hi * 0.33).toInt().coerceIn(1, 255) val a = (lo * 0.8 + hi * 0.33).toInt().coerceIn(1, 255)
runCatching { runCatching {
lv.vibrate( lv.vibrate(
if (amplitudeControlled) oneShot(a, durationMs) if (bind.amplitudeControlled) oneShot(a, durationMs)
else oneShot(VibrationEffect.DEFAULT_AMPLITUDE, durationMs) else oneShot(VibrationEffect.DEFAULT_AMPLITUDE, durationMs)
) )
} }
@@ -215,28 +225,29 @@ class GamepadFeedback(private val handle: Long) {
private fun dispatchHidout(buf: ByteBuffer, n: Int) { private fun dispatchHidout(buf: ByteBuffer, n: Int) {
buf.rewind() buf.rewind()
val pad = buf.get().toInt() and 0xFF // wire pad index the event is addressed to
when (buf.get()) { // kind tag when (buf.get()) { // kind tag
TAG_LED -> { TAG_LED -> {
val r = buf.get().toInt() and 0xFF val r = buf.get().toInt() and 0xFF
val g = buf.get().toInt() and 0xFF val g = buf.get().toInt() and 0xFF
val b = buf.get().toInt() and 0xFF val b = buf.get().toInt() and 0xFF
Log.i(TAG, "hidout Led r=$r g=$g b=$b") // verification line Log.i(TAG, "hidout pad=$pad Led r=$r g=$g b=$b") // verification line
if (Build.VERSION.SDK_INT >= 33) setLightbar(Color.rgb(r, g, b)) if (Build.VERSION.SDK_INT >= 33) setLightbar(pad, Color.rgb(r, g, b))
} }
TAG_PLAYER_LEDS -> { TAG_PLAYER_LEDS -> {
val bits = buf.get().toInt() and 0x1F val bits = buf.get().toInt() and 0x1F
val player = playerIndexForBits(bits) val player = playerIndexForBits(bits)
Log.i(TAG, "hidout PlayerLeds bits=$bits player=$player") // verification line Log.i(TAG, "hidout pad=$pad PlayerLeds bits=$bits player=$player") // verification line
if (Build.VERSION.SDK_INT >= 33) setPlayerId(player) if (Build.VERSION.SDK_INT >= 33) setPlayerId(pad, player)
} }
TAG_TRIGGER -> { TAG_TRIGGER -> {
val which = buf.get().toInt() and 0xFF // 0 = L2, 1 = R2 val which = buf.get().toInt() and 0xFF // 0 = L2, 1 = R2
val effLen = n - 2 val effLen = n - 3 // [pad][kind][which] header, then the effect block
val mode = if (effLen > 0) buf.get().toInt() and 0xFF else 0 val mode = if (effLen > 0) buf.get().toInt() and 0xFF else 0
// No public adaptive-trigger API on Android — parse-validate the mode + log only. // No public adaptive-trigger API on Android — parse-validate the mode + log only.
Log.i( Log.i(
TAG, TAG,
"hidout Trigger which=$which effLen=$effLen mode=0x%02x (adaptive triggers unsupported on Android)".format(mode), "hidout pad=$pad Trigger which=$which effLen=$effLen mode=0x%02x (adaptive triggers unsupported on Android)".format(mode),
) )
} }
else -> Log.d(TAG, "hidout: unknown kind, dropped") else -> Log.d(TAG, "hidout: unknown kind, dropped")
@@ -253,37 +264,46 @@ class GamepadFeedback(private val handle: Long) {
else -> Integer.bitCount(bits and 0x1F).coerceIn(1, 4) else -> Integer.bitCount(bits and 0x1F).coerceIn(1, 4)
} }
private fun bindLights(dev: InputDevice?) { /** The lights binding for the controller on wire pad [pad], or null (no live pad / no lights / < API 33). Cached by device id. */
if (dev == null) { private fun lightBindFor(pad: Int): LightBind? {
Log.i(TAG, "lights: no controller connected — lightbar/playerLed no-op (emulator path)") if (Build.VERSION.SDK_INT < 33) return null
return val dev = router?.deviceForPad(pad) ?: return null
if (lightBinds.containsKey(dev.id)) return lightBinds[dev.id]
val bind = bindLights(dev)
lightBinds[dev.id] = bind
return bind
} }
private fun bindLights(dev: InputDevice): LightBind? {
val lm = dev.lightsManager val lm = dev.lightsManager
var rgb: Light? = null
var player: Light? = null
for (l in lm.lights) { for (l in lm.lights) {
if (rgbLight == null && l.hasRgbControl()) rgbLight = l if (rgb == null && l.hasRgbControl()) rgb = l
if (playerLight == null && l.type == Light.LIGHT_TYPE_PLAYER_ID) playerLight = l if (player == null && l.type == Light.LIGHT_TYPE_PLAYER_ID) player = l
} }
if (rgbLight == null && playerLight == null) { if (rgb == null && player == null) {
Log.i(TAG, "lights: controller '${dev.name}' exposes no controllable lights — no-op") Log.i(TAG, "lights: controller '${dev.name}' exposes no controllable lights — no-op")
return return null
} }
lightsSession = lm.openSession() val session = lm.openSession()
Log.i(TAG, "lights: bound rgb=${rgbLight != null} playerLed=${playerLight != null}") Log.i(TAG, "lights: bound rgb=${rgb != null} playerLed=${player != null} for '${dev.name}'")
return LightBind(session, rgb, player)
} }
private fun setLightbar(argb: Int) { private fun setLightbar(pad: Int, argb: Int) {
val s = lightsSession ?: return val bind = lightBindFor(pad) ?: return
val l = rgbLight ?: return val l = bind.rgb ?: return
runCatching { runCatching {
s.requestLights(LightsRequest.Builder().addLight(l, LightState.Builder().setColor(argb).build()).build()) bind.session.requestLights(LightsRequest.Builder().addLight(l, LightState.Builder().setColor(argb).build()).build())
} }
} }
private fun setPlayerId(player: Int) { private fun setPlayerId(pad: Int, player: Int) {
val s = lightsSession ?: return val bind = lightBindFor(pad) ?: return
val l = playerLight ?: return val l = bind.player ?: return
runCatching { runCatching {
s.requestLights(LightsRequest.Builder().addLight(l, LightState.Builder().setPlayerId(player).build()).build()) bind.session.requestLights(LightsRequest.Builder().addLight(l, LightState.Builder().setPlayerId(player).build()).build())
} }
} }
} }
@@ -0,0 +1,204 @@
package io.unom.punktfunk.kit
import android.content.Context
import android.hardware.input.InputManager
import android.os.Handler
import android.os.Looper
import android.view.InputDevice
import android.view.KeyEvent
import android.view.MotionEvent
import java.util.concurrent.ConcurrentHashMap
/**
* Multi-controller router for one stream session — the Android analogue of the Linux client's gamepad
* `Worker`/`Slot` model (`pf-client-core/src/gamepad.rs`) over the shared native-plane wire contract
* (`punktfunk-core/src/input.rs`). Each physical controller (Android `deviceId`) gets a STABLE
* lowest-free wire pad index (0..15) held for its lifetime and freed only on disconnect, so a pad
* dropping never renumbers the others (a game must not see its players shuffle). Every forwarded event
* carries that pad index; a [NativeBridge.nativeSendGamepadArrival] declaring the pad's type is sent
* once BEFORE its first input, a [NativeBridge.nativeSendGamepadRemove] on disconnect. Per-device axis
* state lives in each slot's [Gamepad.AxisMapper] so a second controller can't clobber the first.
* Feedback (rumble / HID) is routed BACK to the originating device by pad index via [deviceForPad].
*
* Selection: forward EVERY real controller (the Linux client's single-player pin has no Android UI
* surface yet — Automatic is the only mode). Lifetime matches the session: constructed on stream
* attach (opening a slot for every already-connected pad, so its Arrival lands before any input),
* released on detach.
*
* A single controller lands on wire index 0, so its per-transition button/axis wire is byte-identical
* to the old single-pad path (plus the Arrival/Remove declarations the contract requires — which an
* older host simply ignores).
*
* Threading: slot mutation + dispatch run on the main thread (Android input dispatch and the
* InputManager hot-plug callbacks both land there). [deviceForPad] is read from the feedback poll
* threads, so the slot table is a [ConcurrentHashMap].
*/
class GamepadRouter(context: Context, private val handle: Long, private val setting: Int) {
/** One forwarded controller: its stable wire pad index, per-device axis state, and held buttons. */
private class Slot(val index: Int, val mapper: Gamepad.AxisMapper) {
/** Forwarded button bits currently held (Gamepad.BTN_*) — for release-on-close + chord detection. */
var held = 0
}
/** deviceId → slot. Concurrent: the feedback poll threads read it via [deviceForPad]. */
private val slots = ConcurrentHashMap<Int, Slot>()
private val inputManager = context.getSystemService(InputManager::class.java)
private val listener = object : InputManager.InputDeviceListener {
override fun onInputDeviceAdded(deviceId: Int) {
InputDevice.getDevice(deviceId)?.let { if (isForwardable(it)) openSlot(it) }
}
override fun onInputDeviceRemoved(deviceId: Int) = closeSlot(deviceId)
override fun onInputDeviceChanged(deviceId: Int) {}
}
init {
inputManager?.registerInputDeviceListener(listener, Handler(Looper.getMainLooper()))
// Open a slot for every controller already connected when the session starts — the pads that
// will never fire onInputDeviceAdded during this session; their Arrival lands before any input.
for (id in InputDevice.getDeviceIds()) {
InputDevice.getDevice(id)?.let { if (isForwardable(it)) openSlot(it) }
}
}
/**
* One gamepad button transition for the device that produced [event] (already resolved to BTN_*
* bit [bit]). Opens the device's slot (declaring its type) if unseen, forwards the bit on the
* slot's pad index, tracks held state, and returns true when this press completed the emergency
* stream-exit chord (Select + Start + L1 + R1) on THIS pad — the caller then leaves the stream
* (mirrors the Linux client's escape chord: any one controller can leave).
*/
fun onButton(event: KeyEvent, bit: Int): Boolean {
val slot = slotFor(event.device) ?: return false
when (event.action) {
KeyEvent.ACTION_DOWN -> {
// repeatCount guard: don't re-send a held button as auto-repeat.
if (event.repeatCount == 0) NativeBridge.nativeSendGamepadButton(handle, bit, true, slot.index)
slot.held = slot.held or bit
if (slot.held and EXIT_CHORD == EXIT_CHORD) {
slot.held = 0
return true
}
}
KeyEvent.ACTION_UP -> {
NativeBridge.nativeSendGamepadButton(handle, bit, false, slot.index)
slot.held = slot.held and bit.inv()
}
}
return false
}
/**
* One joystick MotionEvent — routed to the producing device's own [Gamepad.AxisMapper] (per-device
* state). Returns true if consumed. Only a real gamepad drives a pad: a DualSense/DS4 motion-sensor
* sibling node classifies as bare joystick (no GAMEPAD source class) and reports every pad axis as
* 0, so [isForwardable] filters it out before it can open a slot or clobber axes.
*/
fun onMotion(event: MotionEvent): Boolean {
if (!event.isFromSource(InputDevice.SOURCE_JOYSTICK)) return false
if (event.actionMasked != MotionEvent.ACTION_MOVE) return false
val dev = event.device ?: return false
if (!isForwardable(dev)) return false
val slot = slotFor(dev) ?: return false
slot.mapper.onMotion(event)
return true
}
/**
* The controller currently mapped to wire pad [pad], for feedback routing; null if that index
* holds no live slot (a pad that just unplugged — the update is then dropped). Read from the
* feedback poll threads.
*/
fun deviceForPad(pad: Int): InputDevice? {
for ((deviceId, slot) in slots) {
if (slot.index == pad) return InputDevice.getDevice(deviceId)
}
return null
}
/**
* Flush + drop every slot and unregister the hot-plug listener. Call on session teardown, AFTER
* the feedback poll threads are joined (they read [deviceForPad]).
*/
fun release() {
inputManager?.unregisterInputDeviceListener(listener)
// Snapshot the ids first — closeSlot mutates the map.
for (id in slots.keys.toList()) closeSlot(id)
}
// ---- slots ----
/** A real, non-virtual controller we forward — its source classes include GAMEPAD (excludes a pad's bare-joystick sensor node). */
private fun isForwardable(dev: InputDevice): Boolean =
!dev.isVirtual && dev.sources and InputDevice.SOURCE_GAMEPAD == InputDevice.SOURCE_GAMEPAD
/**
* The slot for [dev], opening one (and declaring the pad) if this device is unseen; null when [dev]
* isn't a forwardable controller or every wire index is taken. The [isForwardable] gate lives here —
* the single lazy-open chokepoint both [onButton] and [onMotion] funnel through — so no entry point
* can open a phantom slot for a virtual/non-gamepad source (the hot-plug listener and init loop
* pre-filter and call [openSlot] directly).
*/
private fun slotFor(dev: InputDevice?): Slot? {
if (dev == null) return null
slots[dev.id]?.let { return it }
if (!isForwardable(dev)) return null
return openSlot(dev)
}
/**
* Open a slot for [dev] on the lowest free wire index, declaring its kind ([NativeBridge.nativeSendGamepadArrival])
* before any input so the host builds a matching virtual device (mixed types across pads).
* Idempotent; null when all 16 wire indices are already forwarded.
*/
private fun openSlot(dev: InputDevice): Slot? {
slots[dev.id]?.let { return it }
val index = lowestFreeIndex() ?: return null // 16 pads already forwarded — drop this one
// Automatic resolves the pad's type from its VID/PID; an explicit setting forces every pad
// to that type (a single global choice — matches the handshake's session-default pref).
val pref = if (setting == Gamepad.PREF_AUTO) Gamepad.prefFor(dev) else setting
NativeBridge.nativeSendGamepadArrival(handle, pref, index)
val slot = Slot(index, Gamepad.AxisMapper(handle, index))
slots[dev.id] = slot
return slot
}
/**
* Flush a slot's held wire state (so nothing sticks host-side), signal the removal, and free its
* index. Safe against an already-gone device — the flush emits wire events only, no device access.
*/
private fun closeSlot(deviceId: Int) {
val slot = slots.remove(deviceId) ?: return
releaseHeld(slot)
NativeBridge.nativeSendGamepadRemove(handle, slot.index)
}
/** Lift every held button + zero the axes/HAT dpad for [slot] (wire events only, all on its index). */
private fun releaseHeld(slot: Slot) {
var bits = slot.held
while (bits != 0) {
val bit = bits and -bits // lowest set bit
NativeBridge.nativeSendGamepadButton(handle, bit, false, slot.index)
bits = bits and bit.inv()
}
slot.held = 0
slot.mapper.reset() // zero sticks/triggers + release the HAT dpad
}
/** Lowest wire index 0..[MAX_PADS) not held by a slot, or null when full — stable lowest-free keeps indices from shuffling on hot-plug. */
private fun lowestFreeIndex(): Int? {
val taken = slots.values.mapTo(HashSet()) { it.index }
for (i in 0 until MAX_PADS) if (i !in taken) return i
return null
}
private companion object {
/** Mirror of `punktfunk-core::input::MAX_PADS` — wire pad indices 0..15. */
const val MAX_PADS = 16
/** Emergency stream-exit chord: Select + Start + L1 + R1 held together (matches the legacy single-pad chord). */
const val EXIT_CHORD = Gamepad.BTN_BACK or Gamepad.BTN_START or Gamepad.BTN_LB or Gamepad.BTN_RB
}
}
@@ -269,26 +269,43 @@ object NativeBridge {
/** One key transition. vk: Windows VK (0 = dropped by Rust). mods: VK modifier mask (0 for now). */ /** One key transition. vk: Windows VK (0 = dropped by Rust). mods: VK modifier mask (0 for now). */
external fun nativeSendKey(handle: Long, vk: Int, down: Boolean, mods: Int) external fun nativeSendKey(handle: Long, vk: Int, down: Boolean, mods: Int)
// ---- Gamepad: one pad forwarded as pad 0 (Rust hardcodes flags=0) ---- // ---- Gamepad: each controller forwarded on its own wire pad index (0..15, low byte of flags) ----
// The pad index is assigned per Android device by GamepadRouter; a single controller lands on 0,
// so its wire is byte-identical to the old single-pad path. The core folds the per-transition
// events into seq'd GamepadState snapshots keyed on this index and owns the per-pad seq.
/** One gamepad button transition. bit: a [Gamepad].BTN_* bit. down: press/release. */ /** One gamepad button transition on wire pad [pad] (0..15). bit: a [Gamepad].BTN_* bit. down: press/release. */
external fun nativeSendGamepadButton(handle: Long, bit: Int, down: Boolean) external fun nativeSendGamepadButton(handle: Long, bit: Int, down: Boolean, pad: Int)
/** One gamepad axis update. axisId: [Gamepad].AXIS_* (0..5). value: stick i16 (+y=up) / trigger 0..255. */ /** One gamepad axis update on wire pad [pad] (0..15). axisId: [Gamepad].AXIS_* (0..5). value: stick i16 (+y=up) / trigger 0..255. */
external fun nativeSendGamepadAxis(handle: Long, axisId: Int, value: Int) external fun nativeSendGamepadAxis(handle: Long, axisId: Int, value: Int, pad: Int)
/**
* Declare the controller KIND presented on wire pad [pad] (0..15) so the host builds a matching
* virtual device (mixed types across pads). pref: a [Gamepad].PREF_* wire byte. Send ONCE when a
* pad opens, BEFORE any of its input; an older host ignores it (that pad then uses the handshake's
* session-default kind — the pre-existing single-pad behaviour on pad 0).
*/
external fun nativeSendGamepadArrival(handle: Long, pref: Int, pad: Int)
/** Signal wire pad [pad] (0..15) was unplugged so the host tears its virtual device down. The core stamps the seq + re-sends. */
external fun nativeSendGamepadRemove(handle: Long, pad: Int)
// ---- Host→client gamepad feedback: Rust pulls block ~100ms, Kotlin renders (see GamepadFeedback) ---- // ---- Host→client gamepad feedback: Rust pulls block ~100ms, Kotlin renders (see GamepadFeedback) ----
/** /**
* Block up to ~100 ms for the next rumble update. Returns `(low shl 16) or high` (each * Block up to ~100 ms for the next rumble update. Returns a packed positive long: bits 49..52 =
* 0..0xFFFF; 0 = stop), or -1 on timeout / session closed. Call from a dedicated poll thread. * wire pad index (0..15), bit 48 = has a v2 lease, bits 32..47 = ttl_ms, bits 16..31 = low, bits
* 0..15 = high (each amplitude 0..0xFFFF; 0/0 = stop), or -1 on timeout / session closed. Kotlin
* routes the update to the controller holding that pad index. Call from a dedicated poll thread.
*/ */
external fun nativeNextRumble(handle: Long): Long external fun nativeNextRumble(handle: Long): Long
/** /**
* Block up to ~100 ms for the next DualSense HID-output event, written into [buf] (a direct * Block up to ~100 ms for the next DualSense HID-output event, written into [buf] (a direct
* ByteBuffer, capacity >= 64) as `[kind][fields…]`: Led=01 r g b, PlayerLeds=02 bits, * ByteBuffer, capacity >= 64) as `[pad][kind][fields…]` (leading pad = the wire pad index to
* Trigger=03 which effect…. Returns the byte count, or -1 on timeout / session closed. * route to): Led=pad 01 r g b, PlayerLeds=pad 02 bits, Trigger=pad 03 which effect…. Returns the
* byte count, or -1 on timeout / session closed.
*/ */
external fun nativeNextHidout(handle: Long, buf: java.nio.ByteBuffer): Int external fun nativeNextHidout(handle: Long, buf: java.nio.ByteBuffer): Int
} }
+130 -41
View File
@@ -15,6 +15,7 @@ use ndk::media::media_format::MediaFormat;
use ndk::native_window::NativeWindow; use ndk::native_window::NativeWindow;
use punktfunk_core::client::NativeClient; use punktfunk_core::client::NativeClient;
use punktfunk_core::error::PunktfunkError; use punktfunk_core::error::PunktfunkError;
use punktfunk_core::reanchor::{GateVerdict, ReanchorGate};
use punktfunk_core::session::Frame; use punktfunk_core::session::Frame;
use std::collections::VecDeque; use std::collections::VecDeque;
use std::ffi::c_void; use std::ffi::c_void;
@@ -208,9 +209,15 @@ fn run_sync(
// pressure the AU stays parked here instead of being dropped (a drop forces a keyframe // pressure the AU stays parked here instead of being dropped (a drop forces a keyframe
// round-trip) and we only pop the next one once it's queued. // round-trip) and we only pop the next one once it's queued.
let mut pending: Option<Frame> = None; let mut pending: Option<Frame> = None;
// Loss recovery: watch the host→client unrecoverable-drop count and ask for an IDR when it // Freeze-until-reanchor: the shared post-loss gate ([`punktfunk_core::reanchor::ReanchorGate`]).
// climbs. // Armed on a frame-index gap or a dropped-count climb, it withholds the decoder's concealed output
let mut last_dropped = client.frames_dropped(); // (released WITHOUT rendering — the SurfaceView keeps the last rendered frame on glass) until a
// proven clean re-anchor lifts it: an IDR (wire FLAG_SOF), an RFI anchor, or the 2nd recovery mark.
// `last_kf_req` throttles the keyframe intents it emits; `recovery_flags` carries each AU's
// user_flags from feed to present (keyed by the codec-echoed pts) so `on_decoded` reads the
// re-anchor signalling the platform decoder doesn't expose.
let mut gate = ReanchorGate::new(client.frames_dropped());
let mut recovery_flags: VecDeque<(u64, u32)> = VecDeque::new();
let mut last_kf_req: Option<Instant> = None; let mut last_kf_req: Option<Instant> = None;
// Skew-corrected latency stats (spec: design/stats-unification.md) use the negotiated // Skew-corrected latency stats (spec: design/stats-unification.md) use the negotiated
// host-minus-client clock offset (0 if the host didn't answer the skew handshake — then the // host-minus-client clock offset (0 if the host didn't answer the skew handshake — then the
@@ -245,9 +252,18 @@ fn run_sync(
Ok(frame) => { Ok(frame) => {
// Loss recovery (RFI): feed the frame index so a forward gap fires a throttled // Loss recovery (RFI): feed the frame index so a forward gap fires a throttled
// reference-frame-invalidation request — an RFI-capable host (AMD LTR / NVENC) // reference-frame-invalidation request — an RFI-capable host (AMD LTR / NVENC)
// recovers with a cheap clean P-frame instead of a full IDR. The frames_dropped // recovers with a cheap clean P-frame instead of a full IDR. The same forward gap
// keyframe path below stays the backstop when the recovery frame itself is lost. // arms the freeze gate so the decoder's concealment is held off the screen until the
let _ = client.note_frame_index(frame.frame_index); // recovery re-anchors. The frames_dropped keyframe path below stays the backstop.
if client.note_frame_index(frame.frame_index) {
gate.arm(Instant::now());
}
// Park this AU's re-anchor flags for the present side (keyed by the pts the codec
// echoes on the output buffer) — unconditional, unlike the HUD's `in_flight` map.
recovery_flags.push_back((frame.pts_ns / 1000, frame.flags));
if recovery_flags.len() > IN_FLIGHT_CAP {
recovery_flags.pop_front();
}
if fed == 0 { if fed == 0 {
let p = &frame.data; let p = &frame.data;
log::info!( log::info!(
@@ -336,6 +352,8 @@ fn run_sync(
&mut in_flight, &mut in_flight,
clock_offset.load(Ordering::Relaxed), clock_offset.load(Ordering::Relaxed),
&tracker, &tracker,
&mut gate,
&mut recovery_flags,
); );
rendered += r; rendered += r;
discarded += d; discarded += d;
@@ -375,21 +393,19 @@ fn run_sync(
work_accum_ns = 0; work_accum_ns = 0;
} }
// Loss recovery: under infinite GOP the only recovery keyframe is one we request. The // Loss recovery + overdue backstop, folded through the gate. Under infinite GOP the only
// reassembler drops unrecoverable AUs (frames_dropped); the decoder then conceals the // recovery keyframe is one we request; the reassembler drops unrecoverable AUs (frames_dropped)
// reference-missing delta frames that follow and renders them without error, so keying off // and the decoder then conceals the reference-missing deltas and renders them without error, so
// a decode error rarely fires. Request an IDR when the drop count climbs, throttled — the // a decode-error trigger rarely fires — the gate arms the freeze on the drop-count climb
// decode stays wedged for several frames until the IDR lands, so requesting every frame // instead. An overdue freeze (held REANCHOR_FREEZE_MAX with no clean re-anchor) re-asks while it
// would flood the control stream. // keeps holding: never resume to gray — a dead stream is the QUIC idle-timeout watchdog's job.
let dropped = client.frames_dropped();
if dropped > last_dropped {
last_dropped = dropped;
let now = Instant::now(); let now = Instant::now();
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { if gate.poll(client.frames_dropped(), now)
&& last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now); last_kf_req = Some(now);
let _ = client.request_keyframe(); let _ = client.request_keyframe();
log::debug!("decode: requested keyframe (loss recovery, dropped={dropped})"); log::debug!("decode: requested keyframe (loss recovery / overdue re-anchor)");
}
} }
} }
@@ -707,8 +723,10 @@ struct OutputReady {
/// internal looper thread) push the codec ones; the feeder thread pushes `Au`. Each carries only /// internal looper thread) push the codec ones; the feeder thread pushes `Au`. Each carries only
/// owned/`Copy` data so the callback closures satisfy the `Send` bound and never touch the codec. /// owned/`Copy` data so the callback closures satisfy the `Send` bound and never touch the codec.
enum DecodeEvent { enum DecodeEvent {
/// A received access unit from the feeder, ready to queue into the decoder. /// A received access unit from the feeder, ready to queue into the decoder. The `bool` is the
Au(Frame), /// feeder's [`NativeClient::note_frame_index`] verdict — `true` when this AU revealed a forward
/// frame-index gap, so the loop arms the freeze gate (the feeder already fired the RFI request).
Au(Frame, bool),
/// An input buffer slot freed (index) — we can queue an AU into it. /// An input buffer slot freed (index) — we can queue an AU into it.
InputAvailable(usize), InputAvailable(usize),
/// A decoded frame is ready (buffer index + echoed pts + the callback-time `decoded` stamp). /// A decoded frame is ready (buffer index + echoed pts + the callback-time `decoded` stamp).
@@ -894,7 +912,12 @@ fn run_async(
let mut discarded: u64 = 0; let mut discarded: u64 = 0;
// AUs larger than the codec input buffer, dropped whole (see `feed`/`feed_ready`). // AUs larger than the codec input buffer, dropped whole (see `feed`/`feed_ready`).
let mut oversized_dropped: u64 = 0; let mut oversized_dropped: u64 = 0;
let mut last_dropped = client.frames_dropped(); // Freeze-until-reanchor gate (see the sync loop for the rationale). Armed on a frame-index gap
// (the feeder's Au verdict), a parked-AU overflow drop, a dropped-count climb, or a recoverable
// codec error; `recovery_flags` carries each AU's user_flags from `dispatch_event` (feed) to
// `present_ready` (present), keyed by the codec-echoed pts.
let mut gate = ReanchorGate::new(client.frames_dropped());
let mut recovery_flags: VecDeque<(u64, u32)> = VecDeque::new();
let mut last_kf_req: Option<Instant> = None; let mut last_kf_req: Option<Instant> = None;
// Productive (dispatch+feed+present) time between displayed frames; reported to ADPF once one is // Productive (dispatch+feed+present) time between displayed frames; reported to ADPF once one is
// presented. The blocking event wait is excluded (idle, not work) — same accounting as the sync loop. // presented. The blocking event wait is excluded (idle, not work) — same accounting as the sync loop.
@@ -920,6 +943,8 @@ fn run_async(
&mut ready, &mut ready,
&mut fmt_dirty, &mut fmt_dirty,
&mut fatal, &mut fatal,
&mut gate,
&mut recovery_flags,
)); ));
} }
// Coalesce every other event already queued into this one work pass — correct newest-only // Coalesce every other event already queued into this one work pass — correct newest-only
@@ -932,6 +957,8 @@ fn run_async(
&mut ready, &mut ready,
&mut fmt_dirty, &mut fmt_dirty,
&mut fatal, &mut fatal,
&mut gate,
&mut recovery_flags,
)); ));
} }
stats.note_skipped(aus_dropped); // parked-AU overflow drops are client-side skips too stats.note_skipped(aus_dropped); // parked-AU overflow drops are client-side skips too
@@ -956,6 +983,8 @@ fn run_async(
&tracker, &tracker,
&mut rendered, &mut rendered,
&mut discarded, &mut discarded,
&mut gate,
&mut recovery_flags,
); );
work_accum_ns += work_t0.elapsed().as_nanos() as i64; work_accum_ns += work_t0.elapsed().as_nanos() as i64;
@@ -987,19 +1016,21 @@ fn run_async(
log::info!("decode: fed={fed} rendered={rendered} discarded={discarded}"); log::info!("decode: fed={fed} rendered={rendered} discarded={discarded}");
} }
} }
// Loss recovery: request an IDR when the reassembler's unrecoverable-drop count climbs (or we // Loss recovery + overdue backstop, folded through the gate. A parked-AU overflow drop is itself
// dropped a parked AU on overflow), throttled so a multi-frame recovery gap doesn't flood the // a loss, so it arms the freeze directly; the gate's `poll` then arms on a dropped-count climb
// control stream. // and re-asks on an overdue freeze. All keyframe intents route through the shared 100 ms
let dropped = client.frames_dropped(); // throttle so a multi-frame recovery gap can't flood the control stream.
if dropped > last_dropped || aus_dropped > 0 {
last_dropped = dropped;
let now = Instant::now(); let now = Instant::now();
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) { if aus_dropped > 0 {
gate.arm(now);
}
if (gate.poll(client.frames_dropped(), now) || aus_dropped > 0)
&& last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now); last_kf_req = Some(now);
let _ = client.request_keyframe(); let _ = client.request_keyframe();
} }
} }
}
let _ = codec.stop(); let _ = codec.stop();
shutdown.store(true, Ordering::SeqCst); // ensure the feeder wakes and exits, then join it shutdown.store(true, Ordering::SeqCst); // ensure the feeder wakes and exits, then join it
@@ -1033,8 +1064,9 @@ fn feeder_loop(
Ok(frame) => { Ok(frame) => {
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference-frame- // Loss recovery (RFI): a forward frame-index gap fires a throttled reference-frame-
// invalidation request so an RFI-capable host recovers with a cheap clean P-frame // invalidation request so an RFI-capable host recovers with a cheap clean P-frame
// instead of a full IDR (the frames_dropped keyframe path is the backstop). // instead of a full IDR (the frames_dropped keyframe path is the backstop). The gap
let _ = client.note_frame_index(frame.frame_index); // verdict rides the Au event so the decode loop arms its freeze gate on the same signal.
let gap = client.note_frame_index(frame.frame_index);
if stats.enabled() { if stats.enabled() {
let received_ns = now_realtime_ns(); let received_ns = now_realtime_ns();
let clock_offset = clock_offset.load(Ordering::Relaxed) as i128; let clock_offset = clock_offset.load(Ordering::Relaxed) as i128;
@@ -1067,7 +1099,7 @@ fn feeder_loop(
} }
} }
} }
if ev_tx.send(DecodeEvent::Au(frame)).is_err() { if ev_tx.send(DecodeEvent::Au(frame, gap)).is_err() {
break; // the decode loop is gone break; // the decode loop is gone
} }
} }
@@ -1079,6 +1111,7 @@ fn feeder_loop(
/// Route one [`DecodeEvent`] into the loop's working sets. Returns `true` only when a parked AU was /// Route one [`DecodeEvent`] into the loop's working sets. Returns `true` only when a parked AU was
/// dropped on overflow (the caller then requests a keyframe). /// dropped on overflow (the caller then requests a keyframe).
#[allow(clippy::too_many_arguments)] // two call sites; the freeze gate + flag map are threaded in
fn dispatch_event( fn dispatch_event(
ev: DecodeEvent, ev: DecodeEvent,
pending_aus: &mut VecDeque<Frame>, pending_aus: &mut VecDeque<Frame>,
@@ -1086,9 +1119,20 @@ fn dispatch_event(
ready: &mut Vec<OutputReady>, ready: &mut Vec<OutputReady>,
fmt_dirty: &mut bool, fmt_dirty: &mut bool,
fatal: &mut bool, fatal: &mut bool,
gate: &mut ReanchorGate,
recovery_flags: &mut VecDeque<(u64, u32)>,
) -> bool { ) -> bool {
match ev { match ev {
DecodeEvent::Au(f) => { DecodeEvent::Au(f, gap) => {
// A forward frame-index gap arms the freeze; park this AU's flags for the present side to
// fold `on_decoded` (keyed by the pts the codec will echo).
if gap {
gate.arm(Instant::now());
}
recovery_flags.push_back((f.pts_ns / 1000, f.flags));
if recovery_flags.len() > IN_FLIGHT_CAP {
recovery_flags.pop_front();
}
pending_aus.push_back(f); pending_aus.push_back(f);
if pending_aus.len() > FRAME_PARK_CAP { if pending_aus.len() > FRAME_PARK_CAP {
pending_aus.pop_front(); // sustained overflow — drop oldest, signal a keyframe request pending_aus.pop_front(); // sustained overflow — drop oldest, signal a keyframe request
@@ -1109,6 +1153,10 @@ fn dispatch_event(
DecodeEvent::Error { fatal: f } => { DecodeEvent::Error { fatal: f } => {
if f { if f {
*fatal = true; *fatal = true;
} else {
// A recoverable/transient codec error is a decode hiccup on a broken reference chain —
// arm the freeze so the concealed output it recovers into is held off the screen.
gate.arm(Instant::now());
} }
} }
} }
@@ -1180,6 +1228,8 @@ fn present_ready(
tracker: &DisplayTracker, tracker: &DisplayTracker,
rendered: &mut u64, rendered: &mut u64,
discarded: &mut u64, discarded: &mut u64,
gate: &mut ReanchorGate,
recovery_flags: &mut VecDeque<(u64, u32)>,
) { ) {
if ready.is_empty() { if ready.is_empty() {
return; return;
@@ -1192,10 +1242,16 @@ fn present_ready(
note_decoded_pts(stats, &mut g, clock_offset, o.pts_us, o.decoded_ns); note_decoded_pts(stats, &mut g, clock_offset, o.pts_us, o.decoded_ns);
} }
} }
// Fold EVERY output through the gate in pts (== decode) order — even the ones newest-wins discards —
// so the two-mark re-anchor count stays correct; the newest's verdict decides whether it reaches
// glass (`false` = withheld concealment; the SurfaceView keeps the last rendered frame frozen on).
let now = Instant::now();
let last = ready.len() - 1; let last = ready.len() - 1;
let mut skipped: u64 = 0; let mut skipped: u64 = 0;
for (i, o) in ready.drain(..).enumerate() { for (i, o) in ready.drain(..).enumerate() {
let render = i == last; let flags = take_flags(recovery_flags, o.pts_us);
let present = gate.on_decoded(flags, false, now) == GateVerdict::Present;
let render = i == last && present;
match codec.release_output_buffer_by_index(o.index, render) { match codec.release_output_buffer_by_index(o.index, render) {
Ok(()) if render => { Ok(()) if render => {
*rendered += 1; *rendered += 1;
@@ -1215,7 +1271,7 @@ fn present_ready(
} }
} }
} }
stats.note_skipped(skipped); // HUD `skipped` counter (newest-wins drops); no-op while hidden stats.note_skipped(skipped); // HUD `skipped` counter (newest-wins + held-off drops); no-op hidden
} }
/// React to an output-format change by signalling the stream's HDR dataspace on the Surface (SDR /// React to an output-format change by signalling the stream's HDR dataspace on the Surface (SDR
@@ -1411,19 +1467,30 @@ fn drain(
in_flight: &mut VecDeque<(u64, i128)>, in_flight: &mut VecDeque<(u64, i128)>,
clock_offset: i64, clock_offset: i64,
tracker: &DisplayTracker, tracker: &DisplayTracker,
gate: &mut ReanchorGate,
recovery_flags: &mut VecDeque<(u64, u32)>,
) -> (u64, u64) { ) -> (u64, u64) {
// Newest ready buffer so far (presented after the loop) with its HUD metadata — // Newest ready buffer so far (presented after the loop) with its HUD metadata —
// `Some((pts_us, decoded_ns))` only while the HUD is visible (the stamp read is gated). // `Some((pts_us, decoded_ns))` only while the HUD is visible. `held_present` is the freeze gate's
// verdict for that newest buffer (`false` = a post-loss concealment to withhold).
let mut held: Option<(OutputBuffer<'_>, Option<(u64, i128)>)> = None; let mut held: Option<(OutputBuffer<'_>, Option<(u64, i128)>)> = None;
let mut held_present = true;
let mut discarded: u64 = 0; let mut discarded: u64 = 0;
let mut wait = first_wait; let mut wait = first_wait;
loop { loop {
match codec.dequeue_output_buffer(wait) { match codec.dequeue_output_buffer(wait) {
Ok(DequeuedOutputBufferInfoResult::Buffer(buf)) => { Ok(DequeuedOutputBufferInfoResult::Buffer(buf)) => {
wait = Duration::ZERO; // only the first dequeue may block // Only the first dequeue may block; later ones poll (wait == ZERO).
wait = Duration::ZERO;
// Fold every dequeued frame through the gate in pts (== decode) order — even the ones
// the newest-wins policy discards — so the two-mark re-anchor count stays correct; the
// verdict of the newest (last folded) buffer decides whether it reaches glass.
let pts_us = buf.info().presentation_time_us().max(0) as u64;
let flags = take_flags(recovery_flags, pts_us);
held_present =
gate.on_decoded(flags, false, Instant::now()) == GateVerdict::Present;
let meta = if stats.enabled() { let meta = if stats.enabled() {
// The dequeue IS the sync loop's decoded-availability instant. // The dequeue IS the sync loop's decoded-availability instant.
let pts_us = buf.info().presentation_time_us().max(0) as u64;
let decoded_ns = now_realtime_ns(); let decoded_ns = now_realtime_ns();
note_decoded_pts(stats, in_flight, clock_offset, pts_us, decoded_ns); note_decoded_pts(stats, in_flight, clock_offset, pts_us, decoded_ns);
Some((pts_us, decoded_ns)) Some((pts_us, decoded_ns))
@@ -1469,16 +1536,19 @@ fn drain(
} }
} }
} }
// Present the newest ready frame, if any, and park its metadata for the render callback. // Present the newest ready frame — UNLESS the gate is withholding it as a post-loss concealment,
// in which case release it without rendering (the SurfaceView keeps the last rendered frame frozen
// on glass) and count it as a discard rather than a display.
let mut rendered = 0; let mut rendered = 0;
if let Some((buf, meta)) = held { if let Some((buf, meta)) = held {
match codec.release_output_buffer(buf, true) { match codec.release_output_buffer(buf, held_present) {
Ok(()) => { Ok(()) if held_present => {
rendered = 1; rendered = 1;
if let Some((pts_us, decoded_ns)) = meta { if let Some((pts_us, decoded_ns)) = meta {
tracker.note_rendered(pts_us, decoded_ns); tracker.note_rendered(pts_us, decoded_ns);
} }
} }
Ok(()) => discarded += 1, // held off the screen — awaiting a clean re-anchor
Err(e) => log::warn!("decode: release_output_buffer: {e}"), Err(e) => log::warn!("decode: release_output_buffer: {e}"),
} }
} }
@@ -1520,6 +1590,25 @@ fn note_decoded_pts(
stats.note_decoded(e2e_us, decode_us); stats.note_decoded(e2e_us, decode_us);
} }
/// The AU `user_flags` for a decoded output, keyed by the echoed `presentationTimeUs`. Recovery
/// signalling (FLAG_SOF IDR marker / RECOVERY_ANCHOR / RECOVERY_POINT) rides the AU's flags, which are
/// only in scope at feed time — so the feed side parks `(pts_us, flags)` here and the present side
/// looks them up to fold [`ReanchorGate::on_decoded`]. Decode order == input order (low-latency, no
/// B-frames), so this evicts entries older than `pts_us` as it goes; a miss (probe filler, or an entry
/// aged past the cap) reads `0` — no recovery flags, decoded normally.
fn take_flags(map: &mut VecDeque<(u64, u32)>, pts_us: u64) -> u32 {
while let Some(&(p, f)) = map.front() {
if p > pts_us {
break; // future frame — leave it for its own output buffer
}
map.pop_front();
if p == pts_us {
return f;
}
}
0
}
/// Map the decoder's reported output colour to a BT.2020 HDR dataspace, or `None` for SDR. The /// Map the decoder's reported output colour to a BT.2020 HDR dataspace, or `None` for SDR. The
/// integer values are the Android MediaFormat colour constants the NDK shares: COLOR_TRANSFER /// integer values are the Android MediaFormat colour constants the NDK shares: COLOR_TRANSFER
/// ST2084 = 6 (PQ/HDR10), HLG = 7; COLOR_RANGE FULL = 1, LIMITED = 2 (the host encodes limited). /// ST2084 = 6 (PQ/HDR10), HLG = 7; COLOR_RANGE FULL = 1, LIMITED = 2 (the host encodes limited).
+42 -30
View File
@@ -24,12 +24,13 @@ const TAG_PLAYER_LEDS: u8 = 0x02;
const TAG_TRIGGER: u8 = 0x03; const TAG_TRIGGER: u8 = 0x03;
/// `NativeBridge.nativeNextRumble(handle): Long` — block up to ~100 ms for the next rumble update. /// `NativeBridge.nativeNextRumble(handle): Long` — block up to ~100 ms for the next rumble update.
/// Returns a packed positive long: bit 48 = "has a v2 lease", bits 32..47 = `ttl_ms`, bits 16..31 = /// Returns a packed positive long: bits 49..52 = wire `pad` index (0..15), bit 48 = "has a v2 lease",
/// `low`, bits 0..15 = `high` (`low`/`high` 0..=0xFFFF, `0/0` = stop). The lease flag is /// bits 32..47 = `ttl_ms`, bits 16..31 = `low`, bits 0..15 = `high` (`low`/`high` 0..=0xFFFF, `0/0` =
/// out-of-band so ANY 16-bit `ttl_ms` — including 0xFFFF — is unambiguous (no in-band sentinel to /// stop). The lease flag is out-of-band so ANY 16-bit `ttl_ms` — including 0xFFFF — is unambiguous (no
/// collide with a real 65535 ms lease). No lease (legacy host) → bit 48 clear, and Kotlin falls /// in-band sentinel to collide with a real 65535 ms lease). No lease (legacy host) → bit 48 clear, and
/// back to its long one-shot. `-1` on timeout / session closed (all packed values are positive, so /// Kotlin falls back to its long one-shot. `-1` on timeout / session closed (all packed values are
/// `-1` stays unambiguous). Pad index is dropped (single-pad model). Run from a Kotlin poll thread. /// positive, so `-1` stays unambiguous). Kotlin routes the update back to the controller holding that
/// wire `pad` index (multi-pad rumble). Run from a Kotlin poll thread.
#[no_mangle] #[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextRumble( pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextRumble(
_env: JNIEnv, _env: JNIEnv,
@@ -46,14 +47,19 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextRumble(
// threads (and joins them — unbounded) before nativeClose frees the handle. // threads (and joins them — unbounded) before nativeClose frees the handle.
let h = unsafe { &*(handle as *const SessionHandle) }; let h = unsafe { &*(handle as *const SessionHandle) };
match h.client.next_rumble_ttl(PULL_TIMEOUT) { match h.client.next_rumble_ttl(PULL_TIMEOUT) {
Ok((_pad, low, high, ttl)) => { Ok((pad, low, high, ttl)) => {
// The reorder gate already ran in the core, so this update is fresh. Encode the // The reorder gate already ran in the core, so this update is fresh. Encode the
// Option out-of-band: a real lease sets bit 48 and carries ttl_ms verbatim. // Option out-of-band: a real lease sets bit 48 and carries ttl_ms verbatim. The pad
// index rides above the lease flag (bits 49..52), keeping the whole word positive.
let (lease_flag, ttl_bits) = match ttl { let (lease_flag, ttl_bits) = match ttl {
Some(ms) => (1i64 << 48, jlong::from(ms) << 32), Some(ms) => (1i64 << 48, jlong::from(ms) << 32),
None => (0, 0), None => (0, 0),
}; };
lease_flag | ttl_bits | (jlong::from(low) << 16) | jlong::from(high) (jlong::from(pad & 0xF) << 49)
| lease_flag
| ttl_bits
| (jlong::from(low) << 16)
| jlong::from(high)
} }
Err(_) => -1, // NoFrame (timeout) or Closed — Kotlin loops on its running flag Err(_) => -1, // NoFrame (timeout) or Closed — Kotlin loops on its running flag
} }
@@ -61,10 +67,12 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextRumble(
} }
/// `NativeBridge.nativeNextHidout(handle, buf): Int` — block up to ~100 ms for the next DualSense /// `NativeBridge.nativeNextHidout(handle, buf): Int` — block up to ~100 ms for the next DualSense
/// HID-output event, written into the caller's direct ByteBuffer as `[kind][fields…]`: /// HID-output event, written into the caller's direct ByteBuffer as `[pad][kind][fields…]` (the
/// Led → `[0x01][r][g][b]` (len 4) /// leading `pad` is the wire pad index the event is addressed to, so Kotlin routes it to that
/// PlayerLeds → `[0x02][bits]` (len 2) /// controller — multi-pad HID feedback):
/// Trigger → `[0x03][which][effect…]` (len 2 + effect.len()) /// Led → `[pad][0x01][r][g][b]` (len 5)
/// PlayerLeds → `[pad][0x02][bits]` (len 3)
/// Trigger → `[pad][0x03][which][effect…]` (len 3 + effect.len())
/// Returns the byte count written, or `-1` on timeout / session closed / buffer too small. /// Returns the byte count written, or `-1` on timeout / session closed / buffer too small.
#[no_mangle] #[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextHidout( pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextHidout(
@@ -97,33 +105,37 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeNextHidout(
// SAFETY: `ptr`/`cap` describe the direct ByteBuffer's backing store, valid for this call. // SAFETY: `ptr`/`cap` describe the direct ByteBuffer's backing store, valid for this call.
let out = unsafe { std::slice::from_raw_parts_mut(ptr, cap) }; let out = unsafe { std::slice::from_raw_parts_mut(ptr, cap) };
// out[0] = wire pad index; out[1] = kind tag; the rest is the per-kind payload.
let n = match ev { let n = match ev {
HidOutput::Led { r, g, b, .. } => { HidOutput::Led { pad, r, g, b } => {
if cap < 4 { if cap < 5 {
return -1; return -1;
} }
out[0] = TAG_LED; out[0] = pad;
out[1] = r; out[1] = TAG_LED;
out[2] = g; out[2] = r;
out[3] = b; out[3] = g;
4 out[4] = b;
5
} }
HidOutput::PlayerLeds { bits, .. } => { HidOutput::PlayerLeds { pad, bits } => {
if cap < 2 { if cap < 3 {
return -1; return -1;
} }
out[0] = TAG_PLAYER_LEDS; out[0] = pad;
out[1] = bits; out[1] = TAG_PLAYER_LEDS;
2 out[2] = bits;
3
} }
HidOutput::Trigger { which, effect, .. } => { HidOutput::Trigger { pad, which, effect } => {
let n = 2 + effect.len(); let n = 3 + effect.len();
if cap < n { if cap < n {
return -1; // the raw DS5 trigger block is ~11 bytes; Kotlin allocates 64 return -1; // the raw DS5 trigger block is ~11 bytes; Kotlin allocates 64
} }
out[0] = TAG_TRIGGER; out[0] = pad;
out[1] = which; out[1] = TAG_TRIGGER;
out[2..n].copy_from_slice(&effect); out[2] = which;
out[3..n].copy_from_slice(&effect);
n n
} }
HidOutput::TrackpadHaptic { .. } => { HidOutput::TrackpadHaptic { .. } => {
+64 -13
View File
@@ -145,13 +145,19 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendKey(
} }
// ---- Gamepad: Kotlin captures (KeyEvent/MotionEvent) → NativeClient::send_input --------------- // ---- Gamepad: Kotlin captures (KeyEvent/MotionEvent) → NativeClient::send_input ---------------
// Single-pad model: exactly one controller, forwarded as pad 0 (flags = 0). Buttons carry the // Multi-pad model: each physical controller is forwarded on its own wire pad index (0..15), carried
// gamepad::BTN_* bit in `code` and pressed/released in `x` (1/0); axes carry the gamepad::AXIS_* id // in the low byte of `flags` on every per-pad event — the Kotlin side (`GamepadRouter`) assigns a
// in `code` and the value in `x` (sticks i16 32768..32767, +y = up; triggers 0..255). The host // stable lowest-free index per Android device and threads it here. Buttons carry the gamepad::BTN_*
// accumulates the incremental events into its virtual xpad. Wire contract: input.rs::gamepad. // bit in `code` and pressed/released in `x` (1/0); axes carry the gamepad::AXIS_* id in `code` and
// the value in `x` (sticks i16 32768..32767, +y = up; triggers 0..255). The host accumulates the
// incremental events per pad into a matching virtual device. The core input task folds these into
// the seq'd GamepadState snapshots (keyed on this same `flags` index) and owns the per-pad seq — so
// the only thing this layer must get right is the index. Wire contract: input.rs::gamepad. A single
// controller lands on index 0, so its wire is byte-identical to the old single-pad path.
/// `NativeBridge.nativeSendGamepadButton(handle, bit, down)` — one gamepad button transition. /// `NativeBridge.nativeSendGamepadButton(handle, bit, down, pad)` — one gamepad button transition on
/// `bit`: a `gamepad::BTN_*` bit (e.g. BTN_A = 0x1000). `down`: 1=press, 0=release. /// wire pad index `pad`. `bit`: a `gamepad::BTN_*` bit (e.g. BTN_A = 0x1000). `down`: 1=press,
/// 0=release. `pad`: wire pad index 0..15 (rides `flags`).
#[no_mangle] #[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepadButton( pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepadButton(
_env: JNIEnv, _env: JNIEnv,
@@ -159,21 +165,21 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepad
handle: jlong, handle: jlong,
bit: jint, bit: jint,
down: jboolean, down: jboolean,
pad: jint,
) { ) {
// flags = 0: pad index 0 — single-pad model.
send_event( send_event(
handle, handle,
InputKind::GamepadButton, InputKind::GamepadButton,
bit as u32, bit as u32,
i32::from(down != 0), i32::from(down != 0),
0, 0,
0, pad as u32,
); );
} }
/// `NativeBridge.nativeSendGamepadAxis(handle, axisId, value)` — one gamepad axis update. /// `NativeBridge.nativeSendGamepadAxis(handle, axisId, value, pad)` — one gamepad axis update on wire
/// `axisId`: a `gamepad::AXIS_*` id (LS_X=0..RT=5). `value`: stick i16 (32768..32767, +y=up) or /// pad index `pad`. `axisId`: a `gamepad::AXIS_*` id (LS_X=0..RT=5). `value`: stick i16
/// trigger 0..255. /// (32768..32767, +y=up) or trigger 0..255. `pad`: wire pad index 0..15 (rides `flags`).
#[no_mangle] #[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepadAxis( pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepadAxis(
_env: JNIEnv, _env: JNIEnv,
@@ -181,7 +187,52 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepad
handle: jlong, handle: jlong,
axis_id: jint, axis_id: jint,
value: jint, value: jint,
pad: jint,
) { ) {
// flags = 0: pad index 0 — single-pad model. send_event(
send_event(handle, InputKind::GamepadAxis, axis_id as u32, value, 0, 0); handle,
InputKind::GamepadAxis,
axis_id as u32,
value,
0,
pad as u32,
);
}
/// `NativeBridge.nativeSendGamepadArrival(handle, pref, pad)` — declare the controller KIND presented
/// on wire pad index `pad` so the host builds a matching virtual device (mixed types — pad 0 a
/// DualSense, pad 1 an Xbox pad). `pref`: the `GamepadPref` wire byte (rides `code`). `pad`: wire pad
/// index 0..15 (rides `flags`). Sent ONCE when a pad opens, BEFORE any of its input; the core re-sends
/// it a few times against datagram loss, and an older host ignores the unknown tag (that pad then uses
/// the session-default kind from the handshake — the pre-existing single-pad behaviour on pad 0).
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepadArrival(
_env: JNIEnv,
_this: JObject,
handle: jlong,
pref: jint,
pad: jint,
) {
send_event(
handle,
InputKind::GamepadArrival,
pref as u32,
0,
0,
pad as u32,
);
}
/// `NativeBridge.nativeSendGamepadRemove(handle, pad)` — signal that wire pad index `pad` was
/// unplugged so the host tears its virtual device down. `pad` (rides `flags`) is the only field; the
/// core stamps the per-pad seq (in the snapshot seq space, so a reordered snapshot can't resurrect the
/// pad) and arms a re-send burst against datagram loss. An older host ignores the unknown tag.
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeSendGamepadRemove(
_env: JNIEnv,
_this: JObject,
handle: jlong,
pad: jint,
) {
send_event(handle, InputKind::GamepadRemove, 0, 0, 0, pad as u32);
} }
+13 -15
View File
@@ -2,24 +2,22 @@
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"> <plist version="1.0">
<dict> <dict>
<!-- Custom keys merged into the auto-generated Info.plist (GENERATE_INFOPLIST_FILE=YES <key>CADisableMinimumFrameDurationOnPhone</key>
supplies the rest). NSBonjourServices is required for NWBrowser to browse this <true/>
service type on iOS/tvOS — without it the system blocks the browse and discovery <key>GCSupportedGameControllers</key>
returns nothing. Kept OUT of the synchronized App/ + Sources/ groups so it isn't <array>
auto-added as a bundle resource (which collides with Info.plist processing). --> <dict>
<key>ProfileName</key>
<string>ExtendedGamepad</string>
</dict>
<dict>
<key>ProfileName</key>
<string>MicroGamepad</string>
</dict>
</array>
<key>NSBonjourServices</key> <key>NSBonjourServices</key>
<array> <array>
<string>_punktfunk._udp</string> <string>_punktfunk._udp</string>
</array> </array>
<!-- Standard-algorithm crypto only (AES-GCM via the Rust core) — exempt from export
compliance, but the key must be declared or every TestFlight build stalls on the
compliance question. -->
<key>ITSAppUsesNonExemptEncryption</key>
<false/>
<!-- Allow CADisplayLink above 60 Hz on ProMotion iPhones: without this key the system
silently caps the link at 60 even when SessionPresenter asks for the stream's rate
via preferredFrameRateRange, so a 120 fps stream would present at half rate. -->
<key>CADisableMinimumFrameDurationOnPhone</key>
<true/>
</dict> </dict>
</plist> </plist>
+3 -1
View File
@@ -40,6 +40,8 @@ let package = Package(
// its manifest breaks SwiftPM whole-graph validation on macOS, and only the // its manifest breaks SwiftPM whole-graph validation on macOS, and only the
// Punktfunk-tvOS target links it; the #if os(tvOS) import never compiles here.) // Punktfunk-tvOS target links it; the #if os(tvOS) import never compiles here.)
.executableTarget(name: "PunktfunkClient", dependencies: ["PunktfunkKit"]), .executableTarget(name: "PunktfunkClient", dependencies: ["PunktfunkKit"]),
.testTarget(name: "PunktfunkKitTests", dependencies: ["PunktfunkKit"]), // PunktfunkCore is a direct dep too so the wire tests can name the C ABI's
// `PunktfunkInputEvent` / `PUNKTFUNK_INPUT_KIND_*` when asserting the gamepad byte layout.
.testTarget(name: "PunktfunkKitTests", dependencies: ["PunktfunkKit", "PunktfunkCore"]),
] ]
) )
@@ -436,6 +436,7 @@
INFOPLIST_KEY_CFBundleDisplayName = Punktfunk; INFOPLIST_KEY_CFBundleDisplayName = Punktfunk;
INFOPLIST_KEY_GCSupportsControllerUserInteraction = YES; INFOPLIST_KEY_GCSupportsControllerUserInteraction = YES;
INFOPLIST_KEY_GCSupportsGameMode = YES; INFOPLIST_KEY_GCSupportsGameMode = YES;
INFOPLIST_KEY_ITSAppUsesNonExemptEncryption = NO;
INFOPLIST_KEY_LSApplicationCategoryType = "public.app-category.games"; INFOPLIST_KEY_LSApplicationCategoryType = "public.app-category.games";
INFOPLIST_KEY_NSLocalNetworkUsageDescription = "Punktfunk connects directly to your punktfunk host on the local network to stream video, audio, and input."; INFOPLIST_KEY_NSLocalNetworkUsageDescription = "Punktfunk connects directly to your punktfunk host on the local network to stream video, audio, and input.";
INFOPLIST_KEY_NSMicrophoneUsageDescription = "Your microphone is streamed to the connected punktfunk host, where it appears as a virtual microphone."; INFOPLIST_KEY_NSMicrophoneUsageDescription = "Your microphone is streamed to the connected punktfunk host, where it appears as a virtual microphone.";
@@ -477,6 +478,7 @@
INFOPLIST_KEY_CFBundleDisplayName = Punktfunk; INFOPLIST_KEY_CFBundleDisplayName = Punktfunk;
INFOPLIST_KEY_GCSupportsControllerUserInteraction = YES; INFOPLIST_KEY_GCSupportsControllerUserInteraction = YES;
INFOPLIST_KEY_GCSupportsGameMode = YES; INFOPLIST_KEY_GCSupportsGameMode = YES;
INFOPLIST_KEY_ITSAppUsesNonExemptEncryption = NO;
INFOPLIST_KEY_LSApplicationCategoryType = "public.app-category.games"; INFOPLIST_KEY_LSApplicationCategoryType = "public.app-category.games";
INFOPLIST_KEY_NSLocalNetworkUsageDescription = "Punktfunk connects directly to your punktfunk host on the local network to stream video, audio, and input."; INFOPLIST_KEY_NSLocalNetworkUsageDescription = "Punktfunk connects directly to your punktfunk host on the local network to stream video, audio, and input.";
INFOPLIST_KEY_NSMicrophoneUsageDescription = "Your microphone is streamed to the connected punktfunk host, where it appears as a virtual microphone."; INFOPLIST_KEY_NSMicrophoneUsageDescription = "Your microphone is streamed to the connected punktfunk host, where it appears as a virtual microphone.";
@@ -49,6 +49,13 @@
ReferencedContainer = "container:Punktfunk.xcodeproj"> ReferencedContainer = "container:Punktfunk.xcodeproj">
</BuildableReference> </BuildableReference>
</BuildableProductRunnable> </BuildableProductRunnable>
<EnvironmentVariables>
<EnvironmentVariable
key = "PUNKTFUNK_BILINEAR_LUMA"
value = "1"
isEnabled = "YES">
</EnvironmentVariable>
</EnvironmentVariables>
</LaunchAction> </LaunchAction>
<ProfileAction <ProfileAction
buildConfiguration = "Release" buildConfiguration = "Release"
@@ -419,9 +419,10 @@ final class SessionModel: ObservableObject {
micChannel: defaults.integer(forKey: DefaultsKey.micChannel), micChannel: defaults.integer(forKey: DefaultsKey.micChannel),
micEnabled: defaults.object(forKey: DefaultsKey.micEnabled) as? Bool ?? true) micEnabled: defaults.object(forKey: DefaultsKey.micEnabled) as? Bool ?? true)
self.audio = audio self.audio = audio
// Gamepads: forward GamepadManager's active controller as pad 0 and render the // Gamepads: forward every controller GamepadManager selected each on its own wire pad
// host's feedback (rumble always; lightbar/player-LEDs/adaptive-triggers when the // index (a pin forwards only one, Automatic forwards all) and render the host's feedback
// session's virtual pad is a DualSense). Same trust gate as audio nothing is // back to the pad it's addressed to (rumble always; lightbar/player-LEDs/adaptive-triggers
// when a pad's virtual device is a DualSense). Same trust gate as audio nothing is
// forwarded during the trust prompt. // forwarded during the trust prompt.
let capture = GamepadCapture(connection: conn, manager: .shared) let capture = GamepadCapture(connection: conn, manager: .shared)
// The cross-client escape chord (hold L1+R1+Start+Select 1.5 s) on tvOS the only // The cross-client escape chord (hold L1+R1+Start+Select 1.5 s) on tvOS the only
@@ -133,8 +133,10 @@ extension SettingsView {
.foregroundStyle(.secondary) .foregroundStyle(.secondary)
} }
Spacer() Spacer()
if gamepads.active?.id == controller.id { // Every forwarded controller is surfaced (not just the primary `active`) with its
Text("In use") // wire pad index as a player number a pin forwards only one, Automatic forwards all.
if let pad = gamepads.padIndex(for: controller) {
Text("Player \(pad + 1)")
.font(.geist(11, .semibold, relativeTo: .caption2)) .font(.geist(11, .semibold, relativeTo: .caption2))
.padding(.horizontal, 8) .padding(.horizontal, 8)
.padding(.vertical, 3) .padding(.vertical, 3)
@@ -21,10 +21,10 @@ struct SettingsView: View {
@AppStorage(DefaultsKey.streamWidth) var width = 1920 @AppStorage(DefaultsKey.streamWidth) var width = 1920
@AppStorage(DefaultsKey.streamHeight) var height = 1080 @AppStorage(DefaultsKey.streamHeight) var height = 1080
@AppStorage(DefaultsKey.streamHz) var hz = 60 @AppStorage(DefaultsKey.streamHz) var hz = 60
// Default ON: a windowed session streams at the window's native pixels (1:1, no scaling) so it // Opt-in (default OFF): the explicit mode below is used and never auto-resized. When ON, a
// stays pixel-exact instead of the presenter resampling a fixed-mode frame into the window. // windowed session instead streams at the window's native pixels (1:1, no scaling) so it stays
// Off falls back to the explicit mode below (fixed output, scaled to non-matching windows). // pixel-exact rather than the presenter resampling a fixed-mode frame into the window.
@AppStorage(DefaultsKey.matchWindow) var matchWindow = true @AppStorage(DefaultsKey.matchWindow) var matchWindow = false
@AppStorage(DefaultsKey.compositor) var compositor = 0 @AppStorage(DefaultsKey.compositor) var compositor = 0
@AppStorage(DefaultsKey.gamepadType) var gamepadType = 0 @AppStorage(DefaultsKey.gamepadType) var gamepadType = 0
@AppStorage(DefaultsKey.bitrateKbps) var bitrateKbps = 0 @AppStorage(DefaultsKey.bitrateKbps) var bitrateKbps = 0
@@ -59,6 +59,26 @@ public extension PunktfunkInputEvent {
make(PUNKTFUNK_INPUT_KIND_GAMEPAD_AXIS.rawValue, code: axis, x: value, y: 0, flags: pad) make(PUNKTFUNK_INPUT_KIND_GAMEPAD_AXIS.rawValue, code: axis, x: value, y: 0, flags: pad)
} }
/// Declare a pad's controller KIND (`InputKind::GamepadArrival`): `pref` is the
/// `GamepadType` wire byte (Auto=0, Xbox360=1, DualSense=2, XboxOne=3, DualShock4=4,
/// SteamController=5, SteamDeck=6), `pad` the wire index. Sent once when a controller slot
/// opens BEFORE that pad's first input so the host builds a matching virtual device and a
/// session can mix types (pad 0 a DualSense, pad 1 an Xbox pad). The core re-sends it a few
/// times against datagram loss and folds per-pad state behind it; a host that predates the tag
/// ignores it and uses the session-default kind from the handshake. Idempotent on the host.
static func gamepadArrival(pref: UInt32, pad: UInt32) -> PunktfunkInputEvent {
make(PUNKTFUNK_INPUT_KIND_GAMEPAD_ARRIVAL.rawValue, code: pref, x: 0, y: 0, flags: pad)
}
/// A pad disconnected (`InputKind::GamepadRemove`): `flags` = pad index. The client sends the
/// bare index; the core stamps the per-pad removal seq (`encode_gamepad_remove`) in the shared
/// snapshot seq space and arms a loss-resistant re-send burst, so the host tears the pad's
/// virtual device down and no reordered snapshot can resurrect it. A host that predates the tag
/// ignores it (the pad then lingers until session end the pre-existing behaviour).
static func gamepadRemove(pad: UInt32) -> PunktfunkInputEvent {
make(PUNKTFUNK_INPUT_KIND_GAMEPAD_REMOVE.rawValue, code: 0, x: 0, y: 0, flags: pad)
}
// Touch (host-side: libei ei_touchscreen on the virtual output). `id` distinguishes // Touch (host-side: libei ei_touchscreen on the virtual output). `id` distinguishes
// fingers and is reusable after touchUp; coordinates are absolute pixels on the // fingers and is reusable after touchUp; coordinates are absolute pixels on the
// client's touch surface, whose size rides in `flags` so the host can rescale // client's touch surface, whose size rides in `flags` so the host can rescale
@@ -450,6 +450,21 @@ public final class PunktfunkConnection {
_ = punktfunk_connection_note_frame_index(h, frameIndex, nil) _ = punktfunk_connection_note_frame_index(h, frameIndex, nil)
} }
/// Like `noteFrameIndex`, but also reports whether the core saw a FORWARD frame-index gap the
/// signal that intervening frames were lost and the following AUs reference a picture that never
/// arrived. The post-loss re-anchor gate arms its display freeze on a gap (the earliest, most
/// precise loss trigger ahead of the `framesDropped` climb). Same core side effect as
/// `noteFrameIndex` (the throttled RFI request); call it for every received AU. Returns false
/// after close.
public func noteFrameIndexGap(_ frameIndex: UInt32) -> Bool {
abiLock.lock()
defer { abiLock.unlock() }
guard let h = handle, !closeRequested else { return false }
var gap = false
_ = punktfunk_connection_note_frame_index(h, frameIndex, &gap)
return gap
}
/// Cumulative access units the hostclient reassembler dropped as unrecoverable (FEC couldn't /// Cumulative access units the hostclient reassembler dropped as unrecoverable (FEC couldn't
/// rebuild them). The video pump polls this and calls `requestKeyframe()` when it climbs the /// rebuild them). The video pump polls this and calls `requestKeyframe()` when it climbs the
/// correct loss trigger under the host's infinite GOP, where unrecoverable loss yields /// correct loss trigger under the host's infinite GOP, where unrecoverable loss yields
@@ -1,24 +1,33 @@
// Gamepad capture punktfunk/1 datagrams. Forwards exactly ONE controller whatever // Gamepad capture punktfunk/1 datagrams. Forwards EVERY controller GamepadManager selected
// GamepadManager selected as pad 0, for the lifetime of a streaming session. // each on its own stable wire pad index (pf-client-core's slot model) for the lifetime of a
// streaming session. One physical controller with no pin is player 0 (byte-identical to the old
// single-pad path); a pin forwards only that one, also as pad 0.
// //
// The wire is incremental (one button/axis transition per 18-byte event, accumulated // Each forwarded controller gets a `Slot`: its open GC handlers plus the wire state (buttons,
// host-side into the virtual pad see punktfunk_core::input::gamepad), so we snapshot the // axes, touchpad fingers, motion throttle) for its pad index isolated per device so two
// full GCExtendedGamepad state on every valueChanged and diff against the previous // controllers never clobber each other. On connect a slot opens (GamepadArrival declares its
// snapshot. Sticks are ±32767 with +y = up (GC already matches, no flip), triggers 0...255. // kind, then input flows); on disconnect / pin change / stop it closes (held state flushed to
// rest on the wire, then GamepadRemove tells the host to tear the pad's virtual device down).
//
// The wire is incremental (one button/axis transition per 18-byte event, accumulated host-side
// into the virtual pad see punktfunk_core::input::gamepad), so we snapshot the full
// GCExtendedGamepad state on every valueChanged and diff against the previous snapshot. Sticks
// are ±32767 with +y = up (GC already matches, no flip), triggers 0...255. The core folds these
// per-pad transitions into idempotent, sequence-numbered snapshots keyed on the same pad index,
// so all this layer must get right is the index one controller per slot, one slot per index.
// //
// PlayStation-pad extras ride the rich-input plane (0xCC): touchpad contacts normalized // PlayStation-pad extras ride the rich-input plane (0xCC): touchpad contacts normalized
// 0...65535 (origin top-left, +y down GC's ±1/+y-up is converted here) and motion // 0...65535 (origin top-left, +y down GC's ±1/+y-up is converted here) and motion samples in
// samples in raw DualSense sensor units (gyro 20 LSB per deg/s, accel 10000 LSB per g // raw DualSense sensor units (gyro 20 LSB per deg/s, accel 10000 LSB per g derived from the
// derived from the host's fixed calibration blob; the conversion lives in ONE place, // host's fixed calibration blob; the conversion lives in ONE place, `Wire`, so a live sign/scale
// `Wire`, so a live sign/scale correction is a one-line change). The host ignores both // correction is a one-line change). The host ignores both unless a pad's virtual device is a
// unless the session's virtual pad is a DualSense or DualShock 4 both carry a touchpad // DualSense or DualShock 4 both carry a touchpad and motion, so the capture below covers either
// and motion, so the capture below covers either (`GCDualShockGamepad` exposes the same // (`GCDualShockGamepad` exposes the same `touchpad*` surface as `GCDualSenseGamepad`).
// `touchpad*` surface as `GCDualSenseGamepad`).
// //
// Unlike mouse/keyboard capture, gamepad forwarding is NOT gated on the mouse-capture // Unlike mouse/keyboard capture, gamepad forwarding is NOT gated on the mouse-capture toggle a
// toggle a controller can't click local UI, so it always drives the host while the app // controller can't click local UI, so it always drives the host while the app is active. On
// is active. On deactivation, controller switch, or stop, every held control is released // deactivation, controller switch, or stop, every held control is released on the wire (the host
// on the wire (the host pad would otherwise stay stuck on the last state). // pad would otherwise stay stuck on the last state).
#if os(macOS) #if os(macOS)
import AppKit import AppKit
@@ -33,17 +42,35 @@ import GameController
public final class GamepadCapture { public final class GamepadCapture {
private let connection: PunktfunkConnection private let connection: PunktfunkConnection
private let manager: GamepadManager private let manager: GamepadManager
private var activeSub: AnyCancellable? private var forwardedSub: AnyCancellable?
private var observers: [NSObjectProtocol] = [] private var observers: [NSObjectProtocol] = []
private var bound: GCController?
/// App inactive GC stops delivering; everything is released and stays silent. /// App inactive GC stops delivering; everything is released and stays silent.
private var suspended = false private var suspended = false
// Last wire state (the diff base also what releaseAll() unwinds). /// One forwarded controller: the open device plus the last wire state for its pad index (the
private var buttons: UInt32 = 0 /// diff base also what `flush` unwinds). Held per Slot so two controllers never clobber each
private var axes: [Int32] = [0, 0, 0, 0, 0, 0] /// other's held buttons/axes/fingers. Mirrors pf-client-core's `Slot`.
private var fingerActive: [Bool] = [false, false] private final class Slot {
private var lastMotionNs: UInt64 = 0 let controller: GCController
/// Wire pad index (GamepadManager's stable lowest-free assignment), threaded onto every
/// event this controller sends the low byte of `flags`.
let pad: UInt32
/// The controller KIND declared to the host (GamepadArrival) when the slot opened.
let pref: PunktfunkConnection.GamepadType
var buttons: UInt32 = 0
var axes: [Int32] = [0, 0, 0, 0, 0, 0]
var fingerActive: [Bool] = [false, false]
var lastMotionNs: UInt64 = 0
init(controller: GCController, pad: UInt32, pref: PunktfunkConnection.GamepadType) {
self.controller = controller
self.pad = pad
self.pref = pref
}
}
/// Open forwarded controllers, one Slot per physical pad on its own wire index. Reconciled
/// against `manager.forwarded` (empty until a session's `start`, cleared by `stop`).
private var slots: [Slot] = []
/// Motion forwarding floor: 4 ms between samples ( 250 Hz, the DualSense's own rate). /// Motion forwarding floor: 4 ms between samples ( 250 Hz, the DualSense's own rate).
private static let motionIntervalNs: UInt64 = 4_000_000 private static let motionIntervalNs: UInt64 = 4_000_000
@@ -71,10 +98,14 @@ public final class GamepadCapture {
} }
public func start() { public func start() {
// Fires immediately with the current selection, then on every change a switch // Session-scoped index assignment: a controller pinned before the session forwards as
// releases the old controller's wire state before the new one takes over. // pad 0 (pf-client-core assigns indices at slot-open time, not app-launch time).
activeSub = manager.$active.sink { [weak self] dc in manager.resetForwardingAssignment()
MainActor.assumeIsolated { self?.rebind(to: dc?.controller) } // Fires immediately with the current forwarded set, then on every change a connect,
// disconnect, or pin change reconciles the open slots against it (opening/closing devices
// and flushing wire state so nothing sticks down).
forwardedSub = manager.$forwarded.sink { [weak self] list in
MainActor.assumeIsolated { self?.reconcile(list) }
} }
#if os(macOS) #if os(macOS)
let resign = NSApplication.willResignActiveNotification let resign = NSApplication.willResignActiveNotification
@@ -97,53 +128,56 @@ public final class GamepadCapture {
MainActor.assumeIsolated { MainActor.assumeIsolated {
guard let self else { return } guard let self else { return }
self.suspended = false self.suspended = false
if let ext = self.bound?.extendedGamepad { self.sync(ext) } // Re-send every open pad's current state (GC delivered nothing while inactive).
for slot in self.slots {
if let ext = slot.controller.extendedGamepad { self.sync(slot, ext) }
}
} }
}) })
} }
public func stop() { public func stop() {
releaseAll() closeAllSlots()
rebind(to: nil) forwardedSub = nil
activeSub = nil
observers.forEach { NotificationCenter.default.removeObserver($0) } observers.forEach { NotificationCenter.default.removeObserver($0) }
observers.removeAll() observers.removeAll()
} }
private func rebind(to controller: GCController?) { /// Bring `slots` in line with the forwarded set: close any slot no longer wanted (flushing its
guard controller !== bound else { return } /// held wire state and sending GamepadRemove first) and open any newly-forwarded controller into
releaseAll() /// its assigned wire index. A controller that stays forwarded keeps its slot untouched, so a
if let ext = bound?.extendedGamepad { /// second pad connecting never disturbs the first. Mirrors pf-client-core's `reconcile_slots`.
ext.valueChangedHandler = nil private func reconcile(_ forwarded: [GamepadManager.DiscoveredController]) {
let tp = Self.touchpad(ext) let wantIDs = Set(forwarded.map { ObjectIdentifier($0.controller) })
tp?.primary.valueChangedHandler = nil for slot in slots where !wantIDs.contains(ObjectIdentifier(slot.controller)) {
tp?.secondary.valueChangedHandler = nil closeSlot(slot)
} }
// Hand the system gestures back to the OS before letting the old pad go outside a for dc in forwarded where !slots.contains(where: { $0.controller === dc.controller }) {
// stream the share button's screenshot and the Home overlay are the user's, not ours. openSlot(dc)
if let old = bound {
for element in old.physicalInputProfile.elements.values {
element.preferredSystemGestureState = .enabled
} }
// A chord-holding pad may have just unplugged re-evaluate so a stale hold disarms.
updateEscapeChord()
} }
if let motion = bound?.motion {
motion.valueChangedHandler = nil
// Power the sensors back down left active they keep the pad streaming
// gyro/accel over Bluetooth (battery drain) long after the session.
if motion.sensorsRequireManualActivation { motion.sensorsActive = false }
}
bound = controller
guard let c = controller, let ext = c.extendedGamepad else { return }
ext.valueChangedHandler = { [weak self] g, _ in /// Open one forwarded controller on its assigned wire index: attach GC handlers, claim its
MainActor.assumeIsolated { self?.sync(g) } /// system gestures, declare its kind (GamepadArrival before any input), then wake the host
/// pad and send its initial state. Skipped when the pad has no wire index (every slot taken)
/// or exposes no extended profile.
private func openSlot(_ dc: GamepadManager.DiscoveredController) {
guard let pad = manager.padIndex(for: dc), let ext = dc.controller.extendedGamepad else { return }
let c = dc.controller
let slot = Slot(controller: c, pad: UInt32(pad), pref: dc.kind)
slots.append(slot)
ext.valueChangedHandler = { [weak self, weak slot] g, _ in
MainActor.assumeIsolated { if let self, let slot { self.sync(slot, g) } }
} }
// Claim EVERY element's system gesture while this pad drives a stream. The OS attaches // Claim EVERY element's system gesture while this pad drives a stream. The OS attaches
// gestures to several controller buttons share/create local screenshot/recording, // gestures to several controller buttons share/create local screenshot/recording,
// Home Game Center overlay (iOS) / Launchpad's Games folder (macOS) and with a // Home Game Center overlay (iOS) / Launchpad's Games folder (macOS) and with a
// gesture attached the press is the system's, not the game's. During capture the remote // gesture attached the press is the system's, not the game's. During capture the remote
// session IS the game: the share button must reach the host (e.g. Steam screenshots), // session IS the game: the share button must reach the host (e.g. Steam screenshots),
// the PS button must open the host's Steam overlay. Restored to .enabled on unbind. // the PS button must open the host's Steam overlay. Restored to .enabled on close.
for element in c.physicalInputProfile.elements.values { for element in c.physicalInputProfile.elements.values {
element.preferredSystemGestureState = .disabled element.preferredSystemGestureState = .disabled
} }
@@ -153,42 +187,83 @@ public final class GamepadCapture {
// `extendedGamepad.buttonHome` is unreliable/often nil even when the physical element // `extendedGamepad.buttonHome` is unreliable/often nil even when the physical element
// exists. On tvOS the element is absent (reserved) nil, the whole block no-ops. // exists. On tvOS the element is absent (reserved) nil, the whole block no-ops.
if let home = c.physicalInputProfile.buttons[GCInputButtonHome] { if let home = c.physicalInputProfile.buttons[GCInputButtonHome] {
home.pressedChangedHandler = { [weak self] _, _, pressed in home.pressedChangedHandler = { [weak self, weak slot] _, _, pressed in
MainActor.assumeIsolated { self?.sendGuide(down: pressed) } MainActor.assumeIsolated { if let self, let slot { self.sendGuide(slot, down: pressed) } }
} }
} }
// Wake the host pad immediately (pads are created lazily from the first event; // Declare this pad's controller KIND before any of its input, so the host builds a
// a DualSense's UHID handshake + initial lightbar write only start then). // matching virtual device (mixed types pad 0 a DualSense, pad 1 an Xbox pad). The core
connection.send(.gamepadAxis(GamepadWire.axisLSX, value: 0, pad: 0)) // re-sends it a few times against datagram loss; an older host ignores it and uses the
sync(ext) // session-default kind. Then wake the host pad (pads are created lazily from the first
// event; a DualSense's UHID handshake + initial lightbar write only start then).
connection.send(.gamepadArrival(pref: slot.pref.rawValue, pad: slot.pad))
connection.send(.gamepadAxis(GamepadWire.axisLSX, value: 0, pad: slot.pad))
sync(slot, ext)
if let tp = Self.touchpad(ext) { if let tp = Self.touchpad(ext) {
tp.primary.valueChangedHandler = { [weak self] _, x, y in tp.primary.valueChangedHandler = { [weak self, weak slot] _, x, y in
MainActor.assumeIsolated { self?.touch(finger: 0, x: x, y: y) } MainActor.assumeIsolated { if let self, let slot { self.touch(slot, finger: 0, x: x, y: y) } }
} }
tp.secondary.valueChangedHandler = { [weak self] _, x, y in tp.secondary.valueChangedHandler = { [weak self, weak slot] _, x, y in
MainActor.assumeIsolated { self?.touch(finger: 1, x: x, y: y) } MainActor.assumeIsolated { if let self, let slot { self.touch(slot, finger: 1, x: x, y: y) } }
} }
} }
if let motion = c.motion { if let motion = c.motion {
if motion.sensorsRequireManualActivation { motion.sensorsActive = true } if motion.sensorsRequireManualActivation { motion.sensorsActive = true }
motion.valueChangedHandler = { [weak self] m in motion.valueChangedHandler = { [weak self, weak slot] m in
MainActor.assumeIsolated { self?.forwardMotion(m) } MainActor.assumeIsolated { if let self, let slot { self.forwardMotion(slot, m) } }
} }
} }
} }
/// Snapshot the profile into wire state and send every transition since the last one. /// Flush a slot's held wire state (so nothing sticks down host-side) and signal the host to tear
private func sync(_ g: GCExtendedGamepad) { /// its virtual device down (GamepadRemove), then detach GC handlers, hand the system gestures
/// back, and power the sensors down. Wire-only until the GC cleanup, so it is safe even when the
/// device already physically unplugged. Mirrors pf-client-core's `close_slot_at`.
private func closeSlot(_ slot: Slot) {
flush(slot)
// Sent after the flush so the core stamps it with a seq past the zeroing snapshots; the host
// seq-gates it, so a reordered snapshot can't resurrect the removed pad.
connection.send(.gamepadRemove(pad: slot.pad))
let c = slot.controller
if let ext = c.extendedGamepad {
ext.valueChangedHandler = nil
let tp = Self.touchpad(ext)
tp?.primary.valueChangedHandler = nil
tp?.secondary.valueChangedHandler = nil
}
c.physicalInputProfile.buttons[GCInputButtonHome]?.pressedChangedHandler = nil
// Hand the system gestures back to the OS before letting the pad go outside a stream the
// share button's screenshot and the Home overlay are the user's, not ours.
for element in c.physicalInputProfile.elements.values {
element.preferredSystemGestureState = .enabled
}
if let motion = c.motion {
motion.valueChangedHandler = nil
// Power the sensors back down left active they keep the pad streaming gyro/accel
// over Bluetooth (battery drain) long after the session.
if motion.sensorsRequireManualActivation { motion.sensorsActive = false }
}
slots.removeAll { $0 === slot }
}
private func closeAllSlots() {
while let slot = slots.first { closeSlot(slot) }
chordTimer?.invalidate()
chordTimer = nil
}
/// Snapshot the profile into a slot's wire state and send every transition since the last one,
/// tagged with the slot's wire pad index.
private func sync(_ slot: Slot, _ g: GCExtendedGamepad) {
guard !suspended else { return } guard !suspended else { return }
let newButtons = Self.buttonMask(g) let newButtons = Self.buttonMask(g)
updateEscapeChord(newButtons) let changed = newButtons ^ slot.buttons
let changed = newButtons ^ buttons
if changed != 0 { if changed != 0 {
for bit in GamepadWire.allButtons where changed & bit != 0 { for bit in GamepadWire.allButtons where changed & bit != 0 {
connection.send(.gamepadButton(bit, down: newButtons & bit != 0, pad: 0)) connection.send(.gamepadButton(bit, down: newButtons & bit != 0, pad: slot.pad))
} }
buttons = newButtons slot.buttons = newButtons
} }
let newAxes: [Int32] = [ let newAxes: [Int32] = [
Int32((g.leftThumbstick.xAxis.value * 32767).rounded()), Int32((g.leftThumbstick.xAxis.value * 32767).rounded()),
@@ -198,22 +273,23 @@ public final class GamepadCapture {
Int32((g.leftTrigger.value * 255).rounded()), Int32((g.leftTrigger.value * 255).rounded()),
Int32((g.rightTrigger.value * 255).rounded()), Int32((g.rightTrigger.value * 255).rounded()),
] ]
for (i, v) in newAxes.enumerated() where v != axes[i] { for (i, v) in newAxes.enumerated() where v != slot.axes[i] {
connection.send(.gamepadAxis(UInt32(i), value: v, pad: 0)) connection.send(.gamepadAxis(UInt32(i), value: v, pad: slot.pad))
axes[i] = v slot.axes[i] = v
} }
updateEscapeChord()
} }
/// Forward the guide (Home/PS) transition directly it's kept out of `buttonMask` (the legacy /// Forward the guide (Home/PS) transition directly it's kept out of `buttonMask` (the legacy
/// `buttonHome` element is unreliable). Folds into `buttons` so a held PS button is released by /// `buttonHome` element is unreliable). Folds into the slot's `buttons` so a held PS button is
/// `releaseAll` on focus loss just like the others. /// released by `flush` on focus loss / close just like the others.
private func sendGuide(down: Bool) { private func sendGuide(_ slot: Slot, down: Bool) {
guard !suspended else { return } guard !suspended else { return }
let bit = GamepadWire.guide let bit = GamepadWire.guide
let now = down ? (buttons | bit) : (buttons & ~bit) let now = down ? (slot.buttons | bit) : (slot.buttons & ~bit)
guard now != buttons else { return } guard now != slot.buttons else { return }
connection.send(.gamepadButton(bit, down: down, pad: 0)) connection.send(.gamepadButton(bit, down: down, pad: slot.pad))
buttons = now slot.buttons = now
} }
private static func buttonMask(_ g: GCExtendedGamepad) -> UInt32 { private static func buttonMask(_ g: GCExtendedGamepad) -> UInt32 {
@@ -234,7 +310,7 @@ public final class GamepadCapture {
if g.leftShoulder.isPressed { b |= GamepadWire.leftShoulder } if g.leftShoulder.isPressed { b |= GamepadWire.leftShoulder }
if g.rightShoulder.isPressed { b |= GamepadWire.rightShoulder } if g.rightShoulder.isPressed { b |= GamepadWire.rightShoulder }
// guide (Home/PS) is NOT read here it's forwarded directly by the Home button's // guide (Home/PS) is NOT read here it's forwarded directly by the Home button's
// pressedChangedHandler (the legacy `buttonHome` element is unreliable). See `rebind`. // pressedChangedHandler (the legacy `buttonHome` element is unreliable). See `openSlot`.
if g.buttonA.isPressed { b |= GamepadWire.a } if g.buttonA.isPressed { b |= GamepadWire.a }
if g.buttonB.isPressed { b |= GamepadWire.b } if g.buttonB.isPressed { b |= GamepadWire.b }
if g.buttonX.isPressed { b |= GamepadWire.x } if g.buttonX.isPressed { b |= GamepadWire.x }
@@ -262,29 +338,29 @@ public final class GamepadCapture {
return nil return nil
} }
/// One touchpad finger moved. GC reports ±1 positions and snaps to exactly (0, 0) on /// One touchpad finger moved on a slot's pad. GC reports ±1 positions and snaps to exactly
/// lift treated as the lift signal (a real finger landing on the precise center /// (0, 0) on lift treated as the lift signal (a real finger landing on the precise center
/// momentarily reads as a lift; harmless for a 1-in-65k coincidence). /// momentarily reads as a lift; harmless for a 1-in-65k coincidence).
private func touch(finger: Int, x: Float, y: Float) { private func touch(_ slot: Slot, finger: Int, x: Float, y: Float) {
guard !suspended else { return } guard !suspended else { return }
let lifted = x == 0 && y == 0 let lifted = x == 0 && y == 0
if lifted { if lifted {
if fingerActive[finger] { if slot.fingerActive[finger] {
fingerActive[finger] = false slot.fingerActive[finger] = false
connection.sendTouchpad(finger: UInt8(finger), active: false, x: 0, y: 0) connection.sendTouchpad(pad: UInt8(slot.pad), finger: UInt8(finger), active: false, x: 0, y: 0)
} }
return return
} }
fingerActive[finger] = true slot.fingerActive[finger] = true
let w = GamepadWire.touchpad(x: x, y: y) let w = GamepadWire.touchpad(x: x, y: y)
connection.sendTouchpad(finger: UInt8(finger), active: true, x: w.x, y: w.y) connection.sendTouchpad(pad: UInt8(slot.pad), finger: UInt8(finger), active: true, x: w.x, y: w.y)
} }
private func forwardMotion(_ m: GCMotion) { private func forwardMotion(_ slot: Slot, _ m: GCMotion) {
guard !suspended else { return } guard !suspended else { return }
let now = DispatchTime.now().uptimeNanoseconds let now = DispatchTime.now().uptimeNanoseconds
guard now &- lastMotionNs >= Self.motionIntervalNs else { return } guard now &- slot.lastMotionNs >= Self.motionIntervalNs else { return }
lastMotionNs = now slot.lastMotionNs = now
// Total acceleration in g: gravity + user when split, else the raw vector. // Total acceleration in g: gravity + user when split, else the raw vector.
let ax: Float let ax: Float
let ay: Float let ay: Float
@@ -301,6 +377,7 @@ public final class GamepadCapture {
let gs = GamepadWire.gyroLSBPerRadS let gs = GamepadWire.gyroLSBPerRadS
let as_ = GamepadWire.accelLSBPerG let as_ = GamepadWire.accelLSBPerG
connection.sendMotion( connection.sendMotion(
pad: UInt8(slot.pad),
gyro: ( gyro: (
GamepadWire.motionRaw(Float(m.rotationRate.x), scale: gs), GamepadWire.motionRaw(Float(m.rotationRate.x), scale: gs),
GamepadWire.motionRaw(Float(m.rotationRate.y), scale: gs), GamepadWire.motionRaw(Float(m.rotationRate.y), scale: gs),
@@ -313,13 +390,12 @@ public final class GamepadCapture {
)) ))
} }
/// Unwind everything held on the wire: button-ups, neutral axes, lifted fingers. The /// Arm the disconnect timer when ANY forwarded pad holds the full escape chord, disarm the
/// host's virtual pad returns to rest instead of running with the last state. /// moment none do a release, or the holding pad unplugged (pf-client-core's `chord_held` is
/// Arm the disconnect timer when the full chord lands, disarm the moment any of the four /// likewise any-slot). GC events only arrive on state CHANGES, so a held chord needs the timer:
/// releases. Events only arrive on state CHANGES, so a held chord needs the timer the /// the handler won't fire again until something moves.
/// handler won't fire again until something moves. private func updateEscapeChord() {
private func updateEscapeChord(_ newButtons: UInt32) { let held = slots.contains { $0.buttons & Self.escapeChord == Self.escapeChord }
let held = newButtons & Self.escapeChord == Self.escapeChord
if held, chordTimer == nil { if held, chordTimer == nil {
let timer = Timer(timeInterval: Self.disconnectHold, repeats: false) { [weak self] _ in let timer = Timer(timeInterval: Self.disconnectHold, repeats: false) { [weak self] _ in
Task { @MainActor in self?.onDisconnectRequest?() } Task { @MainActor in self?.onDisconnectRequest?() }
@@ -332,20 +408,31 @@ public final class GamepadCapture {
} }
} }
/// Unwind everything a slot holds on the wire: button-ups, neutral axes, lifted fingers. The
/// host's virtual pad returns to rest instead of running with the last state. Wire events only
/// (no GC calls) safe against an already-removed device. Does NOT close the slot or send
/// GamepadRemove (that's `closeSlot`).
private func flush(_ slot: Slot) {
for bit in GamepadWire.allButtons where slot.buttons & bit != 0 {
connection.send(.gamepadButton(bit, down: false, pad: slot.pad))
}
slot.buttons = 0
for (i, v) in slot.axes.enumerated() where v != 0 {
connection.send(.gamepadAxis(UInt32(i), value: 0, pad: slot.pad))
slot.axes[i] = 0
}
for (f, active) in slot.fingerActive.enumerated() where active {
connection.sendTouchpad(pad: UInt8(slot.pad), finger: UInt8(f), active: false, x: 0, y: 0)
slot.fingerActive[f] = false
}
}
/// Flush every open slot's held state (app deactivation) keeps the slots open (GC just stops
/// delivering; resume re-syncs), disarms the escape chord. Distinct from `closeAllSlots`, which
/// also sends GamepadRemove and detaches handlers.
private func releaseAll() { private func releaseAll() {
chordTimer?.invalidate() chordTimer?.invalidate()
chordTimer = nil chordTimer = nil
for bit in GamepadWire.allButtons where buttons & bit != 0 { for slot in slots { flush(slot) }
connection.send(.gamepadButton(bit, down: false, pad: 0))
}
buttons = 0
for (i, v) in axes.enumerated() where v != 0 {
connection.send(.gamepadAxis(UInt32(i), value: 0, pad: 0))
axes[i] = 0
}
for (f, active) in fingerActive.enumerated() where active {
connection.sendTouchpad(finger: UInt8(f), active: false, x: 0, y: 0)
fingerActive[f] = false
}
} }
} }
@@ -1,20 +1,23 @@
// Hostclient gamepad feedback rendering: one drain thread polls the rumble (0xCA) and // Hostclient gamepad feedback rendering: one drain thread polls the rumble (0xCA) and
// HID-output (0xCD) planes and replays them on the active physical controller // HID-output (0xCD) planes and replays each update on the forwarded physical controller it is
// ADDRESSED TO by wire pad index
// //
// rumble CHHapticEngine players (per-handle localities when the pad has them, // rumble CHHapticEngine players (per-handle localities when the pad has them,
// one combined engine otherwise), // one combined engine otherwise), a RumbleRenderer per pad,
// lightbar GCDeviceLight, // lightbar GCDeviceLight,
// player LEDs GCController.playerIndex (the DS bit patterns map to player 14), // player LEDs GCController.playerIndex (the DS bit patterns map to player 14),
// trigger FX DualSenseTriggerEffect.parse GCDualSenseAdaptiveTrigger. // trigger FX DualSenseTriggerEffect.parse GCDualSenseAdaptiveTrigger.
// //
// Only pad 0 is rendered (exactly one controller is forwarded). HID-output traffic exists // Every forwarded controller gets a per-pad feedback slot (its RumbleRenderer + last light /
// only on PlayStation-pad sessions (a DualSense, or a DualShock 4 = lightbar only) the // player-LED / trigger state) keyed on the same wire index GamepadCapture streams it on, so a
// drain always polls both planes with short timeouts and never spins, so an Xbox session // rumble the host aimed at pad 1 drives pad 1's actuator and nothing else. An update for a pad
// just renders rumble. GameController profile mutation // with no live slot (one that just closed) is dropped. HID-output traffic exists only on
// happens on main; CHHapticEngine work on its own serial queue; the drain thread itself // PlayStation-pad sessions (a DualSense, or a DualShock 4 = lightbar only); the drain always
// touches neither. When GamepadManager switches the active controller mid-session, the // polls both planes with short timeouts and never spins, so an Xbox pad just renders rumble.
// old pad is reset (triggers off, player index unset) and the last known feedback state // GameController profile mutation happens on main; CHHapticEngine work on the renderer's serial
// is replayed onto the new one. // queue; the drain thread itself touches neither (it routes rumble to the pad's renderer under a
// lock and hops HID to main). When a controller leaves the forwarded set the old pad is reset
// (triggers off, player index unset) and its renderer silenced.
import Combine import Combine
import Foundation import Foundation
@@ -22,26 +25,40 @@ import GameController
public final class GamepadFeedback { public final class GamepadFeedback {
private let connection: PunktfunkConnection private let connection: PunktfunkConnection
private let manager: GamepadManager
private let flag = StopFlag() private let flag = StopFlag()
private let drainDone = DispatchSemaphore(value: 0) private let drainDone = DispatchSemaphore(value: 0)
private var drainStarted = false private var drainStarted = false
private let rumble = RumbleRenderer(policy: .session) private var forwardedSub: AnyCancellable?
private var activeSub: AnyCancellable?
// Last applied feedback (main-actor) replayed when the active controller changes. /// One forwarded controller's non-rumble feedback state (main-actor) the GC target plus the
@MainActor private var target: GCController? /// last applied lightbar / player-LED / trigger, replayed if the controller on this pad swaps.
@MainActor private var lastLight: (r: UInt8, g: UInt8, b: UInt8)? @MainActor private final class Slot {
@MainActor private var lastPlayerBits: UInt8? var controller: GCController?
@MainActor private var lastTrigger: [DualSenseTriggerEffect?] = [nil, nil] var lastLight: (r: UInt8, g: UInt8, b: UInt8)?
var lastPlayerBits: UInt8?
var lastTrigger: [DualSenseTriggerEffect?] = [nil, nil]
init(controller: GCController?) { self.controller = controller }
}
/// HID / lightbar / player-LED slots, keyed by wire pad index. Main-actor only.
@MainActor private var slots: [UInt8: Slot] = [:]
/// Rumble renderers keyed by wire pad index, guarded by `routingLock` so the background drain
/// thread can route an incoming envelope to the right pad's renderer while the main actor
/// reconciles the set. RumbleRenderer serializes on its own queue, so calling `apply` from the
/// drain thread is safe only the map lookup needs the lock.
private let routingLock = NSLock()
private var rumbleByPad: [UInt8: RumbleRenderer] = [:]
public init(connection: PunktfunkConnection, manager: GamepadManager) { public init(connection: PunktfunkConnection, manager: GamepadManager) {
self.connection = connection self.connection = connection
self.manager = manager
// Capture self weakly in the hop too, so the inner sink's weak capture isn't shadowing // Capture self weakly in the hop too, so the inner sink's weak capture isn't shadowing
// an implicit strong one and the subscription (stored on self) never retain-cycles. // an implicit strong one and the subscription (stored on self) never retain-cycles.
Task { @MainActor [weak self] in Task { @MainActor [weak self] in
guard let self else { return } guard let self else { return }
self.activeSub = manager.$active.sink { [weak self] dc in self.forwardedSub = manager.$forwarded.sink { [weak self] list in
MainActor.assumeIsolated { self?.retarget(dc?.controller) } MainActor.assumeIsolated { self?.reconcile(list) }
} }
} }
} }
@@ -67,6 +84,38 @@ public final class GamepadFeedback {
} }
} }
/// Bring the per-pad feedback slots in line with the forwarded set: drop pads no longer
/// forwarded (silence + release their renderer, reset their controller), add a slot +
/// renderer for each new pad, and retarget a pad whose controller changed (a re-plug into the
/// same freed index) replaying its cached feedback onto the new device.
@MainActor
private func reconcile(_ forwarded: [GamepadManager.DiscoveredController]) {
var want: [UInt8: GCController] = [:]
for dc in forwarded {
if let pad = manager.padIndex(for: dc) { want[pad] = dc.controller }
}
for (pad, slot) in slots where want[pad] == nil {
reset(slot.controller)
slots[pad] = nil
let renderer = withRouting { rumbleByPad.removeValue(forKey: pad) }
renderer?.stop()
}
for (pad, controller) in want {
if let slot = slots[pad] {
guard slot.controller !== controller else { continue }
reset(slot.controller)
slot.controller = controller
withRouting { rumbleByPad[pad]?.retarget(controller) }
replay(slot)
} else {
slots[pad] = Slot(controller: controller)
let renderer = RumbleRenderer(policy: .session)
renderer.retarget(controller)
withRouting { rumbleByPad[pad] = renderer }
}
}
}
public func start() { public func start() {
guard !drainStarted else { return } guard !drainStarted else { return }
drainStarted = true drainStarted = true
@@ -88,19 +137,19 @@ public final class GamepadFeedback {
// rumble/HID latency low while leaving the lock free between polls. // rumble/HID latency low while leaving the lock free between polls.
// //
// Rumble is idempotent state, so drain the plane DRY and apply only the newest // Rumble is idempotent state, so drain the plane DRY and apply only the newest
// level. The old one-datagram-per-cycle shape let a burst outpace the ~125 Hz // level PER PAD. The old one-datagram-per-cycle shape let a burst outpace the
// drain: levels rendered up to ~130 ms late through the core's 16-deep queue, // ~125 Hz drain: levels rendered up to ~130 ms late through the core's 16-deep
// and its drop-newest overflow could shed a stop while stale nonzero states // queue, and its drop-newest overflow could shed a stop while stale nonzero
// queued ahead of it buzzing until the host's next 500 ms refresh. // states queued ahead of it buzzing until the host's next 500 ms refresh.
var newest: (low: UInt16, high: UInt16, ttl: UInt32)? var newestByPad: [UInt8: (low: UInt16, high: UInt16, ttl: UInt32)] = [:]
var rumbleBurst = 0 var rumbleBurst = 0
while rumbleBurst < 64, !flag.isStopped, while rumbleBurst < 64, !flag.isStopped,
let r = try connection.nextRumble2(timeoutMs: 0) { let r = try connection.nextRumble2(timeoutMs: 0) {
if r.pad == 0 { newest = (r.low, r.high, r.ttlMs) } newestByPad[UInt8(truncatingIfNeeded: r.pad)] = (r.low, r.high, r.ttlMs)
rumbleBurst += 1 rumbleBurst += 1
} }
if let n = newest { for (pad, n) in newestByPad {
self?.rumble.apply(low: n.low, high: n.high, ttlMs: n.ttl) self?.routeRumble(pad: pad, low: n.low, high: n.high, ttlMs: n.ttl)
} }
// Drain a BOUNDED burst of hidout events so sustained 0xCD traffic (a game writing // Drain a BOUNDED burst of hidout events so sustained 0xCD traffic (a game writing
// per-frame LED/trigger reports) can't spin here or block stop() past one cycle. // per-frame LED/trigger reports) can't spin here or block stop() past one cycle.
@@ -126,7 +175,7 @@ public final class GamepadFeedback {
thread.start() thread.start()
} }
/// Stop the drain and silence the motors. Blocks until the drain thread exits ( one /// Stop the drain and silence every pad's motors. Blocks until the drain thread exits ( one
/// poll cycle) call off the main actor, before `connection.close()`. /// poll cycle) call off the main actor, before `connection.close()`.
public func stop() { public func stop() {
flag.stop() flag.stop()
@@ -134,17 +183,32 @@ public final class GamepadFeedback {
drainDone.wait() drainDone.wait()
drainStarted = false drainStarted = false
} }
rumble.stop() let renderers = withRouting { () -> [RumbleRenderer] in
// Drop the retarget subscription and the dead session's cached feedback a let r = Array(rumbleByPad.values)
// controller change after teardown must not replay this session's triggers/LEDs. rumbleByPad.removeAll()
Task { @MainActor in return r
self.activeSub = nil
self.lastLight = nil
self.lastPlayerBits = nil
self.lastTrigger = [nil, nil]
self.reset(self.target)
self.target = nil
} }
for r in renderers { r.stop() }
// Drop the subscription and every dead pad's cached feedback a controller change after
// teardown must not replay this session's triggers/LEDs.
Task { @MainActor in
self.forwardedSub = nil
for slot in self.slots.values { self.reset(slot.controller) }
self.slots.removeAll()
}
}
/// Route one rumble envelope to its pad's renderer (drain thread). An update for a pad with no
/// live renderer one that just left the forwarded set is dropped.
private func routeRumble(pad: UInt8, low: UInt16, high: UInt16, ttlMs: UInt32) {
let renderer = withRouting { rumbleByPad[pad] }
renderer?.apply(low: low, high: high, ttlMs: ttlMs)
}
private func withRouting<R>(_ body: () -> R) -> R {
routingLock.lock()
defer { routingLock.unlock() }
return body()
} }
private func render(_ ev: PunktfunkConnection.HidOutputEvent) { private func render(_ ev: PunktfunkConnection.HidOutputEvent) {
@@ -157,40 +221,37 @@ public final class GamepadFeedback {
private func apply(_ ev: PunktfunkConnection.HidOutputEvent) { private func apply(_ ev: PunktfunkConnection.HidOutputEvent) {
switch ev { switch ev {
case let .led(pad, r, g, b): case let .led(pad, r, g, b):
guard pad == 0 else { return } guard let slot = slots[pad] else { return }
lastLight = (r, g, b) slot.lastLight = (r, g, b)
target?.light?.color = GCColor( slot.controller?.light?.color = GCColor(
red: Float(r) / 255, green: Float(g) / 255, blue: Float(b) / 255) red: Float(r) / 255, green: Float(g) / 255, blue: Float(b) / 255)
case let .playerLEDs(pad, bits): case let .playerLEDs(pad, bits):
guard pad == 0 else { return } guard let slot = slots[pad] else { return }
lastPlayerBits = bits slot.lastPlayerBits = bits
target?.playerIndex = Self.playerIndex(forBits: bits) slot.controller?.playerIndex = Self.playerIndex(forBits: bits)
case let .triggerEffect(pad, which, effect): case let .triggerEffect(pad, which, effect):
guard pad == 0, which < 2 else { return } guard which < 2, let slot = slots[pad] else { return }
let parsed = DualSenseTriggerEffect.parse(effect) let parsed = DualSenseTriggerEffect.parse(effect)
lastTrigger[Int(which)] = parsed slot.lastTrigger[Int(which)] = parsed
if let trigger = adaptiveTrigger(which) { if let trigger = adaptiveTrigger(slot.controller, which) {
parsed.apply(to: trigger) parsed.apply(to: trigger)
} }
} }
} }
/// Replay a pad's cached feedback onto its (swapped-in) controller so a re-plug looks the same.
@MainActor @MainActor
private func retarget(_ controller: GCController?) { private func replay(_ slot: Slot) {
guard controller !== target else { return } if let (r, g, b) = slot.lastLight {
reset(target) slot.controller?.light?.color = GCColor(
target = controller
rumble.retarget(controller)
// Replay the session's feedback state so a swapped-in controller looks the same.
if let (r, g, b) = lastLight {
controller?.light?.color = GCColor(
red: Float(r) / 255, green: Float(g) / 255, blue: Float(b) / 255) red: Float(r) / 255, green: Float(g) / 255, blue: Float(b) / 255)
} }
if let bits = lastPlayerBits { if let bits = slot.lastPlayerBits {
controller?.playerIndex = Self.playerIndex(forBits: bits) slot.controller?.playerIndex = Self.playerIndex(forBits: bits)
} }
for which in 0..<2 { for which in 0..<2 {
if let effect = lastTrigger[which], let trigger = adaptiveTrigger(UInt8(which)) { if let effect = slot.lastTrigger[which],
let trigger = adaptiveTrigger(slot.controller, UInt8(which)) {
effect.apply(to: trigger) effect.apply(to: trigger)
} }
} }
@@ -207,8 +268,8 @@ public final class GamepadFeedback {
} }
@MainActor @MainActor
private func adaptiveTrigger(_ which: UInt8) -> GCDualSenseAdaptiveTrigger? { private func adaptiveTrigger(_ controller: GCController?, _ which: UInt8) -> GCDualSenseAdaptiveTrigger? {
guard let ds = target?.extendedGamepad as? GCDualSenseGamepad else { return nil } guard let ds = controller?.extendedGamepad as? GCDualSenseGamepad else { return nil }
return which == 0 ? ds.leftTrigger : ds.rightTrigger return which == 0 ? ds.leftTrigger : ds.rightTrigger
} }
} }
@@ -1,14 +1,18 @@
// Controller discovery + selection, app-lifetime. One GamepadManager (`.shared`) watches // Controller discovery + selection, app-lifetime. One GamepadManager (`.shared`) watches
// GCController connect/disconnect from launch, so the Settings page shows live controller // GCController connect/disconnect from launch, so the Settings page shows live controller
// state without a session, and the session components (GamepadCapture / GamepadFeedback) // state without a session, and the session components (GamepadCapture / GamepadFeedback)
// follow `active` exactly ONE physical controller is forwarded to the host, as pad 0. // follow `forwarded` every forwarded controller is streamed to the host, each on its own
// wire pad index (pf-client-core parity; up to `GamepadWire.maxPads`).
// //
// Selection: the user can pin a controller in Settings (persisted under // Selection (mirrors pf-client-core's `forwarded_ids` + slot model): with no pin, EVERY
// DefaultsKey.gamepadID); with no pin or the pinned one absent the most recently // extended controller is forwarded each assigned a stable lowest-free pad index held for
// connected extended gamepad wins. GCController has no stable hardware serial, so the pin // its forwarded lifetime, so a disconnect frees only its own index and never renumbers the
// is a fingerprint of vendorName|productCategory (+ a connect-order suffix for twins); // others. A pin (Settings, persisted under DefaultsKey.gamepadID) forwards ONLY that one pad
// identical twin controllers may swap a pin across reconnects, which the Settings footer // an explicit single-player choice. `active` stays the single "primary" pad (the pinned
// documents. // one, else the most recently connected extended gamepad) that the Settings / launcher / menu
// UI reads. GCController has no stable hardware serial, so the pin is a fingerprint of
// vendorName|productCategory (+ a connect-order suffix for twins); identical twin controllers
// may swap a pin across reconnects, which the Settings footer documents.
// //
// A singleton (not a SwiftUI environment object) because macOS shows Settings in its own // A singleton (not a SwiftUI environment object) because macOS shows Settings in its own
// `Settings{}` scene there is no common ancestor view to inject from. // `Settings{}` scene there is no common ancestor view to inject from.
@@ -60,9 +64,23 @@ public final class GamepadManager: ObservableObject {
/// Every detected controller, in connect order (Settings lists these). /// Every detected controller, in connect order (Settings lists these).
@Published public private(set) var controllers: [DiscoveredController] = [] @Published public private(set) var controllers: [DiscoveredController] = []
/// The one controller forwarded to the host (pad 0); nil when none qualifies. /// The single "primary" controller the pinned one, else the most recently connected
/// extended gamepad; nil when none qualifies. The Settings / launcher / menu UI and the
/// connect-time `resolveType` read this; the streaming input path uses `forwarded`.
@Published public private(set) var active: DiscoveredController? @Published public private(set) var active: DiscoveredController?
/// The controllers forwarded to the host this session, in wire-pad-index preference order
/// (pf-client-core's `forwarded_ids`): a pin forwards ONLY the pinned pad; Automatic forwards
/// every extended controller. GamepadCapture opens a slot per entry and GamepadFeedback routes
/// feedback back to it, each on the index from `padIndex(for:)`.
@Published public private(set) var forwarded: [DiscoveredController] = []
/// Stable wire pad index (0..<`GamepadWire.maxPads`) per forwarded controller, keyed by
/// GCController identity. Lowest-free, held while the controller stays forwarded a
/// disconnect frees only its own index so the others never renumber (pf-client-core's
/// `lowest_free_index`). Recomputed by `assignPadIndices` whenever `forwarded` changes.
private var padIndexByController: [ObjectIdentifier: UInt8] = [:]
/// The user's pinned controller fingerprint ("" = automatic). Persisted; updating it /// The user's pinned controller fingerprint ("" = automatic). Persisted; updating it
/// reselects immediately, so a Settings Picker can bind straight to this. /// reselects immediately, so a Settings Picker can bind straight to this.
@Published public var preferredID: String { @Published public var preferredID: String {
@@ -159,7 +177,52 @@ public final class GamepadManager: ObservableObject {
let candidates = controllers.filter(\.isExtended) let candidates = controllers.filter(\.isExtended)
// The pin wins when present; otherwise the most recently connected extended pad // The pin wins when present; otherwise the most recently connected extended pad
// (list is in connect order). A stale pin falls back to automatic. // (list is in connect order). A stale pin falls back to automatic.
active = candidates.last { $0.id == preferredID } ?? candidates.last let pinned = candidates.last { $0.id == preferredID }
active = pinned ?? candidates.last
// Forwarded set (pf-client-core's `forwarded_ids`): a pin forwards ONLY the pinned pad
// (explicit single-player); Automatic forwards every extended controller in connect order
// (oldestnewest), so a game's player numbers are stable across hot-plug churn.
let next = pinned.map { [$0] } ?? candidates
// Update the pad-index assignment BEFORE publishing `forwarded`: @Published emits in
// `willSet`, so GamepadCapture/GamepadFeedback reconcile against `padIndex(for:)` the
// instant this assignment lands a stale map here would skip a newly-forwarded pad.
assignPadIndices(for: next)
forwarded = next
}
/// Assign each forwarded controller a stable wire pad index (lowest-free, held while it stays
/// forwarded) mirrors pf-client-core's slot model, where a disconnect frees only its own
/// index and the others keep theirs. A controller already holding an index keeps it across the
/// churn; a slot beyond `GamepadWire.maxPads` goes unassigned (that pad is not forwarded).
private func assignPadIndices(for next: [DiscoveredController]) {
let live = Set(next.map { ObjectIdentifier($0.controller) })
padIndexByController = padIndexByController.filter { live.contains($0.key) }
for dc in next {
let key = ObjectIdentifier(dc.controller)
guard padIndexByController[key] == nil,
let free = Self.lowestFreeIndex(Set(padIndexByController.values)) else { continue }
padIndexByController[key] = free
}
}
/// The lowest wire pad index not already taken, or nil when all `GamepadWire.maxPads` are in
/// use (pf-client-core's `lowest_free_index`).
private static func lowestFreeIndex(_ taken: Set<UInt8>) -> UInt8? {
(0..<UInt8(GamepadWire.maxPads)).first { !taken.contains($0) }
}
/// The wire pad index a forwarded controller streams on, or nil when it isn't forwarded.
public func padIndex(for controller: DiscoveredController) -> UInt8? {
padIndexByController[ObjectIdentifier(controller.controller)]
}
/// Drop every pad-index assignment and recompute from the current forwarded set called when
/// a streaming session begins so the assignment starts fresh (a controller pinned before the
/// session forwards as pad 0, not whatever index it held for the Settings list). pf-client-core
/// assigns indices at slot-open time; this reproduces that session-scoped start.
public func resetForwardingAssignment() {
padIndexByController.removeAll()
reselect()
} }
private static func describe(_ c: GCController, id: String) -> DiscoveredController { private static func describe(_ c: GCController, id: String) -> DiscoveredController {
@@ -1,10 +1,14 @@
// The gamepad wire contract shared by capture (GamepadCapture), feedback (GamepadFeedback), // The gamepad wire contract shared by capture (GamepadCapture), feedback (GamepadFeedback),
// and the tests button bits, axis ids, and the touchpad/motion unit conversions. // and the tests the pad count, button bits, axis ids, and the touchpad/motion unit conversions.
import Foundation import Foundation
/// The gamepad wire contract (mirrors `punktfunk_core::input::gamepad`). /// The gamepad wire contract (mirrors `punktfunk_core::input::gamepad`).
public enum GamepadWire { public enum GamepadWire {
/// Gamepads addressable on the wire the pad index rides the low byte of `flags` on every
/// per-pad event, 0...15 (`punktfunk_core::input::MAX_PADS`).
public static let maxPads: Int = 16
public static let dpadUp: UInt32 = 0x0001 public static let dpadUp: UInt32 = 0x0001
public static let dpadDown: UInt32 = 0x0002 public static let dpadDown: UInt32 = 0x0002
public static let dpadLeft: UInt32 = 0x0004 public static let dpadLeft: UInt32 = 0x0004
@@ -85,6 +85,12 @@ public final class InputCapture {
/// its Esc suppression need it in both states). /// its Esc suppression need it in both states).
private var cmdKeysDown: Set<UInt32> = [] private var cmdKeysDown: Set<UInt32> = []
/// Physical Control/Option/Shift keys currently held (Windows VKs, both L/R sides). iPad only:
/// the Q release chord is recognized from the HID stream here (iOS has no NSEvent monitor,
/// like the toggle), so it needs the live modifier state tracked in both forwarding states,
/// exactly like `cmdKeysDown`, and flushed by `releaseAll` when GC delivery stops.
private var chordModifiersDown: Set<UInt32> = []
/// While true, mouse/keyboard flow to the host and key NSEvents are swallowed /// While true, mouse/keyboard flow to the host and key NSEvents are swallowed
/// locally; while false the user is interacting with the local UI (dragging the /// locally; while false the user is interacting with the local UI (dragging the
/// window, clicking the HUD) and nothing is forwarded. Main-queue only. /// window, clicking the HUD) and nothing is forwarded. Main-queue only.
@@ -119,6 +125,21 @@ public final class InputCapture {
public var onDisconnect: (() -> Void)? public var onDisconnect: (() -> Void)?
public var onCycleStats: (() -> Void)? public var onCycleStats: (() -> Void)?
#if os(iOS)
/// Windows VKs of the three modifier classes in the Q release chord, both L/R sides:
/// control (0xA2/0xA3), option (0xA4/0xA5), shift (0xA0/0xA1). Used to sift the HID key stream.
private static let chordModifierVKs: Set<UInt32> = [0xA2, 0xA3, 0xA4, 0xA5, 0xA0, 0xA1]
/// Whether Control AND Option AND Shift are all currently held (either side of each counts)
/// the modifier precondition for the iPad Q release chord.
private var hasReleaseChordModifiers: Bool {
let m = chordModifiersDown
return (m.contains(0xA2) || m.contains(0xA3)) // control
&& (m.contains(0xA4) || m.contains(0xA5)) // option
&& (m.contains(0xA0) || m.contains(0xA1)) // shift
}
#endif
/// Fired when a newer InputCapture takes the process-global GC handler slots (the /// Fired when a newer InputCapture takes the process-global GC handler slots (the
/// singletons hold ONE handler each): the preempted owner must drop its capture /// singletons hold ONE handler each): the preempted owner must drop its capture
/// state its handlers are gone, so it would otherwise sit "captured" with dead /// state its handlers are gone, so it would otherwise sit "captured" with dead
@@ -294,6 +315,7 @@ public final class InputCapture {
/// in another app would otherwise stay "held" here forever hijacking Esc). /// in another app would otherwise stay "held" here forever hijacking Esc).
private func releaseAll() { private func releaseAll() {
cmdKeysDown.removeAll() cmdKeysDown.removeAll()
chordModifiersDown.removeAll()
suppressedVK = nil suppressedVK = nil
for vk in pressedVKs { for vk in pressedVKs {
connection.send(.key(vk, down: false)) connection.send(.key(vk, down: false))
@@ -576,6 +598,13 @@ public final class InputCapture {
self.cmdKeysDown.remove(vk) self.cmdKeysDown.remove(vk)
} }
} }
#if os(iOS)
// Track Control/Option/Shift for the Q release chord below in both forwarding
// states (like `cmdKeysDown`) so a modifier held before capture engaged still counts.
if Self.chordModifierVKs.contains(vk) {
if pressed { self.chordModifiersDown.insert(vk) } else { self.chordModifiersDown.remove(vk) }
}
#endif
// The toggle's Esc checked before the forwarding gate, because in the // The toggle's Esc checked before the forwarding gate, because in the
// engage direction forwarding is already true when this fires. // engage direction forwarding is already true when this fires.
if vk == self.suppressedVK { if vk == self.suppressedVK {
@@ -592,6 +621,18 @@ public final class InputCapture {
} }
#endif #endif
guard self.forwarding else { return } guard self.forwarding else { return }
#if os(iOS)
// Q releases the captured mouse/keyboard (cross-client parity the same combo the
// macOS keyDown monitor handles). Recognized only while forwarding (nothing to release
// otherwise). The Q is latched (`suppressedVK`) so its keyUp can't type into the host;
// the modifiers were forwarded as they went down and are flushed by the release
// path (setCaptured(false) releaseAll). VK 0x51 is layout-independent (physical Q).
if pressed, vk == 0x51, self.hasReleaseChordModifiers {
self.suppressedVK = 0x51
self.onReleaseCapture?()
return
}
#endif
// Release direction of the toggle: GC's Esc-down can beat the NSEvent // Release direction of the toggle: GC's Esc-down can beat the NSEvent
// monitor never type Esc into the host while is held ( is reserved). // monitor never type Esc into the host while is held ( is reserved).
if vk == 0x1B, !self.cmdKeysDown.isEmpty { if vk == 0x1B, !self.cmdKeysDown.isEmpty {
@@ -124,7 +124,16 @@ float2 chromaUV(texture2d<float> lumaTex, texture2d<float> chromaTex, float2 uv)
float3 sampleRgb(texture2d<float> lumaTex, texture2d<float> chromaTex, float2 uv, float3 sampleRgb(texture2d<float> lumaTex, texture2d<float> chromaTex, float2 uv,
constant CscUniform& csc) { constant CscUniform& csc) {
constexpr sampler s(filter::linear, address::clamp_to_edge); constexpr sampler s(filter::linear, address::clamp_to_edge);
float3 yuv = float3(catmullRomLuma(lumaTex, s, uv), #ifdef PF_BILINEAR_LUMA
// DEBUG (PUNKTFUNK_BILINEAR_LUMA=1): plain bilinear luma Catmull-Rom OFF. A/B lever to see if
// the bicubic overshoot contributes to edge fringing. NOTE: at a true 1:1 present both paths
// reduce to the identity texel, so if this toggle VISIBLY changes the picture, the present is
// NOT 1:1 (there's a resample); if it looks identical, the fringing is upstream (codec/source/OS).
float lumaY = lumaTex.sample(s, uv).r;
#else
float lumaY = catmullRomLuma(lumaTex, s, uv);
#endif
float3 yuv = float3(lumaY,
chromaTex.sample(s, chromaUV(lumaTex, chromaTex, uv)).rg); chromaTex.sample(s, chromaUV(lumaTex, chromaTex, uv)).rg);
return saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w, return saturate(float3(dot(csc.r0.xyz, yuv) + csc.r0.w,
dot(csc.r1.xyz, yuv) + csc.r1.w, dot(csc.r1.xyz, yuv) + csc.r1.w,
@@ -250,7 +259,16 @@ public final class MetalVideoPresenter {
let pipelineHDR: MTLRenderPipelineState let pipelineHDR: MTLRenderPipelineState
let pipelineHDRToneMap: MTLRenderPipelineState? let pipelineHDRToneMap: MTLRenderPipelineState?
do { do {
let library = try device.makeLibrary(source: shaderSource, options: nil) // DEBUG A/B lever: PUNKTFUNK_BILINEAR_LUMA=1 compiles the shader with Catmull-Rom OFF
// (plain bilinear luma) by prepending a #define ahead of the source. Default (unset) is
// the normal bicubic path. Read at presenter creation set it in the environment and
// relaunch to flip; the log line confirms which path built.
let bilinearLuma = ProcessInfo.processInfo.environment["PUNKTFUNK_BILINEAR_LUMA"] == "1"
let source = (bilinearLuma ? "#define PF_BILINEAR_LUMA 1\n" : "") + shaderSource
if bilinearLuma {
presenterLog.info("stage2: PUNKTFUNK_BILINEAR_LUMA=1 — Catmull-Rom luma DISABLED (bilinear)")
}
let library = try device.makeLibrary(source: source, options: nil)
let vtx = library.makeFunction(name: "pf_vtx") let vtx = library.makeFunction(name: "pf_vtx")
let sdr = MTLRenderPipelineDescriptor() let sdr = MTLRenderPipelineDescriptor()
sdr.vertexFunction = vtx sdr.vertexFunction = vtx
@@ -590,8 +608,17 @@ public final class MetalVideoPresenter {
let sig = "\(Int(decoded.width))x\(Int(decoded.height))\(Int(drawable.width))x\(Int(drawable.height))|hdr\(hdrActive ? 1 : 0)" let sig = "\(Int(decoded.width))x\(Int(decoded.height))\(Int(drawable.width))x\(Int(drawable.height))|hdr\(hdrActive ? 1 : 0)"
if sig != lastSizeSig { if sig != lastSizeSig {
lastSizeSig = sig lastSizeSig = sig
// Explicit verdict: is the shader presenting 1:1 (decoded == drawable) or resampling? The
// scale ratio makes a residual match-window mismatch obvious. If this says 1:1 but the
// picture is still soft, the resample is downstream of us (macOS compositor a scaled
// display mode, or a fractional-pixel window position), not the shader.
let sx = decoded.width > 0 ? drawable.width / decoded.width : 0
let sy = decoded.height > 0 ? drawable.height / decoded.height : 0
let verdict = decoded == drawable
? "1:1 (no resample)"
: String(format: "RESAMPLE scale=%.4fx%.4f", sx, sy)
let msg = let msg =
"stage2: decoded \(Int(decoded.width))x\(Int(decoded.height)) → drawable \(Int(drawable.width))x\(Int(drawable.height)) hdr=\(hdrActive)" "stage2: decoded \(Int(decoded.width))x\(Int(decoded.height)) → drawable \(Int(drawable.width))x\(Int(drawable.height)) [\(verdict)] hdr=\(hdrActive)"
presenterLog.info("\(msg, privacy: .public)") presenterLog.info("\(msg, privacy: .public)")
} }
} }
@@ -0,0 +1,99 @@
// Swift wrapper around the punktfunk-core C ABI's post-loss re-anchor gate
// (`punktfunk_reanchor_gate_*`, ABI v6). The shared Rust gate (crates/punktfunk-core/src/reanchor.rs)
// is what the Linux/Windows desktop pump and the Android client use directly; the Swift clients reach
// it across the C ABI so the freeze-until-reanchor policy is defined ONCE for every platform.
//
// Why a freeze at all: after unrecoverable loss the host keeps sending delta frames that reference a
// picture the client never got. Hardware decoders (VideoToolbox included) don't reliably error on
// that they CONCEAL, returning a gray/garbage frame with a success status. Presenting those is the
// visible "gray flash with motion" of the loss reports. The gate withholds concealed frames and holds
// the last good picture on glass until a PROVEN clean re-anchor lands an IDR (wire `FLAG_SOF`), an
// RFI recovery anchor (`USER_FLAG_RECOVERY_ANCHOR`), or the 2nd of two intra-refresh recovery marks
// (`USER_FLAG_RECOVERY_POINT`) with a bounded backstop so a lost re-anchor can never freeze forever.
// See punktfunk-planning design/client-reanchor-freeze-parity.md.
//
// Threading: one gate per session. Its calls arrive from two threads the pump thread (`arm` on a
// frame-index gap / a submit failure, `poll` per iteration) and a VideoToolbox decode thread
// (`onDecoded` per decoded frame, `onNoOutput` on a decode error). The raw Rust gate is a plain
// struct behind an opaque pointer with no internal synchronization, so every call is serialized under
// `lock` here the calls are cheap field updates, so contention is negligible. `@unchecked Sendable`:
// the lock enforces the contract.
import Foundation
import PunktfunkCore
final class ReanchorGate: @unchecked Sendable {
private let lock = NSLock()
/// The opaque `ReanchorGate *`. `var` so `reseed` can swap it at session start. Never NULL
/// (`punktfunk_reanchor_gate_new` never returns NULL).
private var ptr: OpaquePointer
/// Seed the baseline with the connection's current `framesDropped` so the first `poll` doesn't
/// read the session's starting drop count as a fresh loss.
init(framesDropped: UInt64) {
ptr = punktfunk_reanchor_gate_new(framesDropped)
}
deinit { punktfunk_reanchor_gate_free(ptr) }
/// Re-anchor the drop-count baseline to `framesDropped` for a (re)started session. The gate is
/// created in the pipeline's init (before a connection exists, seeded 0); `start` calls this once
/// the live connection's count is known so a mid-life connection's non-zero baseline isn't
/// mistaken for loss on the first poll.
func reseed(framesDropped: UInt64) {
lock.lock()
defer { lock.unlock() }
punktfunk_reanchor_gate_free(ptr)
ptr = punktfunk_reanchor_gate_new(framesDropped)
}
/// Arm the freeze: a loss was detected (a frame-index gap, or a decoder wedge). Zeroes the
/// recovery-mark count and (re)sets the backstop deadline.
func arm() {
lock.lock()
punktfunk_reanchor_gate_arm(ptr)
lock.unlock()
}
/// Fold one decoded frame. `flags` is the AU's wire `user_flags`. Returns true to PRESENT the
/// frame, false to WITHHOLD it as a post-loss concealment (hold the last good picture). Pass
/// `decoderKeyframe: false` VideoToolbox doesn't flag IDRs, so the wire `FLAG_SOF` covers it.
func onDecoded(flags: UInt32, decoderKeyframe: Bool = false) -> Bool {
lock.lock()
defer { lock.unlock() }
var present = false
_ = punktfunk_reanchor_gate_on_decoded(ptr, flags, decoderKeyframe, &present)
return present
}
/// A received AU produced no decoded frame (a VideoToolbox decode error). Returns true when the
/// no-output streak has tripped (the gate armed the freeze) and the caller should throttled
/// request a keyframe.
func onNoOutput() -> Bool {
lock.lock()
defer { lock.unlock() }
var requestKf = false
_ = punktfunk_reanchor_gate_on_no_output(ptr, &requestKf)
return requestKf
}
/// Periodic fold of the session's `framesDropped` plus the overdue backstop. Returns true when the
/// caller should throttled request a keyframe (a drop-count climb armed a fresh freeze, or the
/// freeze is overdue and re-asks while it keeps holding).
func poll(framesDropped: UInt64) -> Bool {
lock.lock()
defer { lock.unlock() }
var requestKf = false
_ = punktfunk_reanchor_gate_poll(ptr, framesDropped, &requestKf)
return requestKf
}
/// Whether the gate is currently withholding concealed frames (frozen on the last good picture).
var isHolding: Bool {
lock.lock()
defer { lock.unlock() }
var holding = false
_ = punktfunk_reanchor_gate_is_holding(ptr, &holding)
return holding
}
}
@@ -205,17 +205,33 @@ final class SessionPresenter {
return nil return nil
}() }()
let fit: CGRect = aspect.map { AVMakeRect(aspectRatio: $0, insideRect: bounds) } ?? bounds let fit: CGRect = aspect.map { AVMakeRect(aspectRatio: $0, insideRect: bounds) } ?? bounds
// Snap the sublayer frame to the BACKING PIXEL GRID. AVMakeRect centers the aspect-fit rect,
// so its origin/size are usually fractional points; a metal sublayer whose frame doesn't land
// on whole device pixels is RESAMPLED by the macOS/UIKit compositor during composite a
// uniform "everything looks soft" blur even when the drawable itself is pixel-exact 1:1
// (verified via the stage2 "[1:1 (no resample)]" log while the picture was still soft). Round
// origin AND size to device pixels so the composite is a true 1:1 blit. Idempotent when the
// frame is already aligned (e.g. fullscreen fit == integer bounds), so it's a no-op there.
let scale = contentsScale > 0 ? contentsScale : 1
let snapped = CGRect(
x: (fit.origin.x * scale).rounded() / scale,
y: (fit.origin.y * scale).rounded() / scale,
width: (fit.width * scale).rounded() / scale,
height: (fit.height * scale).rounded() / scale)
// No implicit resize animation; contentsScale tracks the view's backing/display scale. // No implicit resize animation; contentsScale tracks the view's backing/display scale.
CATransaction.begin() CATransaction.begin()
CATransaction.setDisableActions(true) CATransaction.setDisableActions(true)
metalLayer.contentsScale = contentsScale metalLayer.contentsScale = contentsScale
metalLayer.frame = fit metalLayer.frame = snapped
CATransaction.commit() CATransaction.commit()
// Hand the resulting pixel size to the render thread (it must not read layer geometry // Hand the resulting pixel size to the render thread (it must not read layer geometry
// cross-thread) this is what the presenter sizes its drawable to. // cross-thread) this is what the presenter sizes its drawable to. Uses the SNAPPED size so
// the drawable's texel count equals the on-screen device-pixel count exactly (1 texel 1
// device pixel); with the frame snapped, this equals the pre-snap rounded value, so the
// decodeddrawable 1:1 the log confirmed is preserved.
stage2?.setDrawableTarget(CGSize( stage2?.setDrawableTarget(CGSize(
width: (fit.width * contentsScale).rounded(), width: (snapped.width * scale).rounded(),
height: (fit.height * contentsScale).rounded())) height: (snapped.height * scale).rounded()))
#if os(tvOS) #if os(tvOS)
// Push the display's live EDR headroom alongside: > 1 means the TV is composited in an // Push the display's live EDR headroom alongside: > 1 means the TV is composited in an
// HDR mode (the session's AVDisplayManager request landed see StreamViewIOS), and HDR // HDR mode (the session's AVDisplayManager request landed see StreamViewIOS), and HDR
@@ -259,6 +259,10 @@ public final class Stage2Pipeline {
private let endToEndMeter: LatencyMeter? private let endToEndMeter: LatencyMeter?
private let displayMeter: LatencyMeter? private let displayMeter: LatencyMeter?
private let recovery = KeyframeRecovery() private let recovery = KeyframeRecovery()
/// Post-loss freeze-until-reanchor gate (shared core policy via the C ABI). Created here seeded 0;
/// `start` reseeds it to the live connection's drop count. Captured by the decoder callbacks
/// (which withhold concealed frames) and driven by the pump (arm on a gap, poll per iteration).
private let gate = ReanchorGate(framesDropped: 0)
private var token = StopFlag() private var token = StopFlag()
private var offsetNs: Int64 = 0 private var offsetNs: Int64 = 0
/// Signalled when the pump thread exits, so `stop()` can join it (bounded) before `decoder.reset()` /// Signalled when the pump thread exits, so `stop()` can join it (bounded) before `decoder.reset()`
@@ -306,21 +310,29 @@ public final class Stage2Pipeline {
let ring = ring let ring = ring
let recovery = recovery let recovery = recovery
let renderSignal = renderSignal let renderSignal = renderSignal
let gate = gate
self.decoder = VideoDecoder( self.decoder = VideoDecoder(
onDecoded: { frame in onDecoded: { frame in
// Decode stage = receiveddecoded, both client CLOCK_REALTIME (offset 0 no // Decode stage = receiveddecoded, both client CLOCK_REALTIME (offset 0 no
// skew applies). Stamped at decode completion, so it covers every decoded frame, // skew applies). Stamped at decode completion, so it covers every decoded frame,
// including ones the newest-wins ring drops before present. // including ones the re-anchor gate withholds or the newest-wins ring drops.
decodeMeter?.record( decodeMeter?.record(
ptsNs: UInt64(frame.receivedNs), atNs: frame.decodedNs, offsetNs: 0) ptsNs: UInt64(frame.receivedNs), atNs: frame.decodedNs, offsetNs: 0)
// Freeze-until-reanchor: WITHHOLD a decoder-concealed post-loss frame (the gray/
// garbage VideoToolbox returns Ok for a reference-missing delta) don't submit it,
// so the CAMetalLayer keeps its last good drawable on glass. The gate lifts (returns
// present) on a proven clean re-anchor (IDR / RFI anchor / 2nd recovery mark) or the
// bounded backstop. decoderKeyframe=false: VT doesn't flag IDRs, the wire FLAG_SOF does.
guard gate.onDecoded(flags: frame.flags) else { return }
ring.submit(frame) ring.submit(frame)
// FRAME ARRIVAL is the render trigger (never the display link see the header). // FRAME ARRIVAL is the render trigger (never the display link see the header).
renderSignal.signal() renderSignal.signal()
}, },
// Async decode failure (a bad P-frame referencing a lost/corrupt IDR): the pump resets to // Async decode failure (a bad P-frame referencing a lost/corrupt IDR): fold it into the
// re-gate on the next IDR, and we ask the host to send one now (infinite GOP it wouldn't // gate's no-output streak (which arms the freeze after a short run, matching the desktop),
// and when that trips ask the host for a fresh IDR now (infinite GOP it wouldn't
// otherwise come soon). Throttled in KeyframeRecovery. // otherwise come soon). Throttled in KeyframeRecovery.
onDecodeError: { _ in recovery.request() }) onDecodeError: { _ in if gate.onNoOutput() { recovery.request() } })
} }
/// Start pulling AUs into the decoder. MAIN THREAD. `onFrame` fires per AU at receipt (the /// Start pulling AUs into the decoder. MAIN THREAD. `onFrame` fires per AU at receipt (the
@@ -334,6 +346,7 @@ public final class Stage2Pipeline {
) { ) {
offsetNs = connection.clockOffsetNs offsetNs = connection.clockOffsetNs
recovery.bind(connection) // arm host-keyframe recovery for this session recovery.bind(connection) // arm host-keyframe recovery for this session
gate.reseed(framesDropped: connection.framesDropped()) // baseline the freeze to this session
token = StopFlag() // fresh token per start a stop is permanent (like StreamPump) token = StopFlag() // fresh token per start a stop is permanent (like StreamPump)
// Configure the decoder's chroma + the layer's initial colorimetry before the first frame. The // Configure the decoder's chroma + the layer's initial colorimetry before the first frame. The
@@ -348,6 +361,7 @@ public final class Stage2Pipeline {
let recovery = recovery let recovery = recovery
let presenter = presenter let presenter = presenter
let pumpStopped = pumpStopped let pumpStopped = pumpStopped
let reanchorGate = gate
let thread = Thread { let thread = Thread {
defer { pumpStopped.signal() } // let stop() join the pump (bounded) before decoder.reset() defer { pumpStopped.signal() } // let stop() join the pump (bounded) before decoder.reset()
var format: CMVideoFormatDescription? var format: CMVideoFormatDescription?
@@ -379,6 +393,9 @@ public final class Stage2Pipeline {
awaitingIDR = true awaitingIDR = true
} }
if awaitingIDR { recovery.request() } if awaitingIDR { recovery.request() }
// Freeze backstop: a drop-count climb arms the gate (in case the frame-index gap
// below was itself lost), and an overdue freeze re-asks for the re-anchor.
if reanchorGate.poll(framesDropped: dropped) { recovery.request() }
// Drain HDR mastering metadata (0xCE) and hand it to the PRESENTER ( CAEDRMetadata). // Drain HDR mastering metadata (0xCE) and hand it to the PRESENTER ( CAEDRMetadata).
// Polled UNCONDITIONALLY (not gated on connection.isHDR, the fixed Welcome flag): the // Polled UNCONDITIONALLY (not gated on connection.isHDR, the fixed Welcome flag): the
// host sends 0xCE only for HDR, INCLUDING a mid-session SDRHDR transition (a game // host sends 0xCE only for HDR, INCLUDING a mid-session SDRHDR transition (a game
@@ -391,8 +408,10 @@ public final class Stage2Pipeline {
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference- // Loss recovery (RFI): a forward frame-index gap fires a throttled reference-
// frame-invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers // frame-invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers
// with a cheap clean P-frame instead of a full IDR. The framesDropped-driven // with a cheap clean P-frame instead of a full IDR. The framesDropped-driven
// recovery below stays the backstop for when the recovery frame itself is lost. // recovery above stays the backstop for when the recovery frame itself is lost.
connection.noteFrameIndex(au.frameIndex) // The same gap is the earliest, most precise signal to ARM the display freeze
// the following concealed frames are withheld until a clean re-anchor.
if connection.noteFrameIndexGap(au.frameIndex) { reanchorGate.arm() }
onFrame?(au) onFrame?(au)
if let f = connection.videoCodec.formatDescription(fromKeyframe: au.data) { if let f = connection.videoCodec.formatDescription(fromKeyframe: au.data) {
format = f // refreshed on every IDR (mode changes included) format = f // refreshed on every IDR (mode changes included)
@@ -28,6 +28,11 @@ final class StreamPump {
// Coalesced host keyframe requests (100 ms throttle see KeyframeRecovery). // Coalesced host keyframe requests (100 ms throttle see KeyframeRecovery).
let recovery = KeyframeRecovery() let recovery = KeyframeRecovery()
recovery.bind(connection) recovery.bind(connection)
// Post-loss freeze-until-reanchor (shared core policy via the C ABI). Stage-1 has no per-frame
// decode callback, so the gate is folded at ENQUEUE (from the AU's wire flags): a withheld
// frame is still enqueued but flagged DoNotDisplay so the layer's decoder keeps the reference
// chain fed while the last GOOD picture stays on glass until a clean re-anchor lifts it.
let gate = ReanchorGate(framesDropped: connection.framesDropped())
// The layer is non-Sendable but its enqueue/flush are documented thread-safe, and after // The layer is non-Sendable but its enqueue/flush are documented thread-safe, and after
// this point only the pump thread drives it assert that so the @Sendable Thread closure // this point only the pump thread drives it assert that so the @Sendable Thread closure
// may capture it. // may capture it.
@@ -77,13 +82,17 @@ final class StreamPump {
awaitingIDR = true awaitingIDR = true
} }
if awaitingIDR { recovery.request() } if awaitingIDR { recovery.request() }
// Freeze backstop: a drop-count climb arms the gate (should the frame-index gap
// below be lost too), and an overdue freeze re-asks for the re-anchor.
if gate.poll(framesDropped: dropped) { recovery.request() }
guard let au = try connection.nextAU(timeoutMs: 100) else { return true } guard let au = try connection.nextAU(timeoutMs: 100) else { return true }
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference- // Loss recovery (RFI): a forward frame-index gap fires a throttled reference-
// frame-invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers // frame-invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers
// with a cheap clean P-frame instead of a full IDR. The framesDropped-driven // with a cheap clean P-frame instead of a full IDR. The framesDropped-driven
// recovery above stays the backstop for when the recovery frame itself is lost. // recovery above stays the backstop for when the recovery frame itself is lost.
connection.noteFrameIndex(au.frameIndex) // The same gap is the earliest, most precise signal to ARM the display freeze.
if connection.noteFrameIndexGap(au.frameIndex) { gate.arm() }
onFrame?(au) onFrame?(au)
let idrFormat = connection.videoCodec.formatDescription(fromKeyframe: au.data) let idrFormat = connection.videoCodec.formatDescription(fromKeyframe: au.data)
if let f = idrFormat { if let f = idrFormat {
@@ -107,6 +116,7 @@ final class StreamPump {
// delta into a failed layer can't recover it. // delta into a failed layer can't recover it.
if !wasFailed { pumpLog.warning("video: display layer .failed — flushing + re-anchoring") } if !wasFailed { pumpLog.warning("video: display layer .failed — flushing + re-anchoring") }
layer.flush() layer.flush()
gate.arm() // a wedged decoder is a loss freeze until the re-anchor
if idrFormat == nil { if idrFormat == nil {
format = nil format = nil
awaitingIDR = true awaitingIDR = true
@@ -117,6 +127,13 @@ final class StreamPump {
let sample = connection.videoCodec.sampleBuffer(au: au, format: f), let sample = connection.videoCodec.sampleBuffer(au: au, format: f),
!token.isStopped // don't enqueue a stale frame after a restart !token.isStopped // don't enqueue a stale frame after a restart
else { return true } else { return true }
// Freeze-until-reanchor: while holding, WITHHOLD this concealed post-loss frame by
// flagging it DoNotDisplay the layer still decodes it (keeping the reference
// chain fed) but shows the last GOOD picture until a clean re-anchor lifts the
// gate. Folded from the AU's wire flags (stage-1 has no decode callback).
if !gate.onDecoded(flags: au.flags) {
StreamPump.setDoNotDisplay(sample)
}
layer.enqueue(sample) layer.enqueue(sample)
return true return true
} catch { } catch {
@@ -133,6 +150,21 @@ final class StreamPump {
thread.start() thread.start()
} }
/// Flag a sample decode-but-don't-display (`kCMSampleAttachmentKey_DoNotDisplay`). Used to
/// withhold decoder-concealed post-loss frames while the re-anchor gate holds: the layer keeps
/// its reference chain fed without flipping the frozen picture. No-op if the attachments array
/// can't be materialized (then the frame just displays the freeze degrades to the old behavior).
private static func setDoNotDisplay(_ sample: CMSampleBuffer) {
guard let attachments = CMSampleBufferGetSampleAttachmentsArray(
sample, createIfNecessary: true), CFArrayGetCount(attachments) > 0
else { return }
let dict = unsafeBitCast(CFArrayGetValueAtIndex(attachments, 0), to: CFMutableDictionary.self)
CFDictionarySetValue(
dict,
Unmanaged.passUnretained(kCMSampleAttachmentKey_DoNotDisplay).toOpaque(),
Unmanaged.passUnretained(kCFBooleanTrue).toOpaque())
}
/// Stop pumping ( one poll timeout). Does not close the connection. /// Stop pumping ( one poll timeout). Does not close the connection.
func stop() { func stop() {
token.stop() token.stop()
@@ -27,19 +27,40 @@ public struct ReadyFrame: @unchecked Sendable {
/// True when the stream is HDR (BT.2020 PQ): the buffer is 10-bit P010 and the presenter must /// True when the stream is HDR (BT.2020 PQ): the buffer is 10-bit P010 and the presenter must
/// configure EDR + BT.2020 PQ output. Derived from the decoded buffer's pixel format. /// configure EDR + BT.2020 PQ output. Derived from the decoded buffer's pixel format.
public let isHDR: Bool public let isHDR: Bool
/// The AU's wire `user_flags` (`AccessUnit.flags`), threaded through the decode via the frame
/// context so the re-anchor gate can classify this decoded frame (IDR / RFI anchor / recovery
/// mark) at present time the async decode callback has no other access to it. 0 when unknown.
public let flags: UInt32
}
/// Per-frame context threaded through the VideoToolbox frame refcon: the AU's receipt instant (for
/// the decode-stage meter) and its wire `user_flags` (for the re-anchor gate). Retained across the
/// async decode and reclaimed exactly once by the output callback for every frame VideoToolbox
/// accepts, or by `decode`'s error branch for a frame `DecodeFrame` rejected outright (the callback
/// then never fires). A tiny per-frame allocation, the price of smuggling two values (a 64-bit
/// instant plus the flags) through the single `void*` a bit-pattern scalar can't hold.
private final class FrameContext {
let receivedNs: Int64
let flags: UInt32
init(receivedNs: Int64, flags: UInt32) {
self.receivedNs = receivedNs
self.flags = flags
}
} }
/// The C output callback can't capture context, so VideoToolbox hands it the refcon we set at /// The C output callback can't capture context, so VideoToolbox hands it the refcon we set at
/// session creation a pointer back to the owning `VideoDecoder`. The per-frame refcon carries /// session creation a pointer back to the owning `VideoDecoder`. The per-frame refcon is the
/// the AU's `receivedNs` as a pointer bit pattern (a scalar smuggled through the C void*, never /// retained `FrameContext` set at submit; reclaim it here (balancing `passRetained`) and unpack the
/// dereferenced) so the decode stage can be computed against decode-completion. /// AU's receipt instant (for the decode stage) and wire flags (for the re-anchor gate).
private let decoderOutputCallback: VTDecompressionOutputCallback = { private let decoderOutputCallback: VTDecompressionOutputCallback = {
refcon, frameRefcon, status, _, imageBuffer, pts, _ in refcon, frameRefcon, status, _, imageBuffer, pts, _ in
guard let refcon else { return } guard let refcon else { return }
let receivedNs = frameRefcon.map { Int64(Int(bitPattern: $0)) } ?? 0 let ctx = frameRefcon.map { Unmanaged<FrameContext>.fromOpaque($0).takeRetainedValue() }
Unmanaged<VideoDecoder>.fromOpaque(refcon) Unmanaged<VideoDecoder>.fromOpaque(refcon)
.takeUnretainedValue() .takeUnretainedValue()
.handleDecoded(status: status, imageBuffer: imageBuffer, pts: pts, receivedNs: receivedNs) .handleDecoded(
status: status, imageBuffer: imageBuffer, pts: pts,
receivedNs: ctx?.receivedNs ?? 0, flags: ctx?.flags ?? 0)
} }
/// Owns a `VTDecompressionSession` rebuilt whenever the format description changes (every IDR / /// Owns a `VTDecompressionSession` rebuilt whenever the format description changes (every IDR /
@@ -117,16 +138,21 @@ public final class VideoDecoder: @unchecked Sendable {
let sample = codec.sampleBuffer(au: au, format: newFormat) let sample = codec.sampleBuffer(au: au, format: newFormat)
else { lock.unlock(); return false } else { lock.unlock(); return false }
var infoOut = VTDecodeInfoFlags() var infoOut = VTDecodeInfoFlags()
// The AU's receipt instant + wire flags ride through as a retained context; the output
// callback reclaims it. Retain immediately before submit so no early return can leak it.
let ctx = FrameContext(receivedNs: au.receivedNs, flags: au.flags)
let refcon = Unmanaged.passRetained(ctx).toOpaque()
let status = VTDecompressionSessionDecodeFrame( let status = VTDecompressionSessionDecodeFrame(
session, session,
sampleBuffer: sample, sampleBuffer: sample,
flags: [._EnableAsynchronousDecompression], flags: [._EnableAsynchronousDecompression],
// The AU's receipt instant rides through as a bit pattern (nil for 0 the output frameRefcon: refcon,
// callback maps that back to 0); the callback needs it to stamp the decode stage.
frameRefcon: UnsafeMutableRawPointer(bitPattern: Int(au.receivedNs)),
infoFlagsOut: &infoOut) infoFlagsOut: &infoOut)
lock.unlock() lock.unlock()
if status != noErr { if status != noErr {
// DecodeFrame rejected the frame outright the output callback will NOT fire, so
// reclaim the context here (balancing passRetained) to avoid leaking it.
Unmanaged<FrameContext>.fromOpaque(refcon).release()
onDecodeError(status) onDecodeError(status)
return false return false
} }
@@ -231,9 +257,10 @@ public final class VideoDecoder: @unchecked Sendable {
} }
/// VT thread. Stamp decode-completion and enqueue, or report the error. `receivedNs` is the /// VT thread. Stamp decode-completion and enqueue, or report the error. `receivedNs` is the
/// AU's receipt instant threaded through the frame refcon (0 = unknown). /// AU's receipt instant and `flags` its wire `user_flags`, both threaded through the frame refcon
/// (0 = unknown).
fileprivate func handleDecoded( fileprivate func handleDecoded(
status: OSStatus, imageBuffer: CVImageBuffer?, pts: CMTime, receivedNs: Int64 status: OSStatus, imageBuffer: CVImageBuffer?, pts: CMTime, receivedNs: Int64, flags: UInt32
) { ) {
guard status == noErr, let imageBuffer else { guard status == noErr, let imageBuffer else {
onDecodeError(status) onDecodeError(status)
@@ -259,6 +286,6 @@ public final class VideoDecoder: @unchecked Sendable {
onDecoded( onDecoded(
ReadyFrame( ReadyFrame(
ptsNs: ptsNs, receivedNs: receivedNs, decodedNs: decodedNs, ptsNs: ptsNs, receivedNs: receivedNs, decodedNs: decodedNs,
pixelBuffer: imageBuffer, isHDR: isHDR)) pixelBuffer: imageBuffer, isHDR: isHDR, flags: flags))
} }
} }
@@ -661,15 +661,16 @@ public final class StreamLayerView: NSView {
DispatchQueue.main.async { self?.noteDecodedContentSize(width: w, height: h) } DispatchQueue.main.async { self?.noteDecodedContentSize(width: w, height: h) }
overlayDecodedSize?(w, h) overlayDecodedSize?(w, h)
}) })
// Match-window (C3): follow the window's pixel size DEFAULT ON, so a windowed session // Match-window (C3): when ON, follow the window's pixel size so a windowed session streams
// streams 1:1 (pixel-exact) instead of the presenter resampling a fixed-mode frame into a // 1:1 (pixel-exact) instead of the presenter resampling a fixed-mode frame into a
// non-matching window. The first real `layout()` feeds the initial size, so the stream // non-matching window. The first real `layout()` feeds the initial size, so the stream
// converges to the window even though the connect used the explicit/display mode; entering // converges to the window even though the connect used the explicit/display mode; entering
// fullscreen reports the full-display px, restoring a native-res 1:1 present there too. // fullscreen reports the full-display px, restoring a native-res 1:1 present there too.
// `?? true` so an unset default matches the Settings toggle (which also defaults on). // OPT-IN `?? false` matches the Settings toggle (which also defaults off); an unset
// default keeps the explicit mode.
let follower = MatchWindowFollower( let follower = MatchWindowFollower(
connection: connection, connection: connection,
enabled: UserDefaults.standard.object(forKey: DefaultsKey.matchWindow) as? Bool ?? true) enabled: UserDefaults.standard.object(forKey: DefaultsKey.matchWindow) as? Bool ?? false)
follower.onResizeTarget = onResizeTarget // resize overlay START signal (instant, on the follower) follower.onResizeTarget = onResizeTarget // resize overlay START signal (instant, on the follower)
matchFollower = follower matchFollower = follower
layoutPresenter() layoutPresenter()
@@ -24,7 +24,9 @@
// (== locked): GCMouse forwards only WHILE locked, the UIKit indirect path (motion, buttons AND // (== locked): GCMouse forwards only WHILE locked, the UIKit indirect path (motion, buttons AND
// scroll) only while NOT locked so a pointer that emits both channels under lock can't double-send. // scroll) only while NOT locked so a pointer that emits both channels under lock can't double-send.
// Hardware keyboard forwarding shares InputCapture with macOS auto-engaged when streaming // Hardware keyboard forwarding shares InputCapture with macOS auto-engaged when streaming
// starts, toggles (detected from the HID stream; there is no NSEvent monitor here). // starts, toggles and Q releases (both detected from the HID stream; there is no NSEvent
// monitor here). Q is the cross-client Ctrl+Alt+Shift+Q it un-captures so the Magic Keyboard
// trackpad drives the local iPad UI again.
// //
// The public type is named StreamView like its macOS twin (each is platform-gated), so // The public type is named StreamView like its macOS twin (each is platform-gated), so
// the SwiftUI app layer is identical on both platforms. // the SwiftUI app layer is identical on both platforms.
@@ -337,7 +339,19 @@ public final class StreamViewController: StreamViewControllerBase {
x: p.x, y: p.y, surfaceWidth: p.w, surfaceHeight: p.h) x: p.x, y: p.y, surfaceWidth: p.w, surfaceHeight: p.h)
} }
streamView.onPointerButton = { [weak self] button, down in streamView.onPointerButton = { [weak self] button, down in
guard let self, self.inputCapture?.gcMouseForwarding == false else { return } guard let self else { return }
// Released a trackpad/mouse click into the video RE-ENGAGES capture (the iPad
// analogue of macOS's `mouseDown engageCapture(fromClick:)`, and the click-mirror of
// the / Q keyboard toggles). Only the button-DOWN engages; that click is the local
// engage gesture, so it's suppressed toward the host (`fromClick`) and never forwarded
// its release is swallowed by InputCapture's suppress latch, whichever path delivers it.
// (Finger taps are untouched: touch always plays directly, so only the indirect pointer
// re-captures.) Captured already the absolute path forwards the button as before.
if !self.captured {
if down, self.captureEnabled { self.setCaptured(true, fromClick: true) }
return
}
guard self.inputCapture?.gcMouseForwarding == false else { return }
self.inputCapture?.sendMouseButton(button, pressed: down) self.inputCapture?.sendMouseButton(button, pressed: down)
} }
streamView.onScroll = { [weak self] dx, dy in streamView.onScroll = { [weak self] dx, dy in
@@ -350,19 +364,27 @@ public final class StreamViewController: StreamViewControllerBase {
guard let self else { return } guard let self else { return }
self.setCaptured(!self.captured) self.setCaptured(!self.captured)
} }
// Q (cross-client parity with macOS/Windows/Linux) releases the captured pointer +
// keyboard so the Magic Keyboard trackpad returns to driving the local iPad UI. Detected
// from the HID stream in InputCapture (no NSEvent monitor on iOS); unlike the toggle it
// only ever RELEASES re-pressing it while already released is a no-op (setCaptured guards).
capture.onReleaseCapture = { [weak self] in
self?.setCaptured(false)
}
capture.onPreempted = { [weak self] in capture.onPreempted = { [weak self] in
self?.setCaptured(false) self?.setCaptured(false)
} }
capture.start() capture.start()
inputCapture = capture inputCapture = capture
// Match-window (C3): follow the scene's pixel size DEFAULT ON, so a resizable iPad scene // Match-window (C3): when ON, follow the scene's pixel size so a resizable iPad scene
// streams 1:1 (pixel-exact) instead of the presenter resampling a fixed-mode frame into it. // streams 1:1 (pixel-exact) instead of the presenter resampling a fixed-mode frame into it.
// `viewDidLayoutSubviews` feeds it covers Stage Manager / Split View resizes and rotation. // `viewDidLayoutSubviews` feeds it covers Stage Manager / Split View resizes and rotation.
// iPhone is a fixed full-screen scene, so this naturally no-ops (reports the device mode). // iPhone is a fixed full-screen scene, so this naturally no-ops (reports the device mode).
// `?? true` so an unset default matches the Settings toggle (which also defaults on). // OPT-IN `?? false` matches the Settings toggle (which also defaults off); an unset
// default keeps the explicit mode.
let follower = MatchWindowFollower( let follower = MatchWindowFollower(
connection: connection, connection: connection,
enabled: UserDefaults.standard.object(forKey: DefaultsKey.matchWindow) as? Bool ?? true) enabled: UserDefaults.standard.object(forKey: DefaultsKey.matchWindow) as? Bool ?? false)
follower.onResizeTarget = onResizeTarget follower.onResizeTarget = onResizeTarget
matchFollower = follower matchFollower = follower
#endif #endif
@@ -422,6 +444,19 @@ public final class StreamViewController: StreamViewControllerBase {
) { [weak self] _ in ) { [weak self] _ in
self?.syncPointerLock() self?.syncPointerLock()
}) })
// The Stream menu's "Release Mouse" (Q) posts this the discoverable menu surface for
// the RELEASED state. While CAPTURED the combo is recognized from the HID stream in
// InputCapture (onReleaseCapture) before the menu sees it, so in practice this fires as a
// not-captured no-op (setCaptured guards it); wired for honesty + a non-GC fallback. Only the
// foreground-active scene's stream acts the iPad analogue of macOS's key-window guard, so a
// second Stage Manager scene isn't released out from under the user.
observers.append(NotificationCenter.default.addObserver(
forName: .punktfunkReleaseCapture, object: nil, queue: .main
) { [weak self] _ in
guard let self,
self.view.window?.windowScene?.activationState == .foregroundActive else { return }
self.setCaptured(false)
})
if captureEnabled { if captureEnabled {
setCaptured(true) // entering a session is the deliberate "capture me" moment setCaptured(true) // entering a session is the deliberate "capture me" moment
@@ -556,11 +591,15 @@ public final class StreamViewController: StreamViewControllerBase {
} }
#if os(iOS) #if os(iOS)
private func setCaptured(_ on: Bool) { /// `fromClick` marks a click-driven engage (the released-state pointer click that re-captures):
/// that click's press/release are suppressed toward the host it's the local engage gesture,
/// not a host click exactly as macOS's `engageCapture(fromClick:)` does. Keyboard-driven
/// engages () pass false so a normal click still reaches the host.
private func setCaptured(_ on: Bool, fromClick: Bool = false) {
if on { if on {
// `connection != nil` is the session-active gate (presenter internals are opaque here). // `connection != nil` is the session-active gate (presenter internals are opaque here).
guard captureEnabled, !captured, connection != nil else { return } guard captureEnabled, !captured, connection != nil else { return }
inputCapture?.setForwarding(true) inputCapture?.setForwarding(true, suppressClick: fromClick)
captured = true captured = true
} else { } else {
guard captured else { return } guard captured else { return }
@@ -3,6 +3,7 @@
// player-LED-bits GCControllerPlayerIndex map. All pure functions. // player-LED-bits GCControllerPlayerIndex map. All pure functions.
import GameController import GameController
import PunktfunkCore
import XCTest import XCTest
@testable import PunktfunkKit @testable import PunktfunkKit
@@ -40,6 +41,43 @@ final class GamepadWireTests: XCTestCase {
XCTAssertEqual(GamepadWire.axisRT, 5) XCTAssertEqual(GamepadWire.axisRT, 5)
} }
func testPadIndexRidesFlagsOnEveryPerPadEvent() {
// The wire pad index is the low byte of `flags` (punktfunk_core::input) on button + axis.
let btn = PunktfunkInputEvent.gamepadButton(GamepadWire.a, down: true, pad: 3)
XCTAssertEqual(btn.kind, UInt8(PUNKTFUNK_INPUT_KIND_GAMEPAD_BUTTON.rawValue))
XCTAssertEqual(btn.code, GamepadWire.a)
XCTAssertEqual(btn.x, 1)
XCTAssertEqual(btn.flags, 3)
let axis = PunktfunkInputEvent.gamepadAxis(GamepadWire.axisRT, value: 200, pad: 5)
XCTAssertEqual(axis.kind, UInt8(PUNKTFUNK_INPUT_KIND_GAMEPAD_AXIS.rawValue))
XCTAssertEqual(axis.code, GamepadWire.axisRT)
XCTAssertEqual(axis.x, 200)
XCTAssertEqual(axis.flags, 5)
// Single-controller path stays byte-identical: pad 0 flags 0, exactly as before.
XCTAssertEqual(PunktfunkInputEvent.gamepadButton(GamepadWire.a, down: false, pad: 0).flags, 0)
XCTAssertEqual(PunktfunkInputEvent.gamepadAxis(GamepadWire.axisLSX, value: 0, pad: 0).flags, 0)
}
func testArrivalAndRemoveWireLayout() {
// GamepadArrival (kind 14): code = the GamepadType wire byte, flags = pad index.
let arrival = PunktfunkInputEvent.gamepadArrival(
pref: PunktfunkConnection.GamepadType.dualSense.rawValue, pad: 2)
XCTAssertEqual(arrival.kind, UInt8(PUNKTFUNK_INPUT_KIND_GAMEPAD_ARRIVAL.rawValue))
XCTAssertEqual(arrival.code, PunktfunkConnection.GamepadType.dualSense.rawValue) // 2
XCTAssertEqual(arrival.flags, 2)
// The GamepadType raw values ARE the GamepadPref wire bytes the host resolves.
XCTAssertEqual(PunktfunkConnection.GamepadType.xbox360.rawValue, 1)
XCTAssertEqual(PunktfunkConnection.GamepadType.dualSense.rawValue, 2)
XCTAssertEqual(PunktfunkConnection.GamepadType.xboxOne.rawValue, 3)
XCTAssertEqual(PunktfunkConnection.GamepadType.dualShock4.rawValue, 4)
// GamepadRemove (kind 13): flags = pad index (the core stamps the per-pad seq).
let remove = PunktfunkInputEvent.gamepadRemove(pad: 7)
XCTAssertEqual(remove.kind, UInt8(PUNKTFUNK_INPUT_KIND_GAMEPAD_REMOVE.rawValue))
XCTAssertEqual(remove.flags, 7)
// 16 addressable pads (punktfunk_core::input::MAX_PADS).
XCTAssertEqual(GamepadWire.maxPads, 16)
}
func testTouchpadConversionCorners() { func testTouchpadConversionCorners() {
// GC ±1 with +y up wire 0...65535 with origin top-left, +y down. // GC ±1 with +y up wire 0...65535 with origin top-left, +y down.
let topLeft = GamepadWire.touchpad(x: -1, y: 1) let topLeft = GamepadWire.touchpad(x: -1, y: 1)
+7 -6
View File
@@ -347,13 +347,14 @@ pub fn show(
stream.add(stats_row.widget()); stream.add(stats_row.widget());
let input = adw::PreferencesGroup::builder().title("Input").build(); let input = adw::PreferencesGroup::builder().title("Input").build();
// Which physical controller forwards as pad 0: automatic = the most recently connected // Controller forwarding: Automatic forwards EVERY real controller, each as its own pad
// real pad (Steam's virtual pad skipped). A pin is persisted by stable key // (Steam's virtual pad skipped); pinning one restricts the session to that single
// (`Settings::forward_pad`), so it survives restarts — and disconnects: an offline // controller (single-player). The pin is persisted by stable key (`Settings::forward_pad`),
// pinned pad keeps its entry here instead of silently snapping back to Automatic. // so it survives restarts — and disconnects: an offline pinned pad keeps its entry here
// instead of silently snapping back to Automatic.
let pads = gamepads.pads(); let pads = gamepads.pads();
let saved_pin = settings.borrow().forward_pad.clone(); let saved_pin = settings.borrow().forward_pad.clone();
let mut pad_names = vec!["Automatic (most recent)".to_string()]; let mut pad_names = vec!["Automatic (all controllers)".to_string()];
let mut pad_keys: Vec<String> = Vec::new(); let mut pad_keys: Vec<String> = Vec::new();
for p in &pads { for p in &pads {
let kind = p.kind_label(); let kind = p.kind_label();
@@ -379,7 +380,7 @@ pub fn show(
if pads.is_empty() { if pads.is_empty() {
"No controllers detected" "No controllers detected"
} else { } else {
"Exactly one controller is forwarded to the host" "All controllers are forwarded, each as its own player; pick one to force single-player"
}, },
&pad_names.iter().map(String::as_str).collect::<Vec<_>>(), &pad_names.iter().map(String::as_str).collect::<Vec<_>>(),
); );
+4 -11
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "punktfunk-client-windows" name = "punktfunk-client-windows"
description = "Native Windows punktfunk/1 client — WinUI 3 (windows-reactor) shell, D3D11/SwapChainPanel present, FFmpeg decode, WASAPI audio, SDL3 gamepads" description = "Native Windows punktfunk/1 client — WinUI 3 (windows-reactor) shell, SDL3 gamepads; streaming runs in the spawned punktfunk-session binary"
version.workspace = true version.workspace = true
edition.workspace = true edition.workspace = true
rust-version.workspace = true rust-version.workspace = true
@@ -57,13 +57,10 @@ windows = { git = "https://github.com/microsoft/windows-rs", rev = "a4f7b2cb7c63
"Win32_UI_WindowsAndMessaging", "Win32_UI_WindowsAndMessaging",
] } ] }
# Video decode (same FFmpeg pin as the host/Linux client) — software HEVC on the GPU-less dev # FFmpeg — used only to enumerate which codecs this client can decode (probe::decodable_codecs),
# box; D3D11VA hardware decode is a follow-up for the real-GPU box. # advertised to the host on the speed-test connect. Same pin as the host/Linux client. (Real
# decode + present live in the spawned punktfunk-session binary.)
ffmpeg-next = "8" ffmpeg-next = "8"
opus = "0.3"
# Audio render + mic capture (the WASAPI analogue of the Linux client's PipeWire backend).
wasapi = "0.23"
# Gamepads: capture + feedback (full DualSense fidelity needs hidapi). SDL3 is cross-platform; # Gamepads: capture + feedback (full DualSense fidelity needs hidapi). SDL3 is cross-platform;
# built from source via the bundled CMake on Windows (no system SDL3). # built from source via the bundled CMake on Windows (no system SDL3).
@@ -71,12 +68,8 @@ sdl3 = { version = "0.18", features = ["build-from-source", "hidapi"] }
mdns-sd = "0.20" mdns-sd = "0.20"
async-channel = "2" async-channel = "2"
# The decoded-frame channel (session pump → render thread): crossbeam because the render loop
# blocks with `recv_timeout`, which async-channel has no sync analogue of.
crossbeam-channel = "0.5"
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
anyhow = "1"
tracing = "0.1" tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+5 -205
View File
@@ -6,10 +6,7 @@
use super::style::*; use super::style::*;
use super::{AppCtx, Screen, Svc, Target}; use super::{AppCtx, Screen, Svc, Target};
use crate::discovery::DiscoveredHost; use crate::discovery::DiscoveredHost;
use crate::session::{self, SessionEvent, SessionParams, Stats}; use crate::trust::{self, KnownHost, KnownHosts};
use crate::trust::{self, KnownHost, KnownHosts, Settings};
use crate::video::DecoderPref;
use punktfunk_core::config::{CompositorPref, GamepadPref, Mode};
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
@@ -117,82 +114,6 @@ pub(crate) fn initiate_launch(
); );
} }
/// The mode to request: explicit settings, with `0` fields resolved to the native size/refresh
/// of the display our window is on (mirrors the Linux/Swift clients' native-display default).
pub(crate) fn resolve_mode(s: &Settings) -> Mode {
let mut mode = Mode {
width: s.width,
height: s.height,
refresh_hz: s.refresh_hz,
};
if mode.width == 0 || mode.refresh_hz == 0 {
if let Some((w, h, hz)) = current_display_mode() {
if mode.width == 0 {
(mode.width, mode.height) = (w, h);
}
if mode.refresh_hz == 0 {
mode.refresh_hz = hz;
}
}
}
// No display info (headless session, RDP oddities) — a sane floor.
if mode.width == 0 {
(mode.width, mode.height) = (1920, 1080);
}
if mode.refresh_hz == 0 {
mode.refresh_hz = 60;
}
mode
}
/// The current mode (physical pixels + refresh) of the display our window occupies:
/// `MonitorFromWindow` on the foreground window — ours, the user just clicked in it — then
/// `EnumDisplaySettingsW(ENUM_CURRENT_SETTINGS)` on that monitor's device. Defaults to the
/// primary display when we're not foreground (e.g. a scripted connect).
fn current_display_mode() -> Option<(u32, u32, u32)> {
use windows::core::PCWSTR;
use windows::Win32::Graphics::Gdi::{
EnumDisplaySettingsW, GetMonitorInfoW, MonitorFromWindow, DEVMODEW, ENUM_CURRENT_SETTINGS,
MONITORINFO, MONITORINFOEXW,
};
use windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow;
unsafe {
let monitor = MonitorFromWindow(
GetForegroundWindow(),
windows::Win32::Graphics::Gdi::MONITOR_DEFAULTTOPRIMARY,
);
let mut info = MONITORINFOEXW::default();
info.monitorInfo.cbSize = std::mem::size_of::<MONITORINFOEXW>() as u32;
if !GetMonitorInfoW(
monitor,
&mut info as *mut MONITORINFOEXW as *mut MONITORINFO,
)
.as_bool()
{
return None;
}
let mut dm = DEVMODEW {
dmSize: std::mem::size_of::<DEVMODEW>() as u16,
..Default::default()
};
if !EnumDisplaySettingsW(
PCWSTR(info.szDevice.as_ptr()),
ENUM_CURRENT_SETTINGS,
&mut dm,
)
.as_bool()
{
return None;
}
// dmDisplayFrequency of 0/1 means "hardware default" — unusable as a mode request.
(dm.dmPelsWidth > 0 && dm.dmDisplayFrequency > 1).then_some((
dm.dmPelsWidth,
dm.dmPelsHeight,
dm.dmDisplayFrequency,
))
}
}
/// Tunables that differ between the normal connect and the no-PIN "request access" flow. /// Tunables that differ between the normal connect and the no-PIN "request access" flow.
/// `Default` is the normal connect: short handshake budget, persist *unpaired* on TOFU, and the /// `Default` is the normal connect: short handshake budget, persist *unpaired* on TOFU, and the
/// plain "Connecting" screen. /// plain "Connecting" screen.
@@ -220,9 +141,7 @@ pub(crate) struct ConnectOpts {
/// so it can't loop. /// so it can't loop.
wake_on_fail: bool, wake_on_fail: bool,
/// A library title id (`steam:570`, …) the host launches during the connect handshake — /// A library title id (`steam:570`, …) the host launches during the connect handshake —
/// the library page's tap-to-play. Spawn mode passes it as `--launch`; the legacy /// the library page's tap-to-play, passed to the spawned session child as `--launch`.
/// in-process path has no launch plumbing (it predates the library and is slated for
/// deletion).
launch: Option<String>, launch: Option<String>,
} }
@@ -265,128 +184,11 @@ fn connect_with(
opts: ConnectOpts, opts: ConnectOpts,
) { ) {
// Session-always: every stream runs in the spawned punktfunk-session Vulkan binary. // Session-always: every stream runs in the spawned punktfunk-session Vulkan binary.
// The in-process D3D11VA path below stays reachable via the "Streaming engine" connect_spawn(ctx, target, pin, set_screen, set_status, opts)
// setting / PUNKTFUNK_BUILTIN_STREAM=1 as the A/B baseline until its deletion.
if !super::use_builtin_stream(ctx) {
return connect_spawn(ctx, target, pin, set_screen, set_status, opts);
}
let s = ctx.settings.lock().unwrap().clone();
let gamepad_pref = match GamepadPref::from_name(&s.gamepad) {
Some(GamepadPref::Auto) | None => ctx.gamepad.auto_pref(),
Some(explicit) => explicit,
};
let handle = session::start(SessionParams {
host: target.addr.clone(),
port: target.port,
mode: resolve_mode(&s),
compositor: CompositorPref::from_name(&s.compositor).unwrap_or(CompositorPref::Auto),
gamepad: gamepad_pref,
bitrate_kbps: s.bitrate_kbps,
audio_channels: s.audio_channels,
mic_enabled: s.mic_enabled,
hdr_enabled: s.hdr_enabled,
decoder: DecoderPref::from_name(&s.decoder),
preferred_codec: s.preferred_codec(),
pin,
identity: ctx.identity.clone(),
connect_timeout: opts.connect_timeout,
});
set_status.call(String::new());
set_screen.call(if opts.awaiting_approval {
Screen::RequestAccess
} else {
Screen::Connecting
});
let tofu = pin.is_none();
let persist_paired = opts.persist_paired;
let cancel = opts.cancel;
let wake_on_fail = opts.wake_on_fail;
let ctx = ctx.clone();
let (shared, gamepad) = (ctx.shared.clone(), ctx.gamepad.clone());
let (ss, st) = (set_screen.clone(), set_status.clone());
let target = target.clone();
std::thread::spawn(move || loop {
let event = match handle.events.recv_blocking() {
Ok(e) => e,
Err(_) => {
gamepad.detach();
ss.call(Screen::Hosts);
break;
}
};
// A cancelled request-access connect that resolved late (the host approved or the park
// timed out after the user walked away): tear down silently. Cancel already returned the
// UI to the host list; dropping `event` (and with it any connector) closes the connection
// without popping a stream or a stray error over the screen a new session may own.
if cancel.as_ref().is_some_and(|c| c.load(Ordering::SeqCst)) {
break;
}
match event {
SessionEvent::Connected {
connector,
fingerprint,
..
} => {
if persist_paired || tofu {
// Request-access: the operator approved this device, so record the host as a
// trusted PAIRED host — future connects are then silent (rule 1), exactly like
// after a PIN ceremony. A plain TOFU connect persists it *unpaired* (pinned).
let mut k = KnownHosts::load();
k.upsert(KnownHost {
name: target.name.clone(),
addr: target.addr.clone(),
port: target.port,
fp_hex: trust::hex(&fingerprint),
paired: persist_paired,
last_used: None,
mac: target.mac.clone(),
});
let _ = k.save();
}
trust::touch_last_used(&trust::hex(&fingerprint));
gamepad.attach(connector.clone());
*shared.stats.lock().unwrap() = Stats::default(); // clear any prior session's numbers
*shared.handoff.lock().unwrap() =
Some((connector, handle.frames.clone(), handle.stop.clone()));
ss.call(Screen::Stream);
}
SessionEvent::Failed {
msg,
trust_rejected,
} => {
st.call(msg);
gamepad.detach();
if trust_rejected {
// Pinned-fingerprint mismatch / pairing required → re-pair via the PIN screen.
// The host ANSWERED, so this never takes the wake fallback.
*shared.target.lock().unwrap() = target.clone();
ss.call(Screen::Pair);
} else if wake_on_fail {
// The dial-first attempt to a non-advertising host failed — it may genuinely
// be asleep. NOW wake and wait (its resolved redial uses default opts, so a
// second failure lands on the host list, not back here).
wake_and_connect(&ctx, target.clone(), &ss, &st);
} else {
ss.call(Screen::Hosts);
}
break;
}
SessionEvent::Ended(err) => {
// `None` = the user ended the session themselves (the disconnect shortcut) —
// return to the host list silently; an error banner would read as a failure.
st.call(err.unwrap_or_default());
gamepad.detach();
ss.call(Screen::Hosts);
break;
}
SessionEvent::Stats(s) => *shared.stats.lock().unwrap() = s,
}
});
} }
/// Spawn-mode connect: run the stream in the punktfunk-session binary and translate its /// Spawn-mode connect: run the stream in the punktfunk-session binary and translate its
/// stdout contract into the same navigation the in-process event loop drove. The child /// stdout contract into the app's connect-flow navigation. The child
/// NEVER connects unpinned — a stored/ceremony pin, else the host's advertised /// NEVER connects unpinned — a stored/ceremony pin, else the host's advertised
/// fingerprint (TOFU: persisted once the child reports ready, which proves the host /// fingerprint (TOFU: persisted once the child reports ready, which proves the host
/// really holds that identity, mirroring the GTK shell); no fingerprint at all routes to /// really holds that identity, mirroring the GTK shell); no fingerprint at all routes to
@@ -723,9 +525,7 @@ pub(crate) fn request_access_page(
.on_click(move || { .on_click(move || {
// Return the UI immediately; trip the flag this request's event loop // Return the UI immediately; trip the flag this request's event loop
// captured so it tears down silently when the connect resolves (see // captured so it tears down silently when the connect resolves (see
// ConnectOpts::cancel). Spawn mode: killing the parked child IS the abort // ConnectOpts::cancel). Killing the parked session child IS the abort.
// (builtin mode's in-process connect is blocking with none — it just
// resolves/times out later).
if let Some(c) = ctx.shared.cancel.lock().unwrap().as_ref() { if let Some(c) = ctx.shared.cancel.lock().unwrap().as_ref() {
c.store(true, Ordering::SeqCst); c.store(true, Ordering::SeqCst);
} }
+2 -4
View File
@@ -1,8 +1,7 @@
//! The Shortcuts screen: a short note on the in-stream capture model plus a reference of the //! The Shortcuts screen: a short note on the in-stream capture model plus a reference of the
//! keyboard shortcuts — reached from the Shortcuts button on the host list. The Windows //! keyboard shortcuts — reached from the Shortcuts button on the host list. The Windows
//! counterpart of the GTK client's Keyboard Shortcuts window; the bindings themselves live in //! counterpart of the GTK client's Keyboard Shortcuts window; the bindings themselves live in
//! the session window (and [`crate::input`] for the legacy builtin path), so both clients //! the session window, so both clients document the same set.
//! document the same set.
use super::style::*; use super::style::*;
use super::Screen; use super::Screen;
@@ -10,8 +9,7 @@ use windows_reactor::*;
/// The in-stream keyboard shortcuts, in the GTK Shortcuts window's order: the chord, then what it /// The in-stream keyboard shortcuts, in the GTK Shortcuts window's order: the chord, then what it
/// does. Read-only — the keyboard bindings live in the session window (`pf-presenter`'s run /// does. Read-only — the keyboard bindings live in the session window (`pf-presenter`'s run
/// loop; the legacy builtin path's in [`crate::input`]), the controller chord in its gamepad /// loop), the controller chord in its gamepad service.
/// service.
const STREAM_SHORTCUTS: &[(&str, &str)] = &[ const STREAM_SHORTCUTS: &[(&str, &str)] = &[
("F11 / Alt+Enter", "Toggle fullscreen"), ("F11 / Alt+Enter", "Toggle fullscreen"),
( (
+15 -42
View File
@@ -1,8 +1,8 @@
//! The WinUI 3 (windows-reactor) application shell. //! The WinUI 3 (windows-reactor) application shell.
//! //!
//! Declarative React-like model: this root component routes on a `Screen` value held in //! Declarative React-like model: this root component routes on a `Screen` value held in
//! `use_async_state` so background threads (discovery, the session pump) can drive navigation. //! `use_async_state` so background threads (discovery, the spawned session's stdout reader) can
//! Each screen lives in its own submodule: //! drive navigation. Each screen lives in its own submodule:
//! //!
//! * [`hosts`] — saved/discovered/manual host list, plus per-host forget + speed test //! * [`hosts`] — saved/discovered/manual host list, plus per-host forget + speed test
//! * [`connect`] — the trust gate and session lifecycle glue (connect / request-access flows) //! * [`connect`] — the trust gate and session lifecycle glue (connect / request-access flows)
@@ -10,7 +10,7 @@
//! * [`speed`] — the per-host network speed test (probe burst over the real data plane) //! * [`speed`] — the per-host network speed test (probe burst over the real data plane)
//! * [`settings`] — persisted preferences · [`licenses`] — the license notices screen · //! * [`settings`] — persisted preferences · [`licenses`] — the license notices screen ·
//! [`help`] — the in-stream keyboard-shortcuts reference (reached from the host list) //! [`help`] — the in-stream keyboard-shortcuts reference (reached from the host list)
//! * [`stream`] — the live stream: `SwapChainPanel` + D3D11 presenter + HUD overlay //! * [`stream`] — the stream status card (the stream itself runs in the spawned session window)
//! * [`style`] — the shared look (cards, pills, monograms), following the windows-reactor //! * [`style`] — the shared look (cards, pills, monograms), following the windows-reactor
//! gallery: Mica backdrop, a centred max-width column, theme brushes (`ThemeRef`) //! gallery: Mica backdrop, a centred max-width column, theme brushes (`ThemeRef`)
//! //!
@@ -19,9 +19,6 @@
//! marks it dirty and re-renders it; an `AsyncSetState` written from a background thread does //! marks it dirty and re-renders it; an `AsyncSetState` written from a background thread does
//! NOT (the child is pruned when its props are unchanged) — so everything thread-driven //! NOT (the child is pruned when its props are unchanged) — so everything thread-driven
//! (discovery, HUD stats, speed-test results) is held as *root* state and passed down as props. //! (discovery, HUD stats, speed-test results) is held as *root* state and passed down as props.
//! The present + decoded-frame handoff crosses to the UI thread through a `Mutex` side-channel
//! and thread-locals (the windows-reactor SwapChainPanel sample's pattern), since the per-frame
//! present must not go through state/rerender.
mod connect; mod connect;
mod help; mod help;
@@ -36,7 +33,6 @@ mod style;
use crate::discovery::{self, DiscoveredHost}; use crate::discovery::{self, DiscoveredHost};
use crate::gamepad::GamepadService; use crate::gamepad::GamepadService;
use crate::session::Stats;
use crate::trust::{KnownHosts, Settings}; use crate::trust::{KnownHosts, Settings};
use hosts::HostsProps; use hosts::HostsProps;
use punktfunk_core::client::NativeClient; use punktfunk_core::client::NativeClient;
@@ -45,7 +41,6 @@ use std::collections::HashMap;
use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicBool;
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
use std::time::Duration; use std::time::Duration;
use stream::StreamProps;
use windows_reactor::*; use windows_reactor::*;
#[derive(Clone, PartialEq)] #[derive(Clone, PartialEq)]
@@ -88,7 +83,7 @@ pub(crate) struct Target {
} }
/// Stable app services handed to the page components as props. Each routed screen that uses /// Stable app services handed to the page components as props. Each routed screen that uses
/// hooks (`hosts_page`/`pair_page`/`stream_page`/`speed_page`) is mounted as its own /// hooks (`hosts_page`/`pair_page`/`speed_page`/`library_page`) is mounted as its own
/// `component(...)`, so its hooks live in an isolated slot list — calling them on the shared /// `component(...)`, so its hooks live in an isolated slot list — calling them on the shared
/// parent `cx` would change the hook order whenever the screen changes (reactor's /// parent `cx` would change the hook order whenever the screen changes (reactor's
/// Rules-of-Hooks guard aborts). /// Rules-of-Hooks guard aborts).
@@ -115,18 +110,12 @@ impl PartialEq for Svc {
} }
} }
/// Cross-thread handoff from the session pump (off-thread) to the stream page (UI thread): /// Cross-thread shell state driven off the UI thread: the current target, the live spawned
/// the connector (input sends), the decoded-frame channel (render thread), and the session's /// session child (Disconnect/Cancel kill it) and its latest stats line, plus the connect-flow
/// stop flag (the disconnect shortcut trips it). /// cancel flag and the discovery/library/speed-test generation guards.
#[derive(Default)] #[derive(Default)]
pub(crate) struct Shared { pub(crate) struct Shared {
#[allow(clippy::type_complexity)]
pub(crate) handoff:
Mutex<Option<(Arc<NativeClient>, crate::session::FrameRx, Arc<AtomicBool>)>>,
pub(crate) target: Mutex<Target>, pub(crate) target: Mutex<Target>,
/// Latest stream stats, written by the session's event loop and mirrored into reactor state
/// by the HUD poll thread to drive the overlay.
pub(crate) stats: Mutex<Stats>,
/// The live session child (spawn mode) — the status page's Disconnect and the /// The live session child (spawn mode) — the status page's Disconnect and the
/// request-access Cancel kill it. A FRESH handle is installed per spawn. /// request-access Cancel kill it. A FRESH handle is installed per spawn.
pub(crate) session: Mutex<crate::spawn::SessionChild>, pub(crate) session: Mutex<crate::spawn::SessionChild>,
@@ -157,14 +146,6 @@ pub struct AppCtx {
pub(crate) shared: Arc<Shared>, pub(crate) shared: Arc<Shared>,
} }
/// The legacy in-process streaming path (SwapChainPanel + D3D11VA) instead of the
/// spawned punktfunk-session window: the `PUNKTFUNK_BUILTIN_STREAM=1` env override — a
/// developer A/B knob only (the former Settings "Streaming engine" pick is gone), removed
/// with the legacy path once the Vulkan session is fully validated.
pub(crate) fn use_builtin_stream(_ctx: &AppCtx) -> bool {
std::env::var_os("PUNKTFUNK_BUILTIN_STREAM").is_some_and(|v| v == "1")
}
pub fn run(identity: (String, String), gamepad: GamepadService) -> windows_reactor::Result<()> { pub fn run(identity: (String, String), gamepad: GamepadService) -> windows_reactor::Result<()> {
let ctx = Arc::new(AppCtx { let ctx = Arc::new(AppCtx {
identity, identity,
@@ -302,10 +283,9 @@ fn root(cx: &mut RenderCx, ctx: &Arc<AppCtx>) -> Element {
} }
}); });
// HUD sample: the session event loop writes `shared.stats` and the input hooks track capture // HUD sample: the spawned session child's latest `stats:` line, mirrored into root state so
// state; this poll thread mirrors both into root state so the stream page gets them as a // the stream status page gets it as a *prop* (thread-driven state must be root state — see the
// *prop* (thread-driven state must be root state — see the module docs). The compare in // module docs). The compare in `AsyncSetState::call` makes the idle case free.
// `AsyncSetState::call` makes the idle case free.
cx.use_effect((), { cx.use_effect((), {
let shared = ctx.shared.clone(); let shared = ctx.shared.clone();
let set_hud = set_hud.clone(); let set_hud = set_hud.clone();
@@ -315,10 +295,6 @@ fn root(cx: &mut RenderCx, ctx: &Arc<AppCtx>) -> Element {
.spawn(move || loop { .spawn(move || loop {
std::thread::sleep(std::time::Duration::from_millis(400)); std::thread::sleep(std::time::Duration::from_millis(400));
set_hud.call(stream::HudSample { set_hud.call(stream::HudSample {
stats: *shared.stats.lock().unwrap(),
captured: crate::input::is_captured(),
visible: crate::input::hud_visible(),
present: crate::render::present_stats(),
stats_line: shared.stats_line.lock().unwrap().clone(), stats_line: shared.stats_line.lock().unwrap().clone(),
}); });
}) })
@@ -525,16 +501,13 @@ fn root(cx: &mut RenderCx, ctx: &Arc<AppCtx>) -> Element {
state: library, state: library,
}, },
), ),
// Spawn mode (the default): the stream runs in the punktfunk-session child's own // The stream runs in the punktfunk-session child's own window; this screen is a
// window; this screen is a status page (no hooks — inline is sound). The legacy // status page (no hooks — inline is sound).
// in-process SwapChainPanel page stays behind the "Streaming engine" setting / Screen::Stream => stream::session_page(ctx, &hud),
// PUNKTFUNK_BUILTIN_STREAM=1.
Screen::Stream if !use_builtin_stream(ctx) => stream::session_page(ctx, &hud),
Screen::Stream => component(stream::stream_page, StreamProps { svc, hud }),
}; };
// The Stream screen owns the SwapChainPanel + per-frame present; never wrap it in an animated // The Stream screen is a plain status card (the session child owns the real stream window);
// opacity/offset layer. Everything else slides + fades in on navigation. // it's shown without the navigation entrance tween. Everything else slides + fades in.
if matches!(screen, Screen::Stream) { if matches!(screen, Screen::Stream) {
return body; return body;
} }
+9 -8
View File
@@ -267,12 +267,13 @@ pub(crate) fn settings_page(
); );
// --- Input ----------------------------------------------------------------------------- // --- Input -----------------------------------------------------------------------------
// Which physical controller forwards as pad 0: automatic = the most recently connected. // Controller forwarding: Automatic forwards EVERY real controller, each as its own pad;
// Persisted by stable key (`Settings::forward_pad`, GTK parity) so the pin survives // pinning one restricts the session to that single controller (single-player). Persisted
// restarts AND reaches the spawned session binary, whose service applies the same key. // by stable key (`Settings::forward_pad`, GTK parity) so the pin survives restarts AND
// reaches the spawned session binary, whose service applies the same key.
let pads = ctx.gamepad.pads(); let pads = ctx.gamepad.pads();
let (fwd_names, fwd_i) = { let (fwd_names, fwd_i) = {
let mut names = vec!["Automatic (most recent)".to_string()]; let mut names = vec!["Automatic (all controllers)".to_string()];
names.extend(pads.iter().map(|p| { names.extend(pads.iter().map(|p| {
let kind = p.kind_label(); let kind = p.kind_label();
if kind.is_empty() { if kind.is_empty() {
@@ -301,16 +302,16 @@ pub(crate) fn settings_page(
} else { } else {
keys.get(sel - 1).cloned() keys.get(sel - 1).cloned()
}; };
// Apply live (the in-process service, legacy builtin streams) and persist — // Apply live to the gamepad service and persist — the spawned session
// the spawned session reads `forward_pad` at connect. // reads `forward_pad` at connect.
svc.set_pinned(key.clone()); svc.set_pinned(key.clone());
let mut s = ctx2.settings.lock().unwrap(); let mut s = ctx2.settings.lock().unwrap();
s.forward_pad = key.unwrap_or_default(); s.forward_pad = key.unwrap_or_default();
s.save(); s.save();
}) })
.tooltip( .tooltip(
"Exactly one controller is forwarded to the host; \u{201C}Automatic\u{201D} \ "Every connected controller is forwarded, each as its own player. Pick one \
picks the most recently connected.", to force single-player \u{2014} only it reaches the host.",
) )
}; };
let (pad_names, pad_i) = presets(GAMEPADS, |v| { let (pad_names, pad_i) = presets(GAMEPADS, |v| {
+1 -1
View File
@@ -4,7 +4,7 @@
use super::style::*; use super::style::*;
use super::{Screen, Svc}; use super::{Screen, Svc};
use crate::session::run_speed_probe; use crate::probe::run_speed_probe;
use windows_reactor::*; use windows_reactor::*;
/// Speed-test lifecycle. Held as ROOT state (the probe worker completes it via /// Speed-test lifecycle. Held as ROOT state (the probe worker completes it via
+7 -311
View File
@@ -1,163 +1,20 @@
//! The stream page: a `SwapChainPanel` whose composition swapchain is created (and bound) once on //! The stream status page: streams run in the spawned `punktfunk-session` child's own window,
//! the UI thread, then handed — presenter and all — to the dedicated render thread //! so the shell shows a status card in the app's card language — host header, the child's live
//! ([`crate::render`]), which presents decoded frames at stream cadence. The page itself only //! `stats:` line as a chip row + stage lines, the in-window shortcuts, and a Disconnect.
//! forwards panel size/DPI changes and draws the status-chip HUD overlay (mode · decode path ·
//! HDR · fps/goodput · end-to-end latency + stage equation · capture hint).
use super::style::{edges, uniform}; use super::style::{edges, uniform};
use super::Svc;
use crate::present::Presenter;
use crate::render::{self, RenderThread};
use crate::session::Stats;
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::Mode;
use std::cell::RefCell;
use std::sync::Arc; use std::sync::Arc;
use windows_reactor::*; use windows_reactor::*;
/// One HUD refresh: the latest session stats, the input hooks' capture state, and the render /// One HUD refresh: the session child's latest formatted `stats:` line, mirrored into root state
/// thread's display-side window. Mirrored into root state by the poll thread (`pf-hud`) and /// by the poll thread (`pf-hud`) and passed down as a prop.
/// passed down as a prop.
#[derive(Clone, Default, PartialEq)] #[derive(Clone, Default, PartialEq)]
pub(crate) struct HudSample { pub(crate) struct HudSample {
pub(crate) stats: Stats, /// The session child's latest formatted `stats:` line, for the status page. Empty before the
pub(crate) captured: bool, /// child's first stats window.
/// Whether the stats overlay should be shown — the Settings default at stream start, then
/// whatever Ctrl+Alt+Shift+S last set (see [`crate::input::hud_visible`]). Carried in the
/// sample so a live toggle changes the sample and re-renders the page (the stream page is a
/// child component — only a changed prop re-renders it).
pub(crate) visible: bool,
/// The render thread's glass-side window (presents/s, skips, end-to-end p50/p95, display
/// stage p50) — see [`crate::render::present_stats`].
pub(crate) present: crate::render::PresentStats,
/// Spawn mode: the session child's latest formatted `stats:` line, for the status
/// page. Empty in builtin mode / before the first window.
pub(crate) stats_line: String, pub(crate) stats_line: String,
} }
/// Props for the stream page: the services plus the live HUD sample that drives the overlay
/// (compared by value, so each new sample re-renders the overlay).
#[derive(Clone)]
pub(crate) struct StreamProps {
pub(crate) svc: Svc,
pub(crate) hud: HudSample,
}
impl PartialEq for StreamProps {
fn eq(&self, other: &Self) -> bool {
self.svc == other.svc && self.hud == other.hud
}
}
thread_local! {
/// Frames + host clock offset, stashed by the mount effect for `on_mounted` (which fires
/// later, once the native panel exists).
static PENDING: RefCell<Option<(crate::session::FrameRx, std::sync::Arc<std::sync::atomic::AtomicI64>)>> = const { RefCell::new(None) };
/// The live render thread; stopped + joined by the unmount cleanup (before panel teardown).
static RENDER: RefCell<Option<RenderThread>> = const { RefCell::new(None) };
}
/// The app window's DPI (96 when the window can't be found — then DIPs == pixels). Reactor's
/// `on_resize` reports DIPs and exposes no CompositionScale, so the window DPI is the scale.
fn window_dpi() -> u32 {
use windows::Win32::UI::HiDpi::GetDpiForWindow;
use windows::Win32::UI::WindowsAndMessaging::FindWindowW;
unsafe {
FindWindowW(None, windows::core::w!("Punktfunk"))
.ok()
.map(|h| GetDpiForWindow(h))
.filter(|d| *d > 0)
.unwrap_or(96)
}
}
pub(crate) fn stream_page(props: &StreamProps, cx: &mut RenderCx) -> Element {
let ctx = &props.svc.ctx;
// Take the connector + frames handoff once on mount; keep the connector alive (and for input)
// in a use_ref, stash frames for `on_mounted`, install the input hooks. The cleanup stops the
// render thread FIRST (it must not present into a panel that's tearing down), then removes
// the input hooks.
let connector_ref = cx.use_ref::<Option<Arc<NativeClient>>>(None);
cx.use_effect_with_cleanup((), {
let shared = ctx.shared.clone();
let (inhibit, show_stats) = {
let s = ctx.settings.lock().unwrap();
(s.inhibit_shortcuts, s.show_stats)
};
let connector_ref = connector_ref.clone();
move || {
if let Some((connector, frames, stop)) = shared.handoff.lock().unwrap().take() {
let mode = connector.mode();
let clock_offset = connector.clock_offset_shared();
connector_ref.set(Some(connector.clone()));
PENDING.with(|c| *c.borrow_mut() = Some((frames, clock_offset)));
crate::input::install(connector, mode, inhibit, show_stats, stop);
}
Some(|| {
RENDER.with(|c| {
if let Some(mut rt) = c.borrow_mut().take() {
rt.stop_and_join();
}
});
PENDING.with(|c| c.borrow_mut().take());
crate::input::uninstall();
})
}
});
let mode = connector_ref.borrow().as_ref().map(|c| c.mode());
let host = ctx.shared.target.lock().unwrap().name.clone();
let mut layers: Vec<Element> = vec![swap_chain_panel()
.on_mounted(|panel| {
// Placeholder size — the first `on_resize` (fired after the first layout pass)
// resizes to the panel's real pixel size.
let dpi = window_dpi();
match Presenter::new(1280, 720, dpi) {
Ok(p) => {
if let Err(e) = panel.set_swap_chain(p.swap_chain()) {
tracing::error!(error = %e, "set_swap_chain");
return;
}
if let Some((frames, clock_offset)) = PENDING.with(|c| c.borrow_mut().take()) {
let shared = render::RenderShared::new(1280, 720, dpi);
RENDER.with(|cell| {
*cell.borrow_mut() =
Some(render::spawn(p, frames, shared, clock_offset));
});
tracing::info!(dpi, "stream presenter bound — render thread started");
}
}
Err(e) => tracing::error!(error = %e, "create presenter"),
}
})
.on_resize(|w, h| {
// DIPs → physical pixels; the presenter maps back via SetMatrixTransform.
let dpi = window_dpi();
let px = |v: f64| (v * f64::from(dpi) / 96.0).round() as u32;
RENDER.with(|cell| {
if let Some(rt) = cell.borrow().as_ref() {
rt.shared().set_dpi(dpi);
rt.shared().set_size(px(w), px(h));
}
});
})
.into()];
// The overlay follows the LIVE visibility (Settings default, then Ctrl+Alt+Shift+S): the page
// re-renders on every HUD sample (~400 ms), so a toggle takes effect promptly mid-stream.
if props.hud.visible {
layers.push(hud_overlay(&props.hud, mode, &host));
}
// Flash the shortcut key set for the first few seconds of every session, regardless of the
// HUD setting — so "how do I get back out" is answered the moment the stream comes up (parity
// with the GTK client's stream-start hint). Uptime drives it, so it needs no timer/state: the
// HUD poll re-renders the page each second and the banner drops once the session passes the
// threshold.
if props.hud.stats.uptime_secs < START_HINT_SECS {
layers.push(start_hint());
}
grid(layers).into()
}
/// Spawn mode's Stream screen: the stream runs in the punktfunk-session child's own /// Spawn mode's Stream screen: the stream runs in the punktfunk-session child's own
/// window, so the shell shows a status card in the app's card language — monogram + /// window, so the shell shows a status card in the app's card language — monogram +
/// host header, the child's live `stats:` line as a chip row + stage lines, the /// host header, the child's live `stats:` line as a chip row + stage lines, the
@@ -277,164 +134,3 @@ pub(crate) fn session_page(ctx: &Arc<super::AppCtx>, hud: &HudSample) -> Element
.vertical_alignment(VerticalAlignment::Center) .vertical_alignment(VerticalAlignment::Center)
.into() .into()
} }
/// How long the stream-start shortcut banner stays up (seconds of session uptime).
const START_HINT_SECS: u32 = 6;
/// The stream-start shortcut banner: the full client key set on a translucent pill, bottom-centre,
/// shown for [`START_HINT_SECS`] at the start of every session (see the call site). Independent of
/// the stats overlay, so it appears even with the HUD turned off.
fn start_hint() -> Element {
border(
text_block(
"Click the stream to capture \u{00B7} Ctrl+Alt+Shift+Q releases \u{00B7} \
Ctrl+Alt+Shift+D disconnects \u{00B7} Ctrl+Alt+Shift+S stats \u{00B7} F11 fullscreen",
)
.font_size(12.0)
.semibold()
.foreground(Color::rgb(235, 235, 235)),
)
.background(Color::rgb(0, 0, 0))
.corner_radius(10.0)
.padding(edges(14.0, 8.0, 14.0, 8.0))
.opacity(0.82)
.horizontal_alignment(HorizontalAlignment::Center)
.vertical_alignment(VerticalAlignment::Bottom)
.margin(edges(0.0, 0.0, 0.0, 28.0))
.into()
}
/// A small chip for the dark HUD: coloured text on a translucent dark fill.
fn hud_chip(text: &str, color: Color) -> Border {
border(
text_block(text)
.font_size(11.0)
.semibold()
.foreground(color),
)
.background(Color::rgb(38, 38, 38))
.corner_radius(8.0)
.padding(edges(8.0, 2.0, 8.0, 2.0))
}
/// The negotiated wire codec's display name (`quic::CODEC_*` bit → label).
fn codec_name(bits: u8) -> &'static str {
match bits {
punktfunk_core::quic::CODEC_H264 => "H.264",
punktfunk_core::quic::CODEC_AV1 => "AV1",
_ => "HEVC",
}
}
/// `mm:ss` (or `h:mm:ss`) session time.
fn fmt_uptime(secs: u32) -> String {
let (h, m, s) = (secs / 3600, secs / 60 % 60, secs % 60);
if h > 0 {
format!("{h}:{m:02}:{s:02}")
} else {
format!("{m}:{s:02}")
}
}
/// The streaming HUD overlay (top-right), unified stats vocabulary (design/stats-unification.md):
/// a chip row (mode · codec · decode path · HDR), a stream line (received fps · goodput ·
/// presenter fps), the end-to-end headline (capture→on-glass p50/p95, host-clock corrected), the
/// stage equation (= host + network + decode + display when the host reports 0xCF timings, else
/// the combined = host+network + decode + display; stage p50s), a session line
/// (host · time · loss/skips), and the shortcut hints. Layered over the `SwapChainPanel` in the
/// same grid cell.
fn hud_overlay(hud: &HudSample, mode: Option<Mode>, host: &str) -> Element {
let stats = &hud.stats;
let present = &hud.present;
let res = mode
.map(|m| format!("{}\u{00D7}{}@{}", m.width, m.height, m.refresh_hz))
.unwrap_or_else(|| "\u{2014}".into());
let mut chips: Vec<Element> = vec![
hud_chip(&res, Color::rgb(235, 235, 235)).into(),
hud_chip(codec_name(stats.codec), Color::rgb(180, 190, 255)).into(),
];
chips.push(if stats.hardware {
hud_chip("GPU decode", Color::rgb(120, 220, 150)).into()
} else {
hud_chip("CPU decode", Color::rgb(240, 190, 90)).into()
});
if stats.hdr {
chips.push(hud_chip("HDR", Color::rgb(255, 205, 90)).into());
}
// Received fps + goodput, plus the presenter's own rate (Moonlight's "Rendering frame rate"
// analog — how often the display actually gets a new frame).
let stream_line = format!(
"{:.0} fps \u{00B7} {:.1} Mb/s \u{00B7} display {} fps",
stats.fps, stats.mbps, present.fps
);
// The headline: end-to-end capture→displayed, measured directly post-Present (never the sum
// of the stage percentiles). `(same-host clock)` flags an uncorrected clock (offset == 0:
// same host, or the host skipped the skew handshake).
let mut e2e_line = format!(
"end-to-end {:.1} ms p50 \u{00B7} {:.1} p95 \u{00B7} capture\u{2192}on-glass",
present.e2e_p50_ms, present.e2e_p95_ms
);
if stats.same_host {
e2e_line.push_str(" (same-host clock)");
}
// The equation: the stages tile the headline interval per frame; the window p50s only
// approximately sum (percentiles aren't additive). With per-AU 0xCF host timings the opaque
// `host+network` term splits into `host` (host capture→sent) + `network` (the remainder);
// an old host emits none and the combined term stays.
let stage_line = if stats.split {
format!(
"= host {:.1} + network {:.1} + decode {:.1} + display {:.1}",
stats.host_ms, stats.net_ms, stats.decode_ms, present.display_p50_ms
)
} else {
format!(
"= host+network {:.1} + decode {:.1} + display {:.1}",
stats.hostnet_ms, stats.decode_ms, present.display_p50_ms
)
};
let mut session_bits: Vec<String> = Vec::new();
if !host.is_empty() {
session_bits.push(host.to_string());
}
// `lost` = unrecoverable network drops (session-cumulative); `skipped` = the render thread's
// newest-wins drops last window (expected when the stream outpaces the display).
session_bits.push(fmt_uptime(stats.uptime_secs));
session_bits.push(format!("{} lost", stats.dropped));
if present.skipped > 0 {
session_bits.push(format!("{} skipped", present.skipped));
}
let session_line = session_bits.join(" \u{00B7} ");
let hint = if hud.captured {
"Ctrl+Alt+Shift+Q releases the mouse \u{00B7} Ctrl+Alt+Shift+D disconnects \u{00B7} \
Ctrl+Alt+Shift+S stats \u{00B7} F11 fullscreen"
} else {
"Click the stream to capture \u{00B7} Ctrl+Alt+Shift+D disconnects \u{00B7} \
Ctrl+Alt+Shift+S stats \u{00B7} F11 fullscreen"
};
let dim = |t: &str| {
text_block(t)
.font_size(11.0)
.foreground(Color::rgb(210, 210, 210))
};
border(
vstack((
hstack(chips).spacing(6.0),
dim(&stream_line),
dim(&e2e_line),
dim(&stage_line),
dim(&session_line),
text_block(hint)
.font_size(11.0)
.foreground(Color::rgb(150, 150, 150)),
))
.spacing(6.0),
)
.background(Color::rgb(0, 0, 0))
.corner_radius(10.0)
.padding(uniform(10.0))
.opacity(0.82)
.horizontal_alignment(HorizontalAlignment::Right)
.vertical_alignment(VerticalAlignment::Top)
.margin(uniform(12.0))
.into()
}
-308
View File
@@ -1,308 +0,0 @@
//! Audio: playback (decoded PCM → a WASAPI shared-mode render stream) and the microphone
//! uplink (WASAPI capture → Opus → 0xCB datagrams, the inverse of the host's virtual mic).
//!
//! The WASAPI analogue of the Linux client's PipeWire backend. Playback mirrors the host's
//! virtual-mic producer's adaptive jitter buffer: the session pump pushes 5 ms Opus-decoded
//! chunks on the network clock; the WASAPI render thread pulls whole event-driven quanta on
//! the device clock. Prime to ~3 quanta before producing, cap the ring so latency stays
//! bounded, re-prime after a real drain.
//!
//! WASAPI objects are COM-apartment-bound and not `Send`, so they live on a dedicated thread
//! (the same discipline as the host's `wasapi_cap`); only the channel + stop flag + join
//! handle cross the boundary.
use anyhow::{anyhow, Context, Result};
use punktfunk_core::client::NativeClient;
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{Receiver, SyncSender, TrySendError};
use std::sync::Arc;
use std::time::Duration;
use wasapi::{DeviceEnumerator, Direction, SampleType, StreamMode, WaveFormat};
const SAMPLE_RATE: usize = 48_000;
/// The microphone uplink stays stereo (the host's virtual mic is stereo). The render path is
/// multichannel — its channel count + block align are runtime, driven by the host-resolved layout.
const CHANNELS: usize = 2;
/// Mic frames are 20 ms (960 samples/channel) — any size ≤ 120 ms is fine host-side.
const MIC_FRAME: usize = 960;
pub struct AudioPlayer {
pcm_tx: SyncSender<Vec<f32>>,
stop: Arc<AtomicBool>,
thread: Option<std::thread::JoinHandle<()>>,
}
impl AudioPlayer {
/// Spawn the WASAPI render thread for `channels` (2/6/8, canonical wire order
/// FL FR FC LFE RL RR SL SR). Failure (no render endpoint on this box) is survivable — the
/// caller streams video-only.
pub fn spawn(channels: u8) -> Result<AudioPlayer> {
// 64 × 5 ms = 320 ms of slack between the pump and the WASAPI loop.
let (pcm_tx, pcm_rx) = std::sync::mpsc::sync_channel::<Vec<f32>>(64);
let stop = Arc::new(AtomicBool::new(false));
let (ready_tx, ready_rx) = std::sync::mpsc::sync_channel::<Result<()>>(1);
let stop_t = stop.clone();
let thread = std::thread::Builder::new()
.name("punktfunk-audio".into())
.spawn(move || {
if let Err(e) = render_thread(pcm_rx, stop_t, ready_tx, channels) {
tracing::warn!(error = format!("{e:#}"), "audio playback thread ended");
}
})
.context("spawn audio thread")?;
match ready_rx.recv_timeout(Duration::from_secs(3)) {
Ok(Ok(())) => {
tracing::info!(channels, "WASAPI render: 48 kHz f32 (default endpoint)");
Ok(AudioPlayer {
pcm_tx,
stop,
thread: Some(thread),
})
}
Ok(Err(e)) => Err(e),
Err(_) => Err(anyhow!(
"wasapi render init timed out (no render endpoint?)"
)),
}
}
/// Queue one interleaved f32 chunk (in the session's channel layout). Drops the chunk if the
/// WASAPI side is wedged (the renderer conceals the gap; never block the session pump).
pub fn push(&self, pcm: Vec<f32>) {
if let Err(TrySendError::Disconnected(_)) = self.pcm_tx.try_send(pcm) {
// Thread already dead — Drop will reap it; nothing to do per-chunk.
}
}
}
impl Drop for AudioPlayer {
fn drop(&mut self) {
self.stop.store(true, Ordering::SeqCst);
if let Some(t) = self.thread.take() {
let _ = t.join();
}
}
}
fn render_thread(
pcm_rx: Receiver<Vec<f32>>,
stop: Arc<AtomicBool>,
ready: SyncSender<Result<()>>,
channels: u8,
) -> Result<()> {
if let Err(e) = wasapi::initialize_mta()
.ok()
.context("CoInitializeEx (MTA)")
{
let _ = ready.send(Err(e));
return Ok(());
}
let res = (|| -> Result<()> {
// F32LE interleaved: channels × 4 bytes/sample. Stereo (channels == 2) is byte-identical
// to the old fixed path (mask 0x3, block align 8).
let block_align = channels as usize * 4;
let device = DeviceEnumerator::new()
.context("DeviceEnumerator")?
.get_default_device(&Direction::Render)
.context("default render endpoint")?;
let mut audio_client = device.get_iaudioclient().context("IAudioClient")?;
// The explicit dwChannelMask is the wire order (FL FR FC LFE RL RR SL SR); 5.1 = 0x3F,
// 7.1 = 0x63F. WASAPI delivers channels in ascending mask-bit order, which equals the wire
// order, so the render mapping is the identity — no permute. `autoconvert` (below) lets the
// audio engine downmix when the endpoint has fewer speakers.
let desired = WaveFormat::new(
32,
32,
&SampleType::Float,
SAMPLE_RATE,
channels as usize,
Some(punktfunk_core::audio::wasapi_channel_mask(channels)),
);
let (default_period, _min_period) =
audio_client.get_device_period().context("device period")?;
let mode = StreamMode::EventsShared {
autoconvert: true,
buffer_duration_hns: default_period,
};
audio_client
.initialize_client(&desired, &Direction::Render, &mode)
.context("initialize render client")?;
let h_event = audio_client.set_get_eventhandle().context("event handle")?;
let render_client = audio_client
.get_audiorenderclient()
.context("IAudioRenderClient")?;
audio_client.start_stream().context("start render stream")?;
let _ = ready.send(Ok(()));
// Adaptive jitter buffer, in f32-byte units (same shape as the host's virtual mic).
let mut ring: VecDeque<u8> = VecDeque::new();
let mut primed = false;
let mut out = Vec::new(); // per-quantum scratch, reused across iterations
while !stop.load(Ordering::Relaxed) {
if h_event.wait_for_event(100).is_err() {
continue;
}
// Drain everything the pump has queued into the ring.
while let Ok(chunk) = pcm_rx.try_recv() {
for s in chunk {
ring.extend(s.to_le_bytes());
}
}
let avail_frames = audio_client
.get_available_space_in_frames()
.context("available space")? as usize;
if avail_frames == 0 {
continue;
}
let want_bytes = avail_frames * block_align;
// Prime to ~3 quanta; cap at ~1 quantum of slack beyond that; re-prime on drain.
let target = (3 * want_bytes).clamp(720 * block_align, 9600 * block_align);
let cap = target.max(want_bytes) + want_bytes;
if ring.len() > cap {
ring.drain(..ring.len() - cap);
}
if !primed && ring.len() >= target {
primed = true;
}
out.clear();
out.resize(want_bytes, 0);
if primed {
let n = ring.len().min(want_bytes);
for (dst, b) in out.iter_mut().zip(ring.drain(..n)) {
*dst = b;
}
}
if ring.is_empty() {
primed = false;
}
render_client
.write_to_device(avail_frames, &out, None)
.context("write_to_device")?;
}
audio_client.stop_stream().ok();
Ok(())
})();
if let Err(ref e) = res {
let _ = ready.send(Err(anyhow!("{e:#}")));
}
res
}
/// The microphone uplink: capture the default input device, Opus-encode 20 ms chunks, ship
/// them as 0xCB datagrams into the host's virtual mic source.
pub struct MicStreamer {
stop: Arc<AtomicBool>,
thread: Option<std::thread::JoinHandle<()>>,
}
impl MicStreamer {
pub fn spawn(connector: Arc<NativeClient>) -> Result<MicStreamer> {
let stop = Arc::new(AtomicBool::new(false));
let stop_t = stop.clone();
let thread = std::thread::Builder::new()
.name("punktfunk-mic".into())
.spawn(move || {
if let Err(e) = mic_thread(&connector, stop_t) {
tracing::warn!(error = format!("{e:#}"), "mic uplink thread ended");
}
})
.context("spawn mic thread")?;
Ok(MicStreamer {
stop,
thread: Some(thread),
})
}
}
impl Drop for MicStreamer {
fn drop(&mut self) {
self.stop.store(true, Ordering::SeqCst);
if let Some(t) = self.thread.take() {
let _ = t.join();
}
}
}
fn mic_thread(connector: &Arc<NativeClient>, stop: Arc<AtomicBool>) -> Result<()> {
wasapi::initialize_mta()
.ok()
.context("CoInitializeEx (MTA)")?;
let mut encoder = opus::Encoder::new(
SAMPLE_RATE as u32,
opus::Channels::Stereo,
opus::Application::Voip,
)
.map_err(|e| anyhow!("opus encoder: {e}"))?;
let _ = encoder.set_bitrate(opus::Bitrate::Bits(64_000));
let device = DeviceEnumerator::new()
.context("DeviceEnumerator")?
.get_default_device(&Direction::Capture)
.context("default capture endpoint (no microphone?)")?;
let mut audio_client = device.get_iaudioclient().context("IAudioClient")?;
let desired = WaveFormat::new(32, 32, &SampleType::Float, SAMPLE_RATE, CHANNELS, None);
let (default_period, _min_period) =
audio_client.get_device_period().context("device period")?;
let mode = StreamMode::EventsShared {
autoconvert: true,
buffer_duration_hns: default_period,
};
audio_client
.initialize_client(&desired, &Direction::Capture, &mode)
.context("initialize capture client")?;
let h_event = audio_client.set_get_eventhandle().context("event handle")?;
let capture_client = audio_client
.get_audiocaptureclient()
.context("IAudioCaptureClient")?;
audio_client
.start_stream()
.context("start capture stream")?;
let mut bytes: VecDeque<u8> = VecDeque::new();
let mut ring: VecDeque<f32> = VecDeque::new();
let mut out = vec![0u8; 4000];
let mut seq = 0u32;
while !stop.load(Ordering::Relaxed) {
if h_event.wait_for_event(100).is_err() {
continue;
}
loop {
match capture_client.get_next_packet_size() {
Ok(Some(0)) | Ok(None) => break,
Ok(Some(_n)) => {
capture_client
.read_from_device_to_deque(&mut bytes)
.context("read capture")?;
}
Err(e) => return Err(anyhow!("get_next_packet_size: {e}")),
}
}
let whole = (bytes.len() / 4) * 4;
for c in bytes.drain(..whole).collect::<Vec<u8>>().chunks_exact(4) {
ring.push_back(f32::from_le_bytes([c[0], c[1], c[2], c[3]]));
}
// Ship every complete 20 ms stereo frame.
while ring.len() >= MIC_FRAME * CHANNELS {
let pcm: Vec<f32> = ring.drain(..MIC_FRAME * CHANNELS).collect();
match encoder.encode_float(&pcm, &mut out) {
Ok(len) => {
let pts = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0);
let _ = connector.send_mic(seq, pts, out[..len].to_vec());
seq = seq.wrapping_add(1);
}
Err(e) => tracing::debug!(error = %e, "opus mic encode"),
}
}
}
audio_client.stop_stream().ok();
Ok(())
}
+6 -46
View File
@@ -74,30 +74,26 @@ fn pref_for_type(t: sdl3::gamepad::GamepadType) -> GamepadPref {
} }
enum Ctl { enum Ctl {
Attach(Arc<NativeClient>),
Detach,
Pin(Option<String>), Pin(Option<String>),
} }
#[derive(Clone)] #[derive(Clone)]
pub struct GamepadService { pub struct GamepadService {
pads: Arc<Mutex<Vec<PadInfo>>>, pads: Arc<Mutex<Vec<PadInfo>>>,
active: Arc<Mutex<Option<PadInfo>>>,
// `Arc<Mutex<…>>` (not a bare `Sender`, which is `!Sync`) so the service is `Sync` — the // `Arc<Mutex<…>>` (not a bare `Sender`, which is `!Sync`) so the service is `Sync` — the
// WinUI app shares it across the UI thread and the session-pump thread (attach/detach). // WinUI app shares it across the UI thread and the settings-pin path.
ctl: Arc<Mutex<Sender<Ctl>>>, ctl: Arc<Mutex<Sender<Ctl>>>,
} }
impl GamepadService { impl GamepadService {
pub fn start() -> GamepadService { pub fn start() -> GamepadService {
let pads = Arc::new(Mutex::new(Vec::new())); let pads = Arc::new(Mutex::new(Vec::new()));
let active = Arc::new(Mutex::new(None));
let (ctl, ctl_rx) = std::sync::mpsc::channel(); let (ctl, ctl_rx) = std::sync::mpsc::channel();
let (p, a) = (pads.clone(), active.clone()); let p = pads.clone();
if let Err(e) = std::thread::Builder::new() if let Err(e) = std::thread::Builder::new()
.name("punktfunk-gamepad".into()) .name("punktfunk-gamepad".into())
.spawn(move || { .spawn(move || {
if let Err(e) = run(&p, &a, &ctl_rx) { if let Err(e) = run(&p, &ctl_rx) {
tracing::warn!(error = %e, "gamepad service ended — pads disabled"); tracing::warn!(error = %e, "gamepad service ended — pads disabled");
} }
}) })
@@ -106,7 +102,6 @@ impl GamepadService {
} }
GamepadService { GamepadService {
pads, pads,
active,
ctl: Arc::new(Mutex::new(ctl)), ctl: Arc::new(Mutex::new(ctl)),
} }
} }
@@ -116,33 +111,13 @@ impl GamepadService {
self.pads.lock().unwrap().clone() self.pads.lock().unwrap().clone()
} }
pub fn active(&self) -> Option<PadInfo> {
self.active.lock().unwrap().clone()
}
/// Pin the forwarded controller by stable key (`PadInfo::key`) — `None` = automatic. /// Pin the forwarded controller by stable key (`PadInfo::key`) — `None` = automatic.
/// The pin survives the pad disconnecting: it re-applies the moment a matching /// The pin survives the pad disconnecting: it re-applies the moment a matching
/// controller shows up again (same semantics as `pf-client-core`'s service). /// controller shows up again (same semantics as `pf-client-core`'s service). The spawned
/// `punktfunk-session` binary owns the actual forwarding; this persists the selection.
pub fn set_pinned(&self, key: Option<String>) { pub fn set_pinned(&self, key: Option<String>) {
let _ = self.ctl.lock().unwrap().send(Ctl::Pin(key)); let _ = self.ctl.lock().unwrap().send(Ctl::Pin(key));
} }
pub fn attach(&self, connector: Arc<NativeClient>) {
let _ = self.ctl.lock().unwrap().send(Ctl::Attach(connector));
}
pub fn detach(&self) {
let _ = self.ctl.lock().unwrap().send(Ctl::Detach);
}
/// What "Automatic" resolves to right now — the virtual pad matching the physical one
/// (Swift parity); no pad connected leaves the host's own default.
pub fn auto_pref(&self) -> GamepadPref {
match self.active() {
Some(p) => p.pref,
None => GamepadPref::Auto,
}
}
} }
fn send(connector: &NativeClient, kind: InputKind, code: u32, x: i32) { fn send(connector: &NativeClient, kind: InputKind, code: u32, x: i32) {
@@ -404,11 +379,7 @@ impl Worker {
} }
#[allow(clippy::too_many_lines)] #[allow(clippy::too_many_lines)]
fn run( fn run(pads_out: &Mutex<Vec<PadInfo>>, ctl: &Receiver<Ctl>) -> Result<(), String> {
pads_out: &Mutex<Vec<PadInfo>>,
active_out: &Mutex<Option<PadInfo>>,
ctl: &Receiver<Ctl>,
) -> Result<(), String> {
// Off-main-thread + no video subsystem: keep SDL away from signals, poll pads on its own // Off-main-thread + no video subsystem: keep SDL away from signals, poll pads on its own
// thread. // thread.
sdl3::hint::set("SDL_NO_SIGNAL_HANDLERS", "1"); sdl3::hint::set("SDL_NO_SIGNAL_HANDLERS", "1");
@@ -437,23 +408,12 @@ fn run(
let mut list: Vec<PadInfo> = w.order.iter().filter_map(|&id| w.pad_info(id)).collect(); let mut list: Vec<PadInfo> = w.order.iter().filter_map(|&id| w.pad_info(id)).collect();
list.reverse(); // most recent first — the Settings list order list.reverse(); // most recent first — the Settings list order
*pads_out.lock().unwrap() = list; *pads_out.lock().unwrap() = list;
*active_out.lock().unwrap() = w.active_id().and_then(|id| w.pad_info(id));
}; };
loop { loop {
// Control plane from the UI thread. // Control plane from the UI thread.
loop { loop {
match ctl.try_recv() { match ctl.try_recv() {
Ok(Ctl::Attach(c)) => {
w.attached = Some(c);
w.last_axis = [i32::MIN; 6];
w.set_sensors(true);
}
Ok(Ctl::Detach) => {
w.flush_held();
w.set_sensors(false);
w.attached = None;
}
Ok(Ctl::Pin(key)) => { Ok(Ctl::Pin(key)) => {
let before = w.active_id(); let before = w.active_id();
w.pinned = key; w.pinned = key;
+7 -230
View File
@@ -1,104 +1,14 @@
//! The single Direct3D 11 device shared by the video decoder (D3D11VA hardware decode) and the //! DXGI adapter enumeration for the Settings "GPU" picker.
//! presenter (the `SwapChainPanel` composition swapchain + the present draw).
//! //!
//! Zero-copy hardware decode requires FFmpeg to decode HEVC into `ID3D11Texture2D`s created by the //! Streaming (decode + present) runs in the spawned `punktfunk-session` binary; the shell only
//! **same** device the presenter binds as shader resources and draws with — a texture from one //! needs the list of real (hardware) adapters to offer on a multi-GPU box (a hybrid laptop or an
//! device can't be sampled by another. So the device is created once, here, and both subsystems //! eGPU). The picked adapter description is persisted (`crate::trust::Settings::adapter`) and read
//! pull it from a process-global `OnceLock` (initialised on whichever thread asks first: the //! by the session child at connect (`PUNKTFUNK_ADAPTER` remains the session binary's env override).
//! session pump when it builds the decoder, or the UI thread when it builds the presenter).
//!
//! **Adapter selection** (matters on hybrid boxes — e.g. an Intel iGPU driving the panel next to
//! an NVIDIA dGPU): `PUNKTFUNK_ADAPTER` (index or case-insensitive name substring, a debugging
//! override) wins; else the persisted Settings GPU pick ([`crate::trust::Settings::adapter`], the
//! Settings-page selector on multi-GPU boxes); else the adapter whose output owns the monitor our
//! window is on — that's the adapter DWM composes that monitor with, so presents are copy-free
//! and decode runs on the near GPU; else the default adapter. Deliberately NOT "the adapter with
//! the best decoder": if the monitor's adapter can't decode the codec we demote to software,
//! which beats a per-frame cross-adapter present copy. The device is cached **keyed by the
//! resolved preference**, so a Settings change takes effect at the next session (the pump and the
//! presenter both resolve at session start and read the same value) without an app restart.
//!
//! `PUNKTFUNK_D3D_DEBUG=1` adds the D3D11 debug layer (validation messages in the debugger /
//! DebugView) — invaluable for present-path bugs, which D3D11 otherwise drops silently.
//!
//! **Thread-safety.** windows-rs COM interfaces are deliberately `!Send`/`!Sync` — thread-safety
//! is per-object, not universal. An `ID3D11Device` and its immediate context become free-threaded
//! once `ID3D11Multithread::SetMultithreadProtected(TRUE)` is set, which FFmpeg's D3D11VA backend
//! does inside `av_hwdevice_ctx_init` (it installs an `ID3D11Multithread`-based default lock when we
//! leave `AVD3D11VADeviceContext.lock` null). The decoder then uses FFmpeg's separate
//! `ID3D11VideoContext` for decode while the presenter uses the immediate context for draw; under
//! multithread protection D3D serialises the two internally, and decode/draw touch disjoint context
//! state. That makes the `unsafe impl Send + Sync` below sound for exactly this usage.
use anyhow::{anyhow, Result};
use std::sync::{Arc, Mutex};
use windows::core::Interface; use windows::core::Interface;
use windows::Win32::Graphics::Direct3D::{
D3D_DRIVER_TYPE, D3D_DRIVER_TYPE_HARDWARE, D3D_DRIVER_TYPE_UNKNOWN, D3D_DRIVER_TYPE_WARP,
D3D_FEATURE_LEVEL_11_0, D3D_FEATURE_LEVEL_11_1,
};
use windows::Win32::Graphics::Direct3D11::{
D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Multithread,
D3D11_CREATE_DEVICE_BGRA_SUPPORT, D3D11_CREATE_DEVICE_DEBUG, D3D11_CREATE_DEVICE_FLAG,
D3D11_CREATE_DEVICE_VIDEO_SUPPORT, D3D11_SDK_VERSION,
};
use windows::Win32::Graphics::Dxgi::{CreateDXGIFactory1, IDXGIAdapter, IDXGIFactory1}; use windows::Win32::Graphics::Dxgi::{CreateDXGIFactory1, IDXGIAdapter, IDXGIFactory1};
pub struct SharedDevice { /// The adapter's human-readable description.
pub device: ID3D11Device,
pub context: ID3D11DeviceContext,
/// True when this is a real GPU (hardware) adapter — a precondition for D3D11VA decode. WARP
/// (the GPU-less dev box) creates fine for present but cannot hardware-decode HEVC, so the
/// decoder skips straight to the software path there.
pub hardware: bool,
}
// Sound for our usage — see the module docs: the device + immediate context are free-threaded under
// the multithread protection FFmpeg installs, and decode (video context) / present (immediate
// context) never share mutable context state.
unsafe impl Send for SharedDevice {}
unsafe impl Sync for SharedDevice {}
/// The shared device, cached with the GPU preference it was resolved from (empty = automatic).
/// Re-created when the preference changes — in practice only between sessions: within one session
/// the decoder and the presenter both call [`shared`] at session start with the same value.
static SHARED: Mutex<Option<(String, Arc<SharedDevice>)>> = Mutex::new(None);
/// The user's decode/present GPU preference: the `PUNKTFUNK_ADAPTER` env (debugging override)
/// wins, else the persisted Settings pick; empty = automatic.
fn adapter_pref() -> String {
std::env::var("PUNKTFUNK_ADAPTER")
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.unwrap_or_else(|| crate::trust::Settings::load().adapter)
}
/// The process-shared D3D11 device for the current GPU preference, created (or re-created after
/// a preference change) on demand. `None` only if D3D11 device creation fails for both a hardware
/// adapter and WARP (effectively never — WARP is always present).
pub fn shared() -> Option<Arc<SharedDevice>> {
let pref = adapter_pref();
let mut cached = SHARED.lock().unwrap();
if let Some((key, dev)) = cached.as_ref() {
if *key == pref {
return Some(dev.clone());
}
}
match create_device(&pref) {
Ok(d) => {
let d = Arc::new(d);
*cached = Some((pref, d.clone()));
Some(d)
}
Err(e) => {
tracing::error!(error = %e, "shared D3D11 device creation failed — no present/decode");
None
}
}
}
/// The adapter's human-readable description, for the logs.
fn adapter_name(adapter: &IDXGIAdapter) -> String { fn adapter_name(adapter: &IDXGIAdapter) -> String {
unsafe { unsafe {
adapter adapter
@@ -112,7 +22,7 @@ fn adapter_name(adapter: &IDXGIAdapter) -> String {
} }
} }
/// Every DXGI adapter, in enumeration order (`PUNKTFUNK_ADAPTER=<index>` uses these indices). /// Every DXGI adapter, in enumeration order.
fn all_adapters() -> Vec<IDXGIAdapter> { fn all_adapters() -> Vec<IDXGIAdapter> {
let factory: IDXGIFactory1 = match unsafe { CreateDXGIFactory1() } { let factory: IDXGIFactory1 = match unsafe { CreateDXGIFactory1() } {
Ok(f) => f, Ok(f) => f,
@@ -144,136 +54,3 @@ pub fn adapter_names() -> Vec<String> {
.map(adapter_name) .map(adapter_name)
.collect() .collect()
} }
/// Resolve an explicit adapter: a non-empty `pref` (index or case-insensitive name substring, from
/// env or Settings) wins; else the adapter whose output owns the monitor the app window is on (see
/// module docs); else `None` → the default adapter (also the headless-CLI path with no window).
fn resolve_adapter(pref: &str) -> Option<IDXGIAdapter> {
let adapters = all_adapters();
if !pref.is_empty() {
let found = if let Ok(idx) = pref.parse::<usize>() {
adapters.get(idx).cloned()
} else {
let needle = pref.to_lowercase();
adapters
.iter()
.find(|a| adapter_name(a).to_lowercase().contains(&needle))
.cloned()
};
match &found {
Some(a) => tracing::info!(pref, adapter = %adapter_name(a), "GPU preference matched"),
None => tracing::warn!(pref, "GPU preference matched no adapter — using automatic"),
}
if found.is_some() {
return found;
}
}
// The adapter driving the monitor our window sits on: DWM composes that monitor with it, so
// presenting from it is copy-free (a hybrid box's other adapter would pay a cross-adapter
// copy per frame).
let monitor = unsafe {
use windows::Win32::Graphics::Gdi::{MonitorFromWindow, MONITOR_DEFAULTTONULL};
use windows::Win32::UI::WindowsAndMessaging::FindWindowW;
let hwnd = FindWindowW(None, windows::core::w!("Punktfunk")).ok()?;
MonitorFromWindow(hwnd, MONITOR_DEFAULTTONULL)
};
if monitor.is_invalid() {
return None;
}
for adapter in &adapters {
let mut oi = 0u32;
while let Ok(output) = unsafe { adapter.EnumOutputs(oi) } {
oi += 1;
if let Ok(desc) = unsafe { output.GetDesc() } {
if desc.Monitor == monitor {
tracing::info!(adapter = %adapter_name(adapter), "using the window's monitor adapter");
return Some(adapter.clone());
}
}
}
}
None
}
fn create_device(pref: &str) -> Result<SharedDevice> {
// Preference order: the resolved adapter (or the default hardware adapter) with video support
// (enables D3D11VA); the same without the VIDEO flag (a driver that rejects it still presents +
// software-decodes); finally WARP for the GPU-less box. BGRA_SUPPORT is required for the
// composition swapchain in every case. An explicit adapter requires D3D_DRIVER_TYPE_UNKNOWN.
let adapter = resolve_adapter(pref);
let attempts: [(Option<&IDXGIAdapter>, D3D_DRIVER_TYPE, bool, bool); 3] = match &adapter {
Some(a) => [
(Some(a), D3D_DRIVER_TYPE_UNKNOWN, true, true),
(Some(a), D3D_DRIVER_TYPE_UNKNOWN, false, true),
(None, D3D_DRIVER_TYPE_WARP, false, false),
],
None => [
(None, D3D_DRIVER_TYPE_HARDWARE, true, true),
(None, D3D_DRIVER_TYPE_HARDWARE, false, true),
(None, D3D_DRIVER_TYPE_WARP, false, false),
],
};
// The debug layer needs the SDK layers installed (Graphics Tools); when they're missing the
// creation fails, so each attempt retries without the flag rather than failing the ladder.
let debug = std::env::var("PUNKTFUNK_D3D_DEBUG").is_ok_and(|v| v == "1");
for (adapter, driver, video, hardware) in attempts {
let mut flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT;
if video {
flags |= D3D11_CREATE_DEVICE_VIDEO_SUPPORT;
}
let flag_sets: &[D3D11_CREATE_DEVICE_FLAG] = if debug {
&[flags | D3D11_CREATE_DEVICE_DEBUG, flags]
} else {
&[flags]
};
for &flags in flag_sets {
let mut device = None;
let mut context = None;
let r = unsafe {
D3D11CreateDevice(
adapter,
driver,
None,
flags,
Some(&[D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0]),
D3D11_SDK_VERSION,
Some(&mut device),
None,
Some(&mut context),
)
};
if r.is_ok() {
let (device, context) = (device.unwrap(), context.unwrap());
// Make the device + immediate context free-threaded: the decoder (D3D11VA video
// context, pump thread) and the presenter (immediate context, render thread) both
// touch this device. FFmpeg also sets this during hwdevice init, but doing it up
// front keeps the cross-thread `Send`/`Sync` sound from the moment the device exists.
if let Ok(mt) = context.cast::<ID3D11Multithread>() {
unsafe {
let _ = mt.SetMultithreadProtected(true); // returns the prior state; ignore
}
}
tracing::info!(
adapter = %adapter.map(adapter_name).unwrap_or_else(|| if hardware {
"default".into()
} else {
"WARP (software)".into()
}),
video,
debug = (flags & D3D11_CREATE_DEVICE_DEBUG).0 != 0,
"shared D3D11 device created"
);
return Ok(SharedDevice {
device,
context,
hardware,
});
}
}
}
Err(anyhow!(
"D3D11CreateDevice failed for both hardware and WARP"
))
}
-742
View File
@@ -1,742 +0,0 @@
//! Stream input: Win32 low-level keyboard + mouse hooks forwarding to the host while the WinUI
//! window is focused and the pointer is captured.
//!
//! windows-reactor exposes no raw key-down/up or pointer-position/wheel events (only keyboard
//! *accelerators* and pointer button-state), which is insufficient for a game stream. So this
//! drops below XAML to `WH_KEYBOARD_LL` / `WH_MOUSE_LL`, installed on the UI thread when the
//! stream page mounts and removed when it unmounts.
//!
//! **Pointer lock.** While captured the cursor is *locked* the way a game-streaming client locks
//! it (Moonlight/Parsec): the OS cursor is hidden + confined to the window (`ClipCursor`), and
//! every physical move is turned into a **relative** delta (`InputKind::MouseMove`) — we read the
//! offset from the window centre, ship it (scaled screen→host through the Contain-fit factor, with
//! sub-pixel remainder carried so slow drags aren't lost), then warp the cursor back to centre so
//! it never reaches a screen edge. This is why the old absolute path froze: swallowing
//! `WM_MOUSEMOVE` pinned the OS cursor, so `pt` never travelled and the absolute coordinate
//! snapped to one point. Keys carry the **US-positional VK** for the pressed physical key (the
//! punktfunk wire contract shared by every first-party client — see [`scan_to_positional_vk`]):
//! the hook's layout-resolved `vkCode` must NOT go on the wire, or a non-US pair re-maps
//! positions through two layouts (German: y↔z swapped, ü lands on ö).
//!
//! **Capture state machine** (parity with the GTK/Swift clients): capture engages at stream
//! start, **Ctrl+Alt+Shift+Q** releases it (handing the cursor back to the local desktop), and a
//! **click on the stream** re-engages it. Losing foreground also releases the lock so the cursor
//! is never stranded; regaining it while still captured re-locks. When "capture system
//! shortcuts" is off in Settings, Alt+Tab / Alt+Esc / Ctrl+Esc / the Win keys act on the local
//! desktop instead of being forwarded. **Ctrl+Alt+Shift+D disconnects** the session (consumed
//! locally, works captured or released while our window is foreground): it trips the session's
//! stop flag, the pump winds down, and the event loop navigates back to the host list.
//! **Ctrl+Alt+Shift+S** toggles the stats overlay live and **F11** toggles fullscreen — both are
//! client-local shortcuts (consumed, never forwarded), matching the GTK client's stream key set.
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::Mode;
use punktfunk_core::input::{InputEvent, InputKind};
use std::collections::HashSet;
use std::sync::atomic::{AtomicBool, AtomicIsize, Ordering};
use std::sync::{Arc, Mutex};
use windows::core::BOOL;
use windows::Win32::Foundation::{HWND, LPARAM, LRESULT, POINT, RECT, WPARAM};
use windows::Win32::Graphics::Gdi::ClientToScreen;
use windows::Win32::System::LibraryLoader::GetModuleHandleW;
use windows::Win32::UI::Input::KeyboardAndMouse::{VK_D, VK_F11, VK_Q, VK_S};
use windows::Win32::UI::Shell::{DefSubclassProc, RemoveWindowSubclass, SetWindowSubclass};
use windows::Win32::UI::WindowsAndMessaging::{
CallNextHookEx, ClipCursor, EnumChildWindows, GetClientRect, GetForegroundWindow, SetCursor,
SetCursorPos, SetWindowsHookExW, ShowCursor, UnhookWindowsHookEx, HC_ACTION, HHOOK,
KBDLLHOOKSTRUCT, LLKHF_EXTENDED, LLMHF_INJECTED, MSLLHOOKSTRUCT, WH_KEYBOARD_LL, WH_MOUSE_LL,
WM_KEYUP, WM_LBUTTONDOWN, WM_LBUTTONUP, WM_MBUTTONDOWN, WM_MBUTTONUP, WM_MOUSEHWHEEL,
WM_MOUSEMOVE, WM_MOUSEWHEEL, WM_RBUTTONDOWN, WM_RBUTTONUP, WM_SETCURSOR, WM_SYSKEYUP,
WM_XBUTTONDOWN, WM_XBUTTONUP,
};
struct State {
connector: Arc<NativeClient>,
mode: Mode,
/// The session's stop flag (Ctrl+Alt+Shift+D trips it; the pump then ends the session).
stop: Arc<AtomicBool>,
/// Our window handle, stored as the raw `isize` so `State` is `Send` (`HWND` is not).
hwnd: isize,
/// User intent: forward input to the host (toggled by Ctrl+Alt+Shift+Q / click-to-capture).
captured: bool,
/// Forward system shortcuts (Alt+Tab, Win, …) to the host; off = they act locally.
inhibit_shortcuts: bool,
/// The OS pointer is currently locked (hidden + confined + recentering). Tracks the real
/// `ClipCursor`/`ShowCursor` state so we engage/disengage exactly once per transition.
locked: bool,
/// Lock geometry, captured when the lock engages: the confinement rect (screen coordinates,
/// also the click-to-capture hit test), its centre (the cursor is warped here after every
/// move), and the screen→host scale (the Contain-fit display scale's inverse). Stable while
/// locked — the window can't be moved or resized with the cursor confined inside it.
clip: RECT,
center_x: i32,
center_y: i32,
scale: f32,
/// Sub-pixel remainder of the screen→host scale, carried so slow drags aren't truncated away.
acc_x: f32,
acc_y: f32,
/// Modifier state, tracked from the hook's own event stream (see `kbd_proc`).
ctrl: bool,
alt: bool,
shift: bool,
held_keys: HashSet<u8>,
held_buttons: HashSet<u32>,
}
// `State` carries no `!Send` handle (hwnd is an `isize`), so the static is sound. The hook procs
// run on the same UI thread that installs/removes the hooks, so the lock is uncontended.
static STATE: Mutex<Option<State>> = Mutex::new(None);
static KBD_HOOK: AtomicIsize = AtomicIsize::new(0);
static MOUSE_HOOK: AtomicIsize = AtomicIsize::new(0);
/// Mirror of `State::captured` for lock-free reads off the UI thread (the HUD poll).
static CAPTURED: AtomicBool = AtomicBool::new(false);
/// Live stats-overlay visibility. Seeded from `Settings::show_stats` at `install`, then toggled by
/// Ctrl+Alt+Shift+S for the session (parity with the GTK client's live `s` toggle); the HUD poll
/// reads it lock-free to drive the overlay.
static HUD_VISIBLE: AtomicBool = AtomicBool::new(false);
/// Whether the pointer lock currently wants the OS cursor hidden. Read lock-free by
/// [`cursor_subclass_proc`] (which runs on the UI thread inside `WM_SETCURSOR`) so it can override
/// WinUI's per-move arrow re-assertion — a one-shot `ShowCursor(false)` alone loses that race
/// because the content island re-sets the arrow every time the pointer moves.
static CURSOR_HIDDEN: AtomicBool = AtomicBool::new(false);
/// Our `SetWindowSubclass` id on the WinUI window + its content-island children (any stable value;
/// scopes the subclass so install/remove target exactly our proc).
const CURSOR_SUBCLASS_ID: usize = 0x7066_6375; // 'pfcu'
/// Whether stream input is currently captured (drives the HUD's release/capture hint).
pub fn is_captured() -> bool {
CAPTURED.load(Ordering::Relaxed)
}
/// Whether the stats overlay should be shown: the Settings default at stream start, then whatever
/// Ctrl+Alt+Shift+S last set for the session. Read by the HUD poll thread.
pub fn hud_visible() -> bool {
HUD_VISIBLE.load(Ordering::Relaxed)
}
/// Set the capture intent and engage/release the pointer lock to match.
fn set_captured(st: &mut State, on: bool) {
st.captured = on;
CAPTURED.store(on, Ordering::Relaxed);
set_locked(st, on);
if !on {
flush_held(st); // release held keys/buttons so nothing sticks on the host
}
}
/// Install the hooks for a streaming session. Call from the UI thread once the window is shown.
/// `inhibit_shortcuts` forwards system shortcuts (Alt+Tab, Win, …) to the host; off = local.
/// `show_stats` seeds the stats-overlay visibility that Ctrl+Alt+Shift+S then toggles live.
/// `stop` is the session's stop flag, tripped by the disconnect shortcut.
pub fn install(
connector: Arc<NativeClient>,
mode: Mode,
inhibit_shortcuts: bool,
show_stats: bool,
stop: Arc<AtomicBool>,
) {
HUD_VISIBLE.store(show_stats, Ordering::Relaxed);
let hwnd = unsafe { GetForegroundWindow() };
let mut st = State {
connector,
mode,
stop,
hwnd: hwnd.0 as isize,
captured: false,
inhibit_shortcuts,
locked: false,
clip: RECT::default(),
center_x: 0,
center_y: 0,
scale: 1.0,
acc_x: 0.0,
acc_y: 0.0,
ctrl: false,
alt: false,
shift: false,
held_keys: HashSet::new(),
held_buttons: HashSet::new(),
};
// Capture immediately (the window is foreground at mount, like Moonlight grabbing on stream
// start).
set_captured(&mut st, true);
*STATE.lock().unwrap() = Some(st);
unsafe {
let hinst = GetModuleHandleW(None).ok();
if let Ok(h) = SetWindowsHookExW(WH_KEYBOARD_LL, Some(kbd_proc), hinst.map(Into::into), 0) {
KBD_HOOK.store(h.0 as isize, Ordering::SeqCst);
}
if let Ok(h) = SetWindowsHookExW(WH_MOUSE_LL, Some(mouse_proc), hinst.map(Into::into), 0) {
MOUSE_HOOK.store(h.0 as isize, Ordering::SeqCst);
}
}
tracing::info!(
inhibit_shortcuts,
"stream input hooks installed — pointer locked (Ctrl+Alt+Shift+Q toggles capture)"
);
}
/// Remove the hooks, release the pointer lock, and flush any held keys/buttons (so nothing
/// sticks down on the host).
pub fn uninstall() {
unsafe {
let k = KBD_HOOK.swap(0, Ordering::SeqCst);
if k != 0 {
let _ = UnhookWindowsHookEx(HHOOK(k as *mut _));
}
let m = MOUSE_HOOK.swap(0, Ordering::SeqCst);
if m != 0 {
let _ = UnhookWindowsHookEx(HHOOK(m as *mut _));
}
}
if let Some(mut st) = STATE.lock().unwrap().take() {
// Hand the cursor back + flush held state.
set_captured(&mut st, false);
// Drop the WM_SETCURSOR subclass so the long-lived app window (reused for the host list
// once the stream ends) is left pristine — set_captured already cleared CURSOR_HIDDEN.
remove_cursor_subclass(HWND(st.hwnd as *mut _));
// Fullscreen is a streaming-only mode: if F11 put us there, drop back to a normal window
// so the GUI (the host list) is never left borderless-fullscreen after the stream ends.
exit_fullscreen(HWND(st.hwnd as *mut _));
}
}
/// Release every held key/button on the host, so nothing sticks down when capture is dropped
/// (toggled off) or the session ends.
fn flush_held(st: &mut State) {
let c = st.connector.clone();
for vk in st.held_keys.drain() {
send(&c, InputKind::KeyUp, vk as u32, 0, 0, 0);
}
for b in st.held_buttons.drain() {
send(&c, InputKind::MouseButtonUp, b, 0, 0, 0);
}
}
/// Subclass proc on the WinUI window + its content-island children: while the pointer lock wants
/// the cursor hidden ([`CURSOR_HIDDEN`]), answer `WM_SETCURSOR` ourselves with `SetCursor(None)`
/// and return TRUE — halting WinUI's default handling before it re-asserts the arrow. This is what
/// actually keeps the cursor hidden while captured; the sibling `ShowCursor(false)` cannot, because
/// WinUI re-sets the arrow on every pointer move (the content island answers `WM_SETCURSOR` itself,
/// which a low-level mouse hook never sees). When not hidden, we defer to the chain untouched.
unsafe extern "system" fn cursor_subclass_proc(
hwnd: HWND,
msg: u32,
wparam: WPARAM,
lparam: LPARAM,
_id: usize,
_ref: usize,
) -> LRESULT {
if msg == WM_SETCURSOR && CURSOR_HIDDEN.load(Ordering::Relaxed) {
unsafe {
let _ = SetCursor(None);
}
return LRESULT(1); // handled — suppress the framework's arrow re-assertion
}
unsafe { DefSubclassProc(hwnd, msg, wparam, lparam) }
}
unsafe extern "system" fn subclass_install_cb(child: HWND, _l: LPARAM) -> BOOL {
unsafe {
let _ = SetWindowSubclass(child, Some(cursor_subclass_proc), CURSOR_SUBCLASS_ID, 0);
}
BOOL(1) // keep enumerating
}
unsafe extern "system" fn subclass_remove_cb(child: HWND, _l: LPARAM) -> BOOL {
unsafe {
let _ = RemoveWindowSubclass(child, Some(cursor_subclass_proc), CURSOR_SUBCLASS_ID);
}
BOOL(1)
}
/// Install [`cursor_subclass_proc`] on the top-level WinUI window and every descendant — the video
/// is a composition SwapChainPanel, so the pointer actually sits over WinUI's internal content-
/// island child window, which is the window that receives `WM_SETCURSOR`. `EnumChildWindows`
/// recurses into all descendants, so one pass covers it. Idempotent (re-installing the same
/// id+proc just refreshes it), so it's safe to call on every lock engage.
fn install_cursor_subclass(top: HWND) {
unsafe {
let _ = SetWindowSubclass(top, Some(cursor_subclass_proc), CURSOR_SUBCLASS_ID, 0);
let _ = EnumChildWindows(Some(top), Some(subclass_install_cb), LPARAM(0));
}
}
/// Remove our subclass from the top-level window and every descendant. Called on teardown so the
/// long-lived app window (reused for the host list after the stream ends) is left pristine.
fn remove_cursor_subclass(top: HWND) {
unsafe {
let _ = RemoveWindowSubclass(top, Some(cursor_subclass_proc), CURSOR_SUBCLASS_ID);
let _ = EnumChildWindows(Some(top), Some(subclass_remove_cb), LPARAM(0));
}
}
/// Engage or release the pointer lock: confine + hide + recentre on, free + show on off.
/// Guarded so the `ClipCursor`/`ShowCursor` calls stay balanced (one each per transition).
/// Engaging captures the lock geometry (rect, centre, screen→host scale) — see `State::clip`.
fn set_locked(st: &mut State, on: bool) {
if on == st.locked {
return;
}
let hwnd = HWND(st.hwnd as *mut _);
unsafe {
if on {
let mut rc = RECT::default();
if GetClientRect(hwnd, &mut rc).is_ok() {
let mut tl = POINT {
x: rc.left,
y: rc.top,
};
let mut br = POINT {
x: rc.right,
y: rc.bottom,
};
let _ = ClientToScreen(hwnd, &mut tl);
let _ = ClientToScreen(hwnd, &mut br);
st.clip = RECT {
left: tl.x,
top: tl.y,
right: br.x,
bottom: br.y,
};
let _ = ClipCursor(Some(&st.clip as *const RECT));
st.center_x = (tl.x + br.x) / 2;
st.center_y = (tl.y + br.y) / 2;
// Screen px → host px: the Contain-fit display scale's inverse, so the host
// cursor tracks the physical mouse 1:1 on screen at any window size.
let (ww, wh) = ((br.x - tl.x).max(1) as f32, (br.y - tl.y).max(1) as f32);
let (vw, vh) = (st.mode.width.max(1) as f32, st.mode.height.max(1) as f32);
st.scale = (ww / vw).min(wh / vh).max(0.01);
let _ = SetCursorPos(st.center_x, st.center_y);
}
// Hide the OS cursor. ShowCursor(false) is the coarse gate; the subclass is what
// actually holds it hidden against WinUI's per-move arrow re-assertion — see
// cursor_subclass_proc / install_cursor_subclass.
let _ = ShowCursor(false);
CURSOR_HIDDEN.store(true, Ordering::Relaxed);
install_cursor_subclass(hwnd);
st.acc_x = 0.0;
st.acc_y = 0.0;
} else {
CURSOR_HIDDEN.store(false, Ordering::Relaxed);
let _ = ClipCursor(None);
let _ = ShowCursor(true);
}
}
st.locked = on;
}
/// The pre-fullscreen window placement, saved on entering fullscreen and restored on leaving it.
/// Module-level (not a `toggle_fullscreen`-local static) so the F11 toggle and the stream-stop exit
/// ([`uninstall`]) share the one saved placement, and its presence is also the "are we fullscreen?"
/// flag for [`exit_fullscreen`]. Only ever touched on the UI thread (the hook proc / the stream
/// page's unmount), but a Mutex keeps the static sound + `Sync`.
static SAVED_PLACEMENT: Mutex<Option<windows::Win32::UI::WindowsAndMessaging::WINDOWPLACEMENT>> =
Mutex::new(None);
/// Whether our top-level window is currently borderless-fullscreen. Entering strips
/// `WS_OVERLAPPEDWINDOW`, so its absence is the flag — no extra state beyond [`SAVED_PLACEMENT`].
fn is_fullscreen(hwnd: HWND) -> bool {
use windows::Win32::UI::WindowsAndMessaging::{
GetWindowLongPtrW, GWL_STYLE, WS_OVERLAPPEDWINDOW,
};
let overlapped = WS_OVERLAPPEDWINDOW.0 as isize;
unsafe { GetWindowLongPtrW(hwnd, GWL_STYLE) & overlapped == 0 }
}
/// Enter borderless fullscreen: remember the window placement, drop the frame
/// (`WS_OVERLAPPEDWINDOW`), and size the window to cover the whole monitor. windows-reactor owns
/// the WinUI window but exposes no fullscreen API, so we drive the HWND directly (parity with the
/// GTK client's F11). The SwapChainPanel follows the resulting `WM_SIZE` like any window resize.
fn enter_fullscreen(hwnd: HWND) {
use windows::Win32::Graphics::Gdi::{
GetMonitorInfoW, MonitorFromWindow, MONITORINFO, MONITOR_DEFAULTTOPRIMARY,
};
use windows::Win32::UI::WindowsAndMessaging::{
GetWindowLongPtrW, GetWindowPlacement, SetWindowLongPtrW, SetWindowPos, GWL_STYLE,
SWP_FRAMECHANGED, SWP_NOOWNERZORDER, SWP_NOZORDER, WINDOWPLACEMENT, WS_OVERLAPPEDWINDOW,
};
let overlapped = WS_OVERLAPPEDWINDOW.0 as isize;
unsafe {
let style = GetWindowLongPtrW(hwnd, GWL_STYLE);
let mut wp = WINDOWPLACEMENT {
length: std::mem::size_of::<WINDOWPLACEMENT>() as u32,
..Default::default()
};
let mut mi = MONITORINFO {
cbSize: std::mem::size_of::<MONITORINFO>() as u32,
..Default::default()
};
let mon = MonitorFromWindow(hwnd, MONITOR_DEFAULTTOPRIMARY);
if GetWindowPlacement(hwnd, &mut wp).is_ok() && GetMonitorInfoW(mon, &mut mi).as_bool() {
*SAVED_PLACEMENT.lock().unwrap() = Some(wp);
SetWindowLongPtrW(hwnd, GWL_STYLE, style & !overlapped);
let r = mi.rcMonitor;
let _ = SetWindowPos(
hwnd,
None,
r.left,
r.top,
r.right - r.left,
r.bottom - r.top,
SWP_NOZORDER | SWP_NOOWNERZORDER | SWP_FRAMECHANGED,
);
}
}
}
/// Leave borderless fullscreen: restore the frame style and the saved placement. A no-op when we
/// aren't fullscreen (nothing saved), so it's safe to call unconditionally on stream stop.
fn exit_fullscreen(hwnd: HWND) {
use windows::Win32::UI::WindowsAndMessaging::{
GetWindowLongPtrW, SetWindowLongPtrW, SetWindowPlacement, SetWindowPos, GWL_STYLE,
SWP_FRAMECHANGED, SWP_NOMOVE, SWP_NOOWNERZORDER, SWP_NOSIZE, SWP_NOZORDER,
WS_OVERLAPPEDWINDOW,
};
let Some(wp) = SAVED_PLACEMENT.lock().unwrap().take() else {
return; // never went fullscreen — nothing to restore
};
let overlapped = WS_OVERLAPPEDWINDOW.0 as isize;
unsafe {
let style = GetWindowLongPtrW(hwnd, GWL_STYLE);
SetWindowLongPtrW(hwnd, GWL_STYLE, style | overlapped);
let _ = SetWindowPlacement(hwnd, &wp);
let _ = SetWindowPos(
hwnd,
None,
0,
0,
0,
0,
SWP_NOMOVE | SWP_NOSIZE | SWP_NOZORDER | SWP_NOOWNERZORDER | SWP_FRAMECHANGED,
);
}
}
/// Toggle borderless fullscreen for our top-level window (F11), the classic Win32 dance split into
/// [`enter_fullscreen`] / [`exit_fullscreen`] so the stream-stop path can force windowed too.
fn toggle_fullscreen(hwnd: isize) {
let hwnd = HWND(hwnd as *mut _);
if is_fullscreen(hwnd) {
exit_fullscreen(hwnd);
} else {
enter_fullscreen(hwnd);
}
}
fn send(c: &NativeClient, kind: InputKind, code: u32, x: i32, y: i32, flags: u32) {
let _ = c.send_input(&InputEvent {
kind,
_pad: [0; 3],
code,
x,
y,
flags,
});
}
/// System shortcuts that act on the LOCAL desktop when "capture system shortcuts" is off:
/// the Win keys, Alt+Tab, and Alt/Ctrl+Esc.
fn is_system_shortcut(st: &State, vk: u16) -> bool {
match vk {
0x5B | 0x5C => true, // L/R Win
0x09 => st.alt, // Alt+Tab
0x1B => st.alt || st.ctrl, // Alt+Esc / Ctrl+Esc
_ => false,
}
}
unsafe extern "system" fn kbd_proc(code: i32, wparam: WPARAM, lparam: LPARAM) -> LRESULT {
if code == HC_ACTION as i32 {
let kb = unsafe { &*(lparam.0 as *const KBDLLHOOKSTRUCT) };
let msg = wparam.0 as u32;
let up = msg == WM_KEYUP || msg == WM_SYSKEYUP;
let vk = kb.vkCode as u16;
let mut guard = STATE.lock().unwrap();
if let Some(st) = guard.as_mut() {
// Track modifier state from the hook's own event stream — reliable even while we
// swallow these keys (GetAsyncKeyState doesn't reflect keys suppressed by our own LL
// hook, which is why the shortcut never fired). Handles the generic + L/R vk codes.
match kb.vkCode {
0x11 | 0xA2 | 0xA3 => st.ctrl = !up, // (L/R)CONTROL
0x12 | 0xA4 | 0xA5 => st.alt = !up, // (L/R)MENU (Alt)
0x10 | 0xA0 | 0xA1 => st.shift = !up, // (L/R)SHIFT
_ => {}
}
let foreground = unsafe { GetForegroundWindow() }.0 as isize == st.hwnd;
if foreground {
// Capture toggle: Ctrl+Alt+Shift+Q (consumed locally, never forwarded).
if !up && vk == VK_Q.0 && st.ctrl && st.alt && st.shift {
let on = !st.captured;
set_captured(st, on);
tracing::info!(captured = on, "capture toggled (Ctrl+Alt+Shift+Q)");
return LRESULT(1);
}
// Disconnect: Ctrl+Alt+Shift+D (consumed locally). Release capture immediately so
// the cursor is free while the session winds down and the UI navigates home.
if !up && vk == VK_D.0 && st.ctrl && st.alt && st.shift {
set_captured(st, false);
// Deliberate user exit → close with QUIT_CLOSE_CODE so the host tears the session
// down immediately instead of holding the keep-alive linger for a reconnect.
st.connector.disconnect_quit();
st.stop.store(true, Ordering::SeqCst);
tracing::info!("disconnect requested (Ctrl+Alt+Shift+D)");
return LRESULT(1);
}
// Toggle the stats overlay: Ctrl+Alt+Shift+S (consumed locally). Seeded from
// Settings at install; this live toggle overrides it for the session — parity
// with the GTK client, where `s` flips the OSD without leaving the stream.
if !up && vk == VK_S.0 && st.ctrl && st.alt && st.shift {
let on = !HUD_VISIBLE.load(Ordering::Relaxed);
HUD_VISIBLE.store(on, Ordering::Relaxed);
tracing::info!(hud = on, "stats overlay toggled (Ctrl+Alt+Shift+S)");
return LRESULT(1);
}
// Toggle fullscreen: F11 (consumed locally, no modifiers — a client shortcut,
// never a wire key). Works captured or released. The window resize changes the
// client rect, so re-lock to recompute the pointer confinement + recentre.
if !up && vk == VK_F11.0 {
toggle_fullscreen(st.hwnd);
if st.locked {
set_locked(st, false);
set_locked(st, true);
}
tracing::info!("fullscreen toggled (F11)");
return LRESULT(1);
}
if st.captured {
// With shortcut capture off, hand Alt+Tab & co. to the local desktop —
// neither forwarded nor swallowed.
if !st.inhibit_shortcuts && is_system_shortcut(st, vk) {
return unsafe { CallNextHookEx(None, code, wparam, lparam) };
}
// Wire key: the US-positional VK for this physical key (module docs), derived
// from the scancode. `vkCode` is layout-semantic and only passes through for
// keys the table doesn't cover — extended keys and everything outside the
// typing area, where positional == semantic (plus injected events with
// scanCode 0 from remapping tools, best-effort).
let ext = (kb.flags.0 & LLKHF_EXTENDED.0) != 0;
let v = if ext {
vk as u8
} else {
scan_to_positional_vk(kb.scanCode as u16).unwrap_or(vk as u8)
};
if up {
if st.held_keys.remove(&v) {
send(&st.connector, InputKind::KeyUp, v as u32, 0, 0, 0);
}
} else {
st.held_keys.insert(v);
send(&st.connector, InputKind::KeyDown, v as u32, 0, 0, 0);
}
return LRESULT(1); // swallow so it reaches the host, not the local OS
}
}
}
}
unsafe { CallNextHookEx(None, code, wparam, lparam) }
}
/// Whether a screen point lies inside the window's CURRENT client area (the click-to-capture
/// hit test — computed fresh per click, since the window can move/resize while released).
fn in_client_area(hwnd: isize, pt: POINT) -> bool {
let hwnd = HWND(hwnd as *mut _);
let mut rc = RECT::default();
if unsafe { GetClientRect(hwnd, &mut rc) }.is_err() {
return false;
}
let mut tl = POINT {
x: rc.left,
y: rc.top,
};
let mut br = POINT {
x: rc.right,
y: rc.bottom,
};
unsafe {
let _ = ClientToScreen(hwnd, &mut tl);
let _ = ClientToScreen(hwnd, &mut br);
}
pt.x >= tl.x && pt.x < br.x && pt.y >= tl.y && pt.y < br.y
}
unsafe extern "system" fn mouse_proc(code: i32, wparam: WPARAM, lparam: LPARAM) -> LRESULT {
if code == HC_ACTION as i32 {
let ms = unsafe { &*(lparam.0 as *const MSLLHOOKSTRUCT) };
let msg = wparam.0 as u32;
let injected = (ms.flags & LLMHF_INJECTED) != 0;
let mut guard = STATE.lock().unwrap();
if let Some(st) = guard.as_mut() {
let foreground = unsafe { GetForegroundWindow() }.0 as isize == st.hwnd;
let want_lock = st.captured && foreground;
if want_lock != st.locked {
set_locked(st, want_lock); // sync to focus changes (e.g. lost foreground)
}
// Click-to-capture: after a Ctrl+Alt+Shift+Q release, a primary click on the stream
// re-engages capture. The click is consumed — it starts the grab, it isn't gameplay.
if !st.captured
&& foreground
&& msg == WM_LBUTTONDOWN
&& !injected
&& in_client_area(st.hwnd, ms.pt)
{
set_captured(st, true);
tracing::info!("capture re-engaged (click on stream)");
return LRESULT(1);
}
if st.locked {
// Skip the synthetic move our own SetCursorPos recentre generates.
if injected {
return unsafe { CallNextHookEx(None, code, wparam, lparam) };
}
let c = st.connector.clone();
match msg {
WM_MOUSEMOVE => {
let dx = (ms.pt.x - st.center_x) as f32;
let dy = (ms.pt.y - st.center_y) as f32;
if dx != 0.0 || dy != 0.0 {
st.acc_x += dx / st.scale;
st.acc_y += dy / st.scale;
let (hx, hy) = (st.acc_x.trunc() as i32, st.acc_y.trunc() as i32);
st.acc_x -= hx as f32;
st.acc_y -= hy as f32;
if hx != 0 || hy != 0 {
send(&c, InputKind::MouseMove, 0, hx, hy, 0);
}
}
let _ = unsafe { SetCursorPos(st.center_x, st.center_y) };
}
WM_LBUTTONDOWN => button(st, 1, true),
WM_LBUTTONUP => button(st, 1, false),
WM_RBUTTONDOWN => button(st, 3, true),
WM_RBUTTONUP => button(st, 3, false),
WM_MBUTTONDOWN => button(st, 2, true),
WM_MBUTTONUP => button(st, 2, false),
WM_XBUTTONDOWN => button(st, 3 + ((ms.mouseData >> 16) as u16 as u32), true),
WM_XBUTTONUP => button(st, 3 + ((ms.mouseData >> 16) as u16 as u32), false),
WM_MOUSEWHEEL => send(
&c,
InputKind::MouseScroll,
0,
(ms.mouseData >> 16) as i16 as i32,
0,
0,
),
WM_MOUSEHWHEEL => send(
&c,
InputKind::MouseScroll,
1,
(ms.mouseData >> 16) as i16 as i32,
0,
0,
),
_ => {}
}
return LRESULT(1); // swallow inside the locked window
}
}
}
unsafe { CallNextHookEx(None, code, wparam, lparam) }
}
fn button(st: &mut State, id: u32, down: bool) {
let c = st.connector.clone();
if down {
st.held_buttons.insert(id);
send(&c, InputKind::MouseButtonDown, id, 0, 0, 0);
} else if st.held_buttons.remove(&id) {
send(&c, InputKind::MouseButtonUp, id, 0, 0, 0);
}
}
/// Set-1 make scancode → US-positional VK for the layout-**variant** typing area (letters, digit
/// row, OEM punctuation, the ISO 102nd key) — the exact inverse of the host injector's positional
/// table and the Windows analogue of the Linux client's `evdev_to_vk`. Keys not listed (F-row,
/// nav cluster, numpad, modifiers — plus every E0-extended key, which the caller filters out)
/// have layout-invariant VKs, so the hook's `vkCode` is already correct for them.
fn scan_to_positional_vk(scan: u16) -> Option<u8> {
Some(match scan {
0x02..=0x0A => (scan - 0x02) as u8 + 0x31, // 1..9
0x0B => 0x30, // 0
0x0C => 0xBD, // -_ VK_OEM_MINUS (DE: ß)
0x0D => 0xBB, // =+ VK_OEM_PLUS
0x10 => 0x51, // Q
0x11 => 0x57, // W
0x12 => 0x45, // E
0x13 => 0x52, // R
0x14 => 0x54, // T
0x15 => 0x59, // Y position (QWERTZ: the Z key)
0x16 => 0x55, // U
0x17 => 0x49, // I
0x18 => 0x4F, // O
0x19 => 0x50, // P
0x1A => 0xDB, // [{ VK_OEM_4 (DE: ü)
0x1B => 0xDD, // ]} VK_OEM_6
0x1E => 0x41, // A
0x1F => 0x53, // S
0x20 => 0x44, // D
0x21 => 0x46, // F
0x22 => 0x47, // G
0x23 => 0x48, // H
0x24 => 0x4A, // J
0x25 => 0x4B, // K
0x26 => 0x4C, // L
0x27 => 0xBA, // ;: VK_OEM_1 (DE: ö)
0x28 => 0xDE, // '" VK_OEM_7 (DE: ä)
0x29 => 0xC0, // `~ VK_OEM_3 (DE: ^)
0x2B => 0xDC, // \| VK_OEM_5
0x2C => 0x5A, // Z position (QWERTZ: the Y key)
0x2D => 0x58, // X
0x2E => 0x43, // C
0x2F => 0x56, // V
0x30 => 0x42, // B
0x31 => 0x4E, // N
0x32 => 0x4D, // M
0x33 => 0xBC, // ,< VK_OEM_COMMA
0x34 => 0xBE, // .> VK_OEM_PERIOD
0x35 => 0xBF, // /? VK_OEM_2
0x56 => 0xE2, // <>| VK_OEM_102 (ISO)
_ => return None,
})
}
#[cfg(test)]
mod tests {
use super::*;
/// The German-scramble regression pins: the physical keys a QWERTZ board labels Z/Y/ö/ü must
/// leave this client as their US-position VKs, regardless of the local layout's vkCode.
#[test]
fn positional_pins_for_the_qwertz_scramble() {
assert_eq!(scan_to_positional_vk(0x15), Some(0x59)); // QWERTZ Z key → VK_Y (US position)
assert_eq!(scan_to_positional_vk(0x2C), Some(0x5A)); // QWERTZ Y key → VK_Z (US position)
assert_eq!(scan_to_positional_vk(0x27), Some(0xBA)); // ö key → VK_OEM_1 (US ;: position)
assert_eq!(scan_to_positional_vk(0x1A), Some(0xDB)); // ü key → VK_OEM_4 (US [{ position)
assert_eq!(scan_to_positional_vk(0x28), Some(0xDE)); // ä key → VK_OEM_7 (US '" position)
assert_eq!(scan_to_positional_vk(0x0C), Some(0xBD)); // ß key → VK_OEM_MINUS (US -_ position)
}
/// Keys outside the layout-variant typing area stay un-mapped (vkCode passes through).
#[test]
fn invariant_keys_fall_through() {
for scan in [
0x01u16, 0x0E, 0x0F, 0x1C, 0x1D, 0x2A, 0x36, 0x38, 0x39, 0x3B, 0x45, 0x57,
] {
assert_eq!(scan_to_positional_vk(scan), None, "scan 0x{scan:02X}");
}
}
/// Exactly the 48 typing-area keys are covered (10 digits + 26 letters + 12 OEM), and every
/// mapping is unique — two physical keys must never collapse onto one wire VK.
#[test]
fn table_covers_the_typing_area_bijectively() {
let mapped: Vec<(u16, u8)> = (0u16..=0xFF)
.filter_map(|sc| scan_to_positional_vk(sc).map(|vk| (sc, vk)))
.collect();
assert_eq!(mapped.len(), 48);
let mut vks: Vec<u8> = mapped.iter().map(|&(_, vk)| vk).collect();
vks.sort_unstable();
vks.dedup();
assert_eq!(vks.len(), 48, "duplicate wire VK in the positional table");
}
}
+15 -162
View File
@@ -1,17 +1,16 @@
//! `punktfunk-client` — the native Windows punktfunk/1 client. //! `punktfunk-client` — the native Windows punktfunk/1 client.
//! //!
//! Pure Rust: `NativeClient` linked as a crate (no C ABI, like the GTK Linux client) · FFmpeg //! Pure Rust: `NativeClient` linked as a crate (no C ABI, like the GTK Linux client) · SDL3
//! decode · WASAPI audio · SDL3 gamepads · a **WinUI 3** shell (windows-reactor) with the video //! gamepads · a **WinUI 3** shell (windows-reactor). Streaming (decode + present + audio) runs in
//! on a `SwapChainPanel` bound to a D3D11 composition swapchain. The trust surface mirrors the //! the spawned `punktfunk-session` Vulkan binary; the shell owns host selection, trust and
//! pairing. The trust surface mirrors the
//! other native clients: persistent identity, trust-on-first-use, SPAKE2 PIN pairing — all in-app //! other native clients: persistent identity, trust-on-first-use, SPAKE2 PIN pairing — all in-app
//! (host list, settings, pairing). `--headless` keeps a CLI connect path for tests/measurement. //! (host list, settings, pairing). Streaming runs in the spawned `punktfunk-session` binary;
//! `--headless --speed-test` keeps a decode-less CLI measurement path.
//! //!
//! Usage: //! Usage:
//! punktfunk-client (open the WinUI 3 window: host list, settings, pairing) //! punktfunk-client (open the WinUI 3 window: host list, settings, pairing)
//! punktfunk-client --discover (list punktfunk hosts on the LAN) //! punktfunk-client --discover (list punktfunk hosts on the LAN)
//! punktfunk-client --headless --connect host[:port] [--pin HEX] [--pair PIN] [--mode WxHxHz]
//! [--bitrate MBPS] [--mic] [--decoder auto|hardware|software] [--no-hdr]
//! (no window; count frames + print stats)
//! punktfunk-client --headless --speed-test --connect host[:port] //! punktfunk-client --headless --speed-test --connect host[:port]
//! (measure the path: probe burst → goodput / loss / recommended bitrate) //! (measure the path: probe burst → goodput / loss / recommended bitrate)
@@ -23,29 +22,19 @@
#[cfg(windows)] #[cfg(windows)]
mod app; mod app;
#[cfg(windows)] #[cfg(windows)]
mod audio;
#[cfg(windows)]
mod discovery; mod discovery;
#[cfg(windows)] #[cfg(windows)]
mod gamepad; mod gamepad;
#[cfg(windows)] #[cfg(windows)]
mod gpu; mod gpu;
#[cfg(windows)] #[cfg(windows)]
mod input; mod probe;
#[cfg(windows)]
mod present;
#[cfg(windows)]
mod render;
#[cfg(windows)]
mod session;
#[cfg(windows)] #[cfg(windows)]
mod shell_window; mod shell_window;
#[cfg(windows)] #[cfg(windows)]
mod spawn; mod spawn;
#[cfg(windows)] #[cfg(windows)]
mod trust; mod trust;
#[cfg(windows)]
mod video;
#[cfg(windows)] #[cfg(windows)]
mod wol; mod wol;
@@ -124,13 +113,11 @@ fn set_app_user_model_id() {
} }
} }
/// `--headless --connect host[:port]`: connect from the CLI, count frames, print stats — the /// `--headless --speed-test --connect host[:port]`: measure the path over the real data plane and
/// Windows analogue of `punktfunk-probe`. /// print the outcome — the Windows analogue of `punktfunk-probe`. The former in-process
/// frame-count connect path went with the legacy builtin stream; real streaming is windowed-only.
#[cfg(windows)] #[cfg(windows)]
fn run_headless_cli(args: &[String], identity: (String, String)) { fn run_headless_cli(args: &[String], identity: (String, String)) {
use punktfunk_core::config::{CompositorPref, GamepadPref, Mode};
use std::time::{Duration, Instant};
let arg = |name: &str| -> Option<String> { let arg = |name: &str| -> Option<String> {
args.iter() args.iter()
.position(|a| a == name) .position(|a| a == name)
@@ -154,7 +141,7 @@ fn run_headless_cli(args: &[String], identity: (String, String)) {
let fp = trust::KnownHosts::load() let fp = trust::KnownHosts::load()
.find_by_addr(&host, port) .find_by_addr(&host, port)
.map(|k| k.fp_hex.clone()); .map(|k| k.fp_hex.clone());
match session::run_speed_probe(&host, port, fp.as_deref(), identity) { match probe::run_speed_probe(&host, port, fp.as_deref(), identity) {
Ok(r) => { Ok(r) => {
let mbps = f64::from(r.throughput_kbps) / 1000.0; let mbps = f64::from(r.throughput_kbps) / 1000.0;
let recommended = f64::from(r.throughput_kbps / 10 * 7) / 1000.0; let recommended = f64::from(r.throughput_kbps / 10 * 7) / 1000.0;
@@ -171,144 +158,10 @@ fn run_headless_cli(args: &[String], identity: (String, String)) {
} }
return; return;
} }
let mode = arg("--mode") // Only --speed-test remains headless: real streaming runs in the windowed app's spawned
.and_then(|m| { // punktfunk-session binary, which the deleted in-process frame-count path was replaced by.
let mut it = m.split(['x', 'X']); eprintln!("--headless supports only --speed-test now \u{2014} run the windowed app to stream");
Some(Mode { std::process::exit(2);
width: it.next()?.parse().ok()?,
height: it.next()?.parse().ok()?,
refresh_hz: it.next()?.parse().ok()?,
})
})
.unwrap_or(Mode {
width: 1280,
height: 720,
refresh_hz: 60,
});
let bitrate_kbps = arg("--bitrate")
.and_then(|b| b.parse::<u32>().ok())
.map(|m| m * 1000)
.unwrap_or(0);
let known = trust::KnownHosts::load();
let mut pin = arg("--pin")
.and_then(|h| trust::parse_hex32(&h))
.or_else(|| {
known
.find_by_addr(&host, port)
.and_then(|k| trust::parse_hex32(&k.fp_hex))
});
if let Some(code) = arg("--pair") {
let name = std::env::var("COMPUTERNAME").unwrap_or_else(|_| "windows-client".into());
match punktfunk_core::client::NativeClient::pair(
&host,
port,
(&identity.0, &identity.1),
code.trim(),
&name,
Duration::from_secs(90),
) {
Ok(fp) => {
let mut k = trust::KnownHosts::load();
k.upsert(trust::KnownHost {
name: host.clone(),
addr: host.clone(),
port,
fp_hex: trust::hex(&fp),
paired: true,
last_used: None,
mac: Vec::new(),
});
let _ = k.save();
tracing::info!(fp = %trust::hex(&fp), "paired");
pin = Some(fp);
}
Err(e) => {
eprintln!("Pairing failed: {e:?}");
std::process::exit(1);
}
}
}
let decoder = arg("--decoder")
.map(|d| crate::video::DecoderPref::from_name(&d))
.unwrap_or_default();
tracing::info!(%host, port, ?mode, tofu = pin.is_none(), ?decoder, "connecting (headless)");
let handle = session::start(session::SessionParams {
host,
port,
mode,
compositor: CompositorPref::Auto,
gamepad: GamepadPref::Auto,
bitrate_kbps,
// Headless CLI path (test/scripting) — stereo baseline; the GUI sources this from settings.
audio_channels: 2,
mic_enabled: flag("--mic"),
hdr_enabled: !flag("--no-hdr"),
decoder,
// `--codec h264|hevc|av1` sets the soft preference; default auto (host decides).
preferred_codec: match arg("--codec").as_deref() {
Some("h264") | Some("avc") => punktfunk_core::quic::CODEC_H264,
Some("hevc") | Some("h265") => punktfunk_core::quic::CODEC_HEVC,
Some("av1") => punktfunk_core::quic::CODEC_AV1,
_ => 0,
},
pin,
identity,
// Headless CLI uses the normal (short) handshake budget; the long request-access wait is a
// GUI-only flow.
connect_timeout: Duration::from_secs(15),
});
let deadline = Instant::now() + Duration::from_secs(60);
let mut frames_seen = 0u64;
loop {
while let Ok(ev) = handle.events.try_recv() {
match ev {
session::SessionEvent::Connected {
mode, fingerprint, ..
} => tracing::info!(?mode, fp = %trust::hex(&fingerprint), "connected"),
// With per-AU 0xCF host timings the combined host+network stage splits into
// host (capture→sent on the host) + net; an old host emits none → combined only.
session::SessionEvent::Stats(s) if s.split => tracing::info!(
fps = format!("{:.0}", s.fps),
mbps = format!("{:.1}", s.mbps),
decode_p50_ms = format!("{:.2}", s.decode_ms),
hostnet_p50_ms = format!("{:.2}", s.hostnet_ms),
host_p50_ms = format!("{:.2}", s.host_ms),
net_p50_ms = format!("{:.2}", s.net_ms),
frames_seen,
"stats"
),
session::SessionEvent::Stats(s) => tracing::info!(
fps = format!("{:.0}", s.fps),
mbps = format!("{:.1}", s.mbps),
decode_p50_ms = format!("{:.2}", s.decode_ms),
hostnet_p50_ms = format!("{:.2}", s.hostnet_ms),
frames_seen,
"stats"
),
session::SessionEvent::Failed { msg, .. } => {
tracing::error!(%msg, "connect failed");
return;
}
session::SessionEvent::Ended(err) => {
tracing::info!(reason = err.as_deref().unwrap_or("done"), "session ended");
return;
}
}
}
while handle.frames.try_recv().is_ok() {
frames_seen += 1;
}
if Instant::now() > deadline {
tracing::info!(frames_seen, "harness deadline — stopping");
handle.stop.store(true, std::sync::atomic::Ordering::SeqCst);
return;
}
std::thread::sleep(Duration::from_millis(2));
}
} }
/// `--discover`: browse the LAN for punktfunk hosts (mDNS) and print them, then exit. /// `--discover`: browse the LAN for punktfunk hosts (mDNS) and print them, then exit.
-915
View File
@@ -1,915 +0,0 @@
//! Direct3D11 presenter for a WinUI 3 `SwapChainPanel`. It draws a decoded frame Contain-fit into a
//! **composition** flip-model swapchain, which the reactor stream page binds to the panel via
//! `SwapChainPanelHandle::set_swap_chain`. After that one UI-thread bind, the presenter lives on
//! the dedicated render thread ([`crate::render`]) — presenting never touches (or is stalled by)
//! the XAML thread.
//!
//! Two frame sources, ONE YCbCr→RGB shader whose conversion rows arrive per frame in a constant
//! buffer (`pf_client_core::video::csc_rows` from the frame's CICP signaling — identical colour
//! math for both sources, and the stream's signaled matrix/range is honored, not assumed):
//!
//! * **GPU (D3D11VA)** — [`crate::video::GpuFrame`] is a slice of the decoder-only NV12/P010
//! texture array. One `CopySubresourceRegion` with a display-size box moves the slice — **both
//! planes; in D3D11 a planar slice is a single subresource** (unlike D3D12) — into our
//! sampleable texture, which per-plane SRVs (R8/R8G8, R16/R16G16) expose to the shaders. The
//! source box is mandatory: the decode array is coded-size (e.g. 1920×1088), the target
//! display-size (1920×1080), and D3D11 silently drops size-mismatched full-resource copies.
//! * **CPU upload** — [`crate::video::CpuFrame`] carries NV12/P010 planes from the software
//! decoder; they upload into two dynamic plane textures feeding the same SRV slots/shaders.
//!
//! **Pacing**: the swapchain is created with `DXGI_SWAP_CHAIN_FLAG_FRAME_LATENCY_WAITABLE_OBJECT`
//! and `SetMaximumFrameLatency(1)` (flagless fallback for odd drivers). The render thread waits
//! on the latency waitable before drawing, so at most one present is ever queued (minimum compose
//! latency) and a stream faster than the display drops frames *before* any GPU work. Every
//! `ResizeBuffers` must re-pass the creation flags — that's `swap_flags`.
//!
//! **HiDPI**: buffers are sized in physical pixels and `IDXGISwapChain2::SetMatrixTransform`
//! (scale 96/DPI) maps them to the panel's DIP coordinate space — without it XAML samples a
//! DIP-sized buffer up and the video is blurry at 125/150 % scaling.
//!
//! **HDR10**: when a frame is BT.2020 PQ the swapchain flips to `R10G10B10A2` +
//! `DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020` (+ HDR10 metadata) via `ResizeBuffers`/
//! `SetColorSpace1`; the shader output is already PQ-encoded so the compositor maps PQ→display. SDR
//! stays 8-bit B8G8R8A8.
//!
//! All `windows` types here come from the same windows-rs commit as `windows-reactor`, so the
//! `IDXGISwapChain1` handed to `set_swap_chain` satisfies reactor's `windows_core::Interface`.
use crate::video::{CpuFrame, DecodedFrame, GpuFrame};
use anyhow::{anyhow, Context, Result};
use windows::core::{Interface, PCSTR};
use windows::Win32::Foundation::{CloseHandle, HANDLE, WAIT_OBJECT_0};
use windows::Win32::Graphics::Direct3D::Fxc::{D3DCompile, D3DCOMPILE_OPTIMIZATION_LEVEL3};
use windows::Win32::Graphics::Direct3D::{
ID3DBlob, D3D_PRIMITIVE_TOPOLOGY_TRIANGLELIST, D3D_SRV_DIMENSION_TEXTURE2D,
};
use windows::Win32::Graphics::Direct3D11::*;
use windows::Win32::Graphics::Dxgi::Common::*;
use windows::Win32::Graphics::Dxgi::*;
use windows::Win32::System::Threading::WaitForSingleObject;
// One vertex shader (fullscreen triangle) + ONE pixel shader for every colour combination:
// tex0 is the luma plane, tex1 the chroma plane, and the YCbCr→RGB conversion arrives as three
// constant-buffer rows precomputed on the CPU per frame (`pf_client_core::video::csc_rows` —
// bit-depth exact, range expansion + the P010 ×65535/65472 high-bit repack folded in). One shader
// honors whatever the stream signals (BT.601/709/2020, full/limited, 8/10-bit) instead of the old
// two hardcoded matrices — a BT.601-signaled stream (a Linux host's RGB-input NVENC) used to
// render with BT.709 coefficients, a constant hue error. A PQ stream's rows yield PQ-encoded
// RGB passed through as-is to the HDR10 swapchain, exactly as before.
const SHADER_HLSL: &str = r#"
struct VSOut { float4 pos : SV_Position; float2 uv : TEXCOORD0; };
VSOut vs_main(uint vid : SV_VertexID) {
float2 uv = float2((vid << 1) & 2, vid & 2);
VSOut o;
o.pos = float4(uv * float2(2, -2) + float2(-1, 1), 0, 1);
o.uv = uv;
return o;
}
Texture2D tex0 : register(t0);
Texture2D tex1 : register(t1);
SamplerState smp : register(s0);
cbuffer Csc : register(b0) {
float4 r0; // rgb[i] = dot(ri.xyz, yuv) + ri.w
float4 r1;
float4 r2;
};
float4 ps_yuv(VSOut i) : SV_Target {
// 4:2:0 chroma is left-cosited (H.273 type 0 — the default inference when unsignaled, and
// what the hosts produce), but sampling the half-res plane at the luma UV assumes CENTER
// siting — a ~0.5-luma-px rightward chroma shift on hard colored edges. Offset +0.25 chroma
// texels to re-align (the same correction the Apple client applies). Self-disables when the
// plane widths match (a full-size 4:4:4 chroma plane has no subsampling to correct).
float lw, lh, cw, ch;
tex0.GetDimensions(lw, lh);
tex1.GetDimensions(cw, ch);
float2 cuv = i.uv;
if (cw < lw) { cuv.x += 0.25 / cw; }
float3 yuv = float3(tex0.Sample(smp, i.uv).r, tex1.Sample(smp, cuv).rg);
float3 rgb = float3(dot(r0.xyz, yuv) + r0.w,
dot(r1.xyz, yuv) + r1.w,
dot(r2.xyz, yuv) + r2.w);
return float4(saturate(rgb), 1.0);
}
"#;
/// The currently bound frame: per-plane SRVs (over the GPU sample texture or the CPU plane
/// textures). Redraws (resize, letterbox) re-present it — the CSC constant buffer still holds
/// this frame's rows, and the swapchain mode was latched by `set_hdr` when the frame arrived.
struct Bound {
y: ID3D11ShaderResourceView,
c: ID3D11ShaderResourceView,
}
pub struct Presenter {
device: ID3D11Device,
context: ID3D11DeviceContext,
vs: ID3D11VertexShader,
ps_yuv: ID3D11PixelShader,
/// Dynamic constant buffer holding the bound frame's three CSC rows (`csc_rows`), rewritten
/// on every bind (colour signaling can flip in-band, e.g. the host's SDR→HDR re-init).
csc_buf: ID3D11Buffer,
sampler: ID3D11SamplerState,
swap: IDXGISwapChain1,
/// Creation flags — MUST be re-passed to every `ResizeBuffers` or it fails.
swap_flags: u32,
/// The frame-latency waitable (owned; closed in `Drop`), `None` on the flagless fallback.
waitable: Option<HANDLE>,
rtv: Option<ID3D11RenderTargetView>,
/// GPU path: sampleable copy target for the decoded slice — `(tex, w, h, ten_bit)`, recreated
/// when the decoded size/bit depth changes. Format must equal the decode array's (NV12/P010).
sample_tex: Option<(ID3D11Texture2D, u32, u32, bool)>,
/// The last GPU frame, held until the NEXT bind so its decode surface stays out of the reuse
/// pool at least until this frame's copy has been queued ahead of any later decoder write.
gpu_frame: Option<GpuFrame>,
/// CPU path: dynamic luma + chroma plane textures + their SRVs — `(y, uv, y_srv, uv_srv, w, h,
/// ten_bit)`, recreated when the decoded size/bit depth changes.
#[allow(clippy::type_complexity)]
plane_tex: Option<(
ID3D11Texture2D,
ID3D11Texture2D,
ID3D11ShaderResourceView,
ID3D11ShaderResourceView,
u32,
u32,
bool,
)>,
bound: Option<Bound>,
/// Source frame dimensions, for the Contain-fit letterbox.
src_w: u32,
src_h: u32,
/// Panel (swapchain) size in physical pixels + the window DPI, updated on resize.
panel_w: u32,
panel_h: u32,
dpi: u32,
/// Whether the swapchain is currently in 10-bit HDR10 (R10G10B10A2 + ST.2084) mode.
hdr: bool,
/// The source's static HDR mastering metadata received over the protocol (`0xCE`), applied via
/// `SetHDRMetaData` so the display tone-maps from the real grade instead of a generic 1000-nit
/// guess. `None` until the first update arrives (then the generic baseline is used).
hdr_meta: Option<punktfunk_core::quic::HdrMeta>,
}
/// Latest source HDR mastering metadata, written by the session pump (`session.rs`, the sole
/// `next_hdr_meta` consumer) and read by the render thread before each present — decoupled so the
/// presenter doesn't need the connector. One session at a time on the client, so a single slot.
pub static LATEST_HDR_META: std::sync::Mutex<Option<punktfunk_core::quic::HdrMeta>> =
std::sync::Mutex::new(None);
impl Presenter {
/// Create the presenter on the process-wide shared D3D11 device (the one the decoder uses), plus
/// the composition swapchain + shaders, sized to the panel in physical pixels at `dpi`.
pub fn new(width: u32, height: u32, dpi: u32) -> Result<Presenter> {
let shared = crate::gpu::shared().ok_or_else(|| anyhow!("no shared D3D11 device"))?;
let device = shared.device.clone();
let context = shared.context.clone();
let (vs, ps_yuv, sampler) = build_pipeline(&device)?;
// The per-frame CSC rows (three float4s). Dynamic: rewritten with Map-discard on bind.
let csc_desc = D3D11_BUFFER_DESC {
ByteWidth: 48,
Usage: D3D11_USAGE_DYNAMIC,
BindFlags: D3D11_BIND_CONSTANT_BUFFER.0 as u32,
CPUAccessFlags: D3D11_CPU_ACCESS_WRITE.0 as u32,
..Default::default()
};
let csc_buf = unsafe {
let mut b = None;
device
.CreateBuffer(&csc_desc, None, Some(&mut b))
.context("CreateBuffer (CSC rows)")?;
b.ok_or_else(|| anyhow!("null CSC constant buffer"))?
};
let (swap, swap_flags) =
create_composition_swapchain(&device, width.max(1), height.max(1))?;
// ≤1 queued present: the render thread blocks on the waitable, so a frame is only drawn
// when the compositor is ready to take it — the newest-wins drain happens after the wait.
let waitable = (swap_flags & DXGI_SWAP_CHAIN_FLAG_FRAME_LATENCY_WAITABLE_OBJECT.0 as u32
!= 0)
.then(|| unsafe {
let sc2: IDXGISwapChain2 = swap.cast().ok()?;
sc2.SetMaximumFrameLatency(1).ok()?;
let h = sc2.GetFrameLatencyWaitableObject();
(!h.is_invalid()).then_some(h)
})
.flatten();
let p = Presenter {
device,
context,
vs,
ps_yuv,
csc_buf,
sampler,
swap,
swap_flags,
waitable,
rtv: None,
sample_tex: None,
gpu_frame: None,
plane_tex: None,
bound: None,
src_w: 1,
src_h: 1,
panel_w: width.max(1),
panel_h: height.max(1),
dpi: dpi.max(96),
hdr: false,
hdr_meta: None,
};
p.apply_dpi_matrix();
Ok(p)
}
/// Block until the swapchain can take another present (≤ `timeout_ms`). True when a present
/// slot is free; also true on the flagless fallback (no throttle available, just present).
pub fn wait_present_slot(&self, timeout_ms: u32) -> bool {
match self.waitable {
Some(h) => unsafe { WaitForSingleObject(h, timeout_ms) == WAIT_OBJECT_0 },
None => true,
}
}
/// Update the source HDR mastering metadata (from the `0xCE` plane). Stored for the next HDR
/// swapchain switch, and applied immediately if already presenting HDR. A no-op when unchanged
/// (so it's cheap to call every frame from the render loop).
pub fn set_hdr_metadata(&mut self, meta: punktfunk_core::quic::HdrMeta) {
if self.hdr_meta == Some(meta) {
return;
}
self.hdr_meta = Some(meta);
if self.hdr {
unsafe { self.apply_hdr_metadata() };
}
}
/// The DXGI swapchain to hand to `SwapChainPanelHandle::set_swap_chain`.
pub fn swap_chain(&self) -> &IDXGISwapChain1 {
&self.swap
}
/// Resize the back buffers to the panel's new size in physical pixels at `dpi` (drops the
/// stale RTV, re-applies the DIP↔pixel matrix).
pub fn resize(&mut self, width: u32, height: u32, dpi: u32) {
let dpi = dpi.max(96);
if width == 0
|| height == 0
|| (width == self.panel_w && height == self.panel_h && dpi == self.dpi)
{
return;
}
self.rtv = None; // release all back-buffer refs before ResizeBuffers
unsafe {
if let Err(e) = self.swap.ResizeBuffers(
0,
width,
height,
DXGI_FORMAT_UNKNOWN,
DXGI_SWAP_CHAIN_FLAG(self.swap_flags as i32),
) {
tracing::warn!(error = %e, "ResizeBuffers failed");
return;
}
}
self.panel_w = width;
self.panel_h = height;
self.dpi = dpi;
self.apply_dpi_matrix();
}
/// Map the pixel-sized buffers into the panel's DIP coordinate space (scale 96/DPI) — XAML
/// otherwise stretches whatever size the buffers are to the panel's DIP bounds (blurry).
fn apply_dpi_matrix(&self) {
let s = 96.0 / self.dpi as f32;
if let Ok(sc2) = self.swap.cast::<IDXGISwapChain2>() {
let m = DXGI_MATRIX_3X2_F {
_11: s,
_22: s,
..Default::default()
};
if let Err(e) = unsafe { sc2.SetMatrixTransform(&m) } {
tracing::warn!(error = %e, "SetMatrixTransform failed");
}
}
}
/// Present one decoded frame (Contain-fit) — or, when `frame` is `None`, re-present the last
/// one (or black). Called from the render thread. Takes the frame by value: the GPU path
/// retains the decoder surface until the next bind.
pub fn present(&mut self, frame: Option<DecodedFrame>) {
match frame {
Some(DecodedFrame::Cpu(c)) => {
if c.hdr != self.hdr {
self.set_hdr(c.hdr);
}
if let Err(e) = self.upload(&c) {
tracing::warn!(error = %e, "frame upload failed");
}
}
Some(DecodedFrame::Gpu(g)) => {
if g.hdr != self.hdr {
self.set_hdr(g.hdr);
}
if let Err(e) = self.bind_gpu(g) {
tracing::warn!(error = %e, "GPU frame bind failed");
}
}
None => {}
}
self.draw();
}
/// Copy the decoded slice into our sampleable texture and build per-plane SRVs over it. The
/// decode array is decoder-only (NVIDIA won't bind a decoder array as a shader resource), so
/// it can't be sampled directly — one GPU-to-GPU copy makes the frame sampleable on every
/// vendor. D3D11 planar semantics: the slice is ONE subresource (both planes copy together),
/// and the source box is display-size (the array is coded-size; a full-resource copy would
/// size-mismatch and be silently dropped).
fn bind_gpu(&mut self, g: GpuFrame) -> Result<()> {
let src: ID3D11Texture2D = unsafe {
let raw = g.texture_ptr();
ID3D11Texture2D::from_raw_borrowed(&raw)
.ok_or_else(|| anyhow!("null D3D11 texture"))?
.clone()
};
self.ensure_sample_tex(g.width, g.height, g.ten_bit)?;
let dst = self.sample_tex.as_ref().unwrap().0.clone();
// Even-aligned luma coordinates (NV12/P010 chroma is 2×2 subsampled).
let src_box = D3D11_BOX {
left: 0,
top: 0,
front: 0,
right: g.width & !1,
bottom: g.height & !1,
back: 1,
};
unsafe {
self.context
.CopySubresourceRegion(&dst, 0, 0, 0, 0, &src, g.index, Some(&src_box));
}
let (fy, fc) = plane_formats(g.ten_bit);
let y = self.plane_srv(&dst, fy)?;
let c = self.plane_srv(&dst, fc)?;
self.write_csc_rows(g.color, g.ten_bit)?;
self.src_w = g.width;
self.src_h = g.height;
self.bound = Some(Bound { y, c });
// Hold the frame until the next bind: its decode surface stays out of the reuse pool
// until this copy is queued ahead of any later decoder write (previous frame drops here).
self.gpu_frame = Some(g);
Ok(())
}
/// Ensure the sampleable copy texture matches the decoded frame's size + bit depth (NV12 for
/// 8-bit, P010 for 10-bit — the same format as the decode array, a `CopySubresourceRegion`
/// requirement), recreating it on a change.
fn ensure_sample_tex(&mut self, w: u32, h: u32, ten_bit: bool) -> Result<()> {
if matches!(&self.sample_tex, Some((_, tw, th, tb)) if *tw == w && *th == h && *tb == ten_bit)
{
return Ok(());
}
let desc = D3D11_TEXTURE2D_DESC {
Width: w,
Height: h,
MipLevels: 1,
ArraySize: 1,
Format: if ten_bit {
DXGI_FORMAT_P010
} else {
DXGI_FORMAT_NV12
},
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: D3D11_USAGE_DEFAULT,
BindFlags: D3D11_BIND_SHADER_RESOURCE.0 as u32,
CPUAccessFlags: 0,
MiscFlags: 0,
};
let tex = unsafe {
let mut t = None;
self.device
.CreateTexture2D(&desc, None, Some(&mut t))
.context("CreateTexture2D (sample target)")?;
t.ok_or_else(|| anyhow!("null sample texture"))?
};
self.sample_tex = Some((tex, w, h, ten_bit));
Ok(())
}
/// A shader-resource view over one plane of a single (non-array) NV12/P010 texture — the
/// R8/R8G8 (or R16/R16G16) format selects the luma vs. chroma plane (the D3D11 video
/// sub-format trick).
fn plane_srv(
&self,
tex: &ID3D11Texture2D,
format: DXGI_FORMAT,
) -> Result<ID3D11ShaderResourceView> {
let desc = D3D11_SHADER_RESOURCE_VIEW_DESC {
Format: format,
ViewDimension: D3D_SRV_DIMENSION_TEXTURE2D,
Anonymous: D3D11_SHADER_RESOURCE_VIEW_DESC_0 {
Texture2D: D3D11_TEX2D_SRV {
MostDetailedMip: 0,
MipLevels: 1,
},
},
};
unsafe {
let mut srv = None;
self.device
.CreateShaderResourceView(tex, Some(&desc), Some(&mut srv))
.context("CreateShaderResourceView (plane)")?;
srv.ok_or_else(|| anyhow!("null SRV"))
}
}
/// Upload a software-decoded frame's two planes into the dynamic plane textures (created to
/// match size/bit depth), feeding the same SRV slots + shaders as the GPU path.
fn upload(&mut self, frame: &CpuFrame) -> Result<()> {
let (w, h) = (frame.width, frame.height);
let rebuild = !matches!(&self.plane_tex,
Some((.., tw, th, tb)) if *tw == w && *th == h && *tb == frame.ten_bit);
if rebuild {
let (fy, fc) = plane_formats(frame.ten_bit);
let y = self.dynamic_tex(w, h, fy)?;
let uv = self.dynamic_tex(w.div_ceil(2), h.div_ceil(2), fc)?;
let y_srv = self.plane_srv(&y, fy)?;
let uv_srv = self.plane_srv(&uv, fc)?;
self.plane_tex = Some((y, uv, y_srv, uv_srv, w, h, frame.ten_bit));
}
let (y, uv, y_srv, uv_srv, ..) = self.plane_tex.as_ref().unwrap();
let bytes = if frame.ten_bit { 2 } else { 1 };
self.map_rows(y, &frame.y, frame.y_stride, w as usize * bytes, h as usize)?;
self.map_rows(
uv,
&frame.uv,
frame.uv_stride,
w.div_ceil(2) as usize * 2 * bytes,
h.div_ceil(2) as usize,
)?;
let (y_srv, uv_srv) = (y_srv.clone(), uv_srv.clone());
self.write_csc_rows(frame.color, frame.ten_bit)?;
self.src_w = w;
self.src_h = h;
self.bound = Some(Bound {
y: y_srv,
c: uv_srv,
});
self.gpu_frame = None; // drop any held GPU frame
Ok(())
}
fn dynamic_tex(&self, w: u32, h: u32, format: DXGI_FORMAT) -> Result<ID3D11Texture2D> {
let desc = D3D11_TEXTURE2D_DESC {
Width: w,
Height: h,
MipLevels: 1,
ArraySize: 1,
Format: format,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: D3D11_USAGE_DYNAMIC,
BindFlags: D3D11_BIND_SHADER_RESOURCE.0 as u32,
CPUAccessFlags: D3D11_CPU_ACCESS_WRITE.0 as u32,
MiscFlags: 0,
};
unsafe {
let mut t = None;
self.device
.CreateTexture2D(&desc, None, Some(&mut t))
.context("CreateTexture2D (plane)")?;
t.ok_or_else(|| anyhow!("null plane texture"))
}
}
/// Recompute the bound frame's YCbCr→RGB rows from its CICP signaling and Map-discard them
/// into the CSC constant buffer. `ten_bit` selects the 10-bit code points AND the P010
/// high-bit repack (the plane SRVs are R16/R16G16 UNORM for 10-bit).
fn write_csc_rows(&self, color: pf_client_core::video::ColorDesc, ten_bit: bool) -> Result<()> {
let rows = pf_client_core::video::csc_rows(color, if ten_bit { 10 } else { 8 }, ten_bit);
unsafe {
let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
self.context
.Map(
&self.csc_buf,
0,
D3D11_MAP_WRITE_DISCARD,
0,
Some(&mut mapped),
)
.context("Map CSC constant buffer")?;
std::ptr::copy_nonoverlapping(
rows.as_ptr() as *const u8,
mapped.pData as *mut u8,
48, // [[f32; 4]; 3]
);
self.context.Unmap(&self.csc_buf, 0);
}
Ok(())
}
/// Map-discard `tex` and copy `rows` rows of `row_bytes` from `src` (stride `src_pitch`).
fn map_rows(
&self,
tex: &ID3D11Texture2D,
src: &[u8],
src_pitch: usize,
row_bytes: usize,
rows: usize,
) -> Result<()> {
unsafe {
let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
self.context
.Map(tex, 0, D3D11_MAP_WRITE_DISCARD, 0, Some(&mut mapped))
.context("Map plane texture")?;
let dst = mapped.pData as *mut u8;
let dst_pitch = mapped.RowPitch as usize;
let n = row_bytes.min(src_pitch);
for r in 0..rows {
std::ptr::copy_nonoverlapping(
src.as_ptr().add(r * src_pitch),
dst.add(r * dst_pitch),
n,
);
}
self.context.Unmap(tex, 0);
}
Ok(())
}
fn draw(&mut self) {
let Ok(rtv) = self.rtv() else {
return;
};
let (pw, ph) = (self.panel_w, self.panel_h);
unsafe {
let c = &self.context;
c.ClearRenderTargetView(&rtv, &[0.0, 0.0, 0.0, 1.0]);
if let Some(bound) = &self.bound {
// Contain-fit viewport: scale to the smaller axis, centre, letterbox the rest.
let (ww, wh, vfw, vfh) = (
pw as f32,
ph as f32,
self.src_w.max(1) as f32,
self.src_h.max(1) as f32,
);
let scale = (ww / vfw).min(wh / vfh);
let (dw, dh) = (vfw * scale, vfh * scale);
let (ox, oy) = ((ww - dw) / 2.0, (wh - dh) / 2.0);
c.OMSetRenderTargets(Some(&[Some(rtv.clone())]), None);
let vp = D3D11_VIEWPORT {
TopLeftX: ox,
TopLeftY: oy,
Width: dw,
Height: dh,
MinDepth: 0.0,
MaxDepth: 1.0,
};
c.RSSetViewports(Some(&[vp]));
c.IASetInputLayout(None);
c.IASetPrimitiveTopology(D3D_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
c.VSSetShader(&self.vs, None);
c.PSSetShader(&self.ps_yuv, None);
c.PSSetConstantBuffers(0, Some(&[Some(self.csc_buf.clone())]));
c.PSSetShaderResources(0, Some(&[Some(bound.y.clone()), Some(bound.c.clone())]));
c.PSSetSamplers(0, Some(&[Some(self.sampler.clone())]));
c.Draw(3, 0);
}
let _ = self.swap.Present(1, DXGI_PRESENT(0));
}
}
/// Switch the swapchain between 8-bit SDR (B8G8R8A8, BT.709) and 10-bit HDR10 (R10G10B10A2,
/// ST.2084 PQ BT.2020). `ResizeBuffers` changes the back-buffer format in place, so the panel
/// binding (`set_swap_chain`) stays valid — no rebind. Both frame sources already produce
/// PQ-encoded BT.2020 for HDR, so the colour space is all the compositor needs.
fn set_hdr(&mut self, on: bool) {
self.rtv = None; // release back-buffer refs before ResizeBuffers
let format = if on {
DXGI_FORMAT_R10G10B10A2_UNORM
} else {
DXGI_FORMAT_B8G8R8A8_UNORM
};
unsafe {
if let Err(e) = self.swap.ResizeBuffers(
0,
self.panel_w,
self.panel_h,
format,
DXGI_SWAP_CHAIN_FLAG(self.swap_flags as i32),
) {
tracing::warn!(error = %e, "ResizeBuffers for HDR switch failed");
return;
}
let colorspace = if on {
DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020
} else {
DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709
};
if let Ok(sc3) = self.swap.cast::<IDXGISwapChain3>() {
// Only set a colour space the swapchain accepts for present (on an SDR desktop the
// DWM still tone-maps HDR10 → SDR, so leaving the default there is fine).
if let Ok(support) = sc3.CheckColorSpaceSupport(colorspace) {
if support & DXGI_SWAP_CHAIN_COLOR_SPACE_SUPPORT_FLAG_PRESENT.0 as u32 != 0 {
if let Err(e) = sc3.SetColorSpace1(colorspace) {
// A silent failure here presents PQ content as SDR gamma (crushed/dark) —
// surface it instead of swallowing it.
tracing::warn!(error = %e, ?colorspace, "SetColorSpace1 failed");
}
} else if on {
tracing::warn!("swapchain rejects BT.2020 PQ present colour space (SDR display?) — DWM tone-maps");
}
}
}
self.hdr = on;
if on {
self.apply_hdr_metadata();
}
}
self.apply_dpi_matrix(); // belt-and-braces: keep the DIP mapping across the format switch
tracing::info!(hdr = on, "swapchain colour mode switched");
}
/// Push the current `DXGI_HDR_METADATA_HDR10` to the swapchain. Uses the source's received
/// mastering metadata when known, else a generic HDR10 baseline. Caller ensures HDR mode.
unsafe fn apply_hdr_metadata(&self) {
if let Ok(sc4) = self.swap.cast::<IDXGISwapChain4>() {
let md = self
.hdr_meta
.map(hdr_meta_to_dxgi)
.unwrap_or_else(generic_hdr10_metadata);
let bytes = std::slice::from_raw_parts(
&md as *const DXGI_HDR_METADATA_HDR10 as *const u8,
std::mem::size_of::<DXGI_HDR_METADATA_HDR10>(),
);
if let Err(e) = sc4.SetHDRMetaData(DXGI_HDR_METADATA_TYPE_HDR10, Some(bytes)) {
tracing::warn!(error = %e, "SetHDRMetaData failed");
}
}
}
fn rtv(&mut self) -> Result<ID3D11RenderTargetView> {
if self.rtv.is_none() {
let back: ID3D11Texture2D = unsafe { self.swap.GetBuffer(0).context("GetBuffer")? };
let rtv = unsafe {
let mut v = None;
self.device
.CreateRenderTargetView(&back, None, Some(&mut v))
.context("CreateRenderTargetView")?;
v.unwrap()
};
self.rtv = Some(rtv);
}
Ok(self.rtv.clone().unwrap())
}
}
impl Drop for Presenter {
fn drop(&mut self) {
if let Some(h) = self.waitable.take() {
unsafe {
let _ = CloseHandle(h);
}
}
}
}
/// Luma + chroma plane view formats for NV12 (8-bit) vs P010 (10-in-16-bit).
fn plane_formats(ten_bit: bool) -> (DXGI_FORMAT, DXGI_FORMAT) {
if ten_bit {
(DXGI_FORMAT_R16_UNORM, DXGI_FORMAT_R16G16_UNORM)
} else {
(DXGI_FORMAT_R8_UNORM, DXGI_FORMAT_R8G8_UNORM)
}
}
/// A composition flip-model swapchain (no HWND) for binding to a XAML `SwapChainPanel`, with the
/// frame-latency waitable when the driver allows it. Returns the swapchain + the flags it was
/// created with (every `ResizeBuffers` must re-pass them).
fn create_composition_swapchain(
device: &ID3D11Device,
width: u32,
height: u32,
) -> Result<(IDXGISwapChain1, u32)> {
let dxdev: IDXGIDevice = device.cast().context("IDXGIDevice cast")?;
let factory: IDXGIFactory2 = unsafe {
let adapter = dxdev.GetAdapter().context("GetAdapter")?;
adapter.GetParent().context("GetParent (IDXGIFactory2)")?
};
let mut desc = DXGI_SWAP_CHAIN_DESC1 {
Width: width,
Height: height,
Format: DXGI_FORMAT_B8G8R8A8_UNORM,
Stereo: false.into(),
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
BufferUsage: DXGI_USAGE_RENDER_TARGET_OUTPUT,
BufferCount: 2,
Scaling: DXGI_SCALING_STRETCH,
SwapEffect: DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL,
// IGNORE (opaque), not PREMULTIPLIED: the video fills the panel with opaque RGB either way.
AlphaMode: DXGI_ALPHA_MODE_IGNORE,
Flags: DXGI_SWAP_CHAIN_FLAG_FRAME_LATENCY_WAITABLE_OBJECT.0 as u32,
};
unsafe {
match factory.CreateSwapChainForComposition(device, &desc, None) {
Ok(sc) => Ok((sc, desc.Flags)),
Err(e) => {
// Odd driver/WARP combinations can reject the waitable — fall back to plain
// Present(1) pacing rather than failing the stream page.
tracing::warn!(error = %e, "waitable swapchain rejected — creating without");
desc.Flags = 0;
let sc = factory
.CreateSwapChainForComposition(device, &desc, None)
.context("CreateSwapChainForComposition")?;
Ok((sc, 0))
}
}
}
}
fn build_pipeline(
device: &ID3D11Device,
) -> Result<(ID3D11VertexShader, ID3D11PixelShader, ID3D11SamplerState)> {
let vs_blob = compile(SHADER_HLSL, "vs_main", "vs_5_0")?;
let yuv_blob = compile(SHADER_HLSL, "ps_yuv", "ps_5_0")?;
unsafe {
let mut vs = None;
device
.CreateVertexShader(blob_bytes(&vs_blob), None, Some(&mut vs))
.context("CreateVertexShader")?;
let mut ps_yuv = None;
device
.CreatePixelShader(blob_bytes(&yuv_blob), None, Some(&mut ps_yuv))
.context("CreatePixelShader (yuv)")?;
let sdesc = D3D11_SAMPLER_DESC {
Filter: D3D11_FILTER_MIN_MAG_MIP_LINEAR,
AddressU: D3D11_TEXTURE_ADDRESS_CLAMP,
AddressV: D3D11_TEXTURE_ADDRESS_CLAMP,
AddressW: D3D11_TEXTURE_ADDRESS_CLAMP,
MaxLOD: D3D11_FLOAT32_MAX,
..Default::default()
};
let mut sampler = None;
device
.CreateSamplerState(&sdesc, Some(&mut sampler))
.context("CreateSamplerState")?;
Ok((vs.unwrap(), ps_yuv.unwrap(), sampler.unwrap()))
}
}
fn compile(src: &str, entry: &str, target: &str) -> Result<ID3DBlob> {
let entry_c = std::ffi::CString::new(entry).unwrap();
let target_c = std::ffi::CString::new(target).unwrap();
let mut code = None;
let mut errors = None;
let r = unsafe {
D3DCompile(
src.as_ptr() as *const _,
src.len(),
PCSTR::null(),
None,
None,
PCSTR(entry_c.as_ptr() as *const u8),
PCSTR(target_c.as_ptr() as *const u8),
D3DCOMPILE_OPTIMIZATION_LEVEL3,
0,
&mut code,
Some(&mut errors),
)
};
if r.is_err() {
let msg = errors
.as_ref()
.map(|b| unsafe {
let p = b.GetBufferPointer() as *const u8;
let n = b.GetBufferSize();
String::from_utf8_lossy(std::slice::from_raw_parts(p, n)).to_string()
})
.unwrap_or_default();
return Err(anyhow!("D3DCompile {entry}: {msg}"));
}
code.ok_or_else(|| anyhow!("D3DCompile produced no bytecode"))
}
fn blob_bytes(blob: &ID3DBlob) -> &[u8] {
unsafe {
let p = blob.GetBufferPointer() as *const u8;
let n = blob.GetBufferSize();
std::slice::from_raw_parts(p, n)
}
}
/// True if any attached display is currently in HDR (BT.2020 PQ) mode. The client advertises HDR
/// caps only when this holds, so an SDR display gets a proper 8-bit BT.709 stream instead of PQ it
/// would mis-tone-map (the washed-out/dark failure); an HDR display self-tone-maps from the
/// mastering metadata. Coarse — checks ANY output, not the app's specific monitor; a mid-session
/// monitor move to/from HDR is a follow-up (the `Reconfigure` downgrade).
pub fn display_supports_hdr() -> bool {
unsafe {
let factory: IDXGIFactory1 = match CreateDXGIFactory1() {
Ok(f) => f,
Err(_) => return false,
};
let mut ai = 0u32;
while let Ok(adapter) = factory.EnumAdapters1(ai) {
ai += 1;
let mut oi = 0u32;
while let Ok(output) = adapter.EnumOutputs(oi) {
oi += 1;
if let Ok(o6) = output.cast::<IDXGIOutput6>() {
if let Ok(desc) = o6.GetDesc1() {
if desc.ColorSpace == DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 {
return true;
}
}
}
}
}
}
false
}
/// The HDR display's colour volume from `IDXGIOutput6::GetDesc1` — the first output currently in
/// HDR (BT.2020 PQ) mode, as [`HdrMeta`](punktfunk_core::quic::HdrMeta) for `Hello::display_hdr`.
/// The host writes this volume into its virtual display's EDID, so host apps tone-map to THIS
/// panel and the PQ stream needs no client-side rescue. Chromaticities come as CIE xy floats
/// (×50000 → ST.2086 units, G/B/R order); luminances as nits floats (max ×10000 → 0.0001-cd/m²
/// units); `MaxFullFrameLuminance` → MaxFALL (whole nits); MaxCLL stays 0 (a display has no
/// content light level). Same ANY-output coarseness as [`display_supports_hdr`] — the session
/// gates on that check first, so both look at the same panel in the single-HDR-display case.
pub fn display_hdr_volume() -> Option<punktfunk_core::quic::HdrMeta> {
let to_2086 = |v: f32| (v * 50000.0).round().clamp(0.0, 65535.0) as u16;
unsafe {
let factory: IDXGIFactory1 = CreateDXGIFactory1().ok()?;
let mut ai = 0u32;
while let Ok(adapter) = factory.EnumAdapters1(ai) {
ai += 1;
let mut oi = 0u32;
while let Ok(output) = adapter.EnumOutputs(oi) {
oi += 1;
let Ok(o6) = output.cast::<IDXGIOutput6>() else {
continue;
};
let Ok(desc) = o6.GetDesc1() else { continue };
if desc.ColorSpace != DXGI_COLOR_SPACE_RGB_FULL_G2084_NONE_P2020 {
continue;
}
return Some(punktfunk_core::quic::HdrMeta {
// ST.2086 order is G, B, R.
display_primaries: [
[to_2086(desc.GreenPrimary[0]), to_2086(desc.GreenPrimary[1])],
[to_2086(desc.BluePrimary[0]), to_2086(desc.BluePrimary[1])],
[to_2086(desc.RedPrimary[0]), to_2086(desc.RedPrimary[1])],
],
white_point: [to_2086(desc.WhitePoint[0]), to_2086(desc.WhitePoint[1])],
max_display_mastering_luminance: (desc.MaxLuminance.max(0.0) * 10_000.0).round()
as u32,
min_display_mastering_luminance: (desc.MinLuminance.max(0.0) * 10_000.0).round()
as u32,
max_cll: 0,
max_fall: desc.MaxFullFrameLuminance.max(0.0).round() as u16,
});
}
}
}
None
}
/// Generic HDR10 mastering metadata: BT.2020 primaries + D65 white, a 1000-nit mastering display,
/// MaxCLL 1000 / MaxFALL 400. The fallback used only until the host's real `0xCE` metadata arrives.
fn generic_hdr10_metadata() -> DXGI_HDR_METADATA_HDR10 {
DXGI_HDR_METADATA_HDR10 {
RedPrimary: [35400, 14600],
GreenPrimary: [8500, 39850],
BluePrimary: [6550, 2300],
WhitePoint: [15635, 16450],
MaxMasteringLuminance: 1000,
MinMasteringLuminance: 1, // 0.0001-nit units → 0.0001 nits
MaxContentLightLevel: 1000,
MaxFrameAverageLightLevel: 400,
}
}
/// Map the protocol's [`HdrMeta`](punktfunk_core::quic::HdrMeta) to `DXGI_HDR_METADATA_HDR10`.
/// Two careful conversions: HdrMeta stores primaries in **ST.2086 G,B,R order**, DXGI wants
/// **R,G,B**; and HdrMeta mastering luminance is in **0.0001-cd/m² units** while DXGI's
/// `MaxMasteringLuminance` is in **whole nits** (MinMasteringLuminance stays 0.0001-nit). Chromaticity
/// units (1/50000) and MaxCLL/MaxFALL (nits) match 1:1.
fn hdr_meta_to_dxgi(m: punktfunk_core::quic::HdrMeta) -> DXGI_HDR_METADATA_HDR10 {
let [g, b, r] = m.display_primaries; // ST.2086 order
DXGI_HDR_METADATA_HDR10 {
RedPrimary: r,
GreenPrimary: g,
BluePrimary: b,
WhitePoint: m.white_point,
MaxMasteringLuminance: m.max_display_mastering_luminance / 10_000, // 0.0001-nit → nit
MinMasteringLuminance: m.min_display_mastering_luminance, // already 0.0001-nit
MaxContentLightLevel: m.max_cll,
MaxFrameAverageLightLevel: m.max_fall,
}
}
+79
View File
@@ -0,0 +1,79 @@
//! Network speed-test probe — the GUI's per-host "Test Network Speed…" ([`crate::app`]'s
//! speed page) and the `--headless --speed-test` CLI.
//!
//! Split out of the former in-process session module: the shared spawned-`punktfunk-session`
//! binary owns real streaming now, but the speed test is a shell-side, decode-less measurement
//! over the real data plane, so it stays here. [`decodable_codecs`] rode along for the same
//! reason — the probe connect still advertises which codecs this client can decode.
use ffmpeg_next as ffmpeg;
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::{CompositorPref, GamepadPref, Mode};
use std::time::{Duration, Instant};
/// The `quic` codec bitfield this client can decode — whatever FFmpeg has a decoder for (HEVC/H.264
/// always; AV1 when built in). Advertised to the host so it never emits a codec we can't decode.
pub fn decodable_codecs() -> u8 {
let _ = ffmpeg::init();
let mut bits = 0u8;
for (id, bit) in [
(ffmpeg::codec::Id::HEVC, punktfunk_core::quic::CODEC_HEVC),
(ffmpeg::codec::Id::H264, punktfunk_core::quic::CODEC_H264),
(ffmpeg::codec::Id::AV1, punktfunk_core::quic::CODEC_AV1),
] {
if ffmpeg::decoder::find(id).is_some() {
bits |= bit;
}
}
bits
}
/// Blocking speed-test probe (the GUI's per-host "Test" and the `--headless --speed-test` CLI):
/// a minimal identified connect (720p60 — the host builds a virtual output, but nothing is
/// decoded), then `request_probe` (a 2 s burst up to the host's 3 Gbps ceiling) polled to
/// completion. Run on a worker thread.
pub fn run_speed_probe(
addr: &str,
port: u16,
fp_hex: Option<&str>,
identity: (String, String),
) -> Result<punktfunk_core::client::ProbeOutcome, String> {
// Pin the saved/advertised fingerprint when we have one; a manual host measures over TOFU.
let pin = fp_hex.and_then(crate::trust::parse_hex32);
let c = NativeClient::connect(
addr,
port,
Mode {
width: 1280,
height: 720,
refresh_hz: 60,
},
CompositorPref::Auto,
GamepadPref::Auto,
0, // bitrate_kbps: host default
0, // video_caps: probe connect, nothing is decoded
2, // audio_channels: stereo baseline
decodable_codecs(),
0, // preferred_codec: no preference
None, // display_hdr: probe connect, nothing presents
None, // launch: no game
pin,
Some(identity),
Duration::from_secs(15),
)
.map_err(|e| format!("connect: {e:?}"))?;
c.request_probe(3_000_000, 2_000)
.map_err(|e| format!("probe: {e:?}"))?;
let deadline = Instant::now() + Duration::from_secs(10);
loop {
std::thread::sleep(Duration::from_millis(250));
if c.probe_result().done {
// Let the last UDP shards land before tearing down.
std::thread::sleep(Duration::from_millis(400));
return Ok(c.probe_result());
}
if Instant::now() > deadline {
return Err("probe timed out".to_string());
}
}
}
-285
View File
@@ -1,285 +0,0 @@
//! The dedicated video render thread: decoded frames flow session pump → bounded channel → here →
//! `Presenter::present`. Presenting off the XAML thread means UI jank (layout, input, dialogs)
//! never stalls video, and a filled present queue never blocks the UI thread — the two failure
//! modes of the old present-from-`on_rendering` design.
//!
//! Pacing: block on the channel (the host paces the stream), then on the swapchain's
//! frame-latency waitable (≤1 queued present — see `present.rs`), then drain to the NEWEST frame
//! so a stream faster than the display drops backlog before any GPU work. The UI thread only
//! writes panel size/DPI into [`RenderShared`] atomics; the loop applies them before the next
//! draw (and redraws the held frame after a resize — fresh back buffers are blank).
use crate::present::Presenter;
use crate::session::{FrameRx, FrameTimes};
use crossbeam_channel::RecvTimeoutError;
use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU32, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// The last 1-second render window, published for the HUD (one render thread at a time):
/// presents/s, frames skipped by the newest-wins drain, the end-to-end (capture→on-glass)
/// p50/p95 and the `display` stage (decoded→displayed) p50, all stamped post-`Present()`, in µs.
/// Zeroed when a render thread starts so a new session never shows the previous one's numbers.
static PRESENT_FPS: AtomicU32 = AtomicU32::new(0);
static PRESENT_SKIPPED: AtomicU32 = AtomicU32::new(0);
static E2E_P50_US: AtomicU64 = AtomicU64::new(0);
static E2E_P95_US: AtomicU64 = AtomicU64::new(0);
static DISPLAY_P50_US: AtomicU64 = AtomicU64::new(0);
/// The last render window's glass-side numbers (see the statics above) — the HUD's headline
/// (end-to-end) and trailing stage (display) come from here.
#[derive(Clone, Copy, Default, PartialEq)]
pub struct PresentStats {
/// Presents per second (includes resize redraws of a held frame).
pub fps: u32,
/// Frames dropped by the newest-wins drain this window (client-side pacing skips).
pub skipped: u32,
/// End-to-end capture→displayed p50, ms (host-clock corrected, measured directly).
pub e2e_p50_ms: f32,
/// End-to-end capture→displayed p95, ms.
pub e2e_p95_ms: f32,
/// `display` stage p50, ms: decoded → displayed, single-clock client-local.
pub display_p50_ms: f32,
}
pub fn present_stats() -> PresentStats {
PresentStats {
fps: PRESENT_FPS.load(Ordering::Relaxed),
skipped: PRESENT_SKIPPED.load(Ordering::Relaxed),
e2e_p50_ms: E2E_P50_US.load(Ordering::Relaxed) as f32 / 1000.0,
e2e_p95_ms: E2E_P95_US.load(Ordering::Relaxed) as f32 / 1000.0,
display_p50_ms: DISPLAY_P50_US.load(Ordering::Relaxed) as f32 / 1000.0,
}
}
/// UI-thread → render-thread state. Size is packed into ONE atomic (w<<32|h) so a resize never
/// tears into a (new-width, old-height) pair.
pub struct RenderShared {
size_px: AtomicU64,
dpi: AtomicU32,
stop: AtomicBool,
}
impl RenderShared {
pub fn new(width: u32, height: u32, dpi: u32) -> Arc<RenderShared> {
Arc::new(RenderShared {
size_px: AtomicU64::new(pack(width, height)),
dpi: AtomicU32::new(dpi),
stop: AtomicBool::new(false),
})
}
pub fn set_size(&self, width: u32, height: u32) {
self.size_px.store(pack(width, height), Ordering::Relaxed);
}
pub fn set_dpi(&self, dpi: u32) {
self.dpi.store(dpi, Ordering::Relaxed);
}
fn snapshot(&self) -> (u32, u32, u32) {
let s = self.size_px.load(Ordering::Relaxed);
((s >> 32) as u32, s as u32, self.dpi.load(Ordering::Relaxed))
}
}
fn pack(w: u32, h: u32) -> u64 {
((w as u64) << 32) | h as u64
}
/// Handle owned by the stream page; stops + joins the thread on unmount (and on drop, so a
/// navigation away can't leak a presenting thread).
pub struct RenderThread {
shared: Arc<RenderShared>,
join: Option<std::thread::JoinHandle<()>>,
}
impl RenderThread {
pub fn shared(&self) -> &Arc<RenderShared> {
&self.shared
}
pub fn stop_and_join(&mut self) {
self.shared.stop.store(true, Ordering::SeqCst);
if let Some(j) = self.join.take() {
let _ = j.join();
}
}
}
impl Drop for RenderThread {
fn drop(&mut self) {
self.stop_and_join();
}
}
/// Moves the presenter (COM interfaces, `!Send` by default) onto the render thread. Sound here:
/// the shared device + immediate context are multithread-protected (see `crate::gpu`), D3D/DXGI
/// objects are apartment-agile, and after this one handoff the swapchain/RTV/context calls happen
/// on exactly the render thread — the same single-owner discipline as `SharedDevice`.
struct SendPresenter(Presenter);
unsafe impl Send for SendPresenter {}
/// Spawn the render thread. `frames` carries `(frame, FrameTimes)`; `clock_offset_ns` maps our
/// wall clock onto the host's so the end-to-end (capture→on-glass) number is cross-machine valid
/// (same math as the pump's host+network stage). A live handle (loaded per present) so
/// mid-stream clock re-syncs keep the number honest after an NTP step / drift.
pub fn spawn(
presenter: Presenter,
frames: FrameRx,
shared: Arc<RenderShared>,
clock_offset_ns: Arc<AtomicI64>,
) -> RenderThread {
let boxed = SendPresenter(presenter);
let shared_w = shared.clone();
let join = std::thread::Builder::new()
.name("pf-render".into())
.spawn(move || run(boxed, frames, shared_w, clock_offset_ns))
.expect("spawn render thread");
RenderThread {
shared,
join: Some(join),
}
}
fn now_ns() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0)
}
/// The window DPI, polled ~1 Hz as belt-and-braces for a monitor move that changes DPI without a
/// `SizeChanged` (same DIP size on both screens). `None` when the window isn't up (headless).
fn poll_window_dpi() -> Option<u32> {
use windows::Win32::UI::HiDpi::GetDpiForWindow;
use windows::Win32::UI::WindowsAndMessaging::FindWindowW;
unsafe {
let hwnd = FindWindowW(None, windows::core::w!("Punktfunk")).ok()?;
match GetDpiForWindow(hwnd) {
0 => None,
d => Some(d),
}
}
}
fn run(
presenter: SendPresenter,
frames: FrameRx,
shared: Arc<RenderShared>,
clock_offset_ns: Arc<AtomicI64>,
) {
let mut p = presenter.0;
let mut applied = (0u32, 0u32, 0u32); // last (w, h, dpi) handed to the presenter
let mut presented = 0u32;
let mut dropped = 0u32;
// 1 s tumbling windows: end-to-end (capture→displayed) and the display stage
// (decoded→displayed), sampled post-Present. Percentiles only (spec: stats-unification.md).
let mut e2e_us: Vec<u64> = Vec::with_capacity(256);
let mut display_us: Vec<u64> = Vec::with_capacity(256);
let mut window_start = Instant::now();
let mut last_dpi_poll = Instant::now();
PRESENT_FPS.store(0, Ordering::Relaxed);
PRESENT_SKIPPED.store(0, Ordering::Relaxed);
E2E_P50_US.store(0, Ordering::Relaxed);
E2E_P95_US.store(0, Ordering::Relaxed);
DISPLAY_P50_US.store(0, Ordering::Relaxed);
loop {
if shared.stop.load(Ordering::SeqCst) {
break;
}
let first = match frames.recv_timeout(Duration::from_millis(50)) {
Ok(f) => Some(f),
Err(RecvTimeoutError::Timeout) => None,
Err(RecvTimeoutError::Disconnected) => break,
};
if last_dpi_poll.elapsed() >= Duration::from_secs(1) {
last_dpi_poll = Instant::now();
if let Some(dpi) = poll_window_dpi() {
shared.set_dpi(dpi);
}
}
let snap = shared.snapshot();
let resized = snap != applied && snap.0 > 0 && snap.1 > 0;
if resized {
p.resize(snap.0, snap.1, snap.2);
applied = snap;
}
if first.is_none() && !resized {
continue; // nothing new to show — don't burn GPU re-presenting a static frame
}
// Throttle to the compositor: with ≤1 present outstanding this returns as DWM frees a
// slot, and frames decoded meanwhile are drained below so the newest is what's drawn.
if !p.wait_present_slot(1000) {
tracing::debug!("frame-latency waitable timed out — presenting anyway");
}
let mut newest = first;
while let Ok(f) = frames.try_recv() {
if newest.is_some() {
dropped += 1;
}
newest = Some(f);
}
// The session pump is the sole 0xCE consumer and stashes the latest here (rare updates).
if let Some(meta) = *crate::present::LATEST_HDR_META.lock().unwrap() {
p.set_hdr_metadata(meta);
}
let times: Option<FrameTimes> = newest.as_ref().map(|(_, t)| *t);
p.present(newest.map(|(f, _)| f));
presented += 1;
if let Some(t) = times {
// The `displayed` point: post-Present() on this thread (the honest best-effort
// presentation instant on Windows — endpoint label `capture→on-glass`).
let displayed_ns = now_ns();
// End-to-end = capture → displayed, host-clock corrected, measured directly
// (never the sum of stage percentiles). Clamped (0, 10 s).
let e2e = (displayed_ns as i128 + clock_offset_ns.load(Ordering::Relaxed) as i128
- t.pts_ns as i128)
.max(0) as u64;
if e2e > 0 && e2e < 10_000_000_000 {
e2e_us.push(e2e / 1000);
}
// `display` stage = decoded → displayed, single-clock client-local.
let disp = displayed_ns.saturating_sub(t.decoded_ns);
if disp < 10_000_000_000 {
display_us.push(disp / 1000);
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
e2e_us.sort_unstable();
display_us.sort_unstable();
let p50 = |v: &[u64]| v.get(v.len() / 2).copied().unwrap_or(0);
// p95 = sorted[min(len*95/100, len-1)] — the empty-window case falls to 0 via `get`.
let p95 = |v: &[u64]| {
v.get((v.len() * 95 / 100).min(v.len().saturating_sub(1)))
.copied()
.unwrap_or(0)
};
tracing::debug!(
presented,
dropped,
e2e_p50_us = p50(&e2e_us),
e2e_p95_us = p95(&e2e_us),
display_p50_us = p50(&display_us),
"render window"
);
PRESENT_FPS.store(presented, Ordering::Relaxed);
PRESENT_SKIPPED.store(dropped, Ordering::Relaxed);
E2E_P50_US.store(p50(&e2e_us), Ordering::Relaxed);
E2E_P95_US.store(p95(&e2e_us), Ordering::Relaxed);
DISPLAY_P50_US.store(p50(&display_us), Ordering::Relaxed);
window_start = Instant::now();
presented = 0;
dropped = 0;
e2e_us.clear();
display_us.clear();
}
}
tracing::info!("render thread exiting");
}
-555
View File
@@ -1,555 +0,0 @@
//! Session controller: one worker thread runs connect → pump (video pull + decode, audio
//! pull + Opus decode, stats), feeding the UI over channels. The UI keeps the
//! `Arc<NativeClient>` from the `Connected` event for direct input sends (no extra hop on
//! the input path) — `NativeClient` is `Sync`, planes stay one-consumer-per-thread:
//! video+audio here, rumble+hidout on the gamepad thread.
//!
//! Ported from the GTK Linux client; the platform-specific pieces are the video decoder
//! (software-only here) and the audio backend (WASAPI). The pump body is identical.
use crate::audio;
use crate::video::{DecodedFrame, Decoder, DecoderPref};
use punktfunk_core::client::NativeClient;
use punktfunk_core::config::{CompositorPref, GamepadPref, Mode};
use punktfunk_core::PunktfunkError;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
pub struct SessionParams {
pub host: String,
pub port: u16,
pub mode: Mode,
pub compositor: CompositorPref,
pub gamepad: GamepadPref,
pub bitrate_kbps: u32,
/// Requested audio channel count (2/6/8); the host echoes the resolved value.
pub audio_channels: u8,
/// Stream the default microphone to the host's virtual mic source.
pub mic_enabled: bool,
/// Advertise 10-bit + HDR10 so the host may upgrade HDR content to a Main10/PQ stream.
pub hdr_enabled: bool,
/// Which video decode backend to use (auto/hardware/software).
pub decoder: DecoderPref,
/// The user's preferred video codec (a `quic::CODEC_*` bit, `0` = auto). Soft — the host honors
/// it when it can emit it, else falls back; the resolved codec drives the decoder.
pub preferred_codec: u8,
/// Pinned host fingerprint; `None` = trust on first use (caller persists the observed one).
pub pin: Option<[u8; 32]>,
pub identity: (String, String),
/// How long to wait for the handshake. The normal path uses a short budget; the
/// "request access" (delegated-approval) path uses a long one, because the host PARKS the
/// connection until the operator clicks Approve in its console (so this must exceed the
/// host's approval window — see `PENDING_APPROVAL_WAIT`).
pub connect_timeout: Duration,
}
#[derive(Clone, Copy, Default, PartialEq)]
pub struct Stats {
/// AUs received (reassembled) per second — actual-elapsed-time denominator.
pub fps: f32,
/// Received payload goodput (excludes FEC overhead).
pub mbps: f32,
/// `decode` stage p50 over the last 1 s window: received → decoded, client-local clock.
pub decode_ms: f32,
/// `host+network` stage p50 over the last 1 s window: capture (`pts_ns`) → received,
/// host-clock corrected via `clock_offset_ns`.
pub hostnet_ms: f32,
/// `host` stage p50 (host capture→sent, from the per-AU 0xCF host-timing plane). Valid only
/// when `split` — an old host emits no 0xCF and the HUD keeps the combined stage.
pub host_ms: f32,
/// `network` stage p50 (`hostnet host`, tiled per frame before taking the percentile).
/// Valid only when `split`.
pub net_ms: f32,
/// True when any 0xCF host timings matched received AUs this window — the HUD then renders
/// `host + network` instead of the combined `host+network` term.
pub split: bool,
/// True when `clock_offset_ns == 0` (host didn't answer the skew handshake / same host) —
/// the HUD appends `(same-host clock)` to the end-to-end line.
pub same_host: bool,
/// True when decoding on the GPU (D3D11VA) vs. CPU (software).
pub hardware: bool,
/// True when the stream is BT.2020 PQ HDR10 (last decoded frame).
pub hdr: bool,
/// The negotiated wire codec (a `quic::CODEC_*` bit) — the HUD's codec chip.
pub codec: u8,
/// Frames lost to unrecoverable network drops since session start (reassembler count; each
/// triggers a keyframe re-request).
pub dropped: u64,
/// Seconds since the stream started.
pub uptime_secs: u32,
}
pub enum SessionEvent {
Connected {
connector: Arc<NativeClient>,
mode: Mode,
fingerprint: [u8; 32],
},
/// `trust_rejected` is set when the connect failed the TLS trust check (a `Crypto`
/// error): for a pinned connect this is the fingerprint-changed signal, so the UI can
/// offer a re-pair (PIN) path rather than a dead-end error.
Failed {
msg: String,
trust_rejected: bool,
},
Ended(Option<String>),
Stats(Stats),
}
/// Per-frame measurement points carried with a decoded frame to the render thread: the host
/// capture clock (`pts_ns`) and our local `decoded` stamp (wall-clock ns). Post-`Present()` the
/// render thread derives the `display` stage (displayed decoded, single-clock) and the
/// end-to-end headline (displayed + clock_offset pts) from them.
#[derive(Clone, Copy)]
pub struct FrameTimes {
pub pts_ns: u64,
pub decoded_ns: u64,
}
/// Decoded frames + their measurement points, session pump → render thread (crossbeam so that
/// thread can block with a timeout — async-channel has no `recv_timeout`).
pub type FrameRx = crossbeam_channel::Receiver<(DecodedFrame, FrameTimes)>;
pub struct SessionHandle {
pub events: async_channel::Receiver<SessionEvent>,
pub frames: FrameRx,
pub stop: Arc<AtomicBool>,
}
/// Blocking speed-test probe (the GUI's per-host "Test" and the `--headless --speed-test` CLI):
/// a minimal identified connect (720p60 — the host builds a virtual output, but nothing is
/// decoded), then `request_probe` (a 2 s burst up to the host's 3 Gbps ceiling) polled to
/// completion. Run on a worker thread.
pub fn run_speed_probe(
addr: &str,
port: u16,
fp_hex: Option<&str>,
identity: (String, String),
) -> Result<punktfunk_core::client::ProbeOutcome, String> {
// Pin the saved/advertised fingerprint when we have one; a manual host measures over TOFU.
let pin = fp_hex.and_then(crate::trust::parse_hex32);
let c = NativeClient::connect(
addr,
port,
Mode {
width: 1280,
height: 720,
refresh_hz: 60,
},
CompositorPref::Auto,
GamepadPref::Auto,
0, // bitrate_kbps: host default
0, // video_caps: probe connect, nothing is decoded
2, // audio_channels: stereo baseline
crate::video::decodable_codecs(),
0, // preferred_codec: no preference
None, // display_hdr: probe connect, nothing presents
None, // launch: no game
pin,
Some(identity),
Duration::from_secs(15),
)
.map_err(|e| format!("connect: {e:?}"))?;
c.request_probe(3_000_000, 2_000)
.map_err(|e| format!("probe: {e:?}"))?;
let deadline = Instant::now() + Duration::from_secs(10);
loop {
std::thread::sleep(Duration::from_millis(250));
if c.probe_result().done {
// Let the last UDP shards land before tearing down.
std::thread::sleep(Duration::from_millis(400));
return Ok(c.probe_result());
}
if Instant::now() > deadline {
return Err("probe timed out".to_string());
}
}
}
pub fn start(params: SessionParams) -> SessionHandle {
let (ev_tx, ev_rx) = async_channel::unbounded();
// Tiny frame queue, newest wins: the pump displaces the oldest when the renderer lags (it
// keeps a Receiver clone for exactly that).
let (frame_tx, frame_rx) = crossbeam_channel::bounded(2);
let stop = Arc::new(AtomicBool::new(false));
let stop_w = stop.clone();
let frame_rx_pump = frame_rx.clone();
std::thread::Builder::new()
.name("punktfunk-session".into())
.spawn(move || pump(params, ev_tx, frame_tx, frame_rx_pump, stop_w))
.expect("spawn session thread");
SessionHandle {
events: ev_rx,
frames: frame_rx,
stop,
}
}
fn now_ns() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0)
}
/// Opus decoder for the audio plane: a plain stereo decoder (the validated path) or a multistream
/// decoder for 5.1/7.1, both behind one `decode_float`. Built from the host-RESOLVED channel count
/// via the shared layout table.
enum AudioDec {
Stereo(opus::Decoder),
Surround(opus::MSDecoder),
}
impl AudioDec {
fn new(channels: u8) -> Result<AudioDec, opus::Error> {
if channels == 2 {
Ok(AudioDec::Stereo(opus::Decoder::new(
48_000,
opus::Channels::Stereo,
)?))
} else {
let l = punktfunk_core::audio::layout_for(channels, false);
Ok(AudioDec::Surround(opus::MSDecoder::new(
48_000, l.streams, l.coupled, l.mapping,
)?))
}
}
fn decode_float(
&mut self,
input: &[u8],
out: &mut [f32],
fec: bool,
) -> Result<usize, opus::Error> {
match self {
AudioDec::Stereo(d) => d.decode_float(input, out, fec),
AudioDec::Surround(d) => d.decode_float(input, out, fec),
}
}
}
fn pump(
params: SessionParams,
ev_tx: async_channel::Sender<SessionEvent>,
frame_tx: crossbeam_channel::Sender<(DecodedFrame, FrameTimes)>,
frame_rx: FrameRx,
stop: Arc<AtomicBool>,
) {
// Advertise 10-bit + HDR10 only when the user enabled HDR AND a display is actually in HDR
// mode: the host then upgrades HDR content to a Main10/PQ stream (its own 10-bit gate still
// applies). On an SDR display we advertise `0` so the host sends a proper 8-bit BT.709 stream
// rather than PQ the panel would mis-tone-map (washed-out/dark). The presenter handles BT.2020
// PQ frames (P010 / X2BGR10).
let hdr_active = params.hdr_enabled && crate::present::display_supports_hdr();
if params.hdr_enabled && !hdr_active {
tracing::info!("HDR enabled in settings but no HDR display detected — requesting SDR");
}
// With HDR active, also report the panel's real colour volume (GetDesc1): the host writes it
// into its virtual display's EDID, so host apps tone-map to THIS panel and the PQ stream
// arrives already inside its volume — the client presents it untouched.
// PUNKTFUNK_CLIENT_PEAK_NITS pins a synthetic volume for A/B runs.
let display_hdr = if hdr_active {
let vol = punktfunk_core::client::display_hdr_env_override()
.or_else(crate::present::display_hdr_volume);
if let Some(m) = vol {
tracing::info!(
max_nits = m.max_display_mastering_luminance / 10_000,
min_millinits = m.min_display_mastering_luminance / 10,
max_fall = m.max_fall,
"advertising this display's HDR volume to the host"
);
}
vol
} else {
None
};
let connector = match NativeClient::connect(
&params.host,
params.port,
params.mode,
params.compositor,
params.gamepad,
params.bitrate_kbps,
if hdr_active {
punktfunk_core::quic::VIDEO_CAP_10BIT | punktfunk_core::quic::VIDEO_CAP_HDR
} else {
0
},
params.audio_channels,
crate::video::decodable_codecs(), // codecs FFmpeg can decode (HEVC/H.264/AV1)
params.preferred_codec, // the user's soft codec preference (0 = auto)
display_hdr,
None, // launch: the Windows client has no library picker yet
params.pin,
Some(params.identity),
params.connect_timeout,
) {
Ok(c) => Arc::new(c),
Err(e) => {
let trust_rejected = matches!(e, PunktfunkError::Crypto);
let msg = match e {
PunktfunkError::Crypto => {
"Host identity rejected — wrong fingerprint, or the host requires pairing"
.to_string()
}
PunktfunkError::Timeout => "Connection timed out".to_string(),
other => format!("Connect failed: {other:?}"),
};
let _ = ev_tx.send_blocking(SessionEvent::Failed {
msg,
trust_rejected,
});
return;
}
};
let _ = ev_tx.send_blocking(SessionEvent::Connected {
connector: connector.clone(),
mode: connector.mode(),
fingerprint: connector.host_fingerprint,
});
// Build the decoder for the codec the host resolved (never assume HEVC).
let codec_id = crate::video::ffmpeg_codec_id(connector.codec);
tracing::info!(
?codec_id,
welcome_codec = connector.codec,
"negotiated video codec"
);
let mut decoder = match Decoder::new(params.decoder, codec_id) {
Ok(d) => d,
Err(e) => {
let _ = ev_tx.send_blocking(SessionEvent::Ended(Some(format!("video decoder: {e}"))));
return;
}
};
let mut hardware = decoder.is_hardware();
let mut hdr = false;
// Audio is best-effort: a session without it still streams. Gamepads are the
// app-lifetime service's job (the UI attaches it on Connected). Build the decoder + playback
// from the host-RESOLVED channel count (never the request), so an older/clamping host that
// resolves stereo is decoded as stereo.
let channels = connector.audio_channels;
let player = audio::AudioPlayer::spawn(channels)
.map_err(|e| tracing::warn!(error = %e, "audio disabled"))
.ok();
let mut opus_dec = AudioDec::new(channels)
.map_err(|e| tracing::warn!(error = %e, "opus decoder failed — audio disabled"))
.ok();
let _mic = params
.mic_enabled
.then(|| {
audio::MicStreamer::spawn(connector.clone())
.map_err(|e| tracing::warn!(error = %e, "mic uplink disabled"))
.ok()
})
.flatten();
// Force an immediate IDR (with in-band parameter sets) rather than waiting for the host's own
// first keyframe — under infinite GOP a late/missed IDR means the decoder sits on
// "PPS id out of range" (a black screen) until one arrives.
let _ = connector.request_keyframe();
// Live host↔client clock offset: loaded per use (Relaxed) so mid-stream re-syncs (an NTP
// step, drift) keep the capture-clock latency stats honest — never cached at session start.
let clock_offset_live = connector.clock_offset_shared();
let mut total_frames = 0u64;
let session_start = Instant::now();
let mut window_start = Instant::now();
let mut frames_n = 0u32;
let mut bytes_n = 0u64;
// 1 s tumbling stage windows (spec: design/stats-unification.md — percentiles, never means).
let mut hostnet_us: Vec<u64> = Vec::with_capacity(256);
let mut decode_us: Vec<u64> = Vec::with_capacity(256);
// Host/network split (Phase 2): received AUs awaiting their 0xCF host timing, `(pts_ns,
// hostnet_us)`, matched as the datagrams arrive. Bounded — an old host never sends any.
let mut pending_split: std::collections::VecDeque<(u64, u64)> =
std::collections::VecDeque::with_capacity(256);
let mut host_us_w: Vec<u64> = Vec::with_capacity(256);
let mut net_us_w: Vec<u64> = Vec::with_capacity(256);
let mut pcm = vec![0f32; 5760 * channels as usize]; // scratch: max Opus frame (120 ms) × channels
// Loss recovery: watch the host→client unrecoverable-drop count and ask for an IDR when it climbs.
let mut last_dropped = connector.frames_dropped();
let mut last_kf_req: Option<Instant> = None;
let end: Option<String> = loop {
if stop.load(Ordering::SeqCst) {
break None;
}
match connector.next_frame(Duration::from_millis(4)) {
Ok(frame) => {
// The `received` point: AU fully reassembled, handed to us, before decode.
let received_ns = now_ns();
// Loss recovery (RFI): a forward frame-index gap fires a throttled reference-frame-
// invalidation request so an RFI-capable host (AMD LTR / NVENC) recovers with a cheap
// clean P-frame instead of a full IDR. The frames_dropped keyframe path below is the
// backstop for when the recovery frame itself is lost.
let _ = connector.note_frame_index(frame.frame_index);
// fps = AUs received per second, Mb/s = received goodput (spec: counted at the
// received point, not the decoded one).
frames_n += 1;
bytes_n += frame.data.len() as u64;
// `host+network` stage: capture → received, host-clock corrected. Clamped (0, 10 s).
let clock_offset = clock_offset_live.load(Ordering::Relaxed);
let hostnet = (received_ns as i128 + clock_offset as i128 - frame.pts_ns as i128)
.max(0) as u64;
if hostnet > 0 && hostnet < 10_000_000_000 {
hostnet_us.push(hostnet / 1000);
// Remember this AU for the 0xCF match below (host/network split).
pending_split.push_back((frame.pts_ns, hostnet / 1000));
if pending_split.len() > 256 {
pending_split.pop_front();
}
}
// A D3D11VA→software demotion (see `Decoder::decode`) starts a FRESH decoder that
// has none of the stream's parameter sets; under infinite GOP it would sit on
// "PPS id out of range" forever. Detect the transition and force a new IDR so the
// rebuilt decoder resynchronizes immediately.
let was_hw = decoder.is_hardware();
let decoded = decoder.decode(&frame.data);
if was_hw && !decoder.is_hardware() {
tracing::info!("decoder demoted to software — requesting keyframe to resync");
let _ = connector.request_keyframe();
}
match decoded {
Ok(Some(decoded)) => {
// The `decoded` point: decoder output frame available.
let decoded_ns = now_ns();
total_frames += 1;
hdr = decoded.hdr();
// The backend can demote D3D11VA → software mid-session on a hardware error.
hardware = decoder.is_hardware();
if total_frames == 1 {
let (w, h) = decoded.dims();
tracing::info!(
width = w,
height = h,
path = if hardware { "d3d11va" } else { "software" },
hdr,
"first frame decoded"
);
}
// `decode` stage: received → decoded, single-clock client-local.
decode_us.push(decoded_ns.saturating_sub(received_ns) / 1000);
// Newest wins: displace the oldest queued frame when the renderer lags.
if let Err(crossbeam_channel::TrySendError::Full(item)) =
frame_tx.try_send((
decoded,
FrameTimes {
pts_ns: frame.pts_ns,
decoded_ns,
},
))
{
let _ = frame_rx.try_recv();
let _ = frame_tx.try_send(item);
}
}
Ok(None) => {}
// Survivable (loss until the next IDR/RFI recovery) — keep feeding.
Err(e) => tracing::debug!(error = %e, "decode error (recovering)"),
}
}
Err(PunktfunkError::NoFrame) => {}
Err(PunktfunkError::Closed) => break Some("Host ended the session".to_string()),
Err(e) => break Some(format!("session: {e:?}")),
}
// Loss recovery: under infinite GOP the only recovery keyframe is one we request. The
// reassembler drops unrecoverable AUs (frames_dropped); the decoder conceals the
// reference-missing delta frames that follow and returns Ok, so keying off a decode error
// rarely fires. Request an IDR when the drop count climbs, throttled.
let dropped = connector.frames_dropped();
if dropped > last_dropped {
last_dropped = dropped;
let now = Instant::now();
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!(dropped, "requested keyframe (loss recovery)");
}
}
// Drain audio between frames (packets land every 5 ms; the queue holds 320 ms).
while let Ok(pkt) = connector.next_audio(Duration::ZERO) {
if let (Some(player), Some(dec)) = (&player, opus_dec.as_mut()) {
match dec.decode_float(&pkt.data, &mut pcm, false) {
// `samples` is per-channel; the interleaved frame is `samples * channels`.
Ok(samples) => player.push(pcm[..samples * channels as usize].to_vec()),
Err(e) => tracing::debug!(error = %e, "opus decode"),
}
}
}
// Drain the HDR static-metadata plane (0xCE): the source's real mastering display + content
// light level. Stash the latest for the UI-thread presenter to apply via SetHDRMetaData —
// this pump is the sole consumer of the plane. Rare (start + on change/keyframe).
while let Ok(meta) = connector.next_hdr_meta(Duration::ZERO) {
*crate::present::LATEST_HDR_META.lock().unwrap() = Some(meta);
}
// Drain the per-AU host-timing plane (0xCF) and match by pts: `host` = the host's own
// capture→sent, `network` = our capture→received minus it — the two tile per frame
// (design/stats-unification.md Phase 2). An old host never emits any; `split` stays false
// and the HUD keeps the combined `host+network` stage.
while let Ok(t) = connector.next_host_timing(Duration::ZERO) {
if let Some(i) = pending_split.iter().position(|(p, _)| *p == t.pts_ns) {
let (_, hn_us) = pending_split.remove(i).unwrap();
host_us_w.push(t.host_us as u64);
net_us_w.push(hn_us.saturating_sub(t.host_us as u64));
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32();
hostnet_us.sort_unstable();
decode_us.sort_unstable();
host_us_w.sort_unstable();
net_us_w.sort_unstable();
let p50 = |v: &[u64]| v.get(v.len() / 2).copied().unwrap_or(0);
let (hostnet_p50, decode_p50) = (p50(&hostnet_us), p50(&decode_us));
let (host_p50, net_p50) = (p50(&host_us_w), p50(&net_us_w));
let split = !host_us_w.is_empty();
tracing::debug!(
fps = frames_n,
hostnet_p50_us = hostnet_p50,
host_p50_us = host_p50,
net_p50_us = net_p50,
split,
decode_p50_us = decode_p50,
total_frames,
"stream window"
);
let _ = ev_tx.try_send(SessionEvent::Stats(Stats {
fps: frames_n as f32 / secs,
mbps: bytes_n as f32 * 8.0 / 1e6 / secs,
decode_ms: decode_p50 as f32 / 1000.0,
hostnet_ms: hostnet_p50 as f32 / 1000.0,
host_ms: host_p50 as f32 / 1000.0,
net_ms: net_p50 as f32 / 1000.0,
split,
same_host: clock_offset_live.load(Ordering::Relaxed) == 0,
hardware,
hdr,
codec: connector.codec,
dropped: last_dropped,
uptime_secs: session_start.elapsed().as_secs() as u32,
}));
window_start = Instant::now();
frames_n = 0;
bytes_n = 0;
hostnet_us.clear();
decode_us.clear();
host_us_w.clear();
net_us_w.clear();
}
};
tracing::info!(
total_frames,
reason = end.as_deref().unwrap_or("user"),
"session ended"
);
stop.store(true, Ordering::SeqCst);
let _ = ev_tx.send_blocking(SessionEvent::Ended(end));
}
-4
View File
@@ -6,10 +6,6 @@
//! [`SpawnEvent`]s a reader thread hands to the app's navigation closure: spinner until //! [`SpawnEvent`]s a reader thread hands to the app's navigation closure: spinner until
//! `{"ready":true}`, banner from the `{"error"|"ended": …}` line, `trust_rejected` //! `{"ready":true}`, banner from the `{"error"|"ended": …}` line, `trust_rejected`
//! routed to the re-pair PIN ceremony, `stats:` lines to the session status page. //! routed to the re-pair PIN ceremony, `stats:` lines to the session status page.
//!
//! The legacy in-process D3D11VA presenter remains reachable via the
//! `PUNKTFUNK_BUILTIN_STREAM=1` env override (`app::use_builtin_stream`) — the
//! developer A/B baseline until its deletion.
use std::io::BufRead as _; use std::io::BufRead as _;
use std::process::{Child, Command, Stdio}; use std::process::{Child, Command, Stdio};
+2 -4
View File
@@ -5,10 +5,8 @@
//! //!
//! The shell is the settings file's only writer; the session only reads it. The shell's //! The shell is the settings file's only writer; the session only reads it. The shell's
//! former private `Settings` copy (≤ 0.8.4: `show_hud`, `engine`) is gone — old files //! former private `Settings` copy (≤ 0.8.4: `show_hud`, `engine`) is gone — old files
//! still load via a serde alias in core, and the legacy in-process presenter is now //! still load via a serde alias in core.
//! reachable only through `PUNKTFUNK_BUILTIN_STREAM=1` (see `app::use_builtin_stream`).
pub use pf_client_core::trust::{ pub use pf_client_core::trust::{
hex, learn_mac, load_or_create_identity, parse_hex32, touch_last_used, KnownHost, KnownHosts, hex, learn_mac, load_or_create_identity, parse_hex32, KnownHost, KnownHosts, Settings,
Settings,
}; };
-653
View File
@@ -1,653 +0,0 @@
//! Video decode: reassembled HEVC access units → frames for the D3D11 presenter.
//!
//! Two backends, picked at session start (override via [`DecoderPref`] / the Settings UI):
//!
//! * **D3D11VA** (any GPU — the vendor-agnostic DXVA path on NVIDIA/AMD/Intel): libavcodec decodes
//! on the GPU into an `ID3D11Texture2D` decode array (decoder-only bind — NVIDIA rejects a
//! decoder array that is also a shader resource). The presenter copies each decoded slice into
//! its own sampleable NV12/P010 texture and converts YUV→RGB in a shader — one cheap GPU-to-GPU
//! copy per frame (no swscale, no CPU readback). The decode array is created by the process-wide
//! shared device ([`crate::gpu`]) the presenter also draws with, so the copy stays on-GPU. This
//! is the big latency/throughput win over software.
//! * **Software**: libavcodec on the CPU + swscale to the same planar layout the hardware path
//! produces (NV12, or P010 for 10-bit) — the presenter uploads the two planes and runs the SAME
//! YUV→RGB shaders, so hw/sw color math is identical. The fallback on a GPU-less box (WARP),
//! when D3D11VA init fails, or when a mid-session hardware error demotes us — the host's
//! IDR/RFI recovery resynchronizes on the next keyframe either way.
//!
//! D3D11VA viability is settled **before the session's first frame** by two probes: the adapter
//! must expose the negotiated codec's DXVA decode profile ([`decode_profile_supported`] — hwaccel
//! init otherwise only fails at the first AU, burning the IDR), and it must be able to create the
//! decode surface pool ([`d3d11va_decode_supported`]). Either failing commits to software decode
//! from frame one (a clean, gap-free stream) instead of dying mid-stream.
//!
//! Both run `AV_CODEC_FLAG_LOW_DELAY`; the host encodes zero-reorder streams (no B-frames, in-band
//! parameter sets on every IDR), so decode is strictly one-in/one-out.
//!
//! HDR is detected in-band from the decoded frame's transfer characteristic (`SMPTE2084` / PQ in the
//! HEVC VUI) — the same signal every other punktfunk client keys off — not from a protocol field.
use anyhow::{anyhow, bail, Context as _, Result};
use ffmpeg::format::Pixel;
use ffmpeg::software::scaling;
use ffmpeg::util::frame::Video as AvFrame;
use ffmpeg_next as ffmpeg;
use pf_client_core::video::ColorDesc;
use std::ffi::c_void;
use std::ptr;
use windows::core::{Interface, GUID};
use windows::Win32::Graphics::Direct3D11::{ID3D11Device, ID3D11VideoDevice};
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT, DXGI_FORMAT_NV12, DXGI_FORMAT_P010};
/// Which decode backend to use; the Settings UI persists this as a string.
#[derive(Clone, Copy, PartialEq, Eq, Debug, Default)]
pub enum DecoderPref {
/// Try D3D11VA, fall back to software.
#[default]
Auto,
/// Force D3D11VA (error out if unavailable, for debugging).
Hardware,
/// Force software decode.
Software,
}
impl DecoderPref {
pub fn from_name(s: &str) -> DecoderPref {
match s {
"hardware" => DecoderPref::Hardware,
"software" => DecoderPref::Software,
_ => DecoderPref::Auto,
}
}
}
pub enum DecodedFrame {
Cpu(CpuFrame),
Gpu(GpuFrame),
}
impl DecodedFrame {
pub fn dims(&self) -> (u32, u32) {
match self {
DecodedFrame::Cpu(c) => (c.width, c.height),
DecodedFrame::Gpu(g) => (g.width, g.height),
}
}
pub fn hdr(&self) -> bool {
match self {
DecodedFrame::Cpu(c) => c.hdr,
DecodedFrame::Gpu(g) => g.hdr,
}
}
}
/// A software-decoded frame in the same planar layout the hardware path produces: an NV12 (or
/// P010 for 10-bit) luma plane + interleaved chroma plane, each with its swscale row stride
/// (≥ the row bytes — swscale pads rows for SIMD). The presenter uploads them into two dynamic
/// plane textures sampled by the same shaders as the D3D11VA path.
pub struct CpuFrame {
pub width: u32,
pub height: u32,
/// Luma plane (`W×H` samples, 1 byte each; 2 for 10-bit) + its row stride in bytes.
pub y: Vec<u8>,
pub y_stride: usize,
/// Interleaved chroma plane (`⌈W/2⌉×⌈H/2⌉` UV pairs) + its row stride in bytes.
pub uv: Vec<u8>,
pub uv_stride: usize,
/// P010 sample layout (10 bits in the high bits of 16) vs NV12. Selects texture/SRV formats.
pub ten_bit: bool,
/// BT.2020 PQ HDR10 vs ordinary BT.709 SDR. Selects the swapchain colour space.
pub hdr: bool,
/// The frame's CICP signaling (HEVC VUI → `AVFrame`), read per-frame — the presenter derives
/// its YCbCr→RGB constant buffer from it (`csc_rows`), so a BT.601-signaled stream (a Linux
/// host's RGB-input NVENC) no longer renders with BT.709 coefficients.
pub color: ColorDesc,
}
/// A decoded frame still on the GPU: a D3D11 texture **array** plus the slice index the decoder
/// wrote this frame into. The presenter copies the slice into its own sampleable texture and
/// converts YUV→RGB in a pixel shader. The underlying surface stays alive — and out of the decoder's
/// reuse pool — for exactly as long as `guard` (an `av_frame_clone` of the decoded frame) lives.
pub struct GpuFrame {
pub width: u32,
pub height: u32,
/// Texture-array slice this frame occupies (`AVFrame::data[1]`).
pub index: u32,
/// The decode pool is P010 (10 bits in the high bits) vs NV12 — from the frames context's
/// `sw_format`. The presenter keys its copy-texture/SRV formats off this: they must match the
/// source array exactly for `CopySubresourceRegion`.
pub ten_bit: bool,
/// BT.2020 PQ HDR10 (ST.2084 transfer) vs ordinary BT.709 SDR. Selects the swapchain colour
/// space only (the host couples 10-bit ⟺ HDR today, but formats key off `ten_bit`).
pub hdr: bool,
/// Per-frame CICP signaling — see [`CpuFrame::color`].
pub color: ColorDesc,
guard: D3d11FrameGuard,
}
impl GpuFrame {
/// The decoder's D3D11 texture array holding this frame's slice, borrowed from the live cloned
/// `AVFrame`. Construct the windows-rs interface on the thread that will use it (the render
/// thread): COM interfaces are `!Send`, but the raw pointer is fine to carry across threads.
pub fn texture_ptr(&self) -> *mut c_void {
unsafe { (*self.guard.0).data[0] as *mut c_void }
}
}
/// Owns a cloned decoded `AVFrame` (which refs the D3D11 surface in the decoder pool). Dropping it
/// releases the surface back for reuse. The clone is plain refcounted data; freeing it from the
/// render thread is fine.
pub struct D3d11FrameGuard(*mut ffmpeg::ffi::AVFrame);
unsafe impl Send for D3d11FrameGuard {}
impl Drop for D3d11FrameGuard {
fn drop(&mut self) {
unsafe { ffmpeg::ffi::av_frame_free(&mut self.0) };
}
}
enum Backend {
D3d11va(D3d11vaDecoder),
Software(SoftwareDecoder),
}
pub struct Decoder {
backend: Backend,
/// The negotiated codec, so a mid-session D3D11VA→software demotion rebuilds for the same codec.
codec_id: ffmpeg::codec::Id,
}
/// Map a negotiated `quic` codec bit to the FFmpeg decoder id the client opens.
pub fn ffmpeg_codec_id(wire: u8) -> ffmpeg::codec::Id {
match wire {
punktfunk_core::quic::CODEC_H264 => ffmpeg::codec::Id::H264,
punktfunk_core::quic::CODEC_AV1 => ffmpeg::codec::Id::AV1,
_ => ffmpeg::codec::Id::HEVC,
}
}
/// The `quic` codec bitfield this client can decode — whatever FFmpeg has a decoder for (HEVC/H.264
/// always; AV1 when built in). Advertised to the host so it never emits a codec we can't decode.
/// Deliberately NOT gated on the DXVA profiles: software decode covers anything FFmpeg can.
pub fn decodable_codecs() -> u8 {
let _ = ffmpeg::init();
let mut bits = 0u8;
for (id, bit) in [
(ffmpeg::codec::Id::HEVC, punktfunk_core::quic::CODEC_HEVC),
(ffmpeg::codec::Id::H264, punktfunk_core::quic::CODEC_H264),
(ffmpeg::codec::Id::AV1, punktfunk_core::quic::CODEC_AV1),
] {
if ffmpeg::decoder::find(id).is_some() {
bits |= bit;
}
}
bits
}
impl Decoder {
pub fn new(pref: DecoderPref, codec_id: ffmpeg::codec::Id) -> Result<Decoder> {
ffmpeg::init().context("ffmpeg init")?;
if pref != DecoderPref::Software {
match D3d11vaDecoder::new(codec_id) {
Ok(d) => {
tracing::info!(?codec_id, "D3D11VA hardware decode active");
return Ok(Decoder {
backend: Backend::D3d11va(d),
codec_id,
});
}
Err(e) => {
if pref == DecoderPref::Hardware {
return Err(e.context("decoder=hardware but D3D11VA failed"));
}
tracing::info!(reason = %e, "D3D11VA unavailable — software decode");
}
}
}
Ok(Decoder {
backend: Backend::Software(SoftwareDecoder::new(codec_id)?),
codec_id,
})
}
/// True for the GPU hardware backend (shown in the stream HUD).
pub fn is_hardware(&self) -> bool {
matches!(self.backend, Backend::D3d11va(_))
}
/// Feed one access unit; returns the decoded frame (the host's streams are one-in/one-out). A
/// software decode error after packet loss is survivable — keep feeding. A D3D11VA error demotes
/// to software for the rest of the session (the next IDR resynchronizes).
pub fn decode(&mut self, au: &[u8]) -> Result<Option<DecodedFrame>> {
match &mut self.backend {
Backend::D3d11va(d) => match d.decode(au) {
Ok(f) => Ok(f.map(DecodedFrame::Gpu)),
Err(e) => {
tracing::warn!(error = %e, "D3D11VA decode failed — falling back to software");
self.backend = Backend::Software(SoftwareDecoder::new(self.codec_id)?);
Ok(None)
}
},
Backend::Software(s) => Ok(s.decode(au)?.map(DecodedFrame::Cpu)),
}
}
}
// --- DXVA decode-profile probe --------------------------------------------------------
/// DXVA decode-profile GUIDs (`dxva.h`), defined locally so no extra windows-rs feature or
/// metadata surface is pulled in for four constants.
const PROFILE_H264_VLD_NOFGT: GUID = GUID::from_u128(0x1b81be68_a0c7_11d3_b984_00c04f2e73c5);
const PROFILE_HEVC_VLD_MAIN: GUID = GUID::from_u128(0x5b11d51b_2f4c_4452_bcc3_09f2a1160cc0);
const PROFILE_HEVC_VLD_MAIN10: GUID = GUID::from_u128(0x107af0e0_ef1a_4d19_aba8_67a163073d13);
const PROFILE_AV1_VLD_PROFILE0: GUID = GUID::from_u128(0xb8be4ccb_cf53_46ba_8d59_d6b8a6da5d2a);
/// Does the shared device's adapter expose a DXVA decode profile for `codec_id`? Checked before
/// building the FFmpeg hwdevice because hwaccel selection (`get_format`) only runs on the FIRST
/// access unit — an unsupported profile would otherwise burn the opening IDR and recover through
/// the mid-stream demotion path instead of committing to software up front. Also logs (once) the
/// adapter's full profile list plus Main10 availability — the forensics for a new GPU/driver.
fn decode_profile_supported(device: &ID3D11Device, codec_id: ffmpeg::codec::Id) -> Result<()> {
let video: ID3D11VideoDevice = device
.cast()
.context("device lacks ID3D11VideoDevice (created without VIDEO_SUPPORT)")?;
let profiles: Vec<GUID> = unsafe {
let n = video.GetVideoDecoderProfileCount();
(0..n)
.filter_map(|i| video.GetVideoDecoderProfile(i).ok())
.collect()
};
log_profiles_once(&profiles);
let (wanted, format, name): (GUID, DXGI_FORMAT, &str) = match codec_id {
ffmpeg::codec::Id::H264 => (PROFILE_H264_VLD_NOFGT, DXGI_FORMAT_NV12, "H.264 VLD NoFGT"),
ffmpeg::codec::Id::HEVC => (PROFILE_HEVC_VLD_MAIN, DXGI_FORMAT_NV12, "HEVC Main"),
ffmpeg::codec::Id::AV1 => (PROFILE_AV1_VLD_PROFILE0, DXGI_FORMAT_NV12, "AV1 Profile 0"),
other => bail!("no DXVA profile known for {other:?}"),
};
let ok = profiles.contains(&wanted)
&& unsafe { video.CheckVideoDecoderFormat(&wanted, format) }
.map(|b| b.as_bool())
.unwrap_or(false);
if !ok {
bail!("adapter exposes no {name} decode profile");
}
// 10-bit (a mid-session HDR upgrade needs Main10): informational — if it's missing the
// decode error → software demotion + keyframe re-request path covers the switch.
if codec_id == ffmpeg::codec::Id::HEVC {
let main10 = profiles.contains(&PROFILE_HEVC_VLD_MAIN10)
&& unsafe { video.CheckVideoDecoderFormat(&PROFILE_HEVC_VLD_MAIN10, DXGI_FORMAT_P010) }
.map(|b| b.as_bool())
.unwrap_or(false);
tracing::info!(main10, "HEVC Main10 (10-bit/HDR) decode profile");
}
Ok(())
}
/// One-time dump of the adapter's DXVA decode profiles.
fn log_profiles_once(profiles: &[GUID]) {
use std::sync::atomic::{AtomicBool, Ordering};
static ONCE: AtomicBool = AtomicBool::new(true);
if ONCE.swap(false, Ordering::Relaxed) {
let list: Vec<String> = profiles.iter().map(|g| format!("{g:?}")).collect();
tracing::info!(count = profiles.len(), profiles = ?list, "adapter DXVA decode profiles");
}
}
// --- software backend ---------------------------------------------------------------
struct SoftwareDecoder {
decoder: ffmpeg::decoder::Video,
/// Rebuilt whenever the decoded format/size **or output format** changes (mid-stream
/// `Reconfigure`, or an 8↔10-bit flip): `(ctx, src_fmt, w, h, dst_fmt)`.
sws: Option<(scaling::Context, Pixel, u32, u32, Pixel)>,
}
impl SoftwareDecoder {
fn new(codec_id: ffmpeg::codec::Id) -> Result<SoftwareDecoder> {
let codec = ffmpeg::decoder::find(codec_id)
.ok_or_else(|| anyhow!("no {codec_id:?} decoder in libavcodec"))?;
let mut ctx = ffmpeg::codec::Context::new_with_codec(codec);
unsafe {
let raw = ctx.as_mut_ptr();
(*raw).flags |= ffmpeg::ffi::AV_CODEC_FLAG_LOW_DELAY as i32;
// Slice threading adds no frame delay (frame threading adds thread_count-1).
(*raw).thread_type = ffmpeg::ffi::FF_THREAD_SLICE;
(*raw).thread_count = 0; // auto
}
let decoder = ctx.decoder().video().context("open video decoder")?;
Ok(SoftwareDecoder { decoder, sws: None })
}
fn decode(&mut self, au: &[u8]) -> Result<Option<CpuFrame>> {
let packet = ffmpeg::Packet::copy(au);
self.decoder
.send_packet(&packet)
.map_err(|e| anyhow!("send_packet: {e}"))?;
let mut frame = AvFrame::empty();
let mut out = None;
while self.decoder.receive_frame(&mut frame).is_ok() {
out = Some(self.convert(&frame)?);
}
Ok(out)
}
/// Convert the decoded planar YUV to the hardware path's layout: NV12 for 8-bit, P010 for
/// 10-bit — a chroma interleave (and 10→16-high-bits shift), NOT a colour conversion. The
/// matrix/range/transfer handling all lives in the presenter's shaders, shared with the
/// D3D11VA path, so software frames are bit-comparable with hardware ones.
fn convert(&mut self, frame: &AvFrame) -> Result<CpuFrame> {
let (fmt, w, h) = (frame.format(), frame.width(), frame.height());
// SAFETY: `frame` wraps a live decoded AVFrame for the duration of this call.
let color = unsafe { ColorDesc::from_raw(frame.as_ptr()) };
let hdr = color.is_pq();
// Source bit depth from the pix-fmt descriptor (stable FFmpeg public API).
let ten_bit = unsafe {
let desc = ffmpeg::ffi::av_pix_fmt_desc_get(fmt.into());
!desc.is_null() && (*desc).comp[0].depth > 8
};
let dst = if ten_bit { Pixel::P010LE } else { Pixel::NV12 };
let rebuild = !matches!(&self.sws, Some((_, f, sw, sh, d)) if *f == fmt && *sw == w && *sh == h && *d == dst);
if rebuild {
let ctx = scaling::Context::get(fmt, w, h, dst, w, h, scaling::Flags::POINT)
.context("swscale context")?;
self.sws = Some((ctx, fmt, w, h, dst));
}
let (sws, ..) = self.sws.as_mut().unwrap();
let mut conv = AvFrame::empty();
sws.run(frame, &mut conv).map_err(|e| anyhow!("sws: {e}"))?;
Ok(CpuFrame {
width: w,
height: h,
y: conv.data(0).to_vec(),
y_stride: conv.stride(0),
uv: conv.data(1).to_vec(),
uv_stride: conv.stride(1),
ten_bit,
hdr,
color,
})
}
}
// --- D3D11VA backend ------------------------------------------------------------------
//
// Raw FFI: ffmpeg-next has no hwaccel wrappers. The COM-typed hwcontext structs are declared here
// (stable FFmpeg public ABI) rather than relied on from ffmpeg-sys bindgen — the generic
// AVHWDeviceContext / AVHWFramesContext (whose payload is an opaque `void *hwctx`) come from
// ffmpeg-sys, and we cast `hwctx` to the structs below. All owned pointers are freed in Drop;
// decoded surfaces transfer out through D3d11FrameGuard.
const AVERROR_EAGAIN: i32 = -11; // -EAGAIN
/// D3D11VA decode surface pool depth: the zero-reorder DPB (12 refs) + the bounded decoded channel
/// (2) + the frame the presenter currently holds (until its copy flushes) + one in-flight decode —
/// 12 is comfortable. A GPU that can't create the pool at all is gated out by
/// `d3d11va_decode_supported` and the session uses software decode.
const DECODE_POOL_SIZE: i32 = 12;
/// `hwcontext_d3d11va.h` — `AVHWDeviceContext::hwctx`. Leaving `lock` null makes FFmpeg install an
/// `ID3D11Multithread` default lock + set multithread protection on `device_context` during init,
/// which is what lets the presenter share this device's immediate context from the render thread.
#[repr(C)]
struct AVD3D11VADeviceContext {
device: *mut c_void, // ID3D11Device*
device_context: *mut c_void, // ID3D11DeviceContext*
video_device: *mut c_void, // ID3D11VideoDevice*
video_context: *mut c_void, // ID3D11VideoContext*
lock: *mut c_void, // void (*)(void*)
unlock: *mut c_void, // void (*)(void*)
lock_ctx: *mut c_void,
}
/// `hwcontext_d3d11va.h` — `AVHWFramesContext::hwctx`. The header is explicit: "The user must at
/// least set D3D11_BIND_DECODER if the frames context is to be used for video decoding" — a
/// user-built frames context gets NO default (BindFlags 0 → `CreateTexture2D` E_INVALIDARG); the
/// automatic OR-in lives only in libavcodec's own frames-param path, which we bypass.
#[repr(C)]
struct AVD3D11VAFramesContext {
texture: *mut c_void, // ID3D11Texture2D* (null → FFmpeg allocates the pool)
bind_flags: u32, // UINT BindFlags
misc_flags: u32, // UINT MiscFlags
texture_infos: *mut c_void, // AVD3D11FrameDescriptor* (FFmpeg-managed)
}
/// `D3D11_BIND_DECODER` — the decode pool's ONLY bind flag. Adding `D3D11_BIND_SHADER_RESOURCE`
/// is what NVIDIA rejects on a decoder texture ARRAY; the presenter samples via its own copy.
const BIND_DECODER: u32 = 0x200;
fn averr(what: &str, code: i32) -> anyhow::Error {
anyhow!("{what}: {}", ffmpeg::Error::from(code))
}
/// libavcodec's `get_format` callback: pick the D3D11 hw surface format and nothing else.
/// Deliberately does NOT build a frames context — with `hw_device_ctx` set and `hw_frames_ctx`
/// left null, libavcodec derives the decode pool itself (`ff_decode_get_hw_frames_ctx`), applying
/// every vendor quirk: DXVA surface alignment (128 for HEVC/AV1), DPB-based pool sizing, and the
/// decoder-only `D3D11_BIND_DECODER` flags. A hand-built context validated on NVIDIA was rejected
/// by Intel at the first `SubmitDecoderBuffers` (E_INVALIDARG) — the vendor-proof path is the one
/// the ffmpeg CLI/mpv ship. Returning anything but `AV_PIX_FMT_D3D11` aborts hardware decode →
/// the session demotes to software.
unsafe extern "C" fn get_format_d3d11(
avctx: *mut ffmpeg::ffi::AVCodecContext,
mut list: *const ffmpeg::ffi::AVPixelFormat,
) -> ffmpeg::ffi::AVPixelFormat {
use ffmpeg::ffi::*;
unsafe {
if (*avctx).hw_device_ctx.is_null() {
return AVPixelFormat::AV_PIX_FMT_NONE;
}
while *list != AVPixelFormat::AV_PIX_FMT_NONE {
if *list == AVPixelFormat::AV_PIX_FMT_D3D11 {
return AVPixelFormat::AV_PIX_FMT_D3D11;
}
list = list.add(1);
}
AVPixelFormat::AV_PIX_FMT_NONE
}
}
/// Predict whether D3D11VA decode will work by doing EXACTLY what the decoder's `get_format` does —
/// allocate an `AVHWFramesContext` (decoder-only pool, no shader-resource bind) and initialize it,
/// which creates the real NV12 decode surface array. On a GPU/driver that can't create the pool this
/// fails here, up front, so the session commits to software decode from the first frame (a clean,
/// gap-free stream) rather than decoding the IDR then dying mid-stream on a texture error that a
/// software demotion can't reliably recover from (the host's infinite GOP won't re-send an IDR).
unsafe fn d3d11va_decode_supported(hw_device: *mut ffmpeg::ffi::AVBufferRef) -> bool {
use ffmpeg::ffi::*;
unsafe {
let frames_ref = av_hwframe_ctx_alloc(hw_device);
if frames_ref.is_null() {
return false;
}
let frames = (*frames_ref).data as *mut AVHWFramesContext;
(*frames).format = AVPixelFormat::AV_PIX_FMT_D3D11;
(*frames).sw_format = AVPixelFormat::AV_PIX_FMT_NV12;
(*frames).width = 1920;
(*frames).height = 1152; // 128-aligned 1080p surface (the HEVC DXVA alignment, see get_format)
(*frames).initial_pool_size = DECODE_POOL_SIZE;
// Decoder-only — matches get_format exactly.
let fhw = (*frames).hwctx as *mut AVD3D11VAFramesContext;
(*fhw).bind_flags = BIND_DECODER;
let r = av_hwframe_ctx_init(frames_ref);
let mut fr = frames_ref;
av_buffer_unref(&mut fr);
r >= 0
}
}
struct D3d11vaDecoder {
ctx: *mut ffmpeg::ffi::AVCodecContext,
hw_device: *mut ffmpeg::ffi::AVBufferRef,
packet: *mut ffmpeg::ffi::AVPacket,
frame: *mut ffmpeg::ffi::AVFrame,
}
// Single-owner pointers, only touched from the session pump thread.
unsafe impl Send for D3d11vaDecoder {}
impl D3d11vaDecoder {
fn new(codec_id: ffmpeg::codec::Id) -> Result<D3d11vaDecoder> {
use ffmpeg::ffi;
let shared = crate::gpu::shared().ok_or_else(|| anyhow!("no shared D3D11 device"))?;
if !shared.hardware {
bail!("shared device is WARP (no hardware video decode)");
}
// The adapter must expose the codec's DXVA profile — checked here, not at the first AU.
decode_profile_supported(&shared.device, codec_id)?;
unsafe {
// Build a D3D11VA hwdevice context around the *shared* device, so decoded textures live
// on the same device the presenter samples + draws with.
let hw_device =
ffi::av_hwdevice_ctx_alloc(ffi::AVHWDeviceType::AV_HWDEVICE_TYPE_D3D11VA);
if hw_device.is_null() {
bail!("av_hwdevice_ctx_alloc(D3D11VA) failed");
}
let devctx = (*hw_device).data as *mut ffi::AVHWDeviceContext;
let d3dctx = (*devctx).hwctx as *mut AVD3D11VADeviceContext;
// Hand FFmpeg an owned ref to the device + immediate context (it Releases them when the
// hwdevice ctx is freed). `into_raw()` transfers a +1 ref without releasing.
(*d3dctx).device = shared.device.clone().into_raw();
(*d3dctx).device_context = shared.context.clone().into_raw();
// lock left null → FFmpeg installs the ID3D11Multithread default lock in init.
let r = ffi::av_hwdevice_ctx_init(hw_device);
if r < 0 {
let mut hw = hw_device;
ffi::av_buffer_unref(&mut hw);
bail!("av_hwdevice_ctx_init: {}", ffmpeg::Error::from(r));
}
// Up-front viability probe (see `d3d11va_decode_supported`): a GPU/driver that can't
// create the decode surface pool commits to software NOW, so it decodes cleanly from the
// first frame instead of failing mid-stream (which a demotion can't reliably recover).
if !d3d11va_decode_supported(hw_device) {
let mut hw = hw_device;
ffi::av_buffer_unref(&mut hw);
bail!("GPU can't create the D3D11VA decode surface pool — using software decode");
}
let codec = ffi::avcodec_find_decoder(codec_id.into());
if codec.is_null() {
let mut hw = hw_device;
ffi::av_buffer_unref(&mut hw);
bail!("no {codec_id:?} decoder");
}
let ctx = ffi::avcodec_alloc_context3(codec);
(*ctx).hw_device_ctx = ffi::av_buffer_ref(hw_device);
(*ctx).get_format = Some(get_format_d3d11);
(*ctx).flags |= ffi::AV_CODEC_FLAG_LOW_DELAY as i32;
// hwaccel: threads only add latency.
(*ctx).thread_count = 1;
// On top of the DPB-based pool libavcodec sizes for us: the bounded decoded channel
// (2) + the frame the presenter holds until its copy flushes + margin.
(*ctx).extra_hw_frames = 4;
let r = ffi::avcodec_open2(ctx, codec, ptr::null_mut());
if r < 0 {
let mut ctx = ctx;
ffi::avcodec_free_context(&mut ctx);
let mut hw = hw_device;
ffi::av_buffer_unref(&mut hw);
bail!("avcodec_open2 (D3D11VA): {}", ffmpeg::Error::from(r));
}
Ok(D3d11vaDecoder {
ctx,
hw_device,
packet: ffi::av_packet_alloc(),
frame: ffi::av_frame_alloc(),
})
}
}
fn decode(&mut self, au: &[u8]) -> Result<Option<GpuFrame>> {
use ffmpeg::ffi;
unsafe {
let r = ffi::av_new_packet(self.packet, au.len() as i32);
if r < 0 {
return Err(averr("av_new_packet", r));
}
ptr::copy_nonoverlapping(au.as_ptr(), (*self.packet).data, au.len());
let r = ffi::avcodec_send_packet(self.ctx, self.packet);
ffi::av_packet_unref(self.packet);
if r < 0 {
return Err(averr("send_packet", r));
}
let mut out = None;
loop {
let r = ffi::avcodec_receive_frame(self.ctx, self.frame);
if r == AVERROR_EAGAIN {
break;
}
if r < 0 {
return Err(averr("receive_frame", r));
}
out = Some(self.lift()?); // newest wins; older guards drop here
ffi::av_frame_unref(self.frame);
}
Ok(out)
}
}
/// Lift the decoded D3D11 surface into a `GpuFrame`. `data[0]` is the texture array, `data[1]`
/// the slice index. We `av_frame_clone` so the surface stays referenced (kept out of the reuse
/// pool) until the presenter drops the guard.
unsafe fn lift(&mut self) -> Result<GpuFrame> {
use ffmpeg::ffi;
unsafe {
if (*self.frame).format != ffi::AVPixelFormat::AV_PIX_FMT_D3D11 as i32 {
bail!("decoder returned a software frame (no D3D11 surface)");
}
// SAFETY: `self.frame` is the live decoded AVFrame for the duration of this call.
let color = ColorDesc::from_raw(self.frame);
let hdr = color.is_pq();
let ten_bit = {
let hwfc = (*self.frame).hw_frames_ctx;
!hwfc.is_null()
&& (*((*hwfc).data as *const ffi::AVHWFramesContext)).sw_format
== ffi::AVPixelFormat::AV_PIX_FMT_P010LE
};
let cloned = ffi::av_frame_clone(self.frame);
if cloned.is_null() {
bail!("av_frame_clone failed");
}
let frame = GpuFrame {
width: (*self.frame).width as u32,
height: (*self.frame).height as u32,
index: (*self.frame).data[1] as usize as u32,
ten_bit,
hdr,
color,
guard: D3d11FrameGuard(cloned),
};
log_layout_once(frame.width, frame.height, frame.index, hdr, ten_bit);
Ok(frame)
}
}
}
impl Drop for D3d11vaDecoder {
fn drop(&mut self) {
use ffmpeg::ffi;
unsafe {
ffi::av_packet_free(&mut self.packet);
ffi::av_frame_free(&mut self.frame);
ffi::avcodec_free_context(&mut self.ctx);
ffi::av_buffer_unref(&mut self.hw_device);
}
}
}
/// One-time dump of the first decoded surface's layout — so a new GPU/driver combination's real
/// format (slice index range, HDR/bit-depth) is visible in the logs without a debugger.
fn log_layout_once(width: u32, height: u32, index: u32, hdr: bool, ten_bit: bool) {
use std::sync::atomic::{AtomicBool, Ordering};
static ONCE: AtomicBool = AtomicBool::new(true);
if ONCE.swap(false, Ordering::Relaxed) {
tracing::info!(
width,
height,
slice = index,
hdr,
ten_bit,
"D3D11VA first frame"
);
}
}
File diff suppressed because it is too large Load Diff
+68 -309
View File
@@ -10,6 +10,7 @@ use crate::audio;
use crate::video::{DecodedFrame, DecodedImage, Decoder}; use crate::video::{DecodedFrame, DecodedImage, Decoder};
use punktfunk_core::client::NativeClient; use punktfunk_core::client::NativeClient;
use punktfunk_core::config::{CompositorPref, GamepadPref, Mode}; use punktfunk_core::config::{CompositorPref, GamepadPref, Mode};
use punktfunk_core::reanchor::{index_gap, GateVerdict, ReanchorGate};
use punktfunk_core::PunktfunkError; use punktfunk_core::PunktfunkError;
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc; use std::sync::Arc;
@@ -99,86 +100,6 @@ pub struct Stats {
pub decoder: &'static str, pub decoder: &'static str,
} }
/// Consecutive no-output AUs that force a keyframe request. ~50 ms at 60 Hz — long
/// enough not to fire on a one-frame decoder hiccup, short enough that a lost initial
/// IDR (or a mid-GOP join) unfreezes almost immediately instead of never.
const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3;
/// Longest the pump holds the last good frame waiting for a post-loss re-anchor keyframe before it
/// gives up and resumes display. After a reference loss the hardware decoder does not error — it
/// conceals the reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as
/// a gray plate with the new frame's motion painted over it) and returns Ok, so displaying them is
/// the "gray frames mid-stream" artifact. We instead freeze on the last good picture until a fresh
/// IDR re-anchors decode — the behaviour NVIDIA already shows (its DISTINCT output image + different
/// concealment reads as a brief freeze, not gray). This cap only bounds the freeze when recovery
/// genuinely stalls (host ignores the request, or an RFI recovery that never emits a keyframe), so a
/// glitch can never become a permanent freeze. A recovery IDR round-trips well under this on any
/// live link.
const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500);
/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
/// latest frame gap before the pump lifts its freeze on an IDR-free stream. TWO, not one: with a
/// continuous rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a
/// loss is only partially healed — stripes swept BEFORE the loss still reference the lost frame — and
/// lifting there would flash a partially-stale picture. The SECOND boundary guarantees a full wave
/// swept entirely after the loss, so the picture is clean. This stays correct under repeated loss
/// because every new gap resets the count. The cost is up to ~2 wave periods of holding the last good
/// frame — the deliberate "hold longer, never show garbage" trade.
///
/// [`USER_FLAG_RECOVERY_POINT`]: punktfunk_core::packet::USER_FLAG_RECOVERY_POINT
const REANCHOR_MARKS_TO_LIFT: u32 = 2;
/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark
/// pushes the freeze deadline out by this much, so a live mark stream (the host actively healing via
/// its wave) keeps the client patiently holding the last good frame instead of tripping the IDR
/// floor mid-heal. Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the
/// marks STOP (heal stalled, or the host isn't running intra-refresh) the deadline lapses and the
/// normal recovery-IDR floor fires, so a real stall still recovers.
const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500);
/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is
/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters
/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small
/// positive delta is a forward gap (missing frames whose dependents will decode against absent
/// references); a delta in the top half is an index behind us.
fn index_gap(expected: u32, got: u32) -> Option<u32> {
let ahead = got.wrapping_sub(expected);
(ahead != 0 && ahead < u32::MAX / 2).then_some(ahead)
}
/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze.
///
/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried
/// [`USER_FLAG_RECOVERY_ANCHOR`](punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR), the host's
/// definitive single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a
/// known-good reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait.
/// `has_mark` — this AU carried [`USER_FLAG_RECOVERY_POINT`](punktfunk_core::packet::USER_FLAG_RECOVERY_POINT),
/// a host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks
/// seen since the latest gap.
///
/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0
/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested
/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only
/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and
/// lifts immediately.
fn reanchor_after_frame(
is_keyframe: bool,
has_anchor: bool,
has_mark: bool,
marks: u32,
) -> (bool, u32) {
let marks = if has_mark {
marks.saturating_add(1)
} else {
marks
};
if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT {
(true, 0)
} else {
(false, marks)
}
}
/// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs). /// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs).
/// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old /// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old
/// host (no 0xCF at all) this just caps the dead-weight ring. /// host (no 0xCF at all) this just caps the dead-weight ring.
@@ -382,27 +303,17 @@ fn pump(
// What actually decoded the last frame — a VAAPI failure demotes mid-session, so // What actually decoded the last frame — a VAAPI failure demotes mid-session, so
// this is read off each frame's image variant rather than fixed at startup. // this is read off each frame's image variant rather than fixed at startup.
let mut dec_path: &'static str = ""; let mut dec_path: &'static str = "";
// Loss recovery: watch the host→client unrecoverable-drop count and ask for an IDR when it climbs.
let mut last_dropped = connector.frames_dropped();
// The stats window keeps its own drop cursor — the OSD shows the per-window delta. // The stats window keeps its own drop cursor — the OSD shows the per-window delta.
let mut window_dropped = last_dropped; let mut window_dropped = connector.frames_dropped();
let mut last_kf_req: Option<Instant> = None; let mut last_kf_req: Option<Instant> = None;
// Consecutive received AUs that produced NO decoded frame (decode error, or the // Freeze-until-reanchor: the shared post-loss gate ([`punktfunk_core::reanchor::ReanchorGate`]).
// decoder swallowed a reference-missing delta and returned nothing). Distinct from // Armed on any loss signal (frame-index gap, dropped-count climb, decoder wedge/demotion), it
// `frames_dropped`, which counts reassembler drops: when the initial IDR is lost (or // withholds the decoder's concealed frames from the presenter — which then redraws the last good
// we join mid-GOP) the reassembler delivers complete-but-undecodable deltas — it // picture — until a proven clean re-anchor (IDR / RFI anchor / second recovery mark) lifts it. It
// never drops, so the drop-count trigger below stays silent and the stream freezes // also owns the no-output streak and the overdue-freeze backstop; the client keeps its own
// on the last good frame. A short streak forces a fresh IDR to re-anchor. // `last_kf_req` request throttle and routes the gate's keyframe intents through it. Seeded with the
let mut no_output_streak = 0u32; // current drop count so the first `poll` doesn't read the baseline as a loss.
// Freeze-until-reanchor: armed the moment we request a recovery keyframe (loss, decode error, or let mut gate = ReanchorGate::new(connector.frames_dropped());
// a no-output streak), it withholds the decoder's concealed frames from the presenter — which
// then redraws the last good picture — until a fresh keyframe re-anchors decode. See
// [`REANCHOR_FREEZE_MAX`] for why this exists and its backstop deadline.
let mut awaiting_reanchor = false;
let mut reanchor_deadline: Option<Instant> = None;
// Host intra-refresh recovery marks seen since the latest gap (see [`REANCHOR_MARKS_TO_LIFT`]).
// Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks.
let mut recovery_marks: u32 = 0;
// The frame_index we expect next (the host numbers frames consecutively). A jump means a frame // The frame_index we expect next (the host numbers frames consecutively). A jump means a frame
// went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms // went missing — the earliest, most reliable signal that the decoder is about to conceal, ~120 ms
// ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of // ahead of `frames_dropped` (the reassembler only declares a straggler lost once it ages out of
@@ -447,9 +358,7 @@ fn pump(
Some(exp) => { Some(exp) => {
if let Some(gap) = index_gap(exp, frame.frame_index) { if let Some(gap) = index_gap(exp, frame.frame_index) {
let now = Instant::now(); let now = Instant::now();
awaiting_reanchor = true; gate.arm(now);
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
next_expected_index = Some(frame.frame_index.wrapping_add(1)); next_expected_index = Some(frame.frame_index.wrapping_add(1));
// The gap carries the PRECISE lost range — [first missing, newest // The gap carries the PRECISE lost range — [first missing, newest
// received - 1] — so this is the one recovery signal that can drive true // received - 1] — so this is the one recovery signal that can drive true
@@ -488,38 +397,14 @@ fn pump(
} }
match decoder.decode(&frame.data) { match decoder.decode(&frame.data) {
Ok(Some(image)) => { Ok(Some(image)) => {
// A decoded frame — the anchor holds. // Fold this decoded frame through the shared freeze gate: it reads the AU's
no_output_streak = 0; // re-anchor wire flags (FLAG_SOF IDR marker / RECOVERY_ANCHOR / RECOVERY_POINT),
// Host-signalled intra-refresh recovery mark: on an IDR-free intra-refresh // takes `image.is_keyframe()` as the ffmpeg keyframe belt, applies the two-mark
// stream this wave-boundary flag is the only clean point the client can honor // rule + the mark-patience backstop, clears the no-output streak, and returns
// (the decoder never flags the re-anchor — the coded frame stays `P`). A live // whether to present this frame or withhold it as a post-loss concealment.
// mark stream also means the host is actively healing, so push the backstop out let present =
// rather than trip a mid-heal IDR (see `RECOVERY_MARK_PATIENCE`). gate.on_decoded(frame.flags, image.is_keyframe(), Instant::now())
let has_mark = == GateVerdict::Present;
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_POINT != 0;
// The host's definitive single-frame re-anchor: an LTR-RFI recovery frame (a
// clean P-frame off a known-good reference), the AMD twin of an IDR re-anchor
// but without the spike. It lifts on the FIRST occurrence.
let has_anchor =
frame.flags & punktfunk_core::packet::USER_FLAG_RECOVERY_ANCHOR != 0;
if has_mark && awaiting_reanchor {
reanchor_deadline = Some(Instant::now() + RECOVERY_MARK_PATIENCE);
}
// A fresh clean re-anchor lifts the freeze and shows this frame: a real intra
// keyframe (IDR, always clean), an LTR-RFI recovery anchor (also whole), OR the
// second recovery mark since the gap (the first wave boundary is only
// half-healed — see `reanchor_after_frame`).
let (lift, marks) = reanchor_after_frame(
image.is_keyframe(),
has_anchor,
has_mark,
recovery_marks,
);
recovery_marks = marks;
if lift {
awaiting_reanchor = false;
reanchor_deadline = None;
}
total_frames += 1; total_frames += 1;
dec_path = match &image { dec_path = match &image {
DecodedImage::Cpu(_) => "software", DecodedImage::Cpu(_) => "software",
@@ -574,19 +459,19 @@ fn pump(
DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)), DecodedImage::VkFrame(v) => Some((v.timeline_sem, v.decode_done_value)),
_ => None, _ => None,
}; };
if awaiting_reanchor { if present {
// Post-loss concealment: withhold this frame (it references a lost/gray
// reference) so the presenter keeps redrawing the last good picture
// rather than flashing the decoder's gray plate. Dropped here — the
// hw-decode stat below still samples via `hw_fence` (raw handle + value,
// valid past the guard). Cleared by the next keyframe or the backstop.
tracing::trace!("holding last frame — awaiting post-loss re-anchor");
} else {
let _ = frame_tx.force_send(DecodedFrame { let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns, pts_ns: frame.pts_ns,
decoded_ns, decoded_ns,
image, image,
}); });
} else {
// Post-loss concealment: withhold this frame (it references a lost/gray
// reference) so the presenter keeps redrawing the last good picture rather
// than flashing the decoder's gray plate. Dropped here — the hw-decode stat
// below still samples via `hw_fence` (raw handle + value, valid past the
// guard). The gate lifts the freeze on the next clean re-anchor / backstop.
tracing::trace!("holding last frame — awaiting post-loss re-anchor");
} }
// `decode` stage: received→decode COMPLETE, single clock. // `decode` stage: received→decode COMPLETE, single clock.
match hw_fence { match hw_fence {
@@ -602,36 +487,35 @@ fn pump(
} }
} }
} }
Ok(None) => no_output_streak += 1, // The decoder produced nothing — under zero-reorder LOW_DELAY (one-in/one-out) that
// Survivable (loss until the next IDR/RFI recovery) — keep feeding. // means it's wedged on missing references with no reassembler drop to trigger
Err(e) => { // recovery. The gate counts the streak and, once it trips, arms the freeze and tells
no_output_streak += 1; // us to (throttled) request a fresh IDR to re-anchor. Both the empty-output and the
tracing::debug!(error = %e, "decode error (recovering)"); // survivable-decode-error arms feed it; a decoded frame resets the streak in
} // `on_decoded`.
} Ok(None) => {
// The decoder has produced nothing for a short run — under zero-reorder
// LOW_DELAY (one-in/one-out) that means it's wedged on missing references
// with no reassembler drop to trigger recovery below. Ask for a fresh IDR
// (throttled), then re-arm the streak so we wait out the request→IDR round
// trip before asking again instead of flooding.
if no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK {
let now = Instant::now(); let now = Instant::now();
// Wedged on missing references: hold the last good frame until re-anchor if gate.on_no_output(now)
// (armed even when the IDR request itself is throttled — the stream is broken && last_kf_req
// regardless of whether we ask again this iteration).
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{ {
last_kf_req = Some(now); last_kf_req = Some(now);
let _ = connector.request_keyframe(); let _ = connector.request_keyframe();
tracing::debug!( tracing::debug!("requested keyframe (decoder produced no output)");
streak = no_output_streak, }
"requested keyframe (decoder produced no output)" }
); // Survivable (loss until the next IDR/RFI recovery) — keep feeding.
no_output_streak = 0; Err(e) => {
tracing::debug!(error = %e, "decode error (recovering)");
let now = Instant::now();
if gate.on_no_output(now)
&& last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!("requested keyframe (decode error recovery)");
}
} }
} }
// The presenter's verdict: hardware frames can't be displayed (GL converter // The presenter's verdict: hardware frames can't be displayed (GL converter
@@ -649,9 +533,7 @@ fn pump(
// through the same throttle as loss recovery below. // through the same throttle as loss recovery below.
if decoder.take_keyframe_request() { if decoder.take_keyframe_request() {
let now = Instant::now(); let now = Instant::now();
awaiting_reanchor = true; gate.arm(now);
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) .is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{ {
@@ -679,41 +561,26 @@ fn pump(
} }
} }
// Loss recovery: under infinite GOP the only recovery keyframe is one we request. The // Loss recovery + overdue backstop, folded through the shared gate. A climb in the
// reassembler drops unrecoverable AUs (frames_dropped); the decoder then conceals the // reassembler's unrecoverable-drop count (`frames_dropped`) means the AUs after the lost one
// reference-missing delta frames that follow and returns Ok, so keying off a decode error // reference a picture we never decoded — the decoder conceals them (gray on RADV) and returns
// rarely fires. Request an IDR when the drop count climbs, throttled — the decode stays // Ok, so a decode-error trigger rarely fires; the gate arms the freeze on the climb instead. An
// wedged for several frames until the IDR lands, so requesting every frame would flood. // overdue freeze (held a full REANCHOR_FREEZE_MAX with no clean re-anchor — a lost recovery IDR,
// or a benign reorder that produced no `frames_dropped`) re-asks while it keeps holding: NEVER
// resume to gray — a genuinely dead stream is the QUIC idle-timeout watchdog's job. Both route
// the gate's keyframe intent through the shared 100 ms throttle; under infinite GOP the only
// recovery keyframe is one we request.
let dropped = connector.frames_dropped(); let dropped = connector.frames_dropped();
if dropped > last_dropped {
last_dropped = dropped;
let now = Instant::now(); let now = Instant::now();
// A dropped AU means the frames after it reference a picture we never decoded — the if gate.poll(dropped, now)
// decoder will conceal them (gray on RADV). Freeze on the last good frame until a fresh && last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
// IDR re-anchors, so the concealment never reaches the screen. {
awaiting_reanchor = true;
recovery_marks = 0;
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now); last_kf_req = Some(now);
let _ = connector.request_keyframe(); let _ = connector.request_keyframe();
tracing::debug!(dropped, "requested keyframe (loss recovery)"); tracing::debug!(
} dropped,
} "requested keyframe (loss recovery / overdue re-anchor)"
// Re-anchor overdue: the freeze has held the whole window with no keyframe — a lost recovery );
// IDR, or a benign reorder that produced no `frames_dropped` and so requested none. Do NOT
// resume to gray (the one thing worse than a freeze): keep holding the last good frame and
// (re-)request a keyframe, throttled + host-coalesced, so a CLEAN re-anchor is what un-freezes
// us. A genuinely dead stream — host gone, link collapsed — is caught by the QUIC idle-timeout
// watchdog (returns to the menu), never by painting the decoder's concealment.
if awaiting_reanchor && reanchor_deadline.is_some_and(|d| Instant::now() >= d) {
let now = Instant::now();
reanchor_deadline = Some(now + REANCHOR_FREEZE_MAX);
if last_kf_req.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100)) {
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!("re-anchor overdue — still holding, re-requesting keyframe");
}
} }
if window_start.elapsed() >= Duration::from_secs(1) { if window_start.elapsed() >= Duration::from_secs(1) {
@@ -836,111 +703,3 @@ fn spawn_audio(
.map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled")) .map_err(|e| tracing::warn!(error = %e, "audio thread failed to start — audio disabled"))
.ok() .ok()
} }
#[cfg(test)]
mod tests {
use super::{index_gap, reanchor_after_frame, REANCHOR_MARKS_TO_LIFT};
// Simulate the pump's re-anchor state across a sequence of decoded frames: each `(is_keyframe,
// has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based)
// at which the freeze first lifts, or `None` if it never does. `gap_before` reset points model a
// fresh loss re-arming the freeze (the pump zeroes the count at every gap/arm site).
fn lift_at(frames: &[(bool, bool)]) -> Option<usize> {
let mut marks = 0u32;
for (i, &(is_kf, has_mark)) in frames.iter().enumerate() {
// The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised
// by `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here.
let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks);
marks = m;
if lift {
return Some(i);
}
}
None
}
#[test]
fn a_single_recovery_mark_does_not_lift() {
// The first wave boundary after a loss is only half-healed — one mark must hold the freeze.
assert_eq!(REANCHOR_MARKS_TO_LIFT, 2);
assert_eq!(lift_at(&[(false, true)]), None);
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false)]),
None
);
}
#[test]
fn the_second_recovery_mark_lifts() {
// Two marks = a full wave swept after the loss → clean re-anchor.
assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1));
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false), (false, true)]),
Some(3)
);
}
#[test]
fn a_real_keyframe_lifts_immediately() {
// An IDR is always a clean anchor — no marks needed.
assert_eq!(lift_at(&[(true, false)]), Some(0));
assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1));
}
#[test]
fn a_fresh_gap_resets_the_mark_count() {
// The pump zeroes `recovery_marks` at each arm site, so one mark before a new gap plus one
// after must NOT lift — the model resets the running count to imitate that.
let mut marks = 0u32;
let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap)
marks = m;
assert_eq!(marks, 1);
marks = 0; // a new gap re-arms the freeze → count reset
let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave
assert!(!lift, "a single post-gap mark must not lift");
assert_eq!(m, 1);
}
#[test]
fn an_rfi_anchor_lifts_immediately() {
// An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good
// reference), so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait.
let (lift, marks) = reanchor_after_frame(false, true, false, 0);
assert!(lift, "an RFI anchor must lift the freeze immediately");
assert_eq!(marks, 0, "a lift resets the running mark count");
// Even with zero prior marks and no keyframe, the anchor alone is sufficient.
let (lift, _) = reanchor_after_frame(false, true, true, 1);
assert!(lift, "an anchor lifts regardless of the pending mark count");
}
#[test]
fn contiguous_indices_are_not_a_gap() {
assert_eq!(index_gap(5, 5), None);
assert_eq!(index_gap(0, 0), None);
}
#[test]
fn a_forward_jump_reports_the_skip_count() {
assert_eq!(index_gap(5, 6), Some(1)); // one frame missing
assert_eq!(index_gap(5, 9), Some(4));
}
#[test]
fn a_straggler_behind_us_is_not_a_gap() {
// The reassembler emitted a newer frame first; the late one must not re-arm.
assert_eq!(index_gap(9, 5), None);
assert_eq!(index_gap(1, 0), None);
}
#[test]
fn the_index_counter_wraps_cleanly() {
// last frame = u32::MAX, so the next expected wraps to 0.
// Contiguous across the wrap.
assert_eq!(index_gap(0, 0), None);
// waiting on u32::MAX, frame 0 arrived → MAX was skipped.
assert_eq!(index_gap(u32::MAX, 0), Some(1));
assert_eq!(index_gap(u32::MAX, 2), Some(3));
// an old frame arriving just after the wrap is still a straggler.
assert_eq!(index_gap(0, u32::MAX), None);
}
}
+141
View File
@@ -13,6 +13,7 @@
use crate::config::{Config, FecConfig, FecScheme, ProtocolPhase, Role}; use crate::config::{Config, FecConfig, FecScheme, ProtocolPhase, Role};
use crate::error::PunktfunkStatus; use crate::error::PunktfunkStatus;
use crate::input::InputEvent; use crate::input::InputEvent;
use crate::reanchor::{GateVerdict, ReanchorGate};
use crate::session::Session; use crate::session::Session;
use crate::stats::Stats; use crate::stats::Stats;
use crate::transport::{loopback_pair, Transport, UdpTransport}; use crate::transport::{loopback_pair, Transport, UdpTransport};
@@ -2620,3 +2621,143 @@ pub unsafe extern "C" fn punktfunk_connection_close(c: *mut PunktfunkConnection)
drop(unsafe { Box::from_raw(c) }); drop(unsafe { Box::from_raw(c) });
} }
} }
// ---- Post-loss re-anchor freeze gate ----
//
// The shared [`ReanchorGate`](crate::reanchor::ReanchorGate) exposed for the Swift client (Rust
// embedders — Android/Windows/Linux — use the struct directly). After an unrecoverable reference
// loss the decoder silently conceals the missing-reference deltas (gray/garbage picture, no error);
// the client freezes on the last good frame and lifts only on a proven clean re-anchor. The gate
// takes time internally (`Instant::now`) so no timestamps cross the boundary. Drive it per session:
// `arm` on a loss (frame-index gap from `punktfunk_connection_note_frame_index`, a decoder
// wedge/demotion), `on_decoded` per decoded frame to gate presentation, `on_no_output` per AU that
// produced nothing, and `poll` each iteration for the dropped-count climb + overdue backstop. Route
// the returned keyframe intents through the client's existing request throttle.
/// Create a re-anchor gate seeded with the session's current `frames_dropped` (so the first
/// [`punktfunk_reanchor_gate_poll`] doesn't read the baseline as a loss). Free with
/// [`punktfunk_reanchor_gate_free`]. Never returns NULL.
#[no_mangle]
pub extern "C" fn punktfunk_reanchor_gate_new(frames_dropped: u64) -> *mut ReanchorGate {
Box::into_raw(Box::new(ReanchorGate::new(frames_dropped)))
}
/// Free a gate created by [`punktfunk_reanchor_gate_new`]. NULL is a no-op.
///
/// # Safety
/// `g` was returned by [`punktfunk_reanchor_gate_new`] and is not used after this call.
#[no_mangle]
pub unsafe extern "C" fn punktfunk_reanchor_gate_free(g: *mut ReanchorGate) {
if !g.is_null() {
drop(unsafe { Box::from_raw(g) });
}
}
/// Arm the freeze: a loss was detected (a frame-index gap, or a decoder wedge/demotion). Zeroes the
/// recovery-mark count and (re-)sets the backstop deadline. NULL is a no-op.
///
/// # Safety
/// `g` is a valid gate handle.
#[no_mangle]
pub unsafe extern "C" fn punktfunk_reanchor_gate_arm(g: *mut ReanchorGate) {
if let Some(g) = unsafe { g.as_mut() } {
g.arm(std::time::Instant::now());
}
}
/// Fold one decoded frame and write to `out_present` whether to display it (`true`) or withhold it as
/// a post-loss concealment (`false`). `flags` is the AU's `user_flags` word ([`PunktfunkFrame::flags`]):
/// the gate reads `FLAG_SOF` (the host's IDR marker), `USER_FLAG_RECOVERY_ANCHOR` and
/// `USER_FLAG_RECOVERY_POINT`. Pass `decoder_keyframe = false` where the platform decoder doesn't flag
/// IDRs (VideoToolbox/MediaCodec) — the wire `FLAG_SOF` covers it.
///
/// # Safety
/// `g` is a valid gate handle; `out_present` is writable or NULL.
#[no_mangle]
pub unsafe extern "C" fn punktfunk_reanchor_gate_on_decoded(
g: *mut ReanchorGate,
flags: u32,
decoder_keyframe: bool,
out_present: *mut bool,
) -> PunktfunkStatus {
guard(|| {
let g = match unsafe { g.as_mut() } {
Some(g) => g,
None => return PunktfunkStatus::NullPointer,
};
let present = g.on_decoded(flags, decoder_keyframe, std::time::Instant::now())
== GateVerdict::Present;
if !out_present.is_null() {
unsafe { *out_present = present };
}
PunktfunkStatus::Ok
})
}
/// A received AU produced no decoded frame. Writes to `out_request_kf` whether the no-output streak has
/// tripped and the client should (throttled) request a keyframe — the gate arms the freeze at the same
/// time.
///
/// # Safety
/// `g` is a valid gate handle; `out_request_kf` is writable or NULL.
#[no_mangle]
pub unsafe extern "C" fn punktfunk_reanchor_gate_on_no_output(
g: *mut ReanchorGate,
out_request_kf: *mut bool,
) -> PunktfunkStatus {
guard(|| {
let g = match unsafe { g.as_mut() } {
Some(g) => g,
None => return PunktfunkStatus::NullPointer,
};
let request = g.on_no_output(std::time::Instant::now());
if !out_request_kf.is_null() {
unsafe { *out_request_kf = request };
}
PunktfunkStatus::Ok
})
}
/// Periodic fold of the session's `frames_dropped` counter plus the overdue backstop. Writes to
/// `out_request_kf` whether the client should (throttled) request a keyframe (a drop-count climb armed
/// a fresh freeze, or the freeze is overdue and re-asks while it keeps holding).
///
/// # Safety
/// `g` is a valid gate handle; `out_request_kf` is writable or NULL.
#[no_mangle]
pub unsafe extern "C" fn punktfunk_reanchor_gate_poll(
g: *mut ReanchorGate,
frames_dropped: u64,
out_request_kf: *mut bool,
) -> PunktfunkStatus {
guard(|| {
let g = match unsafe { g.as_mut() } {
Some(g) => g,
None => return PunktfunkStatus::NullPointer,
};
let request = g.poll(frames_dropped, std::time::Instant::now());
if !out_request_kf.is_null() {
unsafe { *out_request_kf = request };
}
PunktfunkStatus::Ok
})
}
/// Whether the gate is currently withholding concealed frames (frozen on the last good picture).
/// Writes `false` on a NULL gate.
///
/// # Safety
/// `g` is a valid gate handle; `out_holding` is writable or NULL.
#[no_mangle]
pub unsafe extern "C" fn punktfunk_reanchor_gate_is_holding(
g: *const ReanchorGate,
out_holding: *mut bool,
) -> PunktfunkStatus {
guard(|| {
let holding = unsafe { g.as_ref() }.is_some_and(ReanchorGate::is_holding);
if !out_holding.is_null() {
unsafe { *out_holding = holding };
}
PunktfunkStatus::Ok
})
}
+85 -3
View File
@@ -1573,6 +1573,23 @@ async fn worker_main(args: WorkerArgs) {
// Touched pads only: an entry appears on the first gamepad event for that index, so the // Touched pads only: an entry appears on the first gamepad event for that index, so the
// refresh never conjures a virtual pad the embedder didn't drive. // refresh never conjures a virtual pad the embedder didn't drive.
let mut pads: [Option<GamepadSnapshot>; MAX_PADS] = [None; MAX_PADS]; let mut pads: [Option<GamepadSnapshot>; MAX_PADS] = [None; MAX_PADS];
// Per-pad wrapping seq that PERSISTS across a pad's remove/re-add on the same index (the
// snapshot itself is cleared to `None` on removal). A removal takes `seq[idx] + 1` so it
// supersedes every prior snapshot; the re-added pad's first snapshot takes the next value
// after that, so the host's seq gate accepts it instead of rejecting a restarted-at-0 seq.
let mut seq: [u8; MAX_PADS] = [0; MAX_PADS];
// Re-sends of a removal still owed on refresh ticks (the removal rides the lossy datagram
// plane; a single lost one would silently strand a ghost pad on the host — the exact bug
// the removal fixes). Mirrors the host's rumble stop burst: a few time-spread re-sends,
// each with a fresh (higher) seq, and canceled the moment the pad is driven again.
const REMOVE_RESENDS: u8 = 2;
let mut remove_owed: [u8; MAX_PADS] = [0; MAX_PADS];
// Per-pad declared controller kind ([`GamepadArrival`]) + its owed re-sends: the host needs
// the kind before the pad's first frame to build a matching virtual device (mixed types), so
// like the removal it rides the lossy plane with a small time-spread re-send burst.
const ARRIVAL_RESENDS: u8 = 2;
let mut arrival: [Option<u8>; MAX_PADS] = [None; MAX_PADS];
let mut arrival_owed: [u8; MAX_PADS] = [0; MAX_PADS];
let mut refresh = tokio::time::interval(Duration::from_millis(100)); let mut refresh = tokio::time::interval(Duration::from_millis(100));
refresh.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); refresh.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop { loop {
@@ -1584,24 +1601,89 @@ async fn worker_main(args: WorkerArgs) {
&& matches!(ev.kind, InputKind::GamepadButton | InputKind::GamepadAxis) && matches!(ev.kind, InputKind::GamepadButton | InputKind::GamepadAxis)
&& idx < MAX_PADS && idx < MAX_PADS
{ {
// The pad is being driven — cancel any owed removal (a re-plug on this
// index; its fresh snapshot seq already supersedes the removal's).
remove_owed[idx] = 0;
let snap = pads[idx].get_or_insert(GamepadSnapshot { let snap = pads[idx].get_or_insert(GamepadSnapshot {
pad: idx as u8, pad: idx as u8,
..Default::default() ..Default::default()
}); });
// Unknown axis ids don't send (the host's legacy fold drops them too). // Unknown axis ids don't send (the host's legacy fold drops them too).
if snap.fold(&ev) { if snap.fold(&ev) {
snap.seq = snap.seq.wrapping_add(1); seq[idx] = seq[idx].wrapping_add(1);
snap.seq = seq[idx];
let _ = input_conn let _ = input_conn
.send_datagram(snap.to_event().encode().to_vec().into()); .send_datagram(snap.to_event().encode().to_vec().into());
} }
continue; continue;
} }
if gamepad_snapshots && ev.kind == InputKind::GamepadRemove && idx < MAX_PADS {
// Stop refreshing the pad and forward a seq-stamped removal (in the shared
// seq space) so the host tears its virtual device down and no reordered
// snapshot can resurrect it; arm the re-send burst against datagram loss.
// Drop any owed kind declaration too — a re-plug on this index sends its own.
pads[idx] = None;
arrival[idx] = None;
arrival_owed[idx] = 0;
seq[idx] = seq[idx].wrapping_add(1);
remove_owed[idx] = REMOVE_RESENDS;
let rem = crate::input::InputEvent {
flags: crate::input::encode_gamepad_remove(idx as u8, seq[idx]),
..ev
};
let _ = input_conn.send_datagram(rem.encode().to_vec().into());
continue;
}
if gamepad_snapshots && ev.kind == InputKind::GamepadArrival && idx < MAX_PADS {
// Remember the declared kind (`code`) and forward it, arming a re-send burst
// so the host learns it before the pad's first frame even under loss.
arrival[idx] = Some(ev.code as u8);
arrival_owed[idx] = ARRIVAL_RESENDS;
let _ = input_conn.send_datagram(ev.encode().to_vec().into());
continue;
}
let _ = input_conn.send_datagram(ev.encode().to_vec().into()); let _ = input_conn.send_datagram(ev.encode().to_vec().into());
} }
_ = refresh.tick() => { _ = refresh.tick() => {
for snap in pads.iter_mut().flatten() { for idx in 0..MAX_PADS {
snap.seq = snap.seq.wrapping_add(1); // Re-send an owed kind declaration (independent of whether the pad has state
// yet — it may be idle-but-connected). Idempotent on the host.
if arrival_owed[idx] > 0 {
if let Some(kind) = arrival[idx] {
arrival_owed[idx] -= 1;
let arr = crate::input::InputEvent {
kind: InputKind::GamepadArrival,
_pad: [0; 3],
code: kind as u32,
x: 0,
y: 0,
flags: idx as u32,
};
let _ = input_conn.send_datagram(arr.encode().to_vec().into());
} else {
arrival_owed[idx] = 0;
}
}
if let Some(snap) = pads[idx].as_mut() {
seq[idx] = seq[idx].wrapping_add(1);
snap.seq = seq[idx];
let _ = input_conn.send_datagram(snap.to_event().encode().to_vec().into()); let _ = input_conn.send_datagram(snap.to_event().encode().to_vec().into());
} else if remove_owed[idx] > 0 {
// Idempotent removal re-send with a fresh seq (the host drops it as a
// no-op once the pad is already gone, but a re-plug's later snapshot
// still wins by seq).
remove_owed[idx] -= 1;
seq[idx] = seq[idx].wrapping_add(1);
let rem = crate::input::InputEvent {
kind: InputKind::GamepadRemove,
_pad: [0; 3],
code: 0,
x: 0,
y: 0,
flags: crate::input::encode_gamepad_remove(idx as u8, seq[idx]),
};
let _ = input_conn.send_datagram(rem.encode().to_vec().into());
}
} }
} }
} }
+55 -2
View File
@@ -51,6 +51,39 @@ pub enum InputKind {
/// [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); older hosts keep /// [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); older hosts keep
/// receiving the per-transition events. /// receiving the per-transition events.
GamepadState = 12, GamepadState = 12,
/// A pad was unplugged client-side (the native plane's answer to GameStream's
/// `activeGamepadMask`, which the per-transition/snapshot planes otherwise lack — see
/// [`encode_gamepad_remove`]). `flags` packs `seq << 24 | pad`: the low byte is the pad
/// index, the high byte a per-pad wrapping seq sharing the [`GamepadSnapshot`] sequence
/// space. The host clears the pad's `active_mask` bit so its virtual device is torn down,
/// seq-gated against snapshots so one the network reordered past the removal can't resurrect
/// the pad, and the shared seq space keeps the same index reusable by a later re-plug. Sent
/// only to a host that advertised [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE);
/// an older host ignores the unknown tag (the pad then lingers until session end — the
/// pre-existing behaviour).
GamepadRemove = 13,
/// Declares which controller KIND a pad presents so a session can MIX types (pad 0 a
/// DualSense, pad 1 an Xbox pad). `code` = the [`GamepadPref`](crate::config::GamepadPref)
/// wire byte, `flags` = pad index. Sent when the client opens a pad slot — before that pad's
/// first input — and re-sent a few times against datagram loss (like [`GamepadRemove`]). The
/// host resolves the kind to a buildable backend and routes that pad's virtual device to it; a
/// pad the client never declares (an older client, or a fully-lost declaration) falls back to
/// the session-default kind from the handshake. Idempotent (no seq): re-declaring the same kind
/// is a no-op. Meaningful only to a host that advertised
/// [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); an older host ignores the
/// unknown tag (every pad then uses the session-default kind — the pre-existing behaviour).
GamepadArrival = 14,
}
/// Pack a [`InputKind::GamepadRemove`] `flags` word (`seq << 24 | pad`) — the same low-byte-pad /
/// high-byte-seq layout as [`GamepadSnapshot::to_event`], so a removal seq-gates against snapshots.
pub fn encode_gamepad_remove(pad: u8, seq: u8) -> u32 {
((seq as u32) << 24) | (pad as u32)
}
/// Unpack a [`InputKind::GamepadRemove`] `flags` word into `(pad, seq)`.
pub fn decode_gamepad_remove(flags: u32) -> (u8, u8) {
(flags as u8, (flags >> 24) as u8)
} }
/// The gamepad wire contract for [`InputKind::GamepadButton`]/[`InputKind::GamepadAxis`]. /// The gamepad wire contract for [`InputKind::GamepadButton`]/[`InputKind::GamepadAxis`].
@@ -123,6 +156,8 @@ impl InputKind {
10 => TouchMove, 10 => TouchMove,
11 => TouchUp, 11 => TouchUp,
12 => GamepadState, 12 => GamepadState,
13 => GamepadRemove,
14 => GamepadArrival,
_ => return None, _ => return None,
}) })
} }
@@ -321,8 +356,26 @@ mod tests {
}; };
assert_eq!(InputEvent::decode(&e.encode()), Some(e)); assert_eq!(InputEvent::decode(&e.encode()), Some(e));
} }
// 13 (one past GamepadState) is not a valid kind. // GamepadRemove + GamepadArrival are valid kinds; 15 (one past them) is not.
assert_eq!(InputKind::from_u8(13), None); assert_eq!(InputKind::from_u8(13), Some(InputKind::GamepadRemove));
assert_eq!(InputKind::from_u8(14), Some(InputKind::GamepadArrival));
assert_eq!(InputKind::from_u8(15), None);
}
#[test]
fn gamepad_remove_flags_roundtrip() {
for (pad, seq) in [(0u8, 0u8), (3, 200), (15, 255), (7, 1)] {
let flags = encode_gamepad_remove(pad, seq);
assert_eq!(decode_gamepad_remove(flags), (pad, seq));
}
// Layout matches the snapshot's pad/seq packing (low byte pad, high byte seq).
let snap = GamepadSnapshot {
pad: 9,
seq: 123,
..Default::default()
};
let (pad, seq) = decode_gamepad_remove(snap.to_event().flags);
assert_eq!((pad, seq), (9, 123));
} }
#[test] #[test]
+5 -1
View File
@@ -38,6 +38,7 @@ pub mod input;
pub mod packet; pub mod packet;
#[cfg(feature = "quic")] #[cfg(feature = "quic")]
pub mod quic; pub mod quic;
pub mod reanchor;
pub mod session; pub mod session;
pub mod stats; pub mod stats;
pub mod transport; pub mod transport;
@@ -61,7 +62,10 @@ pub use stats::Stats;
/// TTL of a v2 envelope; `punktfunk_connection_next_rumble` is unchanged and drops it). Additive — /// TTL of a v2 envelope; `punktfunk_connection_next_rumble` is unchanged and drops it). Additive —
/// the wire is backward-compatible (the envelope is a length-tolerant tail on 0xCA), so /// the wire is backward-compatible (the envelope is a length-tolerant tail on 0xCA), so
/// [`WIRE_VERSION`] is unchanged. /// [`WIRE_VERSION`] is unchanged.
pub const ABI_VERSION: u32 = 5; /// v6: added the `punktfunk_reanchor_gate_*` surface (post-loss freeze-until-reanchor gate for the
/// Swift client; Rust embedders use [`reanchor::ReanchorGate`] directly). Additive, client-local —
/// no wire change, so [`WIRE_VERSION`] is unchanged.
pub const ABI_VERSION: u32 = 6;
/// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check. /// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
/// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface** /// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
+473
View File
@@ -0,0 +1,473 @@
//! Post-loss display freeze — the shared "freeze-until-reanchor" gate.
//!
//! After an unrecoverable reference loss the hardware decoder does **not** error: it *conceals* the
//! reference-missing delta frames (on RADV, the DPB-and-output-COINCIDE path paints a gray plate with
//! the new frame's motion on top) and returns Ok. Displaying that is the "gray frames mid-stream"
//! artifact. Instead every client freezes on the last good picture — withholds the concealed frames
//! from its presenter, which keeps redrawing the held frame — and lifts the freeze ONLY on a proven
//! clean re-anchor: a real IDR, an LTR-RFI recovery anchor ([`USER_FLAG_RECOVERY_ANCHOR`]), or the
//! second intra-refresh recovery mark ([`USER_FLAG_RECOVERY_POINT`]) since the loss.
//!
//! This module owns that decision so every embedder shares ONE implementation instead of re-deriving
//! it (the Linux/Deck pump in `pf-client-core`, the Windows in-process pump, the Android decode loops,
//! and — over the C ABI — the Apple client). The state machine is time-driven but takes `now` as a
//! parameter so it is unit-testable without a clock; the C ABI wrappers supply `Instant::now()`.
//!
//! [`USER_FLAG_RECOVERY_POINT`]: crate::packet::USER_FLAG_RECOVERY_POINT
//! [`USER_FLAG_RECOVERY_ANCHOR`]: crate::packet::USER_FLAG_RECOVERY_ANCHOR
use crate::packet::{FLAG_SOF, USER_FLAG_RECOVERY_ANCHOR, USER_FLAG_RECOVERY_POINT};
use std::time::{Duration, Instant};
/// Consecutive no-output AUs that force a keyframe request. ~50 ms at 60 Hz — long enough not to fire
/// on a one-frame decoder hiccup, short enough that a lost initial IDR (or a mid-GOP join) unfreezes
/// almost immediately instead of never.
pub const NO_OUTPUT_KEYFRAME_STREAK: u32 = 3;
/// Longest the gate holds the last good frame waiting for a post-loss re-anchor keyframe before it
/// re-asks. After a reference loss the hardware decoder does not error — it conceals the
/// reference-missing deltas (on RADV, the DPB-and-output-COINCIDE path renders them as a gray plate
/// with the new frame's motion painted over it) and returns Ok, so displaying them is the "gray frames
/// mid-stream" artifact. We instead freeze on the last good picture until a fresh IDR re-anchors decode
/// — the behaviour NVIDIA already shows (its DISTINCT output image + different concealment reads as a
/// brief freeze, not gray). This cap only bounds the freeze when recovery genuinely stalls (host
/// ignores the request, or an RFI recovery that never emits a keyframe): the freeze is NEVER lifted to
/// the concealed picture — the deadline re-asks for a keyframe and keeps holding, so a glitch can never
/// become a permanent freeze while a clean re-anchor is what un-freezes. A recovery IDR round-trips well
/// under this on any live link.
pub const REANCHOR_FREEZE_MAX: Duration = Duration::from_millis(500);
/// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
/// latest loss before the gate lifts its freeze on an IDR-free stream. TWO, not one: with a continuous
/// rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a loss is only
/// partially healed — stripes swept BEFORE the loss still reference the lost frame — and lifting there
/// would flash a partially-stale picture. The SECOND boundary guarantees a full wave swept entirely
/// after the loss, so the picture is clean. This stays correct under repeated loss because every fresh
/// arm resets the count. The cost is up to ~2 wave periods of holding the last good frame — the
/// deliberate "hold longer, never show garbage" trade.
///
/// [`USER_FLAG_RECOVERY_POINT`]: crate::packet::USER_FLAG_RECOVERY_POINT
pub const REANCHOR_MARKS_TO_LIFT: u32 = 2;
/// Backstop patience while a host intra-refresh heal is visibly in progress. Each recovery mark pushes
/// the freeze deadline out by this much, so a live mark stream (the host actively healing via its wave)
/// keeps the gate patiently holding the last good frame instead of tripping the IDR floor mid-heal.
/// Must exceed the inter-mark interval (one wave period, ~0.5 s) with margin; if the marks STOP (heal
/// stalled, or the host isn't running intra-refresh) the deadline lapses and the normal recovery-IDR
/// floor fires, so a real stall still recovers.
pub const RECOVERY_MARK_PATIENCE: Duration = Duration::from_millis(1500);
/// Frames skipped when `got` arrives while `expected` was the next index, or `None` if `got` is
/// contiguous (`== expected`) or a straggler we have already passed. Frame indices are u32 counters
/// that wrap, so the "ahead" test is a wrapping subtraction split at the half-space: a small positive
/// delta is a forward gap (missing frames whose dependents will decode against absent references); a
/// delta in the top half is an index behind us.
pub fn index_gap(expected: u32, got: u32) -> Option<u32> {
let ahead = got.wrapping_sub(expected);
(ahead != 0 && ahead < u32::MAX / 2).then_some(ahead)
}
/// Fold one decoded frame into the re-anchor state and decide whether it lifts the post-loss freeze.
///
/// `is_keyframe` — a real IDR (always a clean re-anchor). `has_anchor` — this AU carried
/// [`USER_FLAG_RECOVERY_ANCHOR`](crate::packet::USER_FLAG_RECOVERY_ANCHOR), the host's definitive
/// single-frame re-anchor from an LTR-RFI recovery (a clean P-frame coded against a known-good
/// reference), so it lifts on the FIRST occurrence exactly like an IDR — no two-mark wait. `has_mark` —
/// this AU carried [`USER_FLAG_RECOVERY_POINT`](crate::packet::USER_FLAG_RECOVERY_POINT), a
/// host-signalled intra-refresh wave boundary (only *half* a re-anchor). `marks` — recovery marks seen
/// since the latest loss.
///
/// Returns `(lift, new_marks)`: `lift` clears the freeze; `new_marks` is the running count (reset to 0
/// on a lift). The two-mark rule ([`REANCHOR_MARKS_TO_LIFT`]) lives here so it is unit-tested
/// independent of the pump's channel/decoder plumbing — the first wave boundary after a loss is only
/// partially healed, so a single mark must NOT lift. An anchor (or IDR) is a *whole* re-anchor and
/// lifts immediately.
fn reanchor_after_frame(
is_keyframe: bool,
has_anchor: bool,
has_mark: bool,
marks: u32,
) -> (bool, u32) {
let marks = if has_mark {
marks.saturating_add(1)
} else {
marks
};
if is_keyframe || has_anchor || marks >= REANCHOR_MARKS_TO_LIFT {
(true, 0)
} else {
(false, marks)
}
}
/// Whether a decoded frame should be shown or withheld while the gate is (or isn't) frozen.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum GateVerdict {
/// Present this frame — the gate is not frozen, or this frame is the clean re-anchor that lifts it.
Present,
/// Withhold this frame — it is a post-loss concealment; the presenter keeps the last good picture.
Hold,
}
/// The shared post-loss freeze state machine. A client feeds it three kinds of event — an *arm* (a
/// loss was detected: a frame-index gap, a dropped-count climb, or a decoder wedge/demotion), each
/// *decoded frame* ([`on_decoded`](Self::on_decoded), which decides present-vs-hold and interprets the
/// re-anchor wire flags), and each *no-output* AU ([`on_no_output`](Self::on_no_output)) — plus a
/// periodic [`poll`](Self::poll) that folds the dropped counter and fires the overdue backstop.
///
/// The gate emits *intents* only: [`on_no_output`](Self::on_no_output) and [`poll`](Self::poll) return
/// `true` when the client should ask the host for a keyframe. The client routes that through its own
/// ~100 ms request throttle (and the precise RFI-vs-keyframe range decision stays in the loss-range
/// tracker behind [`crate::client::NativeClient::note_frame_index`]) — the gate never touches the wire.
#[derive(Debug, Clone)]
pub struct ReanchorGate {
/// Frozen on the last good frame, withholding the decoder's concealed output until a clean
/// re-anchor. Armed by any loss signal; cleared only by [`on_decoded`](Self::on_decoded) lifting.
awaiting: bool,
/// Host intra-refresh recovery marks seen since the latest arm (see [`REANCHOR_MARKS_TO_LIFT`]).
/// Reset to 0 whenever the freeze is (re-)armed, so a fresh loss always waits out two fresh marks.
marks: u32,
/// When the freeze becomes overdue and [`poll`](Self::poll) re-asks for a keyframe (holding, never
/// resuming to the concealed picture). `None` when not frozen.
deadline: Option<Instant>,
/// Consecutive received AUs that produced no decoded frame — a decoder wedged on missing references
/// with no reassembler drop to trigger recovery. A short streak forces a fresh IDR.
no_output_streak: u32,
/// The last `frames_dropped` value [`poll`](Self::poll) observed; a climb means the reassembler
/// declared an AU unrecoverable and the following deltas will conceal, so arm.
last_dropped: u64,
}
impl ReanchorGate {
/// Seed the gate with the session's current `frames_dropped` so the first [`poll`](Self::poll)
/// doesn't read the baseline as a loss.
pub fn new(frames_dropped: u64) -> Self {
ReanchorGate {
awaiting: false,
marks: 0,
deadline: None,
no_output_streak: 0,
last_dropped: frames_dropped,
}
}
/// Arm the freeze: a loss was detected (a frame-index gap, a dropped-count climb, or a decoder
/// wedge/demotion). Zeroes the mark count so a fresh loss waits out two fresh recovery marks, and
/// (re-)sets the backstop deadline. Idempotent while already frozen (re-arming just re-zeroes the
/// marks and pushes the deadline — the correct behaviour when a second loss lands mid-freeze).
pub fn arm(&mut self, now: Instant) {
self.awaiting = true;
self.marks = 0;
self.deadline = Some(now + REANCHOR_FREEZE_MAX);
}
/// Fold one decoded frame and decide whether to present or withhold it.
///
/// `wire_flags` is the AU's `user_flags` word ([`crate::session::Frame::flags`] /
/// `PunktfunkFrame.flags`); the gate reads [`FLAG_SOF`](crate::packet::FLAG_SOF) (the host sets it
/// only on IDR AUs — the codec-agnostic keyframe signal the platform decoders don't expose),
/// [`USER_FLAG_RECOVERY_ANCHOR`] and [`USER_FLAG_RECOVERY_POINT`]. `decoder_keyframe` is an optional
/// belt from decoders that flag IDRs themselves (libavcodec's `AV_FRAME_FLAG_KEY` on Linux/Windows);
/// pass `false` where the decoder doesn't (Android MediaCodec, Apple VideoToolbox) and rely on the
/// wire `FLAG_SOF`.
///
/// A decoded frame always clears the no-output streak. When frozen, a live mark stream pushes the
/// backstop out ([`RECOVERY_MARK_PATIENCE`]) so a healing wave isn't pre-empted by a mid-heal IDR.
///
/// [`USER_FLAG_RECOVERY_ANCHOR`]: crate::packet::USER_FLAG_RECOVERY_ANCHOR
/// [`USER_FLAG_RECOVERY_POINT`]: crate::packet::USER_FLAG_RECOVERY_POINT
pub fn on_decoded(
&mut self,
wire_flags: u32,
decoder_keyframe: bool,
now: Instant,
) -> GateVerdict {
self.no_output_streak = 0;
let is_keyframe = decoder_keyframe || (wire_flags & FLAG_SOF as u32 != 0);
let has_anchor = wire_flags & USER_FLAG_RECOVERY_ANCHOR != 0;
let has_mark = wire_flags & USER_FLAG_RECOVERY_POINT != 0;
if has_mark && self.awaiting {
self.deadline = Some(now + RECOVERY_MARK_PATIENCE);
}
let (lift, marks) = reanchor_after_frame(is_keyframe, has_anchor, has_mark, self.marks);
self.marks = marks;
if lift {
self.awaiting = false;
self.deadline = None;
}
if self.awaiting {
GateVerdict::Hold
} else {
GateVerdict::Present
}
}
/// A received AU produced no decoded frame (decode error, or the decoder swallowed a
/// reference-missing delta). Returns `true` when the streak has tripped and the client should
/// (throttled) request a keyframe — arming the freeze at the same time, since the stream is broken
/// regardless of whether the throttle lets the request through this iteration.
pub fn on_no_output(&mut self, now: Instant) -> bool {
self.no_output_streak += 1;
if self.no_output_streak >= NO_OUTPUT_KEYFRAME_STREAK {
self.arm(now);
self.no_output_streak = 0;
true
} else {
false
}
}
/// Periodic fold of the session's `frames_dropped` counter plus the overdue backstop. Returns
/// `true` when the client should (throttled) request a keyframe: either the drop count climbed (a
/// fresh unrecoverable loss — arm the freeze) or the freeze has held a full [`REANCHOR_FREEZE_MAX`]
/// window with no re-anchor (re-ask and keep holding — NEVER resume to the concealed picture; a
/// genuinely dead stream is the QUIC idle-timeout watchdog's job, not the gate's).
pub fn poll(&mut self, frames_dropped: u64, now: Instant) -> bool {
let mut want_keyframe = false;
if frames_dropped > self.last_dropped {
self.last_dropped = frames_dropped;
self.arm(now);
want_keyframe = true;
}
if self.awaiting && self.deadline.is_some_and(|d| now >= d) {
self.deadline = Some(now + REANCHOR_FREEZE_MAX);
want_keyframe = true;
}
want_keyframe
}
/// Whether the gate is currently withholding concealed frames (frozen on the last good picture).
pub fn is_holding(&self) -> bool {
self.awaiting
}
}
#[cfg(test)]
mod tests {
use super::*;
// Simulate the gate's re-anchor state across a sequence of decoded frames: each `(is_keyframe,
// has_mark)` pair is folded through `reanchor_after_frame`, returning the frame index (0-based) at
// which the freeze first lifts, or `None` if it never does. A reset to 0 models a fresh loss
// re-arming the freeze (the gate zeroes the count at every arm site).
fn lift_at(frames: &[(bool, bool)]) -> Option<usize> {
let mut marks = 0u32;
for (i, &(is_kf, has_mark)) in frames.iter().enumerate() {
// The intra-refresh-mark model never carries an LTR-RFI anchor (that path is exercised by
// `an_rfi_anchor_lifts_immediately`), so `has_anchor` is always false here.
let (lift, m) = reanchor_after_frame(is_kf, false, has_mark, marks);
marks = m;
if lift {
return Some(i);
}
}
None
}
#[test]
fn a_single_recovery_mark_does_not_lift() {
// The first wave boundary after a loss is only half-healed — one mark must hold the freeze.
assert_eq!(REANCHOR_MARKS_TO_LIFT, 2);
assert_eq!(lift_at(&[(false, true)]), None);
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false)]),
None
);
}
#[test]
fn the_second_recovery_mark_lifts() {
// Two marks = a full wave swept after the loss → clean re-anchor.
assert_eq!(lift_at(&[(false, true), (false, true)]), Some(1));
assert_eq!(
lift_at(&[(false, false), (false, true), (false, false), (false, true)]),
Some(3)
);
}
#[test]
fn a_real_keyframe_lifts_immediately() {
// An IDR is always a clean anchor — no marks needed.
assert_eq!(lift_at(&[(true, false)]), Some(0));
assert_eq!(lift_at(&[(false, true), (true, false)]), Some(1));
}
#[test]
fn a_fresh_gap_resets_the_mark_count() {
// The gate zeroes `marks` at each arm site, so one mark before a new gap plus one after must
// NOT lift — the model resets the running count to imitate that.
let mut marks = 0u32;
let (_, m) = reanchor_after_frame(false, false, true, marks); // mark #1 (pre-gap)
marks = m;
assert_eq!(marks, 1);
marks = 0; // a new gap re-arms the freeze → count reset
let (lift, m) = reanchor_after_frame(false, false, true, marks); // first mark of the new wave
assert!(!lift, "a single post-gap mark must not lift");
assert_eq!(m, 1);
}
#[test]
fn an_rfi_anchor_lifts_immediately() {
// An LTR-RFI recovery anchor is a WHOLE re-anchor (a clean P-frame off a known-good reference),
// so — like an IDR — it lifts on the FIRST occurrence, no two-mark wait.
let (lift, marks) = reanchor_after_frame(false, true, false, 0);
assert!(lift, "an RFI anchor must lift the freeze immediately");
assert_eq!(marks, 0, "a lift resets the running mark count");
// Even with zero prior marks and no keyframe, the anchor alone is sufficient.
let (lift, _) = reanchor_after_frame(false, true, true, 1);
assert!(lift, "an anchor lifts regardless of the pending mark count");
}
#[test]
fn contiguous_indices_are_not_a_gap() {
assert_eq!(index_gap(5, 5), None);
assert_eq!(index_gap(0, 0), None);
}
#[test]
fn a_forward_jump_reports_the_skip_count() {
assert_eq!(index_gap(5, 6), Some(1)); // one frame missing
assert_eq!(index_gap(5, 9), Some(4));
}
#[test]
fn a_straggler_behind_us_is_not_a_gap() {
// The reassembler emitted a newer frame first; the late one must not re-arm.
assert_eq!(index_gap(9, 5), None);
assert_eq!(index_gap(1, 0), None);
}
#[test]
fn the_index_counter_wraps_cleanly() {
// last frame = u32::MAX, so the next expected wraps to 0.
assert_eq!(index_gap(0, 0), None);
// waiting on u32::MAX, frame 0 arrived → MAX was skipped.
assert_eq!(index_gap(u32::MAX, 0), Some(1));
assert_eq!(index_gap(u32::MAX, 2), Some(3));
// an old frame arriving just after the wrap is still a straggler.
assert_eq!(index_gap(0, u32::MAX), None);
}
// ---- gate-level sequence tests (the whole behavioural contract) ----
const SOF: u32 = FLAG_SOF as u32; // IDR wire flag
const ANCHOR: u32 = USER_FLAG_RECOVERY_ANCHOR;
const POINT: u32 = USER_FLAG_RECOVERY_POINT;
fn t0() -> Instant {
Instant::now()
}
#[test]
fn a_clean_link_never_holds() {
// Disarmed gate presents every frame, keyframe or not, and never asks for anything.
let mut g = ReanchorGate::new(0);
let now = t0();
assert_eq!(g.on_decoded(0, false, now), GateVerdict::Present);
assert_eq!(g.on_decoded(SOF, true, now), GateVerdict::Present);
assert!(!g.is_holding());
assert!(!g.poll(0, now));
}
#[test]
fn a_gap_holds_until_the_wire_keyframe_lifts() {
// Android/Apple path: no decoder keyframe flag, lift comes from the wire FLAG_SOF alone.
let mut g = ReanchorGate::new(0);
let now = t0();
g.arm(now); // frame-index gap
assert!(g.is_holding());
assert_eq!(g.on_decoded(0, false, now), GateVerdict::Hold); // concealed delta withheld
assert_eq!(g.on_decoded(0, false, now), GateVerdict::Hold);
assert_eq!(g.on_decoded(SOF, false, now), GateVerdict::Present); // IDR re-anchors
assert!(!g.is_holding());
assert_eq!(g.on_decoded(0, false, now), GateVerdict::Present); // stays presenting
}
#[test]
fn a_gap_lifts_on_the_first_rfi_anchor() {
let mut g = ReanchorGate::new(0);
let now = t0();
g.arm(now);
assert_eq!(g.on_decoded(0, false, now), GateVerdict::Hold);
assert_eq!(g.on_decoded(ANCHOR, false, now), GateVerdict::Present);
assert!(!g.is_holding());
}
#[test]
fn a_gap_lifts_on_the_second_recovery_mark() {
let mut g = ReanchorGate::new(0);
let now = t0();
g.arm(now);
assert_eq!(g.on_decoded(POINT, false, now), GateVerdict::Hold); // first boundary: half-healed
assert_eq!(g.on_decoded(0, false, now), GateVerdict::Hold);
assert_eq!(g.on_decoded(POINT, false, now), GateVerdict::Present); // second: clean
}
#[test]
fn a_second_gap_mid_freeze_resets_the_marks() {
let mut g = ReanchorGate::new(0);
let now = t0();
g.arm(now);
assert_eq!(g.on_decoded(POINT, false, now), GateVerdict::Hold); // mark #1
g.arm(now); // a fresh loss re-arms → mark count zeroed
assert_eq!(g.on_decoded(POINT, false, now), GateVerdict::Hold); // this is mark #1 of the new wave
assert_eq!(g.on_decoded(POINT, false, now), GateVerdict::Present); // #2 lifts
}
#[test]
fn the_dropped_climb_arms_and_asks() {
let mut g = ReanchorGate::new(5);
let now = t0();
assert!(!g.poll(5, now), "no climb → no ask"); // baseline
assert!(g.poll(6, now), "a climb asks for a keyframe");
assert!(g.is_holding(), "and arms the freeze");
assert!(
!g.poll(6, now),
"same value → no repeat ask from the drop path"
);
}
#[test]
fn the_no_output_streak_trips_at_three() {
let mut g = ReanchorGate::new(0);
let now = t0();
assert!(!g.on_no_output(now));
assert!(!g.on_no_output(now));
assert!(g.on_no_output(now), "third no-output trips the streak");
assert!(g.is_holding());
// A decoded frame resets the streak.
g.on_decoded(SOF, false, now); // lifts + resets streak
assert!(!g.on_no_output(now));
assert!(!g.on_no_output(now));
assert!(g.on_no_output(now));
}
#[test]
fn an_overdue_freeze_re_asks_but_keeps_holding() {
let mut g = ReanchorGate::new(0);
let start = t0();
g.arm(start);
// Before the deadline: holding, no re-ask.
assert!(!g.poll(0, start));
assert!(g.is_holding());
// Past REANCHOR_FREEZE_MAX with no re-anchor: re-ask, still holding.
let later = start + REANCHOR_FREEZE_MAX + Duration::from_millis(1);
assert!(g.poll(0, later), "overdue freeze re-asks for a keyframe");
assert!(g.is_holding(), "but never resumes to the concealed picture");
}
#[test]
fn a_live_mark_stream_pushes_the_deadline_out() {
// A healing wave (marks arriving) must not be pre-empted by the overdue IDR floor.
let mut g = ReanchorGate::new(0);
let start = t0();
g.arm(start);
// A mark past the original freeze deadline pushes it out by RECOVERY_MARK_PATIENCE.
let t = start + REANCHOR_FREEZE_MAX + Duration::from_millis(10);
// mark #1 pushes the deadline out; at a time that WOULD have been overdue on the ORIGINAL
// deadline, poll does not re-ask.
assert_eq!(g.on_decoded(POINT, false, t), GateVerdict::Hold);
assert!(!g.poll(0, t + Duration::from_millis(1)));
assert!(g.is_holding());
}
}
+6
View File
@@ -259,6 +259,12 @@ nvenc = ["dep:nvidia-video-codec-sdk"]
# so the LGPL build suffices and keeps the bundled DLLs LGPL, not GPL) at build time and bundles the # so the LGPL build suffices and keeps the bundled DLLs LGPL, not GPL) at build time and bundles the
# FFmpeg DLLs at runtime. Build the all-vendor GPU host with `--features nvenc,amf-qsv`. # FFmpeg DLLs at runtime. Build the all-vendor GPU host with `--features nvenc,amf-qsv`.
amf-qsv = ["dep:ffmpeg-next"] amf-qsv = ["dep:ffmpeg-next"]
# Raw Vulkan Video HEVC encode on Linux (AMD/Intel) — real reference-frame-invalidation loss
# recovery via explicit DPB reference slots (design/linux-vulkan-video-encode.md), the open-stack
# twin of the direct-NVENC path. OFF by default; pulls NO new dependency (reuses the `ash` Vulkan
# bindings already carried for the dmabuf zero-copy bridge). Runtime-gated further by
# PUNKTFUNK_VULKAN_ENCODE (opt-in for now). Build the AMD/Intel RFI host with `--features vulkan-encode`.
vulkan-encode = []
# Build-time icon/version-info embedding (build.rs; Windows dev/CI hosts only — Linux packaging # Build-time icon/version-info embedding (build.rs; Windows dev/CI hosts only — Linux packaging
# builds of this crate never execute the winresource block). # builds of this crate never execute the winresource block).
+103 -63
View File
@@ -373,7 +373,7 @@ pub fn open_video(
bit_depth: u8, bit_depth: u8,
chroma: ChromaFormat, chroma: ChromaFormat,
) -> Result<Box<dyn Encoder>> { ) -> Result<Box<dyn Encoder>> {
let inner = open_video_backend( let (inner, backend) = open_video_backend(
codec, codec,
format, format,
width, width,
@@ -385,10 +385,12 @@ pub fn open_video(
chroma, chroma,
)?; )?;
// Record what this session encodes on (the mgmt API's "currently used GPU"): the backend label // Record what this session encodes on (the mgmt API's "currently used GPU"): the backend label
// mirrors the dispatch `open_video_backend` just took, the GPU identity is the same selection // is reported by `open_video_backend` from the branch that ACTUALLY opened — not re-derived by
// the capturer was created on ([`crate::gpu::selected_gpu`]). Dropping the returned encoder // mirroring its dispatch, which went stale the moment a backend gained an internal fallback
// ends the record, so the live count is correct by construction. // (the default-on Vulkan Video path falls back to VAAPI on a failed open, and a dispatch
let backend = resolved_backend_label(cuda); // mirror would report "vaapi" for every Vulkan session or vice versa). The GPU identity is the
// same selection the capturer was created on ([`crate::gpu::selected_gpu`]). Dropping the
// returned encoder ends the record, so the live count is correct by construction.
let gpu = if backend == "software" { let gpu = if backend == "software" {
crate::gpu::ActiveGpu { crate::gpu::ActiveGpu {
id: String::new(), id: String::new(),
@@ -418,39 +420,6 @@ pub fn open_video(
})) }))
} }
/// The display label of the backend [`open_video_backend`] resolves — kept in lockstep with its
/// dispatch (`windows_resolved_backend` on Windows; the `PUNKTFUNK_ENCODER`/auto match on Linux).
#[cfg(target_os = "windows")]
fn resolved_backend_label(_cuda: bool) -> &'static str {
match windows_resolved_backend() {
WindowsBackend::Nvenc => "nvenc",
WindowsBackend::Amf => "amf",
WindowsBackend::Qsv => "qsv",
WindowsBackend::Software => "software",
}
}
#[cfg(target_os = "linux")]
fn resolved_backend_label(cuda: bool) -> &'static str {
match crate::config::config().encoder_pref.as_str() {
"nvenc" | "nvidia" | "cuda" => "nvenc",
"vaapi" | "amd" | "intel" => "vaapi",
"software" | "sw" | "openh264" => "software",
_ => {
if cuda || !linux_auto_is_vaapi() {
"nvenc"
} else {
"vaapi"
}
}
}
}
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
fn resolved_backend_label(_cuda: bool) -> &'static str {
"none"
}
/// Ties the [`crate::gpu`] live-session record to the encoder's lifetime; pure delegation /// Ties the [`crate::gpu`] live-session record to the encoder's lifetime; pure delegation
/// otherwise. /// otherwise.
struct TrackedEncoder { struct TrackedEncoder {
@@ -488,6 +457,10 @@ impl Encoder for TrackedEncoder {
} }
} }
/// Open the platform encoder backend. Returns the encoder together with the display label of the
/// branch that ACTUALLY opened (`nvenc`/`vaapi`/`vulkan`/`amf`/`qsv`/`software`) — the label feeds
/// the mgmt API's live-session record, and only the open site knows which internal fallback won
/// (e.g. Vulkan Video falling back to VAAPI).
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
fn open_video_backend( fn open_video_backend(
codec: Codec, codec: Codec,
@@ -499,7 +472,7 @@ fn open_video_backend(
cuda: bool, cuda: bool,
bit_depth: u8, bit_depth: u8,
chroma: ChromaFormat, chroma: ChromaFormat,
) -> Result<Box<dyn Encoder>> { ) -> Result<(Box<dyn Encoder>, &'static str)> {
validate_dimensions(codec, width, height)?; validate_dimensions(codec, width, height)?;
// Refresh/fps must be positive and sane: fps feeds the encoder time_base (`Rational(1, fps)`) // Refresh/fps must be positive and sane: fps feeds the encoder time_base (`Rational(1, fps)`)
// and the pts→ns conversion (`pts * 1e9 / fps`), so 0 builds a 1/0 rational / divides by zero. // and the pts→ns conversion (`pts * 1e9 / fps`), so 0 builds a 1/0 rational / divides by zero.
@@ -528,7 +501,30 @@ fn open_video_backend(
// Linux binary serves any GPU; `PUNKTFUNK_ENCODER` forces a specific backend (and surfaces // Linux binary serves any GPU; `PUNKTFUNK_ENCODER` forces a specific backend (and surfaces
// its errors crisply instead of silently trying the other). // its errors crisply instead of silently trying the other).
let pref = crate::config::config().encoder_pref.as_str(); let pref = crate::config::config().encoder_pref.as_str();
let open_vaapi = || -> Result<Box<dyn Encoder>> { // AMD/Intel opener. Default = libav VAAPI. With `--features vulkan-encode` +
// PUNKTFUNK_VULKAN_ENCODE, an HEVC session instead opens the raw Vulkan Video backend (real
// RFI loss recovery the VAAPI path can't express); a failed open falls back to VAAPI so the
// stream never dies over the new path. `format`/`bit_depth`/`chroma` only matter to VAAPI —
// the Vulkan backend imports the dmabuf and does its own 8-bit 4:2:0 CSC.
let open_amd_intel = || -> Result<(Box<dyn Encoder>, &'static str)> {
#[cfg(feature = "vulkan-encode")]
if matches!(codec, Codec::H265 | Codec::Av1) && vulkan_encode_enabled() {
match vulkan_video::VulkanVideoEncoder::open(codec, width, height, fps, bitrate_bps)
{
Ok(e) => {
tracing::info!(
codec = ?codec,
"Linux Vulkan Video encode (real RFI via DPB reference slots) — \
set PUNKTFUNK_VULKAN_ENCODE=0 for libav VAAPI"
);
return Ok((Box::new(e) as Box<dyn Encoder>, "vulkan"));
}
Err(e) => tracing::warn!(
error = %format!("{e:#}"),
"Vulkan Video encode open failed — falling back to libav VAAPI"
),
}
}
vaapi::VaapiEncoder::open( vaapi::VaapiEncoder::open(
codec, codec,
format, format,
@@ -539,10 +535,10 @@ fn open_video_backend(
bit_depth, bit_depth,
chroma, chroma,
) )
.map(|e| Box::new(e) as Box<dyn Encoder>) .map(|e| (Box::new(e) as Box<dyn Encoder>, "vaapi"))
}; };
match pref { let open_nvidia = || -> Result<(Box<dyn Encoder>, &'static str)> {
"nvenc" | "nvidia" | "cuda" => open_nvenc_probed( open_nvenc_probed(
codec, codec,
format, format,
width, width,
@@ -552,8 +548,32 @@ fn open_video_backend(
cuda, cuda,
bit_depth, bit_depth,
chroma, chroma,
), )
"vaapi" | "amd" | "intel" => open_vaapi(), .map(|e| (e, "nvenc"))
};
match pref {
"nvenc" | "nvidia" | "cuda" => open_nvidia(),
"vaapi" | "amd" | "intel" => open_amd_intel(),
// Force the raw Vulkan Video HEVC backend (real RFI). Needs `--features vulkan-encode`.
"vulkan" | "vulkan-video" => {
#[cfg(feature = "vulkan-encode")]
{
if !matches!(codec, Codec::H265 | Codec::Av1) {
anyhow::bail!(
"the Vulkan Video encoder supports HEVC + AV1; the session negotiated {codec:?}"
);
}
vulkan_video::VulkanVideoEncoder::open(codec, width, height, fps, bitrate_bps)
.map(|e| (Box::new(e) as Box<dyn Encoder>, "vulkan"))
}
#[cfg(not(feature = "vulkan-encode"))]
{
let _ = (format, bit_depth, chroma);
anyhow::bail!(
"PUNKTFUNK_ENCODER=vulkan requires a build with --features vulkan-encode"
)
}
}
// GPU-less software H.264 (openh264) — for a headless / GPU-lost box. Explicit-only: // GPU-less software H.264 (openh264) — for a headless / GPU-lost box. Explicit-only:
// `auto` never picks it (a box with `/dev/nvidiactl` present but a dead driver would // `auto` never picks it (a box with `/dev/nvidiactl` present but a dead driver would
// otherwise wrongly resolve to NVENC). Needs H.264 (openh264 emits only that) and a CPU // otherwise wrongly resolve to NVENC). Needs H.264 (openh264 emits only that) and a CPU
@@ -567,30 +587,20 @@ fn open_video_backend(
} }
let _ = (cuda, bit_depth); // software path is CPU + 8-bit only let _ = (cuda, bit_depth); // software path is CPU + 8-bit only
sw::OpenH264Encoder::open(format, width, height, fps, bitrate_bps) sw::OpenH264Encoder::open(format, width, height, fps, bitrate_bps)
.map(|e| Box::new(e) as Box<dyn Encoder>) .map(|e| (Box::new(e) as Box<dyn Encoder>, "software"))
} }
"auto" | "" => { "auto" | "" => {
// A CUDA frame can ONLY be consumed by NVENC. Otherwise the shared auto decision // A CUDA frame can ONLY be consumed by NVENC. Otherwise the shared auto decision
// (manual web-console GPU preference, else the NVIDIA-presence probe) picks the // (manual web-console GPU preference, else the NVIDIA-presence probe) picks the
// backend — see `linux_auto_is_vaapi`. // backend — see `linux_auto_is_vaapi`.
if cuda || !linux_auto_is_vaapi() { if cuda || !linux_auto_is_vaapi() {
open_nvenc_probed( open_nvidia()
codec,
format,
width,
height,
fps,
bitrate_bps,
cuda,
bit_depth,
chroma,
)
} else { } else {
open_vaapi() open_amd_intel()
} }
} }
other => anyhow::bail!( other => anyhow::bail!(
"unknown PUNKTFUNK_ENCODER={other:?} — use auto (default), nvenc, vaapi, or software" "unknown PUNKTFUNK_ENCODER={other:?} — use auto (default), nvenc, vaapi, vulkan, or software"
), ),
} }
} }
@@ -637,7 +647,7 @@ fn open_video_backend(
bit_depth, bit_depth,
chroma, chroma,
) )
.map(|e| Box::new(e) as Box<dyn Encoder>) .map(|e| (Box::new(e) as Box<dyn Encoder>, "nvenc"))
} }
#[cfg(not(feature = "nvenc"))] #[cfg(not(feature = "nvenc"))]
{ {
@@ -666,7 +676,7 @@ fn open_video_backend(
bit_depth, bit_depth,
chroma, chroma,
) )
.map(|e| Box::new(e) as Box<dyn Encoder>) .map(|e| (Box::new(e) as Box<dyn Encoder>, "amf"))
.map_err(|e| { .map_err(|e| {
e.context( e.context(
"native AMF encode failed to open (update the AMD driver / amfrt64.dll \ "native AMF encode failed to open (update the AMD driver / amfrt64.dll \
@@ -690,7 +700,7 @@ fn open_video_backend(
bit_depth, bit_depth,
chroma, chroma,
) )
.map(|e| Box::new(e) as Box<dyn Encoder>) .map(|e| (Box::new(e) as Box<dyn Encoder>, "qsv"))
} }
#[cfg(not(feature = "amf-qsv"))] #[cfg(not(feature = "amf-qsv"))]
{ {
@@ -717,7 +727,7 @@ fn open_video_backend(
fps, fps,
bitrate_bps.min(SW_BITRATE_CEIL), bitrate_bps.min(SW_BITRATE_CEIL),
) )
.map(|e| Box::new(e) as Box<dyn Encoder>) .map(|e| (Box::new(e) as Box<dyn Encoder>, "software"))
} }
} }
} }
@@ -830,6 +840,22 @@ fn nvenc_direct_enabled() -> bool {
.unwrap_or(true) .unwrap_or(true)
} }
/// Whether the raw Vulkan Video HEVC encode backend is active for AMD/Intel. **Default ON** —
/// on-glass validated 2026-07-12 on an AMD RADV 780M with a real Deck-class client: the pipelined
/// encoder ran a rock-solid 1080p@240 HEVC session and healed loss with clean P-frame recovery
/// anchors (never IDR) via explicit DPB reference slots — real reference-frame invalidation the
/// libavcodec VAAPI path can't express (design/linux-vulkan-video-encode.md). `PUNKTFUNK_VULKAN_ENCODE=0`
/// (also `false`/`no`/`off`) is the libav-VAAPI escape hatch. Only consulted with
/// `--features vulkan-encode`, and a failed open falls back to VAAPI, so an unsupported device
/// (e.g. a Mesa without h265 encode, or an untested Intel/ANV where the path misbehaves at open)
/// degrades gracefully to the old backend rather than breaking the stream.
#[cfg(all(target_os = "linux", feature = "vulkan-encode"))]
fn vulkan_encode_enabled() -> bool {
std::env::var("PUNKTFUNK_VULKAN_ENCODE")
.map(|v| !matches!(v.trim(), "0" | "false" | "no" | "off"))
.unwrap_or(true)
}
/// Cheap, side-effect-free NVIDIA-presence probe for the `auto` backend selector: the NVIDIA /// Cheap, side-effect-free NVIDIA-presence probe for the `auto` backend selector: the NVIDIA
/// kernel driver exposes these device nodes, AMD/Intel boxes have neither. Deliberately does NOT /// kernel driver exposes these device nodes, AMD/Intel boxes have neither. Deliberately does NOT
/// create a CUDA context (that would allocate GPU state on every host that merely *might* be /// create a CUDA context (that would allocate GPU state on every host that merely *might* be
@@ -1185,6 +1211,20 @@ mod sw;
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
#[path = "encode/linux/vaapi.rs"] #[path = "encode/linux/vaapi.rs"]
mod vaapi; mod vaapi;
// Raw Vulkan Video HEVC encode on Linux (AMD/Intel; design/linux-vulkan-video-encode.md) — real RFI
// via explicit DPB reference slots (the app owns the DPB), the open-stack twin of the direct-NVENC
// path. Does an on-GPU RGB→NV12 compute CSC since capture delivers packed-RGB dmabufs. Opt-in behind
// `PUNKTFUNK_VULKAN_ENCODE` until on-glass validated; needs `--features vulkan-encode`.
#[cfg(all(target_os = "linux", feature = "vulkan-encode"))]
#[path = "encode/linux/vulkan_video.rs"]
mod vulkan_video;
// Vendored `VK_KHR_video_encode_av1` bindings (host-only) — the AV1 encode structs our pinned
// `ash 0.38.0+1.3.281` predates (finalized Vulkan 1.3.290). Copied verbatim from ash-master's
// generated code rather than bumping `ash` (which breaks the SDL/Vulkan client). Consumed by
// `vulkan_video.rs` via `super::vk_av1_encode`.
#[cfg(all(target_os = "linux", feature = "vulkan-encode"))]
#[path = "encode/linux/vk_av1_encode.rs"]
mod vk_av1_encode;
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
@@ -0,0 +1,32 @@
#version 450
// RGB(A) -> NV12 (BT.709 limited range). One invocation per chroma sample = 2x2 luma block.
layout(local_size_x = 8, local_size_y = 8) in;
layout(binding = 0) uniform sampler2D rgb; // packed RGB input (sampled; BGRA import ok)
layout(binding = 1, r8) uniform writeonly image2D yImg; // full-res Y
layout(binding = 2, rg8) uniform writeonly image2D uvImg; // half-res UV (interleaved)
float lumaY(vec3 c) { return 16.0/255.0 + 0.1826*c.r + 0.6142*c.g + 0.0620*c.b; }
// Source may be SMALLER than the coded (16-aligned) Y plane — e.g. 1080 source vs 1088 coded. Clamp
// every fetch to the source edge so the alignment-padding rows duplicate the last real row instead
// of reading out of bounds (undefined → green garbage that shows if a client ignores the SPS
// conformance-window crop). `textureSize` gives the bound source's real extent.
void main() {
ivec2 sz = imageSize(yImg);
ivec2 rmax = textureSize(rgb, 0) - 1;
ivec2 uvc = ivec2(gl_GlobalInvocationID.xy);
ivec2 p = uvc * 2;
if (p.x >= sz.x || p.y >= sz.y) return;
vec3 c00 = texelFetch(rgb, min(p, rmax), 0).rgb;
vec3 c10 = texelFetch(rgb, min(p + ivec2(1, 0), rmax), 0).rgb;
vec3 c01 = texelFetch(rgb, min(p + ivec2(0, 1), rmax), 0).rgb;
vec3 c11 = texelFetch(rgb, min(p + ivec2(1, 1), rmax), 0).rgb;
imageStore(yImg, p, vec4(lumaY(c00), 0, 0, 1));
imageStore(yImg, p + ivec2(1, 0), vec4(lumaY(c10), 0, 0, 1));
imageStore(yImg, p + ivec2(0, 1), vec4(lumaY(c01), 0, 0, 1));
imageStore(yImg, p + ivec2(1, 1), vec4(lumaY(c11), 0, 0, 1));
vec3 a = (c00 + c10 + c01 + c11) * 0.25;
float U = 128.0/255.0 - 0.1006*a.r - 0.3386*a.g + 0.4392*a.b;
float V = 128.0/255.0 + 0.4392*a.r - 0.3989*a.g - 0.0403*a.b;
imageStore(uvImg, uvc, vec4(U, V, 0, 1));
}
Binary file not shown.
@@ -0,0 +1,500 @@
//! Vendored `VK_KHR_video_encode_av1` bindings — the AV1-encode structs, `StdVideoEncodeAV1*`
//! types and struct-type constants that our pinned `ash 0.38.0+1.3.281` does not ship (the
//! extension was finalized in Vulkan 1.3.290). Bumping `ash` to git-master (`+1.4.352`, which has
//! them) breaks the *client*: it drops the lifetime on `vk::AllocationCallbacks`, and `sdl3-sys`'s
//! `ash` feature still generates `AllocationCallbacks<'static>`, so the presenter's SDL/Vulkan
//! surface path won't compile. Rather than churn the client for a host-only need, we vendor just
//! the encode-side definitions here, **copied verbatim from ash-master's generated code** (so the
//! layouts are correct-by-construction) and chain them into ash's generic video-encode-queue calls
//! via raw `p_next`, exactly as the HEVC path already chains its rate-control struct.
//!
//! Everything *common* to AV1 (sequence header, tile/quant/loop-filter/CDEF/… sub-structs, the
//! `StdVideoAV1*` enums) is already present in 1.3.281's `ash::vk::native` — AV1 **decode** brought
//! it in — so we reuse those and vendor only the encode-specific pieces. Delete this module and
//! switch to `ash::vk::*` once `ash` publishes a 1.4.x release and `sdl3-sys` regenerates.
#![allow(non_snake_case, non_camel_case_types, dead_code)]
use ash::vk;
use ash::vk::native::{
StdVideoAV1FrameType, StdVideoAV1InterpolationFilter, StdVideoAV1Level, StdVideoAV1Profile,
StdVideoAV1SequenceHeader, StdVideoAV1TxMode,
};
use std::ffi::{c_void, CStr};
/// `VK_KHR_video_encode_av1` extension name — ash 0.38's `ash::khr::video_encode_av1` doesn't exist,
/// so we pass this raw to `enabled_extension_names`.
pub const EXTENSION_NAME: &CStr = c"VK_KHR_video_encode_av1";
// ---------- struct-type (VkStructureType) values — construct via `vk::StructureType::from_raw` ----------
pub const ST_CAPABILITIES: i32 = 1_000_513_000;
pub const ST_SESSION_PARAMETERS_CREATE_INFO: i32 = 1_000_513_001;
pub const ST_PICTURE_INFO: i32 = 1_000_513_002;
pub const ST_DPB_SLOT_INFO: i32 = 1_000_513_003;
pub const ST_PHYSICAL_DEVICE_FEATURES: i32 = 1_000_513_004;
pub const ST_PROFILE_INFO: i32 = 1_000_513_005;
pub const ST_RATE_CONTROL_INFO: i32 = 1_000_513_006;
pub const ST_RATE_CONTROL_LAYER_INFO: i32 = 1_000_513_007;
pub const ST_SESSION_CREATE_INFO: i32 = 1_000_513_009;
pub const ST_GOP_REMAINING_FRAME_INFO: i32 = 1_000_513_010;
/// `VK_VIDEO_CODEC_OPERATION_ENCODE_AV1_BIT_KHR` (bit 18).
pub const VIDEO_CODEC_OPERATION_ENCODE_AV1: u32 = 0x0004_0000;
/// `VK_MAX_VIDEO_AV1_REFERENCES_PER_FRAME_KHR` — LAST..ALTREF (the 7 inter reference names).
pub const MAX_VIDEO_AV1_REFERENCES_PER_FRAME: usize = 7;
/// `STD_VIDEO_AV1_PRIMARY_REF_NONE` — a frame that inherits no CDF/context from any reference
/// (the recovery-anchor lever: a clean P-frame independent of prior probability state).
pub const PRIMARY_REF_NONE: u8 = 7;
/// `VK_VIDEO_ENCODE_AV1_SUPERBLOCK_SIZE_128_BIT_KHR` (bit 1 of the superblock-size flags).
pub const SUPERBLOCK_SIZE_128: u32 = 0x2;
// `VkVideoEncodeAV1PredictionModeKHR`
pub const PREDICTION_MODE_INTRA_ONLY: i32 = 0;
pub const PREDICTION_MODE_SINGLE_REFERENCE: i32 = 1;
// `VkVideoEncodeAV1RateControlGroupKHR`
pub const RC_GROUP_INTRA: i32 = 0;
pub const RC_GROUP_PREDICTIVE: i32 = 1;
pub const RC_GROUP_BIPREDICTIVE: i32 = 2;
// AV1 reference names (index into `reference_name_slot_indices`, which is 0-based over LAST..ALTREF).
pub const REFERENCE_NAME_LAST_FRAME_IDX: usize = 0; // STD_VIDEO_AV1_REFERENCE_NAME_LAST_FRAME - 1
// ---------- bindgen bitfield helper (copied verbatim from ash-master native.rs) ----------
#[repr(C)]
#[derive(Debug, Default, Copy, Clone)]
pub struct __BindgenBitfieldUnit<Storage> {
storage: Storage,
}
impl<Storage> __BindgenBitfieldUnit<Storage> {
#[inline]
pub const fn new(storage: Storage) -> Self {
Self { storage }
}
}
impl<Storage> __BindgenBitfieldUnit<Storage>
where
Storage: AsRef<[u8]> + AsMut<[u8]>,
{
#[inline]
pub fn get_bit(&self, index: usize) -> bool {
let byte_index = index / 8;
let byte = self.storage.as_ref()[byte_index];
let bit_index = if cfg!(target_endian = "big") {
7 - (index % 8)
} else {
index % 8
};
byte & (1 << bit_index) == (1 << bit_index)
}
#[inline]
pub fn set_bit(&mut self, index: usize, val: bool) {
let byte_index = index / 8;
let byte = &mut self.storage.as_mut()[byte_index];
let bit_index = if cfg!(target_endian = "big") {
7 - (index % 8)
} else {
index % 8
};
let mask = 1 << bit_index;
if val {
*byte |= mask;
} else {
*byte &= !mask;
}
}
#[inline]
pub fn get(&self, bit_offset: usize, bit_width: u8) -> u64 {
let mut val = 0;
for i in 0..(bit_width as usize) {
if self.get_bit(i + bit_offset) {
let index = if cfg!(target_endian = "big") {
bit_width as usize - 1 - i
} else {
i
};
val |= 1 << index;
}
}
val
}
#[inline]
pub fn set(&mut self, bit_offset: usize, bit_width: u8, val: u64) {
for i in 0..(bit_width as usize) {
let mask = 1 << i;
let val_bit_is_set = val & mask == mask;
let index = if cfg!(target_endian = "big") {
bit_width as usize - 1 - i
} else {
i
};
self.set_bit(index + bit_offset, val_bit_is_set);
}
}
}
// ---------- Std encode structs (copied from ash-master native.rs; common Std types reused from ash) ----------
#[repr(C, align(4))]
#[derive(Debug, Copy, Clone)]
pub struct StdVideoEncodeAV1PictureInfoFlags {
pub _bitfield_align_1: [u8; 0],
pub _bitfield_1: __BindgenBitfieldUnit<[u8; 4usize]>,
}
impl StdVideoEncodeAV1PictureInfoFlags {
#[inline]
pub fn set_error_resilient_mode(&mut self, val: u32) {
self._bitfield_1.set(0, 1, val as u64)
}
#[inline]
pub fn set_disable_cdf_update(&mut self, val: u32) {
self._bitfield_1.set(1, 1, val as u64)
}
#[inline]
pub fn set_use_superres(&mut self, val: u32) {
self._bitfield_1.set(2, 1, val as u64)
}
#[inline]
pub fn set_render_and_frame_size_different(&mut self, val: u32) {
self._bitfield_1.set(3, 1, val as u64)
}
#[inline]
pub fn set_allow_screen_content_tools(&mut self, val: u32) {
self._bitfield_1.set(4, 1, val as u64)
}
#[inline]
pub fn set_is_filter_switchable(&mut self, val: u32) {
self._bitfield_1.set(5, 1, val as u64)
}
#[inline]
pub fn set_force_integer_mv(&mut self, val: u32) {
self._bitfield_1.set(6, 1, val as u64)
}
#[inline]
pub fn set_frame_size_override_flag(&mut self, val: u32) {
self._bitfield_1.set(7, 1, val as u64)
}
#[inline]
pub fn set_buffer_removal_time_present_flag(&mut self, val: u32) {
self._bitfield_1.set(8, 1, val as u64)
}
#[inline]
pub fn set_allow_intrabc(&mut self, val: u32) {
self._bitfield_1.set(9, 1, val as u64)
}
#[inline]
pub fn set_frame_refs_short_signaling(&mut self, val: u32) {
self._bitfield_1.set(10, 1, val as u64)
}
#[inline]
pub fn set_allow_high_precision_mv(&mut self, val: u32) {
self._bitfield_1.set(11, 1, val as u64)
}
#[inline]
pub fn set_is_motion_mode_switchable(&mut self, val: u32) {
self._bitfield_1.set(12, 1, val as u64)
}
#[inline]
pub fn set_use_ref_frame_mvs(&mut self, val: u32) {
self._bitfield_1.set(13, 1, val as u64)
}
#[inline]
pub fn set_disable_frame_end_update_cdf(&mut self, val: u32) {
self._bitfield_1.set(14, 1, val as u64)
}
#[inline]
pub fn set_allow_warped_motion(&mut self, val: u32) {
self._bitfield_1.set(15, 1, val as u64)
}
#[inline]
pub fn set_reduced_tx_set(&mut self, val: u32) {
self._bitfield_1.set(16, 1, val as u64)
}
#[inline]
pub fn set_skip_mode_present(&mut self, val: u32) {
self._bitfield_1.set(17, 1, val as u64)
}
#[inline]
pub fn set_delta_q_present(&mut self, val: u32) {
self._bitfield_1.set(18, 1, val as u64)
}
#[inline]
pub fn set_delta_lf_present(&mut self, val: u32) {
self._bitfield_1.set(19, 1, val as u64)
}
#[inline]
pub fn set_delta_lf_multi(&mut self, val: u32) {
self._bitfield_1.set(20, 1, val as u64)
}
#[inline]
pub fn set_segmentation_enabled(&mut self, val: u32) {
self._bitfield_1.set(21, 1, val as u64)
}
#[inline]
pub fn set_segmentation_update_map(&mut self, val: u32) {
self._bitfield_1.set(22, 1, val as u64)
}
#[inline]
pub fn set_segmentation_temporal_update(&mut self, val: u32) {
self._bitfield_1.set(23, 1, val as u64)
}
#[inline]
pub fn set_segmentation_update_data(&mut self, val: u32) {
self._bitfield_1.set(24, 1, val as u64)
}
#[inline]
pub fn set_UsesLr(&mut self, val: u32) {
self._bitfield_1.set(25, 1, val as u64)
}
#[inline]
pub fn set_usesChromaLr(&mut self, val: u32) {
self._bitfield_1.set(26, 1, val as u64)
}
#[inline]
pub fn set_show_frame(&mut self, val: u32) {
self._bitfield_1.set(27, 1, val as u64)
}
#[inline]
pub fn set_showable_frame(&mut self, val: u32) {
self._bitfield_1.set(28, 1, val as u64)
}
#[inline]
pub fn set_reserved(&mut self, val: u32) {
self._bitfield_1.set(29, 3, val as u64)
}
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct StdVideoEncodeAV1PictureInfo {
pub flags: StdVideoEncodeAV1PictureInfoFlags,
pub frame_type: StdVideoAV1FrameType,
pub frame_presentation_time: u32,
pub current_frame_id: u32,
pub order_hint: u8,
pub primary_ref_frame: u8,
pub refresh_frame_flags: u8,
pub coded_denom: u8,
pub render_width_minus_1: u16,
pub render_height_minus_1: u16,
pub interpolation_filter: StdVideoAV1InterpolationFilter,
pub TxMode: StdVideoAV1TxMode,
pub delta_q_res: u8,
pub delta_lf_res: u8,
pub ref_order_hint: [u8; 8usize],
pub ref_frame_idx: [i8; 7usize],
pub reserved1: [u8; 3usize],
pub delta_frame_id_minus_1: [u32; 7usize],
pub pTileInfo: *const ash::vk::native::StdVideoAV1TileInfo,
pub pQuantization: *const ash::vk::native::StdVideoAV1Quantization,
pub pSegmentation: *const ash::vk::native::StdVideoAV1Segmentation,
pub pLoopFilter: *const ash::vk::native::StdVideoAV1LoopFilter,
pub pCDEF: *const ash::vk::native::StdVideoAV1CDEF,
pub pLoopRestoration: *const ash::vk::native::StdVideoAV1LoopRestoration,
pub pGlobalMotion: *const ash::vk::native::StdVideoAV1GlobalMotion,
pub pExtensionHeader: *const StdVideoEncodeAV1ExtensionHeader,
pub pBufferRemovalTimes: *const u32,
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct StdVideoEncodeAV1ReferenceInfoFlags {
pub _bitfield_align_1: [u32; 0],
pub _bitfield_1: __BindgenBitfieldUnit<[u8; 4usize]>,
}
impl StdVideoEncodeAV1ReferenceInfoFlags {
#[inline]
pub fn set_disable_frame_end_update_cdf(&mut self, val: u32) {
self._bitfield_1.set(0, 1, val as u64)
}
#[inline]
pub fn set_segmentation_enabled(&mut self, val: u32) {
self._bitfield_1.set(1, 1, val as u64)
}
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct StdVideoEncodeAV1ReferenceInfo {
pub flags: StdVideoEncodeAV1ReferenceInfoFlags,
pub RefFrameId: u32,
pub frame_type: StdVideoAV1FrameType,
pub OrderHint: u8,
pub reserved1: [u8; 3usize],
pub pExtensionHeader: *const StdVideoEncodeAV1ExtensionHeader,
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct StdVideoEncodeAV1ExtensionHeader {
pub temporal_id: u8,
pub spatial_id: u8,
}
// ---------- KHR extension structs (repr(C); lifetimes/PhantomData dropped — layout-identical,
// chained by raw p_next). Flag/enum newtypes flattened to their u32/i32 repr. ----------
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1ProfileInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub std_profile: StdVideoAV1Profile,
}
/// `VkPhysicalDeviceVideoEncodeAV1FeaturesKHR` — the `videoEncodeAV1` feature MUST be enabled at
/// device creation for any `VK_VIDEO_CODEC_OPERATION_ENCODE_AV1` use (a spec requirement RADV may
/// tolerate omitting but validation layers and stricter drivers do not).
#[repr(C)]
#[derive(Copy, Clone)]
pub struct PhysicalDeviceVideoEncodeAV1FeaturesKHR {
pub s_type: vk::StructureType,
pub p_next: *mut c_void,
pub video_encode_av1: vk::Bool32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1CapabilitiesKHR {
pub s_type: vk::StructureType,
pub p_next: *mut c_void,
pub flags: u32,
pub max_level: StdVideoAV1Level,
pub coded_picture_alignment: vk::Extent2D,
pub max_tiles: vk::Extent2D,
pub min_tile_size: vk::Extent2D,
pub max_tile_size: vk::Extent2D,
pub superblock_sizes: u32,
pub max_single_reference_count: u32,
pub single_reference_name_mask: u32,
pub max_unidirectional_compound_reference_count: u32,
pub max_unidirectional_compound_group1_reference_count: u32,
pub unidirectional_compound_reference_name_mask: u32,
pub max_bidirectional_compound_reference_count: u32,
pub max_bidirectional_compound_group1_reference_count: u32,
pub max_bidirectional_compound_group2_reference_count: u32,
pub bidirectional_compound_reference_name_mask: u32,
pub max_temporal_layer_count: u32,
pub max_spatial_layer_count: u32,
pub max_operating_points: u32,
pub min_q_index: u32,
pub max_q_index: u32,
pub prefers_gop_remaining_frames: vk::Bool32,
pub requires_gop_remaining_frames: vk::Bool32,
pub std_syntax_flags: u32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1SessionParametersCreateInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub p_std_sequence_header: *const StdVideoAV1SequenceHeader,
pub p_std_decoder_model_info: *const c_void,
pub std_operating_point_count: u32,
pub p_std_operating_points: *const c_void,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1PictureInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub prediction_mode: i32,
pub rate_control_group: i32,
pub constant_q_index: u32,
pub p_std_picture_info: *const StdVideoEncodeAV1PictureInfo,
pub reference_name_slot_indices: [i32; MAX_VIDEO_AV1_REFERENCES_PER_FRAME],
pub primary_reference_cdf_only: vk::Bool32,
pub generate_obu_extension_header: vk::Bool32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1DpbSlotInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub p_std_reference_info: *const StdVideoEncodeAV1ReferenceInfo,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1RateControlInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub flags: u32,
pub gop_frame_count: u32,
pub key_frame_period: u32,
pub consecutive_bipredictive_frame_count: u32,
pub temporal_layer_count: u32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1QIndexKHR {
pub intra_q_index: u32,
pub predictive_q_index: u32,
pub bipredictive_q_index: u32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1FrameSizeKHR {
pub intra_frame_size: u32,
pub predictive_frame_size: u32,
pub bipredictive_frame_size: u32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1RateControlLayerInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub use_min_q_index: vk::Bool32,
pub min_q_index: VideoEncodeAV1QIndexKHR,
pub use_max_q_index: vk::Bool32,
pub max_q_index: VideoEncodeAV1QIndexKHR,
pub use_max_frame_size: vk::Bool32,
pub max_frame_size: VideoEncodeAV1FrameSizeKHR,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1GopRemainingFrameInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub use_gop_remaining_frames: vk::Bool32,
pub gop_remaining_intra: u32,
pub gop_remaining_predictive: u32,
pub gop_remaining_bipredictive: u32,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct VideoEncodeAV1SessionCreateInfoKHR {
pub s_type: vk::StructureType,
pub p_next: *const c_void,
pub use_max_level: vk::Bool32,
pub max_level: StdVideoAV1Level,
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct StdVideoEncodeAV1OperatingPointInfoFlags {
pub _bitfield_align_1: [u32; 0],
pub _bitfield_1: __BindgenBitfieldUnit<[u8; 4usize]>,
}
#[repr(C)]
#[derive(Copy, Clone)]
pub struct StdVideoEncodeAV1OperatingPointInfo {
pub flags: StdVideoEncodeAV1OperatingPointInfoFlags,
pub operating_point_idc: u16,
pub seq_level_idx: u8,
pub seq_tier: u8,
pub decoder_buffer_delay: u32,
pub encoder_buffer_delay: u32,
pub initial_display_delay_minus_1: u8,
}
/// `vk::StructureType` for a raw `ST_*` constant above.
#[inline]
pub fn stype(raw: i32) -> vk::StructureType {
vk::StructureType::from_raw(raw)
}
File diff suppressed because it is too large Load Diff
+168 -35
View File
@@ -12,8 +12,11 @@
//! Drives the AMF runtime through its **C vtable ABI**: the GPUOpen public headers define //! Drives the AMF runtime through its **C vtable ABI**: the GPUOpen public headers define
//! C-compatible vtable structs for every interface, and FFmpeg's `amfenc.c` (plain C) drives AMF //! C-compatible vtable structs for every interface, and FFmpeg's `amfenc.c` (plain C) drives AMF
//! exclusively through them, so that ABI — not the C++ classes — is the stable, supported //! exclusively through them, so that ABI — not the C++ classes — is the stable, supported
//! surface. The FFI below mirrors ONLY the interfaces/slots we call, pinned to header version //! surface. The FFI below mirrors ONLY the interfaces/slots we call, written against header
//! **v1.4.36** (`AMF_FULL_VERSION` 1.4.36.0, gated at load via `AMFQueryVersion`). The runtime is //! version **v1.4.36** (`AMF_FULL_VERSION` 1.4.36.0). At load the runtime is accepted down to a
//! stable-ABI floor of **v1.4.34** (the `AMFQueryVersion` gate); the 1.4.35/1.4.36-only encoder
//! features are string-keyed properties that degrade individually on older drivers, not vtable
//! changes (see [`sys::AMF_MIN_VERSION`]). The runtime is
//! loaded at runtime from the driver-installed `amfrt64.dll` — exactly as `nvenc.rs` loads //! loaded at runtime from the driver-installed `amfrt64.dll` — exactly as `nvenc.rs` loads
//! `nvEncodeAPI64.dll` — so this compiles unconditionally on Windows (**no build feature, no new //! `nvEncodeAPI64.dll` — so this compiles unconditionally on Windows (**no build feature, no new
//! dependency**). Since Phase 3 (design §7) this is the sole AMD dispatch: a box without a //! dependency**). Since Phase 3 (design §7) this is the sole AMD dispatch: a box without a
@@ -50,6 +53,7 @@ use std::collections::VecDeque;
use std::ffi::c_void; use std::ffi::c_void;
use std::ptr; use std::ptr;
use windows::core::{w, Interface, PCWSTR}; use windows::core::{w, Interface, PCWSTR};
use windows::Win32::Foundation::HMODULE;
use windows::Win32::Graphics::Direct3D11::{ use windows::Win32::Graphics::Direct3D11::{
ID3D11Device, ID3D11DeviceContext, ID3D11Resource, ID3D11Texture2D, D3D11_BIND_RENDER_TARGET, ID3D11Device, ID3D11DeviceContext, ID3D11Resource, ID3D11Texture2D, D3D11_BIND_RENDER_TARGET,
D3D11_BIND_SHADER_RESOURCE, D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT, D3D11_BIND_SHADER_RESOURCE, D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT,
@@ -57,9 +61,15 @@ use windows::Win32::Graphics::Direct3D11::{
use windows::Win32::Graphics::Dxgi::Common::{ use windows::Win32::Graphics::Dxgi::Common::{
DXGI_FORMAT_NV12, DXGI_FORMAT_P010, DXGI_SAMPLE_DESC, DXGI_FORMAT_NV12, DXGI_FORMAT_P010, DXGI_SAMPLE_DESC,
}; };
use windows::Win32::Storage::FileSystem::{
GetFileVersionInfoSizeW, GetFileVersionInfoW, VerQueryValueW, VS_FIXEDFILEINFO,
};
use windows::Win32::System::LibraryLoader::GetModuleFileNameW;
// --------------------------------------------------------------------------------------------- // ---------------------------------------------------------------------------------------------
// Mirrored AMF C ABI (pinned to GPUOpen header release v1.4.36 — amf/public/include). // Mirrored AMF C ABI (written against GPUOpen header release v1.4.36 — amf/public/include; every
// slot below is a base-interface slot whose layout is stable since <= v1.4.34, the loader's
// accepted ABI floor, so the mirror is valid on every runtime the loader admits).
// //
// Layout rules this mirror relies on: every AMF interface is a struct whose sole member is a // Layout rules this mirror relies on: every AMF interface is a struct whose sole member is a
// pointer to a C vtable; derived interfaces PREPEND their base's slots in order (AMFInterface → // pointer to a C vtable; derived interfaces PREPEND their base's slots in order (AMFInterface →
@@ -118,11 +128,24 @@ mod sys {
} }
} }
/// The pinned header version this FFI mirrors: `AMF_FULL_VERSION` for 1.4.36.0 /// The AMF header release this FFI mirror was written against: `AMF_FULL_VERSION` for 1.4.36.0
/// (core/Version.h `AMF_MAKE_FULL_VERSION`). The loader requires the runtime to report at /// (core/Version.h `AMF_MAKE_FULL_VERSION`). This is the version claimed to `AMFInit` — but
/// least this via `AMFQueryVersion`, guaranteeing every vtable slot mirrored below exists at /// capped at the runtime's own reported version (see `load_factory`), so an older-but-accepted
/// the mirrored offset. /// runtime is asked only for the ABI it actually provides.
pub const AMF_PINNED_VERSION: u64 = (1u64 << 48) | (4u64 << 32) | (36u64 << 16); pub const AMF_HEADER_VERSION: u64 = (1u64 << 48) | (4u64 << 32) | (36u64 << 16);
/// The oldest AMF runtime the loader accepts (`AMF_FULL_VERSION` 1.4.34.0 — AMD Adrenalin
/// 24.6.1). This is an **ABI floor, not a feature floor**: every vtable slot mirrored in this
/// module belongs to a base interface (`AMFFactory`/`AMFContext`/`AMFComponent`/`AMFData`/
/// `AMFBuffer`) whose layout has been stable — append-only, no mid-vtable insertions — since
/// well before 1.4.34, so a 1.4.34 runtime is guaranteed to expose every mirrored slot at its
/// mirrored offset. Everything 1.4.35/1.4.36 added that this path can touch (new HQ presets,
/// AV1 B-frame / picture management) is a *string-keyed encoder property*, applied through
/// [`set_prop`] with `required=false` — a runtime that lacks it rejects the property (logged)
/// and the feature degrades, rather than shifting any vtable offset. Below this floor the
/// mirror is not guaranteed to match, so the loader declines cleanly (an old-driver decline,
/// never UB).
pub const AMF_MIN_VERSION: u64 = (1u64 << 48) | (4u64 << 32) | (34u64 << 16);
/// `AMF_SURFACE_FORMAT` (core/Surface.h). /// `AMF_SURFACE_FORMAT` (core/Surface.h).
pub const AMF_SURFACE_NV12: i32 = 1; pub const AMF_SURFACE_NV12: i32 = 1;
@@ -474,6 +497,76 @@ fn amf_ok(r: sys::AmfResult, what: &str) -> Result<()> {
} }
} }
/// Format an `AMF_FULL_VERSION` u64 as `major.minor.patch` (the build field is dropped — every
/// version comparison and log line in this module ignores it).
fn amf_version_str(v: u64) -> String {
format!(
"{}.{}.{}",
(v >> 48) & 0xffff,
(v >> 32) & 0xffff,
(v >> 16) & 0xffff
)
}
/// Best-effort on-disk identity of the loaded `amfrt64.dll`: `(full path, file-version resource)`.
/// The file-version is the driver build baked into the DLL (e.g. `31.0.24033.1003`), which — unlike
/// the AMF runtime version — is directly comparable to the display-driver version. This pins down
/// the Boot Camp failure mode: the display driver can report 25.x while the `amfrt64.dll` actually
/// loaded (System32, via the SYSTEM32-only search) is a stale build whose AMF + file versions lag
/// it. Diagnostics only — any failure yields `None` and never affects the load.
///
/// # Safety
/// `module` must be a live module handle owned by the caller (here, the never-unloaded
/// `amfrt64.dll` from `LoadLibraryExW`).
unsafe fn loaded_dll_identity(module: HMODULE) -> (Option<String>, Option<String>) {
let mut buf = [0u16; 512];
let n = GetModuleFileNameW(Some(module), &mut buf) as usize;
// n == 0 → failed; n >= len → path truncated (no guaranteed NUL) — bail either way. Otherwise
// `GetModuleFileNameW` NUL-terminates at `buf[n]`, so `buf` is a valid PCWSTR for the query.
if n == 0 || n >= buf.len() {
return (None, None);
}
let path = String::from_utf16_lossy(&buf[..n]);
(Some(path), dll_file_version(PCWSTR(buf.as_ptr())))
}
/// Read a DLL's `\`-root `VS_FIXEDFILEINFO` file version as `a.b.c.d`. `None` if the file has no
/// version resource or any step fails (diagnostics only).
///
/// # Safety
/// `path` must be a valid NUL-terminated wide string pointing at a readable file path.
unsafe fn dll_file_version(path: PCWSTR) -> Option<String> {
let size = GetFileVersionInfoSizeW(path, None);
if size == 0 {
return None;
}
let mut block = vec![0u8; size as usize];
GetFileVersionInfoW(path, None, size, block.as_mut_ptr() as *mut c_void).ok()?;
let mut value: *mut c_void = ptr::null_mut();
let mut len: u32 = 0;
let ok = VerQueryValueW(
block.as_ptr() as *const c_void,
w!("\\"),
&mut value,
&mut len,
);
if !ok.as_bool() || value.is_null() || (len as usize) < std::mem::size_of::<VS_FIXEDFILEINFO>()
{
return None;
}
// SAFETY: on success `VerQueryValueW` points `value` at a `VS_FIXEDFILEINFO` living inside
// `block` and valid for `len` bytes (checked >= its size); `block` outlives this read.
let ffi = &*(value as *const VS_FIXEDFILEINFO);
let (ms, ls) = (ffi.dwFileVersionMS, ffi.dwFileVersionLS);
Some(format!(
"{}.{}.{}.{}",
ms >> 16,
ms & 0xffff,
ls >> 16,
ls & 0xffff
))
}
// --------------------------------------------------------------------------------------------- // ---------------------------------------------------------------------------------------------
// Runtime loader (the analogue of nvenc.rs `load_api`): resolve amfrt64.dll's two exports once // Runtime loader (the analogue of nvenc.rs `load_api`): resolve amfrt64.dll's two exports once
// per process, gate on the pinned header version, and keep the factory singleton forever. // per process, gate on the pinned header version, and keep the factory singleton forever.
@@ -493,9 +586,9 @@ unsafe impl Send for AmfLib {}
unsafe impl Sync for AmfLib {} unsafe impl Sync for AmfLib {}
/// Resolve the AMF runtime once per process. `Err` = AMF genuinely unavailable here (no AMD /// Resolve the AMF runtime once per process. `Err` = AMF genuinely unavailable here (no AMD
/// driver / `amfrt64.dll`, or a runtime older than the pinned v1.4.36 headers) — callers fail /// driver / `amfrt64.dll`, or a runtime older than the minimum-supported v1.4.34 — see
/// their open cleanly with an "update the AMD driver" message (the session then fails; since /// [`sys::AMF_MIN_VERSION`]) — callers fail their open cleanly with an "update the AMD driver"
/// Phase 3 there is no libavcodec AMF fallback). /// message (the session then fails; since Phase 3 there is no libavcodec AMF fallback).
fn try_factory() -> std::result::Result<&'static AmfLib, &'static str> { fn try_factory() -> std::result::Result<&'static AmfLib, &'static str> {
static LIB: std::sync::OnceLock<std::result::Result<AmfLib, String>> = static LIB: std::sync::OnceLock<std::result::Result<AmfLib, String>> =
std::sync::OnceLock::new(); std::sync::OnceLock::new();
@@ -520,10 +613,11 @@ fn load_factory() -> std::result::Result<AmfLib, String> {
// System32-only search path keeps a planted DLL out of the SYSTEM-service process (same // System32-only search path keeps a planted DLL out of the SYSTEM-service process (same
// hardening as the NVENC loader). The two transmutes cast the resolved exports to their // hardening as the NVENC loader). The two transmutes cast the resolved exports to their
// documented prototypes (core/Factory.h `AMFQueryVersion_Fn`/`AMFInit_Fn`). // documented prototypes (core/Factory.h `AMFQueryVersion_Fn`/`AMFInit_Fn`).
// `AMFQueryVersion` writes one u64 through a live pointer; `AMFInit` is passed the pinned // `AMFQueryVersion` writes one u64 through a live pointer; `AMFInit` is passed the header
// header version and fills `factory` with the process-global singleton only on AMF_OK // version capped at the runtime's own (never newer than what the runtime provides) and fills
// (null-checked after). The module is never freed, so the factory and both entry points stay // `factory` with the process-global singleton only on AMF_OK (null-checked after). The module
// valid for the process lifetime. // is never freed, so the factory and both entry points stay valid for the process lifetime.
// `loaded_dll_identity` only reads that module's own path + version resource (diagnostics).
unsafe { unsafe {
let module = LoadLibraryExW(w!("amfrt64.dll"), None, LOAD_LIBRARY_SEARCH_SYSTEM32) let module = LoadLibraryExW(w!("amfrt64.dll"), None, LOAD_LIBRARY_SEARCH_SYSTEM32)
.map_err(|e| { .map_err(|e| {
@@ -540,33 +634,75 @@ fn load_factory() -> std::result::Result<AmfLib, String> {
if r != sys::AMF_OK { if r != sys::AMF_OK {
return Err(format!("AMFQueryVersion failed: {} ({r})", result_name(r))); return Err(format!("AMFQueryVersion failed: {} ({r})", result_name(r)));
} }
// The vtable layouts mirrored above are the pinned header's; an older runtime may lack // On-disk identity of the DLL we actually loaded (System32's amfrt64.dll, via the
// trailing slots (or predate an insertion), so require at least the pinned version — an // SYSTEM32-only search above) — the Boot Camp diagnostic: the display driver can read 25.x
// old driver is a clean decline (clear session error), not UB. // while THIS file is a stale build whose AMF + file versions lag it, so an "update the
if version < sys::AMF_PINNED_VERSION { // driver" decline is confusing (they did — this DLL just didn't follow).
let (dll_path, dll_file_ver) = loaded_dll_identity(module);
let dll_desc = format!(
"{}{}",
dll_path.as_deref().unwrap_or("amfrt64.dll"),
dll_file_ver
.as_deref()
.map(|v| format!(" (file version {v})"))
.unwrap_or_default(),
);
// Accept any runtime at or above the ABI floor (AMF_MIN_VERSION): every vtable slot this
// module mirrors predates it, so its layout is guaranteed; 1.4.35/1.4.36-only encoder
// features are string-keyed properties that degrade via `set_prop(required=false)`, not
// vtable changes. Below the floor the mirror is not guaranteed — decline cleanly (a clear
// old-driver session error, never UB).
if version < sys::AMF_MIN_VERSION {
return Err(format!( return Err(format!(
"AMF runtime {}.{}.{} is older than the host's pinned headers 1.4.36 — update \ "AMF runtime {amf} (loaded from {dll_desc}) is older than the minimum supported \
the AMD driver", 1.4.34 update the AMD driver (Adrenalin 24.6.1+; 25.1.1+ for the \
(version >> 48) & 0xffff, fully-validated feature set). If the display driver already reports a newer \
(version >> 32) & 0xffff, version, this amfrt64.dll did not update reboot, then DDU + reinstall so \
(version >> 16) & 0xffff, System32's copy is refreshed.",
amf = amf_version_str(version),
)); ));
} }
// Claim no more than the runtime provides: passing a version NEWER than the runtime can
// make AMFInit reject an otherwise-usable older driver, and this path only ever calls ABI
// present at/below the runtime's version. On a >=1.4.36 runtime this is a no-op (== header).
let init_version = sys::AMF_HEADER_VERSION.min(version);
let mut factory: *mut sys::AmfFactory = ptr::null_mut(); let mut factory: *mut sys::AmfFactory = ptr::null_mut();
let r = init(sys::AMF_PINNED_VERSION, &mut factory); let r = init(init_version, &mut factory);
if r != sys::AMF_OK { if r != sys::AMF_OK {
return Err(format!("AMFInit failed: {} ({r})", result_name(r))); return Err(format!("AMFInit failed: {} ({r})", result_name(r)));
} }
if factory.is_null() { if factory.is_null() {
return Err("AMFInit returned a null factory".into()); return Err("AMFInit returned a null factory".into());
} }
// Visible once per process (this runs inside `try_factory`'s OnceLock init). Both the AMF
// runtime version AND the loaded DLL's path + file version are logged, so a field report
// shows "display driver says 25.x but amfrt64.dll is an old build" at a glance.
if version >= sys::AMF_HEADER_VERSION {
tracing::info!(
amf_version = %amf_version_str(version),
dll = %dll_desc,
"AMF runtime loaded (meets the validated 1.4.36 baseline)"
);
} else {
tracing::warn!(
amf_version = %amf_version_str(version),
dll = %dll_desc,
"AMF runtime is older than the validated 1.4.36 baseline — accepted (the core \
encode ABI is stable), but advanced features (LTR / intra-refresh recovery, AV1 \
coded-size alignment, in-band HDR metadata) validated on 1.4.36 may be \
unavailable on this driver and will degrade individually (see the per-property \
logs below). Update to AMD Adrenalin 25.1.1+ for the fully-validated path."
);
}
Ok(AmfLib { factory, version }) Ok(AmfLib { factory, version })
} }
} }
// --------------------------------------------------------------------------------------------- // ---------------------------------------------------------------------------------------------
// Per-codec property tables (names verified against the pinned v1.4.36 headers — // Per-codec property tables (names verified against the v1.4.36 headers —
// components/VideoEncoderVCE.h, VideoEncoderHEVC.h and VideoEncoderAV1.h; the enum VALUES differ // components/VideoEncoderVCE.h, VideoEncoderHEVC.h and VideoEncoderAV1.h; a name a pre-1.4.36
// runtime doesn't recognise is applied through `set_prop(required=false)`, which logs and
// continues, so an older driver degrades that one feature rather than failing. The enum VALUES differ
// between the codecs, e.g. CBR is 1 on AVC but 3 on HEVC/AV1, SPEED is 1 vs 10 vs 100, and AV1 // between the codecs, e.g. CBR is 1 on AVC but 3 on HEVC/AV1, SPEED is 1 vs 10 vs 100, and AV1
// swaps the ULTRA_LOW_LATENCY/LOW_LATENCY usage values relative to AVC/HEVC). // swaps the ULTRA_LOW_LATENCY/LOW_LATENCY usage values relative to AVC/HEVC).
// --------------------------------------------------------------------------------------------- // ---------------------------------------------------------------------------------------------
@@ -1118,15 +1254,12 @@ impl AmfEncoder {
bit_depth: u8, bit_depth: u8,
chroma: ChromaFormat, chroma: ChromaFormat,
) -> Result<Self> { ) -> Result<Self> {
// The once-per-process runtime version (and the older-than-baseline warning) is logged in
// `load_factory`; this per-session line ties an individual encoder open to that version.
let lib = try_factory().map_err(|e| anyhow!("native AMF unavailable: {e}"))?; let lib = try_factory().map_err(|e| anyhow!("native AMF unavailable: {e}"))?;
tracing::debug!( tracing::debug!(
version = %format!( version = %amf_version_str(lib.version),
"{}.{}.{}", "opening AMF encoder"
(lib.version >> 48) & 0xffff,
(lib.version >> 32) & 0xffff,
(lib.version >> 16) & 0xffff
),
"AMF runtime loaded"
); );
let props = codec_props(codec); let props = codec_props(codec);
// AV1 is RDNA3+ — probe at open (never assume), so a pre-RDNA3 box fails HERE with a clear // AV1 is RDNA3+ — probe at open (never assume), so a pre-RDNA3 box fails HERE with a clear
@@ -3062,7 +3195,7 @@ mod tests {
return; return;
} }
}; };
assert!(lib.version >= sys::AMF_PINNED_VERSION); assert!(lib.version >= sys::AMF_MIN_VERSION);
// SAFETY: same contracts as `ensure_inner`, minus the external device: `CreateContext` // SAFETY: same contracts as `ensure_inner`, minus the external device: `CreateContext`
// fills `ctx` only on AMF_OK; `InitDX11(null)` asks AMF to create its own D3D11 device // fills `ctx` only on AMF_OK; `InitDX11(null)` asks AMF to create its own D3D11 device
// (may fail on exotic boxes — treated as a skip); `CreateComponent` likewise. Guards // (may fail on exotic boxes — treated as a skip); `CreateComponent` likewise. Guards
@@ -425,7 +425,11 @@ impl InputInjector for KwinFakeInjector {
self.fake.touch_frame(); self.fake.touch_frame();
} }
// Gamepads are injected through uinput, not the compositor. // Gamepads are injected through uinput, not the compositor.
InputKind::GamepadState | InputKind::GamepadButton | InputKind::GamepadAxis => {} InputKind::GamepadState
| InputKind::GamepadButton
| InputKind::GamepadAxis
| InputKind::GamepadRemove
| InputKind::GamepadArrival => {}
} }
// Surface protocol errors / disconnects, then push the batch to the compositor. // Surface protocol errors / disconnects, then push the batch to the compositor.
self.queue self.queue
@@ -404,6 +404,8 @@ fn kind_bit(kind: InputKind) -> u32 {
InputKind::GamepadButton => 10, InputKind::GamepadButton => 10,
InputKind::GamepadAxis => 11, InputKind::GamepadAxis => 11,
InputKind::GamepadState => 12, InputKind::GamepadState => 12,
InputKind::GamepadRemove => 13,
InputKind::GamepadArrival => 14,
}; };
1 << i 1 << i
} }
@@ -546,7 +548,11 @@ impl EiState {
InputKind::TouchDown | InputKind::TouchMove | InputKind::TouchUp => { InputKind::TouchDown | InputKind::TouchMove | InputKind::TouchUp => {
DeviceCapability::Touch DeviceCapability::Touch
} }
InputKind::GamepadState | InputKind::GamepadButton | InputKind::GamepadAxis => return, // uinput path (later) InputKind::GamepadState
| InputKind::GamepadButton
| InputKind::GamepadAxis
| InputKind::GamepadRemove
| InputKind::GamepadArrival => return, // uinput path (later)
}; };
self.injected += 1; self.injected += 1;
let n = self.injected; let n = self.injected;
@@ -693,9 +699,11 @@ impl EiState {
Some(t) => t.up(ev.code), Some(t) => t.up(ev.code),
None => emitted = false, None => emitted = false,
}, },
InputKind::GamepadState | InputKind::GamepadButton | InputKind::GamepadAxis => { InputKind::GamepadState
emitted = false | InputKind::GamepadButton
} | InputKind::GamepadAxis
| InputKind::GamepadRemove
| InputKind::GamepadArrival => emitted = false,
} }
if emitted { if emitted {
@@ -254,7 +254,11 @@ impl InputInjector for WlrootsInjector {
tracing::debug!(vk = event.code, "unmapped VK keycode — dropped"); tracing::debug!(vk = event.code, "unmapped VK keycode — dropped");
} }
} }
InputKind::GamepadState | InputKind::GamepadButton | InputKind::GamepadAxis => {} // not yet injected InputKind::GamepadState
| InputKind::GamepadButton
| InputKind::GamepadAxis
| InputKind::GamepadRemove
| InputKind::GamepadArrival => {} // not yet injected
// wlroots has no virtual-touch protocol wired here; touch is the libei path only. // wlroots has no virtual-touch protocol wired here; touch is the libei path only.
InputKind::TouchDown | InputKind::TouchMove | InputKind::TouchUp => {} InputKind::TouchDown | InputKind::TouchMove | InputKind::TouchUp => {}
} }
@@ -301,6 +301,8 @@ impl InputInjector for SendInputInjector {
InputKind::GamepadButton InputKind::GamepadButton
| InputKind::GamepadAxis | InputKind::GamepadAxis
| InputKind::GamepadState | InputKind::GamepadState
| InputKind::GamepadRemove
| InputKind::GamepadArrival
| InputKind::TouchDown | InputKind::TouchDown
| InputKind::TouchMove | InputKind::TouchMove
| InputKind::TouchUp => Ok(()), | InputKind::TouchUp => Ok(()),
+334 -118
View File
@@ -1740,166 +1740,315 @@ const MAX_WIRE_PADS: usize = punktfunk_core::input::MAX_PADS;
/// virtual mic has its own tuning — see [`crate::audio::MicPump`].) /// virtual mic has its own tuning — see [`crate::audio::MicPump`].)
const INJECTOR_REOPEN_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2); const INJECTOR_REOPEN_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2);
/// The session's virtual-gamepad backend, resolved once per session (sessions run serially). /// Per-pad virtual-gamepad router: each pad index is served by a backend of that pad's declared
/// kind ([`InputKind::GamepadArrival`](punktfunk_core::input::InputKind::GamepadArrival)), so ONE
/// session can MIX controller types — pad 0 a DualSense, pad 1 an Xbox pad. A pad the client never
/// declares uses `default` (the session kind resolved from the Hello — the pre-existing single-kind
/// behaviour).
/// ///
/// - `Xbox360` — uinput X-Box-360 pads on Linux ([`GamepadManager`](crate::inject::gamepad::GamepadManager)), /// Backends are created lazily per kind (an empty manager holds no device), and each owns only the
/// the in-tree XUSB companion driver (classic XInput) on Windows. Also the X-Box One/Series identity /// indices routed to it. A manager's `active_mask` unplug sweep stays correct across managers
/// (`PUNKTFUNK_GAMEPAD=xboxone`): the same /// because an index another manager owns is `None` in this one, so the sweep never touches it.
/// backend with the One/Series USB VID/PID so games show One/Series glyphs (XInput-identical
/// otherwise). The Linux pad carries it as a [`PadIdentity`](crate::inject::gamepad::PadIdentity).
/// - `DualSense` (`PUNKTFUNK_GAMEPAD=dualsense`) — virtual DualSense via UHID + `hid-playstation`,
/// so a game sees a *real* DualSense (adaptive triggers, lightbar, touchpad, motion); feedback
/// flows back over the rich HID-output plane.
/// - `DualShock4` (`PUNKTFUNK_GAMEPAD=ps4`) — virtual DualShock 4 via the same UHID path: lightbar,
/// touchpad, motion, rumble (DualSense minus adaptive triggers / player LEDs / mute).
/// ///
/// DualShock 4 + One/Series are Linux-only; DualSense has both a Linux (UHID) and a Windows (UMDF /// - Xbox 360 / One — uinput on Linux ([`GamepadManager`](crate::inject::gamepad::GamepadManager),
/// minidriver) backend. The resolver folds any type a platform can't build into `Xbox360`, so a /// two identities), the XUSB companion driver (classic XInput) on Windows.
/// build never constructs a variant it lacks. /// - DualSense / DualShock 4 — Linux UHID `hid-playstation`, or the Windows UMDF minidriver.
enum PadBackend { /// - Steam Deck — Linux UHID `hid-steam`.
Xbox360(crate::inject::gamepad::GamepadManager), ///
/// [`resolve_pad_kind`] folds any kind a platform can't build into one it can, so this never
/// constructs a manager the build lacks.
struct Pads {
/// Declared (and host-resolved) kind per pad index; `default` until a `GamepadArrival` lands.
kinds: [GamepadPref; MAX_WIRE_PADS],
/// The kind of the manager that currently OWNS a built device at each index (`None` = no
/// device). A live device stays in its manager even if `kinds[idx]` later changes (the rare
/// arrival-after-first-frame reorder), so a pad is never duplicated across managers and its
/// removal always reaches the manager that actually holds it.
owner: [Option<GamepadPref>; MAX_WIRE_PADS],
xbox360: Option<crate::inject::gamepad::GamepadManager>,
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
DualSense(crate::inject::dualsense::DualSenseManager), xboxone: Option<crate::inject::gamepad::GamepadManager>,
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
DualShock4(crate::inject::dualshock4::DualShock4Manager), dualsense: Option<crate::inject::dualsense::DualSenseManager>,
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
SteamDeck(crate::inject::steam_controller::SteamControllerManager), dualshock4: Option<crate::inject::dualshock4::DualShock4Manager>,
#[cfg(target_os = "linux")]
steamdeck: Option<crate::inject::steam_controller::SteamControllerManager>,
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
DualSenseWindows(crate::inject::dualsense_windows::DualSenseWindowsManager), dualsense_win: Option<crate::inject::dualsense_windows::DualSenseWindowsManager>,
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
DualShock4Windows(crate::inject::dualshock4_windows::DualShock4WindowsManager), dualshock4_win: Option<crate::inject::dualshock4_windows::DualShock4WindowsManager>,
} }
impl PadBackend { impl Pads {
/// `kind` is the session's resolved backend (see [`resolve_gamepad`] — client preference, /// `default` is the session kind (see [`resolve_gamepad`]); every pad starts on it until the
/// env var, X-Box 360, in that order). Defensive cfg guard: a non-Linux build can only ever /// client declares its own kind.
/// construct the X-Box backend, whatever the resolution said. fn new(default: GamepadPref) -> Pads {
fn select(kind: GamepadPref) -> PadBackend { let default = resolve_pad_kind(default);
tracing::info!(
default = default.as_str(),
"gamepad backends: per-pad router (session default)"
);
Pads {
kinds: [default; MAX_WIRE_PADS],
owner: [None; MAX_WIRE_PADS],
xbox360: None,
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
match kind { xboxone: None,
GamepadPref::DualSense => { #[cfg(target_os = "linux")]
tracing::info!("gamepad backend: virtual DualSense (UHID hid-playstation)"); dualsense: None,
return PadBackend::DualSense(crate::inject::dualsense::DualSenseManager::new()); #[cfg(target_os = "linux")]
} dualshock4: None,
GamepadPref::DualShock4 => { #[cfg(target_os = "linux")]
tracing::info!("gamepad backend: virtual DualShock 4 (UHID hid-playstation)"); steamdeck: None,
return PadBackend::DualShock4(crate::inject::dualshock4::DualShock4Manager::new());
}
GamepadPref::SteamDeck => {
tracing::info!("gamepad backend: virtual Steam Deck (UHID hid-steam)");
return PadBackend::SteamDeck(
crate::inject::steam_controller::SteamControllerManager::new(),
);
}
GamepadPref::XboxOne => {
tracing::info!("gamepad backend: uinput X-Box One/Series pad");
return PadBackend::Xbox360(crate::inject::gamepad::GamepadManager::with_identity(
crate::inject::gamepad::PadIdentity::xbox_one(),
));
}
_ => {}
}
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
match kind { dualsense_win: None,
GamepadPref::DualSense => { #[cfg(target_os = "windows")]
tracing::info!("gamepad backend: virtual DualSense (Windows UMDF shm channel)"); dualshock4_win: None,
return PadBackend::DualSenseWindows( }
crate::inject::dualsense_windows::DualSenseWindowsManager::new(), }
/// Record a pad's client-declared kind (resolved to a buildable backend). Takes effect on the
/// pad's next frame; the arrival is sent before the pad's first input, so a device already
/// built under the wrong kind is only the rare arrival-after-first-frame reorder — it then
/// keeps the earlier kind until re-plug (no live device swap).
fn set_kind(&mut self, idx: usize, kind: GamepadPref) {
if idx >= MAX_WIRE_PADS {
return;
}
let resolved = resolve_pad_kind(kind);
if self.kinds[idx] != resolved {
tracing::info!(
pad = idx,
kind = resolved.as_str(),
"gamepad kind declared (per-pad)"
); );
} }
GamepadPref::DualShock4 => { self.kinds[idx] = resolved;
tracing::info!("gamepad backend: virtual DualShock 4 (Windows UMDF shm channel)");
return PadBackend::DualShock4Windows(
crate::inject::dualshock4_windows::DualShock4WindowsManager::new(),
);
}
_ => {}
}
let _ = kind;
PadBackend::Xbox360(crate::inject::gamepad::GamepadManager::new())
} }
fn handle(&mut self, ev: &crate::gamestream::gamepad::GamepadEvent) { fn handle(&mut self, ev: &crate::gamestream::gamepad::GamepadEvent) {
match self { use crate::gamestream::gamepad::GamepadEvent;
PadBackend::Xbox360(m) => m.handle(ev), // Present = a create/update frame (the pad's mask bit is set); a cleared bit is the
// removal frame emitted by the native detach path (`GamepadRemove`).
let (idx, present) = match ev {
GamepadEvent::State(f) => {
let idx = f.index as usize;
(idx, f.active_mask & (1 << idx) != 0)
}
GamepadEvent::Arrival { index, .. } => (*index as usize, true),
};
if idx >= MAX_WIRE_PADS {
return;
}
let (kind, new_owner) = route_decision(self.owner[idx], self.kinds[idx], present);
self.owner[idx] = new_owner;
self.route_handle(kind, ev);
}
/// Dispatch a decoded event to the manager for `kind`, creating it lazily.
fn route_handle(&mut self, kind: GamepadPref, ev: &crate::gamestream::gamepad::GamepadEvent) {
match kind {
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::DualSense(m) => m.handle(ev), GamepadPref::DualSense => self
.dualsense
.get_or_insert_with(crate::inject::dualsense::DualSenseManager::new)
.handle(ev),
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::DualShock4(m) => m.handle(ev), GamepadPref::DualShock4 => self
.dualshock4
.get_or_insert_with(crate::inject::dualshock4::DualShock4Manager::new)
.handle(ev),
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::SteamDeck(m) => m.handle(ev), GamepadPref::SteamDeck => self
.steamdeck
.get_or_insert_with(crate::inject::steam_controller::SteamControllerManager::new)
.handle(ev),
#[cfg(target_os = "linux")]
GamepadPref::XboxOne => self
.xboxone
.get_or_insert_with(|| {
crate::inject::gamepad::GamepadManager::with_identity(
crate::inject::gamepad::PadIdentity::xbox_one(),
)
})
.handle(ev),
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
PadBackend::DualSenseWindows(m) => m.handle(ev), GamepadPref::DualSense => self
.dualsense_win
.get_or_insert_with(crate::inject::dualsense_windows::DualSenseWindowsManager::new)
.handle(ev),
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
PadBackend::DualShock4Windows(m) => m.handle(ev), GamepadPref::DualShock4 => self
.dualshock4_win
.get_or_insert_with(
crate::inject::dualshock4_windows::DualShock4WindowsManager::new,
)
.handle(ev),
_ => self
.xbox360
.get_or_insert_with(crate::inject::gamepad::GamepadManager::new)
.handle(ev),
} }
} }
/// Apply a rich client→host event (touchpad / motion). A no-op for the X-Box pad, which has no /// Apply a rich client→host event (touchpad / motion) to the pad's kind manager, if it exists
/// equivalent; the DualSense and DualShock 4 pads both carry a touchpad + motion sensors. /// (rich before the first frame = no device yet = a no-op anyway). The X-Box pads have no rich
fn apply_rich(&mut self, _rich: punktfunk_core::quic::RichInput) { /// plane, so those indices ignore it.
match self { fn apply_rich(&mut self, rich: punktfunk_core::quic::RichInput) {
PadBackend::Xbox360(_) => {} use punktfunk_core::quic::RichInput;
let idx = match rich {
RichInput::Touchpad { pad, .. }
| RichInput::Motion { pad, .. }
| RichInput::TouchpadEx { pad, .. } => pad as usize,
};
// Route to the manager that actually owns the device (falling back to the declared kind
// before the first frame builds it), so a pad's touchpad/motion never lands on the wrong
// backend after a kind change.
let kind = self
.owner
.get(idx)
.copied()
.flatten()
.or_else(|| self.kinds.get(idx).copied())
.unwrap_or(GamepadPref::Xbox360);
match kind {
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::DualSense(m) => m.apply_rich(_rich), GamepadPref::DualSense => {
if let Some(m) = &mut self.dualsense {
m.apply_rich(rich)
}
}
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::DualShock4(m) => m.apply_rich(_rich), GamepadPref::DualShock4 => {
if let Some(m) = &mut self.dualshock4 {
m.apply_rich(rich)
}
}
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::SteamDeck(m) => m.apply_rich(_rich), GamepadPref::SteamDeck => {
if let Some(m) = &mut self.steamdeck {
m.apply_rich(rich)
}
}
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
PadBackend::DualSenseWindows(m) => m.apply_rich(_rich), GamepadPref::DualSense => {
if let Some(m) = &mut self.dualsense_win {
m.apply_rich(rich)
}
}
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
PadBackend::DualShock4Windows(m) => m.apply_rich(_rich), GamepadPref::DualShock4 => {
if let Some(m) = &mut self.dualshock4_win {
m.apply_rich(rich)
}
}
_ => {}
} }
} }
/// Service feedback every cycle. `rumble` carries motor force-feedback on the universal plane /// Service feedback for every instantiated backend each cycle. `rumble` carries motor
/// (every backend); `hidout` carries rich feedback on the HID-output plane — lightbar (both /// force-feedback on the universal plane (every backend, tagged with its own pad index);
/// UHID pads), plus player LEDs / adaptive triggers (DualSense only). The X-Box pad has no /// `hidout` carries rich feedback (lightbar / player LEDs / adaptive triggers) for the UHID/UMDF
/// rich-feedback plane. /// pads. The `&mut` closure re-borrows satisfy `FnMut` for each backend.
fn pump( fn pump(
&mut self, &mut self,
rumble: impl FnMut(u16, u16, u16), mut rumble: impl FnMut(u16, u16, u16),
hidout: impl FnMut(punktfunk_core::quic::HidOutput), mut hidout: impl FnMut(punktfunk_core::quic::HidOutput),
) { ) {
match self { if let Some(m) = &mut self.xbox360 {
PadBackend::Xbox360(m) => { m.pump_rumble(&mut rumble); // the X-Box pad has no rich-feedback plane
let _ = hidout; // the X-Box pad has no rich-feedback plane
m.pump_rumble(rumble)
} }
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::DualSense(m) => m.pump(rumble, hidout), {
#[cfg(target_os = "linux")] if let Some(m) = &mut self.xboxone {
PadBackend::DualShock4(m) => m.pump(rumble, hidout), m.pump_rumble(&mut rumble);
#[cfg(target_os = "linux")] }
PadBackend::SteamDeck(m) => m.pump(rumble, hidout), if let Some(m) = &mut self.dualsense {
m.pump(&mut rumble, &mut hidout);
}
if let Some(m) = &mut self.dualshock4 {
m.pump(&mut rumble, &mut hidout);
}
if let Some(m) = &mut self.steamdeck {
m.pump(&mut rumble, &mut hidout);
}
}
#[cfg(target_os = "windows")] #[cfg(target_os = "windows")]
PadBackend::DualSenseWindows(m) => m.pump(rumble, hidout), {
#[cfg(target_os = "windows")] if let Some(m) = &mut self.dualsense_win {
PadBackend::DualShock4Windows(m) => m.pump(rumble, hidout), m.pump(&mut rumble, &mut hidout);
}
if let Some(m) = &mut self.dualshock4_win {
m.pump(&mut rumble, &mut hidout);
}
} }
} }
/// Keep a virtual UHID pad alive during input silence: re-emit its current HID report if it's /// Keep every instantiated virtual UHID/UMDF pad alive during input silence (re-emit its HID
/// gone quiet, so the kernel `hid-playstation` driver / SDL don't treat a held-steady pad as /// report so the kernel driver / SDL don't drop a held-steady pad). The X-Box pads need no
/// unplugged ("controller disconnected every few seconds"). No-op for the X-Box pad (evdev /// heartbeat (evdev holds last-known state). Per-pad gap timers inside each manager govern the
/// holds last-known state with no periodic-report requirement). Called every input-thread tick; /// actual emit cadence, not this per-tick call.
/// the per-pad gap timer (not the tick rate) governs the actual emit cadence.
fn heartbeat(&mut self) { fn heartbeat(&mut self) {
match self {
PadBackend::Xbox360(_) => {}
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
PadBackend::DualSense(m) => m.heartbeat(std::time::Duration::from_millis(8)), {
#[cfg(target_os = "linux")] let gap = std::time::Duration::from_millis(8);
PadBackend::DualShock4(m) => m.heartbeat(std::time::Duration::from_millis(8)), if let Some(m) = &mut self.dualsense {
#[cfg(target_os = "linux")] m.heartbeat(gap);
PadBackend::SteamDeck(m) => m.heartbeat(std::time::Duration::from_millis(8)), }
#[cfg(target_os = "windows")] if let Some(m) = &mut self.dualshock4 {
PadBackend::DualSenseWindows(m) => m.heartbeat(std::time::Duration::from_millis(8)), m.heartbeat(gap);
#[cfg(target_os = "windows")] }
PadBackend::DualShock4Windows(m) => m.heartbeat(std::time::Duration::from_millis(8)), if let Some(m) = &mut self.steamdeck {
m.heartbeat(gap);
} }
} }
#[cfg(target_os = "windows")]
{
let gap = std::time::Duration::from_millis(8);
if let Some(m) = &mut self.dualsense_win {
m.heartbeat(gap);
}
if let Some(m) = &mut self.dualshock4_win {
m.heartbeat(gap);
}
}
}
}
/// The per-pad routing decision for one frame ([`Pads::handle`]): given `owner` (the manager
/// holding a live device at this index, if any), the client-`declared` kind, and whether this is a
/// create/update frame (`present`) vs a removal, return `(kind to route to, new owner)`.
///
/// A live device stays in its owning manager even if the declared kind later changes (so a pad is
/// never duplicated across managers); the declared kind takes effect only when no device exists
/// yet; a removal routes to the owner's manager (so it tears the right device down) and clears the
/// owner.
fn route_decision(
owner: Option<GamepadPref>,
declared: GamepadPref,
present: bool,
) -> (GamepadPref, Option<GamepadPref>) {
match (owner, present) {
(Some(k), true) => (k, Some(k)), // keep the existing device in its manager
(Some(k), false) => (k, None), // removal → owner's manager, then clear
(None, true) => (declared, Some(declared)), // create in the declared kind's manager
(None, false) => (declared, None), // removal with no device — a harmless no-op
}
}
/// Resolve one client-declared per-pad kind to a backend this host can actually build (mixed
/// types): the platform map + the runtime UHID / Steam-conflict degrades that [`resolve_gamepad`]
/// applies to the session default, minus the Auto/env session logic (a per-pad declaration is
/// always a concrete kind).
fn resolve_pad_kind(kind: GamepadPref) -> GamepadPref {
let chosen = pick_gamepad(
kind,
None,
cfg!(target_os = "linux"),
cfg!(target_os = "windows"),
);
degrade_steam_on_conflict(degrade_if_no_uhid(chosen))
} }
/// One client→host input item, both planes on ONE channel so the input thread wakes the /// One client→host input item, both planes on ONE channel so the input thread wakes the
@@ -1956,8 +2105,9 @@ fn send_rumble(
} }
/// The per-session input thread: route pointer/keyboard events to the host-lifetime injector /// The per-session input thread: route pointer/keyboard events to the host-lifetime injector
/// service (`inj_tx`) and gamepad events to this session's [`PadBackend`] (`gamepad` — the /// service (`inj_tx`) and gamepad events to this session's [`Pads`] router (`gamepad` — the
/// resolved Hello preference: uinput X-Box pads or virtual DualSense pads), with rich /// resolved Hello preference is the per-pad default; clients declare each pad's kind so a session
/// can mix uinput X-Box pads and virtual DualSense pads), with rich
/// client→host input (touchpad / motion, [`ClientInput::Rich`]) applied on arrival and /// client→host input (touchpad / motion, [`ClientInput::Rich`]) applied on arrival and
/// feedback pumped between events — rumble on the universal datagram plane, DualSense /// feedback pumped between events — rumble on the universal datagram plane, DualSense
/// LED/trigger feedback on the HID-output plane. The gamepads are created and torn down with /// LED/trigger feedback on the HID-output plane. The gamepads are created and torn down with
@@ -1975,7 +2125,7 @@ fn input_thread(
inj_tx: std::sync::mpsc::Sender<InputEvent>, inj_tx: std::sync::mpsc::Sender<InputEvent>,
gamepad: GamepadPref, gamepad: GamepadPref,
) { ) {
let mut pads = PadBackend::select(gamepad); let mut pads = Pads::new(gamepad);
// Motion-cadence observability (debug level): inter-arrival percentiles per 5 s window, // Motion-cadence observability (debug level): inter-arrival percentiles per 5 s window,
// the measurement a "gyro feels floaty" report needs. Bounded: 5 s at even a 1 kHz pad // the measurement a "gyro feels floaty" report needs. Bounded: 5 s at even a 1 kHz pad
// is 5000 u32s. // is 5000 u32s.
@@ -2098,6 +2248,44 @@ fn input_thread(
} }
} }
} }
InputKind::GamepadRemove => {
// Mid-session hot-unplug from a snapshot-capable client (the native plane's
// `activeGamepadMask` equivalent). Seq-gated in the SAME per-pad sequence
// space as snapshots, so a snapshot the network reordered past this removal
// is dropped (older seq) and can't resurrect the pad — while a later re-plug
// on the same index arrives with a still-newer seq and is accepted. Clearing
// the `active_mask` bit and re-emitting the frame fires every backend's
// unplug sweep (`inject/*/gamepad.rs`), tearing down just this pad's device.
let (pad, seq) = punktfunk_core::input::decode_gamepad_remove(ev.flags);
let idx = pad as usize;
if idx < MAX_WIRE_PADS
&& punktfunk_core::input::GamepadSnapshot::seq_newer(seq, pad_seq[idx])
{
pad_seq[idx] = Some(seq);
if pad_mask & (1 << idx) != 0 {
pad_mask &= !(1 << idx);
pad_state[idx] = PadState::default();
let frame = pad_state[idx].frame(idx, pad_mask);
pads.handle(&crate::gamestream::gamepad::GamepadEvent::State(frame));
tracing::info!(pad = idx, "gamepad unplugged (native detach)");
}
// Fresh feedback bookkeeping so a later re-plug on this index inherits no
// stale rumble lease/seq (a lease still ticking would buzz the new pad).
rumble_state[idx] = (0, 0);
rumble_seen[idx] = false;
rumble_seq[idx] = 0;
rumble_stop_burst[idx] = 0;
}
}
InputKind::GamepadArrival => {
// Per-pad controller kind declaration (mixed types): route this pad's future
// frames to a backend of the declared kind. `code` = the GamepadPref wire byte,
// `flags` = pad index. Applied before the pad's first frame (the client sends it
// on slot open), so the device is built as the right type from the start.
let idx = ev.flags as usize;
let kind = GamepadPref::from_u8(ev.code as u8);
pads.set_kind(idx, kind);
}
_ => { _ => {
// Track press/release so a mid-press disconnect can be undone below. // Track press/release so a mid-press disconnect can be undone below.
match ev.kind { match ev.kind {
@@ -4766,6 +4954,34 @@ fn build_pipeline(
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn per_pad_route_decision() {
use GamepadPref::{DualSense, Xbox360};
// First frame with no device: create in the declared kind's manager, record ownership.
assert_eq!(
route_decision(None, DualSense, true),
(DualSense, Some(DualSense))
);
// Subsequent frame: stays in the owning manager even if the declared kind now differs
// (the arrival-after-first-frame reorder) — never a second device in another manager.
assert_eq!(
route_decision(Some(DualSense), Xbox360, true),
(DualSense, Some(DualSense))
);
// Removal (cleared bit): routes to the owner so the RIGHT device is torn down, then clears.
assert_eq!(
route_decision(Some(DualSense), Xbox360, false),
(DualSense, None)
);
// Removal with no device is a harmless no-op route (owner stays cleared).
assert_eq!(route_decision(None, Xbox360, false), (Xbox360, None));
// A fresh device after a re-plug picks up the newly-declared kind (owner was cleared).
assert_eq!(
route_decision(None, Xbox360, true),
(Xbox360, Some(Xbox360))
);
}
#[test] #[test]
fn live_mode_pack_roundtrips_and_interval_recovers_hz() { fn live_mode_pack_roundtrips_and_interval_recovers_hz() {
// The live-stats mode slot (H3): pack → unpack is exact for real modes. // The live-stats mode slot (H3): pack → unpack is exact for real modes.
+112 -1
View File
@@ -25,7 +25,10 @@
// TTL of a v2 envelope; `punktfunk_connection_next_rumble` is unchanged and drops it). Additive — // TTL of a v2 envelope; `punktfunk_connection_next_rumble` is unchanged and drops it). Additive —
// the wire is backward-compatible (the envelope is a length-tolerant tail on 0xCA), so // the wire is backward-compatible (the envelope is a length-tolerant tail on 0xCA), so
// [`WIRE_VERSION`] is unchanged. // [`WIRE_VERSION`] is unchanged.
#define ABI_VERSION 5 // v6: added the `punktfunk_reanchor_gate_*` surface (post-loss freeze-until-reanchor gate for the
// Swift client; Rust embedders use [`reanchor::ReanchorGate`] directly). Additive, client-local —
// no wire change, so [`WIRE_VERSION`] is unchanged.
#define ABI_VERSION 6
// The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check. // The punktfunk/1 **wire** version — what `Hello`/`Welcome` carry and hosts equality-check.
// Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface** // Deliberately its own constant: [`ABI_VERSION`] tracks the embeddable **C surface**
@@ -586,6 +589,23 @@
#define ColorInfo_MC_BT2020_NCL 9 #define ColorInfo_MC_BT2020_NCL 9
#endif #endif
// Consecutive no-output AUs that force a keyframe request. ~50 ms at 60 Hz — long enough not to fire
// on a one-frame decoder hiccup, short enough that a lost initial IDR (or a mid-GOP join) unfreezes
// almost immediately instead of never.
#define NO_OUTPUT_KEYFRAME_STREAK 3
// How many host intra-refresh recovery marks ([`USER_FLAG_RECOVERY_POINT`]) must arrive since the
// latest loss before the gate lifts its freeze on an IDR-free stream. TWO, not one: with a continuous
// rolling wave the host marks phase-fixed wave boundaries, so the FIRST boundary after a loss is only
// partially healed — stripes swept BEFORE the loss still reference the lost frame — and lifting there
// would flash a partially-stale picture. The SECOND boundary guarantees a full wave swept entirely
// after the loss, so the picture is clean. This stays correct under repeated loss because every fresh
// arm resets the count. The cost is up to ~2 wave periods of holding the last good frame — the
// deliberate "hold longer, never show garbage" trade.
//
// [`USER_FLAG_RECOVERY_POINT`]: crate::packet::USER_FLAG_RECOVERY_POINT
#define REANCHOR_MARKS_TO_LIFT 2
// Stable C ABI status codes. `Ok` is 0; all errors are negative so callers can // Stable C ABI status codes. `Ok` is 0; all errors are negative so callers can
// test `rc < 0`. Do not renumber existing variants — only append. // test `rc < 0`. Do not renumber existing variants — only append.
enum PunktfunkStatus enum PunktfunkStatus
@@ -658,6 +678,28 @@ enum PunktfunkInputKind
// [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); older hosts keep // [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); older hosts keep
// receiving the per-transition events. // receiving the per-transition events.
PUNKTFUNK_INPUT_KIND_GAMEPAD_STATE = 12, PUNKTFUNK_INPUT_KIND_GAMEPAD_STATE = 12,
// A pad was unplugged client-side (the native plane's answer to GameStream's
// `activeGamepadMask`, which the per-transition/snapshot planes otherwise lack — see
// [`encode_gamepad_remove`]). `flags` packs `seq << 24 | pad`: the low byte is the pad
// index, the high byte a per-pad wrapping seq sharing the [`GamepadSnapshot`] sequence
// space. The host clears the pad's `active_mask` bit so its virtual device is torn down,
// seq-gated against snapshots so one the network reordered past the removal can't resurrect
// the pad, and the shared seq space keeps the same index reusable by a later re-plug. Sent
// only to a host that advertised [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE);
// an older host ignores the unknown tag (the pad then lingers until session end — the
// pre-existing behaviour).
PUNKTFUNK_INPUT_KIND_GAMEPAD_REMOVE = 13,
// Declares which controller KIND a pad presents so a session can MIX types (pad 0 a
// DualSense, pad 1 an Xbox pad). `code` = the [`GamepadPref`](crate::config::GamepadPref)
// wire byte, `flags` = pad index. Sent when the client opens a pad slot — before that pad's
// first input — and re-sent a few times against datagram loss (like [`GamepadRemove`]). The
// host resolves the kind to a buildable backend and routes that pad's virtual device to it; a
// pad the client never declares (an older client, or a fully-lost declaration) falls back to
// the session-default kind from the handshake. Idempotent (no seq): re-declaring the same kind
// is a no-op. Meaningful only to a host that advertised
// [`HOST_CAP_GAMEPAD_STATE`](crate::quic::HOST_CAP_GAMEPAD_STATE); an older host ignores the
// unknown tag (every pad then uses the session-default kind — the pre-existing behaviour).
PUNKTFUNK_INPUT_KIND_GAMEPAD_ARRIVAL = 14,
}; };
#ifndef __cplusplus #ifndef __cplusplus
#if __STDC_VERSION__ >= 202311L #if __STDC_VERSION__ >= 202311L
@@ -692,6 +734,18 @@ typedef struct PunktfunkConnection PunktfunkConnection;
// Opaque session handle. Pointer-only from C. // Opaque session handle. Pointer-only from C.
typedef struct PunktfunkSession PunktfunkSession; typedef struct PunktfunkSession PunktfunkSession;
// The shared post-loss freeze state machine. A client feeds it three kinds of event — an *arm* (a
// loss was detected: a frame-index gap, a dropped-count climb, or a decoder wedge/demotion), each
// *decoded frame* ([`on_decoded`](Self::on_decoded), which decides present-vs-hold and interprets the
// re-anchor wire flags), and each *no-output* AU ([`on_no_output`](Self::on_no_output)) — plus a
// periodic [`poll`](Self::poll) that folds the dropped counter and fires the overdue backstop.
//
// The gate emits *intents* only: [`on_no_output`](Self::on_no_output) and [`poll`](Self::poll) return
// `true` when the client should ask the host for a keyframe. The client routes that through its own
// ~100 ms request throttle (and the precise RFI-vs-keyframe range decision stays in the loss-range
// tracker behind [`crate::client::NativeClient::note_frame_index`]) — the gate never touches the wire.
typedef struct ReanchorGate ReanchorGate;
// Forward-compatible session configuration. The caller MUST set `struct_size` to // Forward-compatible session configuration. The caller MUST set `struct_size` to
// `sizeof(PunktfunkConfig)`; the core uses it to detect ABI skew. // `sizeof(PunktfunkConfig)`; the core uses it to detect ABI skew.
typedef struct { typedef struct {
@@ -1715,6 +1769,63 @@ void punktfunk_connection_disconnect_quit(PunktfunkConnection *c);
void punktfunk_connection_close(PunktfunkConnection *c); void punktfunk_connection_close(PunktfunkConnection *c);
#endif #endif
// Create a re-anchor gate seeded with the session's current `frames_dropped` (so the first
// [`punktfunk_reanchor_gate_poll`] doesn't read the baseline as a loss). Free with
// [`punktfunk_reanchor_gate_free`]. Never returns NULL.
ReanchorGate *punktfunk_reanchor_gate_new(uint64_t frames_dropped);
// Free a gate created by [`punktfunk_reanchor_gate_new`]. NULL is a no-op.
//
// # Safety
// `g` was returned by [`punktfunk_reanchor_gate_new`] and is not used after this call.
void punktfunk_reanchor_gate_free(ReanchorGate *g);
// Arm the freeze: a loss was detected (a frame-index gap, or a decoder wedge/demotion). Zeroes the
// recovery-mark count and (re-)sets the backstop deadline. NULL is a no-op.
//
// # Safety
// `g` is a valid gate handle.
void punktfunk_reanchor_gate_arm(ReanchorGate *g);
// Fold one decoded frame and write to `out_present` whether to display it (`true`) or withhold it as
// a post-loss concealment (`false`). `flags` is the AU's `user_flags` word ([`PunktfunkFrame::flags`]):
// the gate reads `FLAG_SOF` (the host's IDR marker), `USER_FLAG_RECOVERY_ANCHOR` and
// `USER_FLAG_RECOVERY_POINT`. Pass `decoder_keyframe = false` where the platform decoder doesn't flag
// IDRs (VideoToolbox/MediaCodec) — the wire `FLAG_SOF` covers it.
//
// # Safety
// `g` is a valid gate handle; `out_present` is writable or NULL.
PunktfunkStatus punktfunk_reanchor_gate_on_decoded(ReanchorGate *g,
uint32_t flags,
bool decoder_keyframe,
bool *out_present);
// A received AU produced no decoded frame. Writes to `out_request_kf` whether the no-output streak has
// tripped and the client should (throttled) request a keyframe — the gate arms the freeze at the same
// time.
//
// # Safety
// `g` is a valid gate handle; `out_request_kf` is writable or NULL.
PunktfunkStatus punktfunk_reanchor_gate_on_no_output(ReanchorGate *g,
bool *out_request_kf);
// Periodic fold of the session's `frames_dropped` counter plus the overdue backstop. Writes to
// `out_request_kf` whether the client should (throttled) request a keyframe (a drop-count climb armed
// a fresh freeze, or the freeze is overdue and re-asks while it keeps holding).
//
// # Safety
// `g` is a valid gate handle; `out_request_kf` is writable or NULL.
PunktfunkStatus punktfunk_reanchor_gate_poll(ReanchorGate *g,
uint64_t frames_dropped,
bool *out_request_kf);
// Whether the gate is currently withholding concealed frames (frozen on the last good picture).
// Writes `false` on a NULL gate.
//
// # Safety
// `g` is a valid gate handle; `out_holding` is writable or NULL.
PunktfunkStatus punktfunk_reanchor_gate_is_holding(const ReanchorGate *g, bool *out_holding);
#ifdef __cplusplus #ifdef __cplusplus
} // extern "C" } // extern "C"
#endif // __cplusplus #endif // __cplusplus
+5 -1
View File
@@ -73,7 +73,11 @@ build() {
# DT_NEEDED as a plain build (no libcuda/libnvidia-encode), so the binary starts fine driver-less # DT_NEEDED as a plain build (no libcuda/libnvidia-encode), so the binary starts fine driver-less
# and only touches NVIDIA on a CUDA capture frame (default on NVIDIA; PUNKTFUNK_NVENC_DIRECT=0 # and only touches NVIDIA on a CUDA capture frame (default on NVIDIA; PUNKTFUNK_NVENC_DIRECT=0
# opts back to libav). AMD/Intel never reach it — the `cuda` gate leaves them on VAAPI. # opts back to libav). AMD/Intel never reach it — the `cuda` gate leaves them on VAAPI.
cargo build --release --locked --features punktfunk-host/nvenc \ # `punktfunk-host/vulkan-encode` is the AMD/Intel twin: a raw VK_KHR_video_encode_h265 backend with
# real RFI (clean P-frame recovery anchor via DPB reference slots; design/linux-vulkan-video-encode.md).
# Pure Rust `ash` (no new lib, no link-time deps); default on for HEVC (PUNKTFUNK_VULKAN_ENCODE=0
# opts back to libav VAAPI), and a failed open falls back to VAAPI so unsupported devices are safe.
cargo build --release --locked --features punktfunk-host/nvenc,punktfunk-host/vulkan-encode \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-tray -p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-tray
# Management web console (opt-in): the Nitro `bun`-preset .output bundle (Bun.serve TLS), # Management web console (opt-in): the Nitro `bun`-preset .output bundle (Bun.serve TLS),
# built AND run with bun. # built AND run with bun.
+5 -1
View File
@@ -180,7 +180,11 @@ export PUNKTFUNK_BUILD_VERSION="%{version}-%{release}"
# at runtime (no link-time dep; __requires_exclude already drops libcuda), so the binary starts # at runtime (no link-time dep; __requires_exclude already drops libcuda), so the binary starts
# driver-less; the encoder engages only on a CUDA frame (default on NVIDIA; PUNKTFUNK_NVENC_DIRECT=0 # driver-less; the encoder engages only on a CUDA frame (default on NVIDIA; PUNKTFUNK_NVENC_DIRECT=0
# opts back to libav) — the `cuda` gate keeps AMD/Intel on VAAPI regardless. # opts back to libav) — the `cuda` gate keeps AMD/Intel on VAAPI regardless.
cargo build --release --locked --features punktfunk-host/nvenc \ # --features punktfunk-host/vulkan-encode: the AMD/Intel twin — a raw VK_KHR_video_encode_h265 backend
# with real RFI (clean P-frame recovery anchor via DPB reference slots; design/linux-vulkan-video-encode.md).
# Pure Rust `ash` (no new lib / no link-time dep); default on for HEVC (PUNKTFUNK_VULKAN_ENCODE=0 opts
# back to libav VAAPI), and a failed open falls back to VAAPI so unsupported devices degrade gracefully.
cargo build --release --locked --features punktfunk-host/nvenc,punktfunk-host/vulkan-encode \
-p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-tray -p punktfunk-host -p punktfunk-client-linux -p punktfunk-client-session -p punktfunk-tray
%if %{with web} %if %{with web}
@@ -146,6 +146,12 @@ pub struct FramePublisher {
/// Set when a surface is dropped for a descriptor mismatch (a game mode-set the display), cleared on a /// Set when a surface is dropped for a descriptor mismatch (a game mode-set the display), cleared on a
/// matched publish — throttles the drop log to once per mismatch episode (game-capture bug GB1). /// matched publish — throttles the drop log to once per mismatch episode (game-capture bug GB1).
mismatch_logged: bool, mismatch_logged: bool,
/// The render adapter (LUID) this publisher's device + opened ring textures live on. A worker
/// re-adopts a publisher preserved across a swap-chain unassign→reassign flap ONLY when the
/// freshly-assigned swap-chain renders on this SAME adapter (else the opened textures would be
/// cross-device); see [`Self::render_adapter`] + `swap_chain_processor::run_core`.
render_luid_low: u32,
render_luid_high: i32,
} }
// SAFETY: created and used only on the swap-chain processor thread. // SAFETY: created and used only on the swap-chain processor thread.
@@ -356,6 +362,8 @@ impl FramePublisher {
ring_format: unsafe { (*header).dxgi_format }, ring_format: unsafe { (*header).dxgi_format },
generation: header_gen, generation: header_gen,
mismatch_logged: false, mismatch_logged: false,
render_luid_low,
render_luid_high,
}) })
} }
@@ -379,6 +387,13 @@ impl FramePublisher {
cur != self.generation cur != self.generation
} }
/// The render adapter (LUID) this publisher's device + opened ring textures live on. The swap-chain
/// worker re-adopts a publisher preserved across an unassign→reassign flap only when the freshly-
/// assigned swap-chain renders on this same adapter (see the field docs + `run_core`).
pub fn render_adapter(&self) -> (u32, i32) {
(self.render_luid_low, self.render_luid_high)
}
/// Copy `surface` into the next free ring slot and signal the host. Never blocks (0 ms try-acquire). /// Copy `surface` into the next free ring slot and signal the host. Never blocks (0 ms try-acquire).
pub fn publish(&mut self, surface: &ID3D11Texture2D) { pub fn publish(&mut self, surface: &ID3D11Texture2D) {
let ring_len = self.slots.len() as u32; let ring_len = self.slots.len() as u32;
@@ -58,6 +58,17 @@ pub struct MonitorObject {
/// closes an unconsumed channel's handles via [`FrameChannel`]'s `Drop`, so no delivery can leak /// closes an unconsumed channel's handles via [`FrameChannel`]'s `Drop`, so no delivery can leak
/// handles in the WUDFHost table whatever the monitor's fate. /// handles in the WUDFHost table whatever the monitor's fate.
pub frame_channel: Option<crate::frame_transport::FrameChannel>, pub frame_channel: Option<crate::frame_transport::FrameChannel>,
/// A live [`FramePublisher`](crate::frame_transport::FramePublisher) preserved across a swap-chain
/// unassign→reassign flap (STEP 6 sibling-join fix). The OS unassigns a monitor's swap-chain
/// whenever a SIBLING display churns the desktop topology (a second client joining / leaving /
/// resizing), which drops the swap-chain worker — but the HOST-owned ring (header / event /
/// textures) the publisher holds stays valid, and the host only re-delivers the frame channel on a
/// ring RECREATE (a descriptor change), so a fresh worker had nothing to re-attach from and the
/// first client's stream froze (repeat frames forever). The exiting worker stashes its still-live
/// publisher here ([`preserve_publisher`]); the next worker on the SAME render adapter takes it back
/// ([`take_preserved_publisher`]) and resumes publishing into the same ring. Dropped with the
/// `MonitorObject` on teardown (closing its ring handles) if no worker ever reclaims it.
pub preserved_publisher: Option<crate::frame_transport::FramePublisher>,
/// When the entry was created — the watchdog skips still-initializing monitors. /// When the entry was created — the watchdog skips still-initializing monitors.
pub created_at: Instant, pub created_at: Instant,
} }
@@ -318,6 +329,49 @@ pub fn has_frame_channel(target_id: u32) -> bool {
.any(|m| m.target_id == target_id && m.frame_channel.is_some()) .any(|m| m.target_id == target_id && m.frame_channel.is_some())
} }
/// Stash a swap-chain worker's still-live [`FramePublisher`](crate::frame_transport::FramePublisher) on
/// its monitor across a swap-chain unassign→reassign flap (STEP 6 sibling-join fix; see the field docs
/// on [`MonitorObject::preserved_publisher`]). Called from the EXITING worker thread — the caller must
/// NOT hold `MONITOR_MODES` (this locks it), matching the same drop-outside-the-lock discipline the
/// processor teardown paths use. Returns `Err(publisher)` when no monitor with `target_id` exists (a
/// genuine teardown, not a flap: the entry was already removed) so the caller drops it, closing the ring
/// handles. Replacing an already-stashed publisher (should not happen — one worker exits at a time)
/// drops the old one, so it can never accumulate. Returning the publisher in the `Err` makes the
/// `Result` itself `#[must_use]`, so a caller can't silently drop the not-preserved publisher.
pub fn preserve_publisher(
target_id: u32,
publisher: crate::frame_transport::FramePublisher,
) -> Result<(), crate::frame_transport::FramePublisher> {
if target_id == 0 {
return Err(publisher);
}
let mut lock = lock_monitors();
if let Some(m) = lock.iter_mut().find(|m| m.target_id == target_id) {
m.preserved_publisher = Some(publisher);
Ok(())
} else {
Err(publisher)
}
}
/// Take (remove) a preserved [`FramePublisher`](crate::frame_transport::FramePublisher) for a freshly-
/// (re)assigned swap-chain worker (STEP 6 sibling-join fix). The caller re-adopts it ONLY when the new
/// swap-chain's render adapter matches the publisher's ([`FramePublisher::render_adapter`]) — same
/// pooled device, so its context + opened ring textures are still valid; on a mismatch the caller drops
/// it and waits for a fresh channel delivery instead. `None` until a worker has stashed one.
pub fn take_preserved_publisher(
target_id: u32,
) -> Option<crate::frame_transport::FramePublisher> {
if target_id == 0 {
return None;
}
lock_monitors()
.iter_mut()
.find(|m| m.target_id == target_id)?
.preserved_publisher
.take()
}
/// Install a swap-chain processor on the monitor whose handle matches, returning any PREVIOUS processor /// Install a swap-chain processor on the monitor whose handle matches, returning any PREVIOUS processor
/// for the caller to drop OUTSIDE the lock. Dropping a processor RAII-joins its worker thread, so it must /// for the caller to drop OUTSIDE the lock. Dropping a processor RAII-joins its worker thread, so it must
/// never happen while holding `MONITOR_MODES` (the worker would block the whole control plane / risk a /// never happen while holding `MONITOR_MODES` (the worker would block the whole control plane / risk a
@@ -406,6 +460,7 @@ pub fn create_monitor(
adapter_luid_high: 0, adapter_luid_high: 0,
swap_chain_processor: None, swap_chain_processor: None,
frame_channel: None, frame_channel: None,
preserved_publisher: None,
created_at: Instant::now(), created_at: Instant::now(),
}); });
id id
@@ -241,7 +241,31 @@ impl SwapChainProcessor {
// the values via IOCTL_SET_FRAME_CHANNEL, which the control plane stashes on our monitor // the values via IOCTL_SET_FRAME_CHANNEL, which the control plane stashes on our monitor
// (`monitor::take_frame_channel`). Until a delivery lands we just drain — exactly the STEP-5 // (`monitor::take_frame_channel`). Until a delivery lands we just drain — exactly the STEP-5
// behaviour — so a non-IDD-push session never stalls. The stash is polled every ~30 iterations. // behaviour — so a non-IDD-push session never stalls. The stash is polled every ~30 iterations.
let mut publisher: Option<FramePublisher> = None; // STEP 6 sibling-join fix: re-adopt a FramePublisher PRESERVED across a swap-chain
// unassign→reassign flap. When a SIBLING display churns the desktop topology (a second client
// joining / leaving / resizing), the OS reassigns THIS monitor's swap-chain and the previous
// worker exited — but the host-owned ring it published into is still live, and the host only
// re-delivers the frame channel on a ring RECREATE (a descriptor change). Without this the
// fresh worker has nothing to attach to and the first client's stream freezes (repeat frames
// forever). Re-adopt ONLY when the freshly-assigned swap-chain renders on the SAME adapter as
// the preserved publisher (same pooled Direct3DDevice → its immediate context + opened ring
// textures are valid); a mismatch drops it and falls back to a fresh channel delivery. A
// preserved publisher that the host superseded meanwhile (ring recreate → is_stale, or a
// pending delivery) is dropped + replaced by the existing re-attach logic at the loop top.
let mut publisher: Option<FramePublisher> =
crate::monitor::take_preserved_publisher(target_id).and_then(|p| {
if p.render_adapter() == (render_luid_low, render_luid_high) {
dbglog!(
"[pf-vd] swap-chain run_core: re-adopted preserved publisher (target={target_id}) — resuming the host ring across the swap-chain flap"
);
Some(p)
} else {
dbglog!(
"[pf-vd] swap-chain run_core: preserved publisher's render adapter changed (target={target_id}) — dropping it, will re-attach from a fresh channel"
);
None
}
});
let mut frames_since_try: u32 = u32::MAX; // attach attempt on the first loop iteration let mut frames_since_try: u32 = u32::MAX; // attach attempt on the first loop iteration
let mut logged_pending = false; let mut logged_pending = false;
@@ -392,6 +416,17 @@ impl SwapChainProcessor {
break; break;
} }
} }
// STEP 6 sibling-join fix: the drain loop exited (the OS unassigned this swap-chain — typically
// because a SIBLING display churned the desktop topology — or it errored), but the host-owned
// ring the publisher holds is still live. Hand it to the monitor so the NEXT worker assigned to
// this monitor resumes publishing into the same ring instead of freezing (the host re-delivers
// the channel only on a ring recreate). If the monitor is GONE (a genuine teardown, not a flap),
// `preserve_publisher` hands the publisher back inside the `Err` and dropping the returned
// `Result` closes the ring handles here — no leak, no stale ring left behind.
if let Some(p) = publisher.take() {
let _ = crate::monitor::preserve_publisher(target_id, p);
}
} }
} }