42 Commits

Author SHA1 Message Date
enricobuehler caa7a1c735 chore(release): bump workspace version to 0.7.1
apple / swift (push) Successful in 1m5s
audit / cargo-audit (push) Successful in 16s
ci / web (push) Successful in 1m6s
ci / docs-site (push) Successful in 1m24s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 48s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 52s
ci / bench (push) Successful in 5m1s
android-screenshots / screenshots (push) Successful in 47s
ci / rust (push) Successful in 9m48s
windows-host / package (push) Successful in 7m10s
android / android (push) Successful in 3m53s
release / apple (push) Successful in 8m47s
decky / build-publish (push) Successful in 15s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m20s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m14s
deb / build-publish (push) Successful in 2m59s
apple / screenshots (push) Successful in 5m42s
flatpak / build-publish (push) Successful in 4m11s
linux-client-screenshots / screenshots (push) Successful in 1m39s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 10m19s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 10m19s
docker / deploy-docs (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
web-screenshots / screenshots (push) Successful in 2m34s
Cut 0.7.1 to test the GTK + Decky polish batch (57ae00a) on-device: the
host-click / disconnect-chord / gray-screen-recovery / leak fixes on the
Linux client, the Deck launcher perf profile, and the Decky pin/pairing
fixes. The [workspace.package] version (inherited by every crate via
version.workspace) is the release being cut; refresh the 9 workspace
entries in Cargo.lock to match (CI builds --locked). Canary derives from
the tag (scripts/ci/pf-version.sh), so cutting v0.7.1 auto-advances canary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 07:47:58 +00:00
enricobuehler 13dc7fc49f style(linux): rustfmt the keyframe-recovery throttle line
apple / swift (push) Successful in 1m9s
android / android (push) Has been cancelled
apple / screenshots (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
ci / rust (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
flatpak / build-publish (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
Wrap the `last_kf_req.is_none_or(...)` guard to satisfy `cargo fmt --all
--check` (CI Format step).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 07:47:16 +00:00
enricobuehler 57ae00a9c8 fix(clients): GTK + Decky polish batch from live Deck/Windows testing
ci / rust (push) Failing after 41s
apple / swift (push) Successful in 1m8s
ci / web (push) Successful in 55s
ci / docs-site (push) Successful in 1m6s
android / android (push) Successful in 3m20s
deb / build-publish (push) Successful in 2m55s
decky / build-publish (push) Successful in 27s
apple / screenshots (push) Successful in 5m46s
ci / bench (push) Successful in 5m5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 3m20s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m31s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 53s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m18s
docker / deploy-docs (push) Has been cancelled
flatpak / build-publish (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
GTK Linux client:
- hosts/library: clicking a card was dead — the handler was on
  FlowBoxChild::activate (never emitted on click); bridge child-activated
  → child.activate() on the FlowBox (ui_hosts, ui_library).
- stream: the Ctrl+Alt+Shift+D/Q/S chords (and all key forwarding) were
  dropped because the key controller sat on the overlay, which loses focus
  to the header back button after nav.push+fullscreen — move it to the
  window and remove it on teardown.
- video: a mid-session VAAPI decode error rebuilt a software decoder but
  never requested a keyframe, so under the infinite GOP the picture stayed
  gray/frozen forever. Request an IDR on any VAAPI error, keep the hardware
  decoder, and demote to software only after repeated failures.
- stream: fix a per-session Capture↔overlay reference cycle that leaked the
  overlay subtree + the Arc<NativeClient> on every session end — hold the
  overlay weakly.
- stream: accumulate the fractional wheel remainder so precision-scroll
  (Deck trackpad / hi-res wheels) sub-unit deltas aren't dropped.
- gamepad library: keep the launcher smooth on the Deck — freeze the aurora
  and trim the visible card range (fewer 3D offscreen passes) on low-power.
- gamepad: log full pad identity (vid:pid:name:type:virtual) on attach to
  diagnose an empty controller list on the Deck.
- cli: --connect host:<badport> silently did nothing; default to 9777 + warn.
- css: add the missing .pf-neutral pill rule; fix the clipped most-recent
  accent (inset outline instead of a corner-clipped box-shadow bar).

Decky plugin:
- surface the on-screen library browser: label the host-row Games button.
- fix silent pin data-loss — the detached Games modal captured a frozen
  pins array, so pinning a second game clobbered the first; mirror pins in
  a ref and track the modal's pinned ids locally for a live label.
- route pair-required hosts through the pairing modal from the fullscreen
  Stream button (parity with the QAM panel).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 07:37:04 +00:00
enricobuehler 882a3d57f6 chore(release): bump workspace version to 0.7.0
android-screenshots / screenshots (push) Successful in 2m12s
windows-host / package (push) Successful in 6m43s
android / android (push) Successful in 9m24s
release / apple (push) Successful in 9m35s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m8s
apple / swift (push) Successful in 1m10s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m7s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 50s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 53s
linux-client-screenshots / screenshots (push) Successful in 1m41s
apple / screenshots (push) Successful in 5m37s
web-screenshots / screenshots (push) Successful in 2m37s
audit / cargo-audit (push) Successful in 1m12s
ci / web (push) Successful in 1m8s
ci / docs-site (push) Successful in 1m22s
ci / rust (push) Successful in 4m34s
deb / build-publish (push) Successful in 3m0s
ci / bench (push) Successful in 4m57s
decky / build-publish (push) Successful in 12s
flatpak / build-publish (push) Successful in 4m7s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 10m2s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m14s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m42s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m15s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 54s
docker / deploy-docs (push) Successful in 20s
The [workspace.package] version (inherited by every crate via
version.workspace) is the release being cut. Refresh the 9 workspace entries
in Cargo.lock to match (CI builds --locked). Canary derives from the tag now
(scripts/ci/pf-version.sh), so no canary-base edit is needed — cutting v0.7.0
auto-advances canary to 0.8.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 22:45:45 +00:00
enricobuehler fa28fa19a0 Merge remote-tracking branch 'origin/main'
android / android (push) Has been cancelled
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
windows-host / package (push) Has been cancelled
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Has been cancelled
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
flatpak / build-publish (push) Has been cancelled
release / apple (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
2026-07-03 22:41:42 +00:00
enricobuehler 42595b5558 fix(flatpak): keep both channels in the OSTree summary (fixes stable "No such ref")
The surfaced install command
  flatpak install --user https://flatpak.unom.io/io.unom.Punktfunk.flatpakref
failed with "No such ref 'app/io.unom.Punktfunk/x86_64/stable'". The stable
commit's objects are on the server, but the repo *summary* (what flatpak reads
to resolve refs) listed only canary.

Root cause: each CI run builds a fresh SINGLE-branch local OSTree repo,
build-update-repo regenerates the summary from that one branch, and rsync
uploads it without --delete. Objects for both channels accumulate, but the
summary is overwritten every run and only names that run's branch. Canary runs
on every main push, stable only on tags — so a tag published stable, then the
next canary push clobbered the summary back to canary-only.

Fix: seed the local repo from the live server (rsync repo/ DOWN) before the
build, so it carries every published branch; the build only adds this run's
commit and the regenerated+signed summary keeps both channels. Single shared
repo kept (no URL/Caddyfile change; existing installs fixed transparently).
Adds a refs log after build-update-repo as a clobber tripwire. Also adopts
scripts/ci/pf-version.sh for the canary base (see previous commit).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 22:40:35 +00:00
enricobuehler 4de543c146 ci(release): derive canary version from git tags (single source of truth)
Every release workflow hardcoded a canary base version (0.5.0 in
Apple/Android/rpm/flatpak/deb, 0.3 in windows-msix/windows-host/decky) that
had to be hand-bumped on each stable release and wasn't. With stable at
v0.6.0, every canary was a version *behind* stable — e.g. the Apple canary
showed up on TestFlight as 0.5.0 while 0.6.0 was already published.

Add scripts/ci/pf-version.{sh,ps1} (bash + pwsh twin) as the single source of
truth: stable = the vX.Y.Z tag; canary = latest stable tag with minor+1,
patch 0 (v0.6.0 -> 0.7.0), so canary is always exactly one minor ahead of the
newest release with zero maintenance. Falls back to the workspace Cargo.toml
version when no tag is fetchable. All workflows now eval/call it and format
their own channel suffix off $PF_BASE; only the canary branch changed, stable
branches and per-channel suffixes are untouched. channels.md drops the old
manual "bump the canary base" release step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 22:40:25 +00:00
enricobuehler 42d1c74663 fix(apple-client/audio): capture the right channel of a multi-channel mic + diagnostics
apple / swift (push) Successful in 1m5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
android / android (push) Has been cancelled
apple / screenshots (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
ci / rust (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
release / apple (push) Has been cancelled
The mic uplink handed the host pure digital silence on a multi-channel
interface: AVAudioConverter's N→stereo downmix takes channels 0/1, but a
pro interface puts the mic on ONE higher discrete channel. Fold the input
to a mono bus ourselves instead — pick the mic's channel (or sum all) and
resample that to the encoder's 48 kHz stereo, so the silent 0/1 downmix
never happens.

- New "Microphone channel" setting (macOS): Auto (sum every channel — a
  lone hot mic passes at full level) or pin 1-based channel N. Picker
  appears only for multi-channel devices, driven by the device's input
  channel count.
- Diagnostics that make this class of failure self-naming next session:
  log the actual live capture device + format + fold mode, warn on a
  silent UID fallback, and a one-shot silence tripwire on the EXTRACTED
  signal (WARN on 10 s of zeros, else peak dBFS).
- foldToMono extracted as a pure, unit-tested helper (pin / sum-clamp x
  interleaved / deinterleaved / mono / out-of-range).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 00:36:35 +02:00
enricobuehler 136f6e8f0e feat(probe): --mic-burst — real-client mic pacing for jitter-buffer regression tests
apple / swift (push) Successful in 1m8s
android / android (push) Has been cancelled
apple / screenshots (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
ci / rust (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
The steady 5 ms mic-test cadence never trips host-side buffering bugs:
the WASAPI crackle (fixed in the previous commit) only reproduced under
a real client's bursty input tap. --mic-burst paces the tone the same
way (two 20 ms Opus packets every 40 ms), so recording the host mic and
counting silence gaps regression-tests the jitter buffer headlessly.
Validated against the fixed Windows host on the lab box: 15 s of bursty
tone, zero mid-stream gaps >=3 ms (gaps confined to the first 40 ms
priming window).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 22:35:49 +00:00
enricobuehler 00acf5e44e fix(host/audio): WASAPI virtual mic — port the priming jitter buffer (crackling fix)
apple / swift (push) Successful in 1m8s
ci / rust (push) Successful in 1m56s
apple / screenshots (push) Successful in 5m17s
ci / bench (push) Successful in 4m41s
decky / build-publish (push) Successful in 24s
ci / web (push) Successful in 59s
android / android (push) Successful in 3m41s
ci / docs-site (push) Successful in 1m0s
windows-host / package (push) Successful in 7m6s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m39s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 1m7s
deb / build-publish (push) Successful in 9m15s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Has been cancelled
Mac → Windows mic passthrough crackled heavily while the identical
stream was clean on the Linux host. Cause: clients push mic audio in
BURSTS on their own clock (the Mac input tap yields ~two 20 ms Opus
packets every ~42 ms) while the WASAPI render loop pulled a block every
~10 ms device period and greedily drained whatever was queued, padding
the rest with zeros — the queue sat near-empty and most periods
inserted mid-stream silence. The Linux backend has absorbed this since
day one with its priming jitter buffer; the WASAPI loop had none.

Port the same semantics: emit silence until ~48 ms is buffered (covers
the worst inter-burst gap), then play from the cushion (zero-filling
only a momentary shortfall), re-prime only after a genuine full drain
(client went quiet). Queue cap raised 80 → 120 ms for burst headroom;
steady-state added latency ≈ the 48 ms cushion.

Diagnosed live on .173: probe tone recording from CABLE Output proved
the endpoint wiring, then the burst-vs-period math explained the
crackle. Build-verified on Windows; on-glass listen pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 22:31:18 +00:00
enricobuehler 38078fe7ee feat(linux-client): gamepad library launcher — console-style coverflow (--browse)
ci / rust (push) Successful in 1m54s
ci / web (push) Successful in 54s
ci / docs-site (push) Successful in 1m2s
windows-host / package (push) Successful in 6m43s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m12s
ci / bench (push) Successful in 4m47s
apple / swift (push) Successful in 1m9s
android / android (push) Successful in 3m33s
deb / build-publish (push) Successful in 4m36s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 6s
decky / build-publish (push) Successful in 15s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m10s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 51s
release / apple (push) Successful in 8m30s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 48s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 53s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m15s
flatpak / build-publish (push) Successful in 4m4s
apple / screenshots (push) Successful in 5m31s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m48s
docker / deploy-docs (push) Successful in 24s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m20s
A controller-driven, chrome-less library launcher for the Steam Deck flow
(the Decky plugin's "Open library on screen" + pinned games, 8470419):
`--browse host[:port]` opens a paired host's game library as a coverflow
over a drifting aurora — A streams the focused title (the id rides the
Hello), session end returns to the launcher, B quits back to Gaming Mode.
`--connect` gains `--launch <id>` for direct-to-game starts; `--mgmt`
overrides the library port. Scope is deliberately library-only: host
selection/settings stay in the touch UI, pairing stays in the plugin (no
dialog can map under gamescope — every state renders in-page).

- gamepad.rs menu mode: the worker holds the active pad open while idle
  (WITHOUT the Valve HIDAPI drivers — Deck lizard mode survives) and
  translates it through a pure MenuNav state machine: edge-triggered
  buttons, held-state snapshot on entry/detach (the escape chord that ends
  a stream can't ghost-fire in the menu), 380/160 ms stick/dpad repeat,
  menu rumble ticks. Keyboard fallback (arrows/Enter/Esc) drives the same
  handler — fully usable with no pad, no host (PUNKTFUNK_FAKE_LIBRARY).
- Coverflow: ±38° corridor-facing tilt under per-card perspective
  (gsk rotate_3d), dense overlapping side shelves with paint-order
  restacking (gtk::Fixed draws in child order), opaque card faces + a
  darkening veil for the recede (translucency would bleed the stack
  through). The strip lives in an External-policy ScrolledWindow because
  a bare gtk::Fixed measures its TRANSFORMED children and inflates the
  page min-width past the window.
- Spring-driven motion: semi-implicit Euler in ≤8 ms substeps (a raw
  50 ms frame leaves the stiff recoil spring ringing at ω·dt ≈ 1.2 —
  regression-tested), ζ≈0.85 cursor chase + ζ≈0.55 boundary wobble;
  velocity carries across retargets so held-repeat scrolling glides.
- Shot scene `gamepad-library` (GTK animations force-disabled in shot mode
  — nav transitions froze mid-slide in headless captures); shared poster
  fetch extracted to library::spawn_art_fetch.

Verified here: 21 unit tests (MenuNav, cursor stepping, spring
convergence/stability), clippy -D warnings clean, screenshot scene
pixel-checked, --browse smoke runs (fake-library + unpaired) on the
headless session. On-Deck validation pending (virtual-pad input, lizard
mode, rumble via Steam Input, full Decky→browse→stream→launcher loop).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 21:41:43 +00:00
enricobuehler 69609945a3 feat(clients): host/network split in every stats HUD (stats phase 2, client side)
Consumes the 0xCF host-timing plane (449a67c) on all four GUI clients: each
keeps a bounded pending ring of receipt samples keyed by pts, matches the
host's per-AU capture→sent reports against it, and the HUD equation becomes

  = host 3.1 + network 6.7 + decode 2.1 + display 2.3

falling back to the combined `= host+network …` term whenever no timing
matched the window (old host / datagram loss) — same total, one split
fewer, never a misleading zero. Apple additionally gains the split as the
only equation line under the stage-1 fallback presenter (receipt is
presenter-independent), a `nextHostTiming` wrapper with its own plane lock,
and a unit-tested `HostNetworkSplitter`; Android extends the JNI stats
array 16→18 doubles (0–15 unchanged); Windows/Linux thread the split
through `Stats` into the HUD and the headless/debug logs.

Docs updated: design/stats-unification.md Phase 2 → implemented (wire
format, fallback semantics), and the docs-site matrix's Sunshine "Host
processing latency" row is now a direct match (ours includes the paced
send; avg vs p50).

Verified here: linux client clippy -D warnings green on the live tree,
windows stub check + hand-verified diff, android cargo-ndk arm64 check
green, apple loopback test extended (needs the rebuilt xcframework + swift
test on the mac). On-glass: pending on all platforms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 21:31:49 +00:00
enricobuehler 8470419433 feat(decky): pinned-games library + self-update robustness; fix gamepad tab-nav
Decky client batch:
- Pinned games / library picker: per-host game grid (GamePickerModal),
  pin/unpin, one-tap streams surfaced on the Hosts tab and QAM
  (usePins/streamPin/resolvePinHost, new src/library.tsx).
- Self-update + client-update plumbing (main.py check_update, hooks.ts
  applyUpdate) with a CA-bundle-resolving SSL context and per-channel
  manifest polling; steam.ts / punktfunkrun.sh launch tweaks.
- scripts/test-backend.py harness for the backend RPCs; README refresh.

Fix: the fullscreen page wrapped <Tabs> in an overflow-visible box, so
Valve's L1/R1 tab slide + autoFocusContents scrollIntoView panned
#GamepadUI itself — the whole Steam UI slid left until a tab was clicked.
Clip the Tabs wrapper (overflow:hidden), matching Valve's own Tabs
containers. (On-glass verification pending — Deck offline this session.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:25:07 +00:00
enricobuehler 449a67ce8d feat(protocol): per-AU host-timing plane (0xCF) — split host+network latency (stats phase 2)
The unified-stats equation's host+network stage was one opaque number
because the wire carried nothing but pts_ns. Now the host reports its own
share per frame: when the client's Hello sets VIDEO_CAP_HOST_TIMING (0x08),
the send thread emits a 13-byte 0xCF datagram — [tag][pts_ns u64][host_us
u32] — right after the AU's last packet leaves the socket, so host_us =
capture→fully-sent (capture read/convert, encode, FEC+seal, paced send)
against the same anchor the wire pts carries. Clients correlate by pts_ns
and derive network = (received + clock_offset − pts) − host_us; the two
terms tile per frame by construction.

Back-compat is free in all four combinations: old clients ignore unknown
datagram tags, old hosts ignore unknown cap bits (client keeps the combined
stage). The hardened data-plane format is untouched — this rides the
established QUIC side-plane pattern (0xC8…0xCE). NativeClient ORs the bit
in unconditionally and exposes next_host_timing(); the C ABI gains
PunktfunkHostTiming + punktfunk_connection_next_host_timing (additive).
The synthetic host emits 0xCF too, so pure-loopback protocol tests cover
the plane.

The probe reports the split (host_p50/p95_us · net_p50/p95_us) and is our
direct analogue of Sunshine's "host processing latency" — ours additionally
includes the paced send.

Validated on loopback (synthetic host + probe, debug build): 240/240 AUs
matched, host_p50 6.5 ms + net_p50 6.4 ms ≈ capture→received p50 13.0 ms.
Core suite + new 0xCF roundtrip/truncation test green; host+core+probe
clippy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 21:22:12 +00:00
enricobuehler 09a5957c6d feat(clients): unified stats vocabulary across every client + Moonlight comparison docs
One stat model everywhere (design/stats-unification.md): four measurement
points (capture/received/decoded/displayed), three stages that tile the
interval exactly, and a HUD that shows the addition explicitly —

  end-to-end 14.2 ms p50 · 19.8 p95 · capture→on-glass
  = host+network 9.8 + decode 2.1 + display 2.3

replacing each client's ad-hoc mix of overlapping absolutes (the Apple HUD's
three arrow lines that looked sequential but weren't), mean-vs-median decode
times (Windows/Linux), missing same-host-clock flags (Windows/Linux), and
three different names for the same capture→received measurement (probe's
"reassembled", Apple/Android's "client", Windows/Linux's post-decode "lat").

Per client: Apple threads receivedNs through the VT decode via the frame
refcon bit pattern so the decode stage exists at all (stage-1 fallback
honestly degrades to a capture→received headline); Windows carries
FrameTimes through the existing frame channel to the render thread and adds
e2e p50/p95 post-Present; Linux stamps received at AU pop and rides
decoded_ns on DecodedFrame to the paintable-set site; Android pairs receipt
stamps with MediaCodec output buffers via the codec's pts round-trip (JNI
stats array 14→16 doubles, indexes 0-13 unchanged). fps now uniformly counts
received AUs; lost/(received+lost) per window, hidden at zero.

docs-site gains "Understanding the Stats Overlay": what each line means, why
the equation only approximately sums (percentiles), and a line-by-line
Moonlight/Sunshine matrix — including that Moonlight has no end-to-end
number and its "network latency" is an ENet control RTT, so punktfunk's
headline must not be compared against any single Moonlight line.

Verified here: linux client + probe + core check/clippy/fmt green, android
native cargo-ndk arm64 check green. Pending: Windows CI + on-glass, swift
test on the mac, on-device Android.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 21:01:29 +00:00
enricobuehler c7630ff5dc fix(host/audio): mic pump — open handshake on Linux + rapid-death backoff
apple / swift (push) Successful in 1m8s
apple / screenshots (push) Successful in 5m18s
android / android (push) Successful in 3m21s
windows-host / package (push) Successful in 6m58s
ci / rust (push) Successful in 1m58s
ci / web (push) Successful in 50s
ci / docs-site (push) Successful in 1m1s
ci / bench (push) Successful in 4m49s
deb / build-publish (push) Successful in 4m37s
decky / build-publish (push) Successful in 14s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 6s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m17s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m59s
docker / deploy-docs (push) Successful in 18s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m34s
Found by a live boot-order test (host started before the user session's
PipeWire): PwMicSource::open returned Ok before the daemon connection was
attempted, so a PipeWire that wasn't running surfaced as an instantly-dead
instance instead of an open failure — and the pump churned
open→die→reopen at heartbeat rate (1 Hz "virtual mic ready" log spam)
instead of backing off.

- PwMicSource::open now has a bring-up handshake (mirrors the Windows
  backend): ready only after connect + stream connect succeed, so a
  down daemon is an open ERROR and the pump's backoff engages.
- The pump triages deaths: an instance that lived >= 5 s (a one-off
  daemon restart) reopens immediately with the backoff reset; one that
  died right after opening counts as a failed open and backs off
  (2 s → 60 s cap). New pump test rapid_death_backs_off.

Re-validated live: host started with PipeWire stopped → throttled
"unavailable" warns, zero churn; daemon started → mic node up on the
next retry; exactly one pump + one loop thread (no leak).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 20:58:06 +00:00
enricobuehler 2c7ded0f3c fix(host/audio): rebuild mic passthrough — eager, self-healing virtual mic on both hosts
apple / swift (push) Successful in 1m7s
ci / rust (push) Successful in 1m57s
ci / web (push) Successful in 59s
android / android (push) Successful in 3m19s
ci / docs-site (push) Successful in 1m0s
apple / screenshots (push) Successful in 5m12s
windows-host / package (push) Successful in 7m2s
ci / bench (push) Successful in 4m52s
decky / build-publish (push) Successful in 14s
deb / build-publish (push) Successful in 4m37s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 8s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 6s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m14s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m40s
docker / deploy-docs (push) Successful in 18s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m28s
Mic passthrough silently died on real hosts. Root causes, all fixed:

- No liveness anywhere: a PipeWire restart (Linux) or any WASAPI device
  error (Windows) killed the backend worker; push() fed the dead queue
  for the rest of the host's life. VirtualMic now has a liveness
  contract (push -> bool, alive(), discard()) and the new shared
  audio::MicPump reopens with backoff, probing on an idle heartbeat so
  the mic heals BETWEEN sessions too. Validated live: systemctl restart
  pipewire -> node back in ~0.5 s, tone flows through the reopened
  backend.

- Lazy creation: the mic device didn't exist until the first 0xCB
  frame, but games bind their capture device at launch and never
  re-follow. The pump opens eagerly at host start (node exists with
  zero clients, elected default source).

- Windows headless dead-end: with VB-CABLE as the ONLY render endpoint
  (exactly what the installer ships), the anti-echo guard rejected the
  cable as the default render endpoint -> mic permanently dead. The new
  wiring_plan (pure, unit-tested on every platform) assigns the mic its
  endpoint FIRST (cable reserved for the mic), points the loopback at a
  DIFFERENT endpoint, and the capture side now yields (explicit
  endpoint or honest error) instead of the mic dying. Plan recomputed
  per (re)open — endpoints churn at boot/logon/driver installs.

- Stale bursts: buffered audio from a previous session played into a
  newly-attached recorder (observed live). Timestamped chunks + a
  consumer-gap check in the process callback age everything past 1 s.

The Linux node mechanism stays the stream-based Audio/Source with
RT_PROCESS + priority.session: the canonical null-audio-sink adapter
recipe was tested on this box (PipeWire 1.6.2) and never gets a clock
(QUANT 0 -> pure silence), and WirePlumber reroutes a feeder targeting
it to the default sink (echo). Decision documented in the module docs.

Live-validated on this box (synthetic host + probe --mic-test,
pw-record): eager node, both attach orderings, PipeWire-restart
self-heal, post-session silence. Windows side compile/CI + on-glass
validation pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 20:41:19 +00:00
enricobuehler b7048446c4 fix(windows-host): IDD-push compose kick — idle desktop no longer fails the attach gate
windows-drivers / probe-and-proto (push) Successful in 24s
apple / swift (push) Successful in 1m8s
ci / rust (push) Successful in 1m42s
windows-drivers / driver-build (push) Successful in 1m45s
ci / web (push) Successful in 54s
android / android (push) Successful in 3m39s
ci / docs-site (push) Successful in 1m8s
deb / build-publish (push) Successful in 4m40s
ci / bench (push) Successful in 4m58s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
decky / build-publish (push) Successful in 25s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
release / apple (push) Successful in 8m9s
windows-host / package (push) Successful in 7m35s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 53s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m27s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 50s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 54s
flatpak / build-publish (push) Successful in 4m26s
apple / screenshots (push) Successful in 5m29s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m46s
docker / deploy-docs (push) Successful in 24s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m21s
DWM presents a display only when something dirties it. On an idle desktop a
perfectly healthy session sat at E_PENDING: the driver attached but no
first frame ever landed, so wait_for_attach's 4 s gate failed the open (and
a mid-session ring recreate hit the same stall against the 3 s
recover-or-drop). A real client escaped only because its own input soon
dirtied the desktop; a headless probe / input-less connect never did.

kick_dwm_compose() injects two net-zero 1 px relative mouse moves via
SendInput — pf-vdisplay has no hardware-cursor plane, so a cursor move is
composited into the frame, a guaranteed real present onto the IDD
swap-chain (the mechanism --input-test always relied on; the pointer ends
where it started). Wired into wait_for_attach (first kick at 600 ms, then
every 800 ms) and, rate-limited, into the GB1 recovery window.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 19:24:57 +00:00
enricobuehler 3039626b87 design(host): add vrr design doc 2026-07-03 19:22:19 +00:00
enricobuehler 3f33ed30ae fix(windows-host): claim the vdisplay single-instance guard eagerly at serve startup
On-glass the lazy (first-session) claim let a second host started while the
freshly-restarted service sat idle win the mutex and ADD a monitor on the
real driver — priority backwards. The claim is now a process-global,
retryable slot (a failed claim is not memoized, so it heals once the other
instance exits), and `serve` claims it before any client can connect;
ensure_device keeps the lazy claim for standalone punktfunk1-host runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 18:57:47 +00:00
enricobuehler 7e31020c1c fix(windows-host): second-host guard — classify ACCESS_DENIED on the instance mutex as in-use
On-glass the SCM service creates Global\punktfunk-vdisplay-manager as
SYSTEM, so a second elevated-admin host's CreateMutexW fails ACCESS_DENIED
(the implicit open is checked against the SYSTEM DACL) before the
ALREADY_EXISTS branch can fire — right refusal, wrong message. Map it to
the same loud "another instance is live" error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 18:53:54 +00:00
enricobuehler fe54aff658 fix(windows-host): cross-plane IDD serialization, linger-expiry race, second-host guard
Batch C of the audit's medium tier (M7+M8+M9):

- M7: GameStream sessions now run the same begin_idd_setup dance as
  punktfunk/1 before creating the shared monitor. A GS connect could
  previously ADD/reconfigure the monitor while a native session was
  mid-build (and vice versa), and its sealed-channel delivery replaced the
  native ring (newest-wins) — each plane could freeze the other. GS has no
  cooperative stop plumbing, so it registers a flag nobody reads: a later
  session signals it, waits the 3 s grace, then force-preempts — the
  intended handover.
- M8: the linger-expiry teardown now runs UNDER the state lock. Running it
  outside let a concurrent acquire see Idle and ADD+isolate while the old
  monitor's pinger-join / CCD-restore / REMOVE were still in flight — a
  failed or de-isolated session exactly at the expiry boundary. A racing
  acquire now waits the few teardown seconds instead. Lock order stays
  state → device everywhere; the pinger takes only the device lock.
- M9: a named mutex (Global\punktfunk-vdisplay-manager) makes a SECOND host
  process fail its vdisplay open loudly instead of firing a startup
  CLEAR_ALL that razes the live host's monitors mid-stream (the admin
  footgun the shared watchdog then masked).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:28:22 +00:00
enricobuehler b46aa15afb fix(windows-drivers): pf-vdisplay robustness — AdapterInitStatus gate, pooled-device TDR check, MMCSS-optional worker
Batch B of the audit's medium tier (M4+M5+M6):

- M4: adapter_init_finished now reads AdapterInitStatus (was ignored) and
  only stashes the adapter on NT_SUCCESS, per the MS sample. A failed async
  init previously produced a HUSK adapter: monitors created on it arrive
  but the OS never assigns a swap-chain — every session black-screens with
  no visible cause (the exact signature live fault-injection produced after
  a WUDFHost kill). Unset adapter → ADD fails cleanly (host-retryable) and
  a re-entrant D0 retries the init; the status is now in the debug log.
- M5: pooled_device checks GetDeviceRemovedReason on a cache hit — a TDR'd
  device was returned for its LUID forever (SetDevice fail-loop, black
  virtual display until device teardown); now it falls through to a fresh
  create.
- M6: an AvSetMmThreadCharacteristicsW failure no longer aborts the worker
  before draining (which stalled the monitor and leaked the WDF swap-chain
  object) — continue unprioritized like the MS sample; revert only if MMCSS
  actually engaged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:20:48 +00:00
enricobuehler 058630f542 feat(decky): visible branded Steam shortcut, one-tap client updates, fullscreen-page polish
- The "Punktfunk" shortcut is no longer hidden: it now ships committed
  artwork (grid/wide/hero/logo/icon, generated by scripts/gen-steam-art.py
  — a pure-stdlib SDF renderer drawing the lens mark + a monoline
  "punktfunk" wordmark) applied via SetCustomArtworkForApp /
  SetShortcutIcon. Existing installs are unhidden and re-arted once per
  ART_VERSION; relaunching the library entry streams to the last host.
- Updates cover the flatpak CLIENT too: check_update compares the
  user-scope installed commit against its remote, applyUpdate runs
  `flatpak update --user` first (awaited) and the plugin reinstall —
  which reloads the panel — last; docs spell out the sudo-less --user
  update ("sudo flatpak update" silently skips per-user installs).
- Fullscreen page: DialogButton stretches to 100% width in the gamepad
  UI, so the Stream/Pair/Refresh/… actions filled whole rows — sized to
  content + right-aligned now; the header drops its Update button (About
  tab + QAM banner keep the flow) and the back button gets a real 40px
  hit target.
- Settings: the disable-Steam-Input note also shows for Automatic — on a
  Deck that now forwards the built-in controller as a Steam Deck pad
  (paddles/trackpads/gyro), which needs Steam Input off for the shortcut.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:16:40 +00:00
enricobuehler e9c1f4083a fix(client-linux): Deck Gaming Mode — auto pad type, real chrome-less fullscreen, leave-to-Gaming-Mode, colour bisect
- "Automatic" gamepad type resolves to the virtual Steam Deck pad on Deck
  hardware (env SteamDeck / DMI Jupiter|Galileo): the built-in 28DE:1205
  identity is invisible at Hello time — the Valve HIDAPI drivers run
  in-session only and Steam Input shadows the pad with its virtual X360 —
  so auto always fell through to Xbox 360. "steamdeck" is now also
  selectable in Settings.
- Chrome-less launches flatten the window CSS (border-radius/box-shadow)
  and fullscreen at startup: gamescope never ACKs the xdg fullscreen
  state, so adwaita kept the floating-CSD rounded corners + shadow
  visible over the stream.
- Gaming-Mode --connect launches quit on session end, so Steam ends the
  "game" and the Deck returns to Gaming Mode — previously the app popped
  to its own hosts page, stranding the user fullscreen and making the
  escape chord read as broken.
- The capture hint is controller-aware; the chromeless hint teaches the
  hold-chord ("hold L1+R1+Start+Select to leave") and a quick chord press
  re-flashes it.
- Colour bisect for the reported off-colours on the VAAPI dmabuf path:
  graphics offload defaults OFF under gamescope (a subsurface hands the
  NV12 CSC to the compositor), PUNKTFUNK_OFFLOAD=1|0 overrides, and each
  colour-signaling change logs whether GDK accepted the BT.709-narrow
  color state (fallback = GDK's BT.601 dmabuf default).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:16:26 +00:00
enricobuehler 20f0d2802f feat(client/android): Snapdragon latency tuning — ADPF pipeline hints, game mode, max-clock decode
Three levers to lower and steady decode latency on Snapdragon (Adreno) devices:

- ADPF (Adaptive Performance Framework): a new dlsym-resolved hint session
  (native/src/adpf.rs; API-33+, resolved at runtime so there's no build-time
  link dependency and libpunktfunk_android.so still loads on API 31/32) tells
  the CPU governor the video pipeline runs a per-frame real-time workload, so it
  keeps those threads on fast cores at high clocks. It now covers all three
  latency-critical threads — the pf-decode feed/drain/present loop, the core
  data-plane pump (UDP receive + FEC reassembly), and the audio thread — via a
  new generic hot-thread registry on NativeClient (register_hot_thread /
  hot_thread_ids; the pump self-registers). The session is built lazily on the
  first presented frame, since ADPF createSession rejects a set containing any
  not-yet-live tid.

- operating-rate -> Short.MAX ("as fast as possible"): pushes the Qualcomm
  decoder to run each frame at max clocks instead of merely sustaining the
  display rate at a power-saving clock that adds per-frame decode latency.

- appCategory="game": makes the app eligible for OEM Game Mode / Game Dashboard
  performance profiles.

The core registry is cross-platform (gettid on Linux/Android, a no-op
elsewhere) — no Android-specific pollution of the shared core. Host workspace +
64 core tests green; Android arm64-v8a + x86_64 (platform 31) build + clippy
clean. On-device Snapdragon validation pending.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 17:16:11 +00:00
enricobuehler 6f8fb15c9b fix(windows-host): self-heal the hostless-zombie pf-vdisplay device (adapter cycle + re-probe)
Fault-injection on-glass showed a killed/crashed WUDFHost leaves the devnode
"started" but HOSTLESS: PnP Status OK, no WUDFHost process, zero device-
interface instances — is_available() then fails every future session at the
vdisplay::open gate (and a reopen inside VdisplayDriver::open finds nothing),
until something cycles the device. Port reset-pf-vdisplay.ps1's adapter
disable→enable step in-process (restart_vdisplay_device): the open gate now
uses ensure_available() (cycle once + bounded re-probe; a genuinely
uninstalled driver — no adapter devnode — still fails fast), and
VdisplayDriver::open retries open_device over a short arrival window after a
cycle, covering the manager's reopen path too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:12:43 +00:00
enricobuehler 89455032a0 fix(windows-host): IDD-push resilience — driver-death recovery, reopenable control device, full interface discovery
Batch A of the audit's medium tier (M1+M2+M3):

- M1 driver-death detection: a dead WUDFHost stops publishing, which at the
  ring is indistinguishable from an idle desktop — SDR sessions streamed a
  frozen frame forever (next_frame's 20 s bail is unreachable once anything
  presented). The ChannelBroker's process handle now doubles as a liveness
  probe (SYNCHRONIZE at OpenProcess); while no fresh frame arrives,
  try_consume polls it (rate-limited) and fails the capturer, landing in the
  session's bounded in-place rebuild.
- M2 reopenable control device: the manager's OnceLock-cached handle is now
  a retire/reopen DeviceSlot — a gone-classified IOCTL failure (driver
  upgrade / WUDFHost restart; pinger, create, or REMOVE) retires the handle
  and the next use reopens + re-handshakes. Retired handles are deliberately
  kept alive forever: bare-HANDLE holders (pinger, ChannelBroker) rely on
  never-closed, and a retired handle only fails IOCTLs. CLEAR_ALL runs on
  the FIRST open only (a reopen races live-ish sessions); acquire retries
  the monitor create once after a reopen. The JOIN path now probes the
  active monitor's WUDFHost pid and preempts a DEAD monitor instead of
  handing the rebuilding session its stale target — without this the whole
  recovery chain starved to the rebuild budget.
- M3 interface discovery: enumerate ALL interface instances with an
  SPINT_ACTIVE filter (a Code-10 devnode at index 0 no longer shadows the
  live interface), HDEVINFO behind RAII (error paths leaked one per probe),
  the raw device handle wrapped before GET_INFO (leaked on handshake
  failure), and the detail-sizing result guarded before the cbSize write.
- pf-driver-proto: SetFrameChannelRequest doc now states the real
  adopt-on-success contract (the old wording invited a driver-side
  close-on-error — a cross-process double-close against the host's reap).
- install: pf_vdisplay_present() passes /connected so a phantom devnode
  can't suppress creating a live ROOT node.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 17:04:19 +00:00
enricobuehler 0da9d8ec10 fix(windows): IDD-push audit highs — keyed-mutex timeout, two per-frame leaks, IDD_PUSH knob, pooled-device threading
Five verified findings from the IDD-push/pf-vdisplay deep audit:

- Keyed-mutex acquire (BOTH endpoints): AcquireSync returns WAIT_TIMEOUT
  (0x102) / WAIT_ABANDONED (0x80) as SUCCESS-severity HRESULTs, which the
  windows-rs Result wrapper erases — a busy slot read as "acquired", so
  driver and host could race the same ring texture (torn frames) and the
  designed busy-skip backpressure was dead code. Both sides now classify
  the raw vtable HRESULT; WAIT_ABANDONED counts as acquired (ownership
  transfers — refusing it would wedge the slot forever).
- Host SDR hot path leaked one ID3D11VideoProcessorInputView per converted
  frame: the D3D11_VIDEO_PROCESSOR_STREAM ManuallyDrop field suppressed the
  release after VideoProcessorBlt. Released by hand now, success or not.
- Driver leaked IddCx's per-acquire surface reference (from_raw_borrowed on
  a TRANSFERRED reference — the MS sample Attach/Reset's it): the swap-chain
  surface set survived swap-chain destruction, the likely true root cause of
  the ~50 MB-per-reconnect VRAM loss that device pooling only mitigated.
  Now adopted via from_raw (publisher or not) and dropped pre-Finished.
- PUNKTFUNK_IDD_PUSH removed: capture is unconditionally IDD-push, but the
  vdisplay manager still gated the lingering-monitor preempt (and render
  pin) on the knob, whose default was OFF — dev/CLI runs reused a lingering
  monitor whose IddCx swap-chain is dead (black reconnect). The preempt and
  the render-GPU pin are now unconditional; host.env comments no longer
  promise the removed DDA/WGC fallback.
- Driver D3D device: dropped D3D11_CREATE_DEVICE_SINGLETHREADED (unsound
  since DEVICE_POOL shares one device across processors) and the pooled
  immediate context is now SetMultithreadProtected — two concurrent
  monitors' workers otherwise race an unlocked context (UB in the UMD).

No wire-contract change (pf-driver-proto untouched); the driver fixes take
effect on the next pf-vdisplay redeploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 16:27:13 +00:00
enricobuehler fbf3fea0c8 fix(packaging/windows): Inno [Code] comment brace-nesting trap broke ISCC
apple / swift (push) Successful in 1m17s
windows-host / package (push) Successful in 7m3s
apple / screenshots (push) Successful in 5m37s
android / android (push) Successful in 3m22s
ci / web (push) Successful in 48s
ci / rust (push) Successful in 1m18s
ci / bench (push) Successful in 4m59s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 4m35s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 10m29s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 10m31s
docker / deploy-docs (push) Successful in 15s
The PublicFwParam doc comment contained a literal code-constant token; Inno's
{ } comments don't nest, so its closing brace ended the comment early and the
trailing text parsed as code ("'BEGIN' expected", compile aborted). Reworded to
avoid the literal braces + added a warning note. Verified: the [Code] section
has no other nested-brace-in-comment traps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 15:27:27 +00:00
enricobuehler c52ae119e1 fix: regenerate Cargo.lock for punktfunk-host's winresource build-dep
apple / swift (push) Successful in 1m10s
audit / cargo-audit (push) Successful in 1m12s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m8s
android / android (push) Successful in 3m49s
ci / web (push) Successful in 48s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m7s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 45s
ci / docs-site (push) Successful in 56s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 50s
release / apple (push) Successful in 8m51s
ci / rust (push) Successful in 5m51s
ci / bench (push) Successful in 4m51s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 3s
deb / build-publish (push) Successful in 4m46s
apple / screenshots (push) Successful in 5m34s
flatpak / build-publish (push) Successful in 5m33s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 10m52s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m43s
docker / deploy-docs (push) Failing after 18s
windows-host / package (push) Has been cancelled
The "Punktfunk Host" identity work added winresource to the host crate but
didn't update the lock, so every --locked CI job failed to resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 15:17:31 +00:00
enricobuehler 5d7aabe8f0 fix(scripts/windows): keep deploy-host.ps1 pure ASCII (em-dash -> hyphen)
apple / swift (push) Successful in 1m8s
ci / rust (push) Failing after 50s
ci / web (push) Successful in 55s
ci / docs-site (push) Successful in 1m8s
android / android (push) Successful in 3m24s
deb / build-publish (push) Failing after 45s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
windows-host / package (push) Failing after 6m18s
apple / screenshots (push) Successful in 5m35s
ci / bench (push) Successful in 4m41s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 3m25s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 3m26s
docker / deploy-docs (push) Successful in 17s
Two comment em-dashes I added tripped the installer-run ASCII guard (PS 5.1
mis-parses non-ASCII on non-UTF-8 locales).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 15:09:17 +00:00
enricobuehler f204a89cef perf(encode/windows): AMF quality=speed + bf=0; drop the useless poll spin
ci / rust (push) Failing after 48s
windows-host / package (push) Failing after 10s
apple / swift (push) Successful in 1m6s
ci / web (push) Successful in 51s
ci / docs-site (push) Successful in 1m8s
android / android (push) Successful in 3m20s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 4s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
apple / screenshots (push) Successful in 5m10s
ci / bench (push) Successful in 4m43s
docker / deploy-docs (push) Successful in 18s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 3m25s
deb / build-publish (push) Failing after 44s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 3m21s
On-box A/B on the .173 Ryzen 7000 iGPU (720p60, real composition via input
injection — an idle virtual desktop composes ~1 fps and gives meaningless
encode timings): the encode-time-first `quality=speed` preset + explicit `bf=0`
cut host-side encode_us from ~36 ms to ~19.5 ms.

The blocking-poll idea from the prior commit was WRONG and is reverted to a
single non-blocking receive (default PUNKTFUNK_FFWIN_POLL_MS=0): libavcodec's
hevc_amf holds ~2 frames before releasing the oldest (needs frame N+2 to flush
N), so a spin between submits provably never yields the owed AU — verified with
a 150 ms cap pegging at exactly 150 ms across every usage preset and pipeline
depth. That ~2-frame buffer is inherent to the libavcodec wrapper, not host
scheduling; the real latency lever is a direct AMF SDK encoder (the AMF
analogue of the direct-NVENC path), tracked as the next AMD work item. The
env knob is retained for a future VCN/driver where a bounded spin can help.

Also measured and rejected: PUNKTFUNK_ZEROCOPY=1 on AMF is ~2x WORSE (68 ms vs
36 ms) — the D3D11 import path adds sync overhead beyond the readback it saves,
so the system-memory default stays. GPU-priority elevation is already
process-wide (dxgi.rs), so it covers the iGPU encode session with no change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 14:57:39 +00:00
enricobuehler 24fa018c70 chore(encode/windows): AMF forensics knobs — PUNKTFUNK_AMF_USAGE + PUNKTFUNK_FFWIN_POLL_MS
apple / swift (push) Successful in 1m6s
ci / web (push) Successful in 53s
deb / build-publish (push) Failing after 44s
windows-host / package (push) Failing after 10s
ci / rust (push) Failing after 49s
android / android (push) Successful in 3m33s
apple / screenshots (push) Successful in 5m18s
ci / docs-site (push) Successful in 57s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 4s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
ci / bench (push) Successful in 5m0s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 3m27s
docker / deploy-docs (push) Successful in 18s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 3m18s
The blocking poll landed but wait_us pegs at exactly the 2-frame-period cap:
AMF holds the AU ~2 frame periods regardless of retrieval. Field knobs to
bisect on-box (usage preset × poll cap) without rebuild cycles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 14:39:36 +00:00
enricobuehler 51a6ca7e02 fix(encode/windows): AMF latency — honor the loop's blocking-poll contract + preset polish
apple / swift (push) Successful in 1m6s
windows-drivers / driver-build (push) Successful in 1m34s
windows-drivers / probe-and-proto (push) Successful in 20s
ci / rust (push) Failing after 47s
ci / web (push) Successful in 52s
windows-host / package (push) Failing after 11s
ci / docs-site (push) Successful in 1m6s
android / android (push) Successful in 3m20s
deb / build-publish (push) Failing after 46s
decky / build-publish (push) Successful in 12s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 13s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 43s
apple / screenshots (push) Successful in 5m11s
docker / deploy-docs (push) Successful in 19s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 3m27s
ci / bench (push) Successful in 4m43s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 3m24s
The session loop's pipeline deferral was designed around direct NVENC, whose
poll() BLOCKS in lock_bitstream; libavcodec's AMF wrapper is truly async
(EAGAIN until the ASIC finishes), so a single non-blocking receive quantized AU
retrieval to the submit cadence: +1–2 frame periods flat (~43 ms p50 at 720p60
on the Ryzen iGPU vs ~3.5 ms of actual encode). FfmpegWinEncoder now tracks
in-flight frames and, while an AU is owed, spin-polls with short sleeps bounded
to ~2 frame periods (an overloaded encoder degrades to next-tick pickup instead
of stalling capture). Also: quality=speed (latency-first, iGPU-class VCN),
explicit bf=0 (h264_amf defaults >0 on RDNA3+), AMF low-latency submission
mode (FFmpeg ≥6.1, ignored on older).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 14:32:41 +00:00
enricobuehler b9fde03f1e feat(security): finish Windows firewall Public opt-in wiring + vuln-disclosure + doc cleanup
Firewall (the service.rs core landed in efb1ba2): scope the web-console rule
(TCP 47992) to Domain+Private by default with a `--allow-public-network` opt-in
that deletes-then-re-adds the rule, and add the installer "Allow connections on
Public networks" task (unchecked) forwarding the flag to `service install` and
`web setup`. Default is now trusted-networks-only; Public is explicit.

Vulnerability disclosure: SECURITY.md (report to security@punktfunk.com, scope,
SLAs, safe harbor), a Gitea issue-template contact link, a README security line,
and a Reporting section on the docs Security page.

Docs: the Security page now documents the Private/Domain firewall default (and
how to fix a misclassified-Public network / opt in); removed internal design-doc
and CLAUDE.md links from the user-facing docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 14:08:17 +00:00
enricobuehler efb1ba26d7 fix(windows): opt-in pad-driver file logs + size-capped service log rotation
Two disk-write fixes:

- pf-xusb/pf-dualsense no longer write C:\Users\Public\pf*-driver.log
  unconditionally — the file log is now opt-in (debug builds, or the
  PFXUSB_DEBUG_LOG / PFDS_DEBUG_LOG system env var), mirroring the audit-§4.4
  fix pf-vdisplay already got: a release driver never writes the world-writable
  Public file (info-leak/DoS surface), and the per-report OUTPUT/SET_STATE hex
  dumps stop being a sustained per-rumble disk-write path during gameplay.
  OutputDebugStringA stays unconditional; the host's driver-silence WARN and
  the gamepad-driver-health failure-mode table now say the log is opt-in.

- service.log/host.log get one-generation rotation: at each (re)open a file
  over 10 MB is renamed to .old, so a crash-restart loop or a RUST_LOG=debug
  left in host.env can't grow the append-forever logs without bound. Rotation
  runs only before an open (never under a live appender — host.log's handle
  lacks FILE_SHARE_DELETE, so a racing rename harmlessly fails).

Windows CI compile/clippy pending (drivers workspace + host are not
Linux-cross-checkable); rides along with the next pad-driver redeploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 14:03:32 +00:00
enricobuehler 1320e3dc66 fix(scripts/windows): deploy-host.ps1 builds all-vendor when an FFmpeg tree exists
windows-host / package (push) Failing after 20s
apple / swift (push) Successful in 1m9s
apple / screenshots (push) Successful in 5m26s
ci / rust (push) Failing after 48s
ci / web (push) Successful in 50s
ci / docs-site (push) Successful in 1m6s
android / android (push) Successful in 3m22s
deb / build-publish (push) Failing after 43s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
ci / bench (push) Successful in 4m40s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Failing after 3m27s
docker / deploy-docs (push) Successful in 6s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Failing after 3m18s
The dev deploy built --features nvenc only, so a web-console GPU preference
pointing at an AMD/Intel adapter made every session die at encoder open
(NV_ENC_ERR_NO_ENCODE_DEVICE) — the exact "can't connect" just hit on the RTX
box's Ryzen iGPU. The script now enables amf-qsv when FFMPEG_DIR (machine env,
process env, or C:\Users\Public\ffmpeg) has a dev tree, and copies the FFmpeg
runtime DLLs next to the exe after a successful build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 13:52:55 +00:00
enricobuehler 1be83575b6 feat(host/windows): "Punktfunk Host" identity in Task Manager (icon + version info)
punktfunk-host.exe embedded no icon or version resources, so Task Manager and
Explorer showed a bare lowercase exe name with a generic icon. build.rs now
embeds the branded .ico + FileDescription "Punktfunk Host" / ProductName
"Punktfunk" via winresource (same pattern as the Windows client and the tray;
Linux packaging builds skip the block). The tray gets a matching "Punktfunk
Tray" description, and the SCM display name moves off lowercase
"punktfunk streaming host" to "Punktfunk Host" (applied idempotently by
`service install` on upgrade).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 13:52:55 +00:00
enricobuehler 4d1d20f832 chore: untrack CLAUDE.md
apple / swift (push) Successful in 1m14s
apple / screenshots (push) Successful in 5m45s
ci / rust (push) Successful in 1m15s
ci / web (push) Successful in 50s
ci / docs-site (push) Successful in 1m5s
android / android (push) Successful in 3m25s
deb / build-publish (push) Successful in 4m35s
ci / bench (push) Successful in 4m55s
decky / build-publish (push) Successful in 11s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 4s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 6s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 10m24s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 10m27s
docker / deploy-docs (push) Successful in 18s
Local per-box assistant instructions (incl. internal environment detail) don't
belong in the published tree; the file stays on disk, now gitignored.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 13:22:23 +00:00
enricobuehler 6e875fea44 fix(apple/ios): the ACTUAL type-checker bomb was pointerSection's footer ternary chain
apple / swift (push) Successful in 1m15s
release / apple (push) Successful in 8m36s
apple / screenshots (push) Successful in 5m44s
ci / rust (push) Successful in 1m31s
ci / web (push) Successful in 56s
android / android (push) Successful in 10m1s
deb / build-publish (push) Successful in 4m33s
ci / bench (push) Successful in 4m52s
ci / docs-site (push) Successful in 1m24s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
decky / build-publish (push) Successful in 11s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 5s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 5s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 5s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 10m20s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 10m19s
docker / deploy-docs (push) Failing after 14s
4f3cd24 split the wrong expression — act's log masking hid the real line number.
The unmasked retry pinpointed it: the pointerSection footer, a ten-segment
string + chain with an isPad ternary nesting four more, evaluated inside the
ViewBuilder. Moved the copy into a plain computed String built with +=
statements (linear to type-check); no text change. The two remaining 5-6
segment chains in Settings are compiled by the passing macOS slice, so they
are proven cheap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 13:17:01 +00:00
enricobuehler 4f3cd24036 fix(apple/ios): split streamModeSection — the inline iOS branch blew the type-checker budget
release / apple (push) Successful in 6m29s
ci / web (push) Successful in 1m2s
ci / docs-site (push) Successful in 1m13s
apple / swift (push) Successful in 1m5s
apple / screenshots (push) Successful in 4m5s
ci / bench (push) Successful in 4m36s
ci / rust (push) Successful in 11m23s
decky / build-publish (push) Successful in 13s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 6s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 6s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 6s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 7s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 5s
deb / build-publish (push) Successful in 4m26s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 9m45s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 9m37s
android / android (push) Successful in 9m37s
docker / deploy-docs (push) Successful in 19s
The Section's iOS content (resolution wheel + 3-way refresh rows + bitrate
rows) as ONE ViewBuilder expression hit "the compiler is unable to type-check
this expression in reasonable time" — failing exactly one build slice, the iOS
archive, so swift test (macOS) and the tvOS/macOS archives never saw it and the
0.6.0 iOS TestFlight upload soft-failed. Extracted iosResolutionWheel /
iosRefreshRows / bitrateRows; no behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 13:06:48 +00:00
141 changed files with 9543 additions and 2230 deletions
+9
View File
@@ -0,0 +1,9 @@
# Shown on the "new issue" chooser so security reports go to the private channel, not a public issue.
blank_issues_enabled: true
contact_links:
- name: 🔒 Report a security vulnerability
url: https://git.unom.io/unom/punktfunk/src/branch/main/SECURITY.md
about: >-
Found a security issue? Please report it privately by email to security@punktfunk.com — do not
open a public issue, so other users aren't exposed before a fix ships. See SECURITY.md for the
full policy.
+2 -1
View File
@@ -78,9 +78,10 @@ jobs:
- name: Version + channel
if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v'))
run: |
eval "$(bash scripts/ci/pf-version.sh)" # -> PF_BASE (one minor ahead of the latest stable tag)
case "$GITHUB_REF" in
refs/tags/v*) VN="${GITHUB_REF_NAME#v}"; TRACK="alpha" ;; # alpha = built-in closed testing
*) VN="0.5.0-ci${GITHUB_RUN_NUMBER}"; TRACK="internal" ;;
*) VN="${PF_BASE}-ci${GITHUB_RUN_NUMBER}"; TRACK="internal" ;;
esac
echo "VERSION_NAME=$VN" >> "$GITHUB_ENV"
echo "PLAY_TRACK=$TRACK" >> "$GITHUB_ENV"
+6 -5
View File
@@ -36,16 +36,17 @@ jobs:
- name: Version + channel
# vX.Y.Z tag -> X.Y.Z, published to the `stable` apt distribution (a real release).
# A main push -> 0.5.0~ciN.g<sha>, published to the `canary` distribution: the '~' sorts
# below the eventual 0.5.0 tag, it climbs monotonically by run number, and the canary base
# stays one minor AHEAD of the latest stable so a stable->canary box re-point still moves
# forward (see channels.md). Computed BEFORE the build so it's stamped into the binary
# A main push -> <next-minor>~ciN.g<sha>, published to the `canary` distribution: the '~' sorts
# below the eventual tag, it climbs monotonically by run number, and the canary base is
# derived one minor AHEAD of the latest stable tag (scripts/ci/pf-version.sh) so a
# stable->canary box re-point still moves forward (see channels.md). Computed BEFORE the build so it's stamped into the binary
# (PUNKTFUNK_BUILD_VERSION -> build.rs -> --version).
run: |
eval "$(bash scripts/ci/pf-version.sh)" # -> PF_BASE (one minor ahead of the latest stable tag)
SHORT=$(echo "$GITHUB_SHA" | cut -c1-8)
case "$GITHUB_REF" in
refs/tags/v*) V="${GITHUB_REF_NAME#v}"; DIST=stable ;;
*) V="0.5.0~ci${GITHUB_RUN_NUMBER}.g${SHORT}"; DIST=canary ;;
*) V="${PF_BASE}~ci${GITHUB_RUN_NUMBER}.g${SHORT}"; DIST=canary ;;
esac
echo "VERSION=$V" >> "$GITHUB_ENV"
echo "DISTRIBUTION=$DIST" >> "$GITHUB_ENV"
+6 -2
View File
@@ -63,7 +63,8 @@ jobs:
pnpm run build # rollup -> clients/decky/dist/index.js
- name: Version + channel + stamp
# Tag vX.Y.Z -> X.Y.Z (stable `latest/` alias + Gitea Release); main push -> 0.3.<run>
# Tag vX.Y.Z -> X.Y.Z (stable `latest/` alias + Gitea Release); main push -> <next-minor>.<run>
# (base one minor ahead of the latest stable tag via scripts/ci/pf-version.sh)
# (`canary/` alias). Decky reads a plugin's INSTALLED version from package.json (NOT
# plugin.json), and the plugin's own update check (clients/decky/main.py check_update)
# compares against it — so the build version is STAMPED into package.json here (mirrored
@@ -72,9 +73,12 @@ jobs:
# (ci10 < ci9), which would break update detection; the run number is monotonic.
working-directory: ${{ gitea.workspace }}
run: |
eval "$(bash scripts/ci/pf-version.sh)" # -> PF_MAJOR/PF_MINOR (base one minor ahead of latest stable)
case "$GITHUB_REF" in
refs/tags/v*) V="${GITHUB_REF_NAME#v}"; ALIAS=latest ;;
*) V="0.3.${GITHUB_RUN_NUMBER}"; ALIAS=canary ;;
# Canary MUST be a plain monotonic numeric semver (see the note above): <major>.<minor>.<run>,
# where major.minor track one minor ahead of the latest stable and the run number climbs.
*) V="${PF_MAJOR}.${PF_MINOR}.${GITHUB_RUN_NUMBER}"; ALIAS=canary ;;
esac
BASE="https://$REGISTRY/api/packages/$OWNER/generic/$PACKAGE"
echo "VERSION=$V" >> "$GITHUB_ENV"
+48 -5
View File
@@ -73,15 +73,17 @@ jobs:
- name: Version + channel
# Tag vX.Y.Z -> X.Y.Z on the OSTree `stable` branch (a real release); a main push ->
# 0.5.0-ciN.g<sha> on the `canary` branch. The two branches live side-by-side in one repo
# <next-minor>-ciN.g<sha> on the `canary` branch (base one minor ahead of the latest stable
# tag via scripts/ci/pf-version.sh). The two branches live side-by-side in one repo
# (rsync runs without --delete), each tracked by its own .flatpakref, so `flatpak update`
# on a stable box never jumps to a canary build. The generic-registry version string allows
# letters/dots/hyphens.
run: |
eval "$(bash scripts/ci/pf-version.sh)" # -> PF_BASE (one minor ahead of the latest stable tag)
SHORT=$(echo "$GITHUB_SHA" | cut -c1-8)
case "$GITHUB_REF" in
refs/tags/v*) V="${GITHUB_REF_NAME#v}"; BRANCH=stable; ALIAS=latest ;;
*) V="0.5.0-ci${GITHUB_RUN_NUMBER}.g${SHORT}"; BRANCH=canary; ALIAS=canary ;;
*) V="${PF_BASE}-ci${GITHUB_RUN_NUMBER}.g${SHORT}"; BRANCH=canary; ALIAS=canary ;;
esac
echo "VERSION=$V" >> "$GITHUB_ENV"
echo "BUNDLE=punktfunk-client-${V}.flatpak" >> "$GITHUB_ENV"
@@ -106,6 +108,40 @@ jobs:
python3 /tmp/flatpak-cargo-generator.py /tmp/Cargo.flatpak.lock \
-o packaging/flatpak/cargo-sources.json
- name: Seed the local OSTree repo from the live server (keep BOTH channels in the summary)
# Each CI run builds only ONE branch (canary on main, stable on a tag). The deploy step's
# `flatpak build-update-repo` regenerates the repo SUMMARY from whatever refs are in the
# LOCAL repo, and the rsync publishes it (without --delete). A fresh single-branch local
# repo therefore produces a single-branch summary that CLOBBERS the other channel on the
# server — the exact bug that made `app/io.unom.Punktfunk/x86_64/stable` unresolvable
# ("No such ref") after a canary main-push overwrote the post-release summary, even though
# the stable commit's objects were still on disk. Fix: mirror the published repo DOWN first,
# so the local repo carries every existing branch; the build below then only ADDS this run's
# commit and the regenerated+signed summary keeps both channels. No-op on a fresh repo (first
# publish) or when the deploy secrets aren't set (the build still produces a valid bundle).
env:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_PORT: ${{ secrets.DEPLOY_PORT }}
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
run: |
set -euo pipefail
if [ -z "${DEPLOY_HOST:-}" ] || [ -z "${DEPLOY_SSH_KEY:-}" ]; then
echo "::warning::DEPLOY_* not set — no seed; building a fresh single-branch repo."
exit 0
fi
install -d -m700 ~/.ssh
printf '%s\n' "$DEPLOY_SSH_KEY" > ~/.ssh/deploy; chmod 600 ~/.ssh/deploy
SSH="ssh -i $HOME/.ssh/deploy -p ${DEPLOY_PORT:-22} -o StrictHostKeyChecking=accept-new"
DEST="${DEPLOY_USER}@${DEPLOY_HOST}"
mkdir -p "$PWD/repo"
# Pull the currently-published repo (all channels' objects + refs) into the repo the build
# will extend. No --delete: the local repo starts empty, so this only ADDS. A missing
# server repo (very first publish) is fine — we continue with a fresh repo.
rsync -az --info=stats1 -e "$SSH" "$DEST:$DEPLOY_DIR/site/repo/" "$PWD/repo/" \
|| echo "::warning::no published repo to seed (first publish?) — continuing fresh"
echo "seeded refs:"; ls "$PWD/repo/refs/heads/app/$APP_ID/x86_64/" 2>/dev/null || echo " (none)"
- name: Build the flatpak (install deps from Flathub, offline build)
run: |
# --install-deps-from=flathub pulls everything the manifest declares: the GNOME 50
@@ -177,6 +213,10 @@ jobs:
--gpg-sign="$KEYID" --gpg-homedir="$GNUPGHOME"
flatpak build-update-repo --generate-static-deltas \
--gpg-sign="$KEYID" --gpg-homedir="$GNUPGHOME" "$PWD/repo"
# The regenerated summary advertises exactly these refs — must include EVERY channel that
# has ever published (the seed step ensures the other channel's commit is present). If this
# ever shows only one branch on a repo that had two, the seed didn't run — investigate.
echo "published summary advertises:"; ls "$PWD/repo/refs/heads/app/$APP_ID/x86_64/" 2>/dev/null || echo " (none)"
# 2) Build the install descriptors (GPGKey = the committed public key, base64).
GPGKEY="$(base64 -w0 packaging/flatpak/unom-flatpak.gpg)"
rm -rf site && mkdir -p site
@@ -188,9 +228,12 @@ jobs:
Comment=unom Flatpak applications
GPGKey=$GPGKEY
EOF
# Two refs, one per channel — both regenerated every run and rsync'd without --delete, so
# the server always offers both (the stable ref only resolves once a release has built the
# `stable` branch). A box installs ONE; `flatpak update` then tracks that channel's branch.
# Two refs, one per channel. Both descriptor files are regenerated every run and rsync'd
# without --delete; the repo SUMMARY carries both branches because the build was seeded
# from the live repo above (so build-update-repo below re-signs a summary listing every
# published channel, not just this run's). The stable ref resolves for good once any
# release has built the `stable` branch. A box installs ONE; `flatpak update` then tracks
# that channel's branch.
write_ref() { # <filename> <branch> <title>
cat > "site/$1" <<EOF
[Flatpak Ref]
+3 -2
View File
@@ -99,13 +99,14 @@ jobs:
- name: Version from tag
run: |
eval "$(bash scripts/ci/pf-version.sh)" # -> PF_BASE, PF_CHANNEL, PF_STABLE_TAG (single source of truth)
case "$GITHUB_REF" in
refs/tags/v*) V="${GITHUB_REF_NAME#v}"; V="${V%%-*}" ;; # App Store marketing version is numeric X.Y.Z (drop -rc)
*) V="0.5.0" ;; # canary marketing version; the build number disambiguates
*) V="$PF_BASE" ;; # canary marketing version = one minor ahead of the latest stable tag; the build number disambiguates
esac
echo "VERSION=$V" >> "$GITHUB_ENV"
echo "BUILD_NUM=$GITHUB_RUN_NUMBER" >> "$GITHUB_ENV"
echo "version $V build $GITHUB_RUN_NUMBER"
echo "version $V build $GITHUB_RUN_NUMBER (channel $PF_CHANNEL, latest stable ${PF_STABLE_TAG})"
- name: Rust toolchain (mac + iOS + tvOS slices)
run: |
+6 -5
View File
@@ -68,16 +68,17 @@ jobs:
restore-keys: cargo-home-
- name: Version + channel
# vX.Y.Z tag -> X.Y.Z-1 in the base group (a real release); main push -> 0.5.0-0.ciN.g<sha>
# in the `<base>-canary` group, whose "0." release sorts below the eventual 0.5.0-1 yet
# climbs by run number. The canary base stays one minor ahead of the latest stable so a
# stable->canary box re-point still moves forward. The spec %build stamps
# vX.Y.Z tag -> X.Y.Z-1 in the base group (a real release); main push -> <next-minor>-0.ciN.g<sha>
# in the `<base>-canary` group, whose "0." release sorts below the eventual <next-minor>-1 yet
# climbs by run number. The canary base is derived one minor ahead of the latest stable tag
# (scripts/ci/pf-version.sh) so a stable->canary box re-point still moves forward. The spec %build stamps
# PUNKTFUNK_BUILD_VERSION from these macros into the binary (--version provenance).
run: |
eval "$(bash scripts/ci/pf-version.sh)" # -> PF_BASE (one minor ahead of the latest stable tag)
SHORT=$(echo "$GITHUB_SHA" | cut -c1-8)
case "$GITHUB_REF" in
refs/tags/v*) V="${GITHUB_REF_NAME#v}"; R="1"; GROUP="${{ matrix.group }}" ;;
*) V="0.5.0"; R="0.ci${GITHUB_RUN_NUMBER}.g${SHORT}"; GROUP="${{ matrix.group }}-canary" ;;
*) V="$PF_BASE"; R="0.ci${GITHUB_RUN_NUMBER}.g${SHORT}"; GROUP="${{ matrix.group }}-canary" ;;
esac
echo "PF_VERSION=$V" >> "$GITHUB_ENV"
echo "PF_RELEASE=$R" >> "$GITHUB_ENV"
+5 -2
View File
@@ -16,7 +16,8 @@
# Versioning (free-form; not MSIX's 4-part rule) — single project version:
# vX.Y.Z tag -> X.Y.Z (THE release; published + stable `latest/` alias + attached to the
# unified Gitea Release).
# main push / dispatch -> 0.3.<run_number> (canary; `canary/` alias; climbs by run number).
# main push / dispatch -> <next-minor>.<run_number> (canary; `canary/` alias; base one minor
# ahead of the latest stable tag via scripts/ci/pf-version.ps1, run climbs).
#
# Signing reuses the client's MSIX_CERT_PFX_B64 / MSIX_CERT_PASSWORD secrets (CN=unom). Without them
# an ephemeral self-signed cert is generated and its public .cer published next to the installer
@@ -102,10 +103,12 @@ jobs:
if (-not $env:VBCABLE_DIR) {
"VBCABLE_DIR=C:\Users\Public\vbcable" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
}
$pf = & "$env:GITHUB_WORKSPACE/scripts/ci/pf-version.ps1" # single source of truth: base is one minor ahead of the latest stable tag
$v = if ($env:GITHUB_REF -like 'refs/tags/v*') {
$env:GITHUB_REF_NAME -replace '^v', ''
} else {
"0.3.$($env:GITHUB_RUN_NUMBER)"
# Canary: <major>.<minor>.<run> — major.minor track one minor ahead of stable, run climbs monotonically.
"$($pf.PF_MAJOR).$($pf.PF_MINOR).$($env:GITHUB_RUN_NUMBER)"
}
"HOST_VERSION=$v" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
"PUNKTFUNK_BUILD_VERSION=$v" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
+5 -2
View File
@@ -16,7 +16,8 @@
# vX.Y.Z tag -> X.Y.Z.0 (THE release; any -rc/+meta pre-release suffix is dropped for MSIX).
# Published to the generic registry + the stable `latest/` alias + attached to the
# unified Gitea Release alongside every other platform's artifact.
# main push / dispatch -> 0.3.<run_number>.0 (canary; climbs monotonically by run number).
# main push / dispatch -> <next-minor>.<run_number>.0 (canary; base is one minor ahead of the
# latest stable tag via scripts/ci/pf-version.ps1, run number climbs monotonically).
# Published to the generic registry + the `canary/` alias.
# Both arches share the version; artifacts are arch-suffixed (..._x64.msix / ..._arm64.msix).
#
@@ -78,11 +79,13 @@ jobs:
"CARGO_TARGET_DIR=${{ matrix.td }}" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
"FFMPEG_DIR=${{ matrix.ffmpeg }}" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
rustup target add ${{ matrix.target }}
$pf = & "$env:GITHUB_WORKSPACE/scripts/ci/pf-version.ps1" # single source of truth: base is one minor ahead of the latest stable tag
$parts = if ($env:GITHUB_REF -like 'refs/tags/v*') {
# MSIX needs a purely-numeric 4-part version: drop any -rc/+meta pre-release suffix.
(($env:GITHUB_REF_NAME -replace '^v', '') -replace '[-+].*$', '').Split('.')
} else {
@('0', '3', $env:GITHUB_RUN_NUMBER)
# Canary: <major>.<minor>.<run>.0 — major.minor track one minor ahead of stable, run climbs monotonically.
@($pf.PF_MAJOR, $pf.PF_MINOR, $env:GITHUB_RUN_NUMBER)
}
while ($parts.Count -lt 4) { $parts += '0' }
$v = ($parts[0..3] -join '.')
+3
View File
@@ -31,3 +31,6 @@ xcuserdata/
# Python bytecode (e.g. clients/android/ci tooling)
__pycache__/
*.pyc
# Claude Code project instructions — local to each dev box, not part of the repo.
CLAUDE.md
-585
View File
@@ -1,585 +0,0 @@
# CLAUDE.md — punktfunk
Low-latency desktop/game streaming stack, Linux-first, with a shared Rust protocol core
(`punktfunk-core`) exposed over a C ABI and native clients per platform. Full design:
[`design/implementation-plan.md`](design/implementation-plan.md). Status table: `README.md`.
## Where the work stands
- **Core (`punktfunk-core` + C ABI): complete and hardened.** FEC recovery, loopback-under-loss,
proptests, C ABI harness all green; 13 adversarial-review findings fixed +
regression-tested (`a913042`).
- **GameStream host: working end-to-end with a stock Moonlight client.** Validated live
on this box: pairing (persists across restarts), serverinfo/applist (app catalog from
`~/.config/punktfunk/apps.json` → each entry picks a compositor + nested command), RTSP, ENet
control, audio, and video at the **client's native resolution and refresh** — the host
creates a per-session virtual output via per-compositor `VirtualDisplay` backends:
**KWin** (`zkde_screencast stream_virtual_output`, needs KWin ≥ 6.5.6 headless; >60 Hz via
custom modes), **gamescope** (spawned headless at WxH@Hz, its PipeWire node captured, needs
gamescope ≥ 3.16.22 — older deadlocks on PipeWire ≥ 1.6), **Mutter** (D-Bus
`RecordVirtual` virtual monitor; validated live on headless GNOME Shell 50, zero-copy),
**Sway/wlroots** (`swaymsg create_output` + custom mode, xdpw portal capture with a
managed chooser config; validated live on sway 1.11, zero-copy).
Performance work landed and measured: GPU **zero-copy** on all paths (tiled dmabuf →
EGL/GL → CUDA; LINEAR dmabuf → **Vulkan bridge** → CUDA → NVENC), auto 2-way NVENC
split-encode above ~1 Gpix/s (5K@240), infinite GOP + RFI keyframes (killed the periodic
freeze), encode|send thread split with `sendmmsg` batching. Stable 240 fps at 5120×1440.
Input: mouse/keyboard (libei via RemoteDesktop portal on KWin/GNOME, gamescope's own EIS
socket, wlr protocols on Sway) and **gamepads** (uinput X-Box-360 pads + rumble
back-channel; validated live — pad created/destroyed with the session). Management REST API +
checked-in OpenAPI doc (`mgmt.rs`). **Web-console performance capture** (`stats_recorder.rs`,
design: [`design/stats-capture-plan.md`](design/stats-capture-plan.md)): the operator arms stats
recording from the web console, plays, stops, and reviews the run as graphs (per-stage latency
breakdown · fps new/repeat · goodput · loss/FEC). A shared `Arc<StatsRecorder>` ring (the hot-path
gate is a runtime `AtomicBool`, replacing the startup-only `PUNKTFUNK_PERF`) is fed by **both** the
native `virtual_stream` and the GameStream encode loop at their existing ~2 s/~1 s aggregation
boundary, and finished captures are saved as on-disk recordings
(`~/.config/punktfunk/captures/*.json`) browsable/exportable from the console's **Performance** page
(recharts). Endpoints `/api/v1/stats/*` (bearer-only). *Implemented; not yet on-glass validated.*
**Web-console log view** (`log_capture.rs`): a `tracing` layer tees DEBUG-and-up (independent of
`RUST_LOG`) into a 4096-entry in-memory ring, served cursor-paged at `GET /api/v1/logs`
(bearer-only) → the console's **Logs** page (follow/pause · level filter · search). The Windows
gamepad drivers now stamp attach/heartbeat marks into their shm sections and the host's
`DriverAttach` watcher turns silence into a one-shot diagnosis WARN (driver-store check + CM
devnode problem code) — failure-mode table: [`design/gamepad-driver-health.md`](design/gamepad-driver-health.md).
The Android client gained Settings → **Connected controllers** (device list + VID/PID + resolved
pad type + live input test) for the client end of the same chain. *Log view + driver health:
Linux-tested; Windows/Android sides CI/device-validation pending.*
- **Native protocol (`punktfunk/1`): full session planes, validated live.** QUIC
control plane (`punktfunk-core` `quic` feature: Hello{mode}/Welcome{full Config}/Start), data
plane = the hardened core `Session` over raw UDP with **GF(2¹⁶) Leopard FEC + AES-GCM**
(inexpressible in GameStream), host creates the native virtual output at the client's
requested mode. `punktfunk1-host` is a **persistent listener** (sessions back to back;
`--max-sessions`). QUIC datagrams carry the side planes, demuxed by first byte: input
0xC8 (incl. **gamepads** — incremental events accumulated into the uinput xpad), **Opus
audio** 0xC9 (48 kHz stereo, 5 ms, host→client), **rumble** 0xCA (host→client). **Trust:**
host serves its persistent identity (`~/.config/punktfunk/cert.pem`, shared with GameStream
pairing) and logs the SHA-256 fingerprint; clients pin it, established by a **SPAKE2 PIN pairing
ceremony** (host arms pairing and displays a 4-digit PIN; a PAKE binds both cert fingerprints so an
attacker gets one online guess, no offline dictionary attack) — PIN pairing is the default for new
hosts. **TOFU on first connect** (`endpoint::client_pinned`) stays as an explicit host opt-in
(`punktfunk1-host --allow-tofu` / `serve --open`, advertised as `pair=optional`) for fully trusted LANs;
clients only offer the TOFU "Trust" path for a host that advertised `pair=optional`, route every
other new host straight to the PIN ceremony, and on a pinned-fingerprint change force re-pairing
(no re-TOFU shortcut). Clients present persistent identities via QUIC client auth, the host stores
paired fingerprints (`punktfunk1-paired.json`) and gates sessions with `--require-pairing` (the
default; `--allow-tofu`/`--open` accept unpaired clients).
**LAN auto-discovery**: both `serve` and `punktfunk1-host` advertise the native service over
mDNS (`_punktfunk._udp`, `crate::discovery`) with TXT `proto`/`fp`(cert fingerprint to
pin)/`pair`(required|optional)/`id`; `punktfunk-probe --discover` lists hosts, Apple clients
browse the same service via NWBrowser (validated cross-LAN 2026-06-12).
**Mid-stream mode renegotiation**: `Reconfigure` on the still-open control stream — the
host rebuilds output+encoder at the new mode in ~90 ms while the data plane runs on
(validated live: one .h265 with 720p and 1080p segments). Measured on-box at 720p120: 1680/1680 frames, **p50 0.83 ms**
capture→…→reassembled; audio measured live (~200 pkts/s). A **wall-clock skew handshake**
(`ClockProbe`/`ClockEcho`, 8 NTP rounds after `Start`, `clock_offset_ns`) aligns the client to the
host clock, so that latency is now valid **cross-machine** (`skew_corrected=true`) — measured GNOME
box → dev box over the LAN: **p50 1.30 ms** (the 1.57 ms inter-box clock offset removed).
`punktfunk-probe` is the
working reference client (`--pin`, datagram counters, `--input-test` incl. gamepad).
The embeddable connector (`NativeClient`) exposes it all over the C ABI: `punktfunk_connect`
(pin/TOFU) + `next_au`/`next_audio`/`next_rumble`/`next_hidout`/`send_input`/
`send_rich_input`. **Client-negotiated virtual pad type**: the Hello carries a gamepad
preference byte (same trailing-byte back-compat pattern as the compositor), the Welcome
echoes the resolved backend — precedence: explicit client choice > `PUNKTFUNK_GAMEPAD`
env > uinput Xbox 360. Backends: **Xbox 360** (uinput on Linux / the pf-xusb UMDF driver on
Windows), **Xbox One/Series** (the same
XInput backend with the One/Series USB identity for matching glyphs — no extra game-visible
capability; impulse-trigger rumble is unreachable through a virtual pad), and the UHID
`hid-playstation` pads — **DualSense** (adaptive triggers, lightbar, touchpad, motion) and
**DualShock 4** (lightbar, touchpad, motion, rumble; DualSense minus adaptive triggers / player
LEDs / mute). DualSense and DualShock 4 each have a Linux (UHID `hid-playstation`) **and a Windows
(UMDF minidriver)** backend — `inject/windows/dualsense_windows.rs` + `inject/windows/dualshock4_windows.rs`, one
driver serving either identity per a `device_type` byte the host stamps into shared memory (the DS4
reuses the same SwDeviceCreate game-detection identity fix as the DualSense). One/Series stays
Linux-only and folds into Xbox 360 off it. Clients auto-resolve the type from the physical controller
(DS5→DualSense, DS4→DualShock 4, Xbox One→Xbox One). **Windows uses ZERO external gamepad
dependencies — ViGEmBus is gone.** Xbox 360 (XInput) runs on a UMDF2 **XUSB companion** driver
(`packaging/windows/drivers/pf-xusb/`, `inject/windows/gamepad_windows.rs`) that registers `GUID_DEVINTERFACE_XUSB`
and answers the buffered XInput IOCTLs from a shared section, so classic `XInputGetState`/`SetState`
work with no kernel bus driver (validated live: slot connected, state + rumble round-trip; Xbox One
folds to this 360 path). All three UMDF drivers (DualSense/DS4 + XUSB) are built from source in CI
(`packaging/windows/drivers/`) and installed by the Inno Setup installer via
`punktfunk-host.exe driver install --gamepad`. The gamepad drivers' **business logic is 100 % safe
Rust**: every raw shared-memory / sealed-channel / WDF-request operation lives behind
`pf-umdf-util` (the audited unsafe layer — `section::MappedView` checked accessors, the
`#![forbid(unsafe_code)]` `channel::ChannelClient` state machine, `wdf::Request` tokens), so a
memory-safety bug can only live in that one small crate. The whole drivers workspace is lint-gated
(`deny(unsafe_op_in_unsafe_fn)` + `deny(clippy::undocumented_unsafe_blocks)`) with a
`cargo clippy -D warnings` step in `windows-drivers.yml`; pf-vdisplay stays FFI-bound (D3D11/IddCx)
but every `unsafe {}` there now carries a `// SAFETY:` proof (unsafe-audited, not unsafe-free).
**Multi-pad ready**: the host stamps each pad's index into the device Location (`pszDeviceLocation`),
the driver reads it (`WdfDeviceAllocAndQueryProperty`) to map its own `*-shm-<index>`, and
`UmdfHostProcessSharing=ProcessSharingDisabled` gives each pad its own host (per-pad statics) —
validated live with 2 distinct XInput slots + 2 DualSense pads. (Client-side multi-pad forwarding is
the remaining piece.)
- **Windows host: implemented and shipping (all-vendor, x64-only, Windows 11 22H2+).** The OS floor
is HARD: pf-vdisplay is built against IddCx 1.10 (1.10 stub + HDR `*2` DDIs + FP16 caps, no runtime
downgrade) — on Windows 10 (incl. LTSC) / Win11 21H2 the driver installs but the device fails start
with Code 10 `STATUS_DEVICE_POWER_FAILURE` (field-reported 2026-07); the installer gates on
`MinVersion=10.0.22621`. `#[cfg(windows)]` backends
behind the same traits as Linux — **IDD-push capture** straight into the in-house all-Rust IddCx
**pf-vdisplay** virtual display (`capture/windows/idd_push.rs`, `vdisplay/windows/pf_vdisplay.rs`;
DXGI Desktop Duplication / WGC as fallbacks, `capture/windows/dxgi.rs`). The host↔driver frame
ring is a **sealed channel** (proto v2, `design/idd-push-security.md`): all shared objects
UNNAMED, handles `DuplicateHandle`d into the driver's WUDFHost and delivered as values over
`IOCTL_SET_FRAME_CHANNEL` (SY+BA-only control device) — only the two endpoint processes can ever
reach a frame (DDA's isolation property in user mode; adopt-on-success handle-ownership contract,
newest-delivery-wins re-attach). *Sealed channel: CI-pending + on-glass revalidation pending.*
The **gamepad SHM channels are sealed the same way** (gamepad proto v2,
`design/gamepad-channel-sealing.md`): the pad DATA sections (`XusbShm`/`PadShm`, now with a
driver-validated `pad_index`) are unnamed + handle-duplicated into the pad WUDFHost
(`gamepad_raii.rs` `PadChannel`); since the HID minidrivers have no control device, the handshake
runs over a tiny named bootstrap mailbox (`Global\pf…-boot-<i>`, pid + handle value only — tampering
is DoS-bounded). *Sealed pad channel: needs both pad drivers redeployed with the host, physical-pad
validation pending.* GPU encode (NVENC
`--features nvenc`; AMD/Intel `--features amf-qsv`), SendInput + the in-house UMDF gamepad drivers
(`inject/windows/`), WASAPI loopback + virtual mic (`audio/windows/wasapi_*`). **Keyboard wire
convention: US-positional VKs** (every first-party client sends the physical key's US-layout VK;
the Windows client derives it from the scancode, NOT the layout-resolved `vkCode`) — the Windows
injector resolves them via a fixed table mirroring the Linux `vk_to_evdev` (never through a
keyboard layout: the SYSTEM service thread's layout re-reads positions as characters — the
German y↔z / ö→ü scramble), while GameStream/Moonlight VKs are layout-semantic
(`KEY_FLAG_SEMANTIC_VK`, resolved under the foreground app's layout, Sunshine's model). Linux
renders positions under the session compositor's layout (libei) or the virtual keyboard's
uploaded keymap (Sway/wlroots — honors `XKB_DEFAULT_LAYOUT` et al., default US); the Android
client reads `KeyEvent.scanCode` first so a user-selected physical-keyboard layout can't
re-map keycodes semantically. Ships as a **signed
Inno Setup installer** that registers a `LocalSystem` SCM service launching into the interactive
session for secure-desktop (UAC/lock-screen) capture (`windows/service.rs`), bundles the
pf-vdisplay driver + the FFmpeg DLLs (+ VB-CABLE for the virtual mic), and is published by
`windows-host.yml`. **Encoder is GPU-aware** (`encode.rs` `open_video` + `windows_resolved_backend`):
`PUNKTFUNK_ENCODER=auto` (the host.env default) reads the **selected render adapter's** vendor →
**NVENC** (NVIDIA, direct SDK, `encode/windows/nvenc.rs`), **AMF** (AMD) / **QSV** (Intel) via libavcodec
(`encode/windows/ffmpeg_win.rs`, the Windows analogue of the Linux VAAPI backend — `WinVendor{Amf,Qsv}`,
system-memory NV12/P010 readback default + opt-in zero-copy D3D11 behind `PUNKTFUNK_ZEROCOPY` with a
system fallback), or software H.264 (`encode/sw.rs`, GPU-less). GameStream codec advertisement is
probed per-GPU on AMF/QSV (`windows_codec_support``serverinfo`, AV1 gated; cached per selected
GPU). **Multi-GPU is first-class** (`gpu.rs`): GPU inventory + a persisted auto/manual preference
(`<config>/gpu-settings.json`, stored by stable PCI identity — LUIDs are per-boot) exposed over
`GET /api/v1/gpus` + `PUT /api/v1/gpus/preference` and a web-console GPU card (Host page: list,
Automatic/Prefer, "In use · backend" badge). One selection — precedence **console preference >
`PUNKTFUNK_RENDER_ADAPTER` > max VRAM**, graceful fallback when the preferred GPU is absent —
feeds `win_adapter::resolve_render_adapter_luid` (capture ring + IddCx render pin), the encoder
vendor auto-detect (previously DXGI adapter 0 — wrong on hybrid boxes like NVIDIA dGPU + Intel
Arc iGPU), and the NVENC 4:4:4 probe; a preference change applies to the next session. On Linux a
matched manual preference picks the VAAPI render node / NVENC-vs-VAAPI auto choice (auto mode
unchanged). *Implemented + unit-tested; not yet on-glass validated on the hybrid box.* **HDR (10-bit)**: WGC
captures the HDR desktop as FP16/Rgb10a2 (DDA FP16 for the secure desktop), the encoder forces HEVC
Main10 + BT.2020 PQ (NVENC ABGR10/P010; AMF/QSV P010 + a swscale Rgb10a2→P010 fallback), the client
auto-detects PQ from the HEVC VUI — gated by `PUNKTFUNK_10BIT` + client `VIDEO_CAP_10BIT`; **Windows
host only** (the Linux host stays 8-bit, blocked upstream). **Vulkan-game HDR over the virtual
display**: NVIDIA/AMD Vulkan ICDs refuse to *advertise* an HDR color space for a surface on an IddCx
indirect display (so Vulkan games — Doom: The Dark Ages, id Tech, etc. — say "device does not support
HDR"), even though the ICD happily *accepts + presents* a forced HDR swapchain there. A tiny always-on
Vulkan **implicit layer** (`packaging/windows/pf-vkhdr-layer/`, `VK_LAYER_PUNKTFUNK_hdr_inject`)
injects the `HDR10_ST2084`/scRGB surface formats into `vkGetPhysicalDeviceSurfaceFormats[2]KHR`,
self-gated on the display's actual advanced-color state (no-op on SDR / real monitors); bundled +
HKLM-registered by the installer. **Live-validated: Doom: The Dark Ages enables HDR over the virtual
display.** **AMF/QSV is CI-green but not yet on-glass validated** (no AMD/Intel Windows box in the
lab); NVENC is live-validated. Newer/less battle-tested than the Linux host. Packaging: `packaging/windows/`.
- **Status tray (`crates/punktfunk-tray`, Windows + Linux).** A small per-user companion binary
showing the host service state at a glance (running / stopped / starting / degraded / failed +
streaming session in the tooltip) with one-click actions: open web console, approve-pairing
shortcut, start/stop/restart, open logs, exit. Status precedence is **service manager first**
(SCM / systemd user unit — a port-squatter can't fake Running), then the new **loopback-only
unauthenticated** `GET /api/v1/local/summary` (counts/booleans only — no PINs/fingerprints/names;
gated in `require_auth` by peer address, needed because `mgmt-token`/`cert.pem` are
SYSTEM/Admins-DACL'd on Windows so a non-elevated tray cannot bearer-auth). Windows:
`#![windows_subsystem = "windows"]` hidden-window + `Shell_NotifyIconW` (per-session `Local\`
mutex, TaskbarCreated re-add, `--quit` for the uninstaller), actions elevate per click via
`ShellExecuteW "runas"` on `punktfunk-host.exe service start|stop|restart` (new `service restart`
subcommand: stop → wait Stopped → start); installed by the Inno `trayicon` task (HKLM Run key).
Linux: ksni (SNI over zbus, `async-io`+`blocking` features), `systemctl --user` actions (no
polkit), `/etc/xdg/autostart` entry whose `--autostart` self-gates (silent exit unless
`~/.config/punktfunk` exists or the unit is enabled); deb/rpm/arch ship binary + autostart +
hicolor icons. Icons generated by `scripts/gen-tray-icons.py` (pure-stdlib; committed .ico/.png,
brand lens + status dot). *Linux live-validated on the headless KDE session (SNI registration,
stop/start transitions, menu-driven start, dbusmenu layout); Windows code MSVC-cross-type-checked
+ clippy-clean but real Windows CI build + on-glass validation pending.*
## What's left
1. **Native clients — decode + present: macOS stage 1 done, first light achieved
(2026-06-10).** PunktfunkKit compiles and is tested on macOS (AnnexB → VideoToolbox →
`AVSampleBufferDisplayLayer`, GCMouse/GCKeyboard capture, `PunktfunkClient` app shell);
validated live Mac ↔ this box at 720p60 — vkcube on glass, input injected via gamescope
EIS. The app speaks the full ABI v2 trust surface: Keychain-persisted client identity
presented on every connect, SPAKE2 PIN pairing UI (host-card context menu + the trust
prompt's "Pair with PIN instead…"), TOFU fingerprint prompt. **Gamepads (2026-06-11):**
controller discovery + selection in Settings (`GamepadManager` — exactly one pad
forwarded as pad 0, auto or pinned; pad TYPE auto-resolves from the physical
controller, user-overridable), capture incl. DualSense touchpad/motion
(`GamepadCapture`/`GamepadWire`; while streaming, EVERY element's
`preferredSystemGestureState` is claimed `.disabled` — share/create reaches the host as
select instead of screenshotting locally, PS/Home reaches the host as guide/`BTN_MODE` =
the Steam-overlay button — restored `.enabled` on unbind), feedback rendering (rumble → CoreHaptics; lightbar /
player LEDs / adaptive triggers → `GCDeviceLight`/`playerIndex`/
`GCDualSenseAdaptiveTrigger` via the table-driven `DualSenseTriggerEffect` parser).
Loopback-tested end to end (`PUNKTFUNK_TEST_FEEDBACK=1` scripted burst); DualSense
motion sign/scale derived, not yet live-verified. **Rumble renderer rewritten
(2026-07-02, `RumbleRenderer.swift`)** around "rumble is idempotent state, divergence
must be bounded": the old per-datagram infinite-duration CoreHaptics players could leak
one dropped async `stop` into a forever-buzzing motor (the stuck-rumble-after-menu bug)
— now finite self-expiring segments with seamless engine-timeline re-arm, newest-wins
dry drain of the 0xCA plane (was 1 datagram/8 ms), dedupe of the host's 500 ms state
refreshes, zero-immediate/ramp-throttled rebakes, escalation to `engine.stop()` on a
throwing player stop, and a 1.6 s staleness watchdog (`Policy.session`; the settings
test panel uses `.manual` = hold). Controller engines use **plain `makePlayer` — never
`makeAdvancedPlayer`**: the controller haptics server (gamecontrollerd) advertises
`adv players: 0`, and iOS 27 beta 2 hard-drops advanced-player loads (XPC decode fault →
CoreHaptics -4811/4097, rumble silently dead). Unit-tested (`RumbleTuningTests`);
stuck-rumble repro on-glass revalidation pending. **Gamepad UI (iOS/iPadOS + macOS,
2026-07-02 rework):** a connected pad swaps the home for a console-style launcher
(`Home/Gamepad*` + `Settings/GamepadSettingsView`) — host carousel with a trailing Add
Host tile (A connect · Y library · X settings · B back), a controller-navigable
settings screen (vertical `GamepadMenuList`, left/right steps values), an add-host
flow with an on-screen controller keyboard (`GamepadKeyboard` — no touch needed
anywhere), and the coverflow library, all over an animated aurora backdrop
(`GamepadScreenBackground`, TimelineView-driven drifting blobs — pure SwiftUI ON
PURPOSE: a .metal lib only reliably bundles in one of the two build systems (SPM vs
xcodeproj synced folders) these sources compile under). Input is the polled
`GamepadMenuInput` (handlers don't fire outside a stream; on (re)start it SNAPSHOTS
held buttons so a handoff press never double-fires), haptics dual-channel (device +
`MenuHaptics` on the pad). macOS: same screens, settings/add-host as sheets (no
fullScreenCover), NSScreen-based mode lists, scroll indicators `.never` (macOS
"always show scroll bars" overrides `.hidden`); launcher/settings/add-host/keyboard
render-verified live on this Mac via `PUNKTFUNK_FORCE_GAMEPAD_UI=1` (dev hook, forces
the mode without a pad). Controller-in-hand on-glass validation still pending on all
platforms. **Touch input (iOS/iPadOS, 2026-07-02):** a 3-way model in Settings —
**Trackpad** (default; the Android client's gesture vocabulary ported 1:1 in
`Input/TouchMouse.swift`: tap=click · two-finger tap=right-click · two-finger drag=scroll ·
tap-then-drag=held drag · three-finger tap=HUD toggle, relative ballistics with the same
px-based acceleration curve), **Direct pointer** (cursor jumps to the finger), **Touch
passthrough** (the previous always-on behavior — real wire touches). Latched per gesture
from `DefaultsKey.touchMode`; not yet on-glass validated. Tests: `swift test` in
`clients/apple` (unit + real-codec round trip),
`test-loopback.sh` (Swift client vs synthetic punktfunk1-hosts on loopback — runs on macOS;
includes the pairing ceremony + `--require-pairing` gate),
`RemoteFirstLightTests` (full pipeline over the LAN). See
[`clients/apple/README.md`](clients/apple/README.md). **Stage 2 presenter is now the DEFAULT**
(stage-1 is the Metal-unavailable / DEBUG fallback): explicit `VTDecompressionSession` decode →
`CAMetalLayer`, presented from the hosting view's **main-runloop `CADisplayLink`** (`renderTick` pops
the newest ready frame per vsync; macOS `displaySyncEnabled = false` is the real fullscreen-judder fix,
~11 ms p50). *(An off-main `CAMetalDisplayLink` and an off-main blocking-render present thread were
both tried and reverted — both measured slower on macOS and iPad.)* **HDR fixed**
(`design/apple-stage2-presenter.md`): the "too bright" bug was a missing reference-white anchor — the
fix keeps the PQ-passthrough shader and adds `CAEDRMetadata.hdr10(…, opticalOutputScale: 203)` +
`wantsExtendedDynamicRangeContent` on the layer (on all platforms; the old `#if os(macOS)` guard left
iOS/tvOS EDR half-engaged), routing the 0xCE mastering metadata to the layer (via `setHdrMeta`) instead
of a never-composited source buffer. **Mid-session SDR↔HDR** is handled: `render` reconciles the layer
per-frame from the decoded `frame.isHDR` (per-mode pixel format `bgra8`/`rgba16Float`), so a game
entering HDR mid-stream just reconfigures (last 0xCE grade cached + re-applied; pump drains 0xCE
unconditionally). **4:4:4 added**: decode format is a 2×2 `(chroma, HDR)` matrix
(`420v/x420/444v/x444`, all biplanar so the shaders are unchanged), advertised (`VIDEO_CAP_444`) only
behind a **hardware-required `VTDecompressionSession` probe** (`Stage444Probe`, validated on M3) with a
Settings opt-out + a bounded pump backstop for an undecodable 4:4:4 session. *HDR brightness + 4:4:4
still need on-glass validation (Windows-HDR / `PUNKTFUNK_444` host).* Next: glass-to-glass numbers via
`tools/latency-probe`.
**Linux stage 1 done, first light 2026-06-12** (`clients/linux`, binary
`punktfunk-client`): GTK4/libadwaita shell linking `punktfunk-core` directly (no C ABI;
`NativeClient` is now `Sync` — mutexed plane receivers), mDNS host list, TOFU + SPAKE2
PIN dialogs (identity shared with client-rs), FFmpeg software HEVC decode (LOW_DELAY,
slice threads) → `GtkGraphicsOffload`-wrapped picture, PipeWire playback (mic-player
jitter ring inverted), SDL3 gamepad capture + rumble/lightbar feedback, keyboard via
exact inverse of the host VK table, absolute mouse + 120-unit scroll. Validated live
against `serve` on this box: 1080p60, steady 60 fps, capture→decoded p50
≈6.4 ms (debug build). `--connect host[:port]` for scripting. **Swift-parity batch +
stage 1.5 (2026-06-12 evening)**: capture state machine (click-to-capture,
Ctrl+Alt+Shift+Q / focus-loss release, held-state flush), app-lifetime SDL gamepad
service (pad pin UI, auto type from the physical pad, DualSense touchpad/motion 0xCC +
raw-DS5-effects trigger/player-LED replay — needs a physical pad to live-verify), mic
uplink (validated live), per-host speed test, compositor pref, native-display mode
default, saved-hosts list, .deb + RPM-subpackage CI (deb.yml/rpm.yml). **VAAPI decode
→ DRM-PRIME dmabuf → `GdkDmabufTexture`** (BT.709 color state; Tier-1 zero-copy on
Intel/AMD, `PUNKTFUNK_DECODER=software|vaapi` override) with a proven fallback ladder —
no VAAPI device (NVIDIA) or mid-session VAAPI error → software decode. **First AMD test
(Steam Deck) hit a green-screen bug, fixed:** FFmpeg's VAAPI export uses
`SEPARATE_LAYERS`, so NV12 arrives as two single-plane layers (R8 luma + GR88 chroma,
one shared fd); the mapper took `layers[0]` only → GTK got a luma-only R8 texture, chroma
read as 0 → green field / red whites. Fix derives the combined fourcc from the decoder
`sw_format` (→ `DRM_FORMAT_NV12`) and flattens all planes across all layers (mpv's
pattern); a first-frame descriptor dump logs the real layout. Awaiting Steam Deck
reconfirm. Next: the stage-2 raw-Wayland
presenter (wp_presentation feedback, tearing-control, Vulkan Video on NVIDIA) —
**wgpu/winit rejected** (no dmabuf import / presentation feedback / shortcuts-inhibit).
**Windows stage 1 done 2026-06-15** (`clients/windows`, binary
`punktfunk-client`): pure-Rust **WinUI 3** UI via **windows-reactor** (a declarative React-like
framework backed by WinUI; PR #4499 added the `SwapChainPanel` widget + `set_swap_chain`). The
video is a **`SwapChainPanel`** bound to a **D3D11 composition swapchain**, presented from a
**dedicated render thread** (`render.rs`, 2026-07-02 rewrite — presenting never touches or is
stalled by the XAML thread): frame-latency-waitable swapchain + `SetMaximumFrameLatency(1)`
(≤1 queued present, newest-wins drain after the wait, so a stream faster than the display drops
backlog before any GPU work), **HiDPI-correct** (pixel-sized buffers + `SetMatrixTransform`
96/DPI — DIP-sized buffers were blurry at 125/150 %), Contain-fit letterbox, WARP fallback.
**FFmpeg decode with a D3D11VA hardware path on all vendors** (`gpu.rs` shares one D3D11 device
between decoder + presenter, adapter picked by console pref `PUNKTFUNK_ADAPTER` > the window's
monitor's adapter > default; `PUNKTFUNK_D3D_DEBUG=1` adds the debug layer): the decode pool is
**decoder-only bind, sized/aligned by libavcodec itself** (get_format returns `AV_PIX_FMT_D3D11`
and lets `hw_device_ctx` drive — three hand-built-frames-context strikes are why: NVIDIA rejects
`DECODER|SHADER_RESOURCE` arrays, `BindFlags=0` fails texture creation, and Intel rejects
non-128-aligned HEVC surfaces at the first `SubmitDecoderBuffers`), a DXVA **profile probe**
before the hwdevice commits software-vs-hardware up front (no burned first IDR), and the
presenter copies the decoded slice with ONE display-size-boxed `CopySubresourceRegion` (a planar
slice is a single subresource in D3D11 — the old two-copy D3D12-style code silently no-opped =
the black screen) into its sampleable NV12/P010 texture → per-plane SRVs + YUV→RGB shaders
(NV12/BT.709, P010/BT.2020-PQ). **Software CPU decode is the fallback** (auto-selected,
`DecoderPref` override, mid-session demotion + keyframe re-request) and now feeds the SAME
shaders (swscale → NV12/P010 planes → two dynamic plane textures) so hw/sw colour math is
identical. **HDR10**: the client advertises 10-bit/HDR (Settings toggle, gated on an HDR
display), detects PQ in-band (`transfer == SMPTE2084`), and flips the swapchain to
`R10G10B10A2` + ST.2084 with HDR10 metadata (0xCE mastering metadata plumbed). **WASAPI** render
+ mic capture, **SDL3** gamepads (rumble/lightbar/DualSense), `mdns-sd` discovery, and the full
trust surface — all **in-app**: a polished WinUI shell (host tiles w/ monogram + status pills,
`InfoBar` errors/hints, `ToggleSwitch` settings, status-chip stream HUD showing GPU/CPU decode +
HDR), host list (live mDNS + saved + manual), settings (resolution/refresh/decoder/bitrate/HDR/
mic), SPAKE2 PIN pairing screen, TOFU, pinned-fp-mismatch re-pair. **Live-validated 2026-07-02
on the hybrid laptop (Intel Arc Pro iGPU + RTX 3500 Ada) against the local Windows host**:
D3D11VA hardware decode 60 fps on BOTH vendors (headless, `PUNKTFUNK_ADAPTER`-forced; NVIDIA
0.2 ms decode, Intel 0.2 ms), software path, and the GUI on glass (real decoded desktop pixels,
GPU-decode HUD chip, ~18 ms capture→decoded p50 over loopback — dominated by the host's 60 Hz
virtual-display capture cadence). HDR-on-glass still pending. **Stream input** is Win32 low-level hooks (`WH_KEYBOARD_LL`/`WH_MOUSE_LL`) — reactor
exposes no raw key/pointer events; native Windows VK + absolute mouse (client-rect Contain-fit) +
wheel, Ctrl+Alt+Shift+Q capture toggle. `--headless`/`--discover` keep CLI paths. Builds + clippy
+ fmt green on **`x86_64-pc-windows-msvc` and `aarch64-pc-windows-msvc`** — the latter
**cross-compiled off the one x64 runner** (no ARM64 runner; the x64 MSVC toolset's ARM64 cross
compiler + a per-arch `FFMPEG_DIR` ARM64 tree, SDL3/libopus build-from-source cross-compile
cleanly), and both ship as signed MSIX (`windows-msix.yml` matrix → `..._x64.msix`/`..._arm64.msix`,
verified: ARM64 binaries + manifest arch). **windows-reactor is unpublished** (git
dep pinned to commit `a4f7b2cb`, bumped 2026-07-02 from `b4129fcc` for `on_pointer_entered`/
`on_pointer_exited` hover events — mechanical renames only: `SymbolGlyph``Symbol`,
`placeholder``placeholder_text`, TextBox `on_changed``on_text_changed`, ToggleSwitch
`on_changed``on_toggled`, `on_menu_item_clicked``on_item_clicked`, SwapChainPanel
`on_ready``on_mounted`; `windows` pinned to the SAME commit so `IDXGISwapChain1` unifies with
`set_swap_chain`). New-model runtime staging: reactor has NO build.rs anymore — the app's own
`build.rs` calls `windows_reactor_setup::as_framework_dependent()` (same-rev build-dep, stages
the bootstrap DLL + resources.pri that pack-msix expects) and `main` calls
`windows_reactor::bootstrap()` before `App` (packaged MSIX: a no-op, the manifest's
`Microsoft.WindowsAppRuntime.2` dependency resolves the runtime). `CARGO_WORKSPACE_DIR` is no
longer required (harmless where still set). Gotcha: `CARGO_HOME` must be an ASCII path
— the `ü` in the dev box's username breaks SDL3's MSVC precompiled-header build. **Parity/cleanup
batch (2026-07-02)**: `app.rs` split into per-screen `app/` modules (mod=root/router · hosts ·
connect · pair · speed · settings · licenses · stream · style; thread-driven state lives in ROOT
`use_async_state` and flows down as props — a child's own async-state write does NOT re-render it);
"Native display" now resolves the real monitor mode at connect (`MonitorFromWindow`
`EnumDisplaySettingsW`, was hardcoded 1080p60); per-host **speed test** (saved-host card button +
`--headless --speed-test`, probe burst → recommended ≈70 % bitrate applied in one tap; bitrate
setting is now a free-form NumberBox); **forget host** (ContentDialog confirm →
`KnownHosts::remove_by_fp`); settings gained forwarded-controller picker + gamepad type + host
compositor + capture-system-shortcuts — the previously-dead `Settings.compositor`/
`inhibit_shortcuts` are now honored (off = Alt+Tab/Alt+Esc/Ctrl+Esc/Win act locally);
**click-to-recapture** after a Ctrl+Alt+Shift+Q release with the HUD hint tracking capture state;
input hook caches lock geometry (no per-move `GetClientRect`), audio jitter-ring trims via
`drain`. Validated on the bare-metal RTX box: `--discover` (3 live LAN hosts), synthetic-host
loopback E2E (TOFU connect → clock skew → HEVC negotiate → shared-D3D11 + D3D11VA init → WASAPI →
session end; synthetic payload isn't decodable so decode output stays unvalidated), speed-test
E2E. The WinUI window itself CANNOT be launched from SSH (session-0 → WinAppSDK 0x80070005,
pre-existing; needs the console session, e.g. PsExec -i 1). **UX batch (2026-07-02 evening,
UIA-smoke-tested on the hybrid laptop)**: host tiles get the WinUI pointer-over fill
(`on_pointer_entered`/`exited` → root hover state → `ControlFillSecondary`), Settings is a stock
**NavigationView** sidebar (Windows-Settings pattern: Display/Video/Input/Audio/About panes,
built-in back arrow, section in root state; the section card is **keyed by section** — an
in-place diff across sections re-sets a reused ComboBox's items, clearing WinUI's selection,
but skips `selected_index` when the values compare equal → blank selection; the key forces a
remount — and the content column rides its own section-switch slide-up tween), new
**"Show the stats overlay (HUD)"** toggle
(`Settings::show_hud`, applies mid-stream via the 400 ms HUD re-render), the Add-host modal
slides up + fades in (root margin/opacity tween, same pattern as screen navigation), and a
self-initiated disconnect (Ctrl+Alt+Shift+D → `Ended(None)`) returns to the host list silently
instead of raising the error banner.
Next: **HDR on-glass validation** (Windows host with `PUNKTFUNK_10BIT` → the HDR laptop
display), then RAWINPUT relative-mouse pointer-lock.
**Android stage 1 done** (`clients/android`, Kotlin app + `native/` Rust JNI core linking
`punktfunk-core`; phone + Android TV): NDK `AMediaCodec` hardware HEVC decode → `SurfaceView` incl.
**HDR10** (Main10/BT.2020 PQ) with low-latency tuning + a live stats HUD (`decode.rs`/`stats.rs`),
Opus/AAudio audio + mic uplink (`audio.rs`/`mic.rs`), gamepad input with rumble/HID feedback
(`feedback.rs`), **native `mdns-sd` mDNS discovery** (`discovery.rs`, polled over JNI — the same
browse the Linux/Windows clients use, replacing the flaky per-OEM `NsdManager`; Kotlin keeps only
the `MulticastLock` + permission UX), SPAKE2 PIN pairing + TOFU (Keystore identity +
known-host store), Compose UI (Connect/Settings/Stream) with D-pad/controller focus nav. Built for
`arm64-v8a` + `x86_64`; published to Google Play (Internal Testing) via `android.yml`
(`ci/play-upload.py`). Touch input is the same 3-way model as iOS (2026-07-02): the existing
Trackpad/Direct mouse modes plus new **real multi-touch passthrough**
(`streamTouchPassthrough``nativeSendTouch` → wire TouchDown/Move/Up), a `TouchMode`
Settings dropdown replacing the old trackpad Boolean (migrated on load); not yet
on-device validated. Next: real-device gamepad/HDR live-verify, presenter/latency polish.
2. **Sub-frame pipelining**: overlap encode and transmit within a frame. Requires a direct
NVENC SDK wrapper (libavcodec only emits whole AUs) — the next big latency lever (~24 ms
at high res).
3. **punktfunk/1 protocol growth.** **Done:** unified host (`serve --gamestream` runs GameStream + the
punktfunk/1 QUIC host in one process; bare `serve` is the secure native-only default — GameStream is
opt-in, trusted-LAN only, security-review #5/#9) with native pairing driven over the mgmt API /
web console (`mod native_pairing`: arm-on-demand → display PIN, paired-device list).
**Done:** PIN pairing is the default, host-gated — the host requires pairing and advertises
`pair=required` unless opted out with `--allow-tofu`/`--open` (then `pair=optional`, accepts
unpaired clients); clients render TOFU only for a `pair=optional` host and force re-pairing on a
fingerprint change. **Done:** concurrent sessions — the accept loop spawns each session
(`--max-concurrent`, default 4, an NVENC bound), each with its own virtual output + encoder, sharing
the host-lifetime input/audio/mic services (shared-desktop multi-view on kwin/mutter/wlroots).
**Done:** delegated pairing approval (§8b-1) — an unpaired device shows up as a pending request in
the web console, one click approves + pins it. Next (see roadmap): gamescope multi-user isolation
(per-session input/audio = independent desktops); §8b-2 peer-push approval from a paired device's
own app.
4. **GameStream host polish**: HDR/10-bit (needs HDR capture + metadata plumbing; `av1_nvenc
-highbitdepth 1` already encodes Main10 from 8-bit input on this box),
reconnect-at-new-mode robustness. AV1 negotiation and surround audio are implemented
and unit/live-capture tested — both still need a live Moonlight confirmation (select
AV1 in a stock client; a real 5.1/7.1 listen incl. FEC under loss).
Box one-time setup is complete: udev rule + `input` group (gamepads validated live),
gamescope 3.16.22 installed system-wide (no PATH override), gnome-shell installed (Mutter
backend validated live). All three compositor backends are live-validated.
## Build / test / run
```sh
cargo build --workspace # green on Linux and macOS
cargo test --workspace # unit + loopback + proptest + C ABI harness (~100 tests)
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all --check
cargo run -p loss-harness # FEC loss-resilience sweep (no network needed)
bash crates/punktfunk-core/tests/c/run.sh # standalone C-ABI link + round-trip proof
```
Generated artifacts are **checked in** and CI fails on drift: `include/punktfunk_core.h`
(cbindgen from `punktfunk-core/src/abi.rs`) and `api/openapi.json` (regenerate with
`cargo run -p punktfunk-host -- openapi > api/openapi.json`; spec lives in `mgmt.rs`).
CI is Gitea Actions (`.gitea/workflows/`, guide: docs-site `ci.md`): `ci.yml` runs the
workspace checks inside the `git.unom.io/unom/punktfunk-rust-ci` image plus web/docs-site
build+typecheck; `docker.yml` builds+pushes the web/docs/rust-ci images (host and native
clients are deliberately NOT containerized); `apple.yml` builds the xcframework and runs
`swift build`/`swift test` on the `macos-arm64` host-mode runner (home-mac-mini-1,
provisioned by `scripts/ci/setup-macos-runner.sh`). Per-client/host release workflows:
`deb.yml`/`rpm.yml`/`flatpak.yml` (Linux client), `android.yml` (Google Play), `windows-msix.yml`
(Windows client), `windows-host.yml` (Windows host installer), `release.yml` (Apple notarized DMG +
TestFlight), `decky.yml` (Steam Deck plugin); Windows builds run on a self-hosted Windows runner
(`home-windows-runner-1`, vmid 210, `windows-amd64:host` label). The runner is reproducible and
**owned by `unom/infra`**, not this repo, since it's shared across unom Windows projects going
forward: `unom/infra`'s `windows-runner/` Packer template bakes a generic Windows 11 template (OS
install + OpenSSH Server + VS Build Tools/NASM/CMake/LLVM + the act_runner/Node/rustup base, no
registration) on Proxmox once; `proxmox/windows-runner/` (Terraform, `bpg/proxmox`) full-clones it
(agent-based IP discovery, no pre-provisioned DHCP reservation needed) and registers the instance
over SSH remote-exec — the same bake-once/clone-fast split `proxmox/unom-1` uses for the Linux CI
host, just without a native Windows cloud-init (registration goes over `remote-exec`/SSH instead of
`initialization{}`; WinRM was tried first but is deprecated in OpenTofu, so this moved to SSH via
Windows' in-box OpenSSH Server). punktfunk layers its own extras on top of that generic runner:
`scripts/ci/provision-windows-wdk.ps1` (WDK + cargo-wdk for the UMDF drivers) and
`scripts/ci/provision-windows-punktfunk-extras.ps1` (FFmpeg x64/ARM64 trees, Inno Setup, the
`aarch64-pc-windows-msvc` rustup target) — both idempotent, and both run automatically at the start
of every Windows CI job via the shared `scripts/ci/ensure-windows-toolchain.ps1` step (a fast no-op
once already provisioned), rather than a separate manually-dispatched provisioning workflow — that
avoided a real footgun once there could be more than one `windows-amd64` runner: a manually
dispatched provisioning workflow has no way to target a *specific* runner instance, so it could
land on an already-provisioned box instead of the one that actually needed it.
## Layout
```
crates/punktfunk-core/ protocol · FEC · crypto · quic (punktfunk/1 control plane, feature-gated)
crates/punktfunk-host/
gamestream/ Moonlight compat: nvhttp · pairing · rtsp · control · stream · gamepad · apps
vdisplay/linux/{kwin,gamescope,mutter,wlroots}.rs per-compositor client-sized virtual outputs
vdisplay/windows/{pf_vdisplay,manager,identity}.rs all-Rust IddCx virtual display (pf-vdisplay)
linux/zerocopy/{egl,cuda,vulkan}.rs dmabuf → CUDA → NVENC (tiled via EGL/GL, LINEAR via Vulkan)
inject/linux/{libei,wlr,gamepad,dualsense,dualshock4,steam_*}.rs Linux input (uinput xpad · UHID pads · virtual Deck)
inject/windows/{sendinput,gamepad_windows,dualsense_windows,dualshock4_windows}.rs Windows input (UMDF shared-mem pads)
encode/linux/{mod,vaapi}.rs · encode/windows/{nvenc,ffmpeg_win}.rs · encode/sw.rs per-GPU encoders (NVENC/CUDA · VAAPI · AMF/QSV) + GPU-less openh264
capture/{linux/,windows/{dxgi,idd_push}}.rs · audio/{linux/,windows/wasapi_*}.rs
windows/{service,install,interactive}.rs SCM service + in-binary driver/web install
capture.rs · encode.rs · audio.rs · gpu.rs · spike.rs · punktfunk1.rs · mgmt.rs · native_pairing.rs · stats_recorder.rs · library.rs
crates/punktfunk-tray/ per-user status tray (Win32 Shell_NotifyIcon · Linux ksni/SNI); icons via scripts/gen-tray-icons.py
clients/probe/ punktfunk/1 reference/probe client (headless test/measurement tool)
clients/linux/ native Linux client (GTK4/libadwaita · FFmpeg · PipeWire · SDL3)
clients/windows/ native Windows client (WinUI 3 via windows-reactor · D3D11 · WASAPI · SDL3)
clients/apple/ native macOS/iOS/tvOS client (Swift · VideoToolbox · GameController)
clients/android/ native Android client (Kotlin app + native/ Rust JNI core over punktfunk-core)
clients/decky/ Steam Deck Decky plugin
packaging/windows/drivers/{pf-vdisplay,pf-dualsense,pf-xusb}/ in-house UMDF drivers (built from source in CI)
packaging/windows/drivers/pf-umdf-util/ audited unsafe layer (safe shm + sealed-channel + WDF request primitives) — gamepad drivers' logic is 100% safe over it
web/ TanStack web console over the mgmt API (status · devices · pairing · GPU selection · performance graphs)
packaging/ apt(deb) · RPM/COPR · Arch/sysext · Flatpak · Bazzite bootc · Windows host installer (per-dir READMEs)
tools/{loss-harness,latency-probe}/ measurement (plan §10)
scripts/ 60-punktfunk.rules · punktfunk-host.service · host.env.example · headless/
include/punktfunk_core.h generated C header
```
## Design invariants — do not regress
- **One core, linked everywhere.** Protocol/FEC/crypto live only in `punktfunk-core`, behind a
stable, versioned C ABI. `tokio`/`quinn` exist only behind the `quic` feature (control
plane); **no async on the per-frame path** — native threads only.
- **Native client resolution, no scaling.** A session gets a virtual output at exactly the
client's WxH@Hz via the `VirtualDisplay` trait (`create(mode) → VirtualOutput { node_id,
remote_fd, preferred_mode, keepalive }`, RAII teardown). There is no cross-compositor
protocol for this — each compositor keeps its own backend.
- **FEC is the wall-breaker.** GF(2⁸) (≤255 shards/block, Moonlight-compatible) and GF(2¹⁶)
Leopard (≤65535 shards/block) — punktfunk/1 negotiates the latter, removing the ~1 Gbps
ceiling.
- **Core security hardening stays intact**: reassembler bounds attacker-controlled fields
before allocating (`ReassemblerLimits`); AES-GCM per-direction nonce salts + seq-as-AAD;
ABI `struct_size` checks. Regression tests exist — keep them green.
- **PipeWire consumer discipline**: our capture streams set `node.dont-reconnect` and tear
down promptly on negotiation timeout — one wedged link head-blocks the daemon's shared
work queue system-wide.
## Running on this box
Headless QEMU VM (Ubuntu 26.04, kernel 7.0), passthrough RTX 5070 Ti (driver 595 **open**
module — a kernel update silently drops it; reinstall `nvidia-driver-595-open`), no KMS
scanout → KWin `--drm` impossible; everything renders offscreen via `renderD128`.
```sh
# compositor session (shell 1, or the systemd unit in scripts/): full headless Plasma.
# The script sets XDG_MENU_PREFIX=plasma- & co. — without it plasmashell runs but the
# launcher menu is EMPTY (no apps, no System Settings).
bash scripts/headless/run-headless-kde.sh 1920x1080
# host (shell 2): bare `serve` is native-only (secure default); add --gamestream for Moonlight compat.
WAYLAND_DISPLAY=wayland-kde XDG_CURRENT_DESKTOP=KDE PUNKTFUNK_VIDEO_SOURCE=virtual \
PUNKTFUNK_ZEROCOPY=1 cargo run -rp punktfunk-host -- serve --gamestream
# punktfunk/1 native loopback test (no Moonlight needed; same env as serve, listener persists
# across sessions — bound it with --max-sessions):
cargo run -rp punktfunk-host -- punktfunk1-host --source virtual --seconds 10 --max-sessions 1
cargo run -rp punktfunk-probe -- --mode 1280x720x120 --out /tmp/a.h265 --input-test # + --pin HEX
```
Pinned crate facts: `ashpd` 0.13 + `pipewire` 0.9 (must match ashpd's) + `ffmpeg-next` 8.x
(`ffmpeg-sys-next` auto-detects the system FFmpeg, so it builds against **FFmpeg 7.x/libavcodec 61
or 8.x/libavcodec 62** — validated live on Ubuntu 26.04 (8) and Bazzite F43 (7.1); the zero-copy
FFI also link-needs `libGL`/`libgbm`/`libcuda` at build time). Env knobs: `PUNKTFUNK_VIDEO_SOURCE=virtual|portal`,
`PUNKTFUNK_COMPOSITOR=kwin|gamescope|mutter`, `PUNKTFUNK_ZEROCOPY=1|0` (Linux default: ON for
VAAPI/AMD/Intel with a one-shot CPU downgrade if the dmabuf offer never negotiates, OFF/opt-in for
NVENC), `PUNKTFUNK_VAAPI_LOW_POWER=1|0` (pin the VAAPI entrypoint; auto = full-feature then VDEnc
fallback for modern Intel), `PUNKTFUNK_NV12=0` (opt OUT of the default GPU RGB→NV12 convert on the
NVIDIA tiled zero-copy path), `PUNKTFUNK_INTRA_REFRESH=1` (opt-in NVENC intra-refresh loss recovery),
`PUNKTFUNK_PIN_CLOCKS=1` (opt-in NVML GPU clock floor, root-gated), `PUNKTFUNK_GAMESCOPE_APP=...`,
`PUNKTFUNK_INPUT_BACKEND=...`, `PUNKTFUNK_PERF=1` (per-stage timing), `PUNKTFUNK_VIDEO_DROP=N` (FEC
test — injects N% wire-packet loss on BOTH the GameStream and native video paths, no netem needed), `PUNKTFUNK_FEC_PCT=N`, `PUNKTFUNK_DSCP=1` (opt-in DSCP/SO_PRIORITY media QoS on the data +
GameStream video/audio sockets; no-op on the wire on Windows without a qWAVE policy),
`PUNKTFUNK_444=1` (full-chroma HEVC 4:4:4, see below).
**HEVC 4:4:4 (full chroma, Range Extensions)**: opt-in via `PUNKTFUNK_444`, negotiated like 10-bit —
the host emits 4:4:4 only when the client advertised `VIDEO_CAP_444` (wire bit `0x04` + ABI
`PUNKTFUNK_VIDEO_CAP_444`), the codec is HEVC, the session is single-process, **and** a GPU probe
(`encode::can_encode_444`, run before the Welcome) confirms support — else it resolves to 4:2:0 and
`Welcome::chroma_format` reflects the real value (honest downgrade; the client reads it via
`punktfunk_connection_chroma_format`). **punktfunk/1-native only** — GameStream/Moonlight stays 4:2:0
(stock clients can't decode 4:4:4). **NVENC is the implemented path**: Linux `hevc_nvenc` feeds a
swscale'd `yuv444p` (RGB-in is always 4:2:0 — verified on the RTX 5070 Ti — so the session forces CPU
RGB capture for 4:4:4); Windows NVENC keeps ARGB input + FREXT profile + `chromaFormatIDC=3` and the
DDA capturer delivers RGB. VAAPI / AMF / QSV **decline** (probe returns false — no validated 4:4:4
hardware in the lab; they'd produce 4:2:0). Software (openh264) is 4:2:0-only. Test with
`PUNKTFUNK_CLIENT_444=1 punktfunk-probe --out x.h265` then `ffprobe x.h265` (expect `pix_fmt yuv444p`).
*Linux NVENC mechanism validated on the RTX 5070 Ti (ffmpeg CLI); Windows NVENC + 10-bit-4:4:4 not yet
on-glass validated.*
## Conventions
- Rust 2021, `rustfmt` + `clippy -D warnings` clean before commit.
- Match the surrounding code's comment density and naming.
- Commit messages end with the Co-Authored-By trailer (see `git log`).
- `pkill` caution on this box: match exact comm names (`pkill -x gamescope-wl`,
`pkill -x punktfunk-host`) — `pkill -f` self-matches the invoking shell.
Generated
+10 -9
View File
@@ -2119,7 +2119,7 @@ dependencies = [
[[package]]
name = "latency-probe"
version = "0.6.0"
version = "0.7.1"
[[package]]
name = "lazy_static"
@@ -2251,7 +2251,7 @@ checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "loss-harness"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"punktfunk-core",
]
@@ -2875,7 +2875,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-android"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"android_logger",
"jni",
@@ -2889,7 +2889,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-linux"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"anyhow",
"async-channel",
@@ -2911,7 +2911,7 @@ dependencies = [
[[package]]
name = "punktfunk-client-windows"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"anyhow",
"async-channel",
@@ -2934,7 +2934,7 @@ dependencies = [
[[package]]
name = "punktfunk-core"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"aes-gcm",
"bytes",
@@ -2964,7 +2964,7 @@ dependencies = [
[[package]]
name = "punktfunk-host"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"aes",
"aes-gcm",
@@ -3027,13 +3027,14 @@ dependencies = [
"windows 0.62.2 (registry+https://github.com/rust-lang/crates.io-index)",
"windows-service",
"winreg",
"winresource",
"x509-parser",
"xkbcommon",
]
[[package]]
name = "punktfunk-probe"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"anyhow",
"mdns-sd",
@@ -3047,7 +3048,7 @@ dependencies = [
[[package]]
name = "punktfunk-tray"
version = "0.6.0"
version = "0.7.1"
dependencies = [
"anyhow",
"ksni",
+1 -1
View File
@@ -17,7 +17,7 @@ members = [
exclude = ["packaging/linux/steam-deck-gadget/usbip-poc"]
[workspace.package]
version = "0.6.0"
version = "0.7.1"
edition = "2021"
rust-version = "1.82"
license = "MIT OR Apache-2.0"
+4 -2
View File
@@ -15,6 +15,9 @@ your local network.
💬 **Community: [Discord](https://discord.gg/kaPNvzMuGU)** — chat, support, and **Android beta
access** · **[r/Punktfunk](https://www.reddit.com/r/Punktfunk/)**.
🔒 **Security:** found a vulnerability? Report it privately to **security@punktfunk.com** — see
[SECURITY.md](SECURITY.md). Please don't open a public issue.
punktfunk pairs a **virtual-display streaming host** with native clients on every platform. It speaks
the existing **GameStream** protocol, so any [Moonlight](https://moonlight-stream.org/) client works
day one — and adds its own faster **`punktfunk/1`** protocol that breaks the ~1 Gbps FEC wall with a
@@ -61,7 +64,7 @@ The **GameStream host works with a stock Moonlight client** — validated live o
and **video at the client's exact resolution and refresh** via a per-session virtual output (KWin,
gamescope, Mutter, and Sway/wlroots backends), encoded with GPU **zero-copy** (dmabuf → CUDA/Vulkan →
NVENC) up to 5120×1440@240. The native **`punktfunk/1`** protocol adds a QUIC control plane and a
GF(2¹⁶) Leopard-FEC + AES-GCM data plane (p50 ~0.8 ms capture→reassembled at 720p120), with
GF(2¹⁶) Leopard-FEC + AES-GCM data plane (p50 ~0.8 ms capture→received at 720p120), with
mid-stream mode renegotiation and a wall-clock skew handshake so latency stays valid across machines.
Both run from **one process**: bare `punktfunk-host serve` is the **secure native-only default**
(`punktfunk/1` + the management API/web console), and `serve --gamestream` additionally enables the
@@ -138,7 +141,6 @@ clients/
web/ web console (TanStack) over the management API — status · devices · pairing · GPUs · performance · logs
packaging/ apt · rpm / COPR · Arch · Flatpak · Bazzite bootc image
docs-site/ public documentation site (Fumadocs) — https://docs.punktfunk.unom.io
design/ design notes & deep-dive plans (index: design/README.md)
include/punktfunk_core.h cbindgen-generated C header (checked in)
tools/ latency-probe · loss-harness (measurement)
```
+69
View File
@@ -0,0 +1,69 @@
# Security Policy
punktfunk is a low-latency desktop/game streaming stack. A host is effectively remote control of a
machine, so we take security reports seriously and appreciate responsible disclosure.
## Reporting a vulnerability
**Please report security issues privately by email to security@punktfunk.com.**
Do **not** open a public issue, pull request, or chat/forum post for a suspected vulnerability — that
exposes other users before a fix exists.
### What to include
The more of this you can give us, the faster we can act:
- The component and version (e.g. `punktfunk-host 0.6.0`, Windows or Linux, which client).
- The impact — what an attacker can do, and from what position (same LAN, a local service account,
admin, a paired client, …).
- Steps to reproduce, a proof-of-concept, or a crash/log if you have one.
- Any suggested fix or mitigation (optional).
## What to expect
We're a small team, so timelines are best-effort, but we commit to:
- **Acknowledge** your report within **3 business days**.
- Give an **initial assessment** (severity + whether we can reproduce) within about **7 days**.
- Keep you updated, and tell you when a fix ships.
- **Credit** you in the advisory / release notes when the fix is public — unless you'd rather stay
anonymous.
We practice **coordinated disclosure**: please give us reasonable time to release a fix before
publishing details. We aim to resolve valid issues within **90 days** and will agree a disclosure
date with you.
## Scope
In scope — the code in this repository:
- The host (`punktfunk-host`), its Windows drivers, and the protocol/crypto core (`punktfunk-core`).
- The native clients (Apple, Linux, Windows, Android), the web management console, and the management
API.
Known limits — documented behavior, not vulnerabilities (see
https://docs.punktfunk.unom.io/docs/security):
- **Admin/SYSTEM already on the host = out of scope.** An attacker who is already administrator or
SYSTEM on the host owns the machine regardless of punktfunk.
- **The virtual display is a real monitor** — any process already in the interactive desktop session
can capture it via the normal OS screen-capture APIs, exactly as it could a physical monitor.
- **GameStream/Moonlight compatibility** (`--gamestream`) uses legacy encryption and is documented as
opt-in, trusted-LAN-only.
- **Public-internet exposure is unsupported** — issues that only arise from exposing the host to the
WAN are expected; keep the host on a trusted LAN or a VPN.
If you're unsure whether something is in scope, report it anyway — we'd rather hear about it.
## Safe harbor
We consider good-faith security research that follows this policy to be authorized, and we won't
pursue legal action against researchers who:
- make a good-faith effort to avoid privacy violations, data loss, and service disruption,
- only test systems they own or have explicit permission to test,
- give us reasonable time to remediate before public disclosure,
- don't exfiltrate more data than needed to demonstrate the issue.
Thank you for helping keep punktfunk and its users safe.
@@ -27,8 +27,15 @@
<uses-feature android:name="android.software.leanback" android:required="false" />
<uses-feature android:name="android.hardware.gamepad" android:required="false" />
<!-- appCategory="game": a game-streaming client IS a game as far as the SoC is concerned.
On Snapdragon devices (and other OEMs with a Game Mode / Game Dashboard) this makes the app
eligible for the vendor's game performance profile — the aggressive CPU/GPU governor and
scheduler treatment games get — which, together with the ADPF hints in the native decode
path, is what keeps clocks up for low, consistent decode latency. Also groups it correctly
under Games in battery/data usage. Advisory: devices without Game Mode ignore it. -->
<application
android:allowBackup="false"
android:appCategory="game"
android:icon="@mipmap/ic_launcher"
android:roundIcon="@mipmap/ic_launcher_round"
android:label="@string/app_name"
@@ -15,11 +15,16 @@ import io.unom.punktfunk.kit.NativeBridge
import kotlin.math.roundToInt
/**
* The live stats overlay — mirrors the Apple client's HUD. Reads the 14-double layout from
* The live stats overlay — the unified HUD (`design/stats-unification.md`, Android v1: headline is
* `capture→decoded`, tiled by `host+network` + `decode`). Reads the 18-double layout from
* [NativeBridge.nativeVideoStats]:
* `[fps, mbps, latP50Ms, latP95Ms, latValid, skew, w, h, hz, dropped, bitDepth, colorPrimaries,
* colorTransfer, chromaFormatIdc]`. The trailing four (present on a current native lib) describe the
* negotiated video feed and render as a codec/depth/colour/chroma line; older layouts just omit it.
* `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skew, w, h, hz, lost, bitDepth, colorPrimaries,
* colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms, netP50Ms]`. Indexes 1013
* (present on a current native lib) describe the negotiated video feed and render as a
* codec/depth/colour/chroma line; 14/15 render as the stage equation — split into
* `host + network + decode` when the Phase-2 terms at 16/17 are nonzero (a current host sends
* per-AU 0xCF timings; an old host leaves them 0 and the combined `host+network` term stands);
* older layouts just omit those lines.
*/
@Composable
internal fun StatsOverlay(s: DoubleArray, modifier: Modifier = Modifier) {
@@ -29,7 +34,7 @@ internal fun StatsOverlay(s: DoubleArray, modifier: Modifier = Modifier) {
val hz = s[8].toInt()
val latValid = s[4] != 0.0
val skew = s[5] != 0.0
val dropped = s[9].toLong()
val lost = s[9].toLong()
Column(
modifier = modifier
.background(Color.Black.copy(alpha = 0.45f), RoundedCornerShape(6.dp))
@@ -50,17 +55,33 @@ internal fun StatsOverlay(s: DoubleArray, modifier: Modifier = Modifier) {
)
}
if (latValid) {
val tag = if (skew) "" else " (same-host)"
val tag = if (skew) "" else " (same-host clock)"
Text(
"capture→client ${"%.1f".format(s[2])}/${"%.1f".format(s[3])} ms p50/p95$tag",
"end-to-end ${"%.1f".format(s[2])} ms p50 · ${"%.1f".format(s[3])} p95 · capture→decoded$tag",
color = Color.White,
fontFamily = FontFamily.Monospace,
fontSize = 12.sp,
)
if (s.size >= 16) {
// Phase-2 split (s[16]/s[17]): render `host + network` separately when the host
// reported its share this window; otherwise the combined term (old host / no
// matched 0xCF timing).
val equation = if (s.size >= 18 && s[16] > 0) {
"= host ${"%.1f".format(s[16])} + network ${"%.1f".format(s[17])} + decode ${"%.1f".format(s[15])}"
} else {
"= host+network ${"%.1f".format(s[14])} + decode ${"%.1f".format(s[15])}"
}
Text(
equation,
color = Color.White,
fontFamily = FontFamily.Monospace,
fontSize = 12.sp,
)
}
}
if (dropped > 0) {
if (lost > 0) {
Text(
"dropped $dropped",
"lost $lost",
color = Color(0xFFFFB0B0),
fontFamily = FontFamily.Monospace,
fontSize = 12.sp,
@@ -105,12 +105,17 @@ object NativeBridge {
/**
* Drain ~1 s of live decode stats for the on-stream HUD, or `null` when no decode thread runs.
* Returns 14 doubles:
* `[fps, mbps, latP50Ms, latP95Ms, latValid, skewCorrected, width, height, refreshHz, framesDropped,
* bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc]`
* (the two flags are 1.0/0.0; the trailing four describe the negotiated video feed — bit depth
* 8/10, CICP primaries/transfer, and the HEVC chroma_format_idc 1=4:2:0 / 3=4:4:4). Poll ~1 Hz;
* each call resets the measurement window.
* Returns 18 doubles (unified stats spec, `design/stats-unification.md`):
* `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skewCorrected, width, height, refreshHz, framesLost,
* bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms,
* netP50Ms]`
* (the two flags are 1.0/0.0; indexes 2/3 are the end-to-end capture→decoded headline; 1013
* describe the negotiated video feed — bit depth 8/10, CICP primaries/transfer, and the HEVC
* chroma_format_idc 1=4:2:0 / 3=4:4:4; 14/15 are the stage p50s tiling the headline —
* `host+network` = capture→received, `decode` = received→decoded; 16/17 split the
* `host+network` term via the host's per-AU 0xCF timings — `host` = the host's capture→sent,
* `network` = the remainder — both 0.0 when no timing matched this window, i.e. an old host).
* Poll ~1 Hz; each call resets the measurement window.
*/
external fun nativeVideoStats(handle: Long): DoubleArray?
+137
View File
@@ -0,0 +1,137 @@
//! Android Adaptive Performance Framework (ADPF) — CPU performance hints for the decode thread.
//!
//! ADPF lets a latency-critical app tell the platform "these threads run a repeating workload with
//! this per-cycle deadline, and here's how long they *actually* took." The kernel's CPU governor
//! (on Qualcomm Snapdragon in particular — its ADPF backend is among the most responsive) then keeps
//! those threads on the fast cores at high clocks instead of migrating them to a little core or
//! down-clocking between frames. For a stream client the win is on the in-process hot path we
//! control — the `pf-decode` feed/drain/present loop — *not* the hardware codec itself (that decodes
//! in the mediacodec service, a separate process we can't hint); keeping our loop from being
//! scheduled late directly trims the jitter between "AU received" and "buffer released to the
//! Surface." It complements the codec-side `operating-rate`/`priority` hints, which push the codec's
//! own clocks.
//!
//! The `APerformanceHint_*` API arrived in NDK **API level 33**. minSdk is 31, so we CANNOT link the
//! symbols directly: a `libpunktfunk_android.so` carrying an unresolved
//! `APerformanceHint_createSession` import fails to load on API 31/32 devices
//! (`System.loadLibrary` throws) even if the code path is never taken. Instead we resolve the
//! entry points from `libandroid.so` with `dlsym` at runtime — absent on < 33 ⇒
//! [`HintSession::create`] returns `None` and the decode loop simply runs without hints.
use std::ffi::c_void;
use std::os::raw::c_int;
// `APerformanceHint_*` function-pointer types. The manager/session handles are opaque, so we treat
// them as `*mut c_void`.
type GetManagerFn = unsafe extern "C" fn() -> *mut c_void;
type CreateSessionFn = unsafe extern "C" fn(*mut c_void, *const i32, usize, i64) -> *mut c_void;
type ReportFn = unsafe extern "C" fn(*mut c_void, i64) -> c_int;
type UpdateTargetFn = unsafe extern "C" fn(*mut c_void, i64) -> c_int;
type CloseFn = unsafe extern "C" fn(*mut c_void);
/// The entry points we use, resolved once from `libandroid.so`, plus the process-wide manager.
struct Api {
create_session: CreateSessionFn,
report: ReportFn,
update_target: UpdateTargetFn,
close: CloseFn,
manager: *mut c_void,
}
/// Resolve the ADPF entry points + the process manager, or `None` on API < 33 (symbols absent) or if
/// the manager is unavailable.
fn resolve_api() -> Option<Api> {
// SAFETY: `dlopen` of an always-present system library with a NUL-terminated name; it returns
// null on failure (checked below). `libandroid.so` is already mapped into every app process, so
// this only bumps its refcount — we intentionally never `dlclose` (process-lifetime handle).
let lib = unsafe { libc::dlopen(c"libandroid.so".as_ptr(), libc::RTLD_NOW) };
if lib.is_null() {
return None;
}
// SAFETY: `dlsym` on the valid handle above with NUL-terminated symbol names; each returns null
// when the symbol is absent (device API < 33), which we check before transmuting the non-null
// pointer to its fn-pointer type (layout-compatible; a resolved symbol is a valid code address).
unsafe {
let get_manager = libc::dlsym(lib, c"APerformanceHint_getManager".as_ptr());
let create_session = libc::dlsym(lib, c"APerformanceHint_createSession".as_ptr());
let report = libc::dlsym(lib, c"APerformanceHint_reportActualWorkDuration".as_ptr());
let update_target = libc::dlsym(lib, c"APerformanceHint_updateTargetWorkDuration".as_ptr());
let close = libc::dlsym(lib, c"APerformanceHint_closeSession".as_ptr());
if get_manager.is_null()
|| create_session.is_null()
|| report.is_null()
|| update_target.is_null()
|| close.is_null()
{
return None; // device API < 33 — no ADPF
}
let get_manager = std::mem::transmute::<*mut c_void, GetManagerFn>(get_manager);
let manager = get_manager();
if manager.is_null() {
return None;
}
Some(Api {
create_session: std::mem::transmute::<*mut c_void, CreateSessionFn>(create_session),
report: std::mem::transmute::<*mut c_void, ReportFn>(report),
update_target: std::mem::transmute::<*mut c_void, UpdateTargetFn>(update_target),
close: std::mem::transmute::<*mut c_void, CloseFn>(close),
manager,
})
}
}
/// A live ADPF hint session bound to a set of thread ids. Dropping it closes the session. Holds raw
/// handles, so it is `!Send`/`!Sync` — created and used only on the `pf-decode` thread.
pub struct HintSession {
api: Api,
session: *mut c_void,
}
impl HintSession {
/// Open a session hinting `tids` with an initial per-frame target of `target_ns` nanoseconds.
/// `None` when ADPF is unavailable (device API < 33) or the platform declines — the caller then
/// runs unhinted (a no-op, not an error).
pub fn create(target_ns: i64, tids: &[i32]) -> Option<Self> {
if target_ns <= 0 || tids.is_empty() {
return None;
}
let api = resolve_api()?;
// SAFETY: `api.manager` is the live process manager returned above; `tids` is a valid slice
// of `len` i32s that `createSession` copies; it returns null on failure (checked).
let session =
unsafe { (api.create_session)(api.manager, tids.as_ptr(), tids.len(), target_ns) };
if session.is_null() {
return None;
}
Some(Self { api, session })
}
/// Report the wall-clock time the hinted thread spent producing the last displayed frame. When
/// it exceeds the session target the governor boosts the cores running the thread; when it
/// stays under, clocks may relax. No-op on a non-positive duration (the API rejects it).
pub fn report_actual(&self, actual_ns: i64) {
if actual_ns <= 0 {
return;
}
// SAFETY: `self.session` is a live session for `self`'s lifetime.
unsafe { (self.api.report)(self.session, actual_ns) };
}
/// Update the per-frame target (e.g. after a mid-session refresh-rate change). Unused today —
/// the decode thread restarts on renegotiation — but kept for that path.
#[allow(dead_code)]
pub fn update_target(&self, target_ns: i64) {
if target_ns <= 0 {
return;
}
// SAFETY: `self.session` is a live session for `self`'s lifetime.
unsafe { (self.api.update_target)(self.session, target_ns) };
}
}
impl Drop for HintSession {
fn drop(&mut self) {
// SAFETY: `self.session` was created by `createSession` and is closed exactly once, here.
unsafe { (self.api.close)(self.session) };
}
}
+4
View File
@@ -324,6 +324,10 @@ fn decode_loop(
counters: Arc<Counters>,
channels: usize,
) {
// Fold this Opus→AAudio thread into the client's hot-thread set so the ADPF session the decode
// thread opens also keeps audio decode on a fast core (registered before the video pump's first
// frame arrives, so it's captured when that session is created). No-op below API 33.
client.register_hot_thread();
// Interleaved f32 samples per millisecond at this layout — the ring's 5 ms reserve check below.
let ms = (SAMPLE_RATE as usize / 1000) * channels;
// Opus decode scratch: worst-case 120 ms frame (5760 samples/ch) × channels.
+168 -9
View File
@@ -9,16 +9,27 @@
use ndk::data_space::DataSpace;
use ndk::media::media_codec::{
DequeuedInputBufferResult, DequeuedOutputBufferInfoResult, MediaCodec, MediaCodecDirection,
OutputBuffer,
};
use ndk::media::media_format::MediaFormat;
use ndk::native_window::{FrameRateCompatibility, NativeWindow};
use punktfunk_core::client::NativeClient;
use punktfunk_core::error::PunktfunkError;
use punktfunk_core::session::Frame;
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// Cap on the pts→received-timestamp map below: MediaCodec holds only a handful of frames in
/// flight, so anything beyond this is stale (codec flushed / HUD toggled) and gets evicted.
const IN_FLIGHT_CAP: usize = 64;
/// Cap on received AUs awaiting their 0xCF host timing (Phase 2 host/network split): the timing
/// datagram trails its AU by at most the wire, so a match lands within a frame or two — anything
/// this deep is a lost datagram (or an old host that never sends any) and gets evicted.
const PENDING_SPLIT_CAP: usize = 256;
/// The decode loop. Runs on the `pf-decode` thread until `shutdown` is set or the session closes.
pub fn run(
client: Arc<NativeClient>,
@@ -61,7 +72,14 @@ pub fn run(
// realtime priority + the target frame rate, so vendor decoders (e.g. Qualcomm) run at full
// clocks instead of a power-saving cadence that adds dequeue latency.
format.set_i32("priority", 0); // 0 = realtime
format.set_i32("operating-rate", mode.refresh_hz as i32);
// Operating rate = the codec's clock hint. Setting it to the display rate merely asks the
// decoder to *sustain* that cadence — a Qualcomm decoder can meet 60/120 fps at a power-saving
// clock that adds a millisecond-plus of decode latency per frame. Setting it to the AOSP
// "unbounded" sentinel (Short.MAX) instead asks the decoder to run each frame at max clocks and
// finish ASAP, minimising per-frame decode latency — the right trade for a real-time stream
// (costs power/heat; the dial to lower if a device thermally throttles over a long session).
// Ignored where unsupported.
format.set_i32("operating-rate", i16::MAX as i32); // 32767 = "as fast as possible"
// HDR static metadata (ST.2086 mastering + content light level): when an HDR session was
// negotiated, set KEY_HDR_STATIC_INFO so the display tone-maps from the source's real grade.
@@ -104,6 +122,25 @@ pub fn run(
);
}
// ADPF: hint the platform that the whole video pipeline — this pf-decode feed/drain/present
// loop, the core's data-plane pump (UDP receive + FEC reassembly), and the audio thread — runs a
// per-frame real-time workload, so the CPU governor keeps those threads on fast cores at high
// clocks instead of down-clocking between frames or parking them on a little core. Snapdragon's
// ADPF backend responds well to this. We register this thread now but create the session lazily
// on the first presented frame: by then the pump + audio threads have registered their ids too,
// and ADPF `createSession` rejects a set with any not-yet-live/dead tid. No-op below API 33.
let frame_period_ns = if mode.refresh_hz > 0 {
1_000_000_000i64 / mode.refresh_hz as i64
} else {
0
};
client.register_hot_thread(); // this decode thread → the pipeline's hot-thread set
let mut hint: Option<crate::adpf::HintSession> = None;
let mut hint_tried = false;
// Accumulates the loop's productive (feed+drain) time between displayed frames; reported to ADPF
// once per rendered frame against the frame-period target.
let mut work_accum_ns: i64 = 0;
let mut fed: u64 = 0;
let mut rendered: u64 = 0;
let mut discarded: u64 = 0;
@@ -115,9 +152,19 @@ pub fn run(
// climbs.
let mut last_dropped = client.frames_dropped();
let mut last_kf_req: Option<Instant> = None;
// Capture→client-receipt latency uses the negotiated host-minus-client clock offset (0 if the
// host didn't answer the skew handshake — then the HUD flags it "same-host").
// Skew-corrected latency stats (spec: design/stats-unification.md) use the negotiated
// host-minus-client clock offset (0 if the host didn't answer the skew handshake — then the
// HUD flags it "(same-host clock)").
let clock_offset = client.clock_offset_ns;
// HUD stage split: receipt timestamps keyed by the pts we queue into the codec, so the decoded
// point (output-buffer dequeue — MediaCodec round-trips presentationTimeUs) can be paired back
// to its receipt for the `decode` stage. Only fed while the HUD is visible.
let mut in_flight: VecDeque<(u64, i128)> = VecDeque::new();
// Phase-2 host/network split (design/stats-unification.md): received AUs awaiting their 0xCF
// host timing, as (pts_ns, capture→received µs). The timings are drained non-blockingly right
// where receipts are recorded and matched by pts; `network = hostnet host` (saturating).
// Only fed while the HUD is visible; an old host never sends a 0xCF, so entries just age out.
let mut pending_split: VecDeque<(u64, u64)> = VecDeque::new();
// The dataspace we've signalled on the Surface so far (None = default/SDR). Set reactively once
// the decoder reports an HDR stream (see `drain`); avoids re-applying every format event.
let mut applied_ds: Option<DataSpace> = None;
@@ -138,15 +185,41 @@ pub fn run(
&p[..p.len().min(6)]
);
}
// HUD stat: capture→client-receipt latency = client_now + (hostclient)
// HUD stat, `received` point: host+network = client_now + (hostclient)
// capture_pts. Gated on the HUD being visible — `enabled` first so the hidden
// steady state skips the wall-clock read and the lock entirely.
// steady state skips the wall-clock read and the lock entirely. The receipt
// stamp is also parked in `in_flight` (keyed by the pts the codec will echo on
// the output buffer) for the decoded-point pairing in `drain`.
if stats.enabled() {
let lat_ns =
now_realtime_ns() + clock_offset as i128 - frame.pts_ns as i128;
let received_ns = now_realtime_ns();
let lat_ns = received_ns + clock_offset as i128 - frame.pts_ns as i128;
let lat_us = (lat_ns > 0 && lat_ns < 10_000_000_000)
.then_some((lat_ns / 1000) as u64);
stats.note(frame.data.len(), lat_us, clock_offset != 0);
stats.note_received(frame.data.len(), lat_us, clock_offset != 0);
in_flight.push_back((frame.pts_ns / 1000, received_ns));
if in_flight.len() > IN_FLIGHT_CAP {
in_flight.pop_front(); // stale — codec never echoed it back
}
// Phase-2 split: park this AU's capture→received sample, then match any
// 0xCF host timings that have arrived — host = the host's own
// capture→sent, network = our capture→received minus it (per-frame
// tiling; saturating in case of clock jitter).
if let Some(hostnet_us) = lat_us {
pending_split.push_back((frame.pts_ns, hostnet_us));
if pending_split.len() > PENDING_SPLIT_CAP {
pending_split.pop_front(); // 0xCF lost / old host — evict
}
}
while let Ok(t) = client.next_host_timing(Duration::ZERO) {
if let Some(i) = pending_split.iter().position(|&(p, _)| p == t.pts_ns)
{
let (_, hostnet_us) = pending_split.remove(i).unwrap();
stats.note_host_split(
t.host_us as u64,
hostnet_us.saturating_sub(t.host_us as u64),
);
}
}
}
pending = Some(frame);
}
@@ -154,6 +227,9 @@ pub fn run(
Err(_) => break, // session closed
}
}
// Time the productive work (feed + drain) only — the `next_frame` poll wait above is idle
// and excluded, so ADPF sees this thread's real per-frame CPU cost, not the poll timeout.
let work_t0 = Instant::now();
if let Some(frame) = pending.take() {
if feed(&codec, &frame.data, frame.pts_ns / 1000) {
fed += 1;
@@ -173,10 +249,48 @@ pub fn run(
} else {
Duration::ZERO
};
let (r, d) = drain(&codec, &window, &mut applied_ds, wait);
let (r, d) = drain(
&codec,
&window,
&mut applied_ds,
wait,
&stats,
&mut in_flight,
clock_offset,
);
rendered += r;
discarded += d;
// ADPF: attribute this iteration's feed+drain time to the frame being produced, and report
// the accumulated per-frame work once one is actually presented (r > 0). Under back-pressure
// the short output-dequeue wait is included in the tally — for a latency-first client,
// biasing the governor toward "boost" is the desired behaviour. Cheap when `hint` is None
// (one `Instant` diff, no report).
work_accum_ns += work_t0.elapsed().as_nanos() as i64;
if r > 0 {
if !hint_tried {
// First presented frame: the pump + audio threads have registered their ids by now.
// Build one ADPF session over the whole pipeline's thread set (empty below API 33,
// or where the platform declines → `None`, and the loop runs unhinted).
hint_tried = true;
let tids = client.hot_thread_ids();
hint = crate::adpf::HintSession::create(frame_period_ns, &tids);
log::info!(
"decode: ADPF hint session {} — {} hot thread(s), target {frame_period_ns} ns",
if hint.is_some() {
"active"
} else {
"unavailable"
},
tids.len(),
);
}
if let Some(h) = &hint {
h.report_actual(work_accum_ns);
}
work_accum_ns = 0;
}
// Loss recovery: under infinite GOP the only recovery keyframe is one we request. The
// reassembler drops unrecoverable AUs (frames_dropped); the decoder then conceals the
// reference-missing delta frames that follow and renders them without error, so keying off
@@ -271,11 +385,19 @@ fn feed(codec: &MediaCodec, au: &[u8], pts_us: u64) -> bool {
/// the caller's input is blocked so the loop waits on decoder progress instead of busy-spinning.
/// Returns `(rendered, discarded)`. Also reacts to `OutputFormatChanged` (which can interleave
/// between buffers — handled without losing the held buffer) to signal HDR on the Surface.
///
/// Each dequeued buffer is also the HUD's `decoded` measurement point (rendered or not — the frame
/// finished decoding either way): end-to-end = decoded + clock_offset capture pts, and the
/// `decode` stage pairs the buffer's echoed presentationTimeUs back to the receipt stamp in
/// `in_flight` (single-clock local difference, no skew involved).
fn drain(
codec: &MediaCodec,
window: &NativeWindow,
applied_ds: &mut Option<DataSpace>,
first_wait: Duration,
stats: &crate::stats::VideoStats,
in_flight: &mut VecDeque<(u64, i128)>,
clock_offset: i64,
) -> (u64, u64) {
let mut held = None; // newest ready buffer so far, presented after the loop
let mut discarded: u64 = 0;
@@ -284,6 +406,9 @@ fn drain(
match codec.dequeue_output_buffer(wait) {
Ok(DequeuedOutputBufferInfoResult::Buffer(buf)) => {
wait = Duration::ZERO; // only the first dequeue may block
if stats.enabled() {
note_decoded(stats, in_flight, clock_offset, &buf);
}
if let Some(stale) = held.replace(buf) {
// A newer frame is ready — drop the held one without rendering.
if let Err(e) = codec.release_output_buffer(stale, false) {
@@ -333,6 +458,40 @@ fn drain(
(rendered, discarded)
}
/// HUD `decoded` point for one dequeued output buffer: build the end-to-end (capture→decoded,
/// skew-corrected, clamped to (0, 10 s)) and `decode` (received→decoded, single-clock local, ≥ 0)
/// samples and hand them to [`crate::stats::VideoStats::note_decoded`]. The codec echoes the input
/// `presentationTimeUs` on the output buffer, which keys the receipt stamp in `in_flight`; entries
/// older than the echoed pts are evicted (decode order == input order here — low-latency, no
/// B-frames — so anything before it was dropped inside the codec or stamped before a flush).
fn note_decoded(
stats: &crate::stats::VideoStats,
in_flight: &mut VecDeque<(u64, i128)>,
clock_offset: i64,
buf: &OutputBuffer<'_>,
) {
let pts_us = buf.info().presentation_time_us().max(0) as u64;
let decoded_ns = now_realtime_ns();
// Pair the echoed pts back to its receipt stamp, evicting stale (older) entries as we go.
let mut received_ns = None;
while let Some(&(p, r)) = in_flight.front() {
if p > pts_us {
break; // future frame — leave it for its own output buffer
}
in_flight.pop_front();
if p == pts_us {
received_ns = Some(r);
break;
}
}
// pts_us is the truncated frame.pts_ns/1000 we queued, so ×1000 re-approximates capture time
// to < 1 µs — negligible against the ms-scale figures shown.
let e2e_ns = decoded_ns + clock_offset as i128 - pts_us as i128 * 1000;
let e2e_us = (e2e_ns > 0 && e2e_ns < 10_000_000_000).then_some((e2e_ns / 1000) as u64);
let decode_us = received_ns.map(|r| ((decoded_ns - r).max(0) / 1000) as u64);
stats.note_decoded(e2e_us, decode_us);
}
/// Map the decoder's reported output colour to a BT.2020 HDR dataspace, or `None` for SDR. The
/// integer values are the Android MediaFormat colour constants the NDK shares: COLOR_TRANSFER
/// ST2084 = 6 (PQ/HDR10), HLG = 7; COLOR_RANGE FULL = 1, LIMITED = 2 (the host encodes limited).
+2
View File
@@ -25,6 +25,8 @@ use jni::objects::JObject;
use jni::sys::jint;
use jni::JNIEnv;
#[cfg(target_os = "android")]
mod adpf;
#[cfg(target_os = "android")]
mod audio;
#[cfg(target_os = "android")]
+23 -11
View File
@@ -72,14 +72,19 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopVideo(
})
}
/// `NativeBridge.nativeVideoStats(handle): DoubleArray?` — drain ~1 s of decode stats for the HUD.
/// Returns 14 doubles
/// `[fps, mbps, latP50Ms, latP95Ms, latValid, skewCorrected, width, height, refreshHz, framesDropped,
/// bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc]`
/// (the two flags are 1.0/0.0; the trailing four describe the negotiated video feed — see below), or
/// `null` when no decode thread is running. Poll ~1 Hz from the UI; each call resets the measurement
/// window. Not android-gated — pure `jni` + connector reads, so it links on the host build too
/// (Kotlin only ever calls it on device).
/// `NativeBridge.nativeVideoStats(handle): DoubleArray?` — drain ~1 s of decode stats for the HUD
/// (unified stats spec, `design/stats-unification.md`). Returns 18 doubles
/// `[fps, mbps, e2eP50Ms, e2eP95Ms, latValid, skewCorrected, width, height, refreshHz, framesLost,
/// bitDepth, colorPrimaries, colorTransfer, chromaFormatIdc, hostNetP50Ms, decodeP50Ms, hostP50Ms,
/// netP50Ms]`
/// (the two flags are 1.0/0.0; indexes 015 match the previous 16-double layout — 013 the original
/// 14-double one with the latency pair re-based to the end-to-end capture→decoded headline, 14/15
/// the stage p50s tiling it: `host+network` = capture→received, `decode` = received→decoded; 16/17
/// are the Phase-2 split of the `host+network` term from the per-AU 0xCF host timings — `host` =
/// the host's capture→sent, `network` = the remainder — both 0.0 when no timing matched this
/// window, i.e. an old host), or `null` when no decode thread is running. Poll ~1 Hz from the UI; each call
/// resets the measurement window. Not android-gated — pure `jni` + connector reads, so it links on
/// the host build too (Kotlin only ever calls it on device).
#[no_mangle]
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats(
env: JNIEnv,
@@ -98,11 +103,11 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats(
let snap = h.stats.drain();
let mode = h.client.mode();
let color = h.client.color;
let buf: [f64; 14] = [
let buf: [f64; 18] = [
snap.fps,
snap.mbps,
snap.lat_p50_ms,
snap.lat_p95_ms,
snap.e2e_p50_ms,
snap.e2e_p95_ms,
if snap.lat_valid { 1.0 } else { 0.0 },
if snap.skew_corrected { 1.0 } else { 0.0 },
mode.width as f64,
@@ -117,6 +122,13 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeVideoStats(
color.primaries as f64,
color.transfer as f64,
h.client.chroma_format as f64,
// Stage p50s tiling the end-to-end headline (appended to keep 013 index-compatible).
snap.hostnet_p50_ms,
snap.decode_p50_ms,
// Phase-2 host/network split of the `host+network` stage (0xCF host timings): 0.0
// when no timing matched this window (old host) — the HUD keeps the combined term.
snap.host_p50_ms,
snap.net_p50_ms,
];
let arr = match env.new_double_array(buf.len() as jsize) {
Ok(a) => a,
+130 -40
View File
@@ -1,8 +1,13 @@
//! Live decode stats for the on-stream HUD (mirrors the Apple client's stats overlay): FPS,
//! receive throughput, and capture→client-receipt latency (p50/p95). The decode thread is the sole
//! writer (`note` per access unit); the JNI accessor `nativeVideoStats` drains a snapshot ~1 Hz and
//! resets the window. Sampling is gated on the HUD actually being visible (`set_enabled`, driven by
//! `nativeSetVideoStatsEnabled`) so the hidden steady state costs one relaxed atomic load per frame.
//! Live decode stats for the on-stream HUD, following the unified stats spec
//! (`design/stats-unification.md`): FPS, receive throughput, and the Android v1 stage split —
//! headline `end-to-end` = capture→decoded (p50/p95) tiled by `host+network` = capture→received
//! and `decode` = received→decoded (stage p50s). When the host emits per-AU 0xCF host timings, the
//! `host+network` term further splits into `host` + `network` (Phase 2, `note_host_split`); an old
//! host emits none and the combined term stands. The decode thread is the sole writer
//! (`note_received` per access unit at receipt, `note_decoded` per decoder output buffer); the JNI
//! accessor `nativeVideoStats` drains a snapshot ~1 Hz and resets the window. Sampling is gated on
//! the HUD actually being visible (`set_enabled`, driven by `nativeSetVideoStatsEnabled`) so the
//! hidden steady state costs one relaxed atomic load per frame.
//! Pure `std` so it compiles on the host build too (the decode thread is android-only, but
//! `SessionHandle` holds the shared handle unconditionally).
@@ -13,9 +18,9 @@ use std::time::Instant;
/// Rolling per-window accumulator. Rates are computed over the actual elapsed wall-time at drain
/// (robust to poll jitter), so a poll that lands at 0.9 s or 1.1 s still reports the right FPS.
pub struct VideoStats {
/// HUD gate: `note` runs on the per-frame decode path, so while the overlay is hidden it (and
/// the caller's latency computation — see `enabled`) early-outs on this flag alone. Off until
/// Kotlin shows the HUD.
/// HUD gate: the samplers run on the per-frame decode path, so while the overlay is hidden
/// they (and the caller's latency computation — see `enabled`) early-out on this flag alone.
/// Off until Kotlin shows the HUD.
enabled: AtomicBool,
inner: Mutex<Inner>,
}
@@ -24,23 +29,52 @@ struct Inner {
window_start: Instant,
frames: u64,
bytes: u64,
/// capture→client-receipt latency samples for this window, in microseconds.
lat_us: Vec<u64>,
/// `end-to-end` = capture→decoded latency samples for this window, in microseconds
/// (skew-corrected clock base).
e2e_us: Vec<u64>,
/// `host+network` stage = capture→received samples, in microseconds (skew-corrected).
hostnet_us: Vec<u64>,
/// Phase-2 split of `host+network` (design/stats-unification.md Phase 2), fed only when the
/// host emits per-AU 0xCF timings: `host` = the host's own capture→sent duration, µs.
host_us: Vec<u64>,
/// The matching `network` term, µs: capture→received minus the host's capture→sent
/// (wire + reassembly). Always pushed in lockstep with `host_us`.
net_us: Vec<u64>,
/// `decode` stage = received→decoded samples, in microseconds (client-local, single clock).
decode_us: Vec<u64>,
/// Whether the host answered the clock-skew handshake (latency is cross-machine valid).
skew_corrected: bool,
}
/// A drained, computed view of one window. `lat_valid` is false when no in-range latency sample
/// landed (then p50/p95 are 0 and the HUD hides the latency line, exactly like the Apple client).
/// A drained, computed view of one window. `lat_valid` is false when no in-range end-to-end sample
/// landed (then the latency figures are 0 and the HUD hides the latency lines, exactly like the
/// Apple client).
pub struct Snapshot {
pub fps: f64,
pub mbps: f64,
pub lat_p50_ms: f64,
pub lat_p95_ms: f64,
/// Headline `end-to-end` (capture→decoded) percentiles, ms.
pub e2e_p50_ms: f64,
pub e2e_p95_ms: f64,
/// Stage p50s (ms): `host+network` (capture→received) and `decode` (received→decoded).
pub hostnet_p50_ms: f64,
pub decode_p50_ms: f64,
/// Phase-2 `host` / `network` split p50s (ms) — 0.0 when no 0xCF timing matched this window
/// (old host / no samples yet), in which case the HUD keeps the combined `host+network` term.
pub host_p50_ms: f64,
pub net_p50_ms: f64,
pub lat_valid: bool,
pub skew_corrected: bool,
}
/// Percentile over a sorted-in-place µs sample vec, in ms. 0.0 when empty.
fn pctl_ms(sorted_us: &[u64], p: f64) -> f64 {
if sorted_us.is_empty() {
return 0.0;
}
let n = sorted_us.len();
sorted_us[((n as f64 * p) as usize).min(n - 1)] as f64 / 1000.0
}
impl VideoStats {
pub fn new() -> VideoStats {
VideoStats {
@@ -49,14 +83,18 @@ impl VideoStats {
window_start: Instant::now(),
frames: 0,
bytes: 0,
lat_us: Vec::with_capacity(256),
e2e_us: Vec::with_capacity(256),
hostnet_us: Vec::with_capacity(256),
host_us: Vec::with_capacity(256),
net_us: Vec::with_capacity(256),
decode_us: Vec::with_capacity(256),
skew_corrected: false,
}),
}
}
/// Whether the HUD wants samples. The decode thread checks this BEFORE building a latency
/// sample, so the per-frame wall-clock read is skipped too while hidden.
/// sample, so the per-frame wall-clock reads are skipped too while hidden.
// Read only by the android-only decode thread; unreferenced on the host build — expected.
#[cfg_attr(not(target_os = "android"), allow(dead_code))]
pub fn enabled(&self) -> bool {
@@ -75,18 +113,23 @@ impl VideoStats {
g.window_start = Instant::now();
g.frames = 0;
g.bytes = 0;
g.lat_us.clear();
g.e2e_us.clear();
g.hostnet_us.clear();
g.host_us.clear();
g.net_us.clear();
g.decode_us.clear();
}
}
/// Record one decoded access unit: its wire size and (if in range) its capture→client latency.
/// Record one received access unit: its wire size and (if in range) its capture→received
/// `host+network` stage sample. Receipt is the fps/goodput counting point per the spec.
// Driven only by the android-only decode thread; unreferenced on the host build — expected.
#[cfg_attr(not(target_os = "android"), allow(dead_code))]
pub fn note(&self, bytes: usize, lat_us: Option<u64>, skew_corrected: bool) {
pub fn note_received(&self, bytes: usize, hostnet_us: Option<u64>, skew_corrected: bool) {
if !self.enabled.load(Ordering::Relaxed) {
return; // HUD hidden — skip the lock (the caller already skipped the clock read)
}
// Poison-proof: `note` runs per-frame on the decode thread, which has no catch_unwind —
// Poison-proof: this runs per-frame on the decode thread, which has no catch_unwind —
// a panic elsewhere must not turn every later lock into a second panic (the counters
// stay consistent regardless).
let mut g = self
@@ -96,14 +139,56 @@ impl VideoStats {
g.frames += 1;
g.bytes += bytes as u64;
g.skew_corrected = skew_corrected;
if let Some(l) = lat_us {
g.lat_us.push(l);
if let Some(l) = hostnet_us {
g.hostnet_us.push(l);
}
}
/// Record one matched host/network split sample (Phase 2): the host's reported capture→sent
/// duration and our capture→received minus it, both µs — one pair per AU whose 0xCF host
/// timing arrived and matched by pts. An old host emits none, leaving the vecs empty and the
/// snapshot p50s at 0 (HUD keeps the combined `host+network` term).
// Driven only by the android-only decode thread; unreferenced on the host build — expected.
#[cfg_attr(not(target_os = "android"), allow(dead_code))]
pub fn note_host_split(&self, host_us: u64, net_us: u64) {
if !self.enabled.load(Ordering::Relaxed) {
return; // HUD hidden — skip the lock
}
// Poison-proof for the same reason as `note_received`.
let mut g = self
.inner
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
g.host_us.push(host_us);
g.net_us.push(net_us);
}
/// Record one decoded output frame: its capture→decoded `end-to-end` sample and its
/// received→decoded `decode` stage sample (either may be absent — e.g. the receipt stamp for
/// this pts predates the HUD being shown).
// Driven only by the android-only decode thread; unreferenced on the host build — expected.
#[cfg_attr(not(target_os = "android"), allow(dead_code))]
pub fn note_decoded(&self, e2e_us: Option<u64>, decode_us: Option<u64>) {
if !self.enabled.load(Ordering::Relaxed) {
return; // HUD hidden — skip the lock (the caller already skipped the clock read)
}
// Poison-proof for the same reason as `note_received`.
let mut g = self
.inner
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some(l) = e2e_us {
g.e2e_us.push(l);
}
if let Some(l) = decode_us {
g.decode_us.push(l);
}
}
/// Compute the window's rates + latency percentiles, then reset for the next window.
pub fn drain(&self) -> Snapshot {
// Poison-proof for the same reason as `note` — a poisoned window still drains fine.
// Poison-proof for the same reason as `note_received` — a poisoned window still drains
// fine.
let mut g = self
.inner
.lock()
@@ -111,26 +196,31 @@ impl VideoStats {
let elapsed = g.window_start.elapsed().as_secs_f64().max(1e-3);
let fps = g.frames as f64 / elapsed;
let mbps = g.bytes as f64 * 8.0 / 1_000_000.0 / elapsed;
let (p50, p95, valid) = if g.lat_us.is_empty() {
(0.0, 0.0, false)
} else {
g.lat_us.sort_unstable();
let n = g.lat_us.len();
let at = |p: f64| g.lat_us[((n as f64 * p) as usize).min(n - 1)] as f64 / 1000.0;
(at(0.50), at(0.95), true)
g.e2e_us.sort_unstable();
g.hostnet_us.sort_unstable();
g.host_us.sort_unstable();
g.net_us.sort_unstable();
g.decode_us.sort_unstable();
let snap = Snapshot {
fps,
mbps,
e2e_p50_ms: pctl_ms(&g.e2e_us, 0.50),
e2e_p95_ms: pctl_ms(&g.e2e_us, 0.95),
hostnet_p50_ms: pctl_ms(&g.hostnet_us, 0.50),
decode_p50_ms: pctl_ms(&g.decode_us, 0.50),
host_p50_ms: pctl_ms(&g.host_us, 0.50),
net_p50_ms: pctl_ms(&g.net_us, 0.50),
lat_valid: !g.e2e_us.is_empty(),
skew_corrected: g.skew_corrected,
};
let skew = g.skew_corrected;
g.window_start = Instant::now();
g.frames = 0;
g.bytes = 0;
g.lat_us.clear();
Snapshot {
fps,
mbps,
lat_p50_ms: p50,
lat_p95_ms: p95,
lat_valid: valid,
skew_corrected: skew,
}
g.e2e_us.clear();
g.hostnet_us.clear();
g.host_us.clear();
g.net_us.clear();
g.decode_us.clear();
snap
}
}
@@ -326,15 +326,21 @@ struct ContentView: View {
onCaptureChange: { [weak model] captured in
model?.mouseCaptured = captured
},
onFrame: { [meter = model.meter, latency = model.latency, offset = conn.clockOffsetNs] au in
onFrame: { [meter = model.meter, latency = model.latency,
split = model.latencySplit, offset = conn.clockOffsetNs] au in
meter.note(byteCount: au.data.count)
latency.record(ptsNs: au.ptsNs, offsetNs: offset)
// The same receipt, keyed by pts, awaiting its 0xCF host timing (the
// host/network split drained by the 1 s stats tick).
split.recordReceipt(
ptsNs: au.ptsNs, receivedNs: au.receivedNs, offsetNs: offset)
},
onSessionEnd: { [weak model] in
Task { @MainActor in model?.sessionEnded() }
},
presentMeter: model.presentLatency,
presentTailMeter: model.presentTail
endToEndMeter: model.endToEnd,
decodeMeter: model.decodeStage,
displayMeter: model.displayStage
)
.overlay(alignment: placement.alignment) {
if captureEnabled && hudEnabled {
@@ -170,7 +170,10 @@ private struct ShotHUD: View {
Text("5120×1440@240 240 fps 812.4 Mb/s")
.font(.system(.caption, design: .monospaced))
}
Text("capture→client 1.3/2.1 ms p50/p95")
Text("end-to-end 2.9 ms p50 · 3.8 p95 · capture→on-glass")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
Text("= host+network 1.3 + decode 0.7 + display 0.9")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
#if os(macOS)
@@ -59,36 +59,62 @@ final class SessionModel: ObservableObject {
@Published var fps = 0
@Published var mbps = 0.0
@Published var totalFrames = 0
/// Captureclient-receipt latency (ms), skew-corrected across machines via the connect-time
/// clock offset p50/p95 for the HUD. `latencyValid` is false until the first sample drains
/// (and whenever no host frames arrived in the last interval). `latencySkewCorrected` = the host
/// The unified latency stages (design/stats-unification.md), ms per 1 s window. `host+network`
/// = capturereceived, skew-corrected across machines via the connect-time clock offset: the
/// stage-2 HUD shows its p50 in the equation line; the stage-1 fallback shows p50/p95 as its
/// `capturereceived` headline. `hostNetworkValid` is false until the first sample drains (and
/// whenever no host frames arrived in the last interval). `hostNetworkSkewCorrected` = the host
/// answered the skew handshake (the number is cross-machine valid, not just same-host).
@Published var latencyP50Ms = 0.0
@Published var latencyP95Ms = 0.0
@Published var latencyValid = false
@Published var latencySkewCorrected = false
/// Capturepresent (glass-to-glass, modulo the host rendercapture term) only the stage-2
/// presenter can stamp this (it owns decode + a CAMetalLayer/display-link present). Stays
/// invalid under stage-1, where the layer presents internally with no per-frame callback.
@Published var presentLatencyP50Ms = 0.0
@Published var presentLatencyP95Ms = 0.0
@Published var presentLatencyValid = false
@Published var presentLatencySkewCorrected = false
/// Decode-completionpresent (the "present tail": ring wait + render + vsync) the term the
/// stage-2 presenter exists to shorten. Both instants are client-side, so no skew applies.
@Published var presentTailP50Ms = 0.0
@Published var presentTailP95Ms = 0.0
@Published var presentTailValid = false
@Published var hostNetworkP50Ms = 0.0
@Published var hostNetworkP95Ms = 0.0
@Published var hostNetworkValid = false
@Published var hostNetworkSkewCorrected = false
/// Phase 2 of the same stage: `host+network` split into its two terms via the host's per-AU
/// 0xCF timing reports (host = capturefully-sent as the host measured it, network = the
/// remainder), matched to receipts by pts in `latencySplit`. `splitValid` is false whenever
/// no timing matched in the window an old host that never emits the plane, or heavy 0xCF
/// loss and the HUD then falls back to the combined `host+network` term.
@Published var hostP50Ms = 0.0
@Published var networkP50Ms = 0.0
@Published var splitValid = false
/// End-to-end = captureon-glass, measured directly per frame (never summed from the stages)
/// the HUD headline. Only the stage-2 presenter can stamp it (it owns decode + a
/// CAMetalLayer/display-link present); stays invalid under stage-1, where the layer presents
/// internally with no per-frame callback.
@Published var endToEndP50Ms = 0.0
@Published var endToEndP95Ms = 0.0
@Published var endToEndValid = false
@Published var endToEndSkewCorrected = false
/// The client-local stage terms of the HUD's equation line (single clock, no skew; p50 only):
/// decode = receiveddecoded, display = decodedon-glass (ring wait + render + vsync the
/// term the stage-2 presenter exists to shorten).
@Published var decodeP50Ms = 0.0
@Published var decodeValid = false
@Published var displayP50Ms = 0.0
@Published var displayValid = false
/// Unrecoverable network frame drops in the last window (FEC couldn't rebuild them) and their
/// share of frames offered, `lost/(received+lost)`. The HUD hides the line while zero.
@Published var lostFrames = 0
@Published var lostPct = 0.0
/// Mirrors StreamView's capture state (it owns the input capture; this drives the
/// HUD's "click to capture" / " releases" hint).
@Published var mouseCaptured = false
let meter = FrameMeter()
/// Capturereceived (the host+network stage), fed per AU at receipt by the stream view's
/// onFrame under both presenters.
let latency = LatencyMeter()
/// Fed by the stage-2 presenter's display link (capturepresent). Passed to StreamView.
let presentLatency = LatencyMeter()
/// Fed by the same present stamp (decode-completionpresent). Passed to StreamView.
let presentTail = LatencyMeter()
/// The host/network split of that same stage: onFrame also records (pts, interval) receipts
/// here, and the 1 s stats tick drains the connection's 0xCF host timings into it under
/// both presenters (the receipt path is presenter-independent).
let latencySplit = HostNetworkSplitter()
/// The stage-2 meters, passed to StreamView: end-to-end (captureon-glass, stamped at
/// present), decode (receiveddecoded), display (decodedon-glass).
let endToEnd = LatencyMeter()
let decodeStage = LatencyMeter()
let displayStage = LatencyMeter()
/// Cumulative reassembler-drop counter at the last stats drain (per-window `lost` delta).
private var lastFramesDropped: UInt64 = 0
private var statsTimer: Timer?
private var audio: SessionAudio?
private var gamepadCapture: GamepadCapture?
@@ -281,7 +307,13 @@ final class SessionModel: ObservableObject {
phase = .idle
fps = 0
mbps = 0
latencyValid = false
hostNetworkValid = false
splitValid = false
endToEndValid = false
decodeValid = false
displayValid = false
lostFrames = 0
lostPct = 0
mouseCaptured = false
}
@@ -306,6 +338,7 @@ final class SessionModel: ObservableObject {
audio.start(
speakerUID: defaults.string(forKey: DefaultsKey.speakerUID) ?? "",
micUID: defaults.string(forKey: DefaultsKey.micUID) ?? "",
micChannel: defaults.integer(forKey: DefaultsKey.micChannel),
micEnabled: defaults.object(forKey: DefaultsKey.micEnabled) as? Bool ?? true)
self.audio = audio
// Gamepads: forward GamepadManager's active controller as pad 0 and render the
@@ -321,6 +354,8 @@ final class SessionModel: ObservableObject {
}
private func startStatsTimer() {
lastFramesDropped = 0 // a fresh connection's cumulative drop counter starts at 0
latencySplit.reset() // no stale receipts/samples from a previous session
let timer = Timer(timeInterval: 1.0, repeats: true) { [weak self] _ in
guard let self else { return }
Task { @MainActor in
@@ -328,28 +363,60 @@ final class SessionModel: ObservableObject {
self.fps = frames
self.mbps = Double(bytes) * 8 / 1_000_000
self.totalFrames = total
// Per-window `lost` = the delta of the connector's cumulative reassembler-drop
// counter (0 after close treat a rewind as no loss rather than underflowing).
let dropped = self.connection?.framesDropped() ?? 0
let lost = dropped >= self.lastFramesDropped
? Int(dropped - self.lastFramesDropped) : 0
self.lastFramesDropped = dropped
self.lostFrames = lost
self.lostPct = lost > 0 ? Double(lost) / Double(frames + lost) * 100 : 0
if let lat = self.latency.drain() {
self.latencyP50Ms = lat.p50Ms
self.latencyP95Ms = lat.p95Ms
self.latencySkewCorrected = lat.skewCorrected
self.latencyValid = true
self.hostNetworkP50Ms = lat.p50Ms
self.hostNetworkP95Ms = lat.p95Ms
self.hostNetworkSkewCorrected = lat.skewCorrected
self.hostNetworkValid = true
} else {
self.latencyValid = false
self.hostNetworkValid = false
}
if let p = self.presentLatency.drain() {
self.presentLatencyP50Ms = p.p50Ms
self.presentLatencyP95Ms = p.p95Ms
self.presentLatencySkewCorrected = p.skewCorrected
self.presentLatencyValid = true
} else {
self.presentLatencyValid = false
// Phase 2: drain the window's per-AU host timings (0xCF) into the splitter
// non-blocking, bounded (a 240 fps window is ~240 reports; the cap only guards
// a pathological burst). `try?` flattens (SE-0230); a throw (.closed during
// teardown) just ends the drain. An old host never emits any splitValid stays
// false and the HUD keeps the combined host+network term.
if let conn = self.connection {
var burst = 0
while burst < 1024, let t = try? conn.nextHostTiming(timeoutMs: 0) {
self.latencySplit.noteHostTiming(ptsNs: t.ptsNs, hostUs: t.hostUs)
burst += 1
}
}
if let t = self.presentTail.drain() {
self.presentTailP50Ms = t.p50Ms
self.presentTailP95Ms = t.p95Ms
self.presentTailValid = true
if let s = self.latencySplit.drain() {
self.hostP50Ms = s.hostP50Ms
self.networkP50Ms = s.networkP50Ms
self.splitValid = true
} else {
self.presentTailValid = false
self.splitValid = false
}
if let e = self.endToEnd.drain() {
self.endToEndP50Ms = e.p50Ms
self.endToEndP95Ms = e.p95Ms
self.endToEndSkewCorrected = e.skewCorrected
self.endToEndValid = true
} else {
self.endToEndValid = false
}
if let d = self.decodeStage.drain() {
self.decodeP50Ms = d.p50Ms
self.decodeValid = true
} else {
self.decodeValid = false
}
if let d = self.displayStage.drain() {
self.displayP50Ms = d.p50Ms
self.displayValid = true
} else {
self.displayValid = false
}
}
}
@@ -1,5 +1,7 @@
// The streaming overlay HUD: mode + fps/throughput, the captureclient (and, under the stage-2
// presenter, capturepresent) latency lines, the platform input hint, and disconnect.
// The streaming overlay HUD: mode + fps/throughput, the unified latency lines
// (design/stats-unification.md end-to-end headline + the stage equation under stage-2, the
// capturereceived headline under the stage-1 fallback), the loss counter, the platform input
// hint, and disconnect.
import PunktfunkKit
import SwiftUI
@@ -18,24 +20,46 @@ struct StreamHUDView: View {
Text("\(connection.width)×\(connection.height)@\(connection.refreshHz) \(model.fps) fps \(model.mbps, specifier: "%.1f") Mb/s")
.font(.system(.caption, design: .monospaced))
}
if model.latencyValid {
// Captureclient-receipt (skew-corrected); excludes the layer's decode+present
// see LatencyMeter. "(same-host)" when the host didn't answer the skew handshake.
Text("capture→client \(model.latencyP50Ms, specifier: "%.1f")/\(model.latencyP95Ms, specifier: "%.1f") ms p50/p95\(model.latencySkewCorrected ? "" : " (same-host)")")
if model.endToEndValid {
// Stage-2: the end-to-end headline (captureon-glass, measured directly, skew-
// corrected) "(same-host clock)" when the host didn't answer the skew handshake.
Text("end-to-end \(model.endToEndP50Ms, specifier: "%.1f") ms p50 · \(model.endToEndP95Ms, specifier: "%.1f") p95 · capture→on-glass\(model.endToEndSkewCorrected ? "" : " (same-host clock)")")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
}
if model.presentLatencyValid {
// Capturepresent (glass-to-glass, modulo host rendercapture) stage-2 presenter
// only; stage-1's layer presents internally with no per-frame stamp.
Text("capture→present \(model.presentLatencyP50Ms, specifier: "%.1f")/\(model.presentLatencyP95Ms, specifier: "%.1f") ms p50/p95\(model.presentLatencySkewCorrected ? "" : " (same-host)")")
// The equation: the stages tiling the headline interval (per-window p50s
// they only approximately sum to the directly-measured total). With a host
// that reports per-AU timings (0xCF) the first term splits into host + network
// (phase 2); an old host keeps the combined term.
if model.hostNetworkValid && model.decodeValid && model.displayValid {
if model.splitValid {
Text("= host \(model.hostP50Ms, specifier: "%.1f") + network \(model.networkP50Ms, specifier: "%.1f") + decode \(model.decodeP50Ms, specifier: "%.1f") + display \(model.displayP50Ms, specifier: "%.1f")")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
} else {
Text("= host+network \(model.hostNetworkP50Ms, specifier: "%.1f") + decode \(model.decodeP50Ms, specifier: "%.1f") + display \(model.displayP50Ms, specifier: "%.1f")")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
}
}
} else if model.hostNetworkValid {
// Stage-1 fallback presenter: the layer decodes + presents internally with no
// per-frame stamp, so the honest headline ends at receipt. The host/network
// split still applies there (receipt is presenter-independent) it becomes the
// only equation line; without it, host+network IS the whole measured interval.
Text("capture→received \(model.hostNetworkP50Ms, specifier: "%.1f") ms p50 · \(model.hostNetworkP95Ms, specifier: "%.1f") p95\(model.hostNetworkSkewCorrected ? "" : " (same-host clock)")")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
if model.splitValid {
Text("= host \(model.hostP50Ms, specifier: "%.1f") + network \(model.networkP50Ms, specifier: "%.1f")")
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
}
}
if model.presentTailValid {
// Decodepresent (the client-local "present tail": ring wait + render + vsync)
// the term the stage-2 presenter shortens; no skew applies (one clock).
Text("decode→present \(model.presentTailP50Ms, specifier: "%.1f")/\(model.presentTailP95Ms, specifier: "%.1f") ms p50/p95")
if model.lostFrames > 0 {
// Unrecoverable network drops this window; hidden while the link is clean.
// String(format:) rather than specifier interpolation: the literal % would
// otherwise land in the LocalizedStringKey's format string as a bogus conversion.
Text(String(format: "lost %d (%.1f%%)", model.lostFrames, model.lostPct))
.font(.system(.caption2, design: .monospaced))
.foregroundStyle(.secondary)
}
@@ -7,63 +7,15 @@ import SwiftUI
extension SettingsView {
// MARK: - Sections (shared)
// NOTE: the Section content is deliberately split into the small named builders below as one
// inline expression the iOS branch (wheel + 3-way refresh + bitrate rows) blew Swift's
// type-checker budget ("unable to type-check this expression in reasonable time"), which
// failed exactly one slice: the iOS archive (macOS/tvOS never compile that branch).
@ViewBuilder var streamModeSection: some View {
Section {
#if os(iOS)
// Touch-first: a rotating wheel of common resolutions (this device's own mode first) and
// a segmented refresh-rate control the same family as the Clock/Timer pickers. The host
// renders a virtual output at exactly the chosen mode, so these are real pixel sizes. The
// last wheel row, "Custom", reveals width/height/refresh fields for an arbitrary mode.
VStack(alignment: .leading, spacing: 4) {
Text("Resolution")
.font(.geist(15, relativeTo: .subheadline))
.foregroundStyle(.secondary)
Picker("Resolution", selection: resolutionSelection) {
ForEach(resolutionChoices, id: \.tag) { choice in
Text(choice.label).tag(choice.tag)
}
}
.labelsHidden()
.pickerStyle(.wheel)
.frame(maxHeight: 140)
}
if isCustomResolution {
// Arbitrary entry: type the exact width × height (and refresh) the host should drive.
HStack {
TextField("Width", value: $width, format: .number.grouping(.never))
.keyboardType(.numberPad)
Text("×")
TextField("Height", value: $height, format: .number.grouping(.never))
.labelsHidden()
.keyboardType(.numberPad)
}
// A row built from an HStack of TextFields otherwise insets its bottom separator to
// the inner content, clipping the hairline under "Width"; pin it to the cell edge.
.alignmentGuide(.listRowSeparatorLeading) { _ in 0 }
LabeledContent("Refresh rate") {
TextField("Hz", value: $hz, format: .number.grouping(.never))
.keyboardType(.numberPad)
.multilineTextAlignment(.trailing)
}
} else if refreshChoices.count > 1 {
VStack(alignment: .leading, spacing: 6) {
Text("Refresh rate")
.font(.geist(15, relativeTo: .subheadline))
.foregroundStyle(.secondary)
Picker("Refresh rate", selection: $hz) {
ForEach(refreshChoices, id: \.self) { rate in
Text("\(rate) Hz").tag(rate)
}
}
.labelsHidden()
.pickerStyle(.segmented)
}
} else {
// A device with a single supported rate (e.g. 60 Hz) has nothing to pick.
LabeledContent("Refresh rate") {
Text("\(hz) Hz").foregroundStyle(.secondary)
}
}
iosResolutionWheel
iosRefreshRows
Button("Use this display's mode") { fillFromMainScreen() }
#elseif os(macOS)
HStack {
@@ -78,23 +30,7 @@ extension SettingsView {
}
#endif
#if !os(tvOS)
Toggle("Automatic bitrate", isOn: automaticBitrate)
if bitrateKbps != 0 {
HStack(spacing: 12) {
Slider(value: bitrateSlider, in: 0...1) {
Text("Bitrate")
}
Text(SpeedTestSheet.mbpsLabel(kbps: bitrateKbps))
.monospacedDigit()
.foregroundStyle(.secondary)
.frame(minWidth: 76, alignment: .trailing)
}
if bitrateKbps > 1_000_000 {
Label(Self.gigabitWarning, systemImage: "exclamationmark.triangle.fill")
.font(.geist(12, relativeTo: .caption))
.foregroundStyle(.orange)
}
}
bitrateRows
#endif
} header: {
Text("Stream mode")
@@ -109,6 +45,67 @@ extension SettingsView {
#if os(iOS)
// MARK: - Stream mode (iOS wheel)
/// Touch-first: a rotating wheel of common resolutions (this device's own mode first) the
/// same family as the Clock/Timer pickers. The host renders a virtual output at exactly the
/// chosen mode, so these are real pixel sizes. The last wheel row, "Custom", reveals
/// width/height/refresh fields for an arbitrary mode (see `iosRefreshRows`).
@ViewBuilder private var iosResolutionWheel: some View {
VStack(alignment: .leading, spacing: 4) {
Text("Resolution")
.font(.geist(15, relativeTo: .subheadline))
.foregroundStyle(.secondary)
Picker("Resolution", selection: resolutionSelection) {
ForEach(resolutionChoices, id: \.tag) { choice in
Text(choice.label).tag(choice.tag)
}
}
.labelsHidden()
.pickerStyle(.wheel)
.frame(maxHeight: 140)
}
}
/// Custom W×H(+Hz) fields, a segmented refresh picker, or a static single-rate row.
@ViewBuilder private var iosRefreshRows: some View {
if isCustomResolution {
// Arbitrary entry: type the exact width × height (and refresh) the host should drive.
HStack {
TextField("Width", value: $width, format: .number.grouping(.never))
.keyboardType(.numberPad)
Text("×")
TextField("Height", value: $height, format: .number.grouping(.never))
.labelsHidden()
.keyboardType(.numberPad)
}
// A row built from an HStack of TextFields otherwise insets its bottom separator to
// the inner content, clipping the hairline under "Width"; pin it to the cell edge.
.alignmentGuide(.listRowSeparatorLeading) { _ in 0 }
LabeledContent("Refresh rate") {
TextField("Hz", value: $hz, format: .number.grouping(.never))
.keyboardType(.numberPad)
.multilineTextAlignment(.trailing)
}
} else if refreshChoices.count > 1 {
VStack(alignment: .leading, spacing: 6) {
Text("Refresh rate")
.font(.geist(15, relativeTo: .subheadline))
.foregroundStyle(.secondary)
Picker("Refresh rate", selection: $hz) {
ForEach(refreshChoices, id: \.self) { rate in
Text("\(rate) Hz").tag(rate)
}
}
.labelsHidden()
.pickerStyle(.segmented)
}
} else {
// A device with a single supported rate (e.g. 60 Hz) has nothing to pick.
LabeledContent("Refresh rate") {
Text("\(hz) Hz").foregroundStyle(.secondary)
}
}
}
/// Sentinel wheel tag for the "Custom" row. Real tags are "WxH" (digits + "x"), so this can't
/// collide with a resolution.
private static let customResolutionTag = "custom"
@@ -156,6 +153,29 @@ extension SettingsView {
}
#endif
#if !os(tvOS)
/// The automatic-bitrate toggle + manual slider (and the >1 Gbps warning) rows.
@ViewBuilder private var bitrateRows: some View {
Toggle("Automatic bitrate", isOn: automaticBitrate)
if bitrateKbps != 0 {
HStack(spacing: 12) {
Slider(value: bitrateSlider, in: 0...1) {
Text("Bitrate")
}
Text(SpeedTestSheet.mbpsLabel(kbps: bitrateKbps))
.monospacedDigit()
.foregroundStyle(.secondary)
.frame(minWidth: 76, alignment: .trailing)
}
if bitrateKbps > 1_000_000 {
Label(Self.gigabitWarning, systemImage: "exclamationmark.triangle.fill")
.font(.geist(12, relativeTo: .caption))
.foregroundStyle(.orange)
}
}
}
#endif
@ViewBuilder var audioSection: some View {
Section {
Picker("Audio channels", selection: $audioChannels) {
@@ -188,6 +208,17 @@ extension SettingsView {
}
}
.disabled(!micEnabled)
// Multi-channel interfaces only: the mic sits on ONE discrete input, so let the user
// pick it. Auto sums every channel (a lone hot mic still passes at full level).
if micChannelCount > 1 {
Picker("Microphone channel", selection: $micChannel) {
Text("Auto (all channels)").tag(0)
ForEach(1...micChannelCount, id: \.self) { ch in
Text("Channel \(ch)").tag(ch)
}
}
.disabled(!micEnabled)
}
#endif
} header: {
Text("Audio")
@@ -204,35 +235,42 @@ extension SettingsView {
/// Touch-input model (iPhone + iPad) plus the iPad-only pointer-capture toggle: lock the
/// mouse/trackpad for relative movement (games) vs forward an absolute cursor position.
@ViewBuilder var pointerSection: some View {
let isPad = UIDevice.current.userInterfaceIdiom == .pad
Section {
Picker("Touch input", selection: $touchMode) {
Text("Trackpad").tag(TouchInputMode.trackpad.rawValue)
Text("Direct pointer").tag(TouchInputMode.pointer.rawValue)
Text("Touch passthrough").tag(TouchInputMode.touch.rawValue)
}
if isPad {
if UIDevice.current.userInterfaceIdiom == .pad {
Toggle("Capture pointer for games", isOn: $pointerCapture)
}
} header: {
Text("Touch & pointer")
} footer: {
Text("Trackpad: your finger nudges the host cursor like a laptop touchpad — tap to "
+ "click, two-finger tap for a right click, two-finger drag to scroll, "
+ "tap-then-drag to hold the button, three-finger tap for the stats overlay. "
+ "Direct pointer: the cursor jumps to your finger. Touch passthrough: real "
+ "multi-touch reaches the host, for apps that understand touch. Applies from "
+ "the next touch."
+ (isPad
? " Pointer capture locks a hardware mouse/trackpad for relative movement "
+ "(mouse-look); off keeps the pointer free and sends absolute positions. "
+ "The lock needs the stream full-screen and frontmost, and falls back "
+ "automatically (Stage Manager, Slide Over)."
: ""))
Text(pointerFooterText)
.font(.geist(12, relativeTo: .caption))
.foregroundStyle(.secondary)
}
}
/// Footer copy for `pointerSection`, built in plain `+=` statements. Deliberately NOT one big
/// `+` chain (with a ternary) inside the ViewBuilder that single expression blew Swift's
/// type-checker budget and was what actually broke the iOS archive.
private var pointerFooterText: String {
var text = "Trackpad: your finger nudges the host cursor like a laptop touchpad — tap to "
text += "click, two-finger tap for a right click, two-finger drag to scroll, "
text += "tap-then-drag to hold the button, three-finger tap for the stats overlay. "
text += "Direct pointer: the cursor jumps to your finger. Touch passthrough: real "
text += "multi-touch reaches the host, for apps that understand touch. Applies from "
text += "the next touch."
if UIDevice.current.userInterfaceIdiom == .pad {
text += " Pointer capture locks a hardware mouse/trackpad for relative movement "
text += "(mouse-look); off keeps the pointer free and sends absolute positions. "
text += "The lock needs the stream full-screen and frontmost, and falls back "
text += "automatically (Stage Manager, Slide Over)."
}
return text
}
#endif
@ViewBuilder var compositorSection: some View {
@@ -283,10 +321,11 @@ extension SettingsView {
Text("Video presenter · debug")
} footer: {
Text("Stage 2 (default) decodes explicitly and presents through Metal with a display "
+ "link — it adds a capture→present (glass-to-glass) latency line in the HUD and "
+ "self-recovers from decode stalls. Stage 1 feeds compressed video straight to the "
+ "system display layer; it freezes on a lost HEVC reference frame, so it's a debug "
+ "fallback only. Applies from the next session.")
+ "link — it gives the HUD the end-to-end (capture→on-glass) headline with the "
+ "host+network/decode/display stage equation and self-recovers from decode "
+ "stalls. Stage 1 feeds compressed video straight to the system display layer; "
+ "it freezes on a lost HEVC reference frame, so it's a debug fallback only. "
+ "Applies from the next session.")
.font(.geist(12, relativeTo: .caption))
.foregroundStyle(.secondary)
}
@@ -61,8 +61,12 @@ struct SettingsView: View {
#if os(macOS)
@AppStorage(DefaultsKey.speakerUID) var speakerUID = ""
@AppStorage(DefaultsKey.micUID) var micUID = ""
@AppStorage(DefaultsKey.micChannel) var micChannel = 0
@State var outputDevices: [AudioDevice] = []
@State var inputDevices: [AudioDevice] = []
// Input channels of the selected mic drives the "Microphone channel" picker, which only
// appears for a multi-channel interface (>1). 0 until the Audio tab loads it.
@State var micChannelCount = 0
#endif
#if os(iOS)
@@ -115,6 +119,12 @@ struct SettingsView: View {
.onAppear {
outputDevices = AudioDevices.outputs()
inputDevices = AudioDevices.inputs()
micChannelCount = AudioDevices.inputChannelCount(forUID: micUID)
}
.onChange(of: micUID) { _, newUID in
// A different mic different channel count; drop a now-out-of-range pin to Auto.
micChannelCount = AudioDevices.inputChannelCount(forUID: newUID)
if micChannel > micChannelCount { micChannel = 0 }
}
.tabItem { Label("Audio", systemImage: "speaker.wave.2") }
@@ -33,6 +33,49 @@ public enum AudioDevices {
}
}
/// Input channel count of the mic the picker would use the device with this UID, or the
/// system default input when `uid` is empty. 0 when it can't be resolved. Drives the
/// "Microphone channel" picker (only shown for multi-channel interfaces).
public static func inputChannelCount(forUID uid: String) -> Int {
let id = uid.isEmpty ? defaultInputDevice() : deviceID(forUID: uid)
guard let id else { return 0 }
return channelCount(id, scope: kAudioObjectPropertyScopeInput)
}
private static func defaultInputDevice() -> AudioDeviceID? {
var address = AudioObjectPropertyAddress(
mSelector: kAudioHardwarePropertyDefaultInputDevice,
mScope: kAudioObjectPropertyScopeGlobal,
mElement: kAudioObjectPropertyElementMain)
var dev = AudioDeviceID(0)
var size = UInt32(MemoryLayout<AudioDeviceID>.size)
guard AudioObjectGetPropertyData(
AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size, &dev) == noErr,
dev != 0
else { return nil }
return dev
}
/// Sum of channels across the device's streams in `scope` (its total input/output channels).
private static func channelCount(
_ id: AudioDeviceID, scope: AudioObjectPropertyScope
) -> Int {
var address = AudioObjectPropertyAddress(
mSelector: kAudioDevicePropertyStreamConfiguration,
mScope: scope,
mElement: kAudioObjectPropertyElementMain)
var size: UInt32 = 0
guard AudioObjectGetPropertyDataSize(id, &address, 0, nil, &size) == noErr, size > 0
else { return 0 }
let raw = UnsafeMutableRawPointer.allocate(
byteCount: Int(size), alignment: MemoryLayout<AudioBufferList>.alignment)
defer { raw.deallocate() }
guard AudioObjectGetPropertyData(id, &address, 0, nil, &size, raw) == noErr else { return 0 }
let abl = UnsafeMutableAudioBufferListPointer(
raw.assumingMemoryBound(to: AudioBufferList.self))
return abl.reduce(0) { $0 + Int($1.mNumberChannels) }
}
private static func all() -> [AudioDeviceID] {
var address = AudioObjectPropertyAddress(
mSelector: kAudioHardwarePropertyDevices,
@@ -62,7 +105,8 @@ public enum AudioDevices {
return AudioObjectGetPropertyDataSize(id, &address, 0, nil, &size) == noErr && size > 0
}
private static func describe(_ id: AudioDeviceID) -> AudioDevice? {
/// UID + human name for a live AudioDeviceID (nil if either property is unreadable).
static func describe(_ id: AudioDeviceID) -> AudioDevice? {
guard let uid = stringProperty(id, kAudioDevicePropertyDeviceUID),
let name = stringProperty(id, kAudioObjectPropertyName)
else { return nil }
@@ -5,9 +5,10 @@
// AVAudioSourceNode pulls from the ring (silence on underrun with re-priming, so a
// network gap costs one dip, not permanent crackle).
//
// mic host: a second AVAudioEngine taps the input device, resamples to 48 kHz
// stereo, slices 20 ms chunks, Opus-encodes, and sendMic()s each packet the host
// feeds them into a virtual PipeWire source.
// mic host: a second AVAudioEngine taps the input device, folds it to one mono bus (the
// chosen channel of a multi-channel interface, or a sum of all channels), resamples to 48 kHz
// stereo, slices 20 ms chunks, Opus-encodes, and sendMic()s each packet the host feeds them
// into a virtual PipeWire source.
//
// Devices are chosen by UID ("" = system default: the engine is then never pinned to a
// concrete device and follows default-device changes). Two engines, not one a single
@@ -68,10 +69,11 @@ public final class SessionAudio {
/// ASYNCHRONOUS: it activates the AVAudioSession off the main thread, then starts the engines on
/// a later main-queue hop (gated by `!flag.isStopped`) so playback is live shortly after, not
/// on return. The mic may start later still if the permission prompt is pending.
public func start(speakerUID: String, micUID: String, micEnabled: Bool) {
public func start(speakerUID: String, micUID: String, micChannel: Int, micEnabled: Bool) {
#if os(macOS)
// No AVAudioSession on macOS start the engines directly (caller's thread, as before).
startEngines(speakerUID: speakerUID, micUID: micUID, micEnabled: micEnabled)
startEngines(
speakerUID: speakerUID, micUID: micUID, micChannel: micChannel, micEnabled: micEnabled)
#else
// Configure + activate the session OFF the main thread (it blocks on the audio server),
// then start the engines back on the main thread once it's active engine routing/format
@@ -81,7 +83,9 @@ public final class SessionAudio {
self.activateAudioSession(micEnabled: micEnabled)
DispatchQueue.main.async { [weak self] in
guard let self, !self.flag.isStopped else { return }
self.startEngines(speakerUID: speakerUID, micUID: micUID, micEnabled: micEnabled)
self.startEngines(
speakerUID: speakerUID, micUID: micUID, micChannel: micChannel,
micEnabled: micEnabled)
}
}
#endif
@@ -115,7 +119,9 @@ public final class SessionAudio {
/// Build + start the playback engine (and the mic uplink when enabled + authorized). Main
/// thread (engine setup); on iOS/tvOS the session is already active by the time this runs.
private func startEngines(speakerUID: String, micUID: String, micEnabled: Bool) {
private func startEngines(
speakerUID: String, micUID: String, micChannel: Int, micEnabled: Bool
) {
startPlayback(speakerUID: speakerUID)
#if os(tvOS)
// No app-accessible microphone input on tvOS playback only.
@@ -123,12 +129,12 @@ public final class SessionAudio {
guard micEnabled else { return }
switch AVCaptureDevice.authorizationStatus(for: .audio) {
case .authorized:
startCapture(micUID: micUID)
startCapture(micUID: micUID, micChannel: micChannel)
case .notDetermined:
AVCaptureDevice.requestAccess(for: .audio) { [weak self] granted in
DispatchQueue.main.async {
guard let self, granted, !self.flag.isStopped else { return }
self.startCapture(micUID: micUID)
self.startCapture(micUID: micUID, micChannel: micChannel)
}
}
default:
@@ -280,7 +286,7 @@ public final class SessionAudio {
// MARK: - Mic (mic host)
#if !os(tvOS)
private func startCapture(micUID: String) {
private func startCapture(micUID: String, micChannel: Int) {
let engine = AVAudioEngine()
let input = engine.inputNode
#if os(macOS)
@@ -300,8 +306,63 @@ public final class SessionAudio {
log.error("no usable input device — mic uplink disabled")
return
}
guard let encoder = try? OpusEncoder(),
let resampler = AVAudioConverter(from: inFormat, to: encoder.pcmFormat),
// Multi-channel-interface handling. A pro interface exposes N discrete inputs with the mic
// on ONE of them, but AVAudioConverter's Nstereo downmix takes channels 0/1 dead
// silence when the mic sits higher up (the classic "host receives zeros"). So we fold the
// input to a single mono bus OURSELVES and resample that. micChannel: 0 = Auto (sum every
// channel a lone hot mic passes at full level), n1 pins 1-based input channel n.
let inChannels = Int(inFormat.channelCount)
let pinnedChannel: Int? = {
guard micChannel >= 1 else { return nil }
let idx = micChannel - 1
guard idx < inChannels else {
log.warning(
"mic channel \(micChannel) out of range (device has \(inChannels)) — mixing all")
return nil
}
return idx
}()
let channelPlan = pinnedChannel.map { "channel \($0 + 1)/\(inChannels)" }
?? (inChannels > 1 ? "mix \(inChannels)ch→mono" : "mono")
// Name the device we're ACTUALLY recording from + its format + how we fold it, once per
// session. This single line localizes the whole class of "host receives silence" failures
// that otherwise need a host-side tone injection to pin down: a UID that silently fell back
// to the default, the wrong device being live, or the wrong channel picked.
#if os(macOS)
if let unit = input.audioUnit, let live = Self.currentDevice(of: unit),
let dev = AudioDevices.describe(live) {
if !micUID.isEmpty, dev.uid != micUID {
log.warning("""
mic selection not honored — requested \(micUID) but capturing from \
\(dev.name) [\(dev.uid)]; the device's UID likely changed (replug) — \
reselect it in Settings
""")
}
log.info("""
mic capture: \(dev.name) [\(dev.uid)] — \(Int(inFormat.sampleRate)) Hz, \
\(inChannels) ch, \(channelPlan)
""")
} else {
log.info("""
mic capture: <device unavailable> — \(Int(inFormat.sampleRate)) Hz, \
\(inChannels) ch, \(channelPlan)
""")
}
#else
log.info(
"mic capture: \(Int(inFormat.sampleRate)) Hz, \(inChannels) ch, \(channelPlan)")
#endif
// Encode a single mono bus (folded from `inFormat` in the tap): the resampler goes
// mono@inputSR the encoder's 48 kHz stereo, so it handles both the rate change and the
// monostereo duplication, and the wrong-channel downmix never happens.
guard let monoFormat = AVAudioFormat(
commonFormat: .pcmFormatFloat32, sampleRate: inFormat.sampleRate,
channels: 1, interleaved: false),
let encoder = try? OpusEncoder(),
let resampler = AVAudioConverter(from: monoFormat, to: encoder.pcmFormat),
let chunk = AVAudioPCMBuffer(
pcmFormat: encoder.pcmFormat, frameCapacity: OpusEncoder.framesPerPacket)
else {
@@ -317,11 +378,59 @@ public final class SessionAudio {
let connection = connection
let flag = flag
// Silence tripwire (tap-confined): a "recording" app can be handed pure digital zeros
// a zeroed input-volume slider, a stale TCC grant, a muted device, OR the wrong channel
// picked and everything downstream looks alive while the host gets silence. Track the
// peak of the EXTRACTED mono bus over the first ~10 s (not the raw device a mic present
// on a channel we didn't grab must still read as silence) and emit exactly ONE verdict.
// This is the log line whose absence made the last occurrence take a host-side tone.
let silenceWindow = Int(inFormat.sampleRate * 10)
let deviceLabel = micUID.isEmpty ? "default input" : micUID
var framesInspected = 0
var inputPeak: Float = 0
var levelReported = false
input.installTap(onBus: 0, bufferSize: 2048, format: inFormat) { buffer, _ in
if flag.isStopped { return }
let frames = Int(buffer.frameLength)
guard frames > 0, let src = buffer.floatChannelData,
let mono = AVAudioPCMBuffer(
pcmFormat: monoFormat, frameCapacity: buffer.frameLength),
let dst = mono.floatChannelData?[0]
else { return }
mono.frameLength = buffer.frameLength
// Fold the multi-channel input down to the one mono bus we encode.
Self.foldToMono(
input: src, frames: frames, channels: Int(buffer.format.channelCount),
interleaved: buffer.format.isInterleaved, pinned: pinnedChannel, out: dst)
if !levelReported {
var localPeak: Float = 0
for i in 0..<frames where abs(dst[i]) > localPeak { localPeak = abs(dst[i]) }
if localPeak > inputPeak { inputPeak = localPeak }
framesInspected += frames
if framesInspected >= silenceWindow {
levelReported = true
if inputPeak == 0 {
log.warning("""
mic uplink has been pure digital SILENCE for 10 s (\(deviceLabel), \
\(channelPlan)) — check the input level (System Settings → Sound → \
Input), Privacy & Security → Microphone, and the Microphone channel in \
Settings; the host is receiving zeros
""")
} else {
let dbfs = 20 * log10(inputPeak)
log.info("""
mic uplink OK — peak \(String(format: "%.1f", dbfs)) dBFS over first \
10 s (\(deviceLabel), \(channelPlan))
""")
}
}
}
let ratio = 48_000 / inFormat.sampleRate
let outCapacity = AVAudioFrameCount(
(Double(buffer.frameLength) * ratio).rounded(.up) + 64)
let outCapacity = AVAudioFrameCount((Double(frames) * ratio).rounded(.up) + 64)
guard let staging = AVAudioPCMBuffer(
pcmFormat: encoder.pcmFormat, frameCapacity: outCapacity)
else { return }
@@ -334,7 +443,7 @@ public final class SessionAudio {
}
fed = true
outStatus.pointee = .haveData
return buffer
return mono
}
guard status != .error, let p = staging.floatChannelData?[0] else { return }
fifo.append(contentsOf: UnsafeBufferPointer(
@@ -378,6 +487,42 @@ public final class SessionAudio {
stateLock.unlock()
log.info("mic uplink started (\(micUID.isEmpty ? "default input" : micUID))")
}
/// Fold `channels` of input (`floatChannelData` layout: `interleaved` one buffer strided by
/// channel count; else one buffer per channel) down to a single mono bus in `out` (`frames`
/// long). `pinned` (0-based, must be `< channels`) copies exactly that channel the fix for a
/// mic on one input of a multi-channel interface; `nil` sums every channel, clamped to
/// [-1, 1], so a lone hot channel still passes at full level instead of the silent 0/1 the
/// default Nstereo downmix would grab. Pure + `internal` for unit testing the index math.
static func foldToMono(
input: UnsafePointer<UnsafeMutablePointer<Float>>, frames: Int, channels: Int,
interleaved: Bool, pinned: Int?, out: UnsafeMutablePointer<Float>
) {
if let ch = pinned, ch < channels {
if interleaved {
let d = input[0]
for i in 0..<frames { out[i] = d[i * channels + ch] }
} else {
let d = input[ch]
for i in 0..<frames { out[i] = d[i] }
}
} else if interleaved {
let d = input[0]
for i in 0..<frames {
var s: Float = 0
for c in 0..<channels { s += d[i * channels + c] }
out[i] = max(-1, min(1, s))
}
} else {
let d0 = input[0]
for i in 0..<frames { out[i] = d0[i] }
for c in 1..<channels {
let d = input[c]
for i in 0..<frames { out[i] += d[i] }
}
if channels > 1 { for i in 0..<frames { out[i] = max(-1, min(1, out[i])) } }
}
}
#endif
#if os(macOS)
@@ -387,5 +532,18 @@ public final class SessionAudio {
unit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0,
&dev, UInt32(MemoryLayout<AudioDeviceID>.size)) == noErr
}
/// Read back the AUHAL's live device the definitive "what are we actually capturing
/// from", which catches a selection that succeeded on paper but silently fell back to
/// the system default (a stale/changed UID, a device that vanished between resolve and
/// start). 0 / an error means we couldn't tell.
private static func currentDevice(of unit: AudioUnit) -> AudioDeviceID? {
var dev = AudioDeviceID(0)
var size = UInt32(MemoryLayout<AudioDeviceID>.size)
let status = AudioUnitGetProperty(
unit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, &dev, &size)
guard status == noErr, dev != 0 else { return nil }
return dev
}
#endif
}
@@ -35,6 +35,10 @@ public struct AccessUnit: Sendable {
public let ptsNs: UInt64
public let frameIndex: UInt32
public let flags: UInt32
/// Client `CLOCK_REALTIME` instant the AU was handed over by the core (post-FEC, decrypted)
/// the **received** measurement point of design/stats-unification.md. The decode stage is
/// `decodedNs - receivedNs`, both client-local (no skew offset applies).
public let receivedNs: Int64
}
/// One Opus audio packet (48 kHz stereo, 5 ms frames) decode with AVAudioConverter
@@ -79,6 +83,9 @@ public final class PunktfunkConnection {
/// Same role for the feedback drain thread (rumble + HID-output two core planes,
/// drained sequentially by one thread).
private let feedbackLock = NSLock()
/// Same role for the host-timing (0xCF) puller its own plane in the core, drained
/// non-blockingly by the app's 1 s stats tick (never contends with the blocking pullers).
private let statsLock = NSLock()
/// Negotiated session mode (host-confirmed).
public private(set) var width: UInt32 = 0
@@ -419,9 +426,13 @@ public final class PunktfunkConnection {
case statusOK:
guard let base = frame.data, frame.len > 0 else { return nil }
let data = Data(bytes: base, count: Int(frame.len)) // copy: ptr valid only until next call
var ts = timespec()
clock_gettime(CLOCK_REALTIME, &ts)
let receivedNs = Int64(ts.tv_sec) * 1_000_000_000 + Int64(ts.tv_nsec)
return AccessUnit(
data: data, ptsNs: frame.pts_ns,
frameIndex: frame.frame_index, flags: frame.flags)
frameIndex: frame.frame_index, flags: frame.flags,
receivedNs: receivedNs)
case statusNoFrame:
return nil
case statusClosed:
@@ -657,6 +668,40 @@ public final class PunktfunkConnection {
}
}
/// One per-AU host-timing report (0xCF): the host's capturefully-sent duration for the
/// access unit whose `AccessUnit.ptsNs` equals `ptsNs` exactly. The stats consumer derives
/// `network = (receivedNs + clockOffsetNs ptsNs) hostUs` the host/network split of the
/// HUD's `host+network` stage (design/stats-unification.md Phase 2).
public struct HostTiming: Sendable, Equatable {
/// The AU's capture stamp (host capture clock matches the AU's `ptsNs`).
public let ptsNs: UInt64
/// Host capturesent duration, µs.
public let hostUs: UInt32
}
/// Pull the next per-AU host timing; nil on timeout, throws `.closed` once the session
/// ended. Best-effort plane: an older host never emits any keep showing the combined
/// `host+network` stage then. Drain non-blockingly (`timeoutMs: 0`) from ONE stats
/// consumer (its own core plane, safe alongside the other pullers).
public func nextHostTiming(timeoutMs: UInt32 = 0) throws -> HostTiming? {
statsLock.lock()
defer { statsLock.unlock() }
guard let h = liveHandle() else { throw PunktfunkClientError.closed }
var out = PunktfunkHostTiming()
let rc = punktfunk_connection_next_host_timing(h, &out, timeoutMs)
switch rc {
case statusOK:
return HostTiming(ptsNs: out.pts_ns, hostUs: out.host_us)
case statusNoFrame:
return nil
case statusClosed:
throw PunktfunkClientError.closed
default:
throw PunktfunkClientError.status(rc)
}
}
/// Send one input event (delivered to the host as a QUIC datagram). Thread-safe;
/// silently dropped after close.
public func send(_ event: PunktfunkInputEvent) {
@@ -676,10 +721,12 @@ public final class PunktfunkConnection {
pumpLock.lock() // pullers exit at their next poll boundary, releasing these
audioLock.lock()
feedbackLock.lock()
statsLock.lock()
abiLock.lock()
let h = handle
handle = nil
abiLock.unlock()
statsLock.unlock()
feedbackLock.unlock()
audioLock.unlock()
pumpLock.unlock()
@@ -24,6 +24,12 @@ public enum DefaultsKey {
public static let micEnabled = "punktfunk.micEnabled"
public static let speakerUID = "punktfunk.speakerUID"
public static let micUID = "punktfunk.micUID"
/// macOS: which input channel of the chosen mic device feeds the host. 0 = "Auto" (sum every
/// channel to mono a mic on a single input of a multi-channel interface passes at full
/// level); n1 pins 1-based input channel n. Multi-channel interfaces expose the mic on ONE
/// discrete channel, and the default Nstereo downmix grabs channels 0/1 (silence when the mic
/// is higher up), so we fold to mono ourselves. Only meaningful for multi-channel devices.
public static let micChannel = "punktfunk.micChannel"
public static let presenter = "punktfunk.presenter"
/// Request a 10-bit BT.2020 PQ (HDR10) stream. On by default; only takes effect when the host
/// has HDR content AND this display supports HDR otherwise the stream stays 8-bit SDR.
@@ -0,0 +1,88 @@
// Splits the unified stats model's `host+network` stage (capturereceived) into its `host`
// (capturefully-sent, reported per AU by the host on the 0xCF plane) and `network`
// (the remainder) terms design/stats-unification.md Phase 2.
//
// Receipt samples are recorded per frame from the pump path; host timings are matched to them
// by exact pts (the 0xCF datagram carries the AU's own `pts_ns`). Best-effort by construction:
// a lost 0xCF datagram, an FEC-dropped AU, or an old host that never emits the plane simply
// contributes no split sample the HUD then keeps the combined `host+network` line. NSLock
// rather than an actor the receipt writer is the non-async pump path (same pattern as
// LatencyMeter/FrameMeter).
import Foundation
/// Per-frame `host` / `network` sampler: `recordReceipt` at AU receipt (pts + the combined
/// capturereceived interval), `noteHostTiming` per drained 0xCF report, `drain` the window's
/// p50s once a second. The pending ring is bounded (drop-oldest) so an old host receipts
/// forever, timings never costs a fixed ~4 KB, not growth.
public final class HostNetworkSplitter: @unchecked Sendable {
private let lock = NSLock()
/// Received AUs awaiting their 0xCF host timing: (pts, combined capturereceived µs).
private var pending: [(ptsNs: UInt64, combinedUs: Int64)] = []
private var hostUsSamples: [Int64] = []
private var networkUsSamples: [Int64] = []
/// ~1 s of frames at 240 fps; beyond it the oldest receipt can no longer expect a match.
private static let pendingCap = 256
public init() {}
/// Record one frame at receipt. `ptsNs` is the host capture clock (the AU's pts),
/// `receivedNs` the client `CLOCK_REALTIME` receipt instant (`AccessUnit.receivedNs`),
/// `offsetNs` the connect-time hostclient clock offset (0 = uncorrected). Same
/// absurd-value clamp as LatencyMeter a sample it would drop must not linger here.
public func recordReceipt(ptsNs: UInt64, receivedNs: Int64, offsetNs: Int64) {
let combinedNs = receivedNs &+ offsetNs &- Int64(bitPattern: ptsNs)
guard combinedNs > 0, combinedNs < 10_000_000_000 else { return }
lock.lock()
pending.append((ptsNs: ptsNs, combinedUs: combinedNs / 1000))
if pending.count > Self.pendingCap {
pending.removeFirst(pending.count - Self.pendingCap)
}
lock.unlock()
}
/// Match one host timing (0xCF) to its receipt: `host` = the reported capturesent,
/// `network` = the combined interval minus it, floored at 0 (the terms tile per frame; a
/// slightly-off skew offset must not produce a negative wire time). Unmatched timings
/// the AU was FEC-dropped, or its receipt raced this drain are simply skipped.
public func noteHostTiming(ptsNs: UInt64, hostUs: UInt32) {
lock.lock()
defer { lock.unlock() }
guard let i = pending.firstIndex(where: { $0.ptsNs == ptsNs }) else { return }
let combinedUs = pending.remove(at: i).combinedUs
hostUsSamples.append(Int64(hostUs))
networkUsSamples.append(max(0, combinedUs - Int64(hostUs)))
}
public struct Split: Sendable {
public let hostP50Ms: Double
public let networkP50Ms: Double
public let count: Int
}
/// The window's p50s since the last drain, then reset (matched samples only; the pending
/// ring survives a receipt may still match a timing drained next tick). `nil` when no
/// timing matched in the interval the caller falls back to the combined stage.
public func drain() -> Split? {
lock.lock()
let host = hostUsSamples.sorted()
let network = networkUsSamples.sorted()
hostUsSamples.removeAll(keepingCapacity: true)
networkUsSamples.removeAll(keepingCapacity: true)
lock.unlock()
guard !host.isEmpty else { return nil }
func p50(_ sorted: [Int64]) -> Double {
Double(sorted[min(sorted.count / 2, sorted.count - 1)]) / 1000.0 // µs ms
}
return Split(hostP50Ms: p50(host), networkP50Ms: p50(network), count: host.count)
}
/// Forget everything (pending receipts + window) a fresh connection starts clean.
public func reset() {
lock.lock()
pending.removeAll()
hostUsSamples.removeAll()
networkUsSamples.removeAll()
lock.unlock()
}
}
@@ -1,23 +1,25 @@
// Per-frame latency sampler for the live HUD: records capture->client-receipt latency and drains
// percentiles on demand. NSLock rather than an actor the writer is the non-async pump/arrival
// path (same pattern as the app's FrameMeter).
// Per-frame latency-stage sampler for the live HUD: records one interval per frame (an end
// instant minus a start instant, both CLOCK_REALTIME ns) and drains percentiles on demand.
// NSLock rather than an actor the writers are the non-async pump/decode/present paths (same
// pattern as the app's FrameMeter).
import Foundation
/// Samples the **capture->client-receipt** latency of each access unit and reports percentiles.
/// Samples one **latency stage** per frame and reports percentiles. One instance per stage of the
/// unified stats model (design/stats-unification.md):
///
/// The latency is `now - pts_ns`, where `pts_ns` is the host's capture wall clock (the AU's pts) and
/// `now` is the client's `CLOCK_REALTIME` instant the AU was received, shifted by the connect-time
/// **clock-skew offset** (`PunktfunkConnection.clockOffsetNs`, host minus client) so the difference
/// is valid across machines. `offsetNs == 0` means an old host that didn't answer the skew handshake
/// (or genuinely synced clocks) the number is then only meaningful same-host.
/// - `host+network` = capturereceived: `record(ptsNs:offsetNs:)` at AU receipt.
/// - `decode` = receiveddecoded and `display` = decodeddisplayed: client-local single-clock
/// stages `record(ptsNs:atNs:offsetNs:)` with the start instant as `ptsNs` and `offsetNs: 0`.
/// - `end-to-end` = capturedisplayed, measured directly (never summed from the stages):
/// `record(ptsNs:atNs:offsetNs:)` at present.
///
/// SCOPE (stage-1 presenter): this covers host capture -> encode -> FEC -> network -> reassembly ->
/// decrypt -> handed to the presenter. It does **not** include the on-device VideoToolbox decode or
/// the `AVSampleBufferDisplayLayer` present that layer decodes and presents compressed samples
/// internally with no per-frame callback. True decode->present (the full glass-to-glass) needs the
/// stage-2 presenter (`VTDecompressionSession` decode-completion + `CAMetalLayer`/display-link
/// present); this meter is the substrate it will extend.
/// For the host-anchored intervals (capture) the sample is `end + offset - pts_ns`, where
/// `pts_ns` is the host's capture wall clock (the AU's pts) and the connect-time **clock-skew
/// offset** (`PunktfunkConnection.clockOffsetNs`, host minus client) makes the difference valid
/// across machines. `offsetNs == 0` means an old host that didn't answer the skew handshake (or
/// genuinely synced clocks) the number is then only meaningful same-host, and the HUD tags the
/// end-to-end line `(same-host clock)`.
public final class LatencyMeter: @unchecked Sendable {
private let lock = NSLock()
private var samplesUs: [Int64] = []
@@ -34,12 +36,16 @@ public final class LatencyMeter: @unchecked Sendable {
record(ptsNs: ptsNs, atNs: nowNs, offsetNs: offsetNs)
}
/// Record one frame whose latency is `atNs + offsetNs - ptsNs` an EXPLICIT client instant
/// rather than now. The stage-2 presenter uses this to stamp capturepresent at the display
/// link's target present time (not the moment the present call ran). All in `CLOCK_REALTIME`.
/// Record one frame whose sample is `atNs + offsetNs - ptsNs` an EXPLICIT end instant
/// rather than now. `ptsNs` is the stage's start point: the AU pts for the host-anchored
/// intervals, or a client stamp (receivedNs / decodedNs, with `offsetNs: 0`) for the local
/// decode/display stages. The stage-2 presenter stamps its present-side samples at the
/// display link's target present time (not the moment the present call ran). All in
/// `CLOCK_REALTIME`.
public func record(ptsNs: UInt64, atNs: Int64, offsetNs: Int64) {
let latNs = atNs &+ offsetNs &- Int64(bitPattern: ptsNs)
// Drop absurd values (a clock step, a wildly wrong offset, or garbage pts).
// Drop absurd values (a clock step, a wildly wrong offset, garbage pts, or a stage whose
// start stamp is missing/after its end) samples are clamped to (0, 10 s).
guard latNs > 0, latNs < 10_000_000_000 else { return }
lock.lock()
samplesUs.append(latNs / 1000)
@@ -38,8 +38,9 @@ final class SessionPresenter {
func start(
connection: PunktfunkConnection,
baseLayer: AVSampleBufferDisplayLayer,
presentMeter: LatencyMeter?,
presentTailMeter: LatencyMeter? = nil,
endToEndMeter: LatencyMeter?,
decodeMeter: LatencyMeter? = nil,
displayMeter: LatencyMeter? = nil,
makeDisplayLink: (AnyObject, Selector) -> CADisplayLink,
onFrame: (@Sendable (AccessUnit) -> Void)?,
onSessionEnd: (@Sendable () -> Void)?
@@ -59,7 +60,8 @@ final class SessionPresenter {
#endif
if !forceStage1,
let pipeline = Stage2Pipeline(
presentMeter: presentMeter, presentTailMeter: presentTailMeter) {
endToEndMeter: endToEndMeter, decodeMeter: decodeMeter,
displayMeter: displayMeter) {
let metal = pipeline.layer
// The opaque metal layer composites OVER the AVSampleBufferDisplayLayer base, which
// sits idle (un-enqueued) in stage-2. contentsScale + frame are set in layout().
@@ -1,7 +1,8 @@
// Stage-2 presenter orchestrator: a pump thread pulls AUs VideoDecoder; the decoder's async output
// drops the newest decoded frame into a 1-slot ring; the hosting view's display link calls `renderTick`
// once per vsync to draw + present the newest ready frame and stamp capturepresent. Mirrors
// StreamPump's lifecycle (one per start; cancel is permanent).
// once per vsync to draw + present the newest ready frame and stamp the unified latency stages
// (end-to-end captureon-glass, plus the decode and display stage terms
// design/stats-unification.md). Mirrors StreamPump's lifecycle (one per start; cancel is permanent).
//
// Threading: the pump runs on its own thread; the decoder callback on a VT thread; `renderTick` +
// `start`/`stop` on the MAIN thread (the view's CADisplayLink fires there). Only the ring (lock-guarded)
@@ -40,8 +41,8 @@ public final class Stage2Pipeline {
private let ring = ReadyRing()
private let presenter: MetalVideoPresenter
private let decoder: VideoDecoder
private let presentMeter: LatencyMeter?
private let presentTailMeter: LatencyMeter?
private let endToEndMeter: LatencyMeter?
private let displayMeter: LatencyMeter?
private let recovery = KeyframeRecovery()
private var token = StopFlag()
private var offsetNs: Int64 = 0
@@ -56,28 +57,41 @@ public final class Stage2Pipeline {
/// The Metal layer the hosting view installs + sizes.
public var layer: CAMetalLayer { presenter.layer }
/// `presentMeter` records capturepresent (the glass-to-glass term); `presentTailMeter`
/// records decode-completionpresent (the ring wait + render the tail stage-2 exists to
/// shorten). Both optional: metering never gates the presenter choice. Returns nil if Metal
/// can't be set up (headless / no GPU) caller falls back to the stage-1 presenter.
public init?(presentMeter: LatencyMeter?, presentTailMeter: LatencyMeter? = nil) {
/// Unified-stats meters (design/stats-unification.md): `endToEndMeter` records the headline
/// end-to-end (captureon-glass, skew-corrected); `decodeMeter` the decode stage
/// (receiveddecoded); `displayMeter` the display stage (decodedon-glass, the ring wait +
/// render + vsync the tail stage-2 exists to shorten). All optional: metering never gates
/// the presenter choice. Returns nil if Metal can't be set up (headless / no GPU) caller
/// falls back to the stage-1 presenter.
public init?(
endToEndMeter: LatencyMeter?,
decodeMeter: LatencyMeter? = nil,
displayMeter: LatencyMeter? = nil
) {
guard let presenter = MetalVideoPresenter.make() else { return nil }
self.presenter = presenter
self.presentMeter = presentMeter
self.presentTailMeter = presentTailMeter
self.endToEndMeter = endToEndMeter
self.displayMeter = displayMeter
let ring = ring
let recovery = recovery
self.decoder = VideoDecoder(
onDecoded: { ring.submit($0) },
onDecoded: { frame in
// Decode stage = receiveddecoded, both client CLOCK_REALTIME (offset 0 no
// skew applies). Stamped at decode completion, so it covers every decoded frame,
// including ones the newest-wins ring drops before present.
decodeMeter?.record(
ptsNs: UInt64(frame.receivedNs), atNs: frame.decodedNs, offsetNs: 0)
ring.submit(frame)
},
// Async decode failure (a bad P-frame referencing a lost/corrupt IDR): the pump resets to
// re-gate on the next IDR, and we ask the host to send one now (infinite GOP it wouldn't
// otherwise come soon). Throttled in KeyframeRecovery.
onDecodeError: { _ in recovery.request() })
}
/// Start pulling AUs into the decoder. MAIN THREAD. `onFrame` fires per AU at receipt (captureclient
/// meter, exactly as stage-1); `onSessionEnd` on close. `clockOffsetNs` (host minus client) makes the
/// present stamp cross-machine valid.
/// Start pulling AUs into the decoder. MAIN THREAD. `onFrame` fires per AU at receipt (the
/// host+network / capturereceived meter, exactly as stage-1); `onSessionEnd` on close.
/// `clockOffsetNs` (host minus client) makes the end-to-end stamp cross-machine valid.
public func start(
connection: PunktfunkConnection,
onFrame: (@Sendable (AccessUnit) -> Void)?,
@@ -174,14 +188,16 @@ public final class Stage2Pipeline {
public func renderTick(targetPresentNs: Int64) {
guard let frame = ring.take() else { return }
let offsetNs = offsetNs
let presentMeter = presentMeter
let presentTailMeter = presentTailMeter
let endToEndMeter = endToEndMeter
let displayMeter = displayMeter
let rendered = presenter.render(frame.pixelBuffer, isHDR: frame.isHDR) { presentedNs in
let atNs = presentedNs ?? targetPresentNs
presentMeter?.record(ptsNs: frame.ptsNs, atNs: atNs, offsetNs: offsetNs)
// Present tail = decode-completion on-glass. Both instants are client
// CLOCK_REALTIME, so no skew offset applies.
presentTailMeter?.record(ptsNs: UInt64(frame.decodedNs), atNs: atNs, offsetNs: 0)
// End-to-end = captureon-glass, measured directly (skew-corrected via the
// connect-time clock offset) the HUD headline.
endToEndMeter?.record(ptsNs: frame.ptsNs, atNs: atNs, offsetNs: offsetNs)
// Display stage = decoded on-glass. Both instants are client CLOCK_REALTIME,
// so no skew offset applies.
displayMeter?.record(ptsNs: UInt64(frame.decodedNs), atNs: atNs, offsetNs: 0)
}
if !rendered { ring.putBack(frame) }
}
@@ -61,7 +61,7 @@ public enum Stage444Probe {
guard created == noErr, let session else { return false }
defer { VTDecompressionSessionInvalidate(session) }
let au = AccessUnit(data: data, ptsNs: 0, frameIndex: 0, flags: 0)
let au = AccessUnit(data: data, ptsNs: 0, frameIndex: 0, flags: 0, receivedNs: 0)
guard let sample = AnnexB.sampleBuffer(au: au, format: format, codec: .hevc) else { return false }
var produced: OSType = 0
@@ -15,6 +15,10 @@ import VideoToolbox
public struct ReadyFrame: @unchecked Sendable {
/// Host capture clock (the AU's pts), in nanoseconds.
public let ptsNs: UInt64
/// Client `CLOCK_REALTIME` instant the AU was received (`AccessUnit.receivedNs`, threaded
/// through the decode via the frame refcon), in nanoseconds. 0 when unknown (a caller that
/// didn't stamp receipt) the decode-stage meter then drops the sample via its sanity guard.
public let receivedNs: Int64
/// Client `CLOCK_REALTIME` instant decode completed, in nanoseconds.
public let decodedNs: Int64
/// The decoded image 8-bit NV12 biplanar (SDR) or 10-bit P010 biplanar (HDR), Metal-compatible.
@@ -25,13 +29,16 @@ public struct ReadyFrame: @unchecked Sendable {
}
/// The C output callback can't capture context, so VideoToolbox hands it the refcon we set at
/// session creation a pointer back to the owning `VideoDecoder`.
/// session creation a pointer back to the owning `VideoDecoder`. The per-frame refcon carries
/// the AU's `receivedNs` as a pointer bit pattern (a scalar smuggled through the C void*, never
/// dereferenced) so the decode stage can be computed against decode-completion.
private let decoderOutputCallback: VTDecompressionOutputCallback = {
refcon, _, status, _, imageBuffer, pts, _ in
refcon, frameRefcon, status, _, imageBuffer, pts, _ in
guard let refcon else { return }
let receivedNs = frameRefcon.map { Int64(Int(bitPattern: $0)) } ?? 0
Unmanaged<VideoDecoder>.fromOpaque(refcon)
.takeUnretainedValue()
.handleDecoded(status: status, imageBuffer: imageBuffer, pts: pts)
.handleDecoded(status: status, imageBuffer: imageBuffer, pts: pts, receivedNs: receivedNs)
}
/// Owns a `VTDecompressionSession` rebuilt whenever the format description changes (every IDR /
@@ -112,7 +119,9 @@ public final class VideoDecoder: @unchecked Sendable {
session,
sampleBuffer: sample,
flags: [._EnableAsynchronousDecompression],
frameRefcon: nil,
// The AU's receipt instant rides through as a bit pattern (nil for 0 the output
// callback maps that back to 0); the callback needs it to stamp the decode stage.
frameRefcon: UnsafeMutableRawPointer(bitPattern: Int(au.receivedNs)),
infoFlagsOut: &infoOut)
lock.unlock()
if status != noErr {
@@ -218,8 +227,11 @@ public final class VideoDecoder: @unchecked Sendable {
return true
}
/// VT thread. Stamp decode-completion and enqueue, or report the error.
fileprivate func handleDecoded(status: OSStatus, imageBuffer: CVImageBuffer?, pts: CMTime) {
/// VT thread. Stamp decode-completion and enqueue, or report the error. `receivedNs` is the
/// AU's receipt instant threaded through the frame refcon (0 = unknown).
fileprivate func handleDecoded(
status: OSStatus, imageBuffer: CVImageBuffer?, pts: CMTime, receivedNs: Int64
) {
guard status == noErr, let imageBuffer else {
onDecodeError(status)
return
@@ -242,6 +254,8 @@ public final class VideoDecoder: @unchecked Sendable {
|| fmt == kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange
|| fmt == kCVPixelFormatType_444YpCbCr10BiPlanarFullRange
onDecoded(
ReadyFrame(ptsNs: ptsNs, decodedNs: decodedNs, pixelBuffer: imageBuffer, isHDR: isHDR))
ReadyFrame(
ptsNs: ptsNs, receivedNs: receivedNs, decodedNs: decodedNs,
pixelBuffer: imageBuffer, isHDR: isHDR))
}
}
@@ -85,39 +85,45 @@ public struct StreamView: NSViewRepresentable {
private let onCaptureChange: ((Bool) -> Void)?
private let onFrame: (@Sendable (AccessUnit) -> Void)?
private let onSessionEnd: (@Sendable () -> Void)?
private let presentMeter: LatencyMeter?
private let presentTailMeter: LatencyMeter?
private let endToEndMeter: LatencyMeter?
private let decodeMeter: LatencyMeter?
private let displayMeter: LatencyMeter?
/// `onFrame`/`onSessionEnd` fire on the pump thread hop to the main actor for UI.
/// `captureEnabled: false` disables input capture entirely while UI (e.g. a trust
/// prompt) is layered over the stream; flipping it to true auto-engages capture
/// once. `onCaptureChange` (main thread) reports engage/release drive the HUD's
/// "click to capture" / " releases" hint with it. `presentMeter` records capturepresent
/// and `presentTailMeter` decodepresent when the stage-2 presenter is active.
/// "click to capture" / " releases" hint with it. The meters record the unified latency
/// stages when the stage-2 presenter is active (design/stats-unification.md):
/// `endToEndMeter` captureon-glass, `decodeMeter` receiveddecoded, `displayMeter`
/// decodedon-glass.
public init(
connection: PunktfunkConnection,
captureEnabled: Bool = true,
onCaptureChange: ((Bool) -> Void)? = nil,
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
onSessionEnd: (@Sendable () -> Void)? = nil,
presentMeter: LatencyMeter? = nil,
presentTailMeter: LatencyMeter? = nil
endToEndMeter: LatencyMeter? = nil,
decodeMeter: LatencyMeter? = nil,
displayMeter: LatencyMeter? = nil
) {
self.connection = connection
self.captureEnabled = captureEnabled
self.onCaptureChange = onCaptureChange
self.onFrame = onFrame
self.onSessionEnd = onSessionEnd
self.presentMeter = presentMeter
self.presentTailMeter = presentTailMeter
self.endToEndMeter = endToEndMeter
self.decodeMeter = decodeMeter
self.displayMeter = displayMeter
}
public func makeNSView(context: Context) -> StreamLayerView {
let view = StreamLayerView()
view.onCaptureChange = onCaptureChange
view.captureEnabled = captureEnabled
view.presentMeter = presentMeter
view.presentTailMeter = presentTailMeter
view.endToEndMeter = endToEndMeter
view.decodeMeter = decodeMeter
view.displayMeter = displayMeter
view.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
return view
}
@@ -125,8 +131,9 @@ public struct StreamView: NSViewRepresentable {
public func updateNSView(_ view: StreamLayerView, context: Context) {
view.onCaptureChange = onCaptureChange
view.captureEnabled = captureEnabled
view.presentMeter = presentMeter
view.presentTailMeter = presentTailMeter
view.endToEndMeter = endToEndMeter
view.decodeMeter = decodeMeter
view.displayMeter = displayMeter
// SwiftUI reuses the NSView across state changes repoint the pump only when the
// connection identity actually changed.
if view.connection !== connection {
@@ -141,10 +148,11 @@ public struct StreamView: NSViewRepresentable {
public final class StreamLayerView: NSView {
private let displayLayer = AVSampleBufferDisplayLayer()
/// Record capturepresent / decodepresent when the stage-2 presenter is active.
/// Consulted at start().
var presentMeter: LatencyMeter?
var presentTailMeter: LatencyMeter?
/// Record the unified latency stages (end-to-end / decode / display) when the stage-2
/// presenter is active. Consulted at start().
var endToEndMeter: LatencyMeter?
var decodeMeter: LatencyMeter?
var displayMeter: LatencyMeter?
/// The shared presenter stack: stage-2 (CAMetalLayer sublayer + display link) with the
/// stage-1 StreamPump displayLayer path as the Metal-unavailable / DEBUG fallback.
private let presenter = SessionPresenter()
@@ -571,8 +579,9 @@ public final class StreamLayerView: NSView {
presenter.start(
connection: connection,
baseLayer: displayLayer,
presentMeter: presentMeter,
presentTailMeter: presentTailMeter,
endToEndMeter: endToEndMeter,
decodeMeter: decodeMeter,
displayMeter: displayMeter,
makeDisplayLink: { displayLink(target: $0, selector: $1) },
onFrame: onFrame,
onSessionEnd: onSessionEnd)
@@ -50,8 +50,9 @@ public struct StreamView: UIViewControllerRepresentable {
private let onCaptureChange: ((Bool) -> Void)?
private let onFrame: (@Sendable (AccessUnit) -> Void)?
private let onSessionEnd: (@Sendable () -> Void)?
private let presentMeter: LatencyMeter?
private let presentTailMeter: LatencyMeter?
private let endToEndMeter: LatencyMeter?
private let decodeMeter: LatencyMeter?
private let displayMeter: LatencyMeter?
public init(
connection: PunktfunkConnection,
@@ -59,24 +60,27 @@ public struct StreamView: UIViewControllerRepresentable {
onCaptureChange: ((Bool) -> Void)? = nil,
onFrame: (@Sendable (AccessUnit) -> Void)? = nil,
onSessionEnd: (@Sendable () -> Void)? = nil,
presentMeter: LatencyMeter? = nil,
presentTailMeter: LatencyMeter? = nil
endToEndMeter: LatencyMeter? = nil,
decodeMeter: LatencyMeter? = nil,
displayMeter: LatencyMeter? = nil
) {
self.connection = connection
self.captureEnabled = captureEnabled
self.onCaptureChange = onCaptureChange
self.onFrame = onFrame
self.onSessionEnd = onSessionEnd
self.presentMeter = presentMeter
self.presentTailMeter = presentTailMeter
self.endToEndMeter = endToEndMeter
self.decodeMeter = decodeMeter
self.displayMeter = displayMeter
}
public func makeUIViewController(context: Context) -> StreamViewController {
let controller = StreamViewController()
controller.onCaptureChange = onCaptureChange
controller.captureEnabled = captureEnabled
controller.presentMeter = presentMeter
controller.presentTailMeter = presentTailMeter
controller.endToEndMeter = endToEndMeter
controller.decodeMeter = decodeMeter
controller.displayMeter = displayMeter
controller.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
return controller
}
@@ -84,8 +88,9 @@ public struct StreamView: UIViewControllerRepresentable {
public func updateUIViewController(_ controller: StreamViewController, context: Context) {
controller.onCaptureChange = onCaptureChange
controller.captureEnabled = captureEnabled
controller.presentMeter = presentMeter
controller.presentTailMeter = presentTailMeter
controller.endToEndMeter = endToEndMeter
controller.decodeMeter = decodeMeter
controller.displayMeter = displayMeter
if controller.connection !== connection {
controller.start(connection: connection, onFrame: onFrame, onSessionEnd: onSessionEnd)
}
@@ -101,10 +106,11 @@ public struct StreamView: UIViewControllerRepresentable {
public final class StreamViewController: UIViewController {
public private(set) var connection: PunktfunkConnection?
private var observers: [NSObjectProtocol] = []
/// Record capturepresent / decodepresent when the stage-2 presenter is active.
/// Consulted at start().
var presentMeter: LatencyMeter?
var presentTailMeter: LatencyMeter?
/// Record the unified latency stages (end-to-end / decode / display) when the stage-2
/// presenter is active. Consulted at start().
var endToEndMeter: LatencyMeter?
var decodeMeter: LatencyMeter?
var displayMeter: LatencyMeter?
/// The shared presenter stack: stage-2 (CAMetalLayer sublayer + display link) with the
/// stage-1 StreamPump displayLayer path as the Metal-unavailable / DEBUG fallback.
private let presenter = SessionPresenter()
@@ -285,8 +291,9 @@ public final class StreamViewController: UIViewController {
presenter.start(
connection: connection,
baseLayer: streamView.displayLayer,
presentMeter: presentMeter,
presentTailMeter: presentTailMeter,
endToEndMeter: endToEndMeter,
decodeMeter: decodeMeter,
displayMeter: displayMeter,
makeDisplayLink: { CADisplayLink(target: $0, selector: $1) },
onFrame: onFrame,
onSessionEnd: onSessionEnd)
@@ -0,0 +1,93 @@
// Multi-channel input mono fold (SessionAudio.foldToMono): the fix for a mic on one channel of
// a multi-channel interface. AVAudioConverter's default Nstereo downmix grabs channels 0/1 dead
// silence when the mic sits higher up so we fold ourselves. This pins the fiddly bits (the
// interleaved stride, channel pinning, the sum-clamp) against regressions without needing hardware.
#if !os(tvOS)
import XCTest
@testable import PunktfunkKit
final class AudioChannelFoldTests: XCTestCase {
/// Drive `foldToMono` over channel data expressed as `[[Float]]`, mirroring the two
/// `floatChannelData` layouts:
/// - deinterleaved: each inner array is one channel (all `frames` long).
/// - interleaved: a single inner array already interleaved (c0f0, c1f0, ), with the real
/// channel count passed separately.
private func fold(
_ planes: [[Float]], frames: Int, channels: Int, interleaved: Bool, pinned: Int?
) -> [Float] {
// One C buffer per plane + a table of pointers to them the shape of floatChannelData.
let buffers: [UnsafeMutablePointer<Float>] = planes.map { plane in
let p = UnsafeMutablePointer<Float>.allocate(capacity: plane.count)
for i in 0..<plane.count { p[i] = plane[i] }
return p
}
let table = UnsafeMutablePointer<UnsafeMutablePointer<Float>>.allocate(
capacity: buffers.count)
for (i, b) in buffers.enumerated() { table[i] = b }
let out = UnsafeMutablePointer<Float>.allocate(capacity: frames)
defer {
buffers.forEach { $0.deallocate() }
table.deallocate()
out.deallocate()
}
SessionAudio.foldToMono(
input: table, frames: frames, channels: channels,
interleaved: interleaved, pinned: pinned, out: out)
return (0..<frames).map { out[$0] }
}
// A pinned channel is copied verbatim the exact fix: mic on a HIGH channel, not 0/1.
func testPinsHigherChannelDeinterleaved() {
let result = fold(
[[0, 0, 0], [0, 0, 0], [0.1, 0.2, 0.3], [0, 0, 0]],
frames: 3, channels: 4, interleaved: false, pinned: 2)
XCTAssertEqual(result, [0.1, 0.2, 0.3])
}
// Same signal, interleaved layout: [c0f0,c1f0,c2f0,c3f0, c0f1,]. Guards the `i*ch + c` stride.
func testPinsHigherChannelInterleaved() {
let interleaved: [Float] = [
0, 0, 0.1, 0,
0, 0, 0.2, 0,
0, 0, 0.3, 0,
]
let result = fold([interleaved], frames: 3, channels: 4, interleaved: true, pinned: 2)
XCTAssertEqual(result, [0.1, 0.2, 0.3])
}
// Auto (pinned: nil): a lone hot channel amid silence passes at FULL level, never attenuated.
func testAutoSumsAllChannelsSoALoneMicSurvives() {
let result = fold(
[[0, 0], [0.4, -0.4], [0, 0]],
frames: 2, channels: 3, interleaved: false, pinned: nil)
XCTAssertEqual(result, [0.4, -0.4])
}
// Two simultaneously-hot channels sum past the unit range clamped, never wraps/overflows.
func testAutoSumClampsToUnitRange() {
let result = fold(
[[0.8, -0.8], [0.9, -0.9]],
frames: 2, channels: 2, interleaved: false, pinned: nil)
XCTAssertEqual(result, [1.0, -1.0])
}
// A plain mono device is passed through untouched (no clamp, no attenuation).
func testMonoIsIdentity() {
let result = fold(
[[0.25, -0.5, 0.75]], frames: 3, channels: 1, interleaved: false, pinned: nil)
XCTAssertEqual(result, [0.25, -0.5, 0.75])
}
// Belt-and-suspenders: an out-of-range pin (the tap already guards, but the setting is
// persisted) is ignored by foldToMono's own `ch < channels` guard, which sums instead of
// reading past the buffer.
func testOutOfRangePinFallsBackToSum() {
let result = fold(
[[0, 0], [0.3, 0.3]],
frames: 2, channels: 2, interleaved: false, pinned: 2)
XCTAssertEqual(result, [0.3, 0.3])
}
}
#endif
@@ -0,0 +1,107 @@
// Unit tests for HostNetworkSplitter (the host/network split of the unified stats model's
// host+network stage design/stats-unification.md Phase 2): pts matching, the per-frame
// tiling arithmetic (network = combined host, floored at 0), drain/reset semantics, the
// bounded pending ring, and the absurd-receipt clamp. All samples use explicit instants, so
// the expectations are exact.
import Foundation
import XCTest
@testable import PunktfunkKit
final class HostNetworkSplitterTests: XCTestCase {
/// An arbitrary host-capture pts (ns) far from zero, like a real CLOCK_REALTIME stamp.
private let basePts: UInt64 = 1_000_000_000_000
private func receipt(_ s: HostNetworkSplitter, pts: UInt64, combinedMs: Int64,
offsetNs: Int64 = 0) {
s.recordReceipt(
ptsNs: pts, receivedNs: Int64(pts) + combinedMs * 1_000_000 - offsetNs,
offsetNs: offsetNs)
}
func testEmptyDrainIsNil() {
XCTAssertNil(HostNetworkSplitter().drain())
}
func testMatchSplitsCombinedIntoHostAndNetwork() {
let s = HostNetworkSplitter()
receipt(s, pts: basePts, combinedMs: 8) // capturereceived 8 ms
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000) // host says 3 ms of it was its own
guard let split = s.drain() else { return XCTFail("expected a matched sample") }
XCTAssertEqual(split.count, 1)
XCTAssertEqual(split.hostP50Ms, 3.0)
XCTAssertEqual(split.networkP50Ms, 5.0, "the two terms tile the combined interval")
XCTAssertNil(s.drain(), "drain resets the window")
}
func testSkewOffsetAppliesToTheCombinedInterval() {
let s = HostNetworkSplitter()
// Client clock 2 ms behind the host: the raw difference alone would read 6 ms.
receipt(s, pts: basePts, combinedMs: 8, offsetNs: 2_000_000)
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000)
XCTAssertEqual(s.drain()?.networkP50Ms, 5.0)
}
func testUnmatchedTimingIsSkipped() {
let s = HostNetworkSplitter()
receipt(s, pts: basePts, combinedMs: 8)
// A timing for an AU we never received (FEC-dropped) must not fabricate a sample.
s.noteHostTiming(ptsNs: basePts + 1, hostUs: 3_000)
XCTAssertNil(s.drain())
}
func testReceiptSurvivesADrainUntilItsTimingArrives() {
let s = HostNetworkSplitter()
receipt(s, pts: basePts, combinedMs: 8)
XCTAssertNil(s.drain(), "no timing matched yet")
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000) // arrives one tick late still matches
XCTAssertEqual(s.drain()?.hostP50Ms, 3.0)
}
func testEachReceiptMatchesOnce() {
let s = HostNetworkSplitter()
receipt(s, pts: basePts, combinedMs: 8)
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000)
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000) // duplicate 0xCF no second sample
XCTAssertEqual(s.drain()?.count, 1)
}
func testNetworkFlooredAtZero() {
let s = HostNetworkSplitter()
// A slightly-off skew offset can make host_us exceed the combined interval.
receipt(s, pts: basePts, combinedMs: 2)
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000)
guard let split = s.drain() else { return XCTFail("expected a sample") }
XCTAssertEqual(split.hostP50Ms, 3.0)
XCTAssertEqual(split.networkP50Ms, 0.0)
}
func testPendingRingDropsOldest() {
let s = HostNetworkSplitter()
for i in 0..<300 { // cap is 256 the first receipts fall out
receipt(s, pts: basePts + UInt64(i), combinedMs: 8)
}
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000) // evicted no match
XCTAssertNil(s.drain())
s.noteHostTiming(ptsNs: basePts + 299, hostUs: 3_000) // newest still pending
XCTAssertEqual(s.drain()?.count, 1)
}
func testAbsurdReceiptsAreDropped() {
let s = HostNetworkSplitter()
receipt(s, pts: basePts, combinedMs: -1) // received before capture clock step
receipt(s, pts: basePts + 1, combinedMs: 20_000) // > 10 s garbage pts/offset
s.noteHostTiming(ptsNs: basePts, hostUs: 1_000)
s.noteHostTiming(ptsNs: basePts + 1, hostUs: 1_000)
XCTAssertNil(s.drain())
}
func testResetForgetsPendingReceipts() {
let s = HostNetworkSplitter()
receipt(s, pts: basePts, combinedMs: 8)
s.reset()
s.noteHostTiming(ptsNs: basePts, hostUs: 3_000)
XCTAssertNil(s.drain(), "a fresh session must not match a previous session's receipts")
}
}
@@ -1,6 +1,10 @@
// Unit tests for LatencyMeter: percentiles, the skew-corrected flag, reset-on-drain, and the
// absurd-value guard. Latencies are constructed by stamping a pts a known interval in the past, so
// the result is that interval plus the (tiny) clock advance between reads asserted with tolerance.
// Unit tests for LatencyMeter (one instance per unified-stats stage see
// design/stats-unification.md): percentiles, the skew-corrected flag, reset-on-drain, the
// absurd-value guard, and the explicit-instant stage form (record(ptsNs:atNs:offsetNs:), used for
// the client-local decode/display stages and the at-present end-to-end stamp). Receipt-path
// latencies are constructed by stamping a pts a known interval in the past, so the result is that
// interval plus the (tiny) clock advance between reads asserted with tolerance; the explicit
// form is exact.
import Foundation
import XCTest
@@ -38,6 +42,26 @@ final class LatencyMeterTests: XCTestCase {
XCTAssertEqual(m.drain()?.skewCorrected, true)
}
func testExplicitStageRecordIsExact() {
let m = LatencyMeter()
// A client-local stage (decode: receiveddecoded) start instant as ptsNs, offset 0.
let receivedNs: Int64 = 1_000_000_000_000
m.record(ptsNs: UInt64(receivedNs), atNs: receivedNs + 3_000_000, offsetNs: 0)
guard let s = m.drain() else { return XCTFail("expected a sample") }
XCTAssertEqual(s.count, 1)
XCTAssertEqual(s.p50Ms, 3.0, "explicit instants make the sample exact")
XCTAssertFalse(s.skewCorrected, "local stages record with offset 0")
}
func testExplicitStageDropsNonPositiveInterval() {
let m = LatencyMeter()
// A stage whose start stamp is missing (0) or after its end must not pollute the window.
let decodedNs: Int64 = 1_000_000_000_000
m.record(ptsNs: 0, atNs: decodedNs, offsetNs: 0) // "start unknown" > 10 s dropped
m.record(ptsNs: UInt64(decodedNs + 1), atNs: decodedNs, offsetNs: 0) // negative dropped
XCTAssertNil(m.drain())
}
func testDropsAbsurdValues() {
let m = LatencyMeter()
let now = nowRealtimeNs()
@@ -25,12 +25,18 @@ final class LoopbackIntegrationTests: XCTestCase {
XCTAssertEqual(conn.resolvedBitrateKbps, 50_000)
// Pull 25 synthetic frames and byte-verify the documented pattern:
// u32 LE frame index, then data[i] = (idx as u8) &+ (i as u8).
// u32 LE frame index, then data[i] = (idx as u8) &+ (i as u8). Alongside, drain the
// per-AU host-timing plane (0xCF) the way the app's stats tick does the connector
// ORs VIDEO_CAP_HOST_TIMING in unconditionally and the synthetic host stamps one
// report per AU, so the pts correlation must hold end to end through the xcframework.
var got = 0
var lastIndex: UInt32 = 0
var receivedPts = Set<UInt64>()
var timings: [PunktfunkConnection.HostTiming] = []
let deadline = Date().addingTimeInterval(30)
while got < 25 {
XCTAssertLessThan(Date(), deadline, "timed out after \(got) frames")
while let t = try conn.nextHostTiming(timeoutMs: 0) { timings.append(t) }
guard let au = try conn.nextAU(timeoutMs: 2000) else { continue }
let idx = au.data.prefix(4).reversed().reduce(UInt32(0)) { ($0 << 8) | UInt32($1) }
for (i, byte) in au.data.enumerated().dropFirst(4) {
@@ -41,10 +47,22 @@ final class LoopbackIntegrationTests: XCTestCase {
}
}
XCTAssertGreaterThan(au.ptsNs, 0)
receivedPts.insert(au.ptsNs)
lastIndex = idx
got += 1
}
XCTAssertGreaterThanOrEqual(lastIndex, 24)
// Belt-and-braces: the last frame's timing lands just after its AU give it a bounded
// grace drain (the stream keeps running, so this must not loop on fresh timings).
var grace = 0
while grace < 64, !timings.contains(where: { receivedPts.contains($0.ptsNs) }),
let t = try conn.nextHostTiming(timeoutMs: 100) {
timings.append(t)
grace += 1
}
XCTAssertTrue(
timings.contains { receivedPts.contains($0.ptsNs) },
"no 0xCF host timing matched a received AU's pts (got \(timings.count) timings)")
// Input goes the other way (enqueue-only; the host logs the count on close)
// including the touch kinds, gamepad events, the rich-input plane (DualSense
@@ -31,7 +31,7 @@ final class Stage444Tests: XCTestCase {
let data = Data(Probe444Blobs.au444_8bit)
let format = try XCTUnwrap(
AnnexB.formatDescription(fromIDR: data, codec: .hevc), "the 4:4:4 blob must yield a format description")
let au = AccessUnit(data: data, ptsNs: 7_000_000, frameIndex: 0, flags: 0)
let au = AccessUnit(data: data, ptsNs: 7_000_000, frameIndex: 0, flags: 0, receivedNs: 0)
let box = FrameBox()
let done = DispatchSemaphore(value: 0)
@@ -38,7 +38,8 @@ final class VideoToolboxRoundTripTests: XCTestCase {
XCTAssertEqual(AnnexB.avcc(from: annexB, codec: .hevc), avccSample)
// 3) Sample buffer real decoder pixels.
let au = AccessUnit(data: annexB, ptsNs: 1_000_000, frameIndex: 0, flags: 0)
let au = AccessUnit(
data: annexB, ptsNs: 1_000_000, frameIndex: 0, flags: 0, receivedNs: 0)
let sample = try XCTUnwrap(AnnexB.sampleBuffer(au: au, format: rebuilt, codec: .hevc))
var session: VTDecompressionSession?
@@ -67,13 +68,14 @@ final class VideoToolboxRoundTripTests: XCTestCase {
}
/// Stage-2 decode half: the same known IDR through `VideoDecoder` assert its async output
/// callback fires with a CVPixelBuffer of the right dimensions, the pts round-trips, and
/// decode-completion is stamped.
/// callback fires with a CVPixelBuffer of the right dimensions, the pts and the receipt stamp
/// round-trip (the latter rides the frame refcon), and decode-completion is stamped.
func testVideoDecoderAsyncCallbackDeliversPixels() throws {
let (formatDesc, avccSample) = try encodeOneHEVCKeyframe()
let annexB = try annexBAU(formatDesc: formatDesc, avccSample: avccSample)
let format = try XCTUnwrap(AnnexB.formatDescription(fromIDR: annexB, codec: .hevc))
let au = AccessUnit(data: annexB, ptsNs: 42_000_000, frameIndex: 0, flags: 0)
let au = AccessUnit(
data: annexB, ptsNs: 42_000_000, frameIndex: 0, flags: 0, receivedNs: 41_000_000)
let box = FrameBox()
let done = DispatchSemaphore(value: 0)
@@ -100,6 +102,8 @@ final class VideoToolboxRoundTripTests: XCTestCase {
XCTAssertEqual(CVPixelBufferGetWidth(ready.pixelBuffer), width)
XCTAssertEqual(CVPixelBufferGetHeight(ready.pixelBuffer), height)
XCTAssertEqual(ready.ptsNs, 42_000_000, "pts round-trips through the decoder")
XCTAssertEqual(
ready.receivedNs, 41_000_000, "receivedNs round-trips through the frame refcon")
XCTAssertGreaterThan(ready.decodedNs, 0, "decode-completion is stamped")
}
+15 -7
View File
@@ -17,10 +17,16 @@ the panel looks and feels native to Gaming Mode.
fingerprint to cross-check against the host's log).
2. **Pair** — for a host that requires it, a gamepad-navigable PIN keypad runs the SPAKE2 pairing
ceremony headlessly, then remembers the host so future streams connect silently.
3. **Stream** — launches fullscreen via a hidden Steam shortcut so gamescope focuses it.
4. **Settings**resolution / refresh / bitrate / gamepad type / host compositor / mic, written
3. **Stream** — launches fullscreen via a branded "Punktfunk" Steam shortcut so gamescope focuses it.
4. **Games**each host row has a games button that opens its **library picker**: pin titles as
one-tap "Stream <Game>" rows in the QAM (jump straight into e.g. Playnite on the host), or
**"Open library on screen"** to launch the client's controller-driven, console-style library
browser (aurora backdrop + poster coverflow; A plays, B returns to Gaming Mode). Pins survive
plugin reinstalls (stored next to the client's config) and follow a host across IP changes
(matched by certificate fingerprint).
5. **Settings** — resolution / refresh / bitrate / gamepad type / host compositor / mic, written
to the client's config.
5. **About** — plugin version, an explicit "Check for updates" button, the setup-guide link, and
6. **About** — plugin version, an explicit "Check for updates" button, the setup-guide link, and
a force-stop for a wedged stream client.
To leave a stream: the in-client controller chord (**L1 + R1 + Start + Select**), or close the
@@ -67,11 +73,13 @@ restart is required for an out-of-band install to appear.
| `src/index.tsx` | Plugin entry: the QAM panel + route registration. |
| `src/page.tsx` | The `/punktfunk` fullscreen page — Hosts (with per-host details) / Settings / About tabs. |
| `src/settings.tsx` · `src/pair.tsx` | Stream-settings section; the gamepad-navigable PIN-pairing modal. |
| `src/hooks.ts` · `src/boundary.tsx` | Shared discovery/update hooks + actions; the render error boundary. |
| `src/steam.ts` | Steam-shortcut launch (`AddShortcut` / `SetAppLaunchOptions` / `RunGame`) — the focus-correct stream start. The shortcut's exe is `/bin/sh` with the wrapper passed as an argument, so the script never needs an exec bit (Decky's zip extraction drops it and the root-owned plugins dir can't be chmodded by the unprivileged backend). |
| `src/library.tsx` | The per-host game picker (pin/unpin, "Open library on screen") + the pinned-game launch helper. |
| `src/hooks.ts` · `src/boundary.tsx` | Shared discovery/update/pins hooks + actions; the render error boundary. |
| `src/steam.ts` | Steam-shortcut launch (`AddShortcut` / `SetAppLaunchOptions` / `RunGame`) — the focus-correct stream start. The shortcut's exe is `/bin/sh` with the wrapper passed as an argument, so the script never needs an exec bit (Decky's zip extraction drops it and the root-owned plugins dir can't be chmodded by the unprivileged backend). Launch extras ride env-prefix tokens: `PF_LAUNCH=<id>` (pinned game) / `PF_BROWSE=1` + `PF_MGMT=<port>` (on-screen library); ids are validated space/quote-free at pin AND launch time. |
| `src/backend.ts` | Typed `callable` bridges to `main.py`. |
| `bin/punktfunkrun.sh` | The launch wrapper the Steam shortcut runs (so the window is focusable). |
| `main.py` | Backend: `discover` (via `avahi-browse`) / `pair` / settings / `kill_stream` / `check_update` (with an explicit CA-bundle search — Decky's embedded Python has no usable default TLS roots on SteamOS). |
| `bin/punktfunkrun.sh` | The launch wrapper the Steam shortcut runs (so the window is focusable); maps `PF_LAUNCH`/`PF_BROWSE`/`PF_MGMT` to `--launch`/`--browse`/`--mgmt`. An older flatpak ignores the flags harmlessly (plain stream / hosts page). |
| `main.py` | Backend: `discover` (via `avahi-browse`) / `pair` / `library` (headless flatpak `--library`, TSV) / pins store (`decky-pinned.json`) / settings / `kill_stream` / `check_update` (with an explicit CA-bundle search — Decky's embedded Python has no usable default TLS roots on SteamOS). |
| `scripts/test-backend.py` | Stdlib-only checks for the backend's pure parsers (TSV, error classes, avahi TXT) + the pins round trip. |
| `plugin.json` · `update.json` | Decky manifest; CI-baked update channel. |
## Limitations / next steps
Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

+24 -2
View File
@@ -11,11 +11,19 @@
#
# Per-session parameters arrive as environment variables, set as the shortcut's Steam launch
# options by the plugin (SteamClient.Apps.SetAppLaunchOptions), so ONE generic shortcut serves
# every host:
# every host (and every pinned game):
# PF_HOST host[:port] to connect to (required)
# PF_LAUNCH library id to launch on connect (optional, e.g. steam:570 — pinned games)
# PF_BROWSE non-empty = open the gamepad library (optional; --browse instead of --connect)
# PF_MGMT management-API port for --browse (optional; client defaults to 47990)
# PF_APPID flatpak app id (default io.unom.Punktfunk)
# PF_FLATPAK override the flatpak binary path (default: `flatpak` on PATH)
#
# Values are plain tokens (the plugin validates launch ids to space/quote-free ASCII before
# they ever reach Steam launch options). An older flatpak without --launch/--browse ignores
# the unknown flags harmlessly (hand-scanned argv): PF_LAUNCH degrades to the plain desktop
# session, PF_BROWSE to the client's hosts page.
#
# Runs as the `deck` user (Steam launched it), so the --user flatpak install is visible and
# WAYLAND_DISPLAY / XDG_RUNTIME_DIR are already correct for gamescope.
#
@@ -33,9 +41,23 @@ if [ -z "${PF_HOST:-}" ]; then
exit 2
fi
echo "punktfunkrun: streaming $APPID --connect $PF_HOST" >&2
# exec so the flatpak client IS the game process — when it exits, Steam ends the "game" and
# Gaming Mode reclaims focus automatically (no manual refocus needed).
# --fullscreen: present the stream chrome-less and fullscreen (the client also auto-detects the
# Deck/gamescope env, and ignores the flag harmlessly on older builds that predate it).
if [ -n "${PF_BROWSE:-}" ]; then
# The gamepad library launcher: browse the host's games on-screen, A streams one,
# session end returns to the launcher, B quits back to Gaming Mode.
echo "punktfunkrun: library $APPID --browse $PF_HOST" >&2
if [ -n "${PF_MGMT:-}" ]; then
exec "$FLATPAK" run --arch=x86_64 "$APPID" --browse "$PF_HOST" --mgmt "$PF_MGMT" --fullscreen
fi
exec "$FLATPAK" run --arch=x86_64 "$APPID" --browse "$PF_HOST" --fullscreen
fi
if [ -n "${PF_LAUNCH:-}" ]; then
# A pinned game: the id rides the session Hello and the host launches that title.
echo "punktfunkrun: streaming $APPID --connect $PF_HOST --launch $PF_LAUNCH" >&2
exec "$FLATPAK" run --arch=x86_64 "$APPID" --connect "$PF_HOST" --launch "$PF_LAUNCH" --fullscreen
fi
echo "punktfunkrun: streaming $APPID --connect $PF_HOST" >&2
exec "$FLATPAK" run --arch=x86_64 "$APPID" --connect "$PF_HOST" --fullscreen
+306 -14
View File
@@ -12,6 +12,11 @@ The backend's jobs are the things Steam can't do:
* **pair(host, port, pin, name)** — run the SPAKE2 PIN ceremony headlessly via the flatpak
client's ``--pair`` mode, capturing the result. Pairing uses the SAME flatpak (so the same
identity store the stream uses), so once paired the stream connects silently.
* **library(host, mgmt_port, fp)** — fetch a paired host's game library headlessly via the
flatpak client's ``--library`` mode (mTLS with the client's own identity; TSV on stdout),
so the picker UI can offer games to pin.
* **get_pins() / set_pins()** — the pinned-games store (``decky-pinned.json`` next to the
client's config, so pins survive plugin reinstalls), annotated with live pairing state.
* **runner_info()** — the absolute path to the launch wrapper + the flatpak app id, handed to
the frontend so it can create/point the Steam shortcut.
* **get_settings() / set_settings()** — read/write the flatpak client's stream settings JSON
@@ -20,11 +25,12 @@ The backend's jobs are the things Steam can't do:
* **check_update()** — poll the registry's per-channel ``manifest.json`` and report whether a
newer build is available (the frontend then drives Decky's own install RPC to apply it).
The TXT-record keys parsed (``proto`` / ``fp`` / ``pair`` / ``id``) are defined by the host
advert in ``crates/punktfunk-host/src/discovery.rs``.
The TXT-record keys parsed (``proto`` / ``fp`` / ``pair`` / ``id`` / ``mgmt``) are defined by
the host advert in ``crates/punktfunk-host/src/discovery.rs``.
"""
import asyncio
import base64
import json
import os
import shutil
@@ -76,6 +82,46 @@ def _runner_path() -> str:
return str(Path(decky.DECKY_PLUGIN_DIR) / "bin" / "punktfunkrun.sh")
def _pins_path() -> Path:
"""The pinned-games store — plugin-owned, but deliberately in the CLIENT's config dir
(like everything else we persist): the plugins dir is root-owned and wiped on
reinstall, while ``~/.config/punktfunk`` survives both."""
return _client_config_dir() / "decky-pinned.json"
def _parse_library_tsv(stdout: str) -> list[dict]:
"""Parse the flatpak client's ``--library`` output: one ``id\\tstore\\ttitle`` line per
game plus a trailing ``N game(s)`` count line (no tabs — it self-skips here). A title
may itself contain tabs, so split at most twice."""
games: list[dict] = []
for line in stdout.splitlines():
parts = line.split("\t", 2)
if len(parts) == 3:
games.append({"id": parts[0], "store": parts[1], "title": parts[2]})
return games
def _classify_library_error(stderr: str) -> str:
"""Map the client's ``library: <LibraryError Display>`` stderr line to a stable error
code for the UI. Substring-matched against the Display strings in
``clients/linux/src/library.rs`` — a wording change degrades to ``client-error``
(generic copy), never a crash."""
s = stderr.lower()
if "didn't recognize this device" in s:
return "not-paired"
if "pinned fingerprint" in s:
return "pin-mismatch"
if "couldn't reach the host" in s:
return "unreachable"
if "management api returned http" in s:
return "http"
if "display" in s or "gtk" in s:
# A flatpak so old it predates --library falls through to GTK init, which fails
# headless from this backend.
return "client-outdated"
return "client-error"
# ----------------------------------------------------------------------------------------
# Self-update check (no Decky store). The plugin is distributed via "Install Plugin from
# URL" pointing at our Gitea generic registry, so the official store never sees it and
@@ -224,6 +270,71 @@ def _flatpak_env() -> dict:
return env
async def _flatpak_capture(args: list[str], timeout: float = 20.0) -> tuple[int, str]:
"""Run ``flatpak <args>`` with the user-session env, merging stderr into stdout. Returns
``(returncode, output)``; ``(-1, "")`` if the binary is missing or the call errors/times out.
Best-effort by design — every caller here treats a failure as "no update / can't tell"."""
flatpak = _flatpak()
if not flatpak:
return -1, ""
proc = None
try:
proc = await asyncio.create_subprocess_exec(
flatpak, *args,
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
env=_flatpak_env(),
)
out, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout)
rc = proc.returncode if proc.returncode is not None else -1
return rc, (out or b"").decode("utf-8", "replace")
except asyncio.TimeoutError:
decky.logger.warning("flatpak %s timed out", " ".join(args))
if proc:
try:
proc.kill()
except ProcessLookupError:
pass
return -1, ""
except Exception: # noqa: BLE001
decky.logger.exception("flatpak %s failed", " ".join(args))
return -1, ""
def _field_from(text: str, name: str) -> str:
"""Pull ``<name>: value`` out of ``flatpak info`` / ``remote-info`` output (e.g. ``Commit``,
``Origin``)."""
prefix = f"{name}:"
for line in text.splitlines():
s = line.strip()
if s.startswith(prefix):
return s.split(":", 1)[1].strip()
return ""
async def _client_update_state() -> dict:
"""Is a newer commit of the flatpak client available in the remote it tracks? The client is a
**per-user** install (so ``sudo flatpak update``, which is system-scope, never touches it), and
it versions independently of this plugin — so we compare the installed commit against the
remote's here and let the QAM offer a user-scope update. Best-effort; all-``False`` on any error
(not installed, no flatpak, offline)."""
state = {"available": False, "installed": "", "remote": ""}
rc, info = await _flatpak_capture(["info", "--user", APP_ID], timeout=10.0)
if rc != 0:
return state # client not installed as a user app / no flatpak
state["installed"] = _field_from(info, "Commit")
origin = _field_from(info, "Origin")
if not origin:
return state
rc, rinfo = await _flatpak_capture(["remote-info", "--user", origin, APP_ID], timeout=25.0)
if rc != 0:
return state # remote unreachable — treat as "up to date", retry next check
state["remote"] = _field_from(rinfo, "Commit")
state["available"] = bool(
state["installed"] and state["remote"] and state["installed"] != state["remote"]
)
return state
def _split_txt(txt: str) -> list[str]:
"""Split an avahi TXT column into tokens, honouring the ``"key=value"`` quoting."""
tokens: list[str] = []
@@ -273,6 +384,11 @@ def _parse_avahi_browse(stdout: str) -> list[dict]:
if props.get("proto") and not props["proto"].startswith("punktfunk/"):
continue
try:
mgmt = int(props.get("mgmt", ""))
except ValueError:
mgmt = 0 # not advertised (standalone punktfunk1-host) — callers default 47990
entry = {
"name": name,
"host": address,
@@ -280,6 +396,8 @@ def _parse_avahi_browse(stdout: str) -> list[dict]:
"pair": props.get("pair", "optional"),
"fp": props.get("fp", ""),
"proto": props.get("proto", ""),
"id": props.get("id", ""),
"mgmt": mgmt,
}
key = props.get("id") or f"{address}:{port}"
existing = out.get(key)
@@ -371,6 +489,136 @@ class Plugin:
reason = (err.strip().splitlines() or out.strip().splitlines() or ["pairing failed"])[-1]
return {"ok": False, "error": reason}
async def library(self, host: str, mgmt_port: int = 0, fp: str = "") -> dict:
"""Fetch a paired host's game library via the flatpak client's headless
``--library`` mode (the client's own mTLS identity + pinned-fingerprint transport —
no trust logic reimplemented here). ``fp`` is passed through whenever the caller
knows the host's cert fingerprint so an IP change can never degrade the pin to a
TOFU accept. Returns ``{ok, games: [{id, store, title}]}`` or
``{ok: False, error: <code>, detail}`` (codes: ``flatpak-not-found`` / ``timeout`` /
``not-paired`` / ``pin-mismatch`` / ``unreachable`` / ``http`` /
``client-outdated`` / ``client-error``)."""
flatpak = _flatpak()
if not flatpak:
return {"ok": False, "error": "flatpak-not-found", "detail": ""}
target = f"{host}:{int(mgmt_port) or 47990}"
argv = [flatpak, "run", "--arch=x86_64", APP_ID, "--library", target]
if fp:
argv += ["--fp", fp]
decky.logger.info("library: fetching %s", target)
proc = None
try:
# Separate pipes (unlike _flatpak_capture): the TSV comes on stdout, the
# client's one-line error reason on stderr. Cold flatpak start on a Deck can
# take seconds — generous timeout, spinner in the UI.
proc = await asyncio.create_subprocess_exec(
*argv,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
env=_flatpak_env(),
)
stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=45.0)
except asyncio.TimeoutError:
if proc:
try:
proc.kill()
except ProcessLookupError:
pass
return {"ok": False, "error": "timeout", "detail": ""}
except Exception as exc: # noqa: BLE001
decky.logger.exception("library fetch failed to launch")
return {"ok": False, "error": "client-error", "detail": str(exc)}
err = stderr.decode(errors="replace")
if proc.returncode != 0:
detail = (err.strip().splitlines() or ["library fetch failed"])[-1]
code = _classify_library_error(err)
decky.logger.warning("library fetch failed (%s): %s", code, detail)
return {"ok": False, "error": code, "detail": detail}
games = _parse_library_tsv(stdout.decode(errors="replace"))
decky.logger.info("library: %d game(s) from %s", len(games), target)
return {"ok": True, "games": games}
async def get_pins(self) -> dict:
"""The pinned games, each annotated with the LIVE ``paired`` state of its host (by
cert fingerprint — an unpaired-since host renders "pairing required" in the QAM)."""
try:
data = json.loads(_pins_path().read_text())
except (OSError, json.JSONDecodeError):
return {"pins": []}
pins = data.get("pins", []) if isinstance(data, dict) else []
paired = _paired_fingerprints()
out = []
for p in pins:
if not isinstance(p, dict) or not p.get("game_id"):
continue
p = dict(p)
p["paired"] = str(p.get("host_fp", "")).lower() in paired
out.append(p)
return {"pins": out}
async def set_pins(self, pins: list) -> dict:
"""Persist the pinned-games list (the frontend sends the whole list — add, remove,
and address-refresh all funnel through here). Validated + deduped on
``(host_fp, game_id)``; written atomically (tmp + rename) — pins are long-lived
user data."""
clean: list[dict] = []
seen: set[tuple[str, str]] = set()
for p in pins if isinstance(pins, list) else []:
if not isinstance(p, dict):
continue
game_id = str(p.get("game_id", ""))
host_fp = str(p.get("host_fp", ""))
if not game_id or not (host_fp or p.get("host")):
continue
key = (host_fp, game_id)
if key in seen:
continue
seen.add(key)
clean.append({
"game_id": game_id,
"title": str(p.get("title", game_id)),
"store": str(p.get("store", "")),
"host_fp": host_fp,
"host_id": str(p.get("host_id", "")),
"host_name": str(p.get("host_name", p.get("host", ""))),
"host": str(p.get("host", "")),
"port": int(p.get("port", 9777) or 9777),
"mgmt": int(p.get("mgmt", 0) or 0),
"added_at": int(p.get("added_at", 0) or 0),
})
try:
d = _client_config_dir()
d.mkdir(parents=True, exist_ok=True)
tmp = _pins_path().with_suffix(".json.tmp")
tmp.write_text(json.dumps({"version": 1, "pins": clean}, indent=2))
os.replace(tmp, _pins_path())
return {"ok": True}
except OSError as exc:
decky.logger.exception("could not write pins")
return {"ok": False, "error": str(exc)}
async def shortcut_art(self) -> dict:
"""The Steam-shortcut artwork shipped with the plugin (``assets/``, generated by
``scripts/gen-steam-art.py``): base64 PNGs for SetCustomArtworkForApp plus the
icon's absolute path for SetShortcutIcon (which wants a file, not bytes). Missing
files are simply omitted — artwork is cosmetic and must never block a launch."""
art: dict = {}
base = Path(decky.DECKY_PLUGIN_DIR) / "assets"
for key, fname in (
("grid", "grid.png"),
("gridwide", "gridwide.png"),
("hero", "hero.png"),
("logo", "logo.png"),
):
try:
art[key] = base64.b64encode((base / fname).read_bytes()).decode()
except OSError:
pass
icon = base / "icon.png"
art["icon_path"] = str(icon) if icon.exists() else ""
return art
async def runner_info(self) -> dict:
"""The wrapper-script path + flatpak app id the frontend needs to create the Steam
shortcut. The shortcut invokes the script through ``/bin/sh`` (see steam.ts), so no
@@ -419,11 +667,37 @@ class Plugin:
return {"ok": False}
return {"ok": True}
async def update_client(self) -> dict:
"""Update the flatpak **client** (io.unom.Punktfunk) in the USER installation — the scope a
Steam Deck install lives in, which ``sudo flatpak update`` (system-scope) never reaches.
Returns whether a new commit was actually pulled. Best-effort; non-fatal."""
flatpak = _flatpak()
if not flatpak:
return {"ok": False, "updated": False, "error": "flatpak-not-found"}
_, before = await _flatpak_capture(["info", "--user", APP_ID], timeout=10.0)
before_commit = _field_from(before, "Commit")
rc, out = await _flatpak_capture(["update", "--user", "-y", APP_ID], timeout=300.0)
if rc != 0:
decky.logger.warning("flatpak client update failed (rc=%s): %s", rc, out[-400:])
return {"ok": False, "updated": False, "error": "update-failed"}
_, after = await _flatpak_capture(["info", "--user", APP_ID], timeout=10.0)
after_commit = _field_from(after, "Commit")
updated = bool(before_commit and after_commit and before_commit != after_commit)
decky.logger.info(
"flatpak client update: %s -> %s (updated=%s)",
before_commit[:10], after_commit[:10], updated,
)
_update_cache["data"] = None # invalidate the cached "update available" snapshot
return {"ok": True, "updated": updated}
async def check_update(self, force: bool = False) -> dict:
"""Is a newer build available in our registry? Compares the installed version
(``package.json``) against the per-channel ``manifest.json`` the CI publishes, and
returns everything the frontend needs to drive Decky's install RPC. Non-fatal: any
failure (no channel baked in, network down) returns ``update_available: False``.
"""Report pending updates for BOTH the plugin and the flatpak client.
The plugin updates via Decky's install RPC (the per-channel ``manifest.json`` the CI
publishes); the **client** updates via ``flatpak update --user`` (a per-user install, so
``sudo flatpak update`` — system-scope — never touches it) and versions independently, so
it's checked here too and applied through :meth:`update_client`. Non-fatal: any failure
leaves the respective ``*_update_available`` ``False``.
"""
current = _installed_version()
cfg = _update_config()
@@ -434,23 +708,37 @@ class Plugin:
"hash": "",
"channel": str(cfg.get("channel", "")),
"update_available": False,
"client_update_available": False,
"client_current": "",
"client_latest": "",
}
manifest_url = cfg.get("manifest")
if not manifest_url:
result["error"] = "update-channel-unknown" # dev / sideloaded build
return result
now = time.monotonic()
cached = _update_cache["data"]
if not force and cached and (now - _update_cache["at"]) < _UPDATE_TTL_S:
return cached
# Client (flatpak) update — checked ALWAYS, even on a dev/sideloaded plugin build.
try:
cu = await _client_update_state()
result["client_update_available"] = bool(cu["available"])
result["client_current"] = (cu["installed"] or "")[:10]
result["client_latest"] = (cu["remote"] or "")[:10]
except Exception: # noqa: BLE001
decky.logger.warning("client update check failed", exc_info=True)
manifest_url = cfg.get("manifest")
if not manifest_url:
result["error"] = "update-channel-unknown" # dev / sideloaded plugin build
_update_cache["at"] = now
_update_cache["data"] = result # the client info is still valid to cache
return result
try:
loop = asyncio.get_running_loop()
manifest = await loop.run_in_executor(None, _fetch_json, manifest_url)
except Exception as exc: # noqa: BLE001
decky.logger.warning("update check failed: %s", exc)
decky.logger.warning("plugin update check failed: %s", exc)
result["error"] = "fetch-failed"
return result # transient — don't cache, retry next open
@@ -461,8 +749,12 @@ class Plugin:
result["update_available"] = bool(result["artifact"]) and (
_semver_tuple(latest) > _semver_tuple(current)
)
if result["update_available"]:
decky.logger.info("update available: %s -> %s (%s)", current, latest, result["channel"])
if result["update_available"] or result["client_update_available"]:
decky.logger.info(
"updates: plugin %s->%s (avail=%s), client->%s (avail=%s)",
current, latest, result["update_available"],
result["client_latest"], result["client_update_available"],
)
_update_cache["at"] = now
_update_cache["data"] = result
return result
+297
View File
@@ -0,0 +1,297 @@
#!/usr/bin/env python3
"""Generate the Steam-shortcut artwork for the Decky plugin (committed, like the tray icons).
The plugin registers a non-Steam shortcut ("Punktfunk") whose grid/hero/logo/icon Steam
would otherwise render as a gray placeholder tile. These assets brand it: the lens mark
(same geometry as scripts/gen-tray-icons.py / web's brand-mark.tsx) over the brand-navy
gradient, plus a monoline "punktfunk" wordmark built from stroke segments ("punktfunk"
needs only p·u·n·k·t·f). The frontend applies them via
SteamClient.Apps.SetCustomArtworkForApp / SetShortcutIcon (src/steam.ts).
Outputs (checked in; re-run only when the brand changes):
clients/decky/assets/grid.png 600 x 900 library capsule (portrait)
clients/decky/assets/gridwide.png 920 x 430 wide capsule (recent games / search)
clients/decky/assets/hero.png 1920 x 620 game-page banner
clients/decky/assets/logo.png transparent overlaid on the hero by Steam
clients/decky/assets/icon.png 256 x 256 list icon (SetShortcutIcon)
Pure stdlib. Unlike the tiny tray icons this rasterizes big surfaces, so edges are
antialiased analytically from signed distances (one sample per pixel) instead of 4x4
supersampling.
"""
import math
import struct
import zlib
from pathlib import Path
HERE = Path(__file__).resolve().parent.parent # clients/decky
OUT = HERE / "assets"
# Brand-mark geometry in its 1000-unit viewbox (identical to gen-tray-icons.py).
R = 194.41
C1 = (403.037, 597.262) # light circle, behind
C2 = (597.8075, 402.8525) # deep circle, in front
BB_MIN = (C1[0] - R, C2[1] - R)
BB_MAX = (C2[0] + R, C1[1] + R)
MARK_CENTER = ((BB_MIN[0] + BB_MAX[0]) / 2, (BB_MIN[1] + BB_MAX[1]) / 2)
MARK_SPAN = BB_MAX[0] - BB_MIN[0]
COL_LIGHT = (0xA7, 0x9F, 0xF8)
COL_DEEP = (0x6C, 0x5B, 0xF3)
COL_HI = (0xD2, 0xC9, 0xFB)
WORD = (0xEF, 0xEC, 0xFD) # wordmark: near-white lavender
BG_TOP = (0x28, 0x1E, 0x46)
BG_BOT = (0x12, 0x0D, 0x22)
# ------------------------------------------------------------------------------------------
# Wordmark: monoline glyphs as polylines in a unit box (y down; x-height top y=0, baseline
# y=1, ascender to -0.5, descender to +1.5). Arcs are sampled into the polylines, so the
# rasterizer only ever measures distance-to-segment; round caps/joins fall out of that.
# ------------------------------------------------------------------------------------------
def _arc(cx, cy, r, a0, a1, n=24):
"""Polyline along a circle arc; degrees, 0 = +x, angles grow clockwise on screen."""
pts = []
for i in range(n + 1):
a = math.radians(a0 + (a1 - a0) * i / n)
pts.append((cx + r * math.cos(a), cy + r * math.sin(a)))
return pts
GLYPHS = {
# letter: (advance, [polyline, ...])
"p": (1.05, [[(0, 0), (0, 1.5)], _arc(0.5, 0.5, 0.5, 0, 360)]),
"u": (1.05, [[(0, 0), (0, 0.5)], _arc(0.5, 0.5, 0.5, 0, 180), [(1, 0), (1, 0.5)]]),
"n": (1.05, [[(0, 0), (0, 1)], _arc(0.5, 0.5, 0.5, 180, 360), [(1, 0.5), (1, 1)]]),
"k": (1.0, [[(0, -0.5), (0, 1)], [(0, 0.62), (0.78, 0)], [(0.30, 0.38), (0.85, 1)]]),
"t": (0.85, [[(0.42, -0.42), (0.42, 1)], [(0, 0), (0.84, 0)]]),
"f": (
0.85,
[[(0.42, 1), (0.42, -0.15)] + _arc(0.75, -0.15, 0.33, 180, 270, 12), [(0, 0), (0.78, 0)]],
),
}
GAP = 0.34 # inter-letter gap, in glyph units
STROKE = 0.26 # stroke thickness, in glyph units
ASCENT, DESCENT = -0.5, 1.5 # glyph-space vertical extent
def word_segments(text):
"""The word's stroke segments [(x1,y1,x2,y2)] in glyph units, plus its unit width."""
segs = []
x = 0.0
for ch in text:
adv, lines = GLYPHS[ch]
for line in lines:
for (x1, y1), (x2, y2) in zip(line, line[1:]):
segs.append((x + x1, y1, x + x2, y2))
x += adv + GAP
return segs, x - GAP
def render_word_alpha(text, unit_px):
"""Coverage (0..255) buffer of the word at `unit_px` pixels per glyph unit."""
segs, width_u = word_segments(text)
half = STROKE / 2 * unit_px
pad = half + 1.5
w = math.ceil(width_u * unit_px + 2 * pad)
h = math.ceil((DESCENT - ASCENT) * unit_px + 2 * pad)
ox, oy = pad, pad - ASCENT * unit_px
px_segs = [(ox + a * unit_px, oy + b * unit_px, ox + c * unit_px, oy + d * unit_px) for a, b, c, d in segs]
# Bucket segments per pixel column range so each pixel tests only nearby strokes.
buf = bytearray(w * h)
for x1, y1, x2, y2 in px_segs:
lo_x = max(0, math.floor(min(x1, x2) - pad))
hi_x = min(w, math.ceil(max(x1, x2) + pad))
lo_y = max(0, math.floor(min(y1, y2) - pad))
hi_y = min(h, math.ceil(max(y1, y2) + pad))
dx, dy = x2 - x1, y2 - y1
len2 = dx * dx + dy * dy
for py in range(lo_y, hi_y):
row = py * w
fy = py + 0.5
for px in range(lo_x, hi_x):
fx = px + 0.5
if len2 > 0:
t = max(0.0, min(1.0, ((fx - x1) * dx + (fy - y1) * dy) / len2))
else:
t = 0.0
d = math.hypot(fx - (x1 + t * dx), fy - (y1 + t * dy))
cov = 0.5 + (half - d)
if cov > 0:
v = min(255, round(min(1.0, cov) * 255))
if v > buf[row + px]:
buf[row + px] = v
return buf, w, h
# ------------------------------------------------------------------------------------------
# Canvas: RGBA bytearray, straight alpha, painted back to front.
# ------------------------------------------------------------------------------------------
class Canvas:
def __init__(self, w, h):
self.w, self.h = w, h
self.buf = bytearray(w * h * 4)
def fill_gradient(self, top, bottom):
for y in range(self.h):
t = y / max(1, self.h - 1)
c = bytes(
(
round(top[0] + (bottom[0] - top[0]) * t),
round(top[1] + (bottom[1] - top[1]) * t),
round(top[2] + (bottom[2] - top[2]) * t),
255,
)
)
self.buf[y * self.w * 4 : (y + 1) * self.w * 4] = c * self.w
def _blend(self, i, rgb, a):
"""`rgb` over the pixel at byte offset i with coverage a (0..1)."""
if a <= 0:
return
b = self.buf
ia = 1.0 - a
da = b[i + 3] / 255.0
oa = a + da * ia
if oa <= 0:
return
for k in range(3):
b[i + k] = round((rgb[k] * a + b[i + k] * da * ia) / oa)
b[i + 3] = round(oa * 255)
def glow(self, cx, cy, radius, rgb, strength):
"""Soft gaussian-ish radial glow (for the mark's halo on the big surfaces)."""
lo_x = max(0, math.floor(cx - 2.2 * radius))
hi_x = min(self.w, math.ceil(cx + 2.2 * radius))
lo_y = max(0, math.floor(cy - 2.2 * radius))
hi_y = min(self.h, math.ceil(cy + 2.2 * radius))
for y in range(lo_y, hi_y):
for x in range(lo_x, hi_x):
d2 = ((x + 0.5 - cx) ** 2 + (y + 0.5 - cy) ** 2) / (radius * radius)
a = strength * math.exp(-2.5 * d2)
if a > 1 / 255:
self._blend((y * self.w + x) * 4, rgb, a)
def mark(self, cx, cy, span):
"""The lens mark centered at (cx, cy) with the given pixel span."""
scale = span / MARK_SPAN
c1 = (cx + (C1[0] - MARK_CENTER[0]) * scale, cy + (C1[1] - MARK_CENTER[1]) * scale)
c2 = (cx + (C2[0] - MARK_CENTER[0]) * scale, cy + (C2[1] - MARK_CENTER[1]) * scale)
r = R * scale
lo_x = max(0, math.floor(min(c1[0], c2[0]) - r - 2))
hi_x = min(self.w, math.ceil(max(c1[0], c2[0]) + r + 2))
lo_y = max(0, math.floor(min(c1[1], c2[1]) - r - 2))
hi_y = min(self.h, math.ceil(max(c1[1], c2[1]) + r + 2))
for y in range(lo_y, hi_y):
for x in range(lo_x, hi_x):
fx, fy = x + 0.5, y + 0.5
cov1 = min(1.0, max(0.0, 0.5 + r - math.hypot(fx - c1[0], fy - c1[1])))
cov2 = min(1.0, max(0.0, 0.5 + r - math.hypot(fx - c2[0], fy - c2[1])))
if cov1 <= 0 and cov2 <= 0:
continue
i = (y * self.w + x) * 4
self._blend(i, COL_LIGHT, cov1)
self._blend(i, COL_DEEP, cov2)
self._blend(i, COL_HI, min(cov1, cov2))
def word(self, text, unit_px, cx, cy):
"""The wordmark centered at (cx, cy); `unit_px` = pixels per glyph unit."""
alpha, w, h = render_word_alpha(text, unit_px)
ox = round(cx - w / 2)
# Optical vertical centering on the x-height band (0..1 in glyph units), not the
# ascender/descender box — the word reads centered that way.
pad = STROKE / 2 * unit_px + 1.5
band_mid = pad - ASCENT * unit_px + 0.5 * unit_px
oy = round(cy - band_mid)
for y in range(h):
ty = y + oy
if not 0 <= ty < self.h:
continue
for x in range(w):
a = alpha[y * w + x]
if a:
tx = x + ox
if 0 <= tx < self.w:
self._blend((ty * self.w + tx) * 4, WORD, a / 255.0)
def round_corners(self, radius):
"""Multiply alpha with a rounded-rect mask (icon)."""
for y in range(self.h):
for x in range(self.w):
dx = max(0.0, max(radius - (x + 0.5), (x + 0.5) - (self.w - radius)))
dy = max(0.0, max(radius - (y + 0.5), (y + 0.5) - (self.h - radius)))
if dx > 0 and dy > 0:
cov = min(1.0, max(0.0, 0.5 + radius - math.hypot(dx, dy)))
i = (y * self.w + x) * 4
self.buf[i + 3] = round(self.buf[i + 3] * cov)
def png(self):
def chunk(tag, data):
return (
struct.pack(">I", len(data))
+ tag
+ data
+ struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)
)
ihdr = struct.pack(">IIBBBBB", self.w, self.h, 8, 6, 0, 0, 0)
raw = b"".join(
b"\x00" + bytes(self.buf[y * self.w * 4 : (y + 1) * self.w * 4]) for y in range(self.h)
)
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def save(name, canvas):
OUT.mkdir(parents=True, exist_ok=True)
out = OUT / name
out.write_bytes(canvas.png())
print(f"wrote {out.relative_to(HERE.parent.parent)} ({canvas.w}x{canvas.h})")
def main():
# Portrait capsule: mark in the upper half, wordmark beneath.
c = Canvas(600, 900)
c.fill_gradient(BG_TOP, BG_BOT)
c.glow(300, 340, 260, COL_DEEP, 0.35)
c.mark(300, 340, 320)
c.word("punktfunk", 44, 300, 640)
save("grid.png", c)
# Wide capsule: mark left, wordmark right of it.
c = Canvas(920, 430)
c.fill_gradient(BG_TOP, BG_BOT)
c.glow(230, 215, 200, COL_DEEP, 0.35)
c.mark(230, 215, 240)
c.word("punktfunk", 40, 620, 220)
save("gridwide.png", c)
# Hero: ambient banner — the mark rides the right third; Steam overlays logo.png itself.
c = Canvas(1920, 620)
c.fill_gradient(BG_TOP, BG_BOT)
c.glow(1500, 310, 330, COL_DEEP, 0.4)
c.mark(1500, 310, 400)
save("hero.png", c)
# Logo (transparent): mark + wordmark side by side, overlaid on the hero by Steam.
c = Canvas(1120, 300)
c.mark(150, 150, 240)
c.word("punktfunk", 62, 660, 155)
save("logo.png", c)
# Icon: brand tile, rounded corners, mark only.
c = Canvas(256, 256)
c.fill_gradient(BG_TOP, BG_BOT)
c.glow(128, 128, 110, COL_DEEP, 0.3)
c.mark(128, 128, 190)
c.round_corners(36)
save("icon.png", c)
if __name__ == "__main__":
main()
+3 -1
View File
@@ -20,12 +20,14 @@ VER="$(python3 -c 'import json;print(json.load(open("package.json"))["version"])
STAGE="$(mktemp -d)"
DEST="$STAGE/$NAME"
mkdir -p "$DEST/dist" "$DEST/bin"
mkdir -p "$DEST/dist" "$DEST/bin" "$DEST/assets"
cp dist/index.js "$DEST/dist/index.js" # ship the bundle only, not the sourcemap
cp main.py plugin.json package.json LICENSE "$DEST/"
# The stream-launch wrapper (target of the Steam shortcut) — must stay executable.
cp bin/punktfunkrun.sh "$DEST/bin/punktfunkrun.sh"
chmod 0755 "$DEST/bin/punktfunkrun.sh"
# Steam-shortcut artwork (grid/hero/logo/icon — scripts/gen-steam-art.py, committed).
cp assets/*.png "$DEST/assets/"
[ -f decky.pyi ] && cp decky.pyi "$DEST/"
[ -f README.md ] && cp README.md "$DEST/"
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
"""Unit checks for main.py's pure helpers — stdlib only, no Decky runtime needed.
Stubs the ``decky`` module (main.py imports it at module level), then asserts the
avahi/TSV/error parsers against fixture strings. The LibraryError fixtures are pinned to
the REAL Display strings in clients/linux/src/library.rs — if those are reworded, the
classifier degrades to ``client-error`` and the matching assertion here fails on purpose.
python3 clients/decky/scripts/test-backend.py
"""
import sys
import types
from pathlib import Path
# ---- stub the decky module before importing main.py ------------------------------------
decky = types.ModuleType("decky")
decky.DECKY_USER_HOME = "/tmp/pf-test-home"
decky.DECKY_PLUGIN_DIR = "/tmp/pf-test-plugin"
class _Log:
def __getattr__(self, _name):
return lambda *a, **k: None
decky.logger = _Log()
sys.modules["decky"] = decky
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
import main # noqa: E402 (the plugin backend)
failures = 0
def check(name: str, cond: bool):
global failures
print(("ok " if cond else "FAIL") + " " + name)
if not cond:
failures += 1
# ---- _parse_library_tsv -----------------------------------------------------------------
tsv = (
"steam:570\tsteam\tDota 2\n"
"custom:abc\tcustom\tTabs\tin\ttitle\n" # tabs inside the title survive (split max 2)
"2 game(s)\n" # the count trailer has no tabs — self-skips
)
games = main._parse_library_tsv(tsv)
check("tsv: two games parsed", len(games) == 2)
check("tsv: fields", games[0] == {"id": "steam:570", "store": "steam", "title": "Dota 2"})
check("tsv: tabs in title preserved", games[1]["title"] == "Tabs\tin\ttitle")
check("tsv: empty input", main._parse_library_tsv("0 game(s)\n") == [])
# ---- _classify_library_error (fixtures = library.rs Display strings) --------------------
check(
"err: not-paired",
main._classify_library_error(
"library: The host didn't recognize this device. Pair with the host first — the "
"library is authorized by this device's certificate (no token needed)."
)
== "not-paired",
)
check(
"err: pin-mismatch",
main._classify_library_error(
"library: The host's certificate doesn't match the pinned fingerprint. "
"Re-pair with a PIN to re-establish trust."
)
== "pin-mismatch",
)
check(
"err: unreachable",
main._classify_library_error(
"library: Couldn't reach the host's management API: connection refused. Check the "
"host is updated and reachable."
)
== "unreachable",
)
check(
"err: http",
main._classify_library_error("library: The management API returned HTTP 500.") == "http",
)
check(
"err: outdated client (GTK init noise)",
main._classify_library_error("cannot open display: \nGtk-WARNING: init failed")
== "client-outdated",
)
check("err: generic fallback", main._classify_library_error("boom") == "client-error")
# ---- _parse_avahi_browse (incl. the new id/mgmt TXT keys) --------------------------------
avahi = (
"+;eth0;IPv4;living-room;_punktfunk._udp;local\n"
"=;eth0;IPv4;living-room;_punktfunk._udp;local;lr.local;192.168.1.42;9777;"
'"proto=punktfunk/1" "fp=aabbcc" "pair=required" "id=abc123" "mgmt=47990"\n'
"=;eth0;IPv6;living-room;_punktfunk._udp;local;lr.local;fe80::1;9777;"
'"proto=punktfunk/1" "fp=aabbcc" "pair=required" "id=abc123" "mgmt=47990"\n'
"=;eth0;IPv4;bare-host;_punktfunk._udp;local;bh.local;192.168.1.77;9777;"
'"proto=punktfunk/1" "fp=ddeeff" "pair=optional"\n'
)
hosts = main._parse_avahi_browse(avahi)
check("avahi: two hosts (id-dedup, IPv4 preferred)", len(hosts) == 2)
lr = next(h for h in hosts if h["name"] == "living-room")
check("avahi: ipv4 wins", lr["host"] == "192.168.1.42")
check("avahi: mgmt parsed", lr["mgmt"] == 47990)
check("avahi: id parsed", lr["id"] == "abc123")
bare = next(h for h in hosts if h["name"] == "bare-host")
check("avahi: mgmt absent -> 0", bare["mgmt"] == 0)
check("avahi: id absent -> empty", bare["id"] == "")
# ---- pins store (round-trip through the real methods, isolated HOME) --------------------
import asyncio # noqa: E402
import shutil # noqa: E402
shutil.rmtree(decky.DECKY_USER_HOME, ignore_errors=True)
plugin = main.Plugin()
pin = {
"game_id": "steam:570",
"title": "Dota 2",
"store": "steam",
"host_fp": "AABBCC",
"host_id": "abc123",
"host_name": "living-room",
"host": "192.168.1.42",
"port": 9777,
"mgmt": 47990,
"added_at": 1780000000,
}
dupe = dict(pin, title="Dota 2 again")
junk = {"title": "no game id"}
res = asyncio.run(plugin.set_pins([pin, dupe, junk]))
check("pins: write ok", res.get("ok") is True)
got = asyncio.run(plugin.get_pins())["pins"]
check("pins: dedup + junk dropped", len(got) == 1)
check("pins: unpaired without known-hosts", got[0]["paired"] is False)
# Mark the host paired in the client's known-hosts store — get_pins must pick it up.
cfg = main._client_config_dir()
cfg.mkdir(parents=True, exist_ok=True)
(cfg / "client-known-hosts.json").write_text(
'{"hosts": [{"name": "living-room", "addr": "192.168.1.42", "port": 9777, '
'"fp_hex": "aabbcc", "paired": true}]}'
)
got = asyncio.run(plugin.get_pins())["pins"]
check("pins: paired via known-hosts fp (case-insensitive)", got[0]["paired"] is True)
shutil.rmtree(decky.DECKY_USER_HOME, ignore_errors=True)
print()
if failures:
print(f"{failures} check(s) FAILED")
sys.exit(1)
print("all checks passed")
+72 -3
View File
@@ -9,6 +9,43 @@ export interface Host {
fp: string; // host cert SHA-256 fingerprint (lowercase hex) from the mDNS advert
proto: string; // advertised protocol, e.g. "punktfunk/1"
paired: boolean; // whether THIS device has already PIN-paired this host (by fingerprint)
id: string; // the host's stable instance id (mDNS TXT `id`; "" when not advertised)
mgmt: number; // management-API port (mDNS TXT `mgmt`; 0 = not advertised → default 47990)
}
// One title from a host's game library (the flatpak client's --library TSV, parsed by the
// backend). `id` is store-qualified (steam:<appid> / custom:<id>) and doubles as the
// launch handle (PF_LAUNCH → the session Hello).
export interface GameEntry {
id: string;
store: string; // "steam" | "custom" | "heroic" | "lutris" | …
title: string;
}
export interface LibraryResult {
ok: boolean;
games?: GameEntry[];
// "flatpak-not-found" | "timeout" | "not-paired" | "pin-mismatch" | "unreachable" |
// "http" | "client-outdated" | "client-error"
error?: string;
detail?: string; // the client's own one-line reason, for the generic error copy
}
// A pinned game — a one-tap stream row in the QAM. The host is identified primarily by
// cert fingerprint (survives IP changes; pairing is fp-keyed too), with the stored
// address as the launch fallback when the host isn't currently advertising.
export interface PinnedGame {
game_id: string;
title: string;
store: string;
host_fp: string;
host_id: string;
host_name: string;
host: string;
port: number;
mgmt: number;
added_at: number; // unix seconds
paired?: boolean; // annotated by get_pins from the client's known-hosts store
}
export interface PairResult {
@@ -38,24 +75,56 @@ export interface StreamSettings {
}
export interface UpdateInfo {
current: string; // installed version (package.json)
latest: string; // newest version in our registry for this channel
current: string; // installed PLUGIN version (package.json)
latest: string; // newest plugin version in our registry for this channel
artifact: string; // immutable zip URL Decky should install
hash: string; // sha256 of that zip (Decky verifies it)
channel: string; // "latest" (stable) | "canary"
update_available: boolean;
update_available: boolean; // a newer PLUGIN build is available
// The flatpak CLIENT (io.unom.Punktfunk) versions independently and is a per-user install, so
// `sudo flatpak update` never touches it — the plugin offers a user-scope update instead.
client_update_available: boolean;
client_current: string; // installed client commit (short) — informational
client_latest: string; // remote client commit (short) — informational
error?: string; // "update-channel-unknown" (dev build) | "fetch-failed"
}
// Steam-shortcut artwork (assets/ in the plugin dir): base64 PNGs keyed grid / gridwide /
// hero / logo, plus the icon's absolute path (SetShortcutIcon wants a file). Keys for
// missing files are absent.
export interface ShortcutArt {
grid?: string;
gridwide?: string;
hero?: string;
logo?: string;
icon_path: string;
}
export const discover = callable<[], Host[]>("discover");
export const pair = callable<
[host: string, port: number, pin: string, name: string],
PairResult
>("pair");
// Fetch a paired host's game library (headless flatpak --library; can take seconds on a
// cold client start — show a spinner). Pass fp whenever known so the pin can't degrade.
export const library = callable<
[host: string, mgmt_port: number, fp: string],
LibraryResult
>("library");
export const getPins = callable<[], { pins: PinnedGame[] }>("get_pins");
export const setPins = callable<[pins: PinnedGame[]], { ok: boolean; error?: string }>(
"set_pins",
);
export const runnerInfo = callable<[], RunnerInfo>("runner_info");
export const shortcutArt = callable<[], ShortcutArt>("shortcut_art");
export const getSettings = callable<[], StreamSettings>("get_settings");
export const setSettings = callable<[settings: StreamSettings], { ok: boolean }>(
"set_settings",
);
export const killStream = callable<[], { ok: boolean }>("kill_stream");
export const checkUpdate = callable<[force: boolean], UpdateInfo>("check_update");
// Update the flatpak client in the user installation (`flatpak update --user -y io.unom.Punktfunk`).
export const updateClient = callable<
[],
{ ok: boolean; updated: boolean; error?: string }
>("update_client");
+236 -33
View File
@@ -1,9 +1,19 @@
// Shared state hooks + user actions for the QAM panel and the fullscreen page.
import { toaster } from "@decky/api";
import { Navigation } from "@decky/ui";
import { useCallback, useEffect, useState } from "react";
import { checkUpdate, discover, Host, UpdateInfo } from "./backend";
import { launchStream } from "./steam";
import { useCallback, useEffect, useRef, useState } from "react";
import {
checkUpdate,
discover,
GameEntry,
getPins,
Host,
PinnedGame,
setPins as setPinsBackend,
updateClient,
UpdateInfo,
} from "./backend";
import { LaunchOpts, launchStream } from "./steam";
export const DOCS_URL = "https://docs.punktfunk.unom.io/docs/steam-deck";
@@ -77,6 +87,11 @@ export function useUpdate() {
return { info, checking, check };
}
/** True when EITHER the plugin or the flatpak client has a pending update. */
export function hasUpdate(info: UpdateInfo | null | undefined): boolean {
return !!info && (info.update_available || info.client_update_available);
}
/** The explicit "Check for updates" action — always ends in a toast so the tap has feedback. */
export async function checkForUpdatesNow(
check: (force: boolean) => Promise<UpdateInfo | null>,
@@ -85,55 +100,243 @@ export async function checkForUpdatesNow(
let body: string;
if (!res || res.error === "fetch-failed") {
body = "Couldnt reach the update server — are you online?";
} else if (hasUpdate(res)) {
const parts: string[] = [];
if (res.update_available) parts.push(`plugin v${res.current} → v${res.latest}`);
if (res.client_update_available) parts.push("client");
body = `Update available: ${parts.join(" + ")}.`;
} else if (res.error === "update-channel-unknown") {
body = "Development build — update checks are disabled.";
} else if (res.update_available) {
body = `Update available: v${res.current} → v${res.latest}.`;
body = "Development build — plugin updates are disabled; the client is up to date.";
} else {
body = `Youre up to date (v${res.current}).`;
body = `Youre up to date (plugin v${res.current}).`;
}
toaster.toast({ title: "Punktfunk", body });
}
export async function applyUpdate(info: UpdateInfo): Promise<void> {
try {
const backend = window.DeckyBackend;
if (backend?.callable) {
// Fire-and-forget: the loader reinstalls + reloads THIS plugin, tearing the panel down
// before any result could arrive — so never await it. Decky shows its own confirm prompt.
void backend.callable("utilities/install_plugin")(
info.artifact,
"punktfunk",
info.latest,
info.hash,
INSTALL_TYPE_UPDATE,
);
/**
* Apply whichever updates are pending. The flatpak CLIENT is updated first (a user-scope
* `flatpak update`, awaited); then, if the PLUGIN itself has an update, Decky's install RPC
* reinstalls it — which reloads the plugin and tears this panel down, so it goes last and is
* fire-and-forget. `check` (when passed) refreshes the panel state after a client-only update so
* the "Update available" button clears.
*/
export async function applyUpdate(
info: UpdateInfo,
check?: (force: boolean) => Promise<UpdateInfo | null>,
): Promise<void> {
if (info.client_update_available) {
toaster.toast({ title: "Punktfunk", body: "Updating the client…" });
try {
const r = await updateClient();
toaster.toast({
title: "Punktfunk",
// Decky's installer also phones the plugin store first, which can hang on some
// networks before the actual install proceeds — set expectations.
body: `Updating to v${info.latest} — confirm Deckys prompt. This can take a couple of minutes.`,
body: !r.ok
? `Client update failed${r.error ? ` (${r.error})` : ""}.`
: r.updated
? "Client updated to the latest version."
: "Client is already up to date.",
});
return;
} catch {
toaster.toast({ title: "Punktfunk", body: "Client update failed." });
}
} catch {
// fall through to the manual path
}
toaster.toast({
title: "Punktfunk",
body: "Update from Decky → Developer → Install Plugin from URL.",
});
if (info.update_available) {
try {
const backend = window.DeckyBackend;
if (backend?.callable) {
// Fire-and-forget: the loader reinstalls + reloads THIS plugin, tearing the panel down
// before any result could arrive — so never await it. Decky shows its own confirm prompt.
void backend.callable("utilities/install_plugin")(
info.artifact,
"punktfunk",
info.latest,
info.hash,
INSTALL_TYPE_UPDATE,
);
toaster.toast({
title: "Punktfunk",
// Decky's installer also phones the plugin store first, which can hang on some
// networks before the actual install proceeds — set expectations.
body: `Updating the plugin to v${info.latest} — confirm Deckys prompt. This can take a couple of minutes.`,
});
return;
}
} catch {
// fall through to the manual path
}
toaster.toast({
title: "Punktfunk",
body: "Update the plugin from Decky → Developer → Install Plugin from URL.",
});
return;
}
// Client-only update (no plugin reinstall): refresh so the button clears.
if (check) void check(true);
}
// ----------------------------------------------------------------------------------------
// Stream launch — via the hidden Steam shortcut (see steam.ts for why).
// ----------------------------------------------------------------------------------------
export async function startStream(h: Host): Promise<void> {
export async function startStream(
h: Host,
opts: LaunchOpts = {},
label?: string,
): Promise<void> {
try {
await launchStream(h.host, h.port);
await launchStream(h.host, h.port, opts);
Navigation.CloseSideMenus();
toaster.toast({ title: "Punktfunk", body: `Starting stream${h.name}` });
toaster.toast({ title: "Punktfunk", body: `Starting ${label ?? "stream"}${h.name}` });
} catch (e) {
toaster.toast({ title: "Punktfunk", body: `Launch failed: ${e}` });
}
}
/** Open the GTK client's gamepad library launcher for a host (`--browse` via PF_BROWSE). */
export async function startBrowse(h: Host): Promise<void> {
try {
await launchStream(h.host, h.port, { browse: true, mgmt: h.mgmt });
Navigation.CloseSideMenus();
toaster.toast({ title: "Punktfunk", body: `Opening library — ${h.name}` });
} catch (e) {
toaster.toast({ title: "Punktfunk", body: `Launch failed: ${e}` });
}
}
// ----------------------------------------------------------------------------------------
// Pinned games — the QAM's one-tap game rows, persisted by the backend next to the
// client's config (survives plugin reinstalls).
// ----------------------------------------------------------------------------------------
export interface PinsApi {
pins: PinnedGame[];
addPin: (h: Host, g: GameEntry) => void;
removePin: (hostFp: string, gameId: string) => void;
isPinned: (hostFp: string, gameId: string) => boolean;
/** Refresh a pin's stored address from a live advert (hosts change IPs). */
updatePinHost: (pin: PinnedGame, h: Host) => void;
refresh: () => Promise<void>;
}
export function usePins(): PinsApi {
const [pins, setPins] = useState<PinnedGame[]>([]);
// A live mirror of `pins`. The Games picker is mounted by Decky's `showModal` into a
// detached portal that captures this hook's callbacks ONCE and never re-renders with fresh
// props, so a mutator closing over the `pins` array reads a frozen base — pinning a second
// game in the same session would compute from the stale `[]` and clobber the first (silent
// data loss). Reading the ref keeps every mutation based on the current set, and lets the
// callbacks keep a stable identity (deps free of `pins`).
const pinsRef = useRef<PinnedGame[]>([]);
pinsRef.current = pins;
const refresh = useCallback(async () => {
try {
setPins((await getPins()).pins);
} catch {
/* backend unavailable — keep the current view */
}
}, []);
useEffect(() => {
void refresh();
}, [refresh]);
// Optimistic local state; the backend validates/dedups and is re-read on failure.
const save = useCallback(
(next: PinnedGame[]) => {
pinsRef.current = next;
setPins(next);
setPinsBackend(next).catch(() => void refresh());
},
[refresh],
);
const addPin = useCallback(
(h: Host, g: GameEntry) => {
const pin: PinnedGame = {
game_id: g.id,
title: g.title,
store: g.store,
host_fp: h.fp,
host_id: h.id,
host_name: h.name,
host: h.host,
port: h.port,
mgmt: h.mgmt,
added_at: Math.floor(Date.now() / 1000),
paired: h.paired,
};
save([
...pinsRef.current.filter(
(p) => !(p.host_fp === pin.host_fp && p.game_id === pin.game_id),
),
pin,
]);
},
[save],
);
const removePin = useCallback(
(hostFp: string, gameId: string) => {
save(pinsRef.current.filter((p) => !(p.host_fp === hostFp && p.game_id === gameId)));
},
[save],
);
const isPinned = useCallback(
(hostFp: string, gameId: string) =>
pins.some((p) => p.host_fp === hostFp && p.game_id === gameId),
[pins],
);
const updatePinHost = useCallback(
(pin: PinnedGame, h: Host) => {
if (pin.host === h.host && pin.port === h.port && pin.mgmt === h.mgmt) {
return;
}
save(
pinsRef.current.map((p) =>
p.host_fp === pin.host_fp && p.game_id === pin.game_id
? { ...p, host: h.host, port: h.port, mgmt: h.mgmt, host_name: h.name }
: p,
),
);
},
[save],
);
return { pins, addPin, removePin, isPinned, updatePinHost, refresh };
}
/**
* The host a pin should launch against right now: match the live mDNS scan by cert
* fingerprint first (pairing is fp-keyed, survives IP changes), then by the host's stable
* id, else fall back to the stored address (host offline or scan flaky — still launch).
*/
export function resolvePinHost(
pin: PinnedGame,
live: Host[],
): { host: Host; online: boolean } {
const fp = pin.host_fp.toLowerCase();
const match =
(fp && live.find((h) => h.fp && h.fp.toLowerCase() === fp)) ||
(pin.host_id && live.find((h) => h.id && h.id === pin.host_id)) ||
undefined;
if (match) {
return { host: match, online: true };
}
return {
host: {
name: pin.host_name || pin.host,
host: pin.host,
port: pin.port,
pair: pin.paired ? "optional" : "required",
fp: pin.host_fp,
proto: "",
paired: !!pin.paired,
id: pin.host_id,
mgmt: pin.mgmt,
},
online: false,
};
}
+49 -6
View File
@@ -12,28 +12,46 @@ import {
} from "@decky/ui";
import { definePlugin, routerHook } from "@decky/api";
import { FC } from "react";
import { FaDownload, FaLock, FaLockOpen, FaSyncAlt, FaTv } from "react-icons/fa";
import { FaDownload, FaLock, FaLockOpen, FaPlay, FaSyncAlt, FaTv } from "react-icons/fa";
import { PluginErrorBoundary } from "./boundary";
import { applyUpdate, checkForUpdatesNow, startStream, useHosts, useUpdate } from "./hooks";
import {
applyUpdate,
checkForUpdatesNow,
hasUpdate,
resolvePinHost,
startStream,
useHosts,
usePins,
useUpdate,
} from "./hooks";
import { streamPin } from "./library";
import { PunktfunkRoute, ROUTE } from "./page";
import { PairModal } from "./pair";
// ----------------------------------------------------------------------------------------
// QAM panel — quick status + entry into the full page + one-tap stream for known hosts.
// QAM panel — quick status + entry into the full page + one-tap stream for known hosts
// and pinned games.
// ----------------------------------------------------------------------------------------
const QamPanel: FC = () => {
const { hosts, scanning, refresh } = useHosts();
const { info: update, checking, check } = useUpdate();
const pins = usePins();
return (
<>
{update?.update_available && (
{hasUpdate(update) && (
<PanelSection title="Update available">
<PanelSectionRow>
<ButtonItem
layout="below"
onClick={() => applyUpdate(update)}
label={`v${update.current} → v${update.latest}`}
onClick={() => applyUpdate(update!, check)}
label={
update!.update_available
? `Plugin v${update!.current} → v${update!.latest}${
update!.client_update_available ? " + client" : ""
}`
: "New client version"
}
description="Installing can take a couple of minutes"
>
<FaDownload style={{ marginRight: "0.5em" }} />
@@ -59,6 +77,31 @@ const QamPanel: FC = () => {
</PanelSectionRow>
</PanelSection>
{/* Pinned games — the "jump straight into Playnite" rows. Pin games from a host's
picker (fullscreen page → host row → games button). */}
{pins.pins.length > 0 && (
<PanelSection title="Games">
{pins.pins.map((pin) => {
const { online } = resolvePinHost(pin, hosts);
return (
<PanelSectionRow key={`${pin.host_fp}:${pin.game_id}`}>
<ButtonItem
layout="below"
onClick={() => streamPin(pin, hosts, pins)}
label={pin.title}
description={`${pin.host_name}${online ? "" : " · offline?"}${
pin.paired ? "" : " · pairing required"
}`}
>
<FaPlay style={{ marginRight: "0.5em" }} />
Stream
</ButtonItem>
</PanelSectionRow>
);
})}
</PanelSection>
)}
<PanelSection title="Hosts">
<PanelSectionRow>
<ButtonItem layout="below" onClick={refresh} disabled={scanning}>
+231
View File
@@ -0,0 +1,231 @@
// The per-host game picker + pinned-game launch helper. The picker fetches a paired
// host's library through the backend (headless flatpak --library — a cold client start
// can take seconds, hence the explicit spinner copy) and pins titles as one-tap rows in
// the QAM's Games section; its header also launches the GTK client's on-screen gamepad
// library (`--browse`).
import { DialogButton, Field, Focusable, ModalRoot, Spinner, showModal } from "@decky/ui";
import { CSSProperties, FC, useEffect, useState } from "react";
import { FaThLarge, FaTv } from "react-icons/fa";
import { GameEntry, Host, library, LibraryResult, PinnedGame } from "./backend";
import { PinsApi, resolvePinHost, startBrowse, startStream } from "./hooks";
import { isSafeLaunchId } from "./steam";
import { PairModal } from "./pair";
/** Human store tag (mirrors the GTK client's `store_label`). */
export function storeLabel(store: string): string {
switch (store) {
case "steam":
return "Steam";
case "custom":
return "Custom";
case "heroic":
return "Heroic";
case "lutris":
return "Lutris";
case "epic":
return "Epic";
case "gog":
return "GOG";
case "xbox":
return "Xbox";
default:
return "Game";
}
}
/**
* Stream a pinned game: resolve the host from the live scan (fp → id → stored address),
* opportunistically refresh a drifted stored address, and route through pairing first if
* this device is no longer paired with the host.
*/
export function streamPin(pin: PinnedGame, live: Host[], pins: PinsApi): void {
const { host, online } = resolvePinHost(pin, live);
if (online) {
pins.updatePinHost(pin, host); // no-op unless the address actually drifted
}
if (!pin.paired) {
showModal(
<PairModal
host={host}
onPaired={() => {
void pins.refresh(); // pick up the now-paired annotation
void startStream(host, { launchId: pin.game_id }, pin.title);
}}
/>,
);
return;
}
void startStream(host, { launchId: pin.game_id }, pin.title);
}
const pickButton: CSSProperties = {
width: "fit-content",
minWidth: "5em",
flexShrink: 0,
};
// Copy per backend error code (LibraryResult.error); `detail` covers the generic case.
function errorCopy(res: LibraryResult): string {
switch (res.error) {
case "not-paired":
return "This Deck isn't paired with the host — pair first, then browse its library.";
case "pin-mismatch":
return "The host's identity changed — re-pair to re-establish trust.";
case "unreachable":
return "Couldn't reach the host's management API. Is the host online and up to date?";
case "timeout":
return "Timed out talking to the host — try again.";
case "flatpak-not-found":
return "The Punktfunk client isn't installed (flatpak io.unom.Punktfunk).";
case "client-outdated":
return "The installed client is too old for library browsing — update it from the About tab.";
default:
return res.detail || "Couldn't fetch the library.";
}
}
// ----------------------------------------------------------------------------------------
// The picker modal: "open on screen" + a pin-toggle list of the host's games.
// ----------------------------------------------------------------------------------------
export const GamePickerModal: FC<{
host: Host;
pins: PinsApi;
clientUpdatePending?: boolean;
closeModal?: () => void;
}> = ({ host, pins, clientUpdatePending, closeModal }) => {
const [result, setResult] = useState<LibraryResult | null>(null);
const [attempt, setAttempt] = useState(0); // bump to refetch (retry / after pairing)
// The modal is a detached `showModal` portal that never re-renders from the page's pin
// state, so `pins.isPinned` would read a frozen snapshot and the Pin/Unpin label would
// never flip within a session. Track this host's pinned ids locally, seeded once from the
// snapshot at open; persistence still goes through the (stale-closure-safe) pins API.
const [pinnedIds, setPinnedIds] = useState<Set<string>>(
() => new Set(pins.pins.filter((p) => p.host_fp === host.fp).map((p) => p.game_id)),
);
const togglePin = (g: GameEntry) => {
const wasPinned = pinnedIds.has(g.id);
setPinnedIds((prev) => {
const next = new Set(prev);
if (wasPinned) next.delete(g.id);
else next.add(g.id);
return next;
});
if (wasPinned) pins.removePin(host.fp, g.id);
else pins.addPin(host, g);
};
useEffect(() => {
let stale = false;
setResult(null);
library(host.host, host.mgmt, host.fp)
.then((res) => {
if (!stale) setResult(res);
})
.catch((e) => {
if (!stale) setResult({ ok: false, error: "client-error", detail: String(e) });
});
return () => {
stale = true;
};
}, [host.host, host.mgmt, host.fp, attempt]);
const games = (result?.ok && result.games) || [];
const sorted = [...games].sort((a, b) => a.title.localeCompare(b.title));
return (
<ModalRoot closeModal={closeModal}>
<div style={{ fontWeight: "bold", fontSize: "1.3em", marginBottom: "0.4em" }}>
{host.name} Games
</div>
<Field
label="Open library on screen"
description="Browse this host's games with the controller, full screen"
childrenContainerWidth="max"
>
<DialogButton
style={pickButton}
onClick={() => {
closeModal?.();
void startBrowse(host);
}}
>
<FaTv style={{ marginRight: "0.4em" }} />
Open
</DialogButton>
</Field>
{clientUpdatePending && (
<Field
focusable={false}
description="A client update is available — direct game launch and on-screen browsing need the latest client."
/>
)}
{result === null && (
<Field
focusable={false}
label={
<span style={{ display: "inline-flex", alignItems: "center", gap: "0.6em" }}>
<Spinner style={{ height: "1em" }} />
Fetching the library
</span>
}
description="This starts the client headlessly — a cold start can take a few seconds."
/>
)}
{result !== null && !result.ok && (
<Field label="Couldn't fetch the library" description={errorCopy(result)} childrenContainerWidth="max">
<Focusable style={{ display: "flex", gap: "0.5em", justifyContent: "flex-end" }}>
{result.error === "not-paired" && (
<DialogButton
style={pickButton}
onClick={() =>
showModal(<PairModal host={host} onPaired={() => setAttempt((n) => n + 1)} />)
}
>
Pair
</DialogButton>
)}
<DialogButton style={pickButton} onClick={() => setAttempt((n) => n + 1)}>
Retry
</DialogButton>
</Focusable>
</Field>
)}
{result?.ok && sorted.length === 0 && (
<Field
focusable={false}
label="No games found"
description="Install Steam titles or add custom entries in the host's web console."
/>
)}
{sorted.length > 0 && (
<div style={{ maxHeight: "55vh", overflowY: "auto" }}>
{sorted.map((g: GameEntry) => {
const pinned = pinnedIds.has(g.id);
const safe = isSafeLaunchId(g.id);
return (
<Field
key={g.id}
label={g.title}
description={
storeLabel(g.store) + (safe ? "" : " · unsupported id — can't be pinned")
}
childrenContainerWidth="max"
>
<DialogButton style={pickButton} disabled={!safe} onClick={() => togglePin(g)}>
<FaThLarge style={{ marginRight: "0.4em" }} />
{pinned ? "Unpin" : "Pin"}
</DialogButton>
</Field>
);
})}
</div>
)}
</ModalRoot>
);
};
+136 -26
View File
@@ -21,17 +21,23 @@ import {
FaLockOpen,
FaPlay,
FaSyncAlt,
FaThLarge,
} from "react-icons/fa";
import { Host, UpdateInfo, killStream } from "./backend";
import { PluginErrorBoundary } from "./boundary";
import {
DOCS_URL,
PinsApi,
applyUpdate,
checkForUpdatesNow,
hasUpdate,
resolvePinHost,
startStream,
useHosts,
usePins,
useUpdate,
} from "./hooks";
import { GamePickerModal, storeLabel, streamPin } from "./library";
import { PairModal } from "./pair";
import { SettingsSection } from "./settings";
import { stopStream } from "./steam";
@@ -52,6 +58,27 @@ const tabScroll: CSSProperties = {
boxSizing: "border-box",
};
// DialogButton stretches to 100% width in the gamepad UI — on a fullscreen row that means a
// screen-wide button. Size action buttons to their content instead (right-aligned by the
// Field's children container).
const actionButton: CSSProperties = {
width: "fit-content",
minWidth: "6em",
flexShrink: 0,
};
// Square icon-only button (details ⓘ, header back arrow) — needs an explicit height too, or
// the zero padding collapses it to the icon's line height.
const iconButton: CSSProperties = {
width: "40px",
minWidth: "40px",
height: "40px",
padding: 0,
flexShrink: 0,
display: "flex",
alignItems: "center",
justifyContent: "center",
};
// ----------------------------------------------------------------------------------------
// Host details — everything the mDNS advert told us, incl. the fingerprint to cross-check
// against the host's own log / web console before trusting it.
@@ -96,7 +123,11 @@ const HostDetailsModal: FC<{ host: Host; closeModal?: () => void }> = ({
// ----------------------------------------------------------------------------------------
// One host row: status icon + address, details / pair / stream actions.
// ----------------------------------------------------------------------------------------
const HostRow: FC<{ host: Host; onPaired: () => void }> = ({ host, onPaired }) => {
const HostRow: FC<{ host: Host; onPaired: () => void; onGames: () => void }> = ({
host,
onPaired,
onGames,
}) => {
// The host's policy is `pair=required`, but if THIS device is already paired we don't need to
// pair again — show it as trusted and go straight to Stream.
const needsPair = host.pair === "required" && !host.paired;
@@ -113,22 +144,37 @@ const HostRow: FC<{ host: Host; onPaired: () => void }> = ({ host, onPaired }) =
}`}
childrenContainerWidth="max"
>
<Focusable style={{ display: "flex", gap: "0.5em" }}>
<Focusable style={{ display: "flex", gap: "0.5em", justifyContent: "flex-end" }}>
<DialogButton
style={{ width: "3em", minWidth: "3em", padding: 0 }}
style={iconButton}
onClick={() => showModal(<HostDetailsModal host={host} />)}
>
<FaInfoCircle />
</DialogButton>
{/* Labeled, not icon-only: this is the entry to the game picker AND the on-screen
library browser, and controller nav has no hover tooltip to explain a bare icon. */}
<DialogButton style={{ ...actionButton, minWidth: "6em" }} onClick={onGames}>
<FaThLarge style={{ marginRight: "0.4em" }} />
Games
</DialogButton>
{needsPair && (
<DialogButton
style={{ minWidth: "5em" }}
style={{ ...actionButton, minWidth: "5em" }}
onClick={() => showModal(<PairModal host={host} onPaired={onPaired} />)}
>
Pair
</DialogButton>
)}
<DialogButton style={{ minWidth: "6em" }} onClick={() => startStream(host)}>
<DialogButton
style={actionButton}
onClick={() =>
needsPair
? showModal(
<PairModal host={host} onPaired={() => startStream(host)} />,
)
: startStream(host)
}
>
<FaPlay style={{ marginRight: "0.4em" }} />
Stream
</DialogButton>
@@ -141,7 +187,9 @@ const HostsTab: FC<{
hosts: Host[];
scanning: boolean;
refresh: () => void;
}> = ({ hosts, scanning, refresh }) => (
pins: PinsApi;
clientUpdatePending: boolean;
}> = ({ hosts, scanning, refresh, pins, clientUpdatePending }) => (
<div style={tabScroll}>
<Field
label="Discover"
@@ -153,7 +201,7 @@ const HostsTab: FC<{
childrenContainerWidth="max"
bottomSeparator={hosts.length ? "standard" : "none"}
>
<DialogButton style={{ minWidth: "8em" }} disabled={scanning} onClick={refresh}>
<DialogButton style={{ ...actionButton, minWidth: "8em" }} disabled={scanning} onClick={refresh}>
{scanning ? (
<Spinner style={{ height: "1em", marginRight: "0.5em" }} />
) : (
@@ -171,8 +219,55 @@ const HostsTab: FC<{
/>
)}
{hosts.map((h) => (
<HostRow key={h.fp || `${h.host}:${h.port}`} host={h} onPaired={refresh} />
<HostRow
key={h.fp || `${h.host}:${h.port}`}
host={h}
onPaired={refresh}
onGames={() =>
showModal(
<GamePickerModal host={h} pins={pins} clientUpdatePending={clientUpdatePending} />,
)
}
/>
))}
{/* Pinned games — also the cleanup surface for pins whose host is gone from the scan. */}
{pins.pins.length > 0 && (
<>
<Field
focusable={false}
label="Pinned games"
description="One-tap streams — they also live in the quick-access menu"
bottomSeparator="standard"
/>
{pins.pins.map((pin) => {
const { online } = resolvePinHost(pin, hosts);
return (
<Field
key={`${pin.host_fp}:${pin.game_id}`}
label={pin.title}
description={`${storeLabel(pin.store)} · ${pin.host_name}${
online ? "" : " · offline?"
}${pin.paired ? "" : " · pairing required"}`}
childrenContainerWidth="max"
>
<Focusable style={{ display: "flex", gap: "0.5em", justifyContent: "flex-end" }}>
<DialogButton style={actionButton} onClick={() => streamPin(pin, hosts, pins)}>
<FaPlay style={{ marginRight: "0.4em" }} />
Play
</DialogButton>
<DialogButton
style={{ ...actionButton, minWidth: "5em" }}
onClick={() => pins.removePin(pin.host_fp, pin.game_id)}
>
Remove
</DialogButton>
</Focusable>
</Field>
);
})}
</>
)}
</div>
);
@@ -212,20 +307,29 @@ const AboutTab: FC<{
childrenContainerWidth="max"
>
<DialogButton
style={{ minWidth: "11em" }}
style={{ ...actionButton, minWidth: "11em" }}
disabled={checking}
onClick={() => void checkForUpdatesNow(check)}
>
{checking ? <Spinner style={{ height: "1em" }} /> : "Check for updates"}
</DialogButton>
</Field>
{update?.update_available && (
{hasUpdate(update) && (
<Field
label={`Update available — v${update.latest}`}
label={
update!.update_available
? `Plugin update — v${update!.latest}${
update!.client_update_available ? " + client" : ""
}`
: "Client update available"
}
description="Installing can take a couple of minutes; Decky reloads the plugin when done"
childrenContainerWidth="max"
>
<DialogButton style={{ minWidth: "9em" }} onClick={() => applyUpdate(update)}>
<DialogButton
style={{ ...actionButton, minWidth: "9em" }}
onClick={() => applyUpdate(update!, check)}
>
<FaDownload style={{ marginRight: "0.4em" }} />
Update
</DialogButton>
@@ -237,7 +341,7 @@ const AboutTab: FC<{
childrenContainerWidth="max"
>
<DialogButton
style={{ minWidth: "8em" }}
style={{ ...actionButton, minWidth: "8em" }}
onClick={() => Navigation.NavigateToExternalWeb(DOCS_URL)}
>
<FaExternalLinkAlt style={{ marginRight: "0.4em" }} />
@@ -254,7 +358,7 @@ const AboutTab: FC<{
description="Force-stop the stream client if a session wedges"
childrenContainerWidth="max"
>
<DialogButton style={{ minWidth: "8em" }} onClick={() => void forceStopStream()}>
<DialogButton style={{ ...actionButton, minWidth: "8em" }} onClick={() => void forceStopStream()}>
Force-stop
</DialogButton>
</Field>
@@ -264,6 +368,7 @@ const AboutTab: FC<{
const PunktfunkPage: FC = () => {
const { hosts, scanning, refresh } = useHosts();
const { info: update, checking, check } = useUpdate();
const pins = usePins();
const [tab, setTab] = useState("hosts");
return (
@@ -275,6 +380,7 @@ const PunktfunkPage: FC = () => {
flexDirection: "column",
}}
>
{/* Header is title + back only — updates live on the About tab (and the QAM banner). */}
<Focusable
style={{
display: "flex",
@@ -285,24 +391,20 @@ const PunktfunkPage: FC = () => {
flexShrink: 0,
}}
>
<DialogButton
style={{ width: "3em", minWidth: "3em", padding: 0 }}
onClick={() => Navigation.NavigateBack()}
>
<DialogButton style={iconButton} onClick={() => Navigation.NavigateBack()}>
<FaArrowLeft />
</DialogButton>
<div className={staticClasses?.Title} style={{ flex: 1, margin: 0 }}>
Punktfunk
</div>
{update?.update_available && (
<DialogButton style={{ minWidth: "9em" }} onClick={() => applyUpdate(update)}>
<FaDownload style={{ marginRight: "0.4em" }} />
Update v{update.latest}
</DialogButton>
)}
</Focusable>
<div style={{ flex: 1, minHeight: 0 }}>
{/* overflow:hidden is load-bearing: Valve's Tabs slides the incoming panel in from the
right on L1/R1, and with autoFocusContents it scrollIntoViews a control inside that
still-offscreen panel. Without a clip here the scroll pans #GamepadUI itself — the whole
Steam UI (top bar included) slides left until you click a tab. Valve's own Tabs always
live in a clipped flex box; match that. */}
<div style={{ flex: 1, minHeight: 0, overflow: "hidden" }}>
<Tabs
activeTab={tab}
onShowTab={(id: string) => setTab(id)}
@@ -311,7 +413,15 @@ const PunktfunkPage: FC = () => {
{
id: "hosts",
title: "Hosts",
content: <HostsTab hosts={hosts} scanning={scanning} refresh={refresh} />,
content: (
<HostsTab
hosts={hosts}
scanning={scanning}
refresh={refresh}
pins={pins}
clientUpdatePending={!!update?.client_update_available}
/>
),
},
{
id: "settings",
+2 -2
View File
@@ -99,10 +99,10 @@ export const SettingsSection: FC = () => {
onChange={(o) => patch({ gamepad: o.data as string })}
/>
</Field>
{s.gamepad === "steamdeck" && (
{(s.gamepad === "steamdeck" || s.gamepad === "auto") && (
<Field
label="⚠ Disable Steam Input"
description="Steam Deck mode forwards the paddles, both trackpads, and gyro to the host. For that, Steam Input must be OFF for Punktfunk: on the game page tap ⚙ → Controller Settings → set Steam Input to Off. Otherwise Steam keeps the Deck's controls and only the sticks + buttons reach the host."
description="On a Deck, Automatic forwards the built-in controller as a Steam Deck pad — paddles, both trackpads, and gyro included. For that, Steam Input must be OFF for Punktfunk: on the game page tap ⚙ → Controller Settings → set Steam Input to Off. Otherwise Steam keeps the Deck's controls and only the sticks + buttons reach the host."
/>
)}
<Field
+109 -19
View File
@@ -8,7 +8,7 @@
// and start it with RunGame. The wrapper then execs
// `flatpak run io.unom.Punktfunk --connect <host>` as a reaper descendant.
import { runnerInfo } from "./backend";
import { runnerInfo, shortcutArt } from "./backend";
// SteamClient is a Steam-internal global injected into the CEF context; it is not fully typed
// by @decky/ui, so declare the surface we use. Signatures verified against MoonDeck + the
@@ -24,24 +24,35 @@ declare const SteamClient: {
SetShortcutName(appId: number, name: string): void;
SetShortcutExe(appId: number, exe: string): void;
SetShortcutStartDir(appId: number, dir: string): void;
SetShortcutIcon(appId: number, iconPath: string): void;
SetAppLaunchOptions(appId: number, options: string): void;
// assetType: 0 = grid (portrait capsule), 1 = hero, 2 = logo, 3 = wide grid.
SetCustomArtworkForApp(
appId: number,
base64Image: string,
imageType: string,
assetType: number,
): Promise<unknown>;
RunGame(gameId: string, _unused: string, _i: number, _j: number): void;
TerminateApp(gameId: string, _b: boolean): void;
};
};
// Steam removed `SteamClient.Apps.SetAppHidden`. Hiding a non-Steam shortcut now goes through
// `collectionStore.SetAppsAsHidden([appId], true)` — but that looks the app up in appStore, which
// only registers a freshly-created shortcut a moment later (calling it immediately throws on a
// null overview). So hiding is BEST-EFFORT + DEFERRED and must NEVER block the launch.
// Steam removed `SteamClient.Apps.SetAppHidden`; visibility goes through
// `collectionStore.SetAppsAsHidden` — but that looks the app up in appStore, which only
// registers a freshly-created shortcut a moment later (calling it immediately throws on a
// null overview). So visibility changes are BEST-EFFORT + DEFERRED, never launch-blocking.
declare const collectionStore:
| { SetAppsAsHidden?: (appIds: number[], hidden: boolean) => void }
| undefined;
function hideShortcut(appId: number): void {
// The shortcut used to be hidden ("implementation detail"); it is user-visible now — it
// carries proper artwork and living in the library is how users relaunch their last host.
// Existing installs still have theirs hidden, so unhide is applied every ensure (idempotent).
function unhideShortcut(appId: number): void {
const attempt = () => {
try {
collectionStore?.SetAppsAsHidden?.([appId], true);
collectionStore?.SetAppsAsHidden?.([appId], false);
} catch {
/* overview not registered yet, or the API changed — cosmetic, ignore */
}
@@ -50,6 +61,40 @@ function hideShortcut(appId: number): void {
setTimeout(attempt, 2500); // fresh shortcut: retry once its app overview lands
}
// Bump when the shipped artwork changes so existing shortcuts re-apply it once.
const ART_VERSION = 1;
const ART_KEY = "punktfunk:shortcutArt";
/**
* Apply the plugin's grid/hero/logo/icon to the shortcut (idempotent, once per ART_VERSION).
* Cosmetic and fully best-effort: any failure is swallowed and retried on the next launch.
*/
async function applyArtwork(appId: number): Promise<void> {
try {
if (localStorage.getItem(ART_KEY) === `${appId}:${ART_VERSION}`) {
return;
}
const art = await shortcutArt();
const assets: [string | undefined, number][] = [
[art.grid, 0],
[art.hero, 1],
[art.logo, 2],
[art.gridwide, 3],
];
for (const [data, assetType] of assets) {
if (data) {
await SteamClient.Apps.SetCustomArtworkForApp(appId, data, "png", assetType);
}
}
if (art.icon_path) {
SteamClient.Apps.SetShortcutIcon(appId, art.icon_path);
}
localStorage.setItem(ART_KEY, `${appId}:${ART_VERSION}`);
} catch (e) {
console.warn("punktfunk: shortcut artwork not applied", e);
}
}
// The shortcut name is user-visible (Steam overlay + library while streaming) — brand-case it.
const SHORTCUT_NAME = "Punktfunk";
@@ -87,10 +132,11 @@ function recallAppId(): number | null {
}
/**
* Ensure exactly one hidden "Punktfunk" shortcut exists (exe = /bin/sh; the wrapper script is
* appended per-launch via the launch options), and return its appId + the current runner path.
* Reuses the remembered shortcut, re-pointing it each time — the plugin dir can change across
* reinstalls, and pre-0.4 shortcuts pointed at the script directly and relied on its exec bit.
* Ensure exactly one "Punktfunk" shortcut exists (exe = /bin/sh; the wrapper script is
* appended per-launch via the launch options), branded and visible in the library, and
* return its appId + the current runner path. Reuses the remembered shortcut, re-pointing
* it each time — the plugin dir can change across reinstalls, pre-0.4 shortcuts pointed at
* the script directly, and pre-0.7 shortcuts were hidden and artless.
*/
async function ensureShortcut(): Promise<{ appId: number; runner: string }> {
const info = await runnerInfo();
@@ -105,14 +151,15 @@ async function ensureShortcut(): Promise<{ appId: number; runner: string }> {
SteamClient.Apps.SetShortcutExe(remembered, SHELL);
SteamClient.Apps.SetShortcutStartDir(remembered, startDir);
SteamClient.Apps.SetShortcutName(remembered, SHORTCUT_NAME);
unhideShortcut(remembered); // pre-0.7 installs hid it
void applyArtwork(remembered); // fire-and-forget — cosmetic, never blocks the launch
return { appId: remembered, runner: info.runner };
}
const appId = await SteamClient.Apps.AddShortcut(SHORTCUT_NAME, SHELL, startDir, "");
SteamClient.Apps.SetShortcutName(appId, SHORTCUT_NAME);
// Hide it from the library — it's an implementation detail, launched programmatically.
// Best-effort + deferred (see hideShortcut); never let it block the launch.
hideShortcut(appId);
unhideShortcut(appId);
void applyArtwork(appId); // fire-and-forget — cosmetic, never blocks the launch
rememberAppId(appId);
return { appId, runner: info.runner };
}
@@ -137,19 +184,62 @@ function disableSteamInputForShortcut(appId: number): void {
}
}
/** Per-launch extras beyond the host target (all optional — {} is the plain stream). */
export interface LaunchOpts {
/** Library id to launch on connect (a pinned game) — rides PF_LAUNCH → `--launch`. */
launchId?: string;
/** Open the gamepad library launcher instead of streaming (PF_BROWSE → `--browse`). */
browse?: boolean;
/** Management-API port for the launcher's library fetch (PF_MGMT; 0/absent = default). */
mgmt?: number;
}
// Launch ids ride Steam launch options as an env-prefix token (`PF_LAUNCH=<id>`), so they
// must be space/quote-free — Steam's tokenizer and the wrapper's env both break otherwise.
// Real ids are `steam:<digits>` / `custom:<slug>`, so this rejects nothing in practice;
// it's VALIDATION, never encoding (the host must match the opaque token verbatim).
const UNSAFE_LAUNCH_ID = /["'\\$`\s]/;
export function isSafeLaunchId(id: string): boolean {
return (
id.length > 0 &&
id.length <= 128 &&
UNSAFE_LAUNCH_ID.exec(id) === null &&
/^[\x21-\x7e]+$/.test(id)
);
}
/**
* Launch a stream to `host:port` fullscreen in Gaming Mode. Encodes the target into the
* shortcut's launch options (so one generic shortcut serves every host), then RunGame.
* Launch a stream to `host:port` fullscreen in Gaming Mode (optionally straight into a
* library title, or into the gamepad library launcher). Encodes the target into the
* shortcut's launch options (so one generic shortcut serves every host and every pinned
* game), then RunGame.
*/
export async function launchStream(host: string, port: number): Promise<void> {
export async function launchStream(
host: string,
port: number,
opts: LaunchOpts = {},
): Promise<void> {
const { appId, runner } = await ensureShortcut();
// Best-effort so the Deck's rich controls reach the client; no-op if the API is absent (the user
// disables Steam Input manually — see the Settings instruction).
disableSteamInputForShortcut(appId);
const target = port && port !== 9777 ? `${host}:${port}` : host;
const env = [`PF_HOST=${target}`];
if (opts.browse) {
env.push("PF_BROWSE=1");
if (opts.mgmt) {
env.push(`PF_MGMT=${Math.floor(opts.mgmt)}`);
}
} else if (opts.launchId) {
if (!isSafeLaunchId(opts.launchId)) {
// Enforced at pin time too (the picker disables Pin) — this is the backstop.
throw new Error(`unsupported launch id: ${opts.launchId}`);
}
env.push(`PF_LAUNCH=${opts.launchId}`);
}
// KEY=value ... %command% args — %command% expands to the shortcut exe (/bin/sh); the wrapper
// script rides behind it as an argument and reads PF_HOST from the environment.
SteamClient.Apps.SetAppLaunchOptions(appId, `PF_HOST=${target} %command% "${runner}"`);
// script rides behind it as an argument and reads PF_* from the environment.
SteamClient.Apps.SetAppLaunchOptions(appId, `${env.join(" ")} %command% "${runner}"`);
SteamClient.Apps.RunGame(gameIdFromAppId(appId), "", -1, 100);
}
+12 -4
View File
@@ -26,6 +26,10 @@ Built in Rust, it links the shared **`punktfunk-core`** directly (no C ABI) and
shows its games (Steam + custom) as a poster grid; click one to launch it in the session.
Fetched from the host's management API over mTLS — paired devices are authorized by their
certificate, no extra host setup.
- **Gamepad library launcher** (`--browse host`) — a console-style, controller-driven coverflow of
a paired host's library (drifting aurora backdrop, center-focus posters, button hints): A plays
the focused title, B quits, L1/R1 jump. Built for the Steam Deck plugin's "Open library" launch;
session end returns to the launcher. Arrow keys/Enter/Esc drive it too (no pad needed).
## Get it
@@ -49,24 +53,28 @@ and SDL3 (with hidapi) development packages.
```sh
# from the repo root
cargo run -p punktfunk-client-linux # launch the app
cargo run -p punktfunk-client-linux -- --discover # list hosts on the LAN, then exit
cargo run -p punktfunk-client-linux -- --connect HOST[:PORT] # skip the host list and connect
cargo run -p punktfunk-client-linux -- --browse HOST # the gamepad library launcher
```
The binary is named **`punktfunk-client`**. Handy flags: `--connect host[:port]` (start a session
immediately — for scripting and the Steam Deck launcher), `--discover [secs]`, and
immediately — for scripting and the Steam Deck launcher) with optional `--launch <id>` (ask the
host to launch that library title, id from `--library`), `--browse host[:port]` (the gamepad
library launcher; `--mgmt <port>` overrides the management port it fetches from),
`--pair <PIN> --connect host[:port]` (run the pairing ceremony headlessly), and
`--library host[:mgmt_port]` (print a host's game library headlessly). Force a decoder with
`PUNKTFUNK_DECODER=software|vaapi`.
`PUNKTFUNK_DECODER=software|vaapi`; `PUNKTFUNK_FAKE_LIBRARY=<file.json>` feeds the launcher
canned entries for UI work with no host.
## Layout
```
src/
main.rs · app.rs entry point, GTK application, primary menu, CSS
cli.rs CLI paths (--connect, headless --pair, screenshot scenes)
cli.rs CLI paths (--connect/--launch, --browse, headless --pair, screenshot scenes)
ui_hosts.rs host card grids (saved + discovered) · add-host dialog · banner
ui_library.rs game-library poster grid (per-host, launches titles)
ui_gamepad_library.rs the --browse gamepad launcher (aurora · coverflow · hint bar)
ui_trust.rs TOFU / PIN-pairing / request-access dialogs
ui_settings.rs resolution · refresh · decoder · bitrate · compositor · mic
ui_stream.rs the stream window (GtkGraphicsOffload present) + input capture
+81 -6
View File
@@ -22,14 +22,44 @@ const CSS: &str = "
color: alpha(currentColor, 0.8); background: alpha(currentColor, 0.1); }
.pf-pill.pf-green { color: @success_color; background: alpha(@success_color, 0.15); }
.pf-pill.pf-accent { color: @accent_color; background: alpha(@accent_color, 0.15); }
.pf-pill.pf-neutral { color: alpha(currentColor, 0.75); background: alpha(currentColor, 0.12); }
.pf-pip { min-width: 8px; min-height: 8px; border-radius: 999px;
background: alpha(currentColor, 0.35); }
.pf-pip.pf-online { background: @success_color; }
.pf-recent { box-shadow: inset 3px 0 0 0 @accent_bg_color; }
/* Most-recent host: a full accent ring drawn as an inset outline so it follows the card's
rounded corners (an `inset` box-shadow bar gets eaten by the 12px corner clip) and leaves
the card's own elevation shadow intact. */
.pf-recent { outline: 2px solid @accent_color; outline-offset: -2px; }
.pf-discovered { border: 1px dashed alpha(currentColor, 0.35); }
.pf-poster { border-radius: 10px; background: alpha(currentColor, 0.08); }
.pf-poster-monogram { font-size: 2.4em; font-weight: bold; color: alpha(currentColor, 0.45); }
.pf-store-badge { color: white; background: rgba(0, 0, 0, 0.55); }
/* Gaming-Mode launches: gamescope displays the window fullscreen but never ACKs the
xdg_toplevel fullscreen state, so GTK keeps the floating-CSD styling — libadwaita's
rounded corners + shadow margin stay visible over the stream. Flatten them outright. */
window.pf-chromeless { border-radius: 0; box-shadow: none; }
/* The gamepad library launcher (`--browse`, ui_gamepad_library) — always-dark console
chrome over the aurora, independent of the desktop theme. */
.pf-gl-page { background: black; color: white; }
.pf-gl-host { font-size: 1.15em; font-weight: bold; color: rgba(255, 255, 255, 0.9); }
.pf-gl-chip { font-size: 0.8em; color: rgba(255, 255, 255, 0.7);
background: rgba(255, 255, 255, 0.08);
border: 1px solid rgba(255, 255, 255, 0.12);
border-radius: 999px; padding: 4px 12px; }
/* Solid face, not glass: coverflow side cards OVERLAP — a translucent card would bleed
the stack through the one on top. */
.pf-gl-poster { border-radius: 16px; background: rgb(30, 30, 37);
border: 1px solid rgba(255, 255, 255, 0.07); }
.pf-gl-dim { background: black; border-radius: 16px; }
.pf-gl-detail-title { font-size: 1.7em; font-weight: bold; color: white; }
.pf-gl-detail-store { font-size: 0.75em; font-weight: 600; letter-spacing: 2px;
color: rgba(255, 255, 255, 0.5); }
.pf-gl-glyph { font-size: 0.85em; font-weight: bold; color: white;
background: rgba(255, 255, 255, 0.14);
border-radius: 999px; min-width: 26px; min-height: 26px; padding: 2px 8px; }
.pf-gl-hint { color: rgba(255, 255, 255, 0.85); }
.pf-gl-status { font-size: 0.85em; color: #ff938a; }
.pf-gl-error-title { font-size: 1.4em; font-weight: bold; color: white; }
";
pub struct App {
@@ -44,9 +74,16 @@ pub struct App {
pub busy: std::cell::Cell<bool>,
/// Steam Deck / Gaming-Mode launch: fullscreen the window (chrome-less) when a stream starts.
pub fullscreen: bool,
/// Quit when the session ends (Gaming-Mode `--connect` launch): the app IS the stream —
/// exiting ends the Steam "game" so the Deck returns to Gaming Mode instead of stranding
/// the user on the client's own hosts page.
pub quit_on_session_end: bool,
/// The hosts page handle (banner + per-card connecting spinner), set right after the
/// page is built — `None` only during construction.
pub hosts: RefCell<Option<Rc<HostsUi>>>,
/// The gamepad library launcher — `Some` only under `--browse`, where it replaces the
/// hosts page as the root (and session end returns here instead of quitting).
pub browse: RefCell<Option<Rc<crate::ui_gamepad_library::LauncherUi>>>,
}
impl App {
@@ -58,11 +95,17 @@ impl App {
self.hosts.borrow().clone()
}
/// Surface a connect failure on the hosts page banner (toast fallback pre-build).
pub fn browse_ui(&self) -> Option<Rc<crate::ui_gamepad_library::LauncherUi>> {
self.browse.borrow().clone()
}
/// Surface a connect failure: the launcher in browse mode, else the hosts page banner
/// (toast fallback pre-build).
pub fn connect_error(&self, msg: &str) {
match self.hosts_ui() {
Some(h) => h.show_error(msg),
None => self.toast(msg),
match (self.browse_ui(), self.hosts_ui()) {
(Some(l), _) => l.show_error(msg),
(_, Some(h)) => h.show_error(msg),
_ => self.toast(msg),
}
}
}
@@ -104,6 +147,14 @@ fn build_ui(gtk_app: &adw::Application) {
}
};
load_css();
// Screenshot scenes must capture settled frames: kill every GTK/libadwaita animation
// (nav-push slides especially — a headless session may starve the frame clock and
// leave a transition frozen mid-flight in the capture).
if crate::cli::shot_scene().is_some() {
if let Some(s) = gtk::Settings::default() {
s.set_gtk_enable_animations(false);
}
}
let nav = adw::NavigationView::new();
let toasts = adw::ToastOverlay::new();
@@ -116,6 +167,14 @@ fn build_ui(gtk_app: &adw::Application) {
.content(&toasts)
.build();
let fullscreen = crate::cli::fullscreen_mode();
if fullscreen {
// Chrome-less shell: no CSD rounding/shadow (see CSS — gamescope never ACKs the
// fullscreen state, so GTK would keep them), and ask for fullscreen up front.
window.add_css_class("pf-chromeless");
window.fullscreen();
}
let app = Rc::new(App {
window: window.clone(),
nav: nav.clone(),
@@ -124,8 +183,12 @@ fn build_ui(gtk_app: &adw::Application) {
identity,
gamepad: crate::gamepad::GamepadService::start(),
busy: std::cell::Cell::new(false),
fullscreen: crate::cli::fullscreen_mode(),
fullscreen,
// (`--browse` makes cli_connect_request None — browse mode returns to the
// launcher on session end instead of quitting.)
quit_on_session_end: fullscreen && crate::cli::cli_connect_request().is_some(),
hosts: RefCell::new(None),
browse: RefCell::new(None),
});
// Re-apply the persisted forwarded-controller pin (stable key; the service matches it
@@ -138,6 +201,18 @@ fn build_ui(gtk_app: &adw::Application) {
}
}
// Browse mode (`--browse host`): the app IS the gamepad library launcher — it becomes
// the ONE root page. No hosts page (whose construction starts the mDNS browse), no
// header-menu actions; `Settings::library_enabled` is deliberately ignored (the flag
// gates the desktop menu item — asking to browse IS the opt-in here).
if let Some((req, paired, mgmt_port)) = crate::cli::cli_browse_request() {
let launcher = crate::ui_gamepad_library::open(app.clone(), req, paired, mgmt_port);
nav.add(&launcher.page);
*app.browse.borrow_mut() = Some(launcher);
window.present();
return;
}
let hosts_ui = Rc::new(crate::ui_hosts::new(
app.settings.clone(),
HostsCallbacks {
+85 -20
View File
@@ -84,19 +84,66 @@ pub fn headless_pair(pin: &str) -> glib::ExitCode {
/// already pinned at this address connects silently on its stored pin; an unknown host is
/// routed to the PIN ceremony (never a silent TOFU connect — `fp_hex`/`pair_optional` are
/// unset, so `initiate_connect`'s manual arm mandates pairing).
///
/// `--launch <id>` asks the host to launch that library title (store-qualified id from
/// `--library`, e.g. `steam:570` — the Decky wrapper's `PF_LAUNCH`); the raw id doubles
/// as the stream title (best-effort — no extra fetch just for a prettier label).
pub fn cli_connect_request() -> Option<ConnectRequest> {
if arg_value("--browse").is_some() {
return None; // browse mode owns the session lifecycle (precedence over --connect)
}
let target = std::env::args().skip_while(|a| a != "--connect").nth(1)?;
let (addr, port) = parse_host_port(&target);
// An unparsable port (`host:notaport`) used to make the whole request `None` → the app
// silently landed on the hosts page with no session and no message. Fall back to the
// native default like the add-host dialog, and say so, instead of doing nothing.
let port = port.unwrap_or_else(|| {
eprintln!("--connect: unparsable port in '{target}', using default 9777");
9777
});
Some(ConnectRequest {
name: addr.clone(),
addr,
port: port?,
port,
fp_hex: None,
pair_optional: false,
launch: None,
launch: arg_value("--launch").map(|id| (id.clone(), id)),
})
}
/// `--browse host[:port]` — open the gamepad library launcher for that host instead of
/// connecting (the Decky wrapper's `PF_BROWSE`; native port, default 9777). The host must
/// already be paired: the stored pin is what lets the launcher fetch the library and
/// connect silently — no dialog can run under gamescope, so an unpaired target renders
/// the launcher's pair-first scene. Returns the request (name + stored fingerprint from
/// the known-hosts store), whether it's paired, and the mgmt port (`--mgmt <port>`, the
/// wrapper's `PF_MGMT`; default 47990 — browse mode runs no mDNS to learn it).
pub fn cli_browse_request() -> Option<(ConnectRequest, bool, u16)> {
let target = arg_value("--browse")?;
let (addr, port) = parse_host_port(&target);
let port = port.unwrap_or(9777);
let known = crate::trust::KnownHosts::load();
let k = known
.hosts
.iter()
.find(|h| h.addr == addr && h.port == port);
let mgmt = arg_value("--mgmt")
.and_then(|p| p.parse().ok())
.unwrap_or(crate::library::DEFAULT_MGMT_PORT);
Some((
ConnectRequest {
name: k.map_or_else(|| addr.clone(), |k| k.name.clone()),
addr,
port,
fp_hex: k.map(|k| k.fp_hex.clone()),
pair_optional: false,
launch: None,
},
k.is_some_and(|k| k.paired),
mgmt,
))
}
/// `--library host[:mgmt_port]` — fetch and print the host's game library over the real
/// mTLS + pinned-fingerprint client, no GTK window (scripting, and the live-API proof
/// that the library HTTP path works against a real host). The pin comes from `--fp HEX`
@@ -219,26 +266,17 @@ pub fn run_shot(app: Rc<App>, scene: &str) {
// no-art placeholders (monogram tiles), and one solid-color texture standing in
// for a loaded poster (the real poster path, minus the network).
"library" | "08-library" => {
let game = |id: &str, store: &str, title: &str| crate::library::GameEntry {
id: id.to_string(),
store: store.to_string(),
title: title.to_string(),
art: crate::library::Artwork::default(),
};
let games = vec![
game("steam:570", "steam", "Dota 2"),
game("steam:1091500", "steam", "Cyberpunk 2077"),
game("custom:emu-1", "custom", "RetroArch"),
game("heroic:fortnite", "heroic", "Fortnite"),
game("gog:witcher3", "gog", "The Witcher 3"),
game("lutris:osu", "lutris", "osu!"),
];
let art = vec![(
"steam:570".to_string(),
solid_texture(300, 450, 0x35, 0x84, 0xe4),
)];
let (games, art) = mock_library();
crate::ui_library::open_mock(app.clone(), mock_req(), games, art);
}
// The gamepad launcher (`--browse`) with the same injected entries — cursor sits
// at 1 so both recede directions show; aurora + easing render frozen (shot mode).
"gamepad-library" | "09-gamepad-library" => {
let (games, art) = mock_library();
let ui = crate::ui_gamepad_library::open_mock(app.clone(), mock_req(), games, art);
app.nav.push(&ui.page);
*app.browse.borrow_mut() = Some(ui);
}
other => tracing::warn!("unknown PUNKTFUNK_SHOT_SCENE={other:?}; showing hosts only"),
}
@@ -268,6 +306,33 @@ pub fn run_shot(app: Rc<App>, scene: &str) {
});
}
/// The mock game set shared by the `library` and `gamepad-library` scenes: mixed stores
/// exercising the badge set, plus one solid-colour poster texture.
fn mock_library() -> (
Vec<crate::library::GameEntry>,
Vec<(String, gtk::gdk::Texture)>,
) {
let game = |id: &str, store: &str, title: &str| crate::library::GameEntry {
id: id.to_string(),
store: store.to_string(),
title: title.to_string(),
art: crate::library::Artwork::default(),
};
let games = vec![
game("steam:570", "steam", "Dota 2"),
game("steam:1091500", "steam", "Cyberpunk 2077"),
game("custom:emu-1", "custom", "RetroArch"),
game("heroic:fortnite", "heroic", "Fortnite"),
game("gog:witcher3", "gog", "The Witcher 3"),
game("lutris:osu", "lutris", "osu!"),
];
let art = vec![(
"steam:570".to_string(),
solid_texture(300, 450, 0x35, 0x84, 0xe4),
)];
(games, art)
}
/// A WxH single-colour RGBA texture — the `library` scene's stand-in for a fetched poster.
fn solid_texture(w: i32, h: i32, r: u8, g: u8, b: u8) -> gtk::gdk::Texture {
let px = [r, g, b, 0xff].repeat((w * h) as usize);
+495 -11
View File
@@ -18,6 +18,17 @@
//! idle host-list window would kill the Deck's system input. The pad list for Settings is
//! built from SDL's ID-based metadata getters, which need no open.
//!
//! **Menu mode is the one idle exception.** The gamepad library launcher (`--browse`)
//! flips [`GamepadService::set_menu_mode`] on for its lifetime: the worker then holds the
//! active pad open and translates its buttons/stick into [`MenuEvent`]s (polled off the
//! open handle each loop — Apple `GamepadMenuInput` parity: edge-triggered buttons,
//! snapshot-on-entry so a button still held from a previous screen or stream can't ghost-
//! fire, stick/dpad direction with initial-delay auto-repeat). The Valve HIDAPI drivers
//! stay OFF — a plain SDL open of the virtual X360 / evdev pad doesn't touch lizard mode —
//! and an attached session always supersedes menu translation (the stream path is
//! untouched); detach re-snapshots so the escape chord that ended the session fires
//! nothing in the menu.
//!
//! This thread is also the single consumer of the rumble and HID-output pull planes.
use punktfunk_core::client::NativeClient;
@@ -50,6 +61,169 @@ const ESCAPE_CHORD: [u32; 4] = [wire::BTN_LB, wire::BTN_RB, wire::BTN_START, wir
/// Hold the [`ESCAPE_CHORD`] at least this long to disconnect (escalates the leave-fullscreen press).
const DISCONNECT_HOLD: Duration = Duration::from_millis(1500);
/// Stick deflection below this is ignored for menu navigation (0.5 of full scale — Apple
/// `GamepadMenuInput` parity; menus want deliberate flicks, not drift).
const MENU_DEADZONE: u16 = 16384;
/// A held direction starts auto-repeating after this initial delay…
const MENU_REPEAT_DELAY: Duration = Duration::from_millis(380);
/// …and then repeats at this cadence until released or changed.
const MENU_REPEAT_INTERVAL: Duration = Duration::from_millis(160);
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MenuDir {
Up,
Down,
Left,
Right,
}
/// One controller action for the launcher UI, translated from the open pad while menu
/// mode is on and no session is attached. Buttons are edge-triggered; `Move` debounces
/// the stick/dpad and auto-repeats ([`MENU_REPEAT_DELAY`]/[`MENU_REPEAT_INTERVAL`]).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MenuEvent {
Move(MenuDir),
/// A — activate the focused item.
Confirm,
/// B — back / quit.
Back,
/// Y (Apple "secondary"; unused by the launcher today, kept for parity).
Secondary,
/// X (Apple "tertiary"; unused).
Tertiary,
/// L1 — jump back 5.
JumpBack,
/// R1 — jump forward 5.
JumpForward,
}
/// Menu haptic pulses — short rumble ticks on the menu pad (never during a stream).
#[derive(Clone, Copy, Debug)]
pub enum MenuPulse {
Move,
Confirm,
Boundary,
}
/// Raw pad state sampled once per worker iteration for menu translation.
#[derive(Clone, Copy, Default)]
struct MenuSample {
/// a, b, x, y, l1, r1 — the order [`MenuNav::poll`] maps to events.
buttons: [bool; 6],
/// Left stick, SDL convention (+y = down).
lx: i16,
ly: i16,
/// up, down, left, right.
dpad: [bool; 4],
}
/// The pure menu-input state machine (no SDL types — unit-tested below). Port of the
/// Swift client's `GamepadMenuInput`: the poll after a [`reset`](Self::reset) adopts the
/// currently-held buttons and direction WITHOUT firing, so a press that crossed a screen
/// handoff (the B that closed a stream, a held A on mode entry) must be released before
/// it can act; buttons fire on the rising edge only.
struct MenuNav {
/// Adopt the next sample silently (set on mode entry / stream detach / pad change).
snapshot_pending: bool,
/// Previous button states, [`MenuSample::buttons`] order.
was: [bool; 6],
dir: Option<MenuDir>,
/// When `dir` engaged — start of the initial-repeat delay.
dir_since: Instant,
last_repeat: Instant,
}
impl MenuNav {
fn new() -> MenuNav {
MenuNav {
snapshot_pending: true,
was: [false; 6],
dir: None,
dir_since: Instant::now(),
last_repeat: Instant::now(),
}
}
/// Arm the snapshot: the next poll adopts held state without firing.
fn reset(&mut self) {
self.snapshot_pending = true;
self.dir = None;
}
/// Direction from the left stick (dominant axis wins past the deadzone), falling back
/// to the discrete dpad. SDL sticks are +y = down.
fn resolve_dir(s: &MenuSample) -> Option<MenuDir> {
let (ax, ay) = (s.lx.unsigned_abs(), s.ly.unsigned_abs());
if ax > MENU_DEADZONE || ay > MENU_DEADZONE {
return Some(if ax >= ay {
if s.lx > 0 {
MenuDir::Right
} else {
MenuDir::Left
}
} else if s.ly > 0 {
MenuDir::Down
} else {
MenuDir::Up
});
}
let [up, down, left, right] = s.dpad;
if left {
Some(MenuDir::Left)
} else if right {
Some(MenuDir::Right)
} else if up {
Some(MenuDir::Up)
} else if down {
Some(MenuDir::Down)
} else {
None
}
}
fn poll(&mut self, s: &MenuSample, now: Instant, out: &mut Vec<MenuEvent>) {
let dir = Self::resolve_dir(s);
if self.snapshot_pending {
self.snapshot_pending = false;
self.was = s.buttons;
self.dir = dir;
self.dir_since = now;
self.last_repeat = now;
return;
}
// buttons order a, b, x, y, l1, r1 → the matching event per index.
const EVENTS: [MenuEvent; 6] = [
MenuEvent::Confirm,
MenuEvent::Back,
MenuEvent::Tertiary,
MenuEvent::Secondary,
MenuEvent::JumpBack,
MenuEvent::JumpForward,
];
for (i, ev) in EVENTS.iter().enumerate() {
if s.buttons[i] && !self.was[i] {
out.push(*ev);
}
self.was[i] = s.buttons[i];
}
if dir != self.dir {
self.dir = dir;
self.dir_since = now;
self.last_repeat = now;
if let Some(d) = dir {
out.push(MenuEvent::Move(d));
}
} else if let Some(d) = dir {
if now.duration_since(self.dir_since) >= MENU_REPEAT_DELAY
&& now.duration_since(self.last_repeat) >= MENU_REPEAT_INTERVAL
{
self.last_repeat = now;
out.push(MenuEvent::Move(d));
}
}
}
}
#[derive(Clone, Debug)]
pub struct PadInfo {
pub name: String,
@@ -114,10 +288,27 @@ fn pref_for_type(t: sdl3::gamepad::GamepadType) -> GamepadPref {
}
}
/// Best-effort "this machine is a Steam Deck". The Gaming-Mode env short-circuits; desktop
/// mode falls back to DMI (Valve board, Jupiter = LCD / Galileo = OLED — readable inside the
/// flatpak sandbox). Cached: the answer can't change while we run.
pub fn is_steam_deck() -> bool {
static DECK: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*DECK.get_or_init(|| {
if std::env::var_os("SteamDeck").is_some() {
return true;
}
let dmi = |f: &str| std::fs::read_to_string(format!("/sys/class/dmi/id/{f}"));
dmi("board_vendor").is_ok_and(|v| v.trim() == "Valve")
&& dmi("product_name").is_ok_and(|p| matches!(p.trim(), "Jupiter" | "Galileo"))
})
}
enum Ctl {
Attach(Arc<NativeClient>),
Detach,
Pin(Option<String>),
MenuMode(bool),
MenuRumble(MenuPulse),
}
#[derive(Clone)]
@@ -131,6 +322,9 @@ pub struct GamepadService {
/// Fires once when the [`ESCAPE_CHORD`] is held past [`DISCONNECT_HOLD`]; the stream page
/// consumes it to end the session (the controller equivalent of Ctrl+Alt+Shift+D).
disconnect_rx: async_channel::Receiver<()>,
/// Menu-navigation events while menu mode is on and no session is attached; the
/// launcher page consumes them.
menu_rx: async_channel::Receiver<MenuEvent>,
}
impl GamepadService {
@@ -140,11 +334,12 @@ impl GamepadService {
let (ctl, ctl_rx) = std::sync::mpsc::channel();
let (escape_tx, escape_rx) = async_channel::unbounded();
let (disconnect_tx, disconnect_rx) = async_channel::unbounded();
let (menu_tx, menu_rx) = async_channel::unbounded();
let (p, a) = (pads.clone(), active.clone());
if let Err(e) = std::thread::Builder::new()
.name("punktfunk-gamepad".into())
.spawn(move || {
if let Err(e) = run(&p, &a, &ctl_rx, &escape_tx, &disconnect_tx) {
if let Err(e) = run(&p, &a, &ctl_rx, &escape_tx, &disconnect_tx, &menu_tx) {
tracing::warn!(error = %e, "gamepad service ended — pads disabled");
}
})
@@ -157,6 +352,7 @@ impl GamepadService {
ctl,
escape_rx,
disconnect_rx,
menu_rx,
}
}
@@ -172,6 +368,25 @@ impl GamepadService {
self.disconnect_rx.clone()
}
/// Menu-navigation events ([`MenuEvent`]) — flowing only while menu mode is on and no
/// session is attached. A fresh clone per call; the launcher spawns a future on it.
pub fn menu_events(&self) -> async_channel::Receiver<MenuEvent> {
self.menu_rx.clone()
}
/// Turn menu mode on/off: while on (and no session attached) the worker holds the
/// active pad open and translates it into [`MenuEvent`]s. The launcher flips this on
/// once for its lifetime — an attached session supersedes translation automatically.
pub fn set_menu_mode(&self, on: bool) {
let _ = self.ctl.send(Ctl::MenuMode(on));
}
/// Play a short menu haptic tick on the menu pad (no-op while a session is attached
/// or no pad is open; best-effort on pads without rumble).
pub fn menu_rumble(&self, pulse: MenuPulse) {
let _ = self.ctl.send(Ctl::MenuRumble(pulse));
}
pub fn pads(&self) -> Vec<PadInfo> {
self.pads.lock().unwrap().clone()
}
@@ -197,8 +412,19 @@ impl GamepadService {
/// What "Automatic" resolves to right now — the virtual pad matching the physical one
/// (Swift parity); no pad connected leaves the host's own default.
///
/// **Steam Deck special case:** this is read at session start, *before* attach — but the
/// Deck's built-in controller is only enumerable with its real 28DE:1205 identity while
/// the Valve HIDAPI drivers run, and those are enabled on attach only (see
/// [`set_valve_hidapi`]); with Steam Input on, SDL sees nothing but Steam's virtual
/// X360 pad anyway. Both cases used to fall through to Xbox 360. On a Deck, a virtual
/// pad (or no pad at all) means the physical controller behind it IS the built-in one —
/// resolve to the Steam Deck virtual pad so the paddles/trackpads/gyro have somewhere
/// to land. A real external controller still wins (it's the one that gets forwarded).
pub fn auto_pref(&self) -> GamepadPref {
match self.active() {
Some(p) if !p.steam_virtual => p.pref,
_ if is_steam_deck() => GamepadPref::SteamDeck,
Some(p) => p.pref,
None => GamepadPref::Auto,
}
@@ -337,6 +563,11 @@ struct Worker<'a> {
chord_since: Option<Instant>,
/// The disconnect signal already fired for the current hold — latched so it fires once.
disconnect_fired: bool,
/// Menu mode ([`GamepadService::set_menu_mode`]): hold the active pad open while idle
/// and translate it into [`MenuEvent`]s. An attached session pauses translation.
menu_mode: bool,
menu_nav: MenuNav,
menu_tx: async_channel::Sender<MenuEvent>,
}
impl Worker<'_> {
@@ -395,12 +626,12 @@ impl Worker<'_> {
})
}
/// Hold exactly the right device: the active pad while a session is attached, nothing
/// otherwise. The single place that decides to open (= grab) hardware; dropping the
/// old handle closes it (`SDL_CloseGamepad`) — on a Deck the firmware watchdog then
/// restores lizard mode.
/// Hold exactly the right device: the active pad while a session is attached or menu
/// mode owns navigation, nothing otherwise. The single place that decides to open
/// (= grab) hardware; dropping the old handle closes it (`SDL_CloseGamepad`) — on a
/// Deck the firmware watchdog then restores lizard mode.
fn sync_open(&mut self) {
let want = if self.attached.is_some() {
let want = if self.attached.is_some() || self.menu_mode {
self.active_id()
} else {
None
@@ -413,7 +644,15 @@ impl Worker<'_> {
match self.subsystem.open(sdl3::sys::joystick::SDL_JoystickID(id)) {
Ok(pad) => {
self.open = Some((id, pad));
self.set_sensors(true);
// Sensors stream only for an attached session (USB/BT bandwidth); the
// menu needs buttons + stick only.
if self.attached.is_some() {
self.set_sensors(true);
} else {
// The menu pad changed under us (hot-plug while the launcher is
// open): adopt the new pad's held state instead of firing it.
self.menu_nav.reset();
}
}
Err(e) => tracing::warn!(id, error = %e, "gamepad open failed"),
}
@@ -619,14 +858,42 @@ impl Worker<'_> {
Ok(Ctl::Detach) => {
self.flush_held();
self.attached = None;
self.sync_open(); // closes the held device
self.sync_open(); // closes the held device (menu mode keeps it)
set_valve_hidapi(false);
if self.menu_mode {
// Back to the launcher: adopt whatever is still physically held
// (the escape chord that ended the session, a lingering B) so it
// can't ghost-fire menu actions.
self.menu_nav.reset();
}
}
Ok(Ctl::Pin(key)) => {
let before = self.active_id();
self.pinned = key;
self.refresh_active(before);
}
Ok(Ctl::MenuMode(on)) => {
self.menu_mode = on;
if on {
self.menu_nav.reset();
}
self.sync_open();
}
Ok(Ctl::MenuRumble(pulse)) => {
if self.attached.is_none() {
if let Some((_, pad)) = self.open.as_mut() {
let (low, high, ms) = match pulse {
// Light high-freq detent — won't jackhammer at repeat rate.
MenuPulse::Move => (0, 0x3000, 25),
// Fuller both-motor thunk.
MenuPulse::Confirm => (0x5000, 0x5000, 60),
// Dull low-freq wall.
MenuPulse::Boundary => (0x6000, 0, 60),
};
let _ = pad.set_rumble(low, high, ms);
}
}
}
Err(std::sync::mpsc::TryRecvError::Empty) => return true,
Err(std::sync::mpsc::TryRecvError::Disconnected) => return false, // app gone
}
@@ -643,7 +910,16 @@ impl Worker<'_> {
if !self.order.contains(&which) {
self.order.push(which);
if let Some(p) = self.pad_info(which) {
tracing::info!(name = p.name, "gamepad attached");
// Full identity: on a Steam Deck this is the one lever for diagnosing an
// empty controller list — it tells you whether SDL sees the physical pad
// (28DE:1205), Steam Input's virtual pad (28DE:11FF), both, or nothing.
tracing::info!(
name = p.name,
key = p.key,
pref = ?p.pref,
steam_virtual = p.steam_virtual,
"gamepad attached"
);
}
self.refresh_active(active);
}
@@ -758,6 +1034,42 @@ impl Worker<'_> {
}
}
/// Sample the open pad and translate it into [`MenuEvent`]s — only while menu mode is
/// on and no session is attached (attach supersedes; SDL events merely wake the loop,
/// so a press is translated the iteration it arrives).
fn menu_poll(&mut self) {
if !self.menu_mode || self.attached.is_some() {
return;
}
let Some((_, pad)) = self.open.as_ref() else {
return;
};
use sdl3::gamepad::{Axis, Button};
let s = MenuSample {
buttons: [
pad.button(Button::South),
pad.button(Button::East),
pad.button(Button::West),
pad.button(Button::North),
pad.button(Button::LeftShoulder),
pad.button(Button::RightShoulder),
],
lx: pad.axis(Axis::LeftX),
ly: pad.axis(Axis::LeftY),
dpad: [
pad.button(Button::DPadUp),
pad.button(Button::DPadDown),
pad.button(Button::DPadLeft),
pad.button(Button::DPadRight),
],
};
let mut out = Vec::new();
self.menu_nav.poll(&s, Instant::now(), &mut out);
for e in out {
let _ = self.menu_tx.try_send(e);
}
}
/// Drain and render the feedback planes — rumble plus HID output (lightbar /
/// player LEDs / adaptive triggers) — on the active pad; this thread is their single
/// consumer. The host re-sends rumble state periodically, so a generous duration with
@@ -821,6 +1133,7 @@ fn run(
ctl: &Receiver<Ctl>,
escape_tx: &async_channel::Sender<()>,
disconnect_tx: &async_channel::Sender<()>,
menu_tx: &async_channel::Sender<MenuEvent>,
) -> Result<(), String> {
// Off-main-thread + no video subsystem: keep SDL away from signals, poll pads on its
// own thread.
@@ -851,6 +1164,9 @@ fn run(
chord_armed: false,
chord_since: None,
disconnect_fired: false,
menu_mode: false,
menu_nav: MenuNav::new(),
menu_tx: menu_tx.clone(),
};
loop {
@@ -865,8 +1181,13 @@ fn run(
// rumble/HID feedback, and the escape-chord hold check all run once per wakeup,
// so their worst case is one timeout (~10 ms attached, imperceptible for
// haptics; DISCONNECT_HOLD is 1500 ms, so 10 ms hold-check granularity is far
// inside tolerance). Idle (no session) wakes lazily at 30 ms for hotplug + ctl.
let timeout = Duration::from_millis(if w.attached.is_some() { 10 } else { 30 });
// inside tolerance; menu mode needs the same cadence for its repeat timing).
// Idle (no session, no menu) wakes lazily at 30 ms for hotplug + ctl.
let timeout = Duration::from_millis(if w.attached.is_some() || w.menu_mode {
10
} else {
30
});
if let Some(event) = pump.wait_event_timeout(timeout) {
w.handle_event(event);
// Drain whatever else queued while we were waiting or handling.
@@ -879,6 +1200,169 @@ fn run(
// new button events; the chord itself is only detected while a session is attached).
w.maybe_fire_disconnect();
w.menu_poll();
w.render_feedback();
}
}
#[cfg(test)]
mod menu_nav_tests {
use super::*;
fn sample() -> MenuSample {
MenuSample::default()
}
fn events(nav: &mut MenuNav, s: &MenuSample, at: Instant) -> Vec<MenuEvent> {
let mut out = Vec::new();
nav.poll(s, at, &mut out);
out
}
#[test]
fn snapshot_adopts_held_state_without_firing() {
let mut nav = MenuNav::new();
let t = Instant::now();
let mut held = sample();
held.buttons[0] = true; // A held on entry
held.lx = 30000; // stick already deflected right
assert!(events(&mut nav, &held, t).is_empty(), "snapshot poll fired");
// Still held: nothing (no rising edge, direction unchanged since snapshot).
assert!(events(&mut nav, &held, t + Duration::from_millis(10)).is_empty());
// Release, then press again → now it fires.
assert!(events(&mut nav, &sample(), t + Duration::from_millis(20)).is_empty());
assert_eq!(
events(&mut nav, &held, t + Duration::from_millis(30)),
vec![MenuEvent::Confirm, MenuEvent::Move(MenuDir::Right)]
);
}
#[test]
fn buttons_fire_on_rising_edge_only() {
let mut nav = MenuNav::new();
let t = Instant::now();
events(&mut nav, &sample(), t); // consume the snapshot
let mut s = sample();
s.buttons[1] = true; // B down
assert_eq!(
events(&mut nav, &s, t + Duration::from_millis(10)),
vec![MenuEvent::Back]
);
for i in 2..20 {
assert!(
events(&mut nav, &s, t + Duration::from_millis(10 * i)).is_empty(),
"held button re-fired"
);
}
}
#[test]
fn reset_rearms_the_snapshot() {
let mut nav = MenuNav::new();
let t = Instant::now();
events(&mut nav, &sample(), t);
nav.reset();
let mut s = sample();
s.buttons[1] = true;
assert!(
events(&mut nav, &s, t + Duration::from_millis(10)).is_empty(),
"post-reset poll fired a held button"
);
}
#[test]
fn direction_repeats_after_delay_at_interval() {
let mut nav = MenuNav::new();
let t = Instant::now();
events(&mut nav, &sample(), t);
let mut s = sample();
s.dpad[3] = true; // dpad right
// Engage: fires immediately.
assert_eq!(
events(&mut nav, &s, t + Duration::from_millis(10)),
vec![MenuEvent::Move(MenuDir::Right)]
);
// Inside the initial delay: silent.
assert!(events(&mut nav, &s, t + Duration::from_millis(300)).is_empty());
// Past the delay: repeats…
assert_eq!(
events(&mut nav, &s, t + Duration::from_millis(400)),
vec![MenuEvent::Move(MenuDir::Right)]
);
// …but not faster than the interval…
assert!(events(&mut nav, &s, t + Duration::from_millis(500)).is_empty());
// …and again once it elapses.
assert_eq!(
events(&mut nav, &s, t + Duration::from_millis(570)),
vec![MenuEvent::Move(MenuDir::Right)]
);
// Release cancels; re-engage fires immediately again.
assert!(events(&mut nav, &sample(), t + Duration::from_millis(580)).is_empty());
assert_eq!(
events(&mut nav, &s, t + Duration::from_millis(590)),
vec![MenuEvent::Move(MenuDir::Right)]
);
}
#[test]
fn direction_change_fires_immediately() {
let mut nav = MenuNav::new();
let t = Instant::now();
events(&mut nav, &sample(), t);
let mut right = sample();
right.lx = 30000;
let mut left = sample();
left.lx = -30000;
assert_eq!(
events(&mut nav, &right, t + Duration::from_millis(10)),
vec![MenuEvent::Move(MenuDir::Right)]
);
assert_eq!(
events(&mut nav, &left, t + Duration::from_millis(20)),
vec![MenuEvent::Move(MenuDir::Left)]
);
}
#[test]
fn direction_resolution() {
// Below the deadzone: nothing.
let mut s = sample();
s.lx = MENU_DEADZONE as i16;
assert_eq!(MenuNav::resolve_dir(&s), None);
// Dominant axis wins; SDL +y = down.
s.lx = 20000;
s.ly = 25000;
assert_eq!(MenuNav::resolve_dir(&s), Some(MenuDir::Down));
s.ly = -25000;
assert_eq!(MenuNav::resolve_dir(&s), Some(MenuDir::Up));
s.lx = 26000;
assert_eq!(MenuNav::resolve_dir(&s), Some(MenuDir::Right));
s.lx = -26000;
assert_eq!(MenuNav::resolve_dir(&s), Some(MenuDir::Left));
// Dpad fallback…
let mut d = sample();
d.dpad[1] = true;
assert_eq!(MenuNav::resolve_dir(&d), Some(MenuDir::Down));
// …but the stick overrides it.
d.lx = 30000;
assert_eq!(MenuNav::resolve_dir(&d), Some(MenuDir::Right));
}
#[test]
fn shoulder_and_face_button_mapping() {
let mut nav = MenuNav::new();
let t = Instant::now();
events(&mut nav, &sample(), t);
let mut s = sample();
s.buttons = [false, false, true, true, true, true]; // x, y, l1, r1
assert_eq!(
events(&mut nav, &s, t + Duration::from_millis(10)),
vec![
MenuEvent::Tertiary,
MenuEvent::Secondary,
MenuEvent::JumpBack,
MenuEvent::JumpForward,
]
);
}
}
+35 -4
View File
@@ -266,6 +266,9 @@ impl SessionUi {
inhibit_shortcuts: self.inhibit,
show_stats: self.show_stats,
chromeless: self.app.fullscreen,
// The attach just went out, so a Deck's built-in pad may not have enumerated
// yet — chromeless (controller-first) shows the chord hint regardless.
pad_connected: self.app.gamepad.active().is_some(),
title,
});
self.app.nav.push(&p.page);
@@ -296,21 +299,49 @@ impl SessionUi {
}
// A pinned connect rejected on trust grounds means the host's cert no
// longer matches the stored pin (rotated cert or impostor) — route to
// the PIN ceremony to re-establish trust rather than dead-ending.
if trust_rejected && !self.tofu {
// the PIN ceremony to re-establish trust rather than dead-ending. Browse
// mode can't: gamescope never maps dialogs, so it renders the advice instead
// (re-pairing is the plugin's job there).
if trust_rejected && !self.tofu && self.app.browse_ui().is_none() {
self.app
.toast("Host fingerprint changed — re-pair with a PIN to continue");
crate::ui_trust::pin_dialog(self.app.clone(), self.req.clone());
} else if trust_rejected && !self.tofu {
self.app
.connect_error("Host identity changed — re-pair from the Punktfunk plugin.");
} else {
// Errors land on the hosts page banner, not a transient toast.
// Errors land on the hosts page banner / launcher strip, not a transient toast.
self.app.connect_error(&format!("Couldn't connect — {msg}"));
}
}
/// `Ended`: detach gamepads, pop back to the hosts page, and surface the reason.
/// `Ended`: detach gamepads, pop back to the launcher (browse mode) or the hosts
/// page, and surface the reason.
fn on_ended(&mut self, err: Option<String>) {
self.close_waiting();
self.app.gamepad.detach();
// Gaming-Mode `--connect` launch: the app IS the stream. Quit so Steam ends the
// "game" and the Deck returns to Gaming Mode — popping to our own hosts page would
// strand the user in a fullscreen shell with no way back.
if self.app.quit_on_session_end {
if let Some(e) = err {
tracing::warn!(error = %e, "session ended");
}
self.app.window.close();
return;
}
// Browse mode: back to the launcher to pick the next game — B there quits to
// Gaming Mode. (The gamepad worker re-opened the pad and armed the held-state
// snapshot on the detach above, so the chord that ended the session fires nothing.)
if let Some(l) = self.app.browse_ui() {
self.app.nav.pop_to_tag("launcher");
l.on_session_ended();
if let Some(e) = err {
self.app.connect_error(&e);
}
self.app.busy.set(false);
return;
}
self.app.nav.pop_to_tag("hosts");
if let Some(h) = self.app.hosts_ui() {
h.set_connecting(None);
+53 -1
View File
@@ -6,8 +6,9 @@
//! verified by its pinned SHA-256 fingerprint (`KnownHost::fp_hex`), not a CA chain.
use serde::Deserialize;
use std::collections::VecDeque;
use std::io::Read;
use std::sync::Arc;
use std::sync::{Arc, Mutex};
use std::time::Duration;
/// The management API's default port — matches `mgmt::DEFAULT_PORT` on the host. A
@@ -181,6 +182,57 @@ pub fn fetch_art(pinned: &ureq::Agent, base: &str, url: &str) -> Result<Vec<u8>,
Ok(bytes)
}
/// Concurrent poster fetches — a handful is plenty for a LAN art proxy without turning a
/// big library into a connection burst.
const ART_WORKERS: usize = 3;
/// Fetch poster bytes for `jobs` (entry id → candidate URLs, walked in order until one
/// loads) on a small worker pool; results stream on the returned channel as they land.
/// Dropping the receiver (the consuming page popped) winds the workers down. Shared by
/// the touch grid and the gamepad launcher — the consumer does its own texture decode on
/// the main loop.
pub fn spawn_art_fetch(
base: String,
identity: (String, String),
pin: Option<[u8; 32]>,
jobs: VecDeque<(String, Vec<String>)>,
) -> async_channel::Receiver<(String, Vec<u8>)> {
let queue = Arc::new(Mutex::new(jobs));
let (tx, rx) = async_channel::unbounded::<(String, Vec<u8>)>();
for _ in 0..ART_WORKERS {
let queue = queue.clone();
let tx = tx.clone();
let base = base.clone();
let identity = identity.clone();
std::thread::Builder::new()
.name("punktfunk-lib-art".into())
.spawn(move || {
let Ok(agent) = agent(&identity, pin) else {
return;
};
loop {
let job = queue.lock().unwrap().pop_front();
let Some((id, candidates)) = job else { break };
for url in &candidates {
match fetch_art(&agent, &base, url) {
Ok(bytes) => {
// Receiver gone (page popped) — stop fetching.
if tx.send_blocking((id, bytes)).is_err() {
return;
}
break;
}
// 404 on a guessed CDN path is routine — try the next kind.
Err(e) => tracing::debug!(%id, url, error = %e, "poster miss"),
}
}
}
})
.expect("spawn art thread");
}
rx
}
fn classify(e: ureq::Error) -> LibraryError {
match e {
ureq::Error::Status(401 | 403, _) => LibraryError::NotPaired,
+2
View File
@@ -26,6 +26,8 @@ mod session;
#[cfg(target_os = "linux")]
mod trust;
#[cfg(target_os = "linux")]
mod ui_gamepad_library;
#[cfg(target_os = "linux")]
mod ui_hosts;
#[cfg(target_os = "linux")]
mod ui_library;
+125 -21
View File
@@ -45,18 +45,55 @@ pub struct SessionParams {
pub connect_timeout: Duration,
}
/// The session pump's share of the unified stats window (design/stats-unification.md):
/// stream facts plus the two stages measured before the presenter. The frame consumer in
/// `ui_stream` contributes the `display` stage and the end-to-end percentiles.
#[derive(Clone, Copy, Default)]
pub struct Stats {
/// AUs received (reassembled) per second, actual-elapsed-time denominator.
pub fps: f32,
/// Received payload bytes × 8 / elapsed (goodput, excludes FEC overhead).
pub mbps: f32,
/// p50 `host+network` stage: capture → received, host-clock corrected (ms).
pub host_net_ms: f32,
/// p50 `host` stage: the host's own capture→fully-sent, from the per-AU 0xCF host
/// timings (design/stats-unification.md Phase 2). Valid only when `split`.
pub host_ms: f32,
/// p50 `network` stage: capture→received minus the host-reported share
/// (`hostnet host`, per-frame, saturating). Valid only when `split`.
pub net_ms: f32,
/// The window had matched host timings — the OSD splits `host+network` into
/// `host + network`. An old host never emits 0xCF, so this stays false and the
/// combined stage renders unchanged.
pub split: bool,
/// p50 `decode` stage: received → decoded, single-clock client-local (ms).
pub decode_ms: f32,
/// Median capture→decoded latency over the last window (host-clock corrected).
pub latency_ms: f32,
/// Unrecoverable network frame drops this window, and their share of
/// received+lost (%). The OSD renders the counter line only when nonzero.
pub lost: u32,
pub lost_pct: f32,
/// The decode path frames actually took this window (`"vaapi"`/`"software"`, empty
/// until the first frame) — the OSD's trailing tag; tracks a mid-session fallback.
pub decoder: &'static str,
}
/// Frames the pump keeps waiting for their 0xCF host timing (pts → capture→received µs).
/// ~2 s at 120 Hz — a timing arrives within a frame or two of its AU, and against an old
/// host (no 0xCF at all) this just caps the dead-weight ring.
const PENDING_SPLIT_CAP: usize = 256;
/// Sort a window of µs samples in place and return `(p50, p95)` per the spec's index
/// rules (`sorted[len/2]`, `sorted[min(len*95/100, len-1)]`); an empty window reads 0.
pub fn window_percentiles(samples: &mut [u64]) -> (u64, u64) {
if samples.is_empty() {
return (0, 0);
}
samples.sort_unstable();
let p50 = samples[samples.len() / 2];
let p95 = samples[(samples.len() * 95 / 100).min(samples.len() - 1)];
(p50, p95)
}
pub enum SessionEvent {
Connected {
connector: Arc<NativeClient>,
@@ -219,13 +256,23 @@ fn pump(
let mut window_start = Instant::now();
let mut frames_n = 0u32;
let mut bytes_n = 0u64;
let mut decode_us_sum = 0u64;
let mut lat_us: Vec<u64> = Vec::with_capacity(256);
// Stage windows (µs samples): `host+network` = capture→received (host-clock
// corrected), `decode` = received→decoded (client-local). p50 per 1 s window.
let mut hostnet_us: Vec<u64> = Vec::with_capacity(256);
let mut decode_us: Vec<u64> = Vec::with_capacity(256);
// Host/network split (Phase 2): frames awaiting their per-AU 0xCF host timing,
// correlated by pts_ns. Bounded — an old host never sends any, so entries just age out.
let mut pending_split: std::collections::VecDeque<(u64, u64)> =
std::collections::VecDeque::with_capacity(PENDING_SPLIT_CAP);
let mut host_us_win: Vec<u64> = Vec::with_capacity(256);
let mut net_us_win: Vec<u64> = Vec::with_capacity(256);
// What actually decoded the last frame — a VAAPI failure demotes mid-session, so
// this is read off each frame's image variant rather than fixed at startup.
let mut dec_path: &'static str = "";
// Loss recovery: watch the host→client unrecoverable-drop count and ask for an IDR when it climbs.
let mut last_dropped = connector.frames_dropped();
// The stats window keeps its own drop cursor — the OSD shows the per-window delta.
let mut window_dropped = last_dropped;
let mut last_kf_req: Option<Instant> = None;
let end: Option<String> = loop {
@@ -237,7 +284,11 @@ fn pump(
// every ~816 ms at 60120 Hz anyway, so this rarely times out mid-stream).
match connector.next_frame(Duration::from_millis(20)) {
Ok(frame) => {
let t0 = Instant::now();
// The `received` point: AU fully reassembled, in hand, before decode.
let received_ns = now_ns();
// fps / goodput count every received AU (spec), decoded or not.
frames_n += 1;
bytes_n += frame.data.len() as u64;
match decoder.decode(&frame.data) {
Ok(Some(image)) => {
total_frames += 1;
@@ -252,18 +303,27 @@ fn pump(
};
tracing::info!(width = w, height = h, path, "first frame decoded");
}
// Latency: our wall clock expressed in the host's capture clock,
// minus the host-stamped capture pts (same math as client-rs).
let lat = (now_ns() as i128 + clock_offset as i128 - frame.pts_ns as i128)
// The `decoded` point — travels with the frame so the presenter
// can measure its `display` stage against it.
let decoded_ns = now_ns();
// `host+network` stage: received expressed in the host's capture
// clock, minus the host-stamped capture pts (clamped (0, 10 s)).
let hn = (received_ns as i128 + clock_offset as i128 - frame.pts_ns as i128)
.max(0) as u64;
if lat > 0 && lat < 10_000_000_000 {
lat_us.push(lat / 1000);
if hn > 0 && hn < 10_000_000_000 {
hostnet_us.push(hn / 1000);
// Remember the sample for the host/network split — matched
// against the AU's 0xCF host timing when it arrives.
if pending_split.len() >= PENDING_SPLIT_CAP {
pending_split.pop_front();
}
pending_split.push_back((frame.pts_ns, hn / 1000));
}
decode_us_sum += t0.elapsed().as_micros() as u64;
frames_n += 1;
bytes_n += frame.data.len() as u64;
// `decode` stage: received→decoded, single clock, no skew.
decode_us.push(decoded_ns.saturating_sub(received_ns) / 1000);
let _ = frame_tx.force_send(DecodedFrame {
pts_ns: frame.pts_ns,
decoded_ns,
image,
});
}
@@ -271,12 +331,39 @@ fn pump(
// Survivable (loss until the next IDR/RFI recovery) — keep feeding.
Err(e) => tracing::debug!(error = %e, "decode error (recovering)"),
}
// A decode error / VAAPI→software demotion asks for a fresh IDR: the infinite
// GOP has no periodic keyframe, so a rebuilt/erroring decoder would stay
// gray/frozen until an unrelated packet drop happened to request one. Route it
// through the same throttle as loss recovery below.
if decoder.take_keyframe_request() {
let now = Instant::now();
if last_kf_req
.is_none_or(|t| now.duration_since(t) >= Duration::from_millis(100))
{
last_kf_req = Some(now);
let _ = connector.request_keyframe();
tracing::debug!("requested keyframe (decoder recovery)");
}
}
}
Err(PunktfunkError::NoFrame) => {}
Err(PunktfunkError::Closed) => break Some("Host ended the session".to_string()),
Err(e) => break Some(format!("session: {e:?}")),
}
// Drain the per-AU host timings (0xCF) non-blockingly and match them to received
// frames by pts: host = the host's own capture→sent, network = our
// capture→received minus it (the two tile per frame by construction). An old
// host never emits any — the deque fills to its cap and the OSD keeps the
// combined `host+network` stage.
while let Ok(t) = connector.next_host_timing(Duration::ZERO) {
if let Some(i) = pending_split.iter().position(|(p, _)| *p == t.pts_ns) {
let (_, hn_us) = pending_split.remove(i).unwrap();
host_us_win.push(t.host_us as u64);
net_us_win.push(hn_us.saturating_sub(t.host_us as u64));
}
}
// Loss recovery: under infinite GOP the only recovery keyframe is one we request. The
// reassembler drops unrecoverable AUs (frames_dropped); the decoder then conceals the
// reference-missing delta frames that follow and returns Ok, so keying off a decode error
@@ -295,30 +382,47 @@ fn pump(
if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32();
lat_us.sort_unstable();
let p50 = lat_us.get(lat_us.len() / 2).copied().unwrap_or(0);
let (hn_p50, _) = window_percentiles(&mut hostnet_us);
let (dec_p50, _) = window_percentiles(&mut decode_us);
// Host/network split — present only when this window matched 0xCF timings.
let split = !host_us_win.is_empty();
let (host_p50, _) = window_percentiles(&mut host_us_win);
let (net_p50, _) = window_percentiles(&mut net_us_win);
let lost = dropped.saturating_sub(window_dropped) as u32;
window_dropped = dropped;
tracing::debug!(
fps = frames_n,
lat_p50_us = p50,
hostnet_p50_us = hn_p50,
host_p50_us = host_p50,
net_p50_us = net_p50,
decode_p50_us = dec_p50,
lost,
total_frames,
"stream window"
);
let _ = ev_tx.try_send(SessionEvent::Stats(Stats {
fps: frames_n as f32 / secs,
mbps: bytes_n as f32 * 8.0 / 1e6 / secs,
decode_ms: if frames_n > 0 {
decode_us_sum as f32 / frames_n as f32 / 1000.0
host_net_ms: hn_p50 as f32 / 1000.0,
host_ms: host_p50 as f32 / 1000.0,
net_ms: net_p50 as f32 / 1000.0,
split,
decode_ms: dec_p50 as f32 / 1000.0,
lost,
lost_pct: if lost > 0 {
lost as f32 * 100.0 / (frames_n + lost) as f32
} else {
0.0
},
latency_ms: p50 as f32 / 1000.0,
decoder: dec_path,
}));
window_start = Instant::now();
frames_n = 0;
bytes_n = 0;
decode_us_sum = 0;
lat_us.clear();
hostnet_us.clear();
decode_us.clear();
host_us_win.clear();
net_us_win.clear();
}
};
File diff suppressed because it is too large Load Diff
+9
View File
@@ -153,6 +153,15 @@ pub fn new(settings: Rc<RefCell<Settings>>, cbs: HostsCallbacks) -> HostsUi {
let disc_heading = heading("On this network");
let disc_flow = make_flow();
// A pointer click (and keyboard activate) emits `child-activated` on the *FlowBox*, never
// the child's own `activate` signal — so bridge it back to the child, where each card wires
// its connect handler (`saved_card`/`discovered_card`). Without this, clicking a card is dead.
for flow in [&saved_flow, &disc_flow] {
flow.connect_child_activated(|_, child| {
child.activate();
});
}
// Shown under the discovered heading while no (unsaved) advert is live yet.
let searching = gtk::Box::new(gtk::Orientation::Horizontal, 8);
let spinner = gtk::Spinner::new();
+10 -40
View File
@@ -14,11 +14,6 @@ use gtk::{gdk, glib};
use std::cell::{Cell, RefCell};
use std::collections::{HashMap, VecDeque};
use std::rc::Rc;
use std::sync::{Arc, Mutex};
/// Concurrent poster fetches — a handful is plenty for a LAN art proxy without turning a
/// big library into a connection burst.
const ART_WORKERS: usize = 3;
/// Everything the page re-renders from. Kept alive by the widget closures (reload/retry/
/// card activation); dropped when the page is popped, which also winds down any in-flight
@@ -76,6 +71,11 @@ fn build(app: Rc<App>, req: ConnectRequest) -> Rc<State> {
.row_spacing(18)
.valign(gtk::Align::Start)
.build();
// Click/keyboard activation fires `child-activated` on the FlowBox, not the child's own
// `activate` — bridge it so each poster's connect handler (below) runs on click.
flow.connect_child_activated(|_, child| {
child.activate();
});
let content = gtk::Box::new(gtk::Orientation::Vertical, 0);
content.set_margin_top(24);
content.set_margin_bottom(24);
@@ -295,39 +295,7 @@ fn load_art(state: &Rc<State>, games: &[GameEntry]) {
}
let identity = state.app.identity.clone();
let pin = state.req.fp_hex.as_deref().and_then(trust::parse_hex32);
let queue = Arc::new(Mutex::new(jobs));
let (tx, rx) = async_channel::unbounded::<(String, Vec<u8>)>();
for _ in 0..ART_WORKERS {
let queue = queue.clone();
let tx = tx.clone();
let base = base.clone();
let identity = identity.clone();
std::thread::Builder::new()
.name("punktfunk-lib-art".into())
.spawn(move || {
let Ok(agent) = library::agent(&identity, pin) else {
return;
};
loop {
let job = queue.lock().unwrap().pop_front();
let Some((id, candidates)) = job else { break };
for url in &candidates {
match library::fetch_art(&agent, &base, url) {
Ok(bytes) => {
// Receiver gone (page popped) — stop fetching.
if tx.send_blocking((id, bytes)).is_err() {
return;
}
break;
}
// 404 on a guessed CDN path is routine — try the next kind.
Err(e) => tracing::debug!(%id, url, error = %e, "poster miss"),
}
}
}
})
.expect("spawn art thread");
}
let rx = library::spawn_art_fetch(base, identity, pin, jobs);
let weak = Rc::downgrade(state);
glib::spawn_future_local(async move {
while let Ok((id, bytes)) = rx.recv().await {
@@ -349,7 +317,8 @@ fn load_art(state: &Rc<State>, games: &[GameEntry]) {
/// The store badge text — `store` comes from the entry (today `steam`/`custom`; future
/// stores per the host's provider list), with the id prefix as a fallback spelling.
fn store_label(store: &str) -> &'static str {
/// Shared with the gamepad launcher's posters.
pub fn store_label(store: &str) -> &'static str {
match store {
"steam" => "Steam",
"custom" => "Custom",
@@ -363,7 +332,8 @@ fn store_label(store: &str) -> &'static str {
}
/// Monogram for the placeholder tile: the first letters of the first two words.
fn initials(title: &str) -> String {
/// Shared with the gamepad launcher's posters.
pub fn initials(title: &str) -> String {
title
.split_whitespace()
.take(2)
+9 -1
View File
@@ -16,7 +16,14 @@ const RESOLUTIONS: &[(u32, u32)] = &[
];
/// `0` = the monitor's native refresh, resolved at connect.
const REFRESH: &[u32] = &[0, 30, 60, 90, 120, 144, 165, 240];
const GAMEPADS: &[&str] = &["auto", "xbox360", "dualsense", "xboxone", "dualshock4"];
const GAMEPADS: &[&str] = &[
"auto",
"xbox360",
"dualsense",
"xboxone",
"dualshock4",
"steamdeck",
];
const COMPOSITORS: &[&str] = &["auto", "kwin", "wlroots", "mutter", "gamescope"];
/// Codec setting values (persisted) paired with their display labels below.
const CODECS: &[&str] = &["auto", "hevc", "h264", "av1"];
@@ -403,6 +410,7 @@ pub fn show(
"DualSense",
"Xbox One",
"DualShock 4",
"Steam Deck",
],
);
let inhibit_row = adw::SwitchRow::builder()
+224 -67
View File
@@ -31,33 +31,78 @@ use std::time::{Duration, Instant};
pub struct StreamPage {
pub page: adw::NavigationPage,
stats_label: gtk::Label,
/// Median capture→paintable-set latency (ms) over the frame consumer's last 1 s
/// window — written there, folded into the OSD on each `Stats` event.
present_ms: Rc<Cell<f32>>,
/// The frame consumer's share of the stats window (end-to-end percentiles + the
/// `display` stage) — written there each 1 s window, folded into the OSD on each
/// `Stats` event.
presented: Rc<PresentedStats>,
/// The stream is HDR (PQ) right now — set by the frame consumer from each frame's
/// signaling (the host can flip SDR↔HDR mid-session, in-band).
hdr: Rc<Cell<bool>>,
/// `clock_offset_ns == 0`: the skew handshake didn't run (or same host) — the
/// end-to-end line carries the `(same-host clock)` flag (spec clock rules).
same_host: bool,
/// `W×H@Hz` for the OSD's first line — fixed at connect, per-session.
mode_line: String,
}
/// Presenter-side window results (design/stats-unification.md): end-to-end =
/// capture→displayed measured directly (p50 + p95), `display` stage = decoded→displayed
/// p50. All ms, refreshed once per 1 s window by the frame consumer.
#[derive(Default)]
struct PresentedStats {
e2e_p50_ms: Cell<f32>,
e2e_p95_ms: Cell<f32>,
display_ms: Cell<f32>,
}
impl StreamPage {
/// Render the canonical unified-stats OSD (design/stats-unification.md — Linux
/// endpoint is paintable-set, headline reads `capture→displayed`).
pub fn update_stats(&self, s: Stats) {
let mut line = format!(
"{:.0} fps · {:.1} Mbit/s · dec {:.1} ms · lat {:.1} ms · present {:.1} ms",
s.fps,
s.mbps,
s.decode_ms,
s.latency_ms,
self.present_ms.get()
);
let mut line1 = format!("{} · {:.0} fps · {:.1} Mb/s", self.mode_line, s.fps, s.mbps);
// Which decoder actually ran this window (vaapi/software) — tracks a fallback.
if !s.decoder.is_empty() {
line.push_str(" · ");
line.push_str(s.decoder);
line1.push_str(" · ");
line1.push_str(s.decoder);
}
if self.hdr.get() {
line.push_str(" · HDR");
line1.push_str(" · HDR");
}
self.stats_label.set_text(&line);
// The equation line: split `host+network` into `host + network` when the host
// reported per-AU timings (0xCF, stats Phase 2); the combined stage otherwise.
let equation = if s.split {
format!(
"= host {:.1} + network {:.1} + decode {:.1} + display {:.1}",
s.host_ms,
s.net_ms,
s.decode_ms,
self.presented.display_ms.get(),
)
} else {
format!(
"= host+network {:.1} + decode {:.1} + display {:.1}",
s.host_net_ms,
s.decode_ms,
self.presented.display_ms.get(),
)
};
let mut text = format!(
"{line1}\n\
end-to-end {:.1} ms p50 · {:.1} p95 · capture→displayed{}\n\
{equation}",
self.presented.e2e_p50_ms.get(),
self.presented.e2e_p95_ms.get(),
if self.same_host {
" (same-host clock)"
} else {
""
},
);
// Counters — only rendered when nonzero this window.
if s.lost > 0 {
text.push_str(&format!("\nlost {} ({:.1}%)", s.lost, s.lost_pct));
}
self.stats_label.set_text(&text);
}
}
@@ -84,6 +129,9 @@ pub struct StreamPageArgs {
/// reveal-on-notify chrome hiding) may never fire — the title bar would stay drawn
/// over the stream. Chrome-less by construction cannot regress that way.
pub chromeless: bool,
/// A controller is connected right now — the capture hint mentions the escape chord.
/// (Chromeless implies a controller-first device, so the chord shows there regardless.)
pub pad_connected: bool,
pub title: String,
}
@@ -119,7 +167,13 @@ fn send_abs(widget: &impl IsA<gtk::Widget>, connector: &NativeClient, x: f64, y:
struct Capture {
connector: Arc<NativeClient>,
window: adw::ApplicationWindow,
overlay: gtk::Overlay,
/// Held WEAKLY. Every input controller + the frame-clock tick are added to this overlay
/// and each captures `Rc<Capture>`; a strong ref back here would close the cycle
/// `overlay → controller → Rc<Capture> → overlay` that GTK can't collect, leaking the
/// whole stream subtree AND the `Arc<NativeClient>` (so `NativeClient::Drop` never runs)
/// on every session end — unbounded growth across the reconnects a Deck does constantly.
/// The live widget tree owns the overlay for the session's lifetime; upgrade at use.
overlay: glib::WeakRef<gtk::Overlay>,
hint: gtk::Label,
inhibit_shortcuts: bool,
captured: Cell<bool>,
@@ -133,13 +187,19 @@ struct Capture {
/// VKs / GameStream button ids currently held — flushed up on release.
held_keys: RefCell<HashSet<u8>>,
held_buttons: RefCell<HashSet<u32>>,
/// Fractional wheel remainder per axis (x, y), in 120-unit WHEEL_DELTA space. Precision
/// scroll surfaces — the Deck trackpad, hi-res wheels, two-finger touchpad — deliver
/// sub-unit deltas; truncating each event drops the tail. Carry it here instead.
scroll_acc: Cell<(f64, f64)>,
}
impl Capture {
/// Send the coalesced pointer position, if any — one datagram, one fresh mode read.
fn flush_pending_motion(&self) {
if let Some((x, y)) = self.pending_abs.take() {
send_abs(&self.overlay, &self.connector, x, y);
if let Some(overlay) = self.overlay.upgrade() {
send_abs(&overlay, &self.connector, x, y);
}
}
}
@@ -147,8 +207,9 @@ impl Capture {
if self.captured.replace(true) {
return;
}
self.overlay
.set_cursor(gdk::Cursor::from_name("none", None).as_ref());
if let Some(overlay) = self.overlay.upgrade() {
overlay.set_cursor(gdk::Cursor::from_name("none", None).as_ref());
}
self.hint.set_visible(false);
if self.inhibit_shortcuts {
if let Some(tl) = self
@@ -165,7 +226,9 @@ impl Capture {
if !self.captured.replace(false) {
return;
}
self.overlay.set_cursor(None);
if let Some(overlay) = self.overlay.upgrade() {
overlay.set_cursor(None);
}
self.hint.set_visible(true);
self.pending_abs.set(None); // never flush motion gathered while captured
if let Some(tl) = self
@@ -197,46 +260,56 @@ pub fn new(args: StreamPageArgs) -> StreamPage {
inhibit_shortcuts,
show_stats,
chromeless,
pad_connected,
title,
} = args;
let w = build_widgets(&window, &title, chromeless);
let w = build_widgets(&window, &title, chromeless, pad_connected);
w.stats_label.set_visible(show_stats);
// OSD line-1 facts, fixed for the session (the mode is negotiated per-session).
let mode = connector.mode();
let mode_line = format!("{}×{}@{}", mode.width, mode.height, mode.refresh_hz);
// Offset 0 = the host didn't answer the skew handshake / same host — flagged on the
// end-to-end line so an uncorrected cross-machine number is never shown silently.
let same_host = clock_offset_ns == 0;
let capture = Rc::new(Capture {
connector,
window: window.clone(),
overlay: w.overlay.clone(),
overlay: w.overlay.downgrade(),
hint: w.hint.clone(),
inhibit_shortcuts,
captured: Cell::new(false),
pending_abs: Cell::new(None),
held_keys: RefCell::new(HashSet::new()),
held_buttons: RefCell::new(HashSet::new()),
scroll_acc: Cell::new((0.0, 0.0)),
});
let present_ms = Rc::new(Cell::new(0.0f32));
let presented = Rc::new(PresentedStats::default());
let hdr = Rc::new(Cell::new(false));
spawn_frame_consumer(
&w.picture,
frames,
clock_offset_ns,
present_ms.clone(),
presented.clone(),
hdr.clone(),
);
attach_keyboard(&w.overlay, &window, &capture, &stop, &w.stats_label);
let key_controller = attach_keyboard(&window, &capture, &stop, &w.stats_label);
attach_mouse(&w.overlay, &capture);
attach_scroll(&w.overlay, &capture);
if !chromeless {
attach_edge_reveal(&w.toolbar, &w.overlay, &window, &capture);
}
let active_handler = attach_capture_lifecycle(&w.overlay, &window, &capture);
let escape_future = spawn_escape_watch(&window, &capture, escape_rx);
let escape_future = spawn_escape_watch(&window, &capture, escape_rx, &w.fs_hint, chromeless);
let disconnect_future = spawn_disconnect_watch(&window, &capture, &stop, disconnect_rx);
wire_teardown(
&w.page,
&window,
&stop,
(w.fs_handler, active_handler),
key_controller,
escape_future,
disconnect_future,
);
@@ -244,8 +317,10 @@ pub fn new(args: StreamPageArgs) -> StreamPage {
StreamPage {
page: w.page,
stats_label: w.stats_label,
present_ms,
presented,
hdr,
same_host,
mode_line,
}
}
@@ -254,6 +329,9 @@ struct PageWidgets {
picture: gtk::Picture,
stats_label: gtk::Label,
hint: gtk::Label,
/// The transient chord/fullscreen-exit hint — the escape watch re-flashes it in
/// chromeless mode.
fs_hint: gtk::Label,
overlay: gtk::Overlay,
toolbar: adw::ToolbarView,
page: adw::NavigationPage,
@@ -264,7 +342,12 @@ struct PageWidgets {
/// The offloaded picture under an overlay (stats HUD, capture hint, fullscreen hint), a
/// header bar with the fullscreen toggle, and the window's fullscreen behavior.
/// `chromeless` (Gaming Mode) builds NO header bar at all — see `StreamPageArgs`.
fn build_widgets(window: &adw::ApplicationWindow, title: &str, chromeless: bool) -> PageWidgets {
fn build_widgets(
window: &adw::ApplicationWindow,
title: &str,
chromeless: bool,
pad_connected: bool,
) -> PageWidgets {
let picture = gtk::Picture::new();
picture.set_content_fit(gtk::ContentFit::Contain);
@@ -273,6 +356,22 @@ fn build_widgets(window: &adw::ApplicationWindow, title: &str, chromeless: bool)
// no-op wrapper. Black letterboxing keeps fullscreen scanout-eligible.
let offload = gtk::GraphicsOffload::new(Some(&picture));
offload.set_black_background(true);
// Whether the raw video dmabuf may be handed to the compositor as a subsurface.
// Under gamescope (chromeless) default OFF: a subsurface makes the COMPOSITOR do the
// NV12→RGB conversion, and gamescope's matrix/range choice for it is outside our
// control (off-colours reported on the Deck) — GTK compositing it itself applies the
// stream's own BT.709-narrow color state. `PUNKTFUNK_OFFLOAD=1|0` overrides either
// way, which also makes the colour question bisectable in one run: offload-off heals →
// compositor conversion; still off → GTK/Mesa import (then try PUNKTFUNK_DECODER=software).
let offload_on = match std::env::var("PUNKTFUNK_OFFLOAD").ok().as_deref() {
Some("0") => false,
Some(_) => true,
None => !chromeless,
};
if !offload_on {
offload.set_enabled(gtk::GraphicsOffloadEnabled::Disabled);
tracing::info!("graphics offload disabled — GTK composites the video itself");
}
let stats_label = gtk::Label::new(None);
stats_label.add_css_class("osd");
@@ -282,9 +381,16 @@ fn build_widgets(window: &adw::ApplicationWindow, title: &str, chromeless: bool)
stats_label.set_margin_start(12);
stats_label.set_margin_top(12);
let hint = gtk::Label::new(Some(
"Click the stream to capture input · Ctrl+Alt+Shift+Q releases · Ctrl+Alt+Shift+D disconnects · Ctrl+Alt+Shift+S stats",
));
// The capture hint speaks the input devices actually present: on a controller-first
// device (chromeless) or with a pad connected it must surface the chord — keyboard-only
// text on a Deck told the user nothing they could press.
let hint = gtk::Label::new(Some(if chromeless {
"Tap the stream to capture input · hold L1 + R1 + Start + Select to leave"
} else if pad_connected {
"Click the stream to capture input · Ctrl+Alt+Shift+Q releases · Ctrl+Alt+Shift+D disconnects · hold L1 + R1 + Start + Select to leave"
} else {
"Click the stream to capture input · Ctrl+Alt+Shift+Q releases · Ctrl+Alt+Shift+D disconnects · Ctrl+Alt+Shift+S stats"
}));
hint.add_css_class("osd");
hint.set_halign(gtk::Align::Center);
hint.set_valign(gtk::Align::End);
@@ -296,7 +402,7 @@ fn build_widgets(window: &adw::ApplicationWindow, title: &str, chromeless: bool)
// devices; the L1+R1+Start+Select chord on a controller). Gaming Mode has no F11,
// no header to reveal, and Steam owns window management — only the chord applies.
let fs_hint = gtk::Label::new(Some(if chromeless {
"L1 + R1 + Start + Select — leave the stream (hold to disconnect)"
"Hold L1 + R1 + Start + Select — leave the stream"
} else {
"F11 · mouse to the top edge · L1 + R1 + Start + Select — exit fullscreen (hold to disconnect)"
}));
@@ -372,6 +478,7 @@ fn build_widgets(window: &adw::ApplicationWindow, title: &str, chromeless: bool)
picture,
stats_label,
hint,
fs_hint,
overlay,
toolbar,
page,
@@ -420,12 +527,13 @@ fn attach_edge_reveal(
/// then draws whatever paintable is current on its own frame clock. Ends itself when the
/// channel closes or the picture is gone.
///
/// Also the capture→present-ish measurement point: at each paintable set the frame's
/// host capture pts is compared against the local wall clock expressed in the host clock
/// (`clock_offset_ns`, same math as the session's decode latency). This is
/// capture→paintable-SET — GTK's own present adds one compositor cycle after this. The
/// 1 s p50 lands on the stats OSD (via `present_ms`) and in a "present window" debug
/// line for headless validation.
/// Also the `displayed` measurement point (design/stats-unification.md): each paintable
/// set stamps the local wall clock, yielding end-to-end = capture→displayed (host-clock
/// corrected via `clock_offset_ns`, p50+p95, measured directly) and the client-local
/// `display` stage = decoded→displayed. This is capture→paintable-SET — GTK's own
/// present adds one compositor cycle after this. The 1 s window results land on the
/// stats OSD (via `PresentedStats`) and in a "present window" debug line for headless
/// validation.
/// One-entry cache of `ColorDesc` → `GdkColorState` (signaling changes at most on an
/// SDR↔HDR flip, never per frame).
#[derive(Default)]
@@ -461,11 +569,15 @@ impl ColorStateCache {
});
}
let state = cicp.build_color_state().ok();
if state.is_none() {
tracing::warn!(
// One line per signaling change — the on-glass colour bisect reads this to tell
// "state applied" from "GDK fell back to its YUV default (BT.601)".
match &state {
Some(_) => tracing::info!(?desc, rgb, "colour signaling → GDK color state"),
None => tracing::warn!(
?desc,
"GDK can't represent this colour signaling — using default"
);
rgb,
"GDK can't represent this colour signaling — using default (YUV: BT.601)"
),
}
self.0 = Some((desc, state.clone()));
state
@@ -476,7 +588,7 @@ fn spawn_frame_consumer(
picture: &gtk::Picture,
frames: async_channel::Receiver<DecodedFrame>,
clock_offset_ns: i64,
present_ms: Rc<Cell<f32>>,
presented_stats: Rc<PresentedStats>,
hdr: Rc<Cell<bool>>,
) {
let picture = picture.downgrade();
@@ -488,7 +600,10 @@ fn spawn_frame_consumer(
let mut yuv_state = ColorStateCache::default();
let mut rgb_state = ColorStateCache::default();
glib::spawn_future_local(async move {
let mut win_lat_us: Vec<u64> = Vec::with_capacity(256);
// Window samples (µs): end-to-end capture→displayed (host-clock corrected) and
// the client-local display stage decoded→displayed.
let mut win_e2e_us: Vec<u64> = Vec::with_capacity(256);
let mut win_disp_us: Vec<u64> = Vec::with_capacity(256);
let mut win_start = Instant::now();
while let Ok(f) = frames.recv().await {
let Some(picture) = picture.upgrade() else {
@@ -561,26 +676,34 @@ fn spawn_frame_consumer(
}
}
}
// Capture→paintable-set latency, host-clock corrected (same math and sanity
// bound as the session's decode-latency window).
// The `displayed` stamp: end-to-end = capture→displayed host-clock corrected
// (same clamp as the session's stage windows); display = decoded→displayed,
// single clock, no skew.
if presented {
let lat = (crate::session::now_ns() as i128 + clock_offset_ns as i128
- f.pts_ns as i128)
.max(0) as u64;
if lat > 0 && lat < 10_000_000_000 {
win_lat_us.push(lat / 1000);
let displayed_ns = crate::session::now_ns();
let e2e = (displayed_ns as i128 + clock_offset_ns as i128 - f.pts_ns as i128).max(0)
as u64;
if e2e > 0 && e2e < 10_000_000_000 {
win_e2e_us.push(e2e / 1000);
}
win_disp_us.push(displayed_ns.saturating_sub(f.decoded_ns) / 1000);
}
if win_start.elapsed() >= Duration::from_secs(1) {
win_lat_us.sort_unstable();
let p50 = win_lat_us.get(win_lat_us.len() / 2).copied().unwrap_or(0);
let frames = win_e2e_us.len();
let (e2e_p50, e2e_p95) = crate::session::window_percentiles(&mut win_e2e_us);
let (disp_p50, _) = crate::session::window_percentiles(&mut win_disp_us);
tracing::debug!(
frames = win_lat_us.len(),
present_p50_us = p50,
frames,
e2e_p50_us = e2e_p50,
e2e_p95_us = e2e_p95,
display_p50_us = disp_p50,
"present window"
);
present_ms.set(p50 as f32 / 1000.0);
win_lat_us.clear();
presented_stats.e2e_p50_ms.set(e2e_p50 as f32 / 1000.0);
presented_stats.e2e_p95_ms.set(e2e_p95 as f32 / 1000.0);
presented_stats.display_ms.set(disp_p50 as f32 / 1000.0);
win_e2e_us.clear();
win_disp_us.clear();
win_start = Instant::now();
}
}
@@ -590,13 +713,20 @@ fn spawn_frame_consumer(
/// Keyboard, capture-phase: the release (Ctrl+Alt+Shift+Q) / disconnect (Ctrl+Alt+Shift+D)
/// / stats (Ctrl+Alt+Shift+S) chords and F11 are handled locally; everything else becomes
/// a VK on the wire while captured.
///
/// The controller lives on the **window**, not the stream overlay: a `NavigationView` push
/// followed by `window.fullscreen()` hands keyboard focus to the pushed page's header back
/// button (a sibling of the overlay), so an overlay-scoped key controller never sees a key and
/// every chord — plus all gameplay key forwarding — is silently dropped until the user clicks
/// the stream. The window is always on the key-propagation path regardless of which child holds
/// focus. Returned so `wire_teardown` can remove it when the page goes away (otherwise the
/// chords would keep firing app-wide against a dead session).
fn attach_keyboard(
overlay: &gtk::Overlay,
window: &adw::ApplicationWindow,
capture: &Rc<Capture>,
stop: &Arc<AtomicBool>,
stats: &gtk::Label,
) {
) -> gtk::EventControllerKey {
let key = gtk::EventControllerKey::new();
key.set_propagation_phase(gtk::PropagationPhase::Capture);
let cap = capture.clone();
@@ -662,7 +792,8 @@ fn attach_keyboard(
}
}
});
overlay.add_controller(key);
window.add_controller(key.clone());
key
}
/// Mouse: absolute motion + buttons — forwarded only while captured; the click that
@@ -681,7 +812,8 @@ fn attach_mouse(overlay: &gtk::Overlay, capture: &Rc<Capture>) {
});
overlay.add_controller(motion);
// The per-tick flush. (The tick callback dies with the overlay, so no teardown.)
// The per-tick flush. The tick callback dies with the overlay (which `Capture` now holds
// only weakly, so it truly can), taking its `Capture` ref with it — no explicit teardown.
let cap = capture.clone();
overlay.add_tick_callback(move |_, _| {
cap.flush_pending_motion();
@@ -691,7 +823,9 @@ fn attach_mouse(overlay: &gtk::Overlay, capture: &Rc<Capture>) {
let click = gtk::GestureClick::builder().button(0).build();
let cap = capture.clone();
click.connect_pressed(move |g, _n, x, y| {
cap.overlay.grab_focus();
if let Some(overlay) = cap.overlay.upgrade() {
overlay.grab_focus();
}
if !cap.captured.get() {
cap.engage(); // the engaging click is suppressed toward the host
return;
@@ -727,16 +861,22 @@ fn attach_scroll(overlay: &gtk::Overlay, capture: &Rc<Capture>) {
}
cap.flush_pending_motion(); // scroll happens at the latest cursor position
// The wire carries WHEEL_DELTA(120) units, positive = up / right; GTK's dy is
// positive = down. Smooth fractions survive — libei's discrete scroll is
// 120-based too.
let vy = (-dy * 120.0) as i32;
// positive = down. libei's discrete scroll is 120-based too. Accumulate the
// fractional remainder so precision-scroll sub-unit deltas aren't lost.
let (mut ax, mut ay) = cap.scroll_acc.get();
ay += -dy * 120.0;
ax += dx * 120.0;
let vy = ay.trunc() as i32;
if vy != 0 {
ay -= f64::from(vy);
send(&cap.connector, InputKind::MouseScroll, 0, vy, 0, 0);
}
let vx = (dx * 120.0) as i32;
let vx = ax.trunc() as i32;
if vx != 0 {
ax -= f64::from(vx);
send(&cap.connector, InputKind::MouseScroll, 1, vx, 0, 0);
}
cap.scroll_acc.set((ax, ay));
glib::Propagation::Stop
});
overlay.add_controller(scroll);
@@ -772,20 +912,30 @@ fn attach_capture_lifecycle(
/// Controller escape chord (gamepad service) → leave fullscreen + release capture. The
/// chord is the only fullscreen exit a controller has (no F11 key; fullscreen hides the
/// chrome). Aborted on page-hidden so a stale future can't act on the shared window.
/// chrome). In chromeless mode there is nothing visible to release INTO — a quick press
/// re-flashes the hold-to-leave hint instead, so an experimenting user learns the hold.
/// Aborted on page-hidden so a stale future can't act on the shared window.
fn spawn_escape_watch(
window: &adw::ApplicationWindow,
capture: &Rc<Capture>,
escape_rx: async_channel::Receiver<()>,
fs_hint: &gtk::Label,
chromeless: bool,
) -> glib::JoinHandle<()> {
let window = window.clone();
let cap = capture.clone();
let fs_hint = fs_hint.clone();
glib::spawn_future_local(async move {
while escape_rx.recv().await.is_ok() {
if window.is_fullscreen() {
window.unfullscreen();
}
cap.release();
if chromeless {
fs_hint.set_visible(true);
let fs_hint = fs_hint.clone();
glib::timeout_add_seconds_local_once(4, move || fs_hint.set_visible(false));
}
}
})
}
@@ -822,12 +972,14 @@ fn wire_teardown(
window: &adw::ApplicationWindow,
stop: &Arc<AtomicBool>,
handlers: (glib::SignalHandlerId, glib::SignalHandlerId),
key_controller: gtk::EventControllerKey,
escape_future: glib::JoinHandle<()>,
disconnect_future: glib::JoinHandle<()>,
) {
let window = window.clone();
let stop_h = stop.clone();
let handlers = RefCell::new(Some(handlers));
let key_controller = RefCell::new(Some(key_controller));
let escape_future = RefCell::new(Some(escape_future));
let disconnect_future = RefCell::new(Some(disconnect_future));
page.connect_hidden(move |_| {
@@ -836,6 +988,11 @@ fn wire_teardown(
window.disconnect(fs);
window.disconnect(active);
}
// The key controller lives on the window (see `attach_keyboard`) — remove it so its
// chords don't keep firing app-wide against a torn-down session.
if let Some(kc) = key_controller.borrow_mut().take() {
window.remove_controller(&kc);
}
if let Some(f) = escape_future.borrow_mut().take() {
f.abort();
}
+46 -6
View File
@@ -24,11 +24,15 @@ use std::os::fd::RawFd;
use std::ptr;
/// One decoded frame headed for the presenter, carrying the host capture timestamp so the
/// UI can measure capture→paintable-set latency at the moment it presents.
/// UI can measure capture→displayed latency at the moment it presents.
pub struct DecodedFrame {
/// Host-clock capture pts (ns) of the AU this image decoded from — compare against
/// the local wall clock + `clock_offset_ns` at paintable-set time.
pub pts_ns: u64,
/// Local wall clock (ns) when the decoder emitted this image — the `decoded`
/// measurement point (design/stats-unification.md); the presenter subtracts it from
/// its paintable-set stamp for the client-local `display` stage.
pub decoded_ns: u64,
pub image: DecodedImage,
}
@@ -132,8 +136,19 @@ pub struct Decoder {
/// The negotiated codec (from the host's Welcome), so a mid-session VAAPI→software demotion
/// rebuilds the software decoder for the SAME codec.
codec_id: ffmpeg::codec::Id,
/// Consecutive VAAPI decode errors — a single transient failure (e.g. a reference-missing
/// frame after packet loss) shouldn't cost the whole session its hardware decoder.
vaapi_fails: u32,
/// Set when the decoder needs a fresh IDR to resynchronize (after an error or a demotion).
/// The pump drains it and asks the host — under the infinite GOP there is no periodic
/// keyframe, so a rebuilt/erroring decoder would otherwise stay gray/frozen forever.
want_keyframe: bool,
}
/// Demote VAAPI→software only after this many consecutive hardware decode errors; a lone
/// transient error just re-requests an IDR and keeps the hardware decoder.
const VAAPI_DEMOTE_AFTER: u32 = 3;
/// Map a negotiated `quic` codec bit to the FFmpeg decoder id the client opens.
pub fn ffmpeg_codec_id(wire: u8) -> ffmpeg::codec::Id {
match wire {
@@ -179,6 +194,8 @@ impl Decoder {
return Ok(Decoder {
backend: Backend::Vaapi(v),
codec_id,
vaapi_fails: 0,
want_keyframe: false,
});
}
Err(e) => {
@@ -192,20 +209,43 @@ impl Decoder {
Ok(Decoder {
backend: Backend::Software(SoftwareDecoder::new(codec_id)?),
codec_id,
vaapi_fails: 0,
want_keyframe: false,
})
}
/// Drain the "please ask the host for an IDR" flag — the pump calls this each iteration
/// (throttled) so a demoted/erroring decoder can resynchronize under the infinite GOP.
pub fn take_keyframe_request(&mut self) -> bool {
std::mem::take(&mut self.want_keyframe)
}
/// Feed one access unit; returns the decoded frame (the host's streams are
/// one-in/one-out). A software decode error after packet loss is survivable — log
/// upstream and keep feeding. A VAAPI error demotes to software for the rest of the
/// session (broken driver, e.g. nvidia-vaapi-driver) — the next IDR resynchronizes.
/// upstream and keep feeding. A VAAPI error re-requests an IDR and retries the hardware
/// decoder; only a persistent streak of failures (a genuinely broken driver, e.g.
/// nvidia-vaapi-driver) demotes to software. Either way `want_keyframe` is set so the
/// pump asks the host for a fresh IDR — under the infinite GOP nothing else resyncs a
/// rebuilt/erroring decoder, so skipping this leaves the picture gray/frozen for good.
pub fn decode(&mut self, au: &[u8]) -> Result<Option<DecodedImage>> {
match &mut self.backend {
Backend::Vaapi(v) => match v.decode(au) {
Ok(f) => Ok(f.map(DecodedImage::Dmabuf)),
Ok(f) => {
self.vaapi_fails = 0;
Ok(f.map(DecodedImage::Dmabuf))
}
Err(e) => {
tracing::warn!(error = %e, "VAAPI decode failed — falling back to software");
self.backend = Backend::Software(SoftwareDecoder::new(self.codec_id)?);
self.vaapi_fails += 1;
self.want_keyframe = true;
if self.vaapi_fails >= VAAPI_DEMOTE_AFTER {
tracing::warn!(error = %e, fails = self.vaapi_fails,
"VAAPI decode failing repeatedly — demoting to software");
self.backend = Backend::Software(SoftwareDecoder::new(self.codec_id)?);
self.vaapi_fails = 0;
} else {
tracing::warn!(error = %e,
"VAAPI decode error — requesting keyframe, keeping hardware decode");
}
Ok(None)
}
},
+1 -1
View File
@@ -27,7 +27,7 @@ GEOMETRY="${GEOMETRY:-1380x860x24}"
SETTLE="${SETTLE:-1.2}"
SHOT_DISPLAY="${SHOT_DISPLAY:-:99}"
if [ "$#" -gt 0 ]; then SCENES=("$@"); else SCENES=(hosts settings trust pair addhost shortcuts library); fi
if [ "$#" -gt 0 ]; then SCENES=("$@"); else SCENES=(hosts settings trust pair addhost shortcuts library gamepad-library); fi
[ -x "$BIN" ] || {
echo "client binary not found: $BIN (build it first: cargo build --release -p punktfunk-client-linux)" >&2
+1 -1
View File
@@ -14,7 +14,7 @@ example of driving the protocol end to end: QUIC control plane, UDP data plane,
- **Receives a real stream**, writes a playable elementary stream (`.h265`/`.h264`/`.av1` — the
extension tracks the **negotiated codec**; the probe advertises all three and the host picks), and
reports per-frame **capture→…→reassembled latency** percentiles (the host stamps each frame with
reports per-frame **capture→received latency** percentiles (the host stamps each frame with
its capture clock).
- **Verification mode** against a synthetic host — byte-checks deterministic test frames.
- **Exercises every plane** with scripted test traffic:
+99 -25
View File
@@ -4,7 +4,7 @@
//! * **verification** (`frames > 0`, synthetic host): byte-checks deterministic test frames;
//! * **stream** (`frames == 0`, virtual host): receives real encoded AUs, writes a playable
//! elementary stream (the dump extension follows the negotiated codec — `.h265`/`.h264`/`.av1`;
//! the probe advertises all three), and reports per-frame **capture→…→reassembled latency**
//! the probe advertises all three), and reports per-frame **capture→received latency**
//! percentiles (the host stamps each frame with its capture wall clock; same-host runs share
//! that clock).
//!
@@ -41,7 +41,7 @@
//! Usage: `punktfunk-probe [--connect HOST:PORT] [--mode WxHxFPS] [--remode WxHxFPS:SECS]
//! [--out FILE] [--bitrate KBPS] [--codec auto|h264|hevc|av1] [--audio-channels 2|6|8]
//! [--launch APP] [--name NAME] [--speed-test KBPS:MS]
//! [--input-test | --mic-test | --touch-test | --rich-input-test]
//! [--input-test | --mic-test [--mic-burst] | --touch-test | --rich-input-test]
//! [--pin HEX | --pair PIN] [--compositor NAME] [--gamepad NAME] | --discover [SECS]`
//! Env: `PUNKTFUNK_CLIENT_10BIT=1` / `PUNKTFUNK_CLIENT_444=1` advertise the 10-bit / 4:4:4 caps.
@@ -65,6 +65,9 @@ struct Args {
input_test: bool,
/// `--mic-test` — stream a synthetic 440 Hz tone as the mic uplink (proves the mic path).
mic_test: bool,
/// `--mic-burst` — pace the mic-test like a real client's input tap (2× 20 ms per 40 ms),
/// the arrival shape that exercises host-side jitter buffering.
mic_burst: bool,
/// `--touch-test` — drag a synthetic finger in a circle (proves the touch path).
touch_test: bool,
/// `--rich-input-test` — drive the DualSense touchpad + motion over 0xCC (host needs
@@ -205,6 +208,7 @@ fn parse_args() -> Args {
out: get("--out").map(String::from),
input_test: argv.iter().any(|a| a == "--input-test"),
mic_test: argv.iter().any(|a| a == "--mic-test"),
mic_burst: argv.iter().any(|a| a == "--mic-burst"),
touch_test: argv.iter().any(|a| a == "--touch-test"),
rich_input_test: argv.iter().any(|a| a == "--rich-input-test"),
pin,
@@ -424,7 +428,9 @@ async fn session(args: Args) -> Result<()> {
// PUNKTFUNK_CLIENT_444=1 advertises VIDEO_CAP_444 (host HEVC 4:4:4 path) — verify the
// resulting chroma with `ffprobe` on the `--out` .h265.
video_caps: {
let mut caps = 0u8;
// Always ask for per-AU host timings (0xCF) — this is a measurement tool, and the
// host/network split is exactly what it exists to report. Old hosts ignore the bit.
let mut caps = punktfunk_core::quic::VIDEO_CAP_HOST_TIMING;
if std::env::var_os("PUNKTFUNK_CLIENT_10BIT").is_some() {
caps |= punktfunk_core::quic::VIDEO_CAP_10BIT;
}
@@ -481,7 +487,7 @@ async fn session(args: Args) -> Result<()> {
.await?;
// Wall-clock skew handshake on the still-private control stream (before --remode/--speed-test
// take it): align our clock to the host's so the per-frame capture→reassembled latency is valid
// take it): align our clock to the host's so the per-frame capture→received latency is valid
// across machines. `None` ⇒ an old host that doesn't answer — fall back to a shared clock (0).
let clock_offset_ns = match punktfunk_core::quic::clock_sync(&mut send, &mut recv).await {
Some(skew) => {
@@ -738,9 +744,16 @@ async fn session(args: Args) -> Result<()> {
});
}
// Mic plane: stream a synthetic 440 Hz tone as the mic uplink (0xCB), Opus-encoded 5 ms
// stereo frames — proves client→host mic passthrough end to end without a real microphone
// (the host decodes it into its virtual PipeWire source; record that source to hear the tone).
// Mic plane: stream a synthetic 440 Hz tone as the mic uplink (0xCB) — proves client→host
// mic passthrough end to end without a real microphone (the host decodes it into its virtual
// source; record that source to hear the tone). Two pacing modes:
// default — Opus 5 ms frames on a steady 5 ms tick (smooth arrival).
// --mic-burst — two 20 ms Opus frames back-to-back every 40 ms, replicating a real
// client's input-tap cadence (the Mac client's AVAudioEngine tap yields
// ~2048-frame buffers → two packets per ~42 ms). This is the arrival
// pattern that exposed the Windows host's missing jitter buffer (constant
// crackle, 2026-07-03): a steady 5 ms stream never trips it. Record the
// host mic and count silence gaps to regression-test host-side buffering.
#[cfg(not(target_os = "linux"))]
if args.mic_test {
tracing::warn!("--mic-test requires Linux (libopus) — skipped");
@@ -748,6 +761,7 @@ async fn session(args: Args) -> Result<()> {
#[cfg(target_os = "linux")]
if args.mic_test {
let conn2 = conn.clone();
let burst = args.mic_burst;
tokio::spawn(async move {
let mut enc =
match opus::Encoder::new(48_000, opus::Channels::Stereo, opus::Application::Voip) {
@@ -758,28 +772,38 @@ async fn session(args: Args) -> Result<()> {
}
};
let _ = enc.set_bitrate(opus::Bitrate::Bits(64_000));
tracing::info!("mic-test: streaming a 440 Hz tone as the mic uplink");
// Frame size + tick per pacing mode; `per_tick` packets are sent back-to-back.
let (frame, tick_ms, per_tick) = if burst {
(960usize, 40u64, 2u32) // 2× 20 ms every 40 ms — the bursty real-client shape
} else {
(240usize, 5u64, 1u32) // 5 ms frames on a smooth tick
};
tracing::info!(burst, "mic-test: streaming a 440 Hz tone as the mic uplink");
let mut phase = 0.0f32;
let step = 2.0 * std::f32::consts::PI * 440.0 / 48_000.0;
let mut pcm = [0f32; 240 * 2]; // 5 ms stereo
let mut pcm = vec![0f32; frame * 2];
let mut out = [0u8; 4000];
let mut interval = tokio::time::interval(std::time::Duration::from_millis(5));
for seq in 0u32.. {
let mut interval = tokio::time::interval(std::time::Duration::from_millis(tick_ms));
let mut seq = 0u32;
'stream: loop {
interval.tick().await;
for f in 0..240 {
let s = (phase.sin()) * 0.25;
phase += step;
if phase > std::f32::consts::PI * 2.0 {
phase -= std::f32::consts::PI * 2.0;
for _ in 0..per_tick {
for f in 0..frame {
let s = (phase.sin()) * 0.25;
phase += step;
if phase > std::f32::consts::PI * 2.0 {
phase -= std::f32::consts::PI * 2.0;
}
pcm[f * 2] = s;
pcm[f * 2 + 1] = s;
}
pcm[f * 2] = s;
pcm[f * 2 + 1] = s;
}
if let Ok(n) = enc.encode_float(&pcm, &mut out) {
let d = punktfunk_core::quic::encode_mic_datagram(seq, now_ns(), &out[..n]);
if conn2.send_datagram(d.into()).is_err() {
break;
if let Ok(n) = enc.encode_float(&pcm, &mut out) {
let d = punktfunk_core::quic::encode_mic_datagram(seq, now_ns(), &out[..n]);
if conn2.send_datagram(d.into()).is_err() {
break 'stream;
}
}
seq = seq.wrapping_add(1);
}
}
tracing::info!("mic-test: done");
@@ -902,6 +926,10 @@ async fn session(args: Args) -> Result<()> {
let audio_bytes = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0));
let rumble_pkts = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0));
let hidout_pkts = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0));
// Per-AU host timings (0xCF) → the stream loop, which matches them to received AUs by pts
// and reports the host/network split. try_send: overflow drops samples, never blocks QUIC.
let (host_timing_tx, host_timing_rx) =
std::sync::mpsc::sync_channel::<punktfunk_core::quic::HostTiming>(512);
{
let (a, ab, r, h) = (
audio_pkts.clone(),
@@ -909,6 +937,7 @@ async fn session(args: Args) -> Result<()> {
rumble_pkts.clone(),
hidout_pkts.clone(),
);
let ht_tx = host_timing_tx;
let conn2 = conn.clone();
// Build a multistream decoder for the host-RESOLVED layout so the probe actually decodes
// the surround stream (not just counts bytes) — the headless validator for the encode path.
@@ -956,6 +985,10 @@ async fn session(args: Args) -> Result<()> {
if h.fetch_add(1, Relaxed) < 12 {
tracing::info!(?hid, "DualSense HID output (0xCD)");
}
} else if let Some(t) = punktfunk_core::quic::decode_host_timing_datagram(&d) {
// Per-AU host timing (0xCF) — forwarded to the stream loop for the
// host/network latency split.
let _ = ht_tx.try_send(t);
}
}
});
@@ -1000,6 +1033,12 @@ async fn session(args: Args) -> Result<()> {
let mut mismatched = 0u32;
let mut bytes = 0u64;
let mut latencies_us: Vec<u64> = Vec::new();
// Host/network split: received AUs awaiting their 0xCF host timing (pts → capture→received
// µs), matched as the datagrams arrive. Bounded — an old host never sends any.
let mut pending_split: std::collections::VecDeque<(u64, u64)> =
std::collections::VecDeque::new();
let mut host_us_v: Vec<u64> = Vec::new();
let mut net_us_v: Vec<u64> = Vec::new();
let mut last_rx = std::time::Instant::now();
let started = std::time::Instant::now();
// Adaptive-FEC loss window: publish a fresh estimate every 750 ms for the LossReport task.
@@ -1051,12 +1090,25 @@ async fn session(args: Args) -> Result<()> {
continue;
}
bytes += frame.data.len() as u64;
// capture→reassembled: our receive instant in the host clock (now + offset)
// capture→received: our receive instant in the host clock (now + offset)
// minus the host's capture pts. offset is 0 same-host / old host.
let lat = (now_ns() as i128 + clock_offset as i128 - frame.pts_ns as i128)
.max(0) as u64;
if lat > 0 && lat < 10_000_000_000 {
latencies_us.push(lat / 1000);
pending_split.push_back((frame.pts_ns, lat / 1000));
if pending_split.len() > 1024 {
pending_split.pop_front();
}
}
// Match any host timings (0xCF) that have arrived: host = the reported
// capture→sent, network = our capture→received minus it (per-frame tiling).
while let Ok(t) = host_timing_rx.try_recv() {
if let Some(i) = pending_split.iter().position(|(p, _)| *p == t.pts_ns) {
let (_, hostnet_us) = pending_split.remove(i).unwrap();
host_us_v.push(t.host_us as u64);
net_us_v.push(hostnet_us.saturating_sub(t.host_us as u64));
}
}
if expected > 0 {
// Verification mode: deterministic content.
@@ -1100,9 +1152,31 @@ async fn session(args: Args) -> Result<()> {
lat_p99_us = pct(0.99),
lat_max_us = latencies_us.last().copied().unwrap_or(0),
skew_corrected,
"punktfunk/1 stream complete (capture→reassembled latency; skew_corrected=true ⇒ \
"punktfunk/1 stream complete (capture→received latency; skew_corrected=true ⇒ \
cross-machine valid, false ⇒ same-host clock)"
);
if !host_us_v.is_empty() {
// The host/network split from the per-AU 0xCF timings (design/stats-unification.md
// Phase 2): host = the host's own capture→sent, network = capture→received minus it.
let pcts = |v: &mut Vec<u64>, p: f64| -> u64 {
if v.is_empty() {
return 0;
}
v.sort_unstable();
v[((v.len() as f64 * p) as usize).min(v.len() - 1)]
};
tracing::info!(
timing_samples = host_us_v.len(),
host_p50_us = pcts(&mut host_us_v, 0.50),
host_p95_us = pcts(&mut host_us_v, 0.95),
net_p50_us = pcts(&mut net_us_v, 0.50),
net_p95_us = pcts(&mut net_us_v, 0.95),
"host/network latency split (host = capture→sent on the host; network = wire + \
reassembly)"
);
} else {
tracing::info!("no host timing datagrams (0xCF) — old host; host+network unsplit");
}
if expected > 0 {
anyhow::ensure!(mismatched == 0, "{mismatched} corrupted frames");
anyhow::ensure!(ok == expected, "received {ok}/{expected} frames");
+46 -19
View File
@@ -2,7 +2,7 @@
//! the UI thread, then handed — presenter and all — to the dedicated render thread
//! ([`crate::render`]), which presents decoded frames at stream cadence. The page itself only
//! forwards panel size/DPI changes and draws the status-chip HUD overlay (mode · decode path ·
//! HDR · fps/throughput/latency · capture hint).
//! HDR · fps/goodput · end-to-end latency + stage equation · capture hint).
use super::style::{edges, uniform};
use super::Svc;
@@ -22,8 +22,9 @@ use windows_reactor::*;
pub(crate) struct HudSample {
pub(crate) stats: Stats,
pub(crate) captured: bool,
/// `(presents/s, skipped/s, capture→presented p50 ms)` — see [`crate::render::present_stats`].
pub(crate) present: (u32, u32, f32),
/// The render thread's glass-side window (presents/s, skips, end-to-end p50/p95, display
/// stage p50) — see [`crate::render::present_stats`].
pub(crate) present: crate::render::PresentStats,
}
/// Props for the stream page: the services plus the live HUD sample that drives the overlay
@@ -171,13 +172,16 @@ fn fmt_uptime(secs: u32) -> String {
}
}
/// The streaming HUD overlay (top-right), mirroring the Apple client: a chip row (mode · codec ·
/// decode path · HDR), a stream line (decode fps / bitrate / decode time), a glass line (display
/// presents + end-to-end latency decoded vs on-glass), a session line (host · time · loss), and
/// the shortcut hints. Layered over the `SwapChainPanel` in the same grid cell.
/// The streaming HUD overlay (top-right), unified stats vocabulary (design/stats-unification.md):
/// a chip row (mode · codec · decode path · HDR), a stream line (received fps · goodput ·
/// presenter fps), the end-to-end headline (capture→on-glass p50/p95, host-clock corrected), the
/// stage equation (= host + network + decode + display when the host reports 0xCF timings, else
/// the combined = host+network + decode + display; stage p50s), a session line
/// (host · time · loss/skips), and the shortcut hints. Layered over the `SwapChainPanel` in the
/// same grid cell.
fn hud_overlay(hud: &HudSample, mode: Option<Mode>, host: &str) -> Element {
let stats = &hud.stats;
let (pfps, skipped, glass_ms) = hud.present;
let present = &hud.present;
let res = mode
.map(|m| format!("{}\u{00D7}{}@{}", m.width, m.height, m.refresh_hz))
.unwrap_or_else(|| "\u{2014}".into());
@@ -193,25 +197,47 @@ fn hud_overlay(hud: &HudSample, mode: Option<Mode>, host: &str) -> Element {
if stats.hdr {
chips.push(hud_chip("HDR", Color::rgb(255, 205, 90)).into());
}
// Received fps + goodput, plus the presenter's own rate (Moonlight's "Rendering frame rate"
// analog — how often the display actually gets a new frame).
let stream_line = format!(
"{:.0} fps \u{00B7} {:.1} Mb/s \u{00B7} decode {:.1} ms",
stats.fps, stats.mbps, stats.decode_ms
"{:.0} fps \u{00B7} {:.1} Mb/s \u{00B7} display {} fps",
stats.fps, stats.mbps, present.fps
);
// End-to-end latency (host-clock corrected): capture→decoded from the pump, capture→on-glass
// from the render thread's post-Present stamp. `skipped` = newest-wins drops (expected when
// the stream outpaces the display); `lost` = unrecoverable network drops.
let glass_line = format!(
"display {pfps} fps \u{00B7} latency {:.1} ms decoded / {glass_ms:.1} ms on-glass",
stats.latency_ms
// The headline: end-to-end capture→displayed, measured directly post-Present (never the sum
// of the stage percentiles). `(same-host clock)` flags an uncorrected clock (offset == 0:
// same host, or the host skipped the skew handshake).
let mut e2e_line = format!(
"end-to-end {:.1} ms p50 \u{00B7} {:.1} p95 \u{00B7} capture\u{2192}on-glass",
present.e2e_p50_ms, present.e2e_p95_ms
);
if stats.same_host {
e2e_line.push_str(" (same-host clock)");
}
// The equation: the stages tile the headline interval per frame; the window p50s only
// approximately sum (percentiles aren't additive). With per-AU 0xCF host timings the opaque
// `host+network` term splits into `host` (host capture→sent) + `network` (the remainder);
// an old host emits none and the combined term stays.
let stage_line = if stats.split {
format!(
"= host {:.1} + network {:.1} + decode {:.1} + display {:.1}",
stats.host_ms, stats.net_ms, stats.decode_ms, present.display_p50_ms
)
} else {
format!(
"= host+network {:.1} + decode {:.1} + display {:.1}",
stats.hostnet_ms, stats.decode_ms, present.display_p50_ms
)
};
let mut session_bits: Vec<String> = Vec::new();
if !host.is_empty() {
session_bits.push(host.to_string());
}
// `lost` = unrecoverable network drops (session-cumulative); `skipped` = the render thread's
// newest-wins drops last window (expected when the stream outpaces the display).
session_bits.push(fmt_uptime(stats.uptime_secs));
session_bits.push(format!("{} lost", stats.dropped));
if skipped > 0 {
session_bits.push(format!("{skipped} skipped"));
if present.skipped > 0 {
session_bits.push(format!("{} skipped", present.skipped));
}
let session_line = session_bits.join(" \u{00B7} ");
let hint = if hud.captured {
@@ -228,7 +254,8 @@ fn hud_overlay(hud: &HudSample, mode: Option<Mode>, host: &str) -> Element {
vstack((
hstack(chips).spacing(6.0),
dim(&stream_line),
dim(&glass_line),
dim(&e2e_line),
dim(&stage_line),
dim(&session_line),
text_block(hint)
.font_size(11.0)
+14 -2
View File
@@ -238,11 +238,23 @@ fn run_headless_cli(args: &[String], identity: (String, String)) {
session::SessionEvent::Connected {
mode, fingerprint, ..
} => tracing::info!(?mode, fp = %trust::hex(&fingerprint), "connected"),
// With per-AU 0xCF host timings the combined host+network stage splits into
// host (capture→sent on the host) + net; an old host emits none → combined only.
session::SessionEvent::Stats(s) if s.split => tracing::info!(
fps = format!("{:.0}", s.fps),
mbps = format!("{:.1}", s.mbps),
decode_p50_ms = format!("{:.2}", s.decode_ms),
hostnet_p50_ms = format!("{:.2}", s.hostnet_ms),
host_p50_ms = format!("{:.2}", s.host_ms),
net_p50_ms = format!("{:.2}", s.net_ms),
frames_seen,
"stats"
),
session::SessionEvent::Stats(s) => tracing::info!(
fps = format!("{:.0}", s.fps),
mbps = format!("{:.1}", s.mbps),
decode_ms = format!("{:.2}", s.decode_ms),
lat_ms = format!("{:.2}", s.latency_ms),
decode_p50_ms = format!("{:.2}", s.decode_ms),
hostnet_p50_ms = format!("{:.2}", s.hostnet_ms),
frames_seen,
"stats"
),
+78 -27
View File
@@ -10,27 +10,46 @@
//! draw (and redraws the held frame after a resize — fresh back buffers are blank).
use crate::present::Presenter;
use crate::session::FrameRx;
use crate::session::{FrameRx, FrameTimes};
use crossbeam_channel::RecvTimeoutError;
use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// The last 1-second render window, published for the HUD (one render thread at a time):
/// presents/s, frames skipped by the newest-wins drain, and the capture→presented p50 in µs.
/// presents/s, frames skipped by the newest-wins drain, the end-to-end (capture→on-glass)
/// p50/p95 and the `display` stage (decoded→displayed) p50, all stamped post-`Present()`, in µs.
/// Zeroed when a render thread starts so a new session never shows the previous one's numbers.
static PRESENT_FPS: AtomicU32 = AtomicU32::new(0);
static PRESENT_SKIPPED: AtomicU32 = AtomicU32::new(0);
static PRESENT_P50_US: AtomicU64 = AtomicU64::new(0);
static E2E_P50_US: AtomicU64 = AtomicU64::new(0);
static E2E_P95_US: AtomicU64 = AtomicU64::new(0);
static DISPLAY_P50_US: AtomicU64 = AtomicU64::new(0);
/// `(presents/s, skipped/s, capture→presented p50 ms)` of the last render window — the HUD's
/// display-side line.
pub fn present_stats() -> (u32, u32, f32) {
(
PRESENT_FPS.load(Ordering::Relaxed),
PRESENT_SKIPPED.load(Ordering::Relaxed),
PRESENT_P50_US.load(Ordering::Relaxed) as f32 / 1000.0,
)
/// The last render window's glass-side numbers (see the statics above) — the HUD's headline
/// (end-to-end) and trailing stage (display) come from here.
#[derive(Clone, Copy, Default, PartialEq)]
pub struct PresentStats {
/// Presents per second (includes resize redraws of a held frame).
pub fps: u32,
/// Frames dropped by the newest-wins drain this window (client-side pacing skips).
pub skipped: u32,
/// End-to-end capture→displayed p50, ms (host-clock corrected, measured directly).
pub e2e_p50_ms: f32,
/// End-to-end capture→displayed p95, ms.
pub e2e_p95_ms: f32,
/// `display` stage p50, ms: decoded → displayed, single-clock client-local.
pub display_p50_ms: f32,
}
pub fn present_stats() -> PresentStats {
PresentStats {
fps: PRESENT_FPS.load(Ordering::Relaxed),
skipped: PRESENT_SKIPPED.load(Ordering::Relaxed),
e2e_p50_ms: E2E_P50_US.load(Ordering::Relaxed) as f32 / 1000.0,
e2e_p95_ms: E2E_P95_US.load(Ordering::Relaxed) as f32 / 1000.0,
display_p50_ms: DISPLAY_P50_US.load(Ordering::Relaxed) as f32 / 1000.0,
}
}
/// UI-thread → render-thread state. Size is packed into ONE atomic (w<<32|h) so a resize never
@@ -101,8 +120,9 @@ impl Drop for RenderThread {
struct SendPresenter(Presenter);
unsafe impl Send for SendPresenter {}
/// Spawn the render thread. `frames` carries `(frame, capture pts_ns)`; `clock_offset_ns` maps our
/// wall clock onto the host's so the logged present latency is end-to-end (same math as the pump).
/// Spawn the render thread. `frames` carries `(frame, FrameTimes)`; `clock_offset_ns` maps our
/// wall clock onto the host's so the end-to-end (capture→on-glass) number is cross-machine valid
/// (same math as the pump's host+network stage).
pub fn spawn(
presenter: Presenter,
frames: FrameRx,
@@ -147,12 +167,17 @@ fn run(presenter: SendPresenter, frames: FrameRx, shared: Arc<RenderShared>, clo
let mut applied = (0u32, 0u32, 0u32); // last (w, h, dpi) handed to the presenter
let mut presented = 0u32;
let mut dropped = 0u32;
let mut lat_us: Vec<u64> = Vec::with_capacity(256);
// 1 s tumbling windows: end-to-end (capture→displayed) and the display stage
// (decoded→displayed), sampled post-Present. Percentiles only (spec: stats-unification.md).
let mut e2e_us: Vec<u64> = Vec::with_capacity(256);
let mut display_us: Vec<u64> = Vec::with_capacity(256);
let mut window_start = Instant::now();
let mut last_dpi_poll = Instant::now();
PRESENT_FPS.store(0, Ordering::Relaxed);
PRESENT_SKIPPED.store(0, Ordering::Relaxed);
PRESENT_P50_US.store(0, Ordering::Relaxed);
E2E_P50_US.store(0, Ordering::Relaxed);
E2E_P95_US.store(0, Ordering::Relaxed);
DISPLAY_P50_US.store(0, Ordering::Relaxed);
loop {
if shared.stop.load(Ordering::SeqCst) {
@@ -198,29 +223,55 @@ fn run(presenter: SendPresenter, frames: FrameRx, shared: Arc<RenderShared>, clo
p.set_hdr_metadata(meta);
}
let pts_ns = newest.as_ref().map(|(_, pts)| *pts);
let times: Option<FrameTimes> = newest.as_ref().map(|(_, t)| *t);
p.present(newest.map(|(f, _)| f));
presented += 1;
if let Some(pts) = pts_ns {
// Capture→presented, host-clock corrected — the glass-side companion to the pump's
// capture→decoded p50.
let lat = (now_ns() as i128 + clock_offset_ns as i128 - pts as i128).max(0) as u64;
if lat > 0 && lat < 10_000_000_000 {
lat_us.push(lat / 1000);
if let Some(t) = times {
// The `displayed` point: post-Present() on this thread (the honest best-effort
// presentation instant on Windows — endpoint label `capture→on-glass`).
let displayed_ns = now_ns();
// End-to-end = capture → displayed, host-clock corrected, measured directly
// (never the sum of stage percentiles). Clamped (0, 10 s).
let e2e =
(displayed_ns as i128 + clock_offset_ns as i128 - t.pts_ns as i128).max(0) as u64;
if e2e > 0 && e2e < 10_000_000_000 {
e2e_us.push(e2e / 1000);
}
// `display` stage = decoded → displayed, single-clock client-local.
let disp = displayed_ns.saturating_sub(t.decoded_ns);
if disp < 10_000_000_000 {
display_us.push(disp / 1000);
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
lat_us.sort_unstable();
let p50 = lat_us.get(lat_us.len() / 2).copied().unwrap_or(0);
tracing::debug!(presented, dropped, present_p50_us = p50, "render window");
e2e_us.sort_unstable();
display_us.sort_unstable();
let p50 = |v: &[u64]| v.get(v.len() / 2).copied().unwrap_or(0);
// p95 = sorted[min(len*95/100, len-1)] — the empty-window case falls to 0 via `get`.
let p95 = |v: &[u64]| {
v.get((v.len() * 95 / 100).min(v.len().saturating_sub(1)))
.copied()
.unwrap_or(0)
};
tracing::debug!(
presented,
dropped,
e2e_p50_us = p50(&e2e_us),
e2e_p95_us = p95(&e2e_us),
display_p50_us = p50(&display_us),
"render window"
);
PRESENT_FPS.store(presented, Ordering::Relaxed);
PRESENT_SKIPPED.store(dropped, Ordering::Relaxed);
PRESENT_P50_US.store(p50, Ordering::Relaxed);
E2E_P50_US.store(p50(&e2e_us), Ordering::Relaxed);
E2E_P95_US.store(p95(&e2e_us), Ordering::Relaxed);
DISPLAY_P50_US.store(p50(&display_us), Ordering::Relaxed);
window_start = Instant::now();
presented = 0;
dropped = 0;
lat_us.clear();
e2e_us.clear();
display_us.clear();
}
}
tracing::info!("render thread exiting");
+103 -30
View File
@@ -46,11 +46,27 @@ pub struct SessionParams {
#[derive(Clone, Copy, Default, PartialEq)]
pub struct Stats {
/// AUs received (reassembled) per second — actual-elapsed-time denominator.
pub fps: f32,
/// Received payload goodput (excludes FEC overhead).
pub mbps: f32,
/// `decode` stage p50 over the last 1 s window: received → decoded, client-local clock.
pub decode_ms: f32,
/// Median capture→decoded latency over the last window (host-clock corrected).
pub latency_ms: f32,
/// `host+network` stage p50 over the last 1 s window: capture (`pts_ns`) → received,
/// host-clock corrected via `clock_offset_ns`.
pub hostnet_ms: f32,
/// `host` stage p50 (host capture→sent, from the per-AU 0xCF host-timing plane). Valid only
/// when `split` — an old host emits no 0xCF and the HUD keeps the combined stage.
pub host_ms: f32,
/// `network` stage p50 (`hostnet host`, tiled per frame before taking the percentile).
/// Valid only when `split`.
pub net_ms: f32,
/// True when any 0xCF host timings matched received AUs this window — the HUD then renders
/// `host + network` instead of the combined `host+network` term.
pub split: bool,
/// True when `clock_offset_ns == 0` (host didn't answer the skew handshake / same host) —
/// the HUD appends `(same-host clock)` to the end-to-end line.
pub same_host: bool,
/// True when decoding on the GPU (D3D11VA) vs. CPU (software).
pub hardware: bool,
/// True when the stream is BT.2020 PQ HDR10 (last decoded frame).
@@ -81,9 +97,19 @@ pub enum SessionEvent {
Stats(Stats),
}
/// Decoded frames + their host-capture `pts_ns`, session pump → render thread (crossbeam so that
/// Per-frame measurement points carried with a decoded frame to the render thread: the host
/// capture clock (`pts_ns`) and our local `decoded` stamp (wall-clock ns). Post-`Present()` the
/// render thread derives the `display` stage (displayed decoded, single-clock) and the
/// end-to-end headline (displayed + clock_offset pts) from them.
#[derive(Clone, Copy)]
pub struct FrameTimes {
pub pts_ns: u64,
pub decoded_ns: u64,
}
/// Decoded frames + their measurement points, session pump → render thread (crossbeam so that
/// thread can block with a timeout — async-channel has no `recv_timeout`).
pub type FrameRx = crossbeam_channel::Receiver<(DecodedFrame, u64)>;
pub type FrameRx = crossbeam_channel::Receiver<(DecodedFrame, FrameTimes)>;
pub struct SessionHandle {
pub events: async_channel::Receiver<SessionEvent>,
@@ -205,7 +231,7 @@ impl AudioDec {
fn pump(
params: SessionParams,
ev_tx: async_channel::Sender<SessionEvent>,
frame_tx: crossbeam_channel::Sender<(DecodedFrame, u64)>,
frame_tx: crossbeam_channel::Sender<(DecodedFrame, FrameTimes)>,
frame_rx: FrameRx,
stop: Arc<AtomicBool>,
) {
@@ -310,8 +336,15 @@ fn pump(
let mut window_start = Instant::now();
let mut frames_n = 0u32;
let mut bytes_n = 0u64;
let mut decode_us_sum = 0u64;
let mut lat_us: Vec<u64> = Vec::with_capacity(256);
// 1 s tumbling stage windows (spec: design/stats-unification.md — percentiles, never means).
let mut hostnet_us: Vec<u64> = Vec::with_capacity(256);
let mut decode_us: Vec<u64> = Vec::with_capacity(256);
// Host/network split (Phase 2): received AUs awaiting their 0xCF host timing, `(pts_ns,
// hostnet_us)`, matched as the datagrams arrive. Bounded — an old host never sends any.
let mut pending_split: std::collections::VecDeque<(u64, u64)> =
std::collections::VecDeque::with_capacity(256);
let mut host_us_w: Vec<u64> = Vec::with_capacity(256);
let mut net_us_w: Vec<u64> = Vec::with_capacity(256);
let mut pcm = vec![0f32; 5760 * channels as usize]; // scratch: max Opus frame (120 ms) × channels
// Loss recovery: watch the host→client unrecoverable-drop count and ask for an IDR when it climbs.
let mut last_dropped = connector.frames_dropped();
@@ -323,7 +356,23 @@ fn pump(
}
match connector.next_frame(Duration::from_millis(4)) {
Ok(frame) => {
let t0 = Instant::now();
// The `received` point: AU fully reassembled, handed to us, before decode.
let received_ns = now_ns();
// fps = AUs received per second, Mb/s = received goodput (spec: counted at the
// received point, not the decoded one).
frames_n += 1;
bytes_n += frame.data.len() as u64;
// `host+network` stage: capture → received, host-clock corrected. Clamped (0, 10 s).
let hostnet = (received_ns as i128 + clock_offset as i128 - frame.pts_ns as i128)
.max(0) as u64;
if hostnet > 0 && hostnet < 10_000_000_000 {
hostnet_us.push(hostnet / 1000);
// Remember this AU for the 0xCF match below (host/network split).
pending_split.push_back((frame.pts_ns, hostnet / 1000));
if pending_split.len() > 256 {
pending_split.pop_front();
}
}
// A D3D11VA→software demotion (see `Decoder::decode`) starts a FRESH decoder that
// has none of the stream's parameter sets; under infinite GOP it would sit on
// "PPS id out of range" forever. Detect the transition and force a new IDR so the
@@ -336,6 +385,8 @@ fn pump(
}
match decoded {
Ok(Some(decoded)) => {
// The `decoded` point: decoder output frame available.
let decoded_ns = now_ns();
total_frames += 1;
hdr = decoded.hdr();
// The backend can demote D3D11VA → software mid-session on a hardware error.
@@ -350,19 +401,17 @@ fn pump(
"first frame decoded"
);
}
// Latency: our wall clock expressed in the host's capture clock,
// minus the host-stamped capture pts (same math as client-rs).
let lat = (now_ns() as i128 + clock_offset as i128 - frame.pts_ns as i128)
.max(0) as u64;
if lat > 0 && lat < 10_000_000_000 {
lat_us.push(lat / 1000);
}
decode_us_sum += t0.elapsed().as_micros() as u64;
frames_n += 1;
bytes_n += frame.data.len() as u64;
// `decode` stage: received → decoded, single-clock client-local.
decode_us.push(decoded_ns.saturating_sub(received_ns) / 1000);
// Newest wins: displace the oldest queued frame when the renderer lags.
if let Err(crossbeam_channel::TrySendError::Full(item)) =
frame_tx.try_send((decoded, frame.pts_ns))
frame_tx.try_send((
decoded,
FrameTimes {
pts_ns: frame.pts_ns,
decoded_ns,
},
))
{
let _ = frame_rx.try_recv();
let _ = frame_tx.try_send(item);
@@ -411,25 +460,47 @@ fn pump(
*crate::present::LATEST_HDR_META.lock().unwrap() = Some(meta);
}
// Drain the per-AU host-timing plane (0xCF) and match by pts: `host` = the host's own
// capture→sent, `network` = our capture→received minus it — the two tile per frame
// (design/stats-unification.md Phase 2). An old host never emits any; `split` stays false
// and the HUD keeps the combined `host+network` stage.
while let Ok(t) = connector.next_host_timing(Duration::ZERO) {
if let Some(i) = pending_split.iter().position(|(p, _)| *p == t.pts_ns) {
let (_, hn_us) = pending_split.remove(i).unwrap();
host_us_w.push(t.host_us as u64);
net_us_w.push(hn_us.saturating_sub(t.host_us as u64));
}
}
if window_start.elapsed() >= Duration::from_secs(1) {
let secs = window_start.elapsed().as_secs_f32();
lat_us.sort_unstable();
let p50 = lat_us.get(lat_us.len() / 2).copied().unwrap_or(0);
hostnet_us.sort_unstable();
decode_us.sort_unstable();
host_us_w.sort_unstable();
net_us_w.sort_unstable();
let p50 = |v: &[u64]| v.get(v.len() / 2).copied().unwrap_or(0);
let (hostnet_p50, decode_p50) = (p50(&hostnet_us), p50(&decode_us));
let (host_p50, net_p50) = (p50(&host_us_w), p50(&net_us_w));
let split = !host_us_w.is_empty();
tracing::debug!(
fps = frames_n,
lat_p50_us = p50,
hostnet_p50_us = hostnet_p50,
host_p50_us = host_p50,
net_p50_us = net_p50,
split,
decode_p50_us = decode_p50,
total_frames,
"stream window"
);
let _ = ev_tx.try_send(SessionEvent::Stats(Stats {
fps: frames_n as f32 / secs,
mbps: bytes_n as f32 * 8.0 / 1e6 / secs,
decode_ms: if frames_n > 0 {
decode_us_sum as f32 / frames_n as f32 / 1000.0
} else {
0.0
},
latency_ms: p50 as f32 / 1000.0,
decode_ms: decode_p50 as f32 / 1000.0,
hostnet_ms: hostnet_p50 as f32 / 1000.0,
host_ms: host_p50 as f32 / 1000.0,
net_ms: net_p50 as f32 / 1000.0,
split,
same_host: clock_offset == 0,
hardware,
hdr,
codec: connector.codec,
@@ -439,8 +510,10 @@ fn pump(
window_start = Instant::now();
frames_n = 0;
bytes_n = 0;
decode_us_sum = 0;
lat_us.clear();
hostnet_us.clear();
decode_us.clear();
host_us_w.clear();
net_us_w.clear();
}
};
+7 -3
View File
@@ -151,9 +151,13 @@ pub mod control {
}
/// `IOCTL_SET_FRAME_CHANNEL` input — the sealed frame channel's bootstrap. Every handle field is a
/// handle VALUE already duplicated into the driver's WUDFHost process by the host; receiving it, the
/// driver OWNS those handles (it closes whatever it doesn't consume — a replaced, invalid, or
/// unmatched delivery must not leak entries in its own handle table).
/// handle VALUE already duplicated into the driver's WUDFHost process by the host. Ownership is
/// **adopt-on-success-only** (`design/idd-push-security.md` invariant 5): the driver owns (and
/// eventually closes) the handles IFF it completes the IOCTL successfully — a replaced or
/// later-unconsumed delivery is then the driver's to close. On ANY error completion (malformed
/// request, unknown `target_id`) the driver must NOT close them: the HOST reaps its remote
/// duplicates (`DUPLICATE_CLOSE_SOURCE`). Exactly one side closes each value; a driver that closed
/// on error would double-close possibly-reused handle values against the host's reap.
///
/// Handle values are only meaningful inside the target process's handle table, so this struct is
/// harmless to any third party: reading it leaks nothing openable, and spoofing it (were the control
+59
View File
@@ -635,6 +635,22 @@ impl PunktfunkHdrMeta {
}
}
/// One access unit's host-side processing time ([`punktfunk_connection_next_host_timing`]):
/// capture → fully sent, i.e. the whole host pipeline (capture read/convert, encode, FEC+seal,
/// paced send). Correlate to the AU whose `PunktfunkFrame::pts_ns` equals `pts_ns`, then
/// `network = (received_instant + clock_offset pts_ns) host_us` — the unified stats HUD's
/// `host` / `network` split (design/stats-unification.md Phase 2). Best-effort: a lost datagram
/// means that frame simply contributes no sample.
#[cfg(feature = "quic")]
#[repr(C)]
#[derive(Clone, Copy)]
pub struct PunktfunkHostTiming {
/// The AU's capture stamp (host capture clock — matches `PunktfunkFrame::pts_ns` exactly).
pub pts_ns: u64,
/// Host capture→sent duration, µs.
pub host_us: u32,
}
/// `PunktfunkRichInput::kind` — a touchpad contact (`finger`/`active`/`x`/`y` valid).
pub const PUNKTFUNK_RICH_TOUCHPAD: u8 = 1;
/// `PunktfunkRichInput::kind` — a motion sample (`gyro`/`accel` valid).
@@ -1759,6 +1775,49 @@ pub unsafe extern "C" fn punktfunk_connection_next_hdr_meta(
})
}
/// Pull the next per-AU host timing (0xCF) into `*out`: the host's capture→sent duration for one
/// access unit, correlated to the AU by `pts_ns` (see [`PunktfunkHostTiming`]).
/// [`PunktfunkStatus::NoFrame`] on timeout, [`PunktfunkStatus::Closed`] once the session ended.
/// A stats consumer drains this non-blockingly (`timeout_ms = 0`) alongside its frame samples;
/// an older host never emits any — keep showing the combined `host+network` stage then. Same
/// threading rules as [`punktfunk_connection_next_rumble`] (one puller, may run alongside the
/// other planes).
///
/// # Safety
/// `c` is a valid connection handle; `out` is writable for one `PunktfunkHostTiming`.
#[cfg(feature = "quic")]
#[no_mangle]
pub unsafe extern "C" fn punktfunk_connection_next_host_timing(
c: *mut PunktfunkConnection,
out: *mut PunktfunkHostTiming,
timeout_ms: u32,
) -> PunktfunkStatus {
guard(|| {
let c = match unsafe { c.as_ref() } {
Some(c) => c,
None => return PunktfunkStatus::NullPointer,
};
if out.is_null() {
return PunktfunkStatus::NullPointer;
}
match c
.inner
.next_host_timing(std::time::Duration::from_millis(timeout_ms as u64))
{
Ok(t) => {
unsafe {
*out = PunktfunkHostTiming {
pts_ns: t.pts_ns,
host_us: t.host_us,
}
};
PunktfunkStatus::Ok
}
Err(e) => e.status(),
}
})
}
/// Read the session's resolved colour signalling + encode bit depth (from the host's Welcome).
/// Each out pointer is filled when non-NULL: `primaries`/`transfer`/`matrix` are CICP code points
/// (BT.709 = 1; BT.2020 = 9; PQ transfer = 16, HLG = 18; BT.2020-NCL matrix = 9), `full_range` is
+99 -5
View File
@@ -140,6 +140,11 @@ const HIDOUT_QUEUE: usize = 32;
/// and low-rate (one on start, re-sent on mastering changes / keyframes); a small ring is ample.
const HDR_META_QUEUE: usize = 8;
/// Host-timing plane depth (0xCF, one datagram per AU). Sized for a 240 fps stream whose stats
/// consumer drains once per second with headroom; overflow drops the newest sample (try_send) —
/// harmless, it's per-frame observability, not state.
const HOST_TIMING_QUEUE: usize = 512;
/// One Opus packet from the host's audio datagram stream (48 kHz stereo, 5 ms frames).
#[derive(Clone, Debug)]
pub struct AudioPacket {
@@ -161,6 +166,9 @@ pub struct NativeClient {
hidout: Mutex<Receiver<HidOutput>>,
/// Inbound static HDR metadata (ST.2086 mastering + content light level) — 0xCE datagrams.
hdr_meta: Mutex<Receiver<HdrMeta>>,
/// Inbound per-AU host capture→send timings — 0xCF datagrams (the client always advertises
/// [`quic::VIDEO_CAP_HOST_TIMING`]; an older host simply never sends any).
host_timing: Mutex<Receiver<crate::quic::HostTiming>>,
input_tx: tokio::sync::mpsc::UnboundedSender<InputEvent>,
/// Outbound mic frames `(seq, pts_ns, opus)` → encoded as 0xCB datagrams by the worker.
mic_tx: tokio::sync::mpsc::UnboundedSender<(u32, u64, Vec<u8>)>,
@@ -176,6 +184,12 @@ pub struct NativeClient {
/// a recovery keyframe under infinite GOP — the correct loss trigger, since unrecoverable loss
/// yields reference-missing frames the decoder silently conceals (a decode-error trigger misses them).
frames_dropped: Arc<AtomicU64>,
/// Kernel ids of the client's latency-critical native threads: the internal data-plane pump
/// (UDP receive + FEC reassembly) plus any embedder plane threads registered via
/// [`NativeClient::register_hot_thread`]. The Android client feeds these to an ADPF hint session
/// so the CPU governor keeps the whole video pipeline on fast cores. Empty on platforms without
/// `gettid` (see [`current_hot_tid`]).
hot_tids: Arc<Mutex<Vec<i32>>>,
worker: Option<std::thread::JoinHandle<()>>,
/// The currently active session mode (the Welcome's, then updated by every accepted
/// [`NativeClient::request_mode`]).
@@ -242,6 +256,32 @@ fn pin_thread_user_interactive() {
#[cfg(not(target_vendor = "apple"))]
fn pin_thread_user_interactive() {}
/// The calling thread's kernel id, for hot-thread performance hints (the Android client's ADPF
/// session today; the consumer is platform-specific). Linux/Android expose `gettid`; elsewhere
/// there's nothing to hint with, so registration is a no-op.
#[cfg(any(target_os = "android", target_os = "linux"))]
fn current_hot_tid() -> Option<i32> {
// SAFETY: `gettid` reads the calling thread's kernel id — an always-safe syscall, no args.
Some(unsafe { libc::gettid() })
}
#[cfg(not(any(target_os = "android", target_os = "linux")))]
fn current_hot_tid() -> Option<i32> {
None
}
/// Record the calling thread's id in the shared hot-thread registry (deduped). Best-effort: a
/// platform without `gettid` or a poisoned lock just skips it — a missed performance hint, not an
/// error on the data path.
fn register_hot_tid(reg: &Mutex<Vec<i32>>) {
if let Some(t) = current_hot_tid() {
if let Ok(mut v) = reg.lock() {
if !v.contains(&t) {
v.push(t);
}
}
}
}
impl NativeClient {
/// Connect to a `punktfunk/1` host and start the session at (up to) `mode`. Blocks until the
/// handshake completes or `timeout` elapses.
@@ -283,6 +323,8 @@ impl NativeClient {
let (rumble_tx, rumble_rx) = std::sync::mpsc::sync_channel::<(u16, u16, u16)>(RUMBLE_QUEUE);
let (hidout_tx, hidout_rx) = std::sync::mpsc::sync_channel::<HidOutput>(HIDOUT_QUEUE);
let (hdr_meta_tx, hdr_meta_rx) = std::sync::mpsc::sync_channel::<HdrMeta>(HDR_META_QUEUE);
let (host_timing_tx, host_timing_rx) =
std::sync::mpsc::sync_channel::<crate::quic::HostTiming>(HOST_TIMING_QUEUE);
let (input_tx, input_rx) = tokio::sync::mpsc::unbounded_channel::<InputEvent>();
let (mic_tx, mic_rx) = tokio::sync::mpsc::unbounded_channel::<(u32, u64, Vec<u8>)>();
let (rich_input_tx, rich_input_rx) = tokio::sync::mpsc::unbounded_channel::<RichInput>();
@@ -292,12 +334,14 @@ impl NativeClient {
let mode_slot = Arc::new(std::sync::Mutex::new(mode));
let probe = Arc::new(Mutex::new(ProbeState::default()));
let frames_dropped = Arc::new(AtomicU64::new(0));
let hot_tids = Arc::new(Mutex::new(Vec::new()));
let host = host.to_string();
let shutdown_w = shutdown.clone();
let mode_slot_w = mode_slot.clone();
let probe_w = probe.clone();
let frames_dropped_w = frames_dropped.clone();
let hot_tids_w = hot_tids.clone();
let ctrl_tx_pump = ctrl_tx.clone(); // the data-plane pump sends adaptive-FEC LossReports
let worker = std::thread::Builder::new()
.name("punktfunk-client".into())
@@ -336,6 +380,7 @@ impl NativeClient {
rumble_tx,
hidout_tx,
hdr_meta_tx,
host_timing_tx,
input_rx,
mic_rx,
rich_input_rx,
@@ -346,6 +391,7 @@ impl NativeClient {
mode_slot: mode_slot_w,
probe: probe_w,
frames_dropped: frames_dropped_w,
hot_tids: hot_tids_w,
}));
})
.map_err(PunktfunkError::Io)?;
@@ -377,6 +423,7 @@ impl NativeClient {
rumble: Mutex::new(rumble_rx),
hidout: Mutex::new(hidout_rx),
hdr_meta: Mutex::new(hdr_meta_rx),
host_timing: Mutex::new(host_timing_rx),
input_tx,
mic_tx,
rich_input_tx,
@@ -385,6 +432,7 @@ impl NativeClient {
shutdown,
worker: Some(worker),
frames_dropped,
hot_tids,
mode: mode_slot,
host_fingerprint: fingerprint,
resolved_compositor,
@@ -526,6 +574,25 @@ impl NativeClient {
self.frames_dropped.load(Ordering::Relaxed)
}
/// Register the calling thread as latency-critical so a later
/// [`hot_thread_ids`](Self::hot_thread_ids) includes it. An embedder calls this from its own
/// plane threads (e.g. the Android client's decode + audio threads) to fold them into the same
/// performance-hint session as the internal data-plane pump. Idempotent per thread; a no-op on
/// platforms without `gettid`.
pub fn register_hot_thread(&self) {
register_hot_tid(&self.hot_tids);
}
/// Kernel ids of the client's latency-critical threads: the internal data-plane pump (UDP
/// receive + FEC reassembly) plus any registered via
/// [`register_hot_thread`](Self::register_hot_thread). The Android client feeds these to an ADPF
/// hint session so the CPU governor keeps the whole video pipeline on fast cores. Empty where
/// thread ids aren't available (platforms without `gettid`); call after the first frame so the
/// pump has registered.
pub fn hot_thread_ids(&self) -> Vec<i32> {
self.hot_tids.lock().map(|v| v.clone()).unwrap_or_default()
}
/// Start a bandwidth speed test: ask the host to burst filler over the data plane at
/// `target_kbps` of goodput for `duration_ms`, *briefly pausing video*. Non-blocking — the
/// measurement accumulates in the background; poll [`NativeClient::probe_result`] until its
@@ -660,6 +727,20 @@ impl NativeClient {
}
}
/// Pull the next per-AU host timing (0xCF): the host's capture→sent duration for one access
/// unit, correlated to the AU by `pts_ns`. Feeds the unified stats HUD's `host` / `network`
/// split (`network = (received + clock_offset pts) host_us`); a stats consumer should
/// drain this non-blockingly alongside its frame samples. An older host never sends any —
/// the HUD then keeps the combined `host+network` stage. Same timeout/closed semantics as
/// [`NativeClient::next_hidout`].
pub fn next_host_timing(&self, timeout: Duration) -> Result<crate::quic::HostTiming> {
match self.host_timing.lock().unwrap().recv_timeout(timeout) {
Ok(t) => Ok(t),
Err(RecvTimeoutError::Timeout) => Err(PunktfunkError::NoFrame),
Err(RecvTimeoutError::Disconnected) => Err(PunktfunkError::Closed),
}
}
/// Queue one input event for delivery as a QUIC datagram.
pub fn send_input(&self, ev: &InputEvent) -> Result<()> {
self.input_tx.send(*ev).map_err(|_| PunktfunkError::Closed)
@@ -713,6 +794,7 @@ struct WorkerArgs {
rumble_tx: SyncSender<(u16, u16, u16)>,
hidout_tx: SyncSender<HidOutput>,
hdr_meta_tx: SyncSender<HdrMeta>,
host_timing_tx: SyncSender<crate::quic::HostTiming>,
input_rx: tokio::sync::mpsc::UnboundedReceiver<InputEvent>,
mic_rx: tokio::sync::mpsc::UnboundedReceiver<(u32, u64, Vec<u8>)>,
rich_input_rx: tokio::sync::mpsc::UnboundedReceiver<RichInput>,
@@ -723,6 +805,7 @@ struct WorkerArgs {
mode_slot: Arc<std::sync::Mutex<Mode>>,
probe: Arc<Mutex<ProbeState>>,
frames_dropped: Arc<AtomicU64>,
hot_tids: Arc<Mutex<Vec<i32>>>,
}
/// The worker: QUIC handshake, then the input/datagram/control tasks + the blocking
@@ -747,6 +830,7 @@ async fn worker_main(args: WorkerArgs) {
rumble_tx,
hidout_tx,
hdr_meta_tx,
host_timing_tx,
mut input_rx,
mut mic_rx,
mut rich_input_rx,
@@ -757,6 +841,7 @@ async fn worker_main(args: WorkerArgs) {
mode_slot,
probe,
frames_dropped,
hot_tids,
} = args;
let setup = async {
let remote: std::net::SocketAddr = format!("{host}:{port}")
@@ -803,8 +888,10 @@ async fn worker_main(args: WorkerArgs) {
launch: launch.clone(),
// The embedder's decode/present caps (e.g. the Windows client advertises
// VIDEO_CAP_10BIT | VIDEO_CAP_HDR). The host only upgrades to a 10-bit / HDR encode
// when the matching bit is set, so `0` stays an 8-bit BT.709 stream.
video_caps,
// when the matching bit is set, so `0` stays an 8-bit BT.709 stream. HOST_TIMING is
// OR'd in unconditionally: every NativeClient build demuxes the 0xCF plane, and the
// bit only asks the host for observability datagrams (never changes the encode).
video_caps: video_caps | crate::quic::VIDEO_CAP_HOST_TIMING,
// Requested surround channel count; the host echoes the resolved value in Welcome.
audio_channels,
// The codecs this client can decode + its soft preference (0 = auto). The host
@@ -1042,6 +1129,11 @@ async fn worker_main(args: WorkerArgs) {
let _ = hdr_meta_tx.try_send(m);
}
}
Some(&crate::quic::HOST_TIMING_MAGIC) => {
if let Some(t) = crate::quic::decode_host_timing_datagram(&d) {
let _ = host_timing_tx.try_send(t);
}
}
_ => {} // unknown tag — a newer host; ignore
}
}
@@ -1063,11 +1155,13 @@ async fn worker_main(args: WorkerArgs) {
// decoder queue — it isn't video.
let pump_shutdown = shutdown.clone();
let pump_probe = probe.clone();
let pump_hot_tids = hot_tids.clone();
let _ = tokio::task::spawn_blocking(move || {
pin_thread_user_interactive(); // feeds frame_tx → the client's user-interactive video pump
// Adaptive-FEC loss reporting: every ADAPT_REPORT_INTERVAL, report the loss observed over the
// window (shards FEC recovered, plus a bump if any frame went unrecoverable) so the host can
// size FEC to the link. Suppressed during a speed test (its FLAG_PROBE filler would skew it).
register_hot_tid(&pump_hot_tids); // this thread does UDP receive + FEC reassembly — hint it
// Adaptive-FEC loss reporting: every ADAPT_REPORT_INTERVAL, report the loss observed over the
// window (shards FEC recovered, plus a bump if any frame went unrecoverable) so the host can
// size FEC to the link. Suppressed during a speed test (its FLAG_PROBE filler would skew it).
const ADAPT_REPORT_INTERVAL: Duration = Duration::from_millis(750);
let mut last_report = Instant::now();
let (mut last_recovered, mut last_received, mut last_dropped) = (0u64, 0u64, 0u64);
+71 -1
View File
@@ -114,6 +114,13 @@ pub const VIDEO_CAP_HDR: u8 = 0x02;
/// [`Welcome::chroma_format`] reflects the real resolved value. Independent of 10-bit/HDR (4:4:4 is a
/// chroma decision, bit depth is a depth decision; the two may combine where the hardware allows).
pub const VIDEO_CAP_444: u8 = 0x04;
/// [`Hello::video_caps`] bit: the client consumes per-AU host-timing datagrams
/// ([`HOST_TIMING_MAGIC`], 0xCF) — the host's capture→send duration per frame, letting the client
/// split its `host+network` latency stage into `host` and `network`
/// (design/stats-unification.md Phase 2). The host emits 0xCF ONLY when this bit is set (an older
/// host ignores it and simply never sends any); a client that doesn't set it keeps the combined
/// stage. Purely observability — never changes what the host encodes.
pub const VIDEO_CAP_HOST_TIMING: u8 = 0x08;
/// [`Hello::video_codecs`] bit: the client can decode H.264 / AVC. The GPU-less **software**
/// encode path (openh264) emits H.264, so a client that wants to stream from a software host MUST
@@ -390,7 +397,7 @@ pub struct ProbeResult {
/// `client → host`, right after [`Start`]: one round of the wall-clock skew handshake. The client
/// stamps `t1_ns` (its monotonic-since-epoch clock) and sends; the host echoes it in [`ClockEcho`]
/// with its own receive/send stamps. A few rounds let the client estimate the host↔client clock
/// offset, so the per-frame `capture→reassembled` latency (the AU `pts_ns` is the host's capture
/// offset, so the per-frame `capture→received` latency (the AU `pts_ns` is the host's capture
/// clock) is meaningful across machines, not just same-host. An old host ignores it (the client
/// times out and assumes a shared clock).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -1601,6 +1608,50 @@ pub fn decode_hdr_meta_datagram(b: &[u8]) -> Option<HdrMeta> {
})
}
/// Per-AU host-timing datagram tag, host → client (see [`HostTiming`]). Next tag after
/// [`HDR_META_MAGIC`]. Emitted once per access unit, right after its last packet left the host's
/// socket, and only when the client advertised [`VIDEO_CAP_HOST_TIMING`].
pub const HOST_TIMING_MAGIC: u8 = 0xCF;
/// One access unit's host-side processing time: capture → fully sent (the whole host pipeline —
/// capture read/convert, encode, FEC+seal, paced send). The client correlates it to the AU by
/// `pts_ns` (the AU's capture stamp, unique per frame) and derives
/// `network = (received + clock_offset pts_ns) host_us`, so the unified-stats equation's
/// `host+network` stage splits into two per-frame-tiling terms. Best-effort like every side-plane
/// datagram: a lost 0xCF just means that frame contributes no host/network sample.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct HostTiming {
/// The AU's capture stamp (host capture clock — matches the AU's `pts_ns` exactly).
pub pts_ns: u64,
/// Host capture→sent duration, µs (saturated at `u32::MAX` ≈ 71 min — far past the 10 s
/// client-side sanity clamp anyway).
pub host_us: u32,
}
/// Wire length of a [`HOST_TIMING_MAGIC`] datagram: tag + u64 pts + u32 µs = 13 bytes.
const HOST_TIMING_LEN: usize = 1 + 8 + 4;
/// Encode a [`HostTiming`] into a [`HOST_TIMING_MAGIC`] datagram.
pub fn encode_host_timing_datagram(t: &HostTiming) -> Vec<u8> {
let mut b = Vec::with_capacity(HOST_TIMING_LEN);
b.push(HOST_TIMING_MAGIC);
b.extend_from_slice(&t.pts_ns.to_le_bytes());
b.extend_from_slice(&t.host_us.to_le_bytes());
b
}
/// Parse a [`HOST_TIMING_MAGIC`] datagram → [`HostTiming`]. `None` on bad tag or a short buffer
/// (the fixed length bounds every read before it happens).
pub fn decode_host_timing_datagram(b: &[u8]) -> Option<HostTiming> {
if b.len() < HOST_TIMING_LEN || b[0] != HOST_TIMING_MAGIC {
return None;
}
Some(HostTiming {
pts_ns: u64::from_le_bytes(b[1..9].try_into().unwrap()),
host_us: u32::from_le_bytes(b[9..13].try_into().unwrap()),
})
}
/// Async framed-message IO over a quinn stream (`u16 LE length || payload`).
pub mod io {
/// Read one framed message (bounded at 64 KiB — control messages are tiny).
@@ -2189,6 +2240,25 @@ mod tests {
assert_eq!(decode_hdr_meta_datagram(&bad), None);
}
#[test]
fn host_timing_datagram_roundtrip_and_truncation() {
let t = HostTiming {
pts_ns: 1_751_500_000_123_456_789, // a realistic 2026 CLOCK_REALTIME capture stamp
host_us: 4_321,
};
let d = encode_host_timing_datagram(&t);
assert_eq!(d[0], HOST_TIMING_MAGIC);
assert_eq!(d.len(), 13);
assert_eq!(decode_host_timing_datagram(&d), Some(t));
// Truncated buffers and a wrong tag are rejected (never partially read).
for n in 0..d.len() {
assert_eq!(decode_host_timing_datagram(&d[..n]), None);
}
let mut bad = d.clone();
bad[0] = HDR_META_MAGIC;
assert_eq!(decode_host_timing_datagram(&bad), None);
}
#[test]
fn hello_start_roundtrip() {
let h = Hello {
+5
View File
@@ -243,3 +243,8 @@ nvenc = ["dep:nvidia-video-codec-sdk"]
# so the LGPL build suffices and keeps the bundled DLLs LGPL, not GPL) at build time and bundles the
# FFmpeg DLLs at runtime. Build the all-vendor GPU host with `--features nvenc,amf-qsv`.
amf-qsv = ["dep:ffmpeg-next"]
# Build-time icon/version-info embedding (build.rs; Windows dev/CI hosts only — Linux packaging
# builds of this crate never execute the winresource block).
[target.'cfg(windows)'.build-dependencies]
winresource = "0.1"
+17
View File
@@ -17,4 +17,21 @@ fn main() {
.unwrap_or_else(|| std::env::var("CARGO_PKG_VERSION").unwrap_or_else(|_| "unknown".into()));
println!("cargo:rustc-env=PUNKTFUNK_VERSION={version}");
println!("cargo:rerun-if-env-changed=PUNKTFUNK_BUILD_VERSION");
// Windows identity resources: the branded icon + version info. Task Manager / Explorer show a
// process by its version-info FileDescription — without one the host appears as a bare
// "punktfunk-host.exe" with no icon. Same winresource pattern as clients/windows and
// punktfunk-tray (cfg(windows) = build HOST, so Linux packaging builds skip it; CARGO_CFG_WINDOWS
// = TARGET).
#[cfg(windows)]
if std::env::var_os("CARGO_CFG_WINDOWS").is_some() {
let icon = "../../packaging/windows/branding/punktfunk.ico";
println!("cargo:rerun-if-changed={icon}");
winresource::WindowsResource::new()
.set_icon_with_id(icon, "1")
.set("FileDescription", "Punktfunk Host")
.set("ProductName", "Punktfunk")
.compile()
.expect("embed windows icon/version resources");
}
}
+450 -5
View File
@@ -42,7 +42,8 @@ pub fn open_audio_capture(channels: u32) -> Result<Box<dyn AudioCapturer>> {
#[cfg(target_os = "windows")]
pub fn open_audio_capture(channels: u32) -> Result<Box<dyn AudioCapturer>> {
audio_control::ensure_wired_once();
// The capture thread runs the audio wiring plan itself (audio_control::wire_now) before
// resolving its endpoint — a fresh plan per open, because Windows endpoints churn.
wasapi_cap::WasapiLoopbackCapturer::open(channels)
.map(|c| Box::new(c) as Box<dyn AudioCapturer>)
}
@@ -57,10 +58,27 @@ pub fn open_audio_capture(_channels: u32) -> Result<Box<dyn AudioCapturer>> {
/// decoded client-mic PCM (interleaved `f32` at [`SAMPLE_RATE`]) into it, and PipeWire delivers
/// it to whichever app records the source — silence when no input is flowing. This is how the
/// client's microphone reaches host applications (mic passthrough).
///
/// **Liveness contract.** Both backends run a worker thread that CAN die under the host's feet
/// (Linux: the PipeWire daemon restarts with the session; Windows: the audio endpoint is
/// invalidated/removed). A dead backend must be observable — [`push`](Self::push) returns `false`
/// and [`alive`](Self::alive) turns false — so the owning [`MicPump`] drops the instance and
/// reopens. Before this contract existed, a single backend death left `push` feeding a dead
/// queue for the rest of the host's life: the historical "mic passthrough works on no host" bug.
pub trait VirtualMic: Send {
/// Push one chunk of interleaved `f32` PCM. Non-blocking — drops if PipeWire is behind
/// (mic audio is lossy/real-time; a stale chunk is worse than a dropped one).
fn push(&self, pcm: &[f32]);
/// Push one chunk of interleaved `f32` PCM. Non-blocking — drops if the backend is behind
/// (mic audio is lossy/real-time; a stale chunk is worse than a dropped one). Returns
/// `false` iff the backend is DEAD (worker thread gone) — the caller must reopen; a merely
/// congested backend drops the chunk and returns `true`.
fn push(&self, pcm: &[f32]) -> bool;
/// Backend liveness without pushing data — lets an idle pump notice a death between
/// sessions, so the mic is already healthy again when the next client connects.
fn alive(&self) -> bool;
/// Drop any buffered-but-unplayed audio. Called after an uplink gap (client muted,
/// session ended) so a recorder never hears a stale burst when audio resumes.
fn discard(&self);
/// The interleaved channel count the source was opened with.
fn channels(&self) -> u32 {
@@ -78,7 +96,8 @@ pub fn open_virtual_mic(channels: u32) -> Result<Box<dyn VirtualMic>> {
#[cfg(target_os = "windows")]
pub fn open_virtual_mic(channels: u32) -> Result<Box<dyn VirtualMic>> {
audio_control::ensure_wired_once();
// The render thread runs the wiring plan itself (audio_control::wire_now) to resolve — and,
// via the plan's default-device changes, to RESERVE — its target endpoint.
wasapi_mic::WasapiVirtualMic::open(channels).map(|m| Box::new(m) as Box<dyn VirtualMic>)
}
@@ -87,6 +106,220 @@ pub fn open_virtual_mic(_channels: u32) -> Result<Box<dyn VirtualMic>> {
anyhow::bail!("virtual mic requires Linux + PipeWire or Windows + a virtual audio device")
}
/// Mic is 48 kHz stereo — matches the Opus stereo decoder and the host→client audio layout.
pub const MIC_CHANNELS: u32 = 2;
/// Bound for the shared mic frame queue (drop-newest when full): the host-lifetime queue is
/// shared across all concurrent sessions and must not grow without limit under a near-line-rate
/// flood (security-review 2026-06-28 S6). 64 × 520 ms frames ≈ 0.31.3 s of slack.
const MIC_QUEUE_CAP: usize = 64;
/// Tuning for [`MicPump`]'s open/reopen/flush behaviour — parameterized so the tests can run the
/// real pump loop in milliseconds instead of seconds.
#[derive(Clone, Copy)]
struct PumpTuning {
/// First-retry delay after a failed backend open; doubles per failure up to `backoff_cap`
/// (a persistently-absent PipeWire session / audio endpoint isn't hammered), resets on
/// success.
backoff_start: std::time::Duration,
backoff_cap: std::time::Duration,
/// Idle liveness-probe interval: with no frames flowing, the pump still notices a dead
/// backend this often and reopens — so the mic is healthy BEFORE the next session starts.
heartbeat: std::time::Duration,
/// An uplink gap longer than this discards the backend's buffered audio before pushing the
/// next frame (a recorder must never hear a stale burst from before a mute/session end).
stale_gap: std::time::Duration,
/// A backend that dies before living this long counts as a FAILED open for backoff purposes
/// (an open that succeeds but dies instantly — e.g. a flapping daemon — must not churn at
/// heartbeat rate); one that lived longer resets the backoff.
stable_after: std::time::Duration,
}
const PUMP_TUNING: PumpTuning = PumpTuning {
backoff_start: std::time::Duration::from_secs(2),
backoff_cap: std::time::Duration::from_secs(60),
heartbeat: std::time::Duration::from_secs(1),
stale_gap: std::time::Duration::from_millis(600),
stable_after: std::time::Duration::from_secs(5),
};
/// Host-lifetime virtual-microphone pump: one thread owns the [`VirtualMic`] backend + an Opus
/// decoder; sessions forward the client's Opus mic frames (0xCB) over a clonable `Send` sender,
/// the thread decodes and feeds the backend.
///
/// The rock-solid properties live HERE, not in the backends:
/// - **Eager**: the backend opens at host start (retrying with backoff), NOT on the first mic
/// frame — so the virtual mic device already exists when host apps/games launch and bind
/// their capture device (most games never re-follow a default-device change mid-run).
/// - **Self-healing**: a dead backend (PipeWire restart, Windows endpoint churn) is detected on
/// every push and on an idle heartbeat, and reopened with backoff. Sessions keep their
/// senders; nothing upstream notices.
/// - **Stale-flush**: buffered audio is discarded after an uplink gap (see [`PumpTuning`]).
///
/// Per-frame Opus DECODE errors stay non-fatal (dropped frame): the mic is shared across every
/// concurrent session, so one paired client's junk frames must not deny everyone's mic
/// (security-review 2026-06-28 S2). The thread exits when every sender is dropped (host
/// shutdown), tearing the backend down.
pub struct MicPump {
tx: std::sync::mpsc::SyncSender<Vec<u8>>,
}
impl MicPump {
/// Start the host-lifetime pump (Linux/Windows). On platforms without a virtual-mic backend
/// the thread just drains and drops frames (sessions still count the datagrams).
pub fn start() -> MicPump {
let (tx, rx) = std::sync::mpsc::sync_channel::<Vec<u8>>(MIC_QUEUE_CAP);
let spawned = std::thread::Builder::new()
.name("punktfunk-mic-pump".into())
.spawn(move || {
#[cfg(any(target_os = "linux", target_os = "windows"))]
pump_thread(rx, || open_virtual_mic(MIC_CHANNELS), PUMP_TUNING);
#[cfg(not(any(target_os = "linux", target_os = "windows")))]
{
tracing::warn!("mic passthrough unsupported on this platform — frames dropped");
for _ in rx {}
}
});
if let Err(e) = spawned {
tracing::error!(error = %e, "mic pump thread spawn failed — mic passthrough disabled");
}
MicPump { tx }
}
/// A sender a session forwards the client's Opus mic frames to (`try_send` — never block a
/// datagram loop). Cloned per session; dropping a clone does NOT stop the pump (it holds
/// the original sender for the host life).
pub fn sender(&self) -> std::sync::mpsc::SyncSender<Vec<u8>> {
self.tx.clone()
}
}
/// Sleep for `dur` while draining (and dropping) queued frames, so a closed/reopening backend
/// never accumulates a stale backlog and senders never see a wedged queue. Returns `false` when
/// every sender is gone (host shutdown).
#[cfg_attr(not(any(target_os = "linux", target_os = "windows")), allow(dead_code))]
fn drain_sleep(rx: &std::sync::mpsc::Receiver<Vec<u8>>, dur: std::time::Duration) -> bool {
use std::sync::mpsc::RecvTimeoutError;
let deadline = std::time::Instant::now() + dur;
loop {
let left = deadline.saturating_duration_since(std::time::Instant::now());
if left.is_zero() {
return true;
}
match rx.recv_timeout(left.min(std::time::Duration::from_millis(250))) {
Ok(_) => {} // drop frames while closed
Err(RecvTimeoutError::Timeout) => {} // keep waiting
Err(RecvTimeoutError::Disconnected) => return false, // host shutdown
}
}
}
/// The pump loop. `opener` is injected so the tests can run the REAL loop against a mock
/// backend; production passes [`open_virtual_mic`].
#[cfg_attr(not(any(target_os = "linux", target_os = "windows")), allow(dead_code))]
fn pump_thread<O>(rx: std::sync::mpsc::Receiver<Vec<u8>>, opener: O, tuning: PumpTuning)
where
O: Fn() -> Result<Box<dyn VirtualMic>>,
{
use std::sync::mpsc::RecvTimeoutError;
use std::time::Instant;
let mut backoff = tuning.backoff_start;
let mut open_fails: u64 = 0;
loop {
// Open phase — eager, from thread start.
let (mic, mut decoder) = loop {
let opened = opener().and_then(|m| {
let d = opus::Decoder::new(SAMPLE_RATE, opus::Channels::Stereo)
.map_err(|e| anyhow::anyhow!("opus decoder: {e}"))?;
Ok((m, d))
});
match opened {
Ok(pair) => break pair,
Err(e) => {
// Throttle (1st, 2nd, 4th, 8th … failure): a box without a PipeWire session
// or virtual audio device would otherwise log every backoff forever.
open_fails += 1;
if open_fails.is_power_of_two() {
tracing::warn!(error = %format!("{e:#}"), attempts = open_fails,
"virtual mic unavailable — retrying with backoff");
}
if !drain_sleep(&rx, backoff) {
return;
}
backoff = (backoff * 2).min(tuning.backoff_cap);
}
}
};
tracing::info!("virtual mic ready (host-lifetime)");
// Drop anything queued while (re)opening — it predates the backend. (The backoff does
// NOT reset here: only an instance that proves stable resets it — see the death triage.)
while rx.try_recv().is_ok() {}
let opened_at = Instant::now();
// Pump phase — runs until the backend dies (break) or the host shuts down (return).
let mut decode_fails: u64 = 0;
let mut pcm = vec![0f32; 5760 * MIC_CHANNELS as usize]; // up to 120 ms scratch
let mut last_push = Instant::now();
loop {
match rx.recv_timeout(tuning.heartbeat) {
Ok(frame) => {
if frame.is_empty() {
continue; // DTX silence — the source underruns to silence on its own
}
if last_push.elapsed() > tuning.stale_gap {
mic.discard();
}
match decoder.decode_float(&frame, &mut pcm, false) {
Ok(samples_per_ch) => {
let total = (samples_per_ch * MIC_CHANNELS as usize).min(pcm.len());
if !mic.push(&pcm[..total]) {
tracing::warn!("virtual mic backend died — reopening");
break;
}
last_push = Instant::now();
decode_fails = 0;
}
Err(e) => {
// Malformed/garbage frame: drop it, keep the shared mic + decoder
// (see the struct docs). Throttled log (1, 2, 4, … fails).
decode_fails += 1;
if decode_fails.is_power_of_two() {
tracing::warn!(error = %e, fails = decode_fails,
"mic opus decode failed — dropping frame");
}
}
}
}
Err(RecvTimeoutError::Timeout) => {
if !mic.alive() {
tracing::warn!("virtual mic backend died while idle — reopening");
break;
}
}
Err(RecvTimeoutError::Disconnected) => {
tracing::debug!("mic pump stopped (host shutting down)");
return;
}
}
}
// Death triage: an instance that lived is a one-off (PipeWire/audio-engine restart) —
// reopen immediately with the backoff reset. One that died right after opening is a
// failed open in disguise (flapping daemon, endpoint racing away): back off like the
// open loop, or the pump would churn open→die→reopen at heartbeat rate.
if opened_at.elapsed() >= tuning.stable_after {
backoff = tuning.backoff_start;
open_fails = 0;
} else {
open_fails += 1;
if !drain_sleep(&rx, backoff) {
return;
}
backoff = (backoff * 2).min(tuning.backoff_cap);
}
}
}
#[cfg(target_os = "windows")]
#[path = "audio/windows/audio_control.rs"]
mod audio_control;
@@ -98,3 +331,215 @@ mod wasapi_cap;
#[cfg(target_os = "windows")]
#[path = "audio/windows/wasapi_mic.rs"]
mod wasapi_mic;
#[cfg_attr(not(target_os = "windows"), allow(dead_code))]
#[path = "audio/wiring_plan.rs"]
pub(crate) mod wiring_plan;
#[cfg(test)]
mod pump_tests {
use super::*;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
/// Mock backend: records pushes/discards, dies on command.
struct MockMic {
alive: Arc<AtomicBool>,
pushed: Arc<AtomicUsize>,
discards: Arc<AtomicUsize>,
}
impl VirtualMic for MockMic {
fn push(&self, pcm: &[f32]) -> bool {
if !self.alive.load(Ordering::Acquire) {
return false;
}
self.pushed.fetch_add(pcm.len(), Ordering::Relaxed);
true
}
fn alive(&self) -> bool {
self.alive.load(Ordering::Acquire)
}
fn discard(&self) {
self.discards.fetch_add(1, Ordering::Relaxed);
}
}
struct Harness {
tx: std::sync::mpsc::SyncSender<Vec<u8>>,
opens: Arc<AtomicUsize>,
alive: Arc<Mutex<Option<Arc<AtomicBool>>>>, // latest instance's kill switch
pushed: Arc<AtomicUsize>,
discards: Arc<AtomicUsize>,
join: std::thread::JoinHandle<()>,
}
/// Run the REAL pump loop against mock backends; `fail_first` opens fail before the first
/// success (exercises the eager retry/backoff path). `dead_on_arrival` opens every instance
/// pre-killed (exercises the rapid-death churn guard). `stable_after` mirrors the tuning
/// field (ZERO = every death counts as stable → immediate reopen, keeping tests fast).
fn start_tuned(fail_first: usize, dead_on_arrival: bool, stable_after: Duration) -> Harness {
let (tx, rx) = std::sync::mpsc::sync_channel::<Vec<u8>>(MIC_QUEUE_CAP);
let opens = Arc::new(AtomicUsize::new(0));
let alive = Arc::new(Mutex::new(None::<Arc<AtomicBool>>));
let pushed = Arc::new(AtomicUsize::new(0));
let discards = Arc::new(AtomicUsize::new(0));
let (opens2, alive2, pushed2, discards2) = (
opens.clone(),
alive.clone(),
pushed.clone(),
discards.clone(),
);
let tuning = PumpTuning {
backoff_start: Duration::from_millis(10),
backoff_cap: Duration::from_millis(40),
heartbeat: Duration::from_millis(20),
stale_gap: Duration::from_millis(80),
stable_after,
};
let join = std::thread::spawn(move || {
pump_thread(
rx,
move || {
let n = opens2.fetch_add(1, Ordering::SeqCst);
if n < fail_first {
anyhow::bail!("backend not up yet (simulated)");
}
let a = Arc::new(AtomicBool::new(!dead_on_arrival));
*alive2.lock().unwrap() = Some(a.clone());
Ok(Box::new(MockMic {
alive: a,
pushed: pushed2.clone(),
discards: discards2.clone(),
}) as Box<dyn VirtualMic>)
},
tuning,
)
});
Harness {
tx,
opens,
alive,
pushed,
discards,
join,
}
}
fn start(fail_first: usize) -> Harness {
start_tuned(fail_first, false, Duration::ZERO)
}
fn wait_until(what: &str, mut cond: impl FnMut() -> bool) {
for _ in 0..200 {
if cond() {
return;
}
std::thread::sleep(Duration::from_millis(10));
}
panic!("timed out waiting for: {what}");
}
fn opus_frame() -> Vec<u8> {
let mut enc = opus::Encoder::new(48_000, opus::Channels::Stereo, opus::Application::Voip)
.expect("opus encoder");
let pcm = [0.1f32; 960 * 2]; // 20 ms stereo
let mut out = vec![0u8; 4000];
let n = enc.encode_float(&pcm, &mut out).expect("encode");
out.truncate(n);
out
}
/// Eager: the backend opens (after transient failures) with NO frame ever sent.
#[test]
fn opens_eagerly_with_backoff() {
let h = start(3);
wait_until("eager open after 3 failures", || {
h.opens.load(Ordering::SeqCst) >= 4 && h.alive.lock().unwrap().is_some()
});
drop(h.tx);
h.join.join().unwrap();
}
/// Frames flow: opus in → PCM pushed to the backend.
#[test]
fn decodes_and_pushes() {
let h = start(0);
wait_until("open", || h.alive.lock().unwrap().is_some());
h.tx.send(opus_frame()).unwrap();
wait_until("pcm pushed", || h.pushed.load(Ordering::SeqCst) > 0);
drop(h.tx);
h.join.join().unwrap();
}
/// A dead backend is noticed WHILE IDLE (heartbeat) and reopened without any traffic.
#[test]
fn reopens_after_idle_death() {
let h = start(0);
wait_until("first open", || h.opens.load(Ordering::SeqCst) >= 1);
wait_until("instance", || h.alive.lock().unwrap().is_some());
h.alive
.lock()
.unwrap()
.as_ref()
.unwrap()
.store(false, Ordering::Release); // kill it
wait_until("reopen after idle death", || {
h.opens.load(Ordering::SeqCst) >= 2
});
drop(h.tx);
h.join.join().unwrap();
}
/// A death detected on push (frame flowing) also reopens, and the frame after reopen flows.
#[test]
fn reopens_after_push_death() {
let h = start(0);
wait_until("instance", || h.alive.lock().unwrap().is_some());
h.alive
.lock()
.unwrap()
.as_ref()
.unwrap()
.store(false, Ordering::Release);
h.tx.send(opus_frame()).unwrap(); // push sees death → reopen
wait_until("reopen", || h.opens.load(Ordering::SeqCst) >= 2);
h.tx.send(opus_frame()).unwrap();
wait_until("pcm after reopen", || h.pushed.load(Ordering::SeqCst) > 0);
drop(h.tx);
h.join.join().unwrap();
}
/// Instances that die immediately after opening must be retried with BACKOFF, not at
/// heartbeat rate — a flapping backend (daemon up but dropping us instantly) would
/// otherwise churn open→die→reopen every heartbeat forever.
#[test]
fn rapid_death_backs_off() {
// Every instance is dead on arrival; stability threshold high so each death counts
// as a failed open. Without the guard: ~1 reopen per heartbeat (20 ms) ≈ 25 opens in
// 500 ms. With backoff 10→20→40 (cap): ≈ 7.
let h = start_tuned(0, true, Duration::from_secs(10));
std::thread::sleep(Duration::from_millis(500));
let opens = h.opens.load(Ordering::SeqCst);
assert!(opens >= 2, "must keep retrying (got {opens})");
assert!(
opens <= 15,
"must back off, not churn per heartbeat (got {opens})"
);
drop(h.tx);
h.join.join().unwrap();
}
/// An uplink gap discards buffered-stale audio before the next frame plays.
#[test]
fn discards_after_gap() {
let h = start(0);
wait_until("instance", || h.alive.lock().unwrap().is_some());
h.tx.send(opus_frame()).unwrap();
wait_until("first push", || h.pushed.load(Ordering::SeqCst) > 0);
std::thread::sleep(Duration::from_millis(150)); // > stale_gap
h.tx.send(opus_frame()).unwrap();
wait_until("discard on gap", || h.discards.load(Ordering::SeqCst) >= 1);
drop(h.tx);
h.join.join().unwrap();
}
}
+301 -179
View File
@@ -16,7 +16,9 @@
use super::{AudioCapturer, VirtualMic, SAMPLE_RATE};
use anyhow::{anyhow, Context, Result};
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError};
use std::sync::Arc;
use std::thread;
use std::time::Duration;
@@ -111,10 +113,28 @@ fn spa_positions(channels: u32) -> [u32; 64] {
/// Virtual microphone: a PipeWire `Audio/Source` node host apps can record from. The host pushes
/// decoded client-mic PCM in; the loop thread's producer callback drains it (silence on
/// underrun) into PipeWire buffers. Mirrors [`PwAudioCapturer`] but inverted (Direction::Output).
///
/// **Why a stream node and not a `support.null-audio-sink` adapter** (the canonical
/// virtual-mic recipe): tested live on this project's headless graph (PipeWire 1.6.2,
/// 2026-07-03), an adapter with `media.class=Audio/Source/Virtual` never gets a clock — the
/// {source, recorder} group runs with QUANT/RATE 0 and delivers pure silence — and WirePlumber
/// rerouted a feeder stream targeting it to the *default sink* instead (which would play the
/// client's voice out of the speakers, straight into the desktop-audio capture: echo). The
/// stream node below, with `RT_PROCESS` + `priority.session` (see the property comments), is
/// validated working on PipeWire 1.4 (Bazzite) and 1.6 (this box) in both attach orderings.
/// Do not "modernize" this to the adapter recipe without re-running that validation.
///
/// **Liveness contract** (see [`VirtualMic`]): the loop thread exits on a core error (PipeWire
/// daemon restart — the node is gone) or a stream error, which flips `alive` — `push` then
/// returns `false` and the owning pump reopens against the new daemon, recreating the node.
pub struct PwMicSource {
pcm: std::sync::mpsc::SyncSender<Vec<f32>>,
pcm: std::sync::mpsc::SyncSender<(std::time::Instant, Vec<f32>)>,
channels: u32,
quit: pipewire::channel::Sender<Terminate>,
/// False once the loop thread has exited (daemon/stream death or teardown).
alive: Arc<AtomicBool>,
/// One-shot flush request, consumed by the process callback (clears the jitter ring).
flush: Arc<AtomicBool>,
}
impl PwMicSource {
@@ -123,21 +143,36 @@ impl PwMicSource {
matches!(channels, 1 | 2),
"virtual mic supports 1 or 2 channels, got {channels}"
);
let (pcm_tx, pcm_rx) = sync_channel::<Vec<f32>>(64);
let (pcm_tx, pcm_rx) = sync_channel::<(std::time::Instant, Vec<f32>)>(64);
let (quit_tx, quit_rx) = pipewire::channel::channel::<Terminate>();
let alive = Arc::new(AtomicBool::new(true));
let flush = Arc::new(AtomicBool::new(false));
// Bring-up handshake (mirrors the Windows backend): a PipeWire that isn't running (host
// service started before the user session) must surface as an open ERROR — engaging the
// pump's backoff — not as an instantly-dead instance the pump would churn on.
let (ready_tx, ready_rx) = sync_channel::<Result<()>>(1);
let (alive_t, flush_t) = (alive.clone(), flush.clone());
thread::Builder::new()
.name("punktfunk-pw-mic".into())
.spawn(move || {
if let Err(e) = mic_pw_thread(pcm_rx, quit_rx, channels) {
if let Err(e) = mic_pw_thread(pcm_rx, quit_rx, channels, flush_t, ready_tx) {
tracing::error!(error = %format!("{e:#}"), "pipewire virtual-mic thread failed");
}
// Whether a clean quit or a daemon death: this instance is done — the pump reopens.
alive_t.store(false, Ordering::Release);
})
.context("spawn pipewire virtual-mic thread")?;
Ok(PwMicSource {
pcm: pcm_tx,
channels,
quit: quit_tx,
})
match ready_rx.recv_timeout(Duration::from_secs(5)) {
Ok(Ok(())) => Ok(PwMicSource {
pcm: pcm_tx,
channels,
quit: quit_tx,
alive,
flush,
}),
Ok(Err(e)) => Err(e),
Err(_) => Err(anyhow!("pipewire virtual-mic init timed out")),
}
}
}
@@ -148,8 +183,24 @@ impl Drop for PwMicSource {
}
impl VirtualMic for PwMicSource {
fn push(&self, pcm: &[f32]) {
let _ = self.pcm.try_send(pcm.to_vec()); // drop if the PipeWire side is behind
fn push(&self, pcm: &[f32]) -> bool {
if !self.alive.load(Ordering::Acquire) {
return false;
}
// Timestamped so the process callback can age out chunks that sat in the channel while
// no recorder was attached (see the staleness logic there).
match self.pcm.try_send((std::time::Instant::now(), pcm.to_vec())) {
Ok(()) => true,
// Behind is fine (drop the chunk); a gone receiver means the loop thread exited.
Err(std::sync::mpsc::TrySendError::Full(_)) => true,
Err(std::sync::mpsc::TrySendError::Disconnected(_)) => false,
}
}
fn alive(&self) -> bool {
self.alive.load(Ordering::Acquire)
}
fn discard(&self) {
self.flush.store(true, Ordering::Release);
}
fn channels(&self) -> u32 {
self.channels
@@ -160,202 +211,273 @@ impl VirtualMic for PwMicSource {
/// the process callback drains into PipeWire buffers (capped, so latency stays bounded).
/// `primed` is a jitter buffer gate — see the process callback.
struct MicUserData {
rx: Receiver<Vec<f32>>,
rx: Receiver<(std::time::Instant, Vec<f32>)>,
ring: VecDeque<f32>,
channels: usize,
primed: bool,
/// One-shot flush request from [`PwMicSource::discard`] (stale-audio drop after a gap).
flush: Arc<AtomicBool>,
/// When the process callback last ran — a long gap means the ring content predates the
/// current consumer (the stream idles with no recorder attached) and must be dropped.
last_run: Option<std::time::Instant>,
}
/// PCM older than this never reaches a recorder: chunks that aged in the channel while no
/// recorder was attached, and ring content from before a consumer gap, are dropped instead of
/// bursting out as stale audio when recording (re)starts.
const MIC_STALE: Duration = Duration::from_secs(1);
fn mic_pw_thread(
pcm_rx: Receiver<Vec<f32>>,
pcm_rx: Receiver<(std::time::Instant, Vec<f32>)>,
quit_rx: pipewire::channel::Receiver<Terminate>,
channels: u32,
flush: Arc<AtomicBool>,
ready: std::sync::mpsc::SyncSender<Result<()>>,
) -> Result<()> {
use pipewire as pw;
use pw::{properties::properties, spa};
use spa::param::audio::{AudioFormat, AudioInfoRaw};
use spa::pod::Pod;
crate::pwinit::ensure_init();
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw mic MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("pw mic Context")?;
let core = context
.connect_rc(None)
.context("pw mic connect (is PipeWire running in this session?)")?;
// The PipeWire objects are lifetime-chained (guards borrow the mainloop/core), so setup and
// the blocking run share one frame; the IIFE lets every setup `?` funnel through the ready
// handshake below (mirrors the Windows render_thread).
let result = (|| -> Result<()> {
crate::pwinit::ensure_init();
let mainloop = pw::main_loop::MainLoopRc::new(None).context("pw mic MainLoop")?;
let context = pw::context::ContextRc::new(&mainloop, None).context("pw mic Context")?;
let core = context
.connect_rc(None)
.context("pw mic connect (is PipeWire running in this session?)")?;
let _quit_guard = quit_rx.attach(mainloop.loop_(), {
let mainloop = mainloop.clone();
move |_| mainloop.quit()
});
let _quit_guard = quit_rx.attach(mainloop.loop_(), {
let mainloop = mainloop.clone();
move |_| mainloop.quit()
});
// media.class=Audio/Source advertises us as a microphone (a recordable source), NOT a
// playback stream — without it, Direction::Output + Playback would route to the speakers.
let stream = pw::stream::StreamBox::new(
&core,
"punktfunk-mic",
properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CLASS => "Audio/Source",
*pw::keys::NODE_NAME => "punktfunk-mic",
*pw::keys::NODE_DESCRIPTION => "Punktfunk Remote Microphone",
// ~5 ms quantum (one Opus frame) so recording apps get smooth low-latency chunks.
*pw::keys::NODE_LATENCY => "240/48000",
// Win WirePlumber's default-source election. This fixes TWO failures (both diagnosed
// live on a Bazzite host, PipeWire 1.4.10):
// 1. Apps that record the *default* input (games, Discord, arecord) get the client's
// mic — the Linux analogue of the Windows host forcing the default recording
// endpoint (audio/windows/audio_control.rs). Without it the source is never the
// default, so default-input recorders hear silence.
// 2. On PipeWire 1.4.x, a *non-default* Audio/Source recorded via `--target` never
// gets a driver assigned — the {source, recorder} group stays orphaned (pw-top:
// QUANT/RATE 0, `driver-node None`), so the RT `process()` callback never fires and
// even an explicitly-selected mic is pure silence. Making it the default source
// keeps WirePlumber driving it, so `process()` runs and audio flows. (PipeWire 1.6
// drives any recorded source regardless, which is why this only bit the 1.4 host.)
// Reproduced with a faithful standalone copy of this node: no priority.session → silent,
// priority.session set → audio, on the same 1.4.10 daemon. Only overrides WirePlumber's
// *auto* default (a user's explicit default.configured.audio.source still wins); the
// value clears typical real-hardware source priorities (~10001900).
"priority.session" => "3000",
},
)
.context("pw mic Stream")?;
let ud = MicUserData {
rx: pcm_rx,
ring: VecDeque::new(),
channels: channels as usize,
primed: false,
};
let _listener = stream
.add_local_listener_with_user_data(ud)
.state_changed(|_s, _ud, old, new| {
tracing::info!(?old, ?new, "pipewire virtual-mic stream state");
})
.param_changed(|_s, _ud, id, param| {
let Some(param) = param else { return };
if id != pw::spa::param::ParamType::Format.as_raw() {
return;
}
let mut info = AudioInfoRaw::default();
if info.parse(param).is_ok() {
tracing::info!(
format = ?info.format(),
rate = info.rate(),
channels = info.channels(),
"virtual-mic format negotiated"
);
}
})
.process(|stream, ud| {
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let Some(mut buffer) = stream.dequeue_buffer() else {
return;
};
// Pull all newly-decoded PCM into the ring.
while let Ok(frame) = ud.rx.try_recv() {
ud.ring.extend(frame);
// Death detection: a core error (the daemon restarted/went away — our remote node no longer
// exists) ends this thread, flipping the owner's `alive` flag so the pump reopens against the
// new daemon. Without this, a PipeWire restart left the loop idling on a dead connection and
// the mic silently broken for the rest of the host's life.
let _core_listener = core
.add_listener_local()
.error({
let mainloop = mainloop.clone();
move |id, _seq, res, message| {
tracing::warn!(
id,
res,
message,
"pipewire core error — virtual mic reopening"
);
mainloop.quit();
}
let stride = 4 * ud.channels; // F32LE interleaved
let datas = buffer.datas_mut();
if datas.is_empty() {
})
.register();
// media.class=Audio/Source advertises us as a microphone (a recordable source), NOT a
// playback stream — without it, Direction::Output + Playback would route to the speakers.
let stream = pw::stream::StreamBox::new(
&core,
"punktfunk-mic",
properties! {
*pw::keys::MEDIA_TYPE => "Audio",
*pw::keys::MEDIA_CLASS => "Audio/Source",
*pw::keys::NODE_NAME => "punktfunk-mic",
*pw::keys::NODE_DESCRIPTION => "Punktfunk Remote Microphone",
// ~5 ms quantum (one Opus frame) so recording apps get smooth low-latency chunks.
*pw::keys::NODE_LATENCY => "240/48000",
// Win WirePlumber's default-source election. This fixes TWO failures (both diagnosed
// live on a Bazzite host, PipeWire 1.4.10):
// 1. Apps that record the *default* input (games, Discord, arecord) get the client's
// mic — the Linux analogue of the Windows host forcing the default recording
// endpoint (audio/windows/audio_control.rs). Without it the source is never the
// default, so default-input recorders hear silence.
// 2. On PipeWire 1.4.x, a *non-default* Audio/Source recorded via `--target` never
// gets a driver assigned — the {source, recorder} group stays orphaned (pw-top:
// QUANT/RATE 0, `driver-node None`), so the RT `process()` callback never fires and
// even an explicitly-selected mic is pure silence. Making it the default source
// keeps WirePlumber driving it, so `process()` runs and audio flows. (PipeWire 1.6
// drives any recorded source regardless, which is why this only bit the 1.4 host.)
// Reproduced with a faithful standalone copy of this node: no priority.session → silent,
// priority.session set → audio, on the same 1.4.10 daemon. Only overrides WirePlumber's
// *auto* default (a user's explicit default.configured.audio.source still wins); the
// value clears typical real-hardware source priorities (~10001900).
"priority.session" => "3000",
},
)
.context("pw mic Stream")?;
let ud = MicUserData {
rx: pcm_rx,
ring: VecDeque::new(),
channels: channels as usize,
primed: false,
flush,
last_run: None,
};
let _listener = stream
.add_local_listener_with_user_data(ud)
.state_changed({
let mainloop = mainloop.clone();
move |_s, _ud, old, new| {
tracing::info!(?old, ?new, "pipewire virtual-mic stream state");
// A stream error is unrecoverable for this instance — exit so the pump reopens.
if matches!(new, pw::stream::StreamState::Error(_)) {
mainloop.quit();
}
}
})
.param_changed(|_s, _ud, id, param| {
let Some(param) = param else { return };
if id != pw::spa::param::ParamType::Format.as_raw() {
return;
}
let data = &mut datas[0];
let want_frames = data.data().map(|s| s.len() / stride).unwrap_or(0);
let want = want_frames * ud.channels; // interleaved samples this quantum needs
static FIRST: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(true);
if FIRST.swap(false, std::sync::atomic::Ordering::Relaxed) {
let mut info = AudioInfoRaw::default();
if info.parse(param).is_ok() {
tracing::info!(
quantum_frames = want_frames,
quantum_ms = want_frames as f32 / 48.0,
"virtual-mic consumer connected"
format = ?info.format(),
rate = info.rate(),
channels = info.channels(),
"virtual-mic format negotiated"
);
}
// Adaptive jitter buffer. The client pushes 5 ms frames; the recorder pulls a
// whole *quantum* (often 2043 ms) from an independent clock. A drain of one
// quantum must not outrun what's buffered, or every call underruns to silence
// (the original ~58% gaps). So prime to ~3 quanta before producing, hold there,
// and re-prime only after a genuine full drain (the client went quiet). The ring
// is capped at a few quanta so latency stays bounded.
let target = (3 * want).clamp(720 * ud.channels, 9600 * ud.channels);
while ud.ring.len() > target.max(want) + want {
ud.ring.pop_front(); // bound latency: drop the oldest beyond ~1 quantum slack
}
if !ud.primed && ud.ring.len() >= target {
ud.primed = true;
}
let n_frames = if let Some(slice) = data.data() {
for k in 0..want {
let s = if ud.primed {
ud.ring.pop_front().unwrap_or(0.0) // silence on a momentary underrun
} else {
0.0 // not yet primed — emit silence while the buffer fills
};
let off = k * 4;
slice[off..off + 4].copy_from_slice(&s.to_le_bytes());
})
.process(|stream, ud| {
let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let Some(mut buffer) = stream.dequeue_buffer() else {
return;
};
// Stale-audio guard, BEFORE pulling new frames: drop the ring when a flush was
// requested (uplink gap — see the pump) or when this callback itself hasn't run
// for a while (the stream idled with no recorder attached; whatever the ring
// holds predates the consumer). A recorder must never hear a burst of old audio.
let now = std::time::Instant::now();
let idled = ud
.last_run
.is_some_and(|t| now.duration_since(t) > MIC_STALE);
if ud.flush.swap(false, std::sync::atomic::Ordering::AcqRel) || idled {
ud.ring.clear();
ud.primed = false;
}
want_frames
} else {
0
};
if ud.ring.is_empty() {
ud.primed = false; // fully drained — re-prime before producing again
ud.last_run = Some(now);
// Pull all newly-decoded PCM into the ring, aging out chunks that sat in the
// channel while nothing consumed them (same staleness rule).
while let Ok((t, frame)) = ud.rx.try_recv() {
if now.duration_since(t) <= MIC_STALE {
ud.ring.extend(frame);
}
}
let stride = 4 * ud.channels; // F32LE interleaved
let datas = buffer.datas_mut();
if datas.is_empty() {
return;
}
let data = &mut datas[0];
let want_frames = data.data().map(|s| s.len() / stride).unwrap_or(0);
let want = want_frames * ud.channels; // interleaved samples this quantum needs
static FIRST: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(true);
if FIRST.swap(false, std::sync::atomic::Ordering::Relaxed) {
tracing::info!(
quantum_frames = want_frames,
quantum_ms = want_frames as f32 / 48.0,
"virtual-mic consumer connected"
);
}
// Adaptive jitter buffer. The client pushes 5 ms frames; the recorder pulls a
// whole *quantum* (often 2043 ms) from an independent clock. A drain of one
// quantum must not outrun what's buffered, or every call underruns to silence
// (the original ~58% gaps). So prime to ~3 quanta before producing, hold there,
// and re-prime only after a genuine full drain (the client went quiet). The ring
// is capped at a few quanta so latency stays bounded.
let target = (3 * want).clamp(720 * ud.channels, 9600 * ud.channels);
while ud.ring.len() > target.max(want) + want {
ud.ring.pop_front(); // bound latency: drop the oldest beyond ~1 quantum slack
}
if !ud.primed && ud.ring.len() >= target {
ud.primed = true;
}
let n_frames = if let Some(slice) = data.data() {
for k in 0..want {
let s = if ud.primed {
ud.ring.pop_front().unwrap_or(0.0) // silence on a momentary underrun
} else {
0.0 // not yet primed — emit silence while the buffer fills
};
let off = k * 4;
slice[off..off + 4].copy_from_slice(&s.to_le_bytes());
}
want_frames
} else {
0
};
if ud.ring.is_empty() {
ud.primed = false; // fully drained — re-prime before producing again
}
let chunk = data.chunk_mut();
*chunk.offset_mut() = 0;
*chunk.stride_mut() = stride as _;
*chunk.size_mut() = (stride * n_frames) as _;
}));
if outcome.is_err() {
tracing::error!("panic in pipewire virtual-mic callback");
}
let chunk = data.chunk_mut();
*chunk.offset_mut() = 0;
*chunk.stride_mut() = stride as _;
*chunk.size_mut() = (stride * n_frames) as _;
}));
if outcome.is_err() {
tracing::error!("panic in pipewire virtual-mic callback");
}
})
.register()
.context("register virtual-mic stream listener")?;
})
.register()
.context("register virtual-mic stream listener")?;
let mut info = AudioInfoRaw::new();
info.set_format(AudioFormat::F32LE);
info.set_rate(SAMPLE_RATE);
info.set_channels(channels);
info.set_position(spa_positions(channels));
let obj = pw::spa::pod::Object {
type_: pw::spa::utils::SpaTypes::ObjectParamFormat.as_raw(),
id: pw::spa::param::ParamType::EnumFormat.as_raw(),
properties: info.into(),
};
let values: Vec<u8> = pw::spa::pod::serialize::PodSerializer::serialize(
std::io::Cursor::new(Vec::new()),
&pw::spa::pod::Value::Object(obj),
)
.context("serialize mic format pod")?
.0
.into_inner();
let mut params = [Pod::from_bytes(&values).context("mic pod from bytes")?];
// RT_PROCESS: run the producer callback on PipeWire's realtime data loop, so the source is a
// *synchronous* graph node that joins its consumer's driver group and is actually driven. Without
// it the node is async/main-loop and, in the host's busy multi-stream graph (desktop-audio +
// video capture + the session), never acquires a driver — it stays suspended and its process()
// never fires, so every recorder hears pure silence (the long-standing "Linux host mic broken").
stream
.connect(
spa::utils::Direction::Output, // we PRODUCE samples (a source)
None,
pw::stream::StreamFlags::AUTOCONNECT
| pw::stream::StreamFlags::MAP_BUFFERS
| pw::stream::StreamFlags::RT_PROCESS,
&mut params,
let mut info = AudioInfoRaw::new();
info.set_format(AudioFormat::F32LE);
info.set_rate(SAMPLE_RATE);
info.set_channels(channels);
info.set_position(spa_positions(channels));
let obj = pw::spa::pod::Object {
type_: pw::spa::utils::SpaTypes::ObjectParamFormat.as_raw(),
id: pw::spa::param::ParamType::EnumFormat.as_raw(),
properties: info.into(),
};
let values: Vec<u8> = pw::spa::pod::serialize::PodSerializer::serialize(
std::io::Cursor::new(Vec::new()),
&pw::spa::pod::Value::Object(obj),
)
.context("pw mic stream connect")?;
.context("serialize mic format pod")?
.0
.into_inner();
let mut params = [Pod::from_bytes(&values).context("mic pod from bytes")?];
mainloop.run();
tracing::debug!("pipewire virtual-mic loop exited (source dropped)");
Ok(())
// RT_PROCESS: run the producer callback on PipeWire's realtime data loop, so the source is a
// *synchronous* graph node that joins its consumer's driver group and is actually driven. Without
// it the node is async/main-loop and, in the host's busy multi-stream graph (desktop-audio +
// video capture + the session), never acquires a driver — it stays suspended and its process()
// never fires, so every recorder hears pure silence (the long-standing "Linux host mic broken").
stream
.connect(
spa::utils::Direction::Output, // we PRODUCE samples (a source)
None,
pw::stream::StreamFlags::AUTOCONNECT
| pw::stream::StreamFlags::MAP_BUFFERS
| pw::stream::StreamFlags::RT_PROCESS,
&mut params,
)
.context("pw mic stream connect")?;
// Setup complete: the daemon connection and stream connect succeeded — report ready,
// then block until quit/death. (A PipeWire that isn't running never reaches this line;
// its connect error surfaces through the handshake as an OPEN failure, so the pump
// backs off instead of churning on instantly-dead instances.)
let _ = ready.send(Ok(()));
mainloop.run();
tracing::debug!("pipewire virtual-mic loop exited (source dropped)");
Ok(())
})();
if let Err(e) = &result {
let _ = ready.send(Err(anyhow!("{e:#}")));
}
result
}
fn pw_thread(
@@ -6,64 +6,39 @@
//! ones, or the loopback re-captures the injected mic (an infinite echo). The installer bundles
//! VB-Audio Virtual Cable (the mic target: its "CABLE Input" render endpoint → "CABLE Output" capture)
//! and the host auto-installs the Steam Streaming pair (a loopback-capable render). This module wires
//! them up at startup so no manual Sound-settings fiddling is ever needed:
//! them up so no manual Sound-settings fiddling is ever needed:
//!
//! * default **PLAYBACK** → a loopback-capable render that is NOT the mic cable (a real output device
//! * the **mic inject target** is assigned FIRST (VB-Cable "CABLE Input" preferred) — mic passthrough
//! is what the cable is bundled for, so it wins the cable even when the cable is the only render
//! endpoint on the box (the loopback then reports itself unavailable instead of echoing);
//! * default **PLAYBACK** → a loopback-capable render that is NOT the mic target (a real output device
//! if one exists, else the Steam Streaming Microphone; **never** the Steam Streaming Speakers, whose
//! loopback is silent — validated live). This is the endpoint [`super::wasapi_cap`] loopback-captures
//! for desktop audio.
//! * default **RECORDING** → the virtual mic's capture endpoint (VB-Cable "CABLE Output") so host apps
//! loopback is silent — validated live). This is the endpoint [`super::wasapi_cap`] captures;
//! * default **RECORDING** → the mic target's capture endpoint (VB-Cable "CABLE Output") so host apps
//! record the client's mic by default.
//!
//! [`super::wasapi_mic::find_device`] then resolves the mic INJECT target to "CABLE Input" — a render
//! candidate that is NOT the default playback — guaranteeing loopback ≠ mic, so there is no echo.
//! The assignment rules are the PURE [`wiring_plan`](super::wiring_plan) module (unit-tested on every
//! platform); this module only enumerates endpoints, applies the plan, and logs. [`wire_now`] runs on
//! every mic/capture (re)open — NOT once per process — because endpoints churn (boot-time
//! registration, hotplug, driver installs) and a stale plan was one of the ways mic passthrough died
//! permanently.
//!
//! Setting a default endpoint uses the undocumented `IPolicyConfig` COM interface (the only way to set
//! a default device programmatically — neither the `windows` nor `wasapi` crate exposes it; it is the
//! same call `mmsys.cpl` makes). Opt out with `PUNKTFUNK_KEEP_DEFAULT` to leave the user's chosen
//! defaults untouched.
//! defaults untouched (the plan is still computed — the mic must still pick a target).
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it.
#![deny(clippy::undocumented_unsafe_blocks)]
use super::wiring_plan::{plan, Endpoint, Wiring};
use anyhow::{anyhow, bail, Result};
use std::ffi::c_void;
use std::sync::Once;
use std::sync::Mutex;
use wasapi::Direction;
/// Run the audio device auto-wiring exactly once per process, before the first capturer/mic opens.
/// Blocks until done so the default playback is set before the loopback captures it. Best-effort:
/// every failure is logged, never fatal (the host then falls back to whatever the current defaults
/// are — exactly the pre-wiring behaviour).
pub(crate) fn ensure_wired_once() {
static WIRED: Once = Once::new();
WIRED.call_once(|| {
if std::env::var_os("PUNKTFUNK_KEEP_DEFAULT").is_some() {
tracing::info!("PUNKTFUNK_KEEP_DEFAULT set — leaving the audio default devices untouched");
return;
}
// Run on a dedicated COM-MTA thread so we never collide with the caller's apartment mode
// (the capture/mic threads each initialize their own COM separately).
let handle = std::thread::Builder::new()
.name("pf-audio-wiring".into())
.spawn(|| {
if wasapi::initialize_mta().ok().is_err() {
tracing::warn!("audio wiring: COM init (MTA) failed — skipping");
return;
}
if let Err(e) = ensure_audio_wiring() {
tracing::warn!(error = %format!("{e:#}"),
"audio auto-wiring failed — mic/desktop audio may need manual device defaults");
}
});
if let Ok(h) = handle {
let _ = h.join();
}
});
}
/// `(friendly_name, endpoint_id)` for every ACTIVE endpoint in direction `dir`.
fn list_endpoints(dir: Direction) -> Vec<(String, String)> {
fn list_endpoints(dir: Direction) -> Vec<Endpoint> {
let mut out = Vec::new();
let Ok(en) = wasapi::DeviceEnumerator::new() else {
return out;
@@ -86,79 +61,85 @@ fn list_endpoints(dir: Direction) -> Vec<(String, String)> {
out
}
/// Pick the loopback + mic-capture devices and set them as the default playback/recording.
fn ensure_audio_wiring() -> Result<()> {
/// Enumerate endpoints, compute the assignment, apply the default-device changes (unless
/// `PUNKTFUNK_KEEP_DEFAULT`), and return the plan for the caller to act on (mic target / loopback
/// echo guard). Must run on a COM-initialized thread (the WASAPI worker threads all
/// `initialize_mta` first). Logged only when the assignment changes, so per-open recomputation
/// stays quiet in the steady state.
pub(crate) fn wire_now() -> Wiring {
let renders = list_endpoints(Direction::Render);
let captures = list_endpoints(Direction::Capture);
if renders.is_empty() {
bail!("no active render endpoints to wire");
}
let want = std::env::var("PUNKTFUNK_MIC_DEVICE")
.ok()
.map(|s| s.to_lowercase());
let wiring = plan(&renders, &captures, want.as_deref());
// A render is unusable as the desktop-audio loopback if it is a VB-Cable endpoint (reserved for
// the mic inject) or the Steam Streaming Speakers (its loopback is silent — validated live).
let excluded_loopback =
|ln: &str| ln.contains("cable") || ln.contains("steam streaming speakers");
// "virtual-ish" = a known virtual cable; a render WITHOUT these markers is a real output device,
// the best loopback source (apps render there and the operator can also hear it).
let virtualish = |ln: &str| {
ln.contains("virtual")
|| ln.contains("cable")
|| ln.contains("steam streaming")
|| ln.contains("voicemeeter")
// Log assignment changes exactly once (first plan included).
static LAST: Mutex<Option<Wiring>> = Mutex::new(None);
let changed = {
let mut last = LAST.lock().unwrap();
let changed = last.as_ref() != Some(&wiring);
*last = Some(wiring.clone());
changed
};
let loopback = renders
.iter()
.find(|(n, _)| {
let ln = n.to_lowercase();
!excluded_loopback(&ln) && !virtualish(&ln)
})
.or_else(|| {
renders
.iter()
.find(|(n, _)| n.to_lowercase().contains("steam streaming microphone"))
})
.or_else(|| {
renders
.iter()
.find(|(n, _)| !excluded_loopback(&n.to_lowercase()))
});
// The virtual mic's CAPTURE endpoint host apps record from — VB-Cable "CABLE Output" preferred.
let mic_capture = captures
.iter()
.find(|(n, _)| n.to_lowercase().contains("cable output"))
.or_else(|| {
captures
.iter()
.find(|(n, _)| n.to_lowercase().contains("steam streaming microphone"))
})
.or_else(|| {
captures.iter().find(|(n, _)| {
let ln = n.to_lowercase();
ln.contains("voicemeeter") || ln.contains("virtual")
})
});
match loopback {
Some((name, id)) => match set_default_endpoint(id) {
Ok(()) => tracing::info!(device = %name,
"audio wiring: default playback = desktop-audio loopback source"),
Err(e) => tracing::warn!(device = %name, error = %format!("{e:#}"),
"audio wiring: failed to set the default playback device"),
},
None => {
tracing::warn!("audio wiring: no usable desktop-audio loopback render endpoint found")
if changed {
tracing::info!(
mic_render = wiring.mic_render.as_ref().map(|(n, _)| n.as_str()),
mic_capture = wiring.mic_capture.as_ref().map(|(n, _)| n.as_str()),
loopback_render = wiring.loopback_render.as_ref().map(|(n, _)| n.as_str()),
renders = ?renders.iter().map(|(n, _)| n.as_str()).collect::<Vec<_>>(),
"audio wiring plan"
);
if wiring.mic_render.is_some() && wiring.loopback_render.is_none() {
tracing::warn!(
"the virtual mic reserved the only usable render endpoint — desktop audio will be \
unavailable until another output device exists (attach one, or let the host \
install the Steam Streaming pair)"
);
}
}
if let Some((name, id)) = mic_capture {
if std::env::var_os("PUNKTFUNK_KEEP_DEFAULT").is_some() {
if changed {
tracing::info!(
"PUNKTFUNK_KEEP_DEFAULT set — leaving the audio default devices untouched"
);
}
return wiring;
}
if let Some((name, id)) = &wiring.loopback_render {
match set_default_endpoint(id) {
Ok(()) => tracing::info!(device = %name,
"audio wiring: default recording = virtual mic (apps record the client's mic)"),
Ok(()) => {
if changed {
tracing::info!(device = %name,
"audio wiring: default playback = desktop-audio loopback source");
}
}
Err(e) => tracing::warn!(device = %name, error = %format!("{e:#}"),
"audio wiring: failed to set the default playback device"),
}
}
if let Some((name, id)) = &wiring.mic_capture {
match set_default_endpoint(id) {
Ok(()) => {
if changed {
tracing::info!(device = %name,
"audio wiring: default recording = virtual mic (apps record the client's mic)");
}
}
Err(e) => tracing::warn!(device = %name, error = %format!("{e:#}"),
"audio wiring: failed to set the default recording device"),
}
}
Ok(())
wiring
}
/// Open a device by endpoint id, with a name for error context.
pub(crate) fn open_endpoint(ep: &Endpoint) -> Result<wasapi::Device> {
wasapi::DeviceEnumerator::new()
.map_err(|e| anyhow!("DeviceEnumerator: {e}"))?
.get_device(&ep.1)
.map_err(|e| anyhow!("open endpoint {:?}: {e}", ep.0))
}
// --- IPolicyConfig (undocumented): set a default audio endpoint by id, for all three roles. ---
@@ -6,7 +6,7 @@
//! COM-apartment-bound and not `Send`, so they live on a dedicated thread (mirrors
//! `linux::PwAudioCapturer`); only the channel + stop flag + join handle are in the struct.
use super::{AudioCapturer, SAMPLE_RATE};
use super::{audio_control, AudioCapturer, SAMPLE_RATE};
use anyhow::{anyhow, Context, Result};
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -109,14 +109,36 @@ fn capture_thread(
}
let res = (|| -> Result<()> {
// Loopback = capture the RENDER endpoint: get the default render device, but open a CAPTURE
// client with loopback=true over it. NOTE: the virtual mic (`super::wasapi_mic`) is guarded
// to NEVER target this same endpoint — otherwise the client's injected mic would be captured
// here and streamed back to the client (infinite echo). Keep that guard in sync if this
// device selection ever changes.
let device = DeviceEnumerator::new()
// client with loopback=true over it. ECHO GUARD: the wiring plan reserves one endpoint for
// the virtual mic (`super::wasapi_mic` writes the client's voice there) — capturing THAT
// endpoint would stream the client's own mic straight back to it. Normally the plan has
// already moved the default playback elsewhere; if the default still IS the mic target
// (PUNKTFUNK_KEEP_DEFAULT, or the cable is the only endpoint), capture the plan's loopback
// endpoint explicitly, or refuse — no desktop audio beats an echo loop.
let wiring = audio_control::wire_now();
let default = DeviceEnumerator::new()
.context("DeviceEnumerator")?
.get_default_device(&Direction::Render)
.context("default render endpoint (loopback needs a render device)")?;
let default_is_mic = match (&wiring.mic_render, default.get_id()) {
(Some((_, mic_id)), Ok(id)) => *mic_id == id,
_ => false,
};
let device = if default_is_mic {
let Some(lb) = &wiring.loopback_render else {
anyhow::bail!(
"the only render endpoint is reserved for the virtual mic (capturing it would \
echo the client's voice back) attach another output device or install the \
Steam Streaming pair to get desktop audio"
);
};
tracing::warn!(mic = %wiring.mic_render.as_ref().unwrap().0, loopback = %lb.0,
"default render endpoint is the virtual-mic target — loopback-capturing the plan's \
endpoint instead");
audio_control::open_endpoint(lb)?
} else {
default
};
let mut audio_client = device.get_iaudioclient().context("IAudioClient")?;
// 48 kHz f32 interleaved in the requested channel layout; autoconvert lets WASAPI's
// shared-mode SRC match the engine mix format to ours (incl. up/downmix to the requested
@@ -3,32 +3,32 @@
//! device and write the client's decoded mic PCM into that device's **render** endpoint; the device's
//! **capture** endpoint then surfaces as a microphone that host apps can record from.
//!
//! Target device, by friendly-name substring (first match wins; override with `PUNKTFUNK_MIC_DEVICE`):
//! VB-Audio "CABLE Input" (bundled by the installer — the preferred, dedicated mic target), the
//! "Steam Streaming Microphone", VoiceMeeter, or anything with "virtual" in the name.
//! [`super::audio_control`] sets the default playback to a DIFFERENT loopback-capable device so the
//! chosen mic is never the endpoint the loopback captures. If no candidate is present we auto-install
//! the Steam Streaming audio pair (see [`install_steam_audio_pair`]); failing that we return an error
//! with install guidance and the host runs without mic passthrough.
//! The target comes from the [`audio_control::wire_now`] plan (recomputed on every open): VB-Audio
//! "CABLE Input" (bundled by the installer — the dedicated mic target), the Steam Streaming
//! Microphone, VoiceMeeter, or anything with "virtual" in the name; `PUNKTFUNK_MIC_DEVICE` overrides.
//! The plan reserves the mic target and points the desktop-audio loopback at a DIFFERENT endpoint, so
//! injecting here can never echo into the host→client audio stream (see
//! [`wiring_plan`](super::wiring_plan) for the precedence rules and the headless cable-only case).
//! If no candidate is present we auto-install the Steam Streaming audio pair (see
//! [`install_steam_audio_pair`]); failing that we return an error with install guidance and the
//! caller (the mic pump) retries with backoff — a cable that appears later (driver install finishing
//! after boot) is picked up without a host restart.
//!
//! **Anti-echo guard (the whole point of this being non-trivial).** The desktop-audio plane
//! ([`super::wasapi_cap`]) loopback-captures the **default render endpoint**. WASAPI loopback
//! captures the *mixed* output of an endpoint — i.e. everything any app renders to it, including
//! what THIS module writes. So if the virtual-mic target is the same device the loopback captures,
//! the client's uplinked mic is captured straight back into the host→client audio stream: an
//! infinite echo. [`find_device`] therefore **excludes the default render endpoint** from the
//! candidates — the mic is guaranteed to land on a different device. (Linux gets this for free: its
//! mic is a dedicated `Audio/Source` node, structurally separate from the monitored sink.)
//! **Liveness.** Any WASAPI error in the render loop (endpoint invalidated/removed, audio engine
//! restart) exits the worker thread, which flips the `alive` flag — [`VirtualMic::push`] then
//! returns `false` and the pump reopens (re-planning, so endpoint churn re-resolves). Before this
//! existed, the first device change silently killed mic passthrough for the rest of the host's life.
//!
//! `push` enqueues decoded interleaved-f32 PCM into a bounded ring (drop-oldest beyond ~80 ms so mic
//! latency stays bounded); a dedicated COM-apartment thread renders it event-driven, filling silence
//! when the client isn't talking. WASAPI objects are `!Send`, so they live entirely on that thread
//! (mirrors `WasapiLoopbackCapturer`).
//! `push` enqueues decoded interleaved-f32 PCM into a bounded ring (drop-oldest beyond ~120 ms so
//! mic latency stays bounded); a dedicated COM-apartment thread renders it event-driven through an
//! adaptive jitter buffer (prime → hold → re-prime, see the render loop — clients arrive in bursts,
//! the device pulls per-period), filling silence when the client isn't talking. WASAPI objects are
//! `!Send`, so they live entirely on that thread (mirrors `WasapiLoopbackCapturer`).
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it.
#![deny(clippy::undocumented_unsafe_blocks)]
use super::{VirtualMic, SAMPLE_RATE};
use super::{audio_control, VirtualMic, SAMPLE_RATE};
use anyhow::{anyhow, Context, Result};
use std::collections::VecDeque;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -41,22 +41,23 @@ use wasapi::{Direction, SampleType, StreamMode, WaveFormat};
const CHANNELS: u32 = 2;
/// 48 kHz stereo f32: 2 channels * 4 bytes.
const BLOCK_ALIGN: usize = 2 * 4;
/// Bound the inject queue at ~80 ms so the passed-through mic stays low-latency (drop oldest beyond).
const MAX_QUEUE_BYTES: usize = (SAMPLE_RATE as usize * 80 / 1000) * BLOCK_ALIGN;
/// Render-endpoint friendly-name substrings (lowercased) we can write into so the device's capture
/// endpoint becomes a host mic. Ordered by preference.
const CANDIDATES: &[&str] = &[
"cable input", // VB-Audio Virtual Cable — bundled by the installer; the preferred dedicated mic target
"steam streaming microphone",
"voicemeeter input",
"voicemeeter aux input",
"virtual",
];
/// Jitter-buffer priming depth (~48 ms): the render loop emits pure silence until this much PCM
/// is queued, then plays from the cushion. Clients deliver mic audio in BURSTS (the Mac client's
/// input tap yields ~two 20 ms Opus packets every ~42 ms) while WASAPI pulls a small block every
/// device period (~10 ms) — with no cushion the queue sits near-empty and most periods insert
/// mid-stream silence: the "crackling mic" (heard live, Mac → Windows host 2026-07-03; the Linux
/// backend's process callback primes the same way and the identical stream was clean there). The
/// depth must cover the worst inter-burst gap (~42 ms), so ~48 ms with re-prime on a full drain.
const PRIME_BYTES: usize = (SAMPLE_RATE as usize * 48 / 1000) * BLOCK_ALIGN;
/// Bound the inject queue at ~120 ms so the passed-through mic stays low-latency (drop oldest
/// beyond): the priming cushion (~48 ms) plus arrival-burst headroom.
const MAX_QUEUE_BYTES: usize = (SAMPLE_RATE as usize * 120 / 1000) * BLOCK_ALIGN;
pub struct WasapiVirtualMic {
queue: Arc<Mutex<VecDeque<u8>>>,
stop: Arc<AtomicBool>,
/// False once the render thread has exited (device error or stop) — the pump's reopen signal.
alive: Arc<AtomicBool>,
join: Option<JoinHandle<()>>,
}
@@ -68,25 +69,29 @@ impl WasapiVirtualMic {
);
let queue = Arc::new(Mutex::new(VecDeque::<u8>::new()));
let stop = Arc::new(AtomicBool::new(false));
let alive = Arc::new(AtomicBool::new(true));
// Bring-up handshake: report the resolved device (or the error) before returning, so a missing
// virtual-mic device surfaces as Err (the caller retries with backoff) not a silent dead thread.
let (ready_tx, ready_rx) = sync_channel::<Result<String>>(1);
let (q, st) = (queue.clone(), stop.clone());
let (q, st, al) = (queue.clone(), stop.clone(), alive.clone());
let join = thread::Builder::new()
.name("punktfunk-wasapi-mic".into())
.spawn(move || {
if let Err(e) = render_thread(q, st, ready_tx) {
tracing::error!(error = %format!("{e:#}"), "wasapi virtual-mic thread failed");
}
// Normal stop or device error alike: this instance is done — the pump reopens.
al.store(false, Ordering::Release);
})
.context("spawn wasapi mic thread")?;
match ready_rx.recv_timeout(Duration::from_secs(3)) {
match ready_rx.recv_timeout(Duration::from_secs(5)) {
Ok(Ok(name)) => {
tracing::info!(device = %name,
"WASAPI virtual mic ready (client mic → this device's render endpoint)");
Ok(WasapiVirtualMic {
queue,
stop,
alive,
join: Some(join),
})
}
@@ -106,9 +111,12 @@ impl Drop for WasapiVirtualMic {
}
impl VirtualMic for WasapiVirtualMic {
fn push(&self, pcm: &[f32]) {
fn push(&self, pcm: &[f32]) -> bool {
if !self.alive.load(Ordering::Acquire) {
return false;
}
let Ok(mut q) = self.queue.lock() else {
return;
return false;
};
q.reserve(pcm.len() * 4);
for &s in pcm {
@@ -119,109 +127,50 @@ impl VirtualMic for WasapiVirtualMic {
let excess = q.len() - MAX_QUEUE_BYTES;
q.drain(..excess);
}
true
}
fn alive(&self) -> bool {
self.alive.load(Ordering::Acquire)
}
fn discard(&self) {
if let Ok(mut q) = self.queue.lock() {
q.clear();
}
}
fn channels(&self) -> u32 {
CHANNELS
}
}
/// The endpoint ID of the device the desktop-audio loopback records (the **default render
/// endpoint**, see [`super::wasapi_cap`]). The virtual mic must never target this device — injecting
/// there echoes the client's mic back into the host→client audio stream. `None` if it can't be
/// resolved (then [`find_device`] can't prove a candidate is safe and falls back to name-only
/// matching — no worse than before the guard existed).
fn default_render_id() -> Option<String> {
wasapi::DeviceEnumerator::new()
.ok()?
.get_default_device(&Direction::Render)
.ok()?
.get_id()
.ok()
}
/// Resolve the virtual-mic target among render endpoints by friendly-name, **excluding the endpoint
/// the loopback captures** (the [`default_render_id`] anti-echo guard). Logs all candidates so a
/// missing/skipped device is diagnosable.
fn find_device() -> Result<wasapi::Device> {
let enumerator = wasapi::DeviceEnumerator::new().context("DeviceEnumerator")?;
let collection = enumerator
.get_device_collection(&Direction::Render)
.context("render device collection")?;
let n = collection.get_nbr_devices().context("device count")?;
let want = std::env::var("PUNKTFUNK_MIC_DEVICE")
.ok()
.map(|s| s.to_lowercase());
// The device the loopback captures — a name match on it is rejected below (would echo).
let loopback_id = default_render_id();
let mut names = Vec::new();
let mut found = None;
let mut skipped_loopback = false;
for i in 0..n {
let Ok(dev) = collection.get_device_at_index(i) else {
continue;
};
let name = dev.get_friendlyname().unwrap_or_default();
let lname = name.to_lowercase();
let hit = match &want {
Some(w) => lname.contains(w),
None => CANDIDATES.iter().any(|c| lname.contains(c)),
};
if hit && found.is_none() {
// Anti-echo guard: never inject into the endpoint the loopback captures.
let is_loopback = match (dev.get_id().ok(), loopback_id.as_deref()) {
(Some(id), Some(lb)) => id == lb,
_ => false,
};
if is_loopback {
skipped_loopback = true;
tracing::warn!(device = %name,
"virtual-mic candidate is the loopback (default render) endpoint — skipping; \
injecting there would echo the client's mic into the desktop-audio stream");
} else {
found = Some(dev);
}
}
names.push(name);
}
found.ok_or_else(|| {
if skipped_loopback {
anyhow!(
"the only virtual-mic candidate among render endpoints {names:?} is the default \
playback device the host loopback-captures injecting there would echo the mic \
back to the client. Add a SEPARATE virtual audio device for the mic (e.g. the Steam \
Streaming Microphone) or set a different default playback device, then reconnect."
)
} else {
anyhow!(
"no virtual-mic device among render endpoints {names:?}. Install VB-Audio Virtual \
Cable or enable Steam Remote Play's microphone (Steam Streaming Microphone), or set \
PUNKTFUNK_MIC_DEVICE=<friendly-name substring>."
)
}
})
}
/// Find the virtual-mic device, and if none exists, try to AUTO-INSTALL one so mic passthrough works
/// out of the box (then re-find). Falls back to the guidance error if nothing can be installed.
fn find_or_install_device() -> Result<wasapi::Device> {
match find_device() {
Ok(d) => Ok(d),
Err(e) => {
tracing::info!("no usable virtual mic device present — attempting auto-install");
// SAFETY: `install_steam_audio_pair` is `unsafe` only because it `LoadLibraryExW`s
// `newdev.dll` and calls `DiInstallDriverW` through a `transmute`d function pointer;
// calling it imposes no extra precondition here (it takes no args and aliases nothing).
// Its internal contract holds: the `DiInstall` type matches the documented
// `BOOL DiInstallDriverW(HWND, PCWSTR, DWORD, PBOOL)` ABI, and it passes a
// NUL-terminated UTF-16 INF path with null/zero optional args. Invoked once on the
// dedicated mic thread.
if unsafe { install_steam_audio_pair() } {
find_device()
} else {
Err(e)
}
/// Resolve the mic inject target from the wiring plan, auto-installing the Steam Streaming pair
/// when nothing usable exists (then re-planning). Runs on the COM-initialized render thread.
fn resolve_target() -> Result<(wasapi::Device, String)> {
let mut wiring = audio_control::wire_now();
if wiring.mic_render.is_none() {
tracing::info!("no usable virtual mic device present — attempting auto-install");
// SAFETY: `install_steam_audio_pair` is `unsafe` only because it `LoadLibraryExW`s
// `newdev.dll` and calls `DiInstallDriverW` through a `transmute`d function pointer;
// calling it imposes no extra precondition here (it takes no args and aliases nothing).
// Its internal contract holds: the `DiInstall` type matches the documented
// `BOOL DiInstallDriverW(HWND, PCWSTR, DWORD, PBOOL)` ABI, and it passes a
// NUL-terminated UTF-16 INF path with null/zero optional args. Invoked once on the
// dedicated mic thread.
if unsafe { install_steam_audio_pair() } {
wiring = audio_control::wire_now();
}
}
let Some(ep) = wiring.mic_render else {
anyhow::bail!(
"no virtual-mic render endpoint on this box. Install VB-Audio Virtual Cable (the host \
installer bundles it) or enable Steam Remote Play's microphone (Steam Streaming \
Microphone), or set PUNKTFUNK_MIC_DEVICE=<friendly-name substring>."
);
};
let name = ep.0.clone();
Ok((audio_control::open_endpoint(&ep)?, name))
}
/// Best-effort: install BOTH Steam Streaming audio devices (the "Steam pair") so mic passthrough
@@ -229,9 +178,9 @@ fn find_or_install_device() -> Result<wasapi::Device> {
/// Play ships `SteamStreamingMicrophone.inf` + `SteamStreamingSpeakers.inf`: the microphone gives the
/// virtual mic a target whose **capture** endpoint apps record from, and the speakers give a
/// **render** endpoint a headless box can loopback-capture that is NOT the mic — so the loopback and
/// the mic land on different devices and never echo (see [`find_device`]). Returns true if either
/// installed. No-op when Steam isn't installed (INFs absent), the install is denied (needs admin —
/// the host runs as SYSTEM), or `PUNKTFUNK_NO_MIC_INSTALL` is set.
/// the mic land on different devices and never echo (see [`super::wiring_plan`]). Returns true if
/// either installed. No-op when Steam isn't installed (INFs absent), the install is denied (needs
/// admin — the host runs as SYSTEM), or `PUNKTFUNK_NO_MIC_INSTALL` is set.
unsafe fn install_steam_audio_pair() -> bool {
// Microphone first (the mic's actual target); speakers second (the distinct desktop-audio sink).
let mic = try_install_steam_audio("SteamStreamingMicrophone.inf");
@@ -320,8 +269,7 @@ fn render_thread(
// Open + start the render stream. The WASAPI objects must outlive the loop, so build them here and
// keep them (a closure that *returned* them would drop them); on any failure report Err and exit.
let setup = (|| -> Result<(wasapi::AudioClient, wasapi::AudioRenderClient, wasapi::Handle, String)> {
let device = find_or_install_device()?;
let name = device.get_friendlyname().unwrap_or_else(|_| "virtual mic".into());
let (device, name) = resolve_target()?;
let mut audio_client = device.get_iaudioclient().context("IAudioClient")?;
// 48 kHz stereo f32; autoconvert lets WASAPI shared-mode SRC match the device mix format.
let desired = WaveFormat::new(
@@ -359,7 +307,19 @@ fn render_thread(
};
let _ = ready.send(Ok(name));
// Any error below (endpoint invalidated/removed, engine restart) propagates out of the loop,
// ending the thread — the `alive` flag flips in the spawn wrapper and the pump reopens.
//
// Adaptive jitter buffer (mirrors the Linux backend's process callback): clients push mic
// audio in bursts on their own clock while the device pulls a block every period from an
// independent clock, so a greedy per-period drain leaves the queue near-empty and pads most
// periods with mid-stream silence — audible as constant crackling. Instead: emit silence
// until [`PRIME_BYTES`] is buffered, then play from the cushion (zero-filling only a
// momentary shortfall), and re-prime only after a genuine FULL drain (the client went quiet —
// between talk spurts the cushion rebuilds, and [`VirtualMic::discard`] resets it across
// session gaps).
let mut buf: Vec<u8> = Vec::new();
let mut primed = false;
while !stop.load(Ordering::Relaxed) {
// The device signals when it wants more data; finite timeout keeps `stop` responsive.
if h_event.wait_for_event(100).is_err() {
@@ -375,13 +335,21 @@ fn render_thread(
if buf.len() < need {
buf.resize(need, 0);
}
// Silence base; overwrite with queued mic PCM (zero-pad the tail when the client is quiet).
// Silence base; overwrite with queued mic PCM once the cushion is primed.
buf[..need].fill(0);
{
let mut q = queue.lock().unwrap();
let n = q.len().min(need);
for (i, b) in q.drain(..n).enumerate() {
buf[i] = b;
if !primed && q.len() >= PRIME_BYTES {
primed = true;
}
if primed {
let n = q.len().min(need);
for (i, b) in q.drain(..n).enumerate() {
buf[i] = b;
}
if q.is_empty() {
primed = false; // fully drained — re-prime before producing again
}
}
}
render_client
@@ -0,0 +1,274 @@
//! Windows audio endpoint assignment — the PURE planning logic behind
//! [`audio_control`](super::audio_control), split out so it compiles (and its unit tests run) on
//! every platform: the precedence rules here encode the hard-won field knowledge, and regressing
//! them must fail CI on Linux too, not only on a Windows box.
//!
//! Two jobs share the render endpoints and must never collide:
//!
//! * the **virtual mic** writes the client's decoded mic PCM into a virtual cable's render
//! endpoint (its capture side surfaces as a host microphone), and
//! * the **desktop-audio loopback** captures a render endpoint's mix for the host→client
//! audio stream.
//!
//! WASAPI loopback captures *everything* an endpoint renders — including what the virtual mic
//! writes — so if both land on the same device the client's voice echoes straight back into the
//! client's own audio stream. The plan therefore assigns the mic its endpoint FIRST (VB-CABLE is
//! bundled by the installer for exactly this) and gives the loopback a *different* one; when only
//! the cable exists (headless box, no other output), the MIC wins and the loopback is honestly
//! unavailable. The old code did the opposite — the mic refused the cable because it was the
//! default render endpoint — which permanently killed mic passthrough in the exact configuration
//! the installer ships (VB-CABLE as the only render device).
/// A `(friendly_name, endpoint_id)` pair as enumerated from WASAPI.
pub(crate) type Endpoint = (String, String);
/// The coherent endpoint assignment for one wiring pass. Computed fresh on every mic/capture
/// (re)open — Windows endpoints churn (boot-time registration, hotplug, driver installs), so a
/// once-per-process plan goes stale.
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct Wiring {
/// Render endpoint RESERVED for the virtual mic (the write target). The loopback must never
/// capture this device.
pub mic_render: Option<Endpoint>,
/// The mic device's CAPTURE side — host apps record this; made the default recording device.
pub mic_capture: Option<Endpoint>,
/// Render endpoint for the desktop-audio loopback; made the default playback device.
pub loopback_render: Option<Endpoint>,
}
/// Render-endpoint friendly-name substrings (lowercased) usable as the virtual-mic write target,
/// ordered by preference. VB-CABLE first: the installer bundles it for this exact purpose.
const MIC_CANDIDATES: &[&str] = &[
"cable input", // VB-Audio Virtual Cable — bundled by the installer
"steam streaming microphone",
"voicemeeter input",
"voicemeeter aux input",
"virtual",
];
/// `(mic render substring, matching capture substring)` — which capture endpoint surfaces the
/// audio written to a given mic render target.
fn capture_for(mic_render_lname: &str) -> &'static [&'static str] {
if mic_render_lname.contains("cable") {
&["cable output"]
} else if mic_render_lname.contains("steam streaming microphone") {
&["steam streaming microphone"]
} else if mic_render_lname.contains("voicemeeter") {
&["voicemeeter out", "voicemeeter"]
} else {
&["virtual"]
}
}
/// A render endpoint no loopback should capture: the VB-CABLE (reserved for the mic even when it
/// isn't the chosen target — capturing a cable someone else feeds echoes too) and the Steam
/// Streaming Speakers, whose loopback is silent (validated live).
fn excluded_from_loopback(lname: &str) -> bool {
lname.contains("cable") || lname.contains("steam streaming speakers")
}
/// A known-virtual device (cables/streaming endpoints). A render WITHOUT these markers is real
/// hardware — the best loopback source (apps render there by default and the operator can also
/// hear it).
fn virtualish(lname: &str) -> bool {
lname.contains("virtual")
|| lname.contains("cable")
|| lname.contains("steam streaming")
|| lname.contains("voicemeeter")
}
/// Compute the assignment. `mic_want` is the operator override (`PUNKTFUNK_MIC_DEVICE`,
/// lowercased): when set it beats the built-in candidate order for the mic target.
pub(crate) fn plan(renders: &[Endpoint], captures: &[Endpoint], mic_want: Option<&str>) -> Wiring {
let find_render = |needle: &str| {
renders
.iter()
.find(|(n, _)| n.to_lowercase().contains(needle))
.cloned()
};
// 1. Mic target first — it has the narrower requirements (must be a virtual cable).
let mic_render = match mic_want {
Some(w) => find_render(w),
None => MIC_CANDIDATES.iter().find_map(|c| find_render(c)),
};
// 2. Its capture side (what host apps record).
let mic_capture = mic_render.as_ref().and_then(|(name, _)| {
capture_for(&name.to_lowercase()).iter().find_map(|c| {
captures
.iter()
.find(|(n, _)| n.to_lowercase().contains(c))
.cloned()
})
});
// 3. Loopback from the REMAINING renders: real hardware > Steam Streaming Microphone (its
// loopback works, unlike the Speakers') > any non-excluded leftover.
let not_mic = |id: &str| mic_render.as_ref().is_none_or(|(_, mid)| mid != id);
let loopback_render = renders
.iter()
.find(|(n, id)| {
let ln = n.to_lowercase();
not_mic(id) && !excluded_from_loopback(&ln) && !virtualish(&ln)
})
.or_else(|| {
renders.iter().find(|(n, id)| {
not_mic(id) && n.to_lowercase().contains("steam streaming microphone")
})
})
.or_else(|| {
renders
.iter()
.find(|(n, id)| not_mic(id) && !excluded_from_loopback(&n.to_lowercase()))
})
.cloned();
Wiring {
mic_render,
mic_capture,
loopback_render,
}
}
#[cfg(test)]
mod tests {
use super::*;
fn ep(name: &str) -> Endpoint {
(name.to_string(), format!("id-{}", name.to_lowercase()))
}
/// The shipped configuration: real output + VB-CABLE. Mic gets the cable, loopback the
/// speakers, recording default = CABLE Output.
#[test]
fn gaming_pc_with_cable() {
let renders = [
ep("Speakers (Realtek HD Audio)"),
ep("CABLE Input (VB-Audio Virtual Cable)"),
];
let captures = [
ep("Microphone (Webcam)"),
ep("CABLE Output (VB-Audio Virtual Cable)"),
];
let w = plan(&renders, &captures, None);
assert_eq!(
w.mic_render.unwrap().0,
"CABLE Input (VB-Audio Virtual Cable)"
);
assert_eq!(
w.mic_capture.unwrap().0,
"CABLE Output (VB-Audio Virtual Cable)"
);
assert_eq!(w.loopback_render.unwrap().0, "Speakers (Realtek HD Audio)");
}
/// THE historical dead-end: headless box where VB-CABLE is the ONLY render endpoint (and
/// therefore the default). The mic must WIN the cable; the loopback is honestly absent.
/// (The old anti-echo guard rejected the cable here → mic permanently dead.)
#[test]
fn headless_cable_only_mic_wins() {
let renders = [ep("CABLE Input (VB-Audio Virtual Cable)")];
let captures = [ep("CABLE Output (VB-Audio Virtual Cable)")];
let w = plan(&renders, &captures, None);
assert!(w.mic_render.is_some(), "mic must claim the only cable");
assert!(w.loopback_render.is_none(), "no echo-safe loopback exists");
}
/// Headless with the Steam pair installed: cable = mic, Steam Streaming Microphone = the
/// loopback (its loopback works; the Speakers' is silent — validated live).
#[test]
fn headless_with_steam_pair() {
let renders = [
ep("CABLE Input (VB-Audio Virtual Cable)"),
ep("Speakers (Steam Streaming Speakers)"),
ep("Speakers (Steam Streaming Microphone)"),
];
let captures = [
ep("CABLE Output (VB-Audio Virtual Cable)"),
ep("Microphone (Steam Streaming Microphone)"),
];
let w = plan(&renders, &captures, None);
assert_eq!(
w.mic_render.unwrap().0,
"CABLE Input (VB-Audio Virtual Cable)"
);
assert_eq!(
w.loopback_render.unwrap().0,
"Speakers (Steam Streaming Microphone)"
);
assert_eq!(
w.mic_capture.unwrap().0,
"CABLE Output (VB-Audio Virtual Cable)"
);
}
/// No cable: the Steam Streaming Microphone doubles as the mic target, and the loopback
/// must NOT then pick the same endpoint (real hardware wins).
#[test]
fn steam_mic_as_target_never_doubles_as_loopback() {
let renders = [
ep("Speakers (Steam Streaming Microphone)"),
ep("Speakers (Realtek HD Audio)"),
];
let captures = [ep("Microphone (Steam Streaming Microphone)")];
let w = plan(&renders, &captures, None);
assert_eq!(
w.mic_render.unwrap().0,
"Speakers (Steam Streaming Microphone)"
);
assert_eq!(w.loopback_render.unwrap().0, "Speakers (Realtek HD Audio)");
}
/// No cable and ONLY the Steam mic: mic wins it, loopback honestly absent (never the same
/// device — that would echo).
#[test]
fn steam_mic_only_no_echo() {
let renders = [ep("Speakers (Steam Streaming Microphone)")];
let captures = [ep("Microphone (Steam Streaming Microphone)")];
let w = plan(&renders, &captures, None);
assert!(w.mic_render.is_some());
assert!(w.loopback_render.is_none());
}
/// Steam Streaming Speakers never become the loopback (silent loopback, validated live) —
/// even when they're the only non-mic endpoint.
#[test]
fn steam_speakers_never_loopback() {
let renders = [
ep("CABLE Input (VB-Audio Virtual Cable)"),
ep("Speakers (Steam Streaming Speakers)"),
];
let w = plan(&renders, &[], None);
assert!(w.loopback_render.is_none());
}
/// Operator override beats the candidate order.
#[test]
fn env_override_wins() {
let renders = [
ep("CABLE Input (VB-Audio Virtual Cable)"),
ep("Voicemeeter Input (VB-Audio Voicemeeter VAIO)"),
];
let captures = [ep("Voicemeeter Out B1 (VB-Audio Voicemeeter VAIO)")];
let w = plan(&renders, &captures, Some("voicemeeter input"));
assert_eq!(
w.mic_render.unwrap().0,
"Voicemeeter Input (VB-Audio Voicemeeter VAIO)"
);
assert_eq!(
w.mic_capture.unwrap().0,
"Voicemeeter Out B1 (VB-Audio Voicemeeter VAIO)"
);
}
/// No virtual device anywhere: no mic target (open fails with guidance), loopback = the
/// real output — desktop audio unaffected.
#[test]
fn no_virtual_device() {
let renders = [ep("Speakers (Realtek HD Audio)")];
let w = plan(&renders, &[], None);
assert!(w.mic_render.is_none());
assert_eq!(w.loopback_render.unwrap().0, "Speakers (Realtek HD Audio)");
}
}
@@ -1129,8 +1129,14 @@ impl VideoConverter {
pInputSurface: std::mem::ManuallyDrop::new(in_view),
..Default::default()
};
self.vctx
.VideoProcessorBlt(&self.vp, &out_view, 0, &[stream])
.context("VideoProcessorBlt")
let blt =
self.vctx
.VideoProcessorBlt(&self.vp, &out_view, 0, std::slice::from_ref(&stream));
// COM in-params never transfer ownership: the Blt only borrowed the input view, and the
// struct's `ManuallyDrop` field suppressed its release — drop it by hand, success or not.
// (Skipping this leaked one view + its UMD allocation PER CONVERTED FRAME — the SDR hot
// path; D3D11 defers the actual destruction until the GPU is done with the blit.)
drop(std::mem::ManuallyDrop::into_inner(stream.pInputSurface));
blt.context("VideoProcessorBlt")
}
}

Some files were not shown because too many files have changed in this diff Show More