docs(release): A/V sync takes a TL;DR slot, and the old audio claim was wrong
android / android (pull_request) Canceled after 0s
apple / swift (pull_request) Canceled after 0s
apple / screenshots (pull_request) Canceled after 0s
ci / rust (pull_request) Canceled after 0s
ci / rust-arm64 (pull_request) Canceled after 0s
ci / web (pull_request) Canceled after 0s
ci / docs-site (pull_request) Canceled after 0s
ci / bun-nix (pull_request) Canceled after 0s
nix / flake (pull_request) Canceled after 0s
windows / build (aarch64-pc-windows-msvc) (pull_request) Canceled after 0s
windows / build (x86_64-pc-windows-msvc) (pull_request) Canceled after 0s
android / android (pull_request) Canceled after 0s
apple / swift (pull_request) Canceled after 0s
apple / screenshots (pull_request) Canceled after 0s
ci / rust (pull_request) Canceled after 0s
ci / rust-arm64 (pull_request) Canceled after 0s
ci / web (pull_request) Canceled after 0s
ci / docs-site (pull_request) Canceled after 0s
ci / bun-nix (pull_request) Canceled after 0s
nix / flake (pull_request) Canceled after 0s
windows / build (aarch64-pc-windows-msvc) (pull_request) Canceled after 0s
windows / build (x86_64-pc-windows-msvc) (pull_request) Canceled after 0s
main moved another 62 commits (a8a4b11f->fca9f42c), taking 0.25.0 to 391 since v0.24.0. Five PRs: decode aliasing (#102), A/V sync (#101), gyro correctness (#99), web console sweep (#100), Apple ATS (#103). THE CORRECTION THAT MATTERED. The notes carried "Audio that falls behind the picture pulls itself back … Android was worst, with no correction at all", describing the jitter ring's buffer-shedding as if it were sync. It never was. The host has stamped `pts_ns` on every audio datagram since long before v0.24.0 and EVERY CLIENT DECODED IT AND NEVER READ IT — verified in the v0.24.0 tree (`crates/punktfunk-host/src/native/audio.rs:162` stamps it; the client audio paths ignore it). Lip-sync was an emergent property of buffer depth, and it got WORSE as video got faster, which is why shaving milliseconds off the audio budget had never helped. That bullet is rewritten to say what is actually true, and A/V sync takes a TL;DR slot. It displaces the settings-BOM bullet, which was the weakest of the six as a HEADLINE: conditional (only if the file was ever saved by PowerShell), partly duplicated by the Windows non-C: entry, and it survives verbatim in Fixed. A/V sync affects every user, every session, every client, with sound on — and unlike most of this release it shipped broken in EVERY release we have ever made. GYRO NEEDS AN UPGRADE NOTE, so it got one. The pipeline was wrong end to end and is now measured against a real controller, which MOVES AIM SENSITIVITY: a pad presented as a DualShock 4 reported gyro 40x fast (host-side), and a PlayStation pad on Android reported ~30% short (client-side). At 40x nobody could have compensated — gyro aim was unusable, not miscalibrated — but the Android ~1.4x change is exactly the size a real person tunes around, so `## Before you update` names it specifically. DELIBERATELY NOT PROMOTED. The decode-aliasing program (#102) reads like a catastrophe — H.264 decoding into a surface it predicted from on 297 of every 300 access units of every stream we emit, on both rungs — but it NEVER SHIPPED: `git ls-tree v0.24.0 crates/` has no pf-vkdecode/pf-dxvadec/pf-vaadec/pf-bitstream. It is a ship-blocker that was cleared, and writing "your picture was subtly wrong" would be false for every reader. It contributes one clause to the decode entry (every path is now checked frame-by-frame against a reference decoder; Windows + Intel AV1 routes through Direct3D) and a full section in the changelog. Same reasoning already applied to #96 and the rav1d abort. Changelog gains the A/V sync mechanism (including that video is the master and continuity outranks sync — the ring refuses a sync request that would break audio on a jittery link) and the aliasing section, with the four independent reasons four gates missed it: a structurally-blind conformance vector, a test that had encoded the bug AS CORRECT, a vacuous assertion that could not fail, and the fact that it streamed clean on glass. gpu_parity is 11 legs, not the 9 an earlier note claimed. Verified after the merge: lock diff versions-only 35/35, `cargo metadata --locked` resolves (39 members), `cargo fmt --all --check` clean in both workspaces, notes body 0 internal-vocabulary hits, Play notes 497/500 by android.yml's own gate. Wire 2, C ABI 17, no new capability bits in this range.
This commit is contained in:
+105
@@ -309,6 +309,111 @@ still `Default`, never an error.
|
||||
now boxed inside each owning wrapper, and create-time arrays are fields of the stored parameters
|
||||
assembled at their final address. The same shape was fixed pre-emptively in H.264/H.265.
|
||||
|
||||
### A/V sync — it did not previously exist
|
||||
|
||||
The host has always stamped `pts_ns` on every audio datagram. **Every client decoded it into
|
||||
`AudioPacket` / `AudioPCM` and never read it.** Video's `pts_ns` was used end to end; audio free-ran
|
||||
at whatever depth its jitter ring reached; nothing compared them. The A/V offset was an emergent
|
||||
property of buffer depths — it moved whenever the ring ratcheted under underrun pressure, and it got
|
||||
**worse every time video got faster**, because a quicker decoder lowers the video leg and leaves
|
||||
audio's where it was. That is why shaving milliseconds off the audio budget had never helped.
|
||||
|
||||
Two host defects were prerequisites:
|
||||
- **`pts_ns` was stamped at encode time**, inside the loop draining an already-accumulated chunk, so
|
||||
every frame of a chunk carried near-identical timestamps describing *when we got round to
|
||||
encoding*. Now derived from the chunk's arrival instant minus queued-frame duration, re-anchored
|
||||
per chunk.
|
||||
- **The host did not pace.** One capture callback hands over a whole quantum (5 ms honoured, **21.3 ms
|
||||
on a VM**, where stock PipeWire raises `min-quantum` to 1024), drained into back-to-back
|
||||
`send_datagram` calls — a 4–5 frame burst then ~21 ms of nothing, which a ring could only absorb by
|
||||
standing a burst period deep. Frames now leave on the audio clock (`FRAME_INTERVAL` 5 ms,
|
||||
`PACE_MAX_SLEEP` 10 ms, `PACE_REANCHOR` 100 ms). Costs no average latency.
|
||||
|
||||
```
|
||||
audio_e2e = (now + buffered_ahead + clock_offset) − pts_ns
|
||||
av_offset = audio_e2e − video_e2e (> 0 ⇒ audio behind the picture)
|
||||
```
|
||||
|
||||
`AvSync` EWMAs it (`AV_EWMA_TAU_MS = 2000`), ignores anything inside `AV_DEADBAND_MS = 10`, waits
|
||||
`AV_MIN_OBSERVATIONS = 100` before a first correction, and **refuses rather than clamps** beyond
|
||||
`AV_SANE_LIMIT_MS = 1000` — a wall-clock step must not steer the ring.
|
||||
|
||||
⭐ **Video is the master, and continuity outranks sync.** `JitterPolicy::set_sync_target` takes only a
|
||||
*request*, clamped between the existing underrun-driven adaptive floor and the hard cap: a link whose
|
||||
jitter genuinely needs more buffer than the picture is away keeps its buffer, and the residual is
|
||||
reported rather than forced. `None`/`nil` reproduces prior behaviour bit-identically, which is how
|
||||
the four rings adopted it one at a time.
|
||||
|
||||
Per client: the Rust desktop reference is a new `video_e2e_ns` atomic beside `clock_offset`, written
|
||||
by the presenter and read by the audio thread. **Android** publishes `OnFrameRendered` — the one
|
||||
place that knows a frame *latched* — **raw, not floor-shaved** (the HUD shaves the OS present floor;
|
||||
sound must reach the ear when light reaches the eye), and stays inert below API 33 rather than
|
||||
substituting the release instant, which targets a future vsync 8–21 ms ahead of glass. **Apple**
|
||||
publishes its `LatencyMeter` sample as an *expiring level*, because that client has a backgrounded
|
||||
keep-alive that keeps audio playing while dropping video decode; its clamp raises the ceiling to the
|
||||
floor rather than `min(max(…))`, which on a device whose callback quantum alone exceeds the hard cap
|
||||
would otherwise hand back the cap, silently below the continuity floor.
|
||||
|
||||
Escape hatches: `PUNKTFUNK_NO_AV_SYNC=1` everywhere, plus
|
||||
`adb shell setprop debug.punktfunk.no_av_sync 1` on Android (a launcher-started app inherits no
|
||||
environment). Observability: `buffer_ms`/`target_ms` had only ever been a `tracing::debug!` line —
|
||||
and on a Deck the client runs under Steam's `reaper` with stdout on a pipe nobody can read, so the
|
||||
one number identifying a deep ring was unobtainable *on the device reporting the latency*. Now on the
|
||||
HUD and in the 1 Hz stats log on every client.
|
||||
|
||||
### Decode-target aliasing — caught before it shipped
|
||||
|
||||
⚠ **None of this ever shipped.** `git ls-tree v0.24.0 crates/` has no `pf-vkdecode`, `pf-dxvadec`,
|
||||
`pf-vaadec` or `pf-bitstream`; v0.24.0's decode rungs were libavcodec. This was a ship-blocker for
|
||||
the new stack, cleared — not a field bug.
|
||||
|
||||
Three of the four native rungs released a picture's surface **inside the plan→submission
|
||||
conversion**, then assigned the decode target a slot. `SlotMap::assign` returns the *lowest free
|
||||
slot* — the one just vacated. The submission then named one surface as both decode target and its own
|
||||
reference: `CurrPicTextureIndex == RefFrameMapTextureIndex[k]` on DXVA, or `pSetupReferenceSlot`
|
||||
sharing an array layer with `pReferenceSlots` on Vulkan. **Decode into the surface you are predicting
|
||||
from.**
|
||||
|
||||
- **AV1 / D3D11VA** — AV1 applies `refresh_frame_flags` *after* decode (7.20), so "read a slot then
|
||||
overwrite it" is the ordinary case: **268 of the vendored vector's 274 frames**, first at frame 6.
|
||||
- **H.264 / both Vulkan and D3D11VA** — `H264Planner` snapshots `dpb_refs` in `begin_picture`, before
|
||||
8.2.5 marking and the C.4.5.3 bump, so a picture the sliding window unmarks and the bump evicts
|
||||
lands in *both* `dpb_refs` and `dpb.removed`. Both conditions coincide only in low-delay H.264 —
|
||||
and NVENC guarantees it (`max_num_ref_frames = 3` alongside `max_dec_frame_buffering = 3`, plus
|
||||
`max_num_reorder_frames = 0`). Result: **297 of every 300 access units of every stream a punktfunk
|
||||
host emits**, at every resolution, on both rungs.
|
||||
- **H.265 is exempt, now measured rather than argued** — 0 of 120 aliases, with a counterfactual that
|
||||
moves the snapshot one call earlier and reproduces 115 of 120.
|
||||
- **VAAPI's exemption was incidental**: the precondition is fully present (117 of 120 AUs) but
|
||||
`plan_to_va` never invents a surface. That held only because three call sites happened to write
|
||||
`free_surface()` and `surface_table()` adjacently; `acquire_target` now returns index, surface and
|
||||
table together so a later edit cannot split them.
|
||||
|
||||
Fix is uniform: the plans grow `release_after_decode`, conversions hand removals back, callers
|
||||
release once the decode op is issued. Costs no slot (`SlotMap::new` allocates `max_dpb_frames + 1`).
|
||||
Both rungs hold the `Result` rather than `?`-ing it so the deferred release runs on failure paths —
|
||||
seven exits sat between conversion and release, each of which would have leaked a slot.
|
||||
|
||||
**Why four gates missed it**, all recorded: the conformance vector is *structurally blind* (level 1.3,
|
||||
no VUI `bitstream_restriction` ⇒ a 7-frame DPB against 2 reference frames, and it reorders) and
|
||||
passed 250/250 for two milestones; **a test had encoded the bug as correct**; another assertion was
|
||||
*vacuous* (it asserted the decode target was never also a reference while handing every picture its
|
||||
own never-reused surface id — distinct integers cannot collide); and **it streamed clean** — *"the
|
||||
2026-08-07 field sessions that looked clean were looking at wrong pixels."*
|
||||
|
||||
`gpu_parity` is now **11 legs** (not 9 — that note was written mid-PR): each decodes a vendored stream,
|
||||
reads back every output frame's NV12, crops to the display region and SHA-256s in *display order*
|
||||
against libavcodec goldens, frame count and flush tail included. The three new legs are our own
|
||||
encoder's output rather than conformance vectors — H.264 because the vector is blind to the shape,
|
||||
H.265 because an exemption with no stream behind it is how the H.264 defect survived two milestones,
|
||||
AV1 because the vector is one tile on all 274 frames while our encoder splits 4K into two tile rows,
|
||||
so every tile array the conversions fill had only ever been written at index 0. `video_vaapi_native`
|
||||
parity is new entirely: 7 legs, bit-identical on RDNA3.
|
||||
|
||||
⚠ Promoting D3D11VA AV1 to `verified` **changes rung selection** on Windows Intel/unknown vendors, not
|
||||
just a label. VAAPI stays `verified = false` deliberately — one vendor, never soaked; flipping it
|
||||
would move `auto` off Vulkan Video on every Linux AMD/Intel client including the Deck.
|
||||
|
||||
### Windows audio substrate
|
||||
|
||||
The host now mints its **own** devnodes from Valve's INFs (`SteamStreamingSpeakers.inf` /
|
||||
|
||||
@@ -9,8 +9,8 @@ This is the largest release so far — a bit over three hundred changes. The sho
|
||||
- **HDR and 10-bit streams stop leaving half your encoder idle** — the cap that held high-refresh HDR at around 130 fps is gone. Update the host.
|
||||
- **Windows hosts make their own "Punktfunk Speakers" and "Punktfunk Microphone"** instead of bundling a third-party audio cable. This needs Steam installed on the host, though it never has to run.
|
||||
- **Controllers, end to end:** a wired DualSense can play a game's fine-grained haptics and its own speaker, and more than twenty separate rumble, trigger and lightbar faults are fixed across every client and both hosts.
|
||||
- **Your settings could silently reset to defaults** if the file was ever saved by PowerShell. Fixed, and a broken settings file now says so instead of quietly starting over.
|
||||
- **A few things need you to act** — a new group on Linux, a firewall port for add-on interfaces. They are right below.
|
||||
- **Sound was never actually lined up with the picture.** Hosts have always marked audio with the moment it was captured, and no app had ever read that mark — so how far behind the picture your sound played was whatever buffering happened to settle on, and it got *worse* every time video got faster. All four apps now measure the gap and close it.
|
||||
- **A few things need you to act** — a new group on Linux, a firewall port for add-on interfaces, and a change to gyro aim sensitivity. They are right below.
|
||||
|
||||
## Before you update
|
||||
|
||||
@@ -22,10 +22,11 @@ Most people need to do nothing. Check this list if any of it applies to you.
|
||||
- **Saving a game with a custom launch command asks for your console password again**, and add-ons may no longer set launch commands at all. A third-party add-on that did will need updating by its author.
|
||||
- **A fresh install now runs the add-on runner by default.** Upgrades are untouched — if you switched it off, it stays off.
|
||||
- **If you set up a Steam Deck with the install script, consider rotating your console password.** It was written to a world-readable file; that is fixed.
|
||||
- **If you play with motion controls, your aim sensitivity will change.** The gyro pipeline was wrong at every stage and is now measured against a real controller, so the numbers moved: a controller presented to games as a DualShock 4 was reporting motion **forty times too fast**, and a PlayStation pad plugged into an Android phone was reporting about **30% short**. If you turned a game's sensitivity down or up to cope, set it back. The Android case is the one people plausibly tuned around — that aim now needs a *higher* in-game sensitivity than you are used to.
|
||||
|
||||
## New
|
||||
|
||||
- **Native GPU video decoding on Linux and Windows.** The desktop apps drive Vulkan Video, Direct3D and VAAPI themselves, with a CPU fallback, and no longer ship a media library at all. Choosing a decoder works exactly as before and your saved preference is migrated. Two real changes: there is no software HEVC decoder any more — none exists that we can ship — so a machine whose graphics card cannot decode HEVC now reconnects on a codec it can finish instead of crawling on the processor; and AV1 is only requested where your card can genuinely decode it, rather than anywhere a processor decoder happened to exist. Update the client.
|
||||
- **Native GPU video decoding on Linux and Windows.** The desktop apps drive Vulkan Video, Direct3D and VAAPI themselves, with a CPU fallback, and no longer ship a media library at all. Choosing a decoder works exactly as before and your saved preference is migrated. Two real changes: there is no software HEVC decoder any more — none exists that we can ship — so a machine whose graphics card cannot decode HEVC now reconnects on a codec it can finish instead of crawling on the processor; and AV1 is only requested where your card can genuinely decode it, rather than anywhere a processor decoder happened to exist. Every decoding path is now checked frame by frame against a reference decoder, so a picture that streams cleanly while being subtly wrong cannot pass unnoticed — which is exactly what was caught during development. On Windows with Intel graphics, AV1 decodes through Direct3D rather than Vulkan, which is far quicker on those cards. Update the client.
|
||||
- **Your DualSense's own haptics and speaker, carried from the host.** The fine-grained feedback in the grips — as distinct from the coarse rumble motors — now crosses the stream to the controller in your hands. Needs a DualSense or DualSense Edge **plugged in by USB** (over Bluetooth the pad exposes no audio device at all), a **Windows host with Steam**, and either the Android app or the desktop session client. Anywhere else nothing changes. A game that only uses ordinary rumble keeps rumbling exactly as before.
|
||||
- **The desktop console answers a mouse and a touchscreen.** Nothing in it had ever been clickable: the button bar is now live on every screen, Tab changes section, and right-click is Back. Saved host tiles gain a menu on Up — Wake, Copy link, Edit and Forget — so a renamed machine or a mistyped address can be fixed without leaving the console.
|
||||
- **Hold Select to press the host's Guide button.** Hold it on its own for about a third of a second and the host sees its Guide button go down, and stay down while you hold — so a longer hold reads as a long-press, which is how a big-screen host opens its Quick Access Menu. A quick tap still goes to the game. On by default on iPhone, iPad and Apple TV, where the system keeps the controller's own Home press for itself; off elsewhere, where the raw press already reaches the host.
|
||||
@@ -44,6 +45,7 @@ Most people need to do nothing. Check this list if any of it applies to you.
|
||||
- **The black screen on VPN-shaped networks heals itself in seconds, mid-stream**, with 0.25.0 on both ends, instead of needing a reconnect.
|
||||
- **The Steam Deck plugin is now a launcher into the app** rather than a second client — see the section below, because some things moved.
|
||||
- **The Android stats overlay stops charging your screen's own delay to the stream.** Your stream is exactly as fast as it was; the headline number gets smaller because it finally measures only the part Punktfunk controls.
|
||||
- **The stats overlay finally shows the audio side** — how much sound is queued ahead of your speakers, and how far that puts it from the picture. Both numbers, because a deep buffer on a rough connection is correct and only the gap tells you the difference. On every client; previously neither existed anywhere you could see.
|
||||
- **Every session now says why it ended**, so quitting a game you launched is no longer reported as trouble, and quitting takes you back to the library you launched it from.
|
||||
|
||||
## Fixed
|
||||
@@ -52,7 +54,7 @@ Most people need to do nothing. Check this list if any of it applies to you.
|
||||
- **HDR and 10-bit streams left half the encoder idle.** A rule written from one old measurement disabled split-frame encoding for every 10-bit stream, capping high-refresh HDR at around 130 fps and making fast motion look wrong. Re-measured across two generations of NVIDIA cards, splitting won every test. Update the host.
|
||||
- **Your settings could silently reset to defaults.** A settings file saved from PowerShell carries an invisible marker that made the app quietly start over — and any other damaged settings, saved-hosts or profiles file did the same without a word. Both now load correctly, or say exactly what is wrong.
|
||||
- **The microphone works again on iPhone, iPad and Mac.** The clients had stopped sending any microphone audio at all — recorded but never reaching the stream, silently and with no error. Separately, game audio on Apple devices clicked and crackled on Wi-Fi that delivers packets in bursts, where every other client sounded clean on the same host. Update the client.
|
||||
- **Audio that falls behind the picture pulls itself back**, instead of staying late for the rest of the session with reconnecting as the only cure. Android was worst, with no correction at all. Update the client.
|
||||
- **Sound is now placed against the picture instead of wherever buffering left it.** Hosts have always marked audio with the moment it was captured, and no app had ever read that mark — so lip-sync was an accident of how deep the audio buffer happened to sit, and it got *worse* as the picture got faster, because a quicker decoder shortens the video side and leaves sound exactly where it was. That is why shaving milliseconds off the audio path had never helped. Every app now measures the gap and steers the buffer to close it — but never so far that sound starts breaking up: on a jittery connection it keeps the buffer and leaves a small gap rather than take the dropout. Hosts also release audio at an even pace instead of in bursts, which mattered most on virtual machines. Update both sides.
|
||||
- **The host stopped pushing your whole desktop mix through Steam's voice channel** on PCs with Steam installed, where a mono or low-rate setting squeezed the entire game mix before encoding. Update the host.
|
||||
- **A host on a network that carries smaller packets than usual no longer streams a permanent black screen** — connected, every gauge green, and nothing on screen, with nothing in either log. The host now measures what the path really carries and warns with the actual diagnosis. Update the host; this works with every client already out there.
|
||||
- **Host discovery no longer goes permanently deaf.** A host missed on first launch is found without restarting the app — the usual trigger being the phone's local-network permission prompt, which the browse never recovered from — and every client now has a Refresh or Rescan control that really re-scans. Update the client.
|
||||
@@ -65,6 +67,10 @@ Most people need to do nothing. Check this list if any of it applies to you.
|
||||
- **Automatic bitrate stops overshooting what your device can actually decode**, and a momentary stall no longer pins your stream at a low bitrate for minutes afterwards. Recovering from a brief freeze also stops making things worse. Update both sides.
|
||||
- **A leftover folder from an uninstalled Sunshine or Apollo is no longer treated as a conflict**, and a host that crashed mid-session gives your screen back the next time it starts.
|
||||
- **Add-ons that read a launcher's database returned zero games on every Linux host**, indistinguishable from an empty library. Update the add-on.
|
||||
- **Motion controls were wrong end to end, and are now measured against a real controller.** A pad presented to games as a DualShock 4 reported gyro forty times too fast; Apple clients sent acceleration upside-down and rotation on the wrong axes, so a turn arrived as a roll; a PlayStation pad on Android reported about 30% short. A controller that stopped sending motion — app backgrounded, pad swapped — also left the virtual pad spinning forever, and one that had never sent any claimed to be in free fall, which some games read as a wild tilt. See the note above about sensitivity. Update both sides; they are independent.
|
||||
- **A Bluetooth controller's gyro now reaches the host from an Android phone.** Previously only a PlayStation pad plugged in by USB had working motion there — a DualSense, DualShock 4, Switch Pro or 8BitDo on Bluetooth had buttons and sticks but a silently dead sensor. Android 12 and newer. And when your controller has a gyro but the session's virtual pad has nowhere to put it, the client now says so on screen and names the setting that fixes it.
|
||||
- **The game library works on any host address on iPhone, iPad, Mac and Apple TV.** It only ever loaded on a home network: over Tailscale, WireGuard, or to a host with a public address, the library came up empty or errored — while streaming to that very same address worked perfectly, which is what made it so confusing. Cover art is also cached on disk now instead of being re-fetched every time. Update the client; the host is unchanged.
|
||||
- **The web console asks its own questions.** Sixteen grey browser confirmation boxes on destructive actions are replaced by the console's own dialogs, and the drop-downs, checkboxes and text areas in the hook editor and library forms are no longer unstyled controls at near-zero contrast. A virtual display can also be placed left of or above the origin again — typing a minus sign used to erase it before you could type the digits.
|
||||
- **The Steam Deck panel shows host names instead of addresses**, and "recreate shortcuts" actually recreates them.
|
||||
|
||||
## If you stream from a Steam Deck
|
||||
|
||||
Reference in New Issue
Block a user