fix(apple): relink stage-4 when the display link stops vending #442

Merged

enricobuehler merged 1 commits from worktree-stage4-link-watchdog into main 2026-08-29 08:48:28 +00:00

Conversation 0 Commits 1 Files Changed 1

+61 -11

enricobuehler commented 2026-08-29 08:35:05 +00:00

Owner

Copy Link

Copy Source

A user reported the stream freezing after a while. Two log bundles from
2026-08-28 (host + iPad Pro client) locate it precisely, and the same defect is
still on main.

The report

The picture froze twice in one session while audio and input kept running,
so it read as a hang rather than a disconnect. Both times the user quit
manually. Both times reconnecting cleared it instantly.

Client: punktfunk-apple 0.31.4, iPad Pro, iOS 27, 2560x1440@120 HEVC over
Tailscale. Host: KWin/Plasma, healthy throughout — capture running, audio
egress clean, session torn down normally on the client's deliberate quit.

What the logs say

Each frozen second logs this:

present  decoded=120 ok=0 fail=0 empty=0 noDrawable=120 inflightMax=0
         glassDeltaMs n=0  latchMs p50=0.00  vendLeadMs p50=0.00
stats    fps=120 presents=0 hostnet_p50=6.7 decode_p50=1.6

Three readings follow:

1. Network and decode are fine. 120 frames per second still arrive and
   decode in 1.6 ms.
2. The display link stopped. vendLeadMs is recorded on every
   metalDisplayLink(_:needsUpdate:) call. Zero samples means the
   CAMetalDisplayLink stopped calling back.
3. noDrawable exactly equals decoded. The render thread only wakes from
   the decoder now; the link never signals it and never stashes a drawable, so
   the last good frame stays on the layer.

The onset second is the same both times:

20:45:04  ok=119  dropped=0        21:01:37  ok=46  dropped=3
20:45:05  ok=0    dropped=3        21:01:38  ok=0   dropped=0

dropped counts a present whose handler reported presentedTime == 0 — the
system took the drawable and never put it on glass.

Why it is permanent

Stage-4 owns no drawable source of its own. Every drawable arrives as
update.drawable; it never calls nextDrawable() itself. So a link that stops
vending is a stream that never presents again.

startDeadlinePresenter clamps the pool to maximumDrawableCount = 2, which
leaves exactly one spare slot. Three presents dropped against that pool
exhausts it, the link can no longer vend, and no present can complete to free a
slot.

This end state is already documented in the same file.
MetalVideoPresenter.swift, in the macOS .transaction branch, carries the
post-mortem: "drawables are never released; after maximumDrawableCount vends
nextDrawable() blocks forever and the stream FREEZES" and "every present
reported presentedTime=0, nothing reached glass."

The structural gap is that stage-4 has no recovery. Stage-3 has
PresentGate.staleAfter = 0.1, described in its own comment as insurance
"rather than freezing the stream". Stage-4 returns early before the gate
exists and passes gate: nil; link.invalidate() is only reached at session
teardown. Only reconnecting recovers, which is exactly what the log shows.

What this changes

One file, +61/−11.

• Stale-link watchdog. The render thread records when it last received a
  drawable. Past 250 ms with frames still decoding and no vend, it retires the
  link generation and starts a fresh one. Invalidating the old link also
  returns the drawables it holds, so this covers an exhausted pool as well as a
  dead link. Only a decoded frame reaches that branch, so a quiet stream never
  trips it.
• Per-generation stop flag. startLink now takes a StopFlag — the type
  the other session workers already use — so a retired link thread exits
  without taking the session with it. The session token still stops every
  generation, so teardown is unchanged.
• Delegate lifetime. withExtendedLifetime now holds the link delegate for
  the thread's life. The old comment claiming "this closure is the strong
  ref" was wrong: delegate is a local, not a capture, and its last use is
  the weak store, so ARC may release it there. It survives in the shipped
  build, so it did not cause these freezes — but it is one optimiser decision
  away from a link that never fires at all.

Why 250 ms

A normal vend wait is bounded by one refresh — 8.3 ms at 120 Hz, 41 ms at a
24 Hz VRR floor. 250 ms is ~30 refreshes at 120 Hz, so it cannot fire on
ordinary phase jitter. The cost of a false positive is one relinked frame; the
cost of missing it is the whole session. Left as a constant rather than another
PUNKTFUNK_* lever: the log line prints the measured stall, so if the value is
wrong the field will say so and it is a one-line change.

Verification

Target	Result
arm64-apple-ios17.0, PunktfunkKit	compiles clean
arm64-apple-tvos17.0, PunktfunkKit	compiles clean
macOS, PunktfunkKit	compiles clean
macOS, swift test	384 tests, 6 skipped, 0 failures

Includes MetalPresenterTests and PresentPacingTests.

The swift test run needed PunktfunkCore.xcframework rebuilt first: the
checked-in bundle's header is stale against the current Rust ABI (it lacks
punktfunk_set_log_callback and twelve other symbols). That breaks
swift build on main too — pre-existing, and not touched here.

What is not verified

No automated test exercises the watchdog firing. The trigger is a display
link that ceases to vend, which has no local repro, and the logic sits inside a
thread closure holding a real CAMetalDisplayLink. Making it testable would
mean injecting startLink — an abstraction for a single caller. The field log
line is the check instead: the relink goes through ClientLog.error, which
writes to the ring that "Send logs to host" uploads, so the next report will
say whether the watchdog fired.

Whether relinking restores vending on a real device is unproven. If
invalidating the link does not release whatever holds the pool, the watchdog
will log every 250 ms rather than cure the freeze. That repeat is itself the
diagnosis, and it arrives in the uploaded log.

Deliberate non-change

Raising maximumDrawableCount back to 3 would widen the margin that ran out
here. It would also give back the measured present-floor fix from 2026-08-13
(Apple TV 4K, os present +32.5 at 60 Hz = 1.95 refresh periods). Left at 2:
the watchdog addresses the freeze without trading latency for it. Worth
revisiting only if the field shows the watchdog firing often.

Rejected paths

• A drawable-leak hunt in the present path. inflightMax returns to 0
  during the freeze, so every addPresentedHandler fired and the presents were
  accounted for. The logs establish the exhaustion chain but not the initial
  trigger for the three presentedTime == 0 returns, so a targeted leak fix
  would have been a guess. The watchdog is correct for any cause.
• A PUNKTFUNK_LINK_STALE_MS lever. See "Why 250 ms" — the safe window is
  ~30x wide, so there is nothing to calibrate yet.

---

User-facing fact changed? n/a — no install step, knob, port, limit or
feature behaviour changes. The only new output is one client log line on a path
that should never run.

A user reported the stream freezing after a while. Two log bundles from 2026-08-28 (host + iPad Pro client) locate it precisely, and the same defect is still on `main`. ## The report The picture froze twice in one session while **audio and input kept running**, so it read as a hang rather than a disconnect. Both times the user quit manually. Both times reconnecting cleared it instantly. Client: `punktfunk-apple 0.31.4`, iPad Pro, iOS 27, 2560x1440@120 HEVC over Tailscale. Host: KWin/Plasma, healthy throughout — capture running, audio egress clean, session torn down normally on the client's deliberate quit. ## What the logs say Each frozen second logs this: ``` present decoded=120 ok=0 fail=0 empty=0 noDrawable=120 inflightMax=0 glassDeltaMs n=0 latchMs p50=0.00 vendLeadMs p50=0.00 stats fps=120 presents=0 hostnet_p50=6.7 decode_p50=1.6 ``` Three readings follow: 1. **Network and decode are fine.** 120 frames per second still arrive and decode in 1.6 ms. 2. **The display link stopped.** `vendLeadMs` is recorded on *every* `metalDisplayLink(_:needsUpdate:)` call. Zero samples means the `CAMetalDisplayLink` stopped calling back. 3. **`noDrawable` exactly equals `decoded`.** The render thread only wakes from the decoder now; the link never signals it and never stashes a drawable, so the last good frame stays on the layer. The onset second is the same both times: ``` 20:45:04 ok=119 dropped=0 21:01:37 ok=46 dropped=3 20:45:05 ok=0 dropped=3 21:01:38 ok=0 dropped=0 ``` `dropped` counts a present whose handler reported `presentedTime == 0` — the system took the drawable and never put it on glass. ## Why it is permanent Stage-4 owns no drawable source of its own. Every drawable arrives as `update.drawable`; it never calls `nextDrawable()` itself. So a link that stops vending is a stream that never presents again. `startDeadlinePresenter` clamps the pool to `maximumDrawableCount = 2`, which leaves exactly one spare slot. Three presents dropped against that pool exhausts it, the link can no longer vend, and no present can complete to free a slot. **This end state is already documented in the same file.** `MetalVideoPresenter.swift`, in the macOS `.transaction` branch, carries the post-mortem: *"drawables are never released; after maximumDrawableCount vends `nextDrawable()` blocks forever and the stream FREEZES"* and *"every present reported presentedTime=0, nothing reached glass."* The structural gap is that **stage-4 has no recovery**. Stage-3 has `PresentGate.staleAfter = 0.1`, described in its own comment as insurance *"rather than freezing the stream"*. Stage-4 returns early before the gate exists and passes `gate: nil`; `link.invalidate()` is only reached at session teardown. Only reconnecting recovers, which is exactly what the log shows. ## What this changes One file, +61/−11. - **Stale-link watchdog.** The render thread records when it last received a drawable. Past 250 ms with frames still decoding and no vend, it retires the link generation and starts a fresh one. Invalidating the old link also returns the drawables it holds, so this covers an exhausted pool as well as a dead link. Only a decoded frame reaches that branch, so a quiet stream never trips it. - **Per-generation stop flag.** `startLink` now takes a `StopFlag` — the type the other session workers already use — so a retired link thread exits without taking the session with it. The session token still stops every generation, so teardown is unchanged. - **Delegate lifetime.** `withExtendedLifetime` now holds the link delegate for the thread's life. The old comment claiming *"this closure is the strong ref"* was wrong: `delegate` is a local, not a capture, and its last use is the weak store, so ARC may release it there. It survives in the shipped build, so it did not cause these freezes — but it is one optimiser decision away from a link that never fires at all. ### Why 250 ms A normal vend wait is bounded by one refresh — 8.3 ms at 120 Hz, 41 ms at a 24 Hz VRR floor. 250 ms is ~30 refreshes at 120 Hz, so it cannot fire on ordinary phase jitter. The cost of a false positive is one relinked frame; the cost of missing it is the whole session. Left as a constant rather than another `PUNKTFUNK_*` lever: the log line prints the measured stall, so if the value is wrong the field will say so and it is a one-line change. ## Verification | Target | Result | | --- | --- | | `arm64-apple-ios17.0`, `PunktfunkKit` | compiles clean | | `arm64-apple-tvos17.0`, `PunktfunkKit` | compiles clean | | macOS, `PunktfunkKit` | compiles clean | | macOS, `swift test` | **384 tests, 6 skipped, 0 failures** | Includes `MetalPresenterTests` and `PresentPacingTests`. The `swift test` run needed `PunktfunkCore.xcframework` rebuilt first: the checked-in bundle's header is stale against the current Rust ABI (it lacks `punktfunk_set_log_callback` and twelve other symbols). That breaks `swift build` on `main` too — pre-existing, and not touched here. ## What is not verified **No automated test exercises the watchdog firing.** The trigger is a display link that ceases to vend, which has no local repro, and the logic sits inside a thread closure holding a real `CAMetalDisplayLink`. Making it testable would mean injecting `startLink` — an abstraction for a single caller. The field log line is the check instead: the relink goes through `ClientLog.error`, which writes to the ring that "Send logs to host" uploads, so the next report will say whether the watchdog fired. **Whether relinking restores vending on a real device is unproven.** If invalidating the link does not release whatever holds the pool, the watchdog will log every 250 ms rather than cure the freeze. That repeat is itself the diagnosis, and it arrives in the uploaded log. ## Deliberate non-change Raising `maximumDrawableCount` back to 3 would widen the margin that ran out here. It would also give back the measured present-floor fix from 2026-08-13 (Apple TV 4K, `os present +32.5` at 60 Hz = 1.95 refresh periods). Left at 2: the watchdog addresses the freeze without trading latency for it. Worth revisiting only if the field shows the watchdog firing often. ## Rejected paths - **A drawable-leak hunt in the present path.** `inflightMax` returns to 0 during the freeze, so every `addPresentedHandler` fired and the presents were accounted for. The logs establish the exhaustion chain but not the initial trigger for the three `presentedTime == 0` returns, so a targeted leak fix would have been a guess. The watchdog is correct for any cause. - **A `PUNKTFUNK_LINK_STALE_MS` lever.** See "Why 250 ms" — the safe window is ~30x wide, so there is nothing to calibrate yet. --- **User-facing fact changed?** n/a — no install step, knob, port, limit or feature behaviour changes. The only new output is one client log line on a path that should never run.

enricobuehler added 1 commit 2026-08-29 08:35:06 +00:00

fix(apple): relink stage-4 when the display link stops vending ... 87bc0883db

Field 2026-08-28, iPad Pro on iOS 27: the picture froze twice in one
session while audio and input kept running, and only reconnecting
cleared it. pf-present logged decode at full rate against ok=0,
noDrawable=120 and no vendLeadMs samples at all, so the
CAMetalDisplayLink had stopped calling back.

Stage-4 has no drawable source of its own: every drawable arrives as
update.drawable. A link that goes quiet is therefore a stream that
never presents again, and nothing noticed. Stage-3's
PresentGate.staleAfter insurance does not reach this path, and the
link is only invalidated at session teardown.

The render thread now tracks how long it has gone without a vend and,
past 250 ms with frames still decoding, retires the link generation
and starts a fresh one. Invalidating the old link also returns the
drawables it holds, so this covers an exhausted pool as well as a
dead link. The relink is logged to the send-logs ring.

Also holds the link delegate for the link thread's lifetime. It is
stored weakly and was kept alive only by a local whose last use is
that store, which ARC is free to release on the spot.

enricobuehler merged commit 21d30b7320 into main 2026-08-29 08:48:28 +00:00

enricobuehler deleted branch worktree-stage4-link-watchdog 2026-08-29 08:48:31 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX FlipperDan HitFrostbite bluelightspecial enricobuehler (Enrico Bühler) kszlim luxus nannku neoney

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#442