docs(design): trim shipped plans, consolidate cluster, add index
Much of design/ described work that has since shipped. Trim each doc to
its durable rationale + still-open items (the code is the source of truth
for shipped detail; git history holds the full originals).
- Shipped plans -> status stubs: stats-capture, gamestream-host-plan,
apple-stage2-presenter, windows-service.
- Trimmed completed-out / open-kept: implementation-plan, hdr-pipeline,
host-latency, gpu-contention (fixed stale status table), game-library,
linux-setup (fixed m0->spike + stale zero-copy claim),
session-aware-host-followups, windows-client-bootstrap,
windows-dualsense-{scoping,game-detection}, windows-virtual-display,
security-review (per-finding status table; #12 still open),
apollo-comparison (shipped backlog collapsed to one-liners).
- Windows-host cluster consolidated: windows-host.md -> redirect into
windows-host-rewrite.md (whose stale scorecard is corrected -- goal1 is
merged, M4 done); windows-secure-desktop.md archived (now a fallback
behind IDD-push primary).
- Kept evergreen: ci.md, gamescope-multiuser.md, windows-build-and-packaging.md.
- New design/README.md: per-doc status table + consolidated open-items
roll-up so nothing is tracked in only one buried doc.
- Repoint 5 code comments to the archived secure-desktop doc path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+106
-244
@@ -1,270 +1,132 @@
|
||||
# HDR pipeline — investigation & implementation plan
|
||||
|
||||
Goal: **true, correct HDR glass-to-glass** for punktfunk, across the host (Windows today;
|
||||
Linux blocked upstream) and every client (Windows / Apple / Android / Linux).
|
||||
> **Status:** Steps 0–3 SHIPPED — protocol/ABI/host in `3526517`, client apply + display
|
||||
> capability-gate in `551012b`. Windows HDR live-validated; Apple/Android CI-compiled (on-glass
|
||||
> pending). Step 4 (Linux) is OPEN, blocked upstream on capture. This doc is trimmed to design
|
||||
> rationale + open items; the shipped code is the source of truth. The original audit (full gap
|
||||
> list, per-file line refs, blocker walkthroughs) is in git history before this trim.
|
||||
|
||||
This is an audit of the current state, the gap list, and a phased plan. It was produced from
|
||||
a full read of every HDR-touching subsystem cross-checked against the HDR10 standards
|
||||
(CICP/H.273 VUI, SMPTE ST.2086 mastering, CEA-861.3 MaxCLL/MaxFALL) and the
|
||||
Sunshine/Apollo/Moonlight reference implementation.
|
||||
Goal: **true, correct HDR glass-to-glass** for punktfunk, across the host (Windows today; Linux
|
||||
blocked upstream) and every client (Windows / Apple / Android / Linux). The plan was produced from a
|
||||
full read of every HDR-touching subsystem cross-checked against the HDR10 standards (CICP/H.273 VUI,
|
||||
SMPTE ST.2086 mastering, CEA-861.3 MaxCLL/MaxFALL) and the Sunshine/Apollo/Moonlight reference.
|
||||
|
||||
> Status legend: **blocker** (HDR can't work) · **correctness** (HDR works but looks wrong) ·
|
||||
> **quality** (correct-ish, missing refinement) · **ok**.
|
||||
The original diagnosis: pixel math + the HEVC VUI we emitted were already correct (self-test
|
||||
validated, matches Apollo), but nothing **measured, signalled, transported, or applied the static HDR
|
||||
metadata** (mastering display colour volume + content light level). The fix was a metadata chain,
|
||||
protocol-first.
|
||||
|
||||
---
|
||||
## What shipped (Steps 0–3)
|
||||
|
||||
## TL;DR
|
||||
- **Step 0 — protocol + ABI carry colour end to end (`3526517`).** `ColorInfo` (4 CICP bytes on
|
||||
`Welcome`) + `HdrMeta` (`0xCE` datagram, bounds-checked); `NativeClient` `color`/`bit_depth` fields
|
||||
+ an `HdrMeta` receiver/demux + `next_hdr_meta`. C ABI: `punktfunk_connect_ex5(... video_caps)`,
|
||||
`next_hdr_meta`, `color_info`, and **fixed `abi.rs:896` `video_caps = 0`** — the one-line root cause
|
||||
that had made Apple's complete (and correct) HDR pipeline dead code. Header regenerated. No
|
||||
rendering changes, CI-testable (round-trip + truncation + SDR back-compat).
|
||||
- **Step 1 — host in-band SEI + complete VUI (`3526517`, live-validated on the RTX box).**
|
||||
Cross-platform byte logic in unit-tested `src/hdr.rs`: `hdr_meta_from_display`,
|
||||
`hevc_mastering_display_sei` SEI **type 137**, `hevc_content_light_level_sei` SEI **type 144**
|
||||
(note: NOT "type 4" — that was a drafting error). Windows `dxgi.rs`/`wgc.rs` read
|
||||
`IDXGIOutput6::GetDesc1` at capture init / output change → `HdrMeta` (MaxCLL/MaxFALL left 0, like
|
||||
Apollo); `nvenc.rs` attaches mastering + CLL SEI on every IDR for HEVC/H.264 and sends the real
|
||||
`0xCE` re-sent each keyframe. In-band SEI is read directly by decoders, so this fixed correctness
|
||||
before clients consumed the protocol and gave an Apollo on-glass parity gate. *Follow-ups:* AV1
|
||||
mastering rides METADATA OBUs (`HDR_MDCV`/`HDR_CLL`), not SEI; the Windows secure-desktop relay
|
||||
still sends only the generic baseline `0xCE` (the helper's in-band SEI carries the real grade).
|
||||
- **Step 2 — clients apply the metadata (`551012b`; Apple/Android CI-compiled).** Each client drains
|
||||
`next_hdr_meta`/`nextHdrMeta` and remaps from the wire form (ST.2086 **G,B,R** order, mastering
|
||||
luminance in 0.0001 cd/m²) to the platform layout: **Windows** `SetHDRMetaData`
|
||||
(`hdr_meta_to_dxgi`: G,B,R→R,G,B reorder, 0.0001-nit→nit), dropping the 1000/1000/400 hardcode;
|
||||
**Apple** `CVBufferSetAttachment` of `kCVImageBufferMasteringDisplayColorVolumeKey` (24-byte BE) +
|
||||
`kCVImageBufferContentLightLevelInfoKey` (4-byte BE) per HDR pixel buffer — the correct path for the
|
||||
`itur_2100_PQ` layer (`CAEDRMetadata` on a PQ layer is ambiguous, deliberately avoided); **Android**
|
||||
`MediaFormat` `KEY_HDR_STATIC_INFO`, a 25-byte CTA-861.3 Type-1 blob (LE, **R,G,B** order, max-lum
|
||||
in **nits-u16**). Apple's connect also flips to `connect_ex5` advertising `videoCap10Bit|videoCapHDR`
|
||||
— the fix that resurrects Apple's previously-dead pipeline.
|
||||
- **Step 3 — display-capability gate (`551012b`).** **Chosen approach: capability-gate, not an
|
||||
in-shader BT.2390 tone-map.** Rationale: with Steps 1–2 the host sends *correct* mastering metadata
|
||||
so an HDR display self-tone-maps; the remaining gap is SDR displays, best fixed by **not advertising
|
||||
HDR you can't present** — the host then sends a proper BT.709 SDR stream instead of PQ the panel
|
||||
mis-tone-maps (washed-out/dark). No guessed curve, deterministic. **Windows**
|
||||
`display_supports_hdr` (any `IDXGIOutput6` colour space == `G2084`); **Apple**
|
||||
`NSScreen.maximumExtendedDynamicRangeColorComponentValue > 1` (macOS) /
|
||||
`UIScreen.main.potentialEDRHeadroom > 1` (iOS); **Android** `Display.getHdrCapabilities`
|
||||
HDR10/HDR10+. Each ANDs with the user's HDR setting before advertising caps and logs when it drops
|
||||
to SDR.
|
||||
|
||||
Our HDR is **correct in isolated islands but broken end-to-end.** The pixel math and the HEVC
|
||||
VUI we *do* emit are right (self-test validated, matches Apollo). What's missing is the
|
||||
**metadata chain**: nothing measures, signals, transports, or applies the *static HDR metadata*
|
||||
(mastering display colour volume + content light level) that tells a display how to tone-map —
|
||||
so every client hardcodes generic values or infers from the bitstream, and one line
|
||||
(`abi.rs:896`, `video_caps = 0`) makes the entire (correct) Apple HDR pipeline dead code.
|
||||
|
||||
---
|
||||
|
||||
## What's already correct (the islands)
|
||||
|
||||
| Stage | Where |
|
||||
|---|---|
|
||||
| Windows host HEVC **VUI** — primaries=9 (BT.2020) / transfer=16 (PQ) / matrix=9 (BT.2020-NCL) / limited range | `encode/nvenc.rs:307-316` |
|
||||
| Windows host **scRGB→BT.2020 PQ** shader (×80 nits → BT.709→2020 → ST.2084 OETF, 10000-nit peak) | `capture/dxgi.rs` — self-test `<1` code error, matches Apollo |
|
||||
| Windows client **P010 decode + YUV→RGB** (BT.2020-NCL, limited→full) + **R10G10B10A2 / G2084-P2020 swapchain** | `present.rs:66-77, 320-370` |
|
||||
| Android client **Main10 decode + reactive DataSpace** (BT2020-PQ/HLG) | `decode.rs:210-227` |
|
||||
| Apple client decode/present **code** (P010 VideoToolbox, BT.2020 PQ Metal, `itur_2100_PQ` + EDR) | correct — but never runs (blocker #2) |
|
||||
|
||||
## Gap list
|
||||
|
||||
### Blockers
|
||||
1. **No color-metadata transport in the protocol** *(the keystone).* The wire carries only
|
||||
`Hello.video_caps` (10BIT/HDR bits) and `Welcome.bit_depth` (8/10) — `quic.rs:127-128`
|
||||
explicitly defers color. No primaries/transfer/matrix/range, **no ST.2086 mastering, no
|
||||
MaxCLL/MaxFALL**. ST.2086/CLL host→client is impossible by construction today.
|
||||
2. **C ABI hardcodes `video_caps = 0`** (`abi.rs:896`) → Apple's complete HDR pipeline is dead
|
||||
code; no ABI embedder can request HDR. One-line root cause.
|
||||
3. **H.264 and AV1 emit zero color signaling on Windows** — the `if self.hdr` VUI block in
|
||||
`nvenc.rs` only writes `hevcConfig`. Any H.264+10-bit or AV1+HDR stream decodes as BT.709 SDR.
|
||||
*(AV1 is **not** a "copy the HEVC VUI" fix — AV1 has no VUI/SEI; it carries
|
||||
primaries/transfer/matrix in the sequence-header `color_config` and mastering/CLL in
|
||||
**METADATA OBUs** `HDR_MDCV`/`HDR_CLL`. Verify whether NVENC's AV1 path accepts them.)*
|
||||
4. **Linux host is 8-bit only end to end** — capture offers only 8-bit PipeWire formats
|
||||
(`capture/linux.rs:443-453, 594-654`; gamescope #2126, portals don't wire PipeWire 1.6
|
||||
BT.2020/PQ); encode downgrades 10-bit (`encode/linux.rs:153-162` TODO, `vaapi.rs:719`) with
|
||||
BT.709 hardcoded. The Windows-style 8-bit→Main10 upconvert shim is not implemented here.
|
||||
5. **Linux client HDR is a complete non-feature** — `video_caps=0`, P010 decode path dead
|
||||
(`video.rs:379`), CICP hardcoded BT.709 (`ui_stream.rs:239-243`), no Wayland
|
||||
color-management (GTK4 0.11 too old).
|
||||
|
||||
### Correctness
|
||||
6. **No host ever emits the ST.2086 mastering or CEA-861.3 CLL SEI.** Windows never reads
|
||||
`IDXGIOutput6::GetDesc1`; `nvenc.rs` never builds an `NV_ENC_SEI_PAYLOAD`; Linux attaches no
|
||||
libavcodec `side_data`. Apollo reads `GetDesc1` and attaches it.
|
||||
7. **Clients hardcode mastering metadata.** `present.rs:584-595` ships fixed
|
||||
`1000-nit / MaxCLL 1000 / MaxFALL 400` (with the literal "the protocol doesn't carry the
|
||||
stream's real mastering metadata yet" comment). Apple/Android set none.
|
||||
8. **HDR→SDR tone-mapping is unaddressed — and it's the common case.** Most client displays are
|
||||
SDR. No client queries display peak; silent `SetColorSpace1`/`SetHDRMetaData` failures present
|
||||
PQ as SDR gamma (crushed/dark). We lean entirely on OS auto-fallback.
|
||||
9. **Windows secure desktop drops HDR to SDR** on lock/UAC (`dxgi.rs:325-368`,
|
||||
`sudovda.rs:234-277`).
|
||||
10. **GameStream silently streams SDR** on a Moonlight HDR request (`mod.rs:48-56`,
|
||||
`rtsp.rs:288-293`) — logged, but no negotiated error. Real Apollo parity needs the Moonlight
|
||||
`SS_HDR_METADATA` blob on the **ENet control channel**, not just in-band.
|
||||
11. **Linux client software path is color-wrong even for SDR** — BT.601 applied to BT.709
|
||||
(`video.rs:162-167`, no `color_state` on the texture). Standalone bug.
|
||||
|
||||
### Quality
|
||||
12. No per-content MaxCLL/MaxFALL (`GetDesc1` doesn't expose it). No encoder-CSC-range vs
|
||||
signaled-range reconciliation (black-crush risk). No automated 10-bit test — `probe` never
|
||||
even reads `Welcome.bit_depth` (`main.rs:396-406`).
|
||||
|
||||
### Out of scope (call out, don't build)
|
||||
- Dynamic metadata: HDR10+ (ST.2094-40) and Dolby Vision RPU. We handle *static* ST.2086 only,
|
||||
with mid-stream changes carried by re-sending the static block (below).
|
||||
- HLG: the colorimetry transfer enum carries `18` from day one (free), but the `0xCE` mastering
|
||||
datagram is **omitted for HLG** (scene-referred, no mastering metadata).
|
||||
|
||||
---
|
||||
|
||||
## Protocol design (the keystone — pure-additive, hardware-free, CI-testable)
|
||||
### Wire format — design decisions worth keeping
|
||||
|
||||
Two layers, both back-compat-safe via the established trailing-bytes / new-datagram-tag patterns.
|
||||
|
||||
### (A) Per-session colorimetry — 4 trailing bytes on `Welcome`
|
||||
- **(A) Per-session colorimetry** — 4 trailing bytes on `Welcome` (offsets 60..64):
|
||||
`colour_primaries` (1=BT.709, 9=BT.2020) · `transfer_characteristics` (1=BT.709, 16=PQ/SMPTE2084,
|
||||
18=HLG) · `matrix_coeffs` (1=BT.709, 9=BT.2020-NCL — **never emit 10 (CL): no client decodes it**) ·
|
||||
`video_full_range_flag`. Decoded with `b.get(60).unwrap_or(1)` so an older host that omits them →
|
||||
BT.709 limited SDR (today's behaviour). A future mirror on `Reconfigured` announces a mid-stream
|
||||
SDR↔HDR / BT.709↔BT.2020 flip (deferred; today a mode switch never changes colour, and `0xCE`
|
||||
re-send covers mastering changes).
|
||||
- **(B) Per-change mastering + CLL** — host→client datagram tag **`0xCE`**, 28 bytes, standard SEI
|
||||
fixed-point (display primaries G,B,R + white point in 1/50000 units; max/min mastering luminance in
|
||||
0.0001 cd/m²; MaxCLL/MaxFALL in nits). ST.2086 is variable, so it rides a datagram rather than the
|
||||
Welcome. **Re-sent on every IDR/RFI keyframe** so a client that dropped the best-effort datagram
|
||||
converges within a GOP; until first receipt the client uses the Welcome transfer + a documented
|
||||
generic default. **Bounds-check length before reading** (reassembler-bounds security invariant —
|
||||
truncation test required). **Omitted entirely for HLG.** Units map straight to DXGI
|
||||
`DXGI_HDR_METADATA_HDR10`, Android `KEY_HDR_STATIC_INFO`, Apple `CAEDRMetadata.hdr10`; the
|
||||
libavcodec/Linux side needs conversion — `AVMasteringDisplayMetadata` stores `AVRational`, not raw
|
||||
fixed-point.
|
||||
|
||||
After the existing `bit_depth` (offset 59), append a fixed 4-byte CICP block at offsets 60..64.
|
||||
(A future mirror on `Reconfigured` will announce a mid-stream SDR↔HDR / BT.709↔BT.2020 flip on the
|
||||
control stream we already use for renegotiation — deferred to Step 1 with the mid-stream-flip work;
|
||||
today a mode switch never changes the colour, and the `0xCE` re-send covers mastering changes.)
|
||||
## Out of scope (accepted — call out, don't build)
|
||||
|
||||
```
|
||||
[60] colour_primaries (CICP: 1=BT.709, 9=BT.2020)
|
||||
[61] transfer_characteristics (1=BT.709, 16=PQ/SMPTE2084, 18=HLG)
|
||||
[62] matrix_coeffs (1=BT.709, 9=BT.2020-NCL) ← never emit 10 (CL): no client decodes it
|
||||
[63] video_full_range_flag (0=limited, 1=full)
|
||||
```
|
||||
- **Dynamic metadata:** HDR10+ (ST.2094-40) and Dolby Vision RPU. We handle *static* ST.2086 only,
|
||||
with mid-stream changes carried by re-sending the static block.
|
||||
- **HLG:** the transfer enum carries `18` from day one (free), but the `0xCE` mastering datagram is
|
||||
omitted for HLG (scene-referred, no mastering metadata).
|
||||
|
||||
Decode with `b.get(60).unwrap_or(1)` etc. — an older host omits them → BT.709 limited SDR
|
||||
(today's behavior). `Welcome` stays `Copy`. Modeled as a `ColorInfo` struct on the wire types
|
||||
and exposed on `NativeClient` (with `bit_depth`) so clients *know* the colorimetry instead of
|
||||
inferring it.
|
||||
## Step 4 — Linux (last; capture blocked upstream) — OPEN
|
||||
|
||||
### (B) Per-change mastering + CLL — a new host→client datagram, tag `0xCE`
|
||||
|
||||
ST.2086 is variable and changes mid-stream, so it rides a datagram (next tag after `0xCD`
|
||||
HIDOUT), demuxed in `client.rs` like AUDIO/RUMBLE/HIDOUT. 28 bytes, standard SEI fixed-point:
|
||||
|
||||
```
|
||||
[0] = 0xCE
|
||||
G.x G.y B.x B.y R.x R.y 6 × u16 LE display primaries, 1/50000 units
|
||||
wp.x wp.y 2 × u16 LE white point, 1/50000 units
|
||||
max_display_mastering_luminance u32 LE 0.0001 cd/m²
|
||||
min_display_mastering_luminance u32 LE 0.0001 cd/m²
|
||||
max_cll u16 LE nits
|
||||
max_fall u16 LE nits
|
||||
```
|
||||
|
||||
- Sent on session start and whenever `GetDesc1`/source mastering changes; **re-sent on every
|
||||
IDR/RFI keyframe** so a client that dropped the (best-effort) datagram converges within a GOP.
|
||||
Until first receipt the client uses the Welcome transfer + a documented generic default.
|
||||
- **Bounds-check length before reading** (reassembler-bounds security invariant) — truncation
|
||||
test required.
|
||||
- **Omitted entirely for HLG.**
|
||||
- Units note: these map straight to DXGI `DXGI_HDR_METADATA_HDR10`, Android `KEY_HDR_STATIC_INFO`,
|
||||
and Apple `CAEDRMetadata.hdr10`. On the **libavcodec/Linux** side they need conversion —
|
||||
`AVMasteringDisplayMetadata` stores `AVRational`, not raw fixed-point.
|
||||
|
||||
### (C) C ABI
|
||||
|
||||
- `punktfunk_connect_ex5(... video_caps: u8)` (ex4 delegates with 0); **fix `abi.rs:896`.**
|
||||
- `punktfunk_connection_next_hdr_meta(c, *mut PunktfunkHdrMeta, timeout_ms)` — new plane,
|
||||
one-puller contract like `next_audio`.
|
||||
- `punktfunk_connection_color_info(c, *mut prim, *mut trc, *mut matrix, *mut range, *mut bit_depth)`.
|
||||
- Regenerate `include/punktfunk_core.h` (cbindgen); `struct_size`/repr(C) guards on new structs.
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Step 0 — Protocol + ABI carry color metadata end to end *(this change)*
|
||||
The dominant cross-cutting blocker; everything else is downstream. No rendering changes, no
|
||||
hardware, CI-testable.
|
||||
|
||||
- **core:** `ColorInfo` + 4 Welcome bytes; `HdrMeta` + `0xCE` codec (bounds-checked);
|
||||
`NativeClient` `color`/`bit_depth` fields + HdrMeta receiver + demux + `next_hdr_meta`.
|
||||
- **C ABI:** `connect_ex5`, `next_hdr_meta`, `color_info`, fix caps=0; regen header.
|
||||
- **host:** populate `Welcome.color` from the negotiated bit-depth/HDR decision; send a `0xCE`
|
||||
(generic default for now) when HDR is negotiated.
|
||||
- **clients:** Windows/Android inherit the demux via shared core; Apple flips to `ex5`.
|
||||
- **validation:** `quic.rs` round-trip + truncation + **SDR back-compat** tests; `probe` logs
|
||||
`bit_depth` + colorimetry; loopback asserts a 10-bit Welcome carries trc=16 and a `0xCE` lands.
|
||||
|
||||
### Step 1 — Host emits correct in-band SEI + complete VUI on all codecs *(landed; RTX-validation pending)*
|
||||
In-band SEI is read directly by decoders, so it fixes correctness even before clients consume
|
||||
the protocol, and gives an Apollo/Moonlight on-glass parity gate.
|
||||
|
||||
- **Single source of truth:** the capturer learns the source display's mastering metadata and
|
||||
exposes it via `Capturer::hdr_meta() -> Option<HdrMeta>`. The stream loop forwards it to the
|
||||
encoder (`Encoder::set_hdr_meta` → in-band SEI) **and** the client (real `0xCE`, re-sent on each
|
||||
keyframe). Pure byte-level logic (float→fixed conversion + the HEVC/H.264 SEI payloads) lives in
|
||||
the unit-tested, cross-platform `src/hdr.rs` (`hdr_meta_from_display`, `hevc_mastering_display_sei`
|
||||
type **137**, `hevc_content_light_level_sei` type **144** — note: NOT "type 4", that was a
|
||||
drafting error).
|
||||
- **Windows (done, CI-compiled / RTX on-glass pending):** `dxgi.rs` + `wgc.rs` read
|
||||
`IDXGIOutput6::GetDesc1` at capture init / output change → `HdrMeta` (MaxCLL/MaxFALL left 0 —
|
||||
GetDesc1 has none, like Apollo). `nvenc.rs` attaches the mastering + CLL SEI on every IDR for
|
||||
HEVC/H.264. (AV1 mastering rides METADATA OBUs, not SEI — follow-up; AV1 `color_config` already
|
||||
lands in Step 0's quick win.)
|
||||
- **Linux encode-ready — DEFERRED into Step 4:** Linux capture is 8-bit only, so signalling
|
||||
BT.2020 PQ + attaching mastering side-data on a downconverted 8-bit stream would be *incorrect*.
|
||||
The libavcodec `side_data` path (with the `AVRational` conversion) lands together with the
|
||||
8-bit→Main10 shim / true 10-bit capture in Step 4.
|
||||
- **Windows secure-desktop relay** (`virtual_stream_relay`) still sends only the generic baseline
|
||||
`0xCE`; the helper's in-band SEI carries the real grade. Wiring the relay's `0xCE` is a follow-up.
|
||||
- **validation (RTX box):** `ffprobe -show_frames` shows mastering + CLL side-data with the
|
||||
display's real luminance and VUI 9/16/9; stock Moonlight shows correct (not washed-out) HDR.
|
||||
Add **encoder-CSC-range == signaled-range** check.
|
||||
|
||||
### Step 2 — Clients apply the metadata *(landed; CI/on-glass validation pending)*
|
||||
All three clients now drain the protocol's `HdrMeta` (`next_hdr_meta` / `nextHdrMeta`) and apply it,
|
||||
each remapping from the wire form (ST.2086 G,B,R order, mastering luminance in 0.0001 cd/m²) to the
|
||||
platform's expected layout:
|
||||
- **Windows (Rust, CI-compiled):** session pump drains `next_hdr_meta` into a `LATEST_HDR_META`
|
||||
slot; `present_newest` applies it via `Presenter::set_hdr_metadata` → real `SetHDRMetaData`
|
||||
(`hdr_meta_to_dxgi`: G,B,R→R,G,B reorder, 0.0001-nit→nit for `MaxMasteringLuminance`), dropping
|
||||
the 1000/1000/400 hardcode. `SetColorSpace1`/`SetHDRMetaData` failures + an SDR-display
|
||||
colour-space rejection are now **logged**, not swallowed.
|
||||
- **Apple (Swift, mac-runner CI):** connect now advertises caps via `punktfunk_connect_ex5`
|
||||
(`SessionModel` computes `videoCap10Bit|videoCapHDR` from `hdrEnabled`) — *this is the fix that
|
||||
resurrects Apple's previously-dead HDR pipeline*. `nextHdrMeta`/`colorInfo` wrappers added; the
|
||||
pump drains `nextHdrMeta` → `VideoDecoder.setHdrMeta` → `CVBufferSetAttachment` of
|
||||
`kCVImageBufferMasteringDisplayColorVolumeKey` (24-byte BE SEI) +
|
||||
`kCVImageBufferContentLightLevelInfoKey` (4-byte BE) on each HDR pixel buffer (the correct path
|
||||
for the itur_2100_PQ layer; `CAEDRMetadata` on a PQ layer is ambiguous and was avoided).
|
||||
- **Android (Rust `decode.rs`, cargo-ndk verified):** when `client.color.is_hdr()`, drain the first
|
||||
`next_hdr_meta` and set `MediaFormat` `hdr-static-info` (`KEY_HDR_STATIC_INFO`) before
|
||||
`configure()` — `android_hdr_static_info` builds the 25-byte CTA-861.3 Type-1 blob (LE, **R,G,B**
|
||||
order, max-lum in **nits-u16**). `Display.getHdrCapabilities` gate deferred (the Surface DataSpace
|
||||
already drives SurfaceFlinger tone-mapping on non-HDR displays).
|
||||
|
||||
### Step 3 — Display-capability gate *(landed; CI/on-glass validation pending)*
|
||||
The common-case correctness step — most client displays are SDR. **Chosen approach: capability-gate**
|
||||
(not an in-shader BT.2390 tone-map). Rationale: with Steps 1–2 the host sends *correct* mastering
|
||||
metadata, so an HDR display self-tone-maps from it; the real remaining gap is SDR displays, best
|
||||
fixed by **not advertising HDR you can't present** — the host then sends a proper BT.709 SDR stream
|
||||
instead of PQ the panel would mis-tone-map (washed-out/dark). No guessed tone-map curve, deterministic.
|
||||
- **Windows** (`present::display_supports_hdr` via DXGI: any `IDXGIOutput6` colour space ==
|
||||
`G2084`): `session.rs` ANDs it with the user's HDR setting before advertising caps; logs when it
|
||||
drops to SDR.
|
||||
- **Apple** (`SessionModel`, main-actor): `NSScreen.maximumExtendedDynamicRangeColorComponentValue
|
||||
> 1` (macOS) / `UIScreen.main.potentialEDRHeadroom > 1` (iOS) ANDed with `hdrEnabled`.
|
||||
- **Android** (`Settings.displaySupportsHdr` via `Display.getHdrCapabilities` HDR10/HDR10+): Kotlin
|
||||
passes it to `nativeConnect`; `session.rs` gates the caps on the new `hdr_enabled` jboolean
|
||||
(cargo-ndk-verified).
|
||||
- **Deferred** (need on-glass / the RTX box): the **mid-session `Reconfigure` "downgrade to SDR"**
|
||||
for a monitor move HDR↔SDR; and confirming the **host produces SDR for an SDR client even off an
|
||||
HDR desktop** — on the native path the per-session SudoVDA follows the negotiated depth (SDR
|
||||
client → SDR virtual display → SDR stream), so it should hold end-to-end; verify the
|
||||
stale-HDR-SudoVDA edge case on the RTX box.
|
||||
|
||||
### Step 4 — Linux (last; capture blocked upstream)
|
||||
- **8-bit→Main10 NVENC upconvert shim** (`encode/linux.rs`) — Main10 transport with correct
|
||||
VUI/SEI without HDR capture (gate so we don't claim HDR transfer on SDR content).
|
||||
- **Linux encode color + side-data (the deferred Step 1c):** set
|
||||
- **(4a) 8-bit→Main10 NVENC upconvert shim** (`encode/linux.rs`) — Main10 transport with correct
|
||||
VUI/SEI without HDR capture (gated so we don't claim HDR transfer on SDR content).
|
||||
- **Linux encode colour + side-data (the deferred Step 1c):** set
|
||||
`color_primaries/trc/colorspace/range` from the negotiated `ColorInfo` and attach
|
||||
`AV_FRAME_DATA_MASTERING_DISPLAY_METADATA` / `CONTENT_LIGHT_LEVEL` side-data (with the
|
||||
`AVRational` conversion) in `encode/linux.rs` + `vaapi.rs` — only once the encoder actually
|
||||
produces 10-bit, so the signalling matches the bits.
|
||||
- True 10-bit capture: offer `ABGR2101010`/`P010` PipeWire formats + read colorimetry; pilot on
|
||||
Sway/wlroots; track gamescope #2126. **Don't block the rest of the plan on it.**
|
||||
- Linux client: `ex5` caps, P010 decode, GdkDmabufTexture CICP from Welcome,
|
||||
`wp_color_management` when GTK ≥ 4.14.
|
||||
`AV_FRAME_DATA_MASTERING_DISPLAY_METADATA` / `CONTENT_LIGHT_LEVEL` side-data (with the `AVRational`
|
||||
conversion) in `encode/linux.rs` + `vaapi.rs` — only once the encoder actually produces 10-bit, so
|
||||
the signalling matches the bits. (Linux capture is 8-bit only, so signalling BT.2020 PQ + attaching
|
||||
mastering side-data on a downconverted 8-bit stream would be *incorrect* — hence deferred out of
|
||||
Step 1.)
|
||||
- **(4b) True 10-bit capture:** offer `ABGR2101010`/`P010` PipeWire formats + read colorimetry; pilot
|
||||
on Sway/wlroots; **blocked on gamescope #2126** (portals don't wire PipeWire 1.6 BT.2020/PQ).
|
||||
**Don't block the rest of the plan on it.**
|
||||
- **(4c) Linux client:** `ex5` caps, P010 decode, GdkDmabufTexture CICP from Welcome,
|
||||
`wp_color_management` when GTK ≥ 4.14. (Also a standalone SDR bug: software path applies BT.601 to
|
||||
BT.709 — needs a BT.601→BT.709 sws + texture `color_state`.)
|
||||
|
||||
## Quick wins (independent, land in parallel)
|
||||
1. `connect_ex5` + fix `abi.rs:896` — resurrects Apple's pipeline *(Step 0)*.
|
||||
2. H.264 VUI + AV1 `color_config` on `nvenc.rs` — closes two latent blockers *(Windows-only,
|
||||
validated in CI / on the RTX box)*.
|
||||
3. `probe` logs `bit_depth` + colorimetry — observability for every later round-trip assertion.
|
||||
4. Linux client BT.601→BT.709 sws + texture `color_state` — standalone SDR correctness bug.
|
||||
5. GameStream silent-downgrade already warns (`rtsp.rs:289`) — keep observable.
|
||||
## Deferred validation (need on-glass / the RTX box)
|
||||
|
||||
- The mid-session `Reconfigure` "downgrade to SDR" for a monitor move HDR↔SDR.
|
||||
- Confirm the **host produces SDR for an SDR client even off an HDR desktop** — on the native path the
|
||||
per-session SudoVDA follows the negotiated depth (SDR client → SDR virtual display → SDR stream), so
|
||||
it should hold end to end; verify the stale-HDR-SudoVDA edge case.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **MaxCLL source:** `GetDesc1` doesn't expose it (Apollo zeroes). Static default, or measure
|
||||
per-frame peak in the PQ shader (only truly-correct, adds a readback)?
|
||||
- **GameStream:** implement `SS_HDR_METADATA` for Moonlight parity, or keep it deliberately SDR
|
||||
and steer HDR users to punktfunk/1?
|
||||
- **GameStream:** implement `SS_HDR_METADATA` (Moonlight `SS_HDR_METADATA` blob on the ENet control
|
||||
channel) for parity, or keep it deliberately SDR and steer HDR users to punktfunk/1?
|
||||
- **HLG:** carry the enum from day one (free) — but do any sources actually produce HLG?
|
||||
- **Linux:** is shipping the 8-bit→Main10 shim as "HDR-capable transport" acceptable, or does it
|
||||
risk advertising HDR we can't truly deliver?
|
||||
- **Linux:** is shipping the 8-bit→Main10 shim as "HDR-capable transport" acceptable, or does it risk
|
||||
advertising HDR we can't truly deliver?
|
||||
|
||||
## Ordering rationale
|
||||
Step 0 first: it's the keystone (metadata transport is the dominant cross-cutter; the ABI line
|
||||
is a one-line root cause) and needs no hardware. Step 1 next: in-band SEI is read directly by
|
||||
decoders, so it fixes correctness even before our clients consume the protocol, and gives an
|
||||
Apollo-parity on-glass gate. Steps 2–3 are mechanical per-client wiring once metadata flows.
|
||||
|
||||
Step 0 first: it's the keystone (metadata transport is the dominant cross-cutter; the ABI
|
||||
`video_caps = 0` line is a one-line root cause) and needs no hardware. Step 1 next: in-band SEI is
|
||||
read directly by decoders, so it fixes correctness even before our clients consume the protocol, and
|
||||
gives an Apollo-parity on-glass gate. Steps 2–3 are mechanical per-client wiring once metadata flows.
|
||||
Linux is last because capture is gated on upstream we don't control; the shim delivers Main10
|
||||
transport without that dependency.
|
||||
|
||||
Hardware dependencies: Step 0 = none (CI); Step 1 = RTX Windows host; Steps 2–3 = a real HDR
|
||||
display per platform; Step 4 = a Linux GPU box + HDR-capable Wayland compositor.
|
||||
Hardware dependencies: Step 0 = none (CI); Step 1 = RTX Windows host; Steps 2–3 = a real HDR display
|
||||
per platform; Step 4 = a Linux GPU box + HDR-capable Wayland compositor.
|
||||
|
||||
Reference in New Issue
Block a user