unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 23 Wiki Activity

fix(video): honor the signaled CSC matrix end-to-end + tvOS HDR presentation

Browse Source 
Clients derive Y'CbCr->RGB from the stream's SIGNALED matrix x range x depth
via shared csc rows (Rust csc_rows + Swift CscRows) instead of hardcoded
709/2020 - a BT.601-signaled stream (a Linux host's RGB-input NVENC) no longer
renders with a constant hue error. Host-side signaling made honest across
NVENC/VAAPI/openh264/GameStream and the session plan's chroma/bit-depth.
Decoded color-bar fixtures (601/709 x limited/full) pin the math in tests on
both cores.

Same presenter, tvOS HDR: tvOS has no Metal EDR API and a bare PQ colorspace
tag composites UNTONE-MAPPED (the "overblown" Apple TV report), so HDR now
splits on the display's live EDR headroom - PQ passthrough when the
per-session AVDisplayManager mode switch landed (a real HDR10 output
tone-maps itself), else an in-shader PQ->SDR tone-map (203-nit reference
white, extended-Reinhard 1000-nit knee, 2020->709) into the proven SDR layer
config. The 10-bit stream keeps its full decode depth either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Enrico Bühler
2026-07-10 16:58:11 +02:00
parent db49904c6d
commit 1fcf9e11ec
26 changed files with 2268 additions and 409 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/punktfunk-host/src/encode/sw.rs
+114 -44
View File
@@ -1,7 +1,13 @@
//! Software H.264 encoder (openh264) — the GPU-less encode path for the Windows host (and a
//! fallback when NVENC is unavailable). Low-latency screen-content config: single-reference,
//! no B-frames (Baseline), bitrate rate-control, in-band SPS/PPS each IDR, BT.709 limited range.
//! no B-frames (Baseline), bitrate rate-control, in-band SPS/PPS each IDR.
//! Synchronous: `submit` encodes immediately and stashes the AU for `poll` (no internal queue).
//!
//! The RGB→YUV conversion is OURS, BT.709 limited range: openh264 writes no colour description
//! into the VUI (unspecified), so decoders fall back to their default — BT.709 limited on every
//! punktfunk client — and the pixels must match that default. The crate's own `YUVBuffer`
//! converter is BT.601 (0.2578/0.5039/0.0977 + 16), which decoded-as-709 is a constant hue
//! error; that's why it is NOT used here.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
@@ -12,19 +18,20 @@ use openh264::encoder::{
BitRate, Complexity, Encoder as Oh264, EncoderConfig, FrameRate, FrameType, IntraFramePeriod,
Profile, RateControlMode, SpsPpsStrategy, UsageType,
};
use openh264::formats::{BgraSliceU8, RgbSliceU8, YUVBuffer};
use openh264::formats::YUVSlices;
use openh264::OpenH264API;
pub struct OpenH264Encoder {
enc: Oh264,
yuv: YUVBuffer,
width: u32,
height: u32,
fps: u32,
src_format: PixelFormat,
/// BGRA scratch for the 3-bpp (Bgr) and R/B-swapped (Rgba/Rgbx) formats openh264 can't wrap
/// directly. Reused across frames.
scratch: Vec<u8>,
/// The converted I420 planes (our BT.709-limited CSC — see the module doc), reused across
/// frames: full-res luma + quarter-res Cb/Cr, tightly packed (stride = width, width/2).
y_plane: Vec<u8>,
u_plane: Vec<u8>,
v_plane: Vec<u8>,
frame_idx: i64,
force_kf: bool,
/// At most one AU per submit (no lookahead), handed back by the next `poll`.
@@ -33,7 +40,7 @@ pub struct OpenH264Encoder {
// openh264's Encoder holds a raw C handle (not auto-Send); it lives on the single encode thread.
// SAFETY: `OpenH264Encoder` wraps `Oh264` (openh264's `Encoder`), which holds a raw C handle to the
// openh264 `ISVCEncoder` and is not auto-`Send`; the other fields (`YUVBuffer`, `Vec`, scalars,
// openh264 `ISVCEncoder` and is not auto-`Send`; the other fields (the plane `Vec`s, scalars,
// `Option<EncodedFrame>`) are plain owned data. The session creates the encoder, calls
// `submit`/`poll`/`flush`, and drops it all on one dedicated encode thread, never sharing it by
// reference across threads, so the C handle is only ever touched from a single thread. Moving the
@@ -62,51 +69,75 @@ impl OpenH264Encoder {
.scene_change_detect(false) // no surprise IDRs (bitrate spikes / freeze)
.adaptive_quantization(true)
.complexity(Complexity::Low) // latency over BD-rate
.profile(Profile::Baseline); // no B-frames; BT.709 limited is the crate default VUI
.profile(Profile::Baseline); // no B-frames; the VUI carries no colour description
let api = OpenH264API::from_source(); // statically-bundled build (default `source` feature)
let enc = Oh264::with_api_config(api, cfg).context("openh264 Encoder::with_api_config")?;
let yuv = YUVBuffer::new(width as usize, height as usize);
let (w, h) = (width as usize, height as usize);
tracing::info!(
"openh264 software encoder: {width}x{height}@{fps} {} Mbps (Baseline, screen-content)",
bps / 1_000_000
);
Ok(Self {
enc,
yuv,
width,
height,
fps,
src_format: format,
scratch: Vec::new(),
y_plane: vec![0; w * h],
u_plane: vec![0; (w / 2) * (h / 2)],
v_plane: vec![0; (w / 2) * (h / 2)],
frame_idx: 0,
force_kf: false,
pending: None,
})
}
/// Normalize a packed source buffer into the reused BGRA `scratch` ([B,G,R,A]). `rgb_order`
/// = source is R,G,B (swap into B,G,R); otherwise source is already B,G,R.
fn normalize_to_bgra(&mut self, src: &[u8], src_bpp: usize, rgb_order: bool) {
/// Convert one packed full-range RGB frame into the I420 planes, BT.709 limited range.
/// `bpp` is the source pixel stride; `ri`/`gi`/`bi` the channel byte offsets within a pixel.
/// Luma per pixel; Cb/Cr from the 2×2 block's averaged RGB (the same box filter the crate's
/// converter used, so only the matrix changed).
fn convert_bt709(&mut self, src: &[u8], bpp: usize, ri: usize, gi: usize, bi: usize) {
let w = self.width as usize;
let h = self.height as usize;
self.scratch.resize(w * h * 4, 0);
for px in 0..(w * h) {
let s = &src[px * src_bpp..px * src_bpp + 3];
let d = &mut self.scratch[px * 4..px * 4 + 4];
if rgb_order {
d[0] = s[2];
d[1] = s[1];
d[2] = s[0];
} else {
d[0] = s[0];
d[1] = s[1];
d[2] = s[2];
let cw = w / 2;
for by in 0..h / 2 {
for bx in 0..cw {
let mut sum = (0f32, 0f32, 0f32);
for (dy, dx) in [(0, 0), (0, 1), (1, 0), (1, 1)] {
let (px, py) = (bx * 2 + dx, by * 2 + dy);
let s = &src[(py * w + px) * bpp..];
let (r, g, b) = (f32::from(s[ri]), f32::from(s[gi]), f32::from(s[bi]));
self.y_plane[py * w + px] = luma709(r, g, b);
sum = (sum.0 + r, sum.1 + g, sum.2 + b);
}
let (cb, cr) = chroma709(sum.0 / 4.0, sum.1 / 4.0, sum.2 / 4.0);
self.u_plane[by * cw + bx] = cb;
self.v_plane[by * cw + bx] = cr;
}
d[3] = 0xff;
}
}
}
/// BT.709 luma coefficients (Kg = 1 − Kr − Kb).
const KR: f32 = 0.2126;
const KB: f32 = 0.0722;
const KG: f32 = 1.0 - KR - KB;
/// One full-range RGB pixel (0..=255 channels) → the BT.709 limited-range 8-bit luma code
/// (16..=235). Kept in lockstep with the client-side inverse (`pf-client-core::video::csc_rows`).
fn luma709(r: f32, g: f32, b: f32) -> u8 {
let y = KR * r + KG * g + KB * b; // full-scale luma, 0..=255
(16.0 + y * (219.0 / 255.0) + 0.5) as u8 // `as` saturates — no manual clamp needed
}
/// (Averaged) full-range RGB → the BT.709 limited-range Cb/Cr codes (16..=240, neutral 128).
fn chroma709(r: f32, g: f32, b: f32) -> (u8, u8) {
let y = KR * r + KG * g + KB * b;
let cb = 128.0 + (b - y) * (224.0 / 255.0) / (2.0 * (1.0 - KB));
let cr = 128.0 + (r - y) * (224.0 / 255.0) / (2.0 * (1.0 - KR));
((cb + 0.5) as u8, (cr + 0.5) as u8)
}
impl Encoder for OpenH264Encoder {
fn submit(&mut self, captured: &CapturedFrame) -> Result<()> {
ensure!(
@@ -139,21 +170,13 @@ impl Encoder for OpenH264Encoder {
self.src_format
);
match self.src_format {
PixelFormat::Rgb => self
.yuv
.read_rgb(RgbSliceU8::new(&bytes[..w * h * 3], (w, h))),
PixelFormat::Bgra | PixelFormat::Bgrx => self
.yuv
.read_rgb(BgraSliceU8::new(&bytes[..w * h * 4], (w, h))),
PixelFormat::Rgba | PixelFormat::Rgbx => {
self.normalize_to_bgra(bytes, 4, true);
self.yuv.read_rgb(BgraSliceU8::new(&self.scratch, (w, h)));
}
PixelFormat::Bgr => {
self.normalize_to_bgra(bytes, 3, false);
self.yuv.read_rgb(BgraSliceU8::new(&self.scratch, (w, h)));
}
// Source pixel stride + R/G/B byte offsets within a pixel — one converter for every
// packed-RGB layout the capturers emit (no BGRA normalization pass needed).
let (bpp, ri, gi, bi) = match self.src_format {
PixelFormat::Rgb => (3, 0, 1, 2),
PixelFormat::Bgr => (3, 2, 1, 0),
PixelFormat::Rgba | PixelFormat::Rgbx => (4, 0, 1, 2),
PixelFormat::Bgra | PixelFormat::Bgrx => (4, 2, 1, 0),
// 10-bit HDR comes only from the GPU NVENC path; the software 8-bit H.264 encoder
// can't represent it (and never receives it — the capturer pairs Rgb10a2 with NVENC).
PixelFormat::Rgb10a2 => {
@@ -166,13 +189,19 @@ impl Encoder for OpenH264Encoder {
"software encoder cannot encode YUV GPU textures (NV12/P010 → NVENC only)"
)
}
}
};
self.convert_bt709(bytes, bpp, ri, gi, bi);
if self.force_kf {
self.enc.force_intra_frame();
self.force_kf = false;
}
let bs = self.enc.encode(&self.yuv).context("openh264 encode")?;
let slices = YUVSlices::new(
(&self.y_plane, &self.u_plane, &self.v_plane),
(w, h),
(w, w / 2, w / 2),
);
let bs = self.enc.encode(&slices).context("openh264 encode")?;
let mut data = Vec::new();
bs.write_vec(&mut data); // AnnexB start codes; SPS/PPS prepended on IDR
if !data.is_empty() {
@@ -225,6 +254,47 @@ mod tests {
use super::*;
use crate::capture::{CapturedFrame, FramePayload, PixelFormat};
/// The BT.709 limited-range anchor points: reference white → (235,128,128), black →
/// (16,128,128), pure red's Cr must hit the positive extreme 240 (it does exactly:
/// 255(1−Kr)·(224/255)/(2(1−Kr)) = 112). ±1 code for float rounding.
#[test]
fn bt709_conversion_anchor_points() {
assert_eq!(luma709(255.0, 255.0, 255.0), 235);
assert_eq!(luma709(0.0, 0.0, 0.0), 16);
assert_eq!(chroma709(255.0, 255.0, 255.0), (128, 128));
assert_eq!(chroma709(0.0, 0.0, 0.0), (128, 128));
let (cb, cr) = chroma709(255.0, 0.0, 0.0);
assert_eq!(cr, 240, "pure red must reach the Cr extreme");
assert!((101..=103).contains(&cb), "red Cb ~102, got {cb}");
let (cb, _) = chroma709(0.0, 0.0, 255.0);
assert_eq!(cb, 240, "pure blue must reach the Cb extreme");
}
/// The 601-vs-709 luma split on pure green (Kg 0.587 vs 0.7152) — guards against anyone
/// "simplifying" the coefficients back to the crate's BT.601 converter (the hue-shift bug
/// this module's own conversion exists to prevent).
#[test]
fn bt709_is_not_bt601() {
// BT.601 green luma: 16 + 219·0.587 = 144.5; BT.709: 16 + 219·0.7152 = 172.6.
let y = luma709(0.0, 255.0, 0.0);
assert!((172..=174).contains(&y), "709 green luma ~173, got {y}");
}
/// A flat gray frame converts to neutral chroma and mid luma across every plane byte
/// (exercises the block loop + plane sizing, not just the per-pixel math).
#[test]
fn converts_flat_gray_to_neutral_planes() {
let (w, h) = (16u32, 8u32);
let mut enc =
OpenH264Encoder::open(PixelFormat::Bgrx, w, h, 60, 1_000_000).expect("open openh264");
let bytes = vec![0x80u8; (w * h * 4) as usize];
enc.convert_bt709(&bytes, 4, 2, 1, 0);
// 16 + 128·(219/255) = 125.9 → 126.
assert!(enc.y_plane.iter().all(|&y| y == 126), "{:?}", &enc.y_plane[..4]);
assert!(enc.u_plane.iter().all(|&u| u == 128));
assert!(enc.v_plane.iter().all(|&v| v == 128));
}
#[test]
fn encodes_synthetic_frame_to_annexb_idr() {
let (w, h, fps) = (1280u32, 720u32, 60u32);