unom/punktfunk
1
0
Fork 0
Code Issues Pull Requests Actions 23 Packages Projects Releases 23 Wiki Activity

fix(video): honor the signaled CSC matrix end-to-end + tvOS HDR presentation

Browse Source 
Clients derive Y'CbCr->RGB from the stream's SIGNALED matrix x range x depth
via shared csc rows (Rust csc_rows + Swift CscRows) instead of hardcoded
709/2020 - a BT.601-signaled stream (a Linux host's RGB-input NVENC) no longer
renders with a constant hue error. Host-side signaling made honest across
NVENC/VAAPI/openh264/GameStream and the session plan's chroma/bit-depth.
Decoded color-bar fixtures (601/709 x limited/full) pin the math in tests on
both cores.

Same presenter, tvOS HDR: tvOS has no Metal EDR API and a bare PQ colorspace
tag composites UNTONE-MAPPED (the "overblown" Apple TV report), so HDR now
splits on the display's live EDR headroom - PQ passthrough when the
per-session AVDisplayManager mode switch landed (a real HDR10 output
tone-maps itself), else an in-shader PQ->SDR tone-map (203-nit reference
white, extended-Reinhard 1000-nit knee, 2020->709) into the proven SDR layer
config. The 10-bit stream keeps its full decode depth either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Enrico Bühler
2026-07-10 16:58:11 +02:00
parent db49904c6d
commit 1fcf9e11ec
26 changed files with 2268 additions and 409 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/punktfunk-host/src/encode/linux/mod.rs
+22 -8
View File
@@ -326,11 +326,19 @@ impl NvencEncoder {
};
}
// NV12 / 4:4:4 paths: we do the RGB→YUV conversion ourselves as BT.709 *limited* range
// (swscale), so signal that in the bitstream VUI (colorspace/range/primaries/transfer) —
// otherwise the client decoder assumes a default and the picture comes out washed-out /
// wrong-contrast. The RGB-input 4:2:0 path leaves these unset (NVENC's internal CSC writes
// its own VUI). Matches the Windows NV12 path's BT.709 limited-range signalling.
// NV12 / 4:4:4 paths: we do the RGB→YUV conversion ourselves as BT.709 (swscale), so
// signal that in the bitstream VUI (colorspace/range/primaries/transfer) — otherwise the
// client decoder assumes a default and the picture comes out washed-out / wrong-contrast.
// The RGB-input 4:2:0 path leaves these unset (NVENC's internal CSC writes its own VUI).
// Matches the Windows NV12 path's BT.709 limited-range signalling.
//
// PUNKTFUNK_444_FULLRANGE=1 (experimental, 4:4:4-only): convert AND signal FULL range —
// recovers the ~12% of code space limited-range quantization gives up, for the exact
// text/UI chroma 4:4:4 exists for. Every punktfunk client honors the signaled range
// (csc_rows / the Apple rows port); ship as default only if the on-glass A/B shows a
// visible win. Linux-only: the Windows path's NVENC-internal CSC range is unmeasured.
let full_range_444 = want_444
&& std::env::var("PUNKTFUNK_444_FULLRANGE").is_ok_and(|v| v.trim() == "1");
if matches!(format, PixelFormat::Nv12) || want_444 {
// SAFETY: same `video` builder — `raw = video.as_mut_ptr()` is the non-null, properly-
// aligned, sole-owned, not-yet-opened `AVCodecContext`. We set its four VUI colour enum
@@ -339,7 +347,11 @@ impl NvencEncoder {
unsafe {
let raw = video.as_mut_ptr();
(*raw).colorspace = ffi::AVColorSpace::AVCOL_SPC_BT709;
(*raw).color_range = ffi::AVColorRange::AVCOL_RANGE_MPEG; // limited/studio
(*raw).color_range = if full_range_444 {
ffi::AVColorRange::AVCOL_RANGE_JPEG // full
} else {
ffi::AVColorRange::AVCOL_RANGE_MPEG // limited/studio
};
(*raw).color_primaries = ffi::AVColorPrimaries::AVCOL_PRI_BT709;
(*raw).color_trc = ffi::AVColorTransferCharacteristic::AVCOL_TRC_BT709;
}
@@ -401,10 +413,12 @@ impl NvencEncoder {
// SAFETY: `sws` is the non-null context from the call above (null-checked). The ITU-709
// coefficient table from `sws_getCoefficients` is a process-lifetime libswscale static,
// reused for src+dst matrices; `sws_setColorspaceDetails` only reads it and writes scalar
// CSC settings into `sws` (limited-range dst: dstRange = 0). No Rust memory is passed.
// CSC settings into `sws` (dstRange matches the VUI: 0 = limited, 1 = the
// PUNKTFUNK_444_FULLRANGE experiment). No Rust memory is passed.
unsafe {
let cs709 = ffi::sws_getCoefficients(SWS_CS_ITU709);
ffi::sws_setColorspaceDetails(sws, cs709, 1, cs709, 0, 0, 1 << 16, 1 << 16);
let dst_range = i32::from(full_range_444);
ffi::sws_setColorspaceDetails(sws, cs709, 1, cs709, dst_range, 0, 1 << 16, 1 << 16);
}
Some(sws)
} else {
crates/punktfunk-host/src/encode/linux/vaapi.rs
+22 -3
View File
@@ -204,8 +204,9 @@ unsafe fn open_vaapi_encoder_mode(
let raw = video.as_mut_ptr();
(*raw).rc_buffer_size = vbv_bits as i32;
(*raw).gop_size = i32::MAX; // no periodic IDR (forced-IDR via pict_type=I on RFI)
// We hand the encoder BT.709 *limited* NV12 (swscale CSC, or scale_vaapi which preserves the
// input range we tag), so signal that VUI — else the client decoder washes the picture out.
// We hand the encoder BT.709 *limited* NV12 (swscale CSC on the CPU path; scale_vaapi pinned
// to `out_color_matrix=bt709:out_range=limited` on the zero-copy path, with the full-range
// RGB input tagged), so signal that VUI — else the client decoder washes the picture out.
(*raw).colorspace = ffi::AVColorSpace::AVCOL_SPC_BT709;
(*raw).color_range = ffi::AVColorRange::AVCOL_RANGE_MPEG;
(*raw).color_primaries = ffi::AVColorPrimaries::AVCOL_PRI_BT709;
@@ -718,6 +719,11 @@ impl DmabufInner {
(*par).format = ffi::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as c_int;
(*par).width = width as c_int;
(*par).height = height as c_int;
// Declare the link's colour up front (full-range RGB — the compositor's desktop) so
// the per-frame tags in `submit` match the negotiated link instead of reading as a
// mid-stream property change.
(*par).color_space = ffi::AVColorSpace::AVCOL_SPC_RGB;
(*par).color_range = ffi::AVColorRange::AVCOL_RANGE_JPEG;
(*par).time_base = ffi::AVRational {
num: 1,
den: fps as c_int,
@@ -751,7 +757,14 @@ impl DmabufInner {
}
init!(src, ptr::null(), "buffer");
init!(hwmap, c"mode=read".as_ptr(), "hwmap");
init!(scale, c"format=nv12".as_ptr(), "scale_vaapi");
// Pin the VPP's output colour to what the encoder's VUI signals (BT.709 limited).
// Without the explicit options the conversion matrix is whatever the driver defaults
// to for an unspecified output (Mesa: BT.601) — a hue shift against the signaled VUI.
init!(
scale,
c"format=nv12:out_color_matrix=bt709:out_range=limited".as_ptr(),
"scale_vaapi"
);
init!(sink, ptr::null(), "buffersink");
let link = |a: *mut ffi::AVFilterContext, b: *mut ffi::AVFilterContext| -> c_int {
@@ -879,6 +892,12 @@ impl DmabufInner {
(*drm).format = ffi::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as c_int;
(*drm).width = self.width as c_int;
(*drm).height = self.height as c_int;
// The dmabuf is the compositor's rendered desktop: full-range RGB. Tag the frame so
// the VPP's colour negotiation sees the real input instead of "unspecified" (an
// untagged input lets the driver pick its own default for the RGB→NV12 conversion —
// Mesa's is BT.601, contradicting the BT.709-limited VUI the encoder signals).
(*drm).color_range = ffi::AVColorRange::AVCOL_RANGE_JPEG;
(*drm).colorspace = ffi::AVColorSpace::AVCOL_SPC_RGB;
(*drm).hw_frames_ctx = ffi::av_buffer_ref(self.drm_frames);
(*drm).data[0] = Box::into_raw(desc) as *mut u8;
// Own the descriptor so it frees with the frame (the fd is owned by the DmabufFrame,
crates/punktfunk-host/src/encode/sw.rs
+114 -44
View File
@@ -1,7 +1,13 @@
//! Software H.264 encoder (openh264) — the GPU-less encode path for the Windows host (and a
//! fallback when NVENC is unavailable). Low-latency screen-content config: single-reference,
//! no B-frames (Baseline), bitrate rate-control, in-band SPS/PPS each IDR, BT.709 limited range.
//! no B-frames (Baseline), bitrate rate-control, in-band SPS/PPS each IDR.
//! Synchronous: `submit` encodes immediately and stashes the AU for `poll` (no internal queue).
//!
//! The RGB→YUV conversion is OURS, BT.709 limited range: openh264 writes no colour description
//! into the VUI (unspecified), so decoders fall back to their default — BT.709 limited on every
//! punktfunk client — and the pixels must match that default. The crate's own `YUVBuffer`
//! converter is BT.601 (0.2578/0.5039/0.0977 + 16), which decoded-as-709 is a constant hue
//! error; that's why it is NOT used here.
// Every `unsafe` block in this file carries a `// SAFETY:` proof; enforce it (unsafe-proof program).
#![deny(clippy::undocumented_unsafe_blocks)]
@@ -12,19 +18,20 @@ use openh264::encoder::{
BitRate, Complexity, Encoder as Oh264, EncoderConfig, FrameRate, FrameType, IntraFramePeriod,
Profile, RateControlMode, SpsPpsStrategy, UsageType,
};
use openh264::formats::{BgraSliceU8, RgbSliceU8, YUVBuffer};
use openh264::formats::YUVSlices;
use openh264::OpenH264API;
pub struct OpenH264Encoder {
enc: Oh264,
yuv: YUVBuffer,
width: u32,
height: u32,
fps: u32,
src_format: PixelFormat,
/// BGRA scratch for the 3-bpp (Bgr) and R/B-swapped (Rgba/Rgbx) formats openh264 can't wrap
/// directly. Reused across frames.
scratch: Vec<u8>,
/// The converted I420 planes (our BT.709-limited CSC — see the module doc), reused across
/// frames: full-res luma + quarter-res Cb/Cr, tightly packed (stride = width, width/2).
y_plane: Vec<u8>,
u_plane: Vec<u8>,
v_plane: Vec<u8>,
frame_idx: i64,
force_kf: bool,
/// At most one AU per submit (no lookahead), handed back by the next `poll`.
@@ -33,7 +40,7 @@ pub struct OpenH264Encoder {
// openh264's Encoder holds a raw C handle (not auto-Send); it lives on the single encode thread.
// SAFETY: `OpenH264Encoder` wraps `Oh264` (openh264's `Encoder`), which holds a raw C handle to the
// openh264 `ISVCEncoder` and is not auto-`Send`; the other fields (`YUVBuffer`, `Vec`, scalars,
// openh264 `ISVCEncoder` and is not auto-`Send`; the other fields (the plane `Vec`s, scalars,
// `Option<EncodedFrame>`) are plain owned data. The session creates the encoder, calls
// `submit`/`poll`/`flush`, and drops it all on one dedicated encode thread, never sharing it by
// reference across threads, so the C handle is only ever touched from a single thread. Moving the
@@ -62,51 +69,75 @@ impl OpenH264Encoder {
.scene_change_detect(false) // no surprise IDRs (bitrate spikes / freeze)
.adaptive_quantization(true)
.complexity(Complexity::Low) // latency over BD-rate
.profile(Profile::Baseline); // no B-frames; BT.709 limited is the crate default VUI
.profile(Profile::Baseline); // no B-frames; the VUI carries no colour description
let api = OpenH264API::from_source(); // statically-bundled build (default `source` feature)
let enc = Oh264::with_api_config(api, cfg).context("openh264 Encoder::with_api_config")?;
let yuv = YUVBuffer::new(width as usize, height as usize);
let (w, h) = (width as usize, height as usize);
tracing::info!(
"openh264 software encoder: {width}x{height}@{fps} {} Mbps (Baseline, screen-content)",
bps / 1_000_000
);
Ok(Self {
enc,
yuv,
width,
height,
fps,
src_format: format,
scratch: Vec::new(),
y_plane: vec![0; w * h],
u_plane: vec![0; (w / 2) * (h / 2)],
v_plane: vec![0; (w / 2) * (h / 2)],
frame_idx: 0,
force_kf: false,
pending: None,
})
}
/// Normalize a packed source buffer into the reused BGRA `scratch` ([B,G,R,A]). `rgb_order`
/// = source is R,G,B (swap into B,G,R); otherwise source is already B,G,R.
fn normalize_to_bgra(&mut self, src: &[u8], src_bpp: usize, rgb_order: bool) {
/// Convert one packed full-range RGB frame into the I420 planes, BT.709 limited range.
/// `bpp` is the source pixel stride; `ri`/`gi`/`bi` the channel byte offsets within a pixel.
/// Luma per pixel; Cb/Cr from the 2×2 block's averaged RGB (the same box filter the crate's
/// converter used, so only the matrix changed).
fn convert_bt709(&mut self, src: &[u8], bpp: usize, ri: usize, gi: usize, bi: usize) {
let w = self.width as usize;
let h = self.height as usize;
self.scratch.resize(w * h * 4, 0);
for px in 0..(w * h) {
let s = &src[px * src_bpp..px * src_bpp + 3];
let d = &mut self.scratch[px * 4..px * 4 + 4];
if rgb_order {
d[0] = s[2];
d[1] = s[1];
d[2] = s[0];
} else {
d[0] = s[0];
d[1] = s[1];
d[2] = s[2];
let cw = w / 2;
for by in 0..h / 2 {
for bx in 0..cw {
let mut sum = (0f32, 0f32, 0f32);
for (dy, dx) in [(0, 0), (0, 1), (1, 0), (1, 1)] {
let (px, py) = (bx * 2 + dx, by * 2 + dy);
let s = &src[(py * w + px) * bpp..];
let (r, g, b) = (f32::from(s[ri]), f32::from(s[gi]), f32::from(s[bi]));
self.y_plane[py * w + px] = luma709(r, g, b);
sum = (sum.0 + r, sum.1 + g, sum.2 + b);
}
let (cb, cr) = chroma709(sum.0 / 4.0, sum.1 / 4.0, sum.2 / 4.0);
self.u_plane[by * cw + bx] = cb;
self.v_plane[by * cw + bx] = cr;
}
d[3] = 0xff;
}
}
}
/// BT.709 luma coefficients (Kg = 1 − Kr − Kb).
const KR: f32 = 0.2126;
const KB: f32 = 0.0722;
const KG: f32 = 1.0 - KR - KB;
/// One full-range RGB pixel (0..=255 channels) → the BT.709 limited-range 8-bit luma code
/// (16..=235). Kept in lockstep with the client-side inverse (`pf-client-core::video::csc_rows`).
fn luma709(r: f32, g: f32, b: f32) -> u8 {
let y = KR * r + KG * g + KB * b; // full-scale luma, 0..=255
(16.0 + y * (219.0 / 255.0) + 0.5) as u8 // `as` saturates — no manual clamp needed
}
/// (Averaged) full-range RGB → the BT.709 limited-range Cb/Cr codes (16..=240, neutral 128).
fn chroma709(r: f32, g: f32, b: f32) -> (u8, u8) {
let y = KR * r + KG * g + KB * b;
let cb = 128.0 + (b - y) * (224.0 / 255.0) / (2.0 * (1.0 - KB));
let cr = 128.0 + (r - y) * (224.0 / 255.0) / (2.0 * (1.0 - KR));
((cb + 0.5) as u8, (cr + 0.5) as u8)
}
impl Encoder for OpenH264Encoder {
fn submit(&mut self, captured: &CapturedFrame) -> Result<()> {
ensure!(
@@ -139,21 +170,13 @@ impl Encoder for OpenH264Encoder {
self.src_format
);
match self.src_format {
PixelFormat::Rgb => self
.yuv
.read_rgb(RgbSliceU8::new(&bytes[..w * h * 3], (w, h))),
PixelFormat::Bgra | PixelFormat::Bgrx => self
.yuv
.read_rgb(BgraSliceU8::new(&bytes[..w * h * 4], (w, h))),
PixelFormat::Rgba | PixelFormat::Rgbx => {
self.normalize_to_bgra(bytes, 4, true);
self.yuv.read_rgb(BgraSliceU8::new(&self.scratch, (w, h)));
}
PixelFormat::Bgr => {
self.normalize_to_bgra(bytes, 3, false);
self.yuv.read_rgb(BgraSliceU8::new(&self.scratch, (w, h)));
}
// Source pixel stride + R/G/B byte offsets within a pixel — one converter for every
// packed-RGB layout the capturers emit (no BGRA normalization pass needed).
let (bpp, ri, gi, bi) = match self.src_format {
PixelFormat::Rgb => (3, 0, 1, 2),
PixelFormat::Bgr => (3, 2, 1, 0),
PixelFormat::Rgba | PixelFormat::Rgbx => (4, 0, 1, 2),
PixelFormat::Bgra | PixelFormat::Bgrx => (4, 2, 1, 0),
// 10-bit HDR comes only from the GPU NVENC path; the software 8-bit H.264 encoder
// can't represent it (and never receives it — the capturer pairs Rgb10a2 with NVENC).
PixelFormat::Rgb10a2 => {
@@ -166,13 +189,19 @@ impl Encoder for OpenH264Encoder {
"software encoder cannot encode YUV GPU textures (NV12/P010 → NVENC only)"
)
}
}
};
self.convert_bt709(bytes, bpp, ri, gi, bi);
if self.force_kf {
self.enc.force_intra_frame();
self.force_kf = false;
}
let bs = self.enc.encode(&self.yuv).context("openh264 encode")?;
let slices = YUVSlices::new(
(&self.y_plane, &self.u_plane, &self.v_plane),
(w, h),
(w, w / 2, w / 2),
);
let bs = self.enc.encode(&slices).context("openh264 encode")?;
let mut data = Vec::new();
bs.write_vec(&mut data); // AnnexB start codes; SPS/PPS prepended on IDR
if !data.is_empty() {
@@ -225,6 +254,47 @@ mod tests {
use super::*;
use crate::capture::{CapturedFrame, FramePayload, PixelFormat};
/// The BT.709 limited-range anchor points: reference white → (235,128,128), black →
/// (16,128,128), pure red's Cr must hit the positive extreme 240 (it does exactly:
/// 255(1−Kr)·(224/255)/(2(1−Kr)) = 112). ±1 code for float rounding.
#[test]
fn bt709_conversion_anchor_points() {
assert_eq!(luma709(255.0, 255.0, 255.0), 235);
assert_eq!(luma709(0.0, 0.0, 0.0), 16);
assert_eq!(chroma709(255.0, 255.0, 255.0), (128, 128));
assert_eq!(chroma709(0.0, 0.0, 0.0), (128, 128));
let (cb, cr) = chroma709(255.0, 0.0, 0.0);
assert_eq!(cr, 240, "pure red must reach the Cr extreme");
assert!((101..=103).contains(&cb), "red Cb ~102, got {cb}");
let (cb, _) = chroma709(0.0, 0.0, 255.0);
assert_eq!(cb, 240, "pure blue must reach the Cb extreme");
}
/// The 601-vs-709 luma split on pure green (Kg 0.587 vs 0.7152) — guards against anyone
/// "simplifying" the coefficients back to the crate's BT.601 converter (the hue-shift bug
/// this module's own conversion exists to prevent).
#[test]
fn bt709_is_not_bt601() {
// BT.601 green luma: 16 + 219·0.587 = 144.5; BT.709: 16 + 219·0.7152 = 172.6.
let y = luma709(0.0, 255.0, 0.0);
assert!((172..=174).contains(&y), "709 green luma ~173, got {y}");
}
/// A flat gray frame converts to neutral chroma and mid luma across every plane byte
/// (exercises the block loop + plane sizing, not just the per-pixel math).
#[test]
fn converts_flat_gray_to_neutral_planes() {
let (w, h) = (16u32, 8u32);
let mut enc =
OpenH264Encoder::open(PixelFormat::Bgrx, w, h, 60, 1_000_000).expect("open openh264");
let bytes = vec![0x80u8; (w * h * 4) as usize];
enc.convert_bt709(&bytes, 4, 2, 1, 0);
// 16 + 128·(219/255) = 125.9 → 126.
assert!(enc.y_plane.iter().all(|&y| y == 126), "{:?}", &enc.y_plane[..4]);
assert!(enc.u_plane.iter().all(|&u| u == 128));
assert!(enc.v_plane.iter().all(|&v| v == 128));
}
#[test]
fn encodes_synthetic_frame_to_annexb_idr() {
let (w, h, fps) = (1280u32, 720u32, 60u32);
crates/punktfunk-host/src/encode/windows/nvenc.rs
+206 -15
View File
@@ -708,11 +708,13 @@ impl NvencD3d11Encoder {
// input — a subsampled NV12/P010 source can't reconstruct full chroma (so the capturer is
// forced to RGB for a 4:4:4 session, and we guard on the input format here too).
//
// ON-GLASS TODO (RTX box): confirm ARGB + chromaFormatIDC=3 + FREXT yields a *true* 4:4:4
// stream. NVENC's RGB→YUV CSC is documented to honor chromaFormatIDC (unlike libavcodec's
// wrapper, which always subsamples RGB to 4:2:0 — hence the Linux path feeds planar YUV444
// instead). If on-glass shows 4:2:0, the follow-up is a BGRA→AYUV shader feeding the native
// `NV_ENC_BUFFER_FORMAT_AYUV` 4:4:4 input format.
// ON-GLASS MEASURED (RTX 5070 Ti, driver 610.43, 2026-07-10 — `nvenc_444_on_glass_probe`
// below + colour-bar analysis): ARGB + chromaFormatIDC=3 + FREXT yields a TRUE 4:4:4
// stream (1-px chroma stripes survive, adjacent-column |dU| ≈ 138), and NVENC's internal
// RGB→YUV conversion FOLLOWS THE CONFIGURED VUI MATRIX (bars match BT.709 within ±1 code
// with our 709 VUI; the same driver produces exact BT.601 when libavcodec's nvenc wrapper
// sets its BT470BG VUI on Linux). The always-written SDR VUI above therefore makes the
// pixels and the signaling agree by construction — no AYUV shader needed.
let rgb_input = matches!(
self.buffer_fmt,
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_ARGB
@@ -752,21 +754,33 @@ impl NvencD3d11Encoder {
}
}
// HDR colour signaling: BT.2020 primaries + SMPTE ST.2084 (PQ) transfer + BT.2020-NCL
// matrix, limited (studio) range — NVENC's RGB→YUV default. HEVC/H.264 carry it in the VUI;
// AV1 has NO VUI, so the SAME CICP code points go in the sequence-header colour config
// (`colorPrimaries`/`transferCharacteristics`/`matrixCoefficients`/`colorRange`). Without
// this a non-HEVC decoder assumes BT.709 SDR → washed-out / colour-shifted HDR.
// Colour signaling, written UNCONDITIONALLY (was HDR-only): the capturer hands NVENC
// pre-converted NV12 (BT.709 limited, the IDD VideoConverter) or P010 (BT.2020 PQ
// limited, the FP16→P010 shader), so the stream must SAY so — an SDR stream with no
// colour description decodes correctly only on clients whose "unspecified" default
// happens to be BT.709 limited (ours are, but Moonlight/third-party/Android-vendor
// decoders default 601 at sub-HD resolutions). HEVC/H.264 carry it in the VUI; AV1 has
// NO VUI, so the SAME CICP code points go in the sequence-header colour config
// (`colorPrimaries`/`transferCharacteristics`/`matrixCoefficients`/`colorRange`).
//
// This is the per-stream colour *description* only. The static mastering-display (ST.2086)
// and content-light (MaxCLL/MaxFALL) metadata — HEVC SEI / AV1 METADATA OBUs — is a
// separate follow-up, as is wiring AV1/H.264 to a true 10-bit (Main10) encode (only HEVC
// sets Main10 above today).
if self.hdr {
let prim = nv::NV_ENC_VUI_COLOR_PRIMARIES::NV_ENC_VUI_COLOR_PRIMARIES_BT2020;
let trc =
nv::NV_ENC_VUI_TRANSFER_CHARACTERISTIC::NV_ENC_VUI_TRANSFER_CHARACTERISTIC_SMPTE2084;
let mat = nv::NV_ENC_VUI_MATRIX_COEFFS::NV_ENC_VUI_MATRIX_COEFFS_BT2020_NCL;
{
let (prim, trc, mat) = if self.hdr {
(
nv::NV_ENC_VUI_COLOR_PRIMARIES::NV_ENC_VUI_COLOR_PRIMARIES_BT2020,
nv::NV_ENC_VUI_TRANSFER_CHARACTERISTIC::NV_ENC_VUI_TRANSFER_CHARACTERISTIC_SMPTE2084,
nv::NV_ENC_VUI_MATRIX_COEFFS::NV_ENC_VUI_MATRIX_COEFFS_BT2020_NCL,
)
} else {
(
nv::NV_ENC_VUI_COLOR_PRIMARIES::NV_ENC_VUI_COLOR_PRIMARIES_BT709,
nv::NV_ENC_VUI_TRANSFER_CHARACTERISTIC::NV_ENC_VUI_TRANSFER_CHARACTERISTIC_BT709,
nv::NV_ENC_VUI_MATRIX_COEFFS::NV_ENC_VUI_MATRIX_COEFFS_BT709,
)
};
match self.codec {
Codec::H265 => {
let vui = &mut cfg.encodeCodecConfig.hevcConfig.hevcVUIParameters;
@@ -1160,6 +1174,24 @@ impl Encoder for NvencD3d11Encoder {
}
_ => nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_ARGB,
};
// 4:4:4 honesty: the FREXT/chromaFormatIDC=3 config engages only on an RGB input (a
// subsampled NV12/P010 source can't reconstruct full chroma). If the capturer handed
// native YUV despite a 4:4:4 negotiation, this session encodes 4:2:0 — clear the flag
// NOW so `caps().chroma_444` (and punktfunk1's post-open cross-check) reports what
// the stream really carries instead of silently claiming full chroma.
if self.chroma_444
&& !matches!(
self.buffer_fmt,
nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_ARGB
| nv::NV_ENC_BUFFER_FORMAT::NV_ENC_BUFFER_FORMAT_ABGR10
)
{
tracing::warn!(
format = ?captured.format,
"4:4:4 negotiated but the capturer delivered subsampled YUV — encoding 4:2:0"
);
self.chroma_444 = false;
}
let device = frame.device.clone();
self.init_session(&device)?;
self.init_device = dev_raw;
@@ -1573,3 +1605,162 @@ pub fn probe_can_encode_444(codec: Codec) -> bool {
ok
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::capture::{dxgi::D3d11Frame, CapturedFrame, FramePayload};
use windows::Win32::Graphics::Direct3D11::{
D3D11_BIND_RENDER_TARGET, D3D11_SUBRESOURCE_DATA, D3D11_TEXTURE2D_DESC,
D3D11_USAGE_DEFAULT,
};
use windows::Win32::Graphics::Dxgi::Common::{
DXGI_FORMAT_B8G8R8A8_UNORM, DXGI_SAMPLE_DESC,
};
use windows::Win32::Graphics::Dxgi::{
CreateDXGIFactory1, IDXGIFactory1, DXGI_ADAPTER_FLAG_SOFTWARE,
};
/// The 8 fully-saturated colour bars the matrix analysis samples (RGB). Saturated primaries
/// separate BT.601 from BT.709 by tens of code points (e.g. pure-green luma 145 vs 173).
const BARS: [(u8, u8, u8); 8] = [
(255, 255, 255), // white
(255, 255, 0), // yellow
(0, 255, 255), // cyan
(0, 255, 0), // green
(255, 0, 255), // magenta
(255, 0, 0), // red
(0, 0, 255), // blue
(0, 0, 0), // black
];
/// BGRA probe pattern: left half = the 8 colour bars (flat patches → matrix measurement),
/// right half = alternating 1-px red/blue columns (the chroma-resolution litmus: true 4:4:4
/// keeps adjacent columns' chroma distinct; an internally-subsampled encode blends them).
fn probe_pattern(w: usize, h: usize) -> Vec<u8> {
let mut px = vec![0u8; w * h * 4];
let bar_w = (w / 2) / BARS.len();
for y in 0..h {
for x in 0..w {
let (r, g, b) = if x < w / 2 {
BARS[(x / bar_w).min(BARS.len() - 1)]
} else if x % 2 == 0 {
(255, 0, 0) // red column
} else {
(0, 0, 255) // blue column
};
let o = (y * w + x) * 4;
px[o] = b;
px[o + 1] = g;
px[o + 2] = r;
px[o + 3] = 255;
}
}
px
}
/// Encode 30 static pattern frames through the real NVENC session (ARGB input, the exact
/// production configuration) at the given chroma and write the Annex-B stream to `path`.
fn encode_pattern(chroma: ChromaFormat, path: &str) {
const W: u32 = 1280;
const H: u32 = 720;
// SAFETY (test-only): straight-line D3D11/DXGI COM calls on one thread; every out-pointer
// is checked before use; the texture/device outlive the encoder (dropped at scope end).
unsafe {
let factory: IDXGIFactory1 = CreateDXGIFactory1().expect("DXGI factory");
let mut adapter = None;
for i in 0.. {
let Ok(a) = factory.EnumAdapters1(i) else { break };
let desc = a.GetDesc1().expect("adapter desc");
if desc.Flags & DXGI_ADAPTER_FLAG_SOFTWARE.0 as u32 == 0 {
adapter = Some(a);
break;
}
}
let adapter = adapter.expect("no hardware DXGI adapter");
let (device, _ctx) =
crate::capture::dxgi::make_device(&adapter).expect("make_device");
let bytes = probe_pattern(W as usize, H as usize);
let init = D3D11_SUBRESOURCE_DATA {
pSysMem: bytes.as_ptr() as *const _,
SysMemPitch: W * 4,
SysMemSlicePitch: 0,
};
let desc = D3D11_TEXTURE2D_DESC {
Width: W,
Height: H,
MipLevels: 1,
ArraySize: 1,
Format: DXGI_FORMAT_B8G8R8A8_UNORM,
SampleDesc: DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: D3D11_USAGE_DEFAULT,
// NVENC registration requires RENDER_TARGET on D3D11 input textures.
BindFlags: D3D11_BIND_RENDER_TARGET.0 as u32,
CPUAccessFlags: 0,
MiscFlags: 0,
};
let mut tex = None;
device
.CreateTexture2D(&desc, Some(&init), Some(&mut tex))
.expect("pattern texture");
let tex = tex.expect("null pattern texture");
let mut enc = NvencD3d11Encoder::open(
Codec::H265,
PixelFormat::Bgra,
W,
H,
60,
100_000_000, // high rate: the 1-px stripes must survive quantization
8,
chroma,
)
.expect("NVENC open");
let mut out = Vec::new();
for i in 0..30u64 {
let frame = CapturedFrame {
width: W,
height: H,
pts_ns: i * 16_666_667,
format: PixelFormat::Bgra,
payload: FramePayload::D3d11(D3d11Frame {
texture: tex.clone(),
device: device.clone(),
}),
};
enc.submit(&frame).expect("submit");
while let Some(au) = enc.poll().expect("poll") {
out.extend_from_slice(&au.data);
}
}
enc.flush().ok();
while let Ok(Some(au)) = enc.poll() {
out.extend_from_slice(&au.data);
}
assert!(!out.is_empty(), "no AUs produced");
let caps444 = enc.caps().chroma_444;
std::fs::write(path, &out).expect("write bitstream");
println!(
"wrote {path}: {} bytes, requested {chroma:?}, caps.chroma_444={caps444}",
out.len()
);
}
}
/// ON-GLASS (RTX box): the measurement gating the AYUV 4:4:4 work — encodes the probe
/// pattern through the REAL ARGB-input NVENC session once with `chromaFormatIDC=3`/FREXT
/// and once as plain 4:2:0, so offline analysis of the two bitstreams answers (1) whether
/// the FREXT stream is truly full-chroma and (2) which matrix NVENC's internal RGB→YUV CSC
/// used (BT.601 vs BT.709 — saturated bars differ by tens of code points). Run with:
/// cargo test -p punktfunk-host --features nvenc -- --ignored nvenc_444_on_glass --nocapture
#[test]
#[ignore = "requires an NVIDIA GPU + driver — run manually on the RTX box"]
fn nvenc_444_on_glass_probe() {
encode_pattern(ChromaFormat::Yuv444, "C:\\Users\\Public\\nvenc444_probe.h265");
encode_pattern(ChromaFormat::Yuv420, "C:\\Users\\Public\\nvenc420_probe.h265");
}
}