punktfunk/design/native-amf-encoder.md

Native AMF encoder — handoff design

Status: PHASES 1 + 2 + 3 IMPLEMENTED (2026-07-06). encode/windows/amf.rs ships the direct-SDK encoder per §3 — FFI pinned to AMF headers v1.4.36, bounded poll, native reset(). Phase 2: AV1 (open-time probe gate; per-codec enum divergences honored — AV1 swaps the ULL/LL usage values and uses GOP=0 + FORCE_FRAME_TYPE_KEY=1), intra-refresh (PUNKTFUNK_INTRA_REFRESH opt-in mirroring Linux NVENC; caps().intra_refresh reflects the driver's actual acceptance), in-band HDR mastering/CLL metadata (*InHDRMetadata host buffer; HEVC + AV1), and the native codec probe. Phase 3: the ffmpeg-AMF dispatch fallback + PUNKTFUNK_AMF_FFMPEG hatch are deleted — AMD dispatch / codec advertisement / 4:4:4 answer are native-only; FFmpeg serves QSV only (ffmpeg_win.rs's AMF machinery is kept solely as the A/B comparator). windows_backend_is_ffmpeg → windows_backend_is_probed. The §7 field-silence gate on Phase 3 was pre-empted on explicit direction — see the §7 gate note for what that costs (VP-format-fallback now fails the session; AMFVideoConverter is the owed native fix). Live-validated on the lab Ryzen iGPU (VCN 3): AVC + HEVC batches across a native in-place reset (Annex-B IDR contract, FIFO pairing); HEVC Main10 P010 with the mastering + CLL prefix SEIs confirmed present in the encoded IDR; intra-refresh property accepted on both codecs; probe honestly answers h264/h265=true, av1=false on this RDNA2 part. The §5.2 latency A/B is measured (amf_latency_ab_bench, 1080p60 HEVC): native encode_us p50 5.18 ms (0.31 frame periods) vs libavcodec-AMF 16.9 ms (1.01) — 3.3× lower, the frame-hold gone. Owed: §5.3 on-glass session behaviors + soak (macOS-client on-glass test in progress), and field validation on ≥2 VCN generations (AV1/RDNA3 has no lab hardware). Companion context: the encode-stall watchdog + Encoder::reset() (punktfunk1.rs / encode.rs, shipped 2026-07-06) and the QSV backend module docs in encode/windows/ffmpeg_win.rs.

1. Why (measured, not speculative)

Three independent reasons, in order of weight:

1. The libavcodec AMF wrapper's structural ~2-frame output hold. hevc_amf/av1_amf need frame N+2 submitted before they release frame N's AU. Measured on the Ryzen 7000 iGPU (VCN, 720p60): encode→retrieve 36 ms p50, dead-stable, invariant across pipeline depth 1/2, every usage preset, and any spin budget (a 150 ms poll spin provably never produced the owed AU — it pegged at exactly 150 ms). See the poll doc comment in ffmpeg_win.rs. The direct-SDK NVENC path retrieves in ~1–2 ms. At 60 Hz this is ~33 ms of pure pipeline latency no FFmpeg-side knob can remove; at 120 Hz it is two whole frame budgets.
2. Silent driver wedges surface as forever-EAGAIN, not errors. The field failure (AMD/Intel streams freezing after ~3–5 min) was invisible because the wrapper's only "not ready" signal is EAGAIN, indistinguishable from a healthy pipeline warming up. The 2026-07-06 watchdog converts that into a bounded rebuild + IDR, but it is a safety net with a ~2 s detection floor. The AMF runtime itself returns typed AMF_RESULT codes (AMF_INPUT_FULL, device-lost, etc.) — a native path sees the wedge on the frame it happens.
3. Feature gaps libavcodec cannot express. No intra-refresh wave (every FEC-unrecoverable loss is answered with a full IDR — the 20–40× frame-size spike the Linux NVENC intra-refresh mode exists to avoid), no in-band HDR mastering SEI (EncoderCaps::supports_hdr_metadata is NVENC-only today), coarse per-frame control.

2. The decision: drop FFmpeg for AMF, keep it for QSV

Native AMF replaces the libavcodec AMF path (phased, §7). FFmpeg stays for QSV.

• QSV via libavcodec with async_depth=1 + low_power VDEnc is already near the hardware latency floor; a direct libvpl port would buy little for its cost. Revisit only if Intel field data shows a QSV-specific gap (separate doc if so).
• Because QSV stays on FFmpeg, the FFmpeg DLLs keep shipping and the amf-qsv build feature keeps existing. Dropping FFmpeg entirely is therefore not on the table here — "drop" means: the AMF dispatch stops going through it once the native path is validated.
• During bring-up the ffmpeg-AMF path remains as an automatic open-failure fallback and an explicit escape hatch (§7), then its AMF dispatch is deleted in Phase 3. Two permanently maintained AMF paths would double the driver-matrix burden, and the one we'd keep "for safety" is precisely the one with the wedge/latency pathology.

3. Architecture

New module crates/punktfunk-host/src/encode/windows/amf.rs implementing crate::encode::Encoder, compiled unconditionally on Windows (no new build-time dependency and no new cargo feature): the AMF runtime is loaded at runtime from the driver-installed amfrt64.dll, exactly as nvenc.rs loads nvEncodeAPI64.dll (load_api). A box without an AMD driver simply fails the open and the dispatch falls through. In-tree FFI decl module (amf_sys submodule or #[repr(C)] blocks in-file, mirroring the small interface subset we use) — model it on how ffmpeg_win.rs mirrors AVD3D11VADeviceContext rather than pulling a binding crate (none is maintained).

3.1 FFI strategy (the load-bearing detail)

The AMF public headers (GPUOpen AMF/amf/public/include) define C-compatible vtable structs for every interface (AMFFactoryVtbl, AMFContextVtbl, AMFComponentVtbl, AMFSurfaceVtbl, AMFDataVtbl, AMFBufferVtbl, AMFVariantStruct, …) — this is not a guess: FFmpeg's amfenc.c is plain C and drives AMF exclusively through those vtables, so the C ABI is the stable, supported surface. Mirror only what we call:

• Entry points: GetProcAddress("AMFQueryVersion") (gate: runtime ≥ the pinned AMF_FULL_VERSION we mirror headers from) and GetProcAddress("AMFInit") → AMFFactory*.
• factory->CreateContext → context->InitDX11(capturer_device, AMF_DX11_1) — the capturer's own ID3D11Device, same-device requirement as every other backend (the capture textures are not shared-handle; see the ensure_inner_d3d11 rebind logic in ffmpeg_win.rs for the device-change lifecycle to replicate).
• factory->CreateComponent(context, name) with AMFVideoEncoderVCE_AVC / AMFVideoEncoder_HEVC / AMFVideoEncoder_AV1 → encoder->Init(AMF_SURFACE_NV12|P010, w, h).
• Per-frame: context->CreateSurfaceFromDX11Native(texture, &surface, observer) → per-surface properties (pts via SetPts, forced-IDR picture type) → encoder->SubmitInput(surface); retrieve via encoder->QueryOutput(&data) (AMF_REPEAT = not ready), AMFBuffer::GetNative/GetSize → EncodedFrame.
• Every mirrored struct/call carries a // SAFETY: proof — the whole encode module tree is under #![deny(clippy::undocumented_unsafe_blocks)] (unsafe-proof program).

Reference implementations to crib from (read both before writing FFI): FFmpeg amfenc.c (the C vtbl usage, property plumbing, result-code handling) and OBS plugins/obs-ffmpeg/texture-amf.cpp (D3D11 texture submission + low-latency streaming config, C++ but the call sequence is what matters). Verify every property name against the pinned SDK headers — names below are from those references and must not be trusted blind.

3.2 Input path (zero-copy by construction)

Own a small D3D11 texture ring (NV12 or P010, D3D11_BIND_RENDER_TARGET | SHADER_RESOURCE, size = pipeline_depth + 2), CopySubresourceRegion the captured texture into the next slot (GPU-local, same pattern as ZeroCopyInner::submit), wrap the slot with CreateSurfaceFromDX11Native, submit. The copy decouples the encoder from the capturer's rotating IDD ring; do NOT wrap the capturer's texture directly. This makes PUNKTFUNK_ZEROCOPY irrelevant for native AMF — there is no readback path to fall back to. Handle the capturer's video-processor format fallback (Bgra/Rgb10a2 instead of NV12/P010 — see pool_mismatch in ffmpeg_win.rs) by returning an open/submit error in Phase 1 so dispatch falls back to the ffmpeg path; an AMFVideoConverter front-end is a Phase 2 option if that fallback ever fires in the field. FramePayload::Cpu (DDA without video processor): same treatment — ffmpeg fallback in Phase 1.

3.3 Retrieval model

Bounded-blocking poll, the vaapi.rs::poll model: after SubmitInput, spin QueryOutput with ~250 µs sleeps up to a budget of min(3/4 frame interval, 12 ms); on expiry return Ok(None) (the session loop keeps the frame in flight and the watchdog arbitrates wedges). VCN encode at streaming settings is ~1–5 ms, so the AU ships the same tick — this is where the ~2-frame hold dies. Expected observable — measure encode_us (submit→AU, in FrameMsg/the web-console stats), not wait_us: on the ffmpeg path the hold hides in encode_us (~2 frame periods) because its non-blocking poll returns EAGAIN in ~2 µs; on the native bounded poll the ASIC wait becomes visible as a few ms of wait_us while encode_us collapses to ~1 frame period or less.

3.4 Encoder configuration (initial property set)

Mirror the intent of the ffmpeg opts block in open_win_encoder (ffmpeg_win.rs:216-247). AVC names given; HEVC/AV1 have _HEVC_/_AV1_ twins — check headers:

Intent	AMF property (verify!)
usage preset (keep PUNKTFUNK_AMF_USAGE mapping)	AMF_VIDEO_ENCODER_USAGE = ULTRA_LOW_LATENCY (default)
CBR, target==peak	RATE_CONTROL_METHOD=CBR, TARGET_BITRATE, PEAK_BITRATE
1-frame VBV (keep PUNKTFUNK_VBV_FRAMES)	VBV_BUFFER_SIZE
HRD + no filler	ENFORCE_HRD=true, FILLER_DATA_ENABLE=false
latency-first quality	QUALITY_PRESET=SPEED
no B-frames (AVC)	B_PIC_PATTERN=0
infinite GOP	IDR_PERIOD=0 (HEVC: GOP_SIZE/NUM_GOPS_PER_IDR — check)
low-latency submission	LOWLATENCY_MODE=true (newer SDKs)
in-band VPS/SPS/PPS on IDR (wire contract: EncodedFrame doc)	HEVC HEADER_INSERTION_MODE=IDR_ALIGNED; AVC HEADER_INSERTION_SPACING — check
SDR/HDR VUI	FULL_RANGE_COLOR=false + color primaries/transfer/matrix props (BT.709 vs BT.2020-PQ, mirroring open_win_encoder)
10-bit	COLOR_BIT_DEPTH=10 + P010 surfaces
per-frame forced IDR	on the input surface: AMF_VIDEO_ENCODER_FORCE_PICTURE_TYPE=IDR
intra-refresh wave (Phase 2)	AVC INTRA_REFRESH_NUM_MBS_PER_SLOT; HEVC CTB twin — check
HDR mastering SEI (Phase 2)	HEVC INPUT_HDR_METADATA (AMFHDRMetadata buffer)

SetProperty failures on optional properties (LOWLATENCY_MODE, intra-refresh) must be log-and-continue, not fatal — availability varies by VCN generation/driver.

3.5 Error + stall semantics (interplay with the 2026-07-06 watchdog)

• SubmitInput → AMF_INPUT_FULL: back-pressure, NOT a wedge — drain and retry, do not reset. (Original prescription "return Err → in-place reset" was wrong, disproven on-glass 2026-07-06: at throughput-ceiling loads — 5120x1440@240 P010 on the lab iGPU — INPUT_FULL → reset → forced IDR → a bigger keyframe → worse overload → a ~320 ms reset/IDR cascade, strictly worse than the libavcodec path's 16-deep input queue riding it out as latency. The log showed dozens of submit failed … AMF_INPUT_FULL … rebuilt in place and zero watchdog stalls.) The shipped handling: submit bounds in-flight surfaces below the input ring depth (pending.len() < RING) by draining finished AUs (buffered in a ready deque for poll, FIFO-preserved) to free a slot before reusing it, and treats a stray INPUT_FULL from SubmitInput the same way (drain + retry the surface). Only a drain that makes NO progress for a bounded budget (INPUT_DRAIN_BUDGET, 200 ms — well under the session watchdog's ~2 s) is a genuine wedge that escalates to Err → the in-place reset. This also closed a latent corruption: the old path let in-flight grow to AMF's internal input queue limit (16) against a ring of 4, so surfaces referenced ring slots already overwritten — the reset masked it. Any other non-OK SubmitInput result: Err.
• QueryOutput → AMF_REPEAT: keep spinning within the poll budget, then Ok(None). AMF_EOF: Ok(None) after flush. Anything else: Err (the loop's poll-error path resets).
• Implement Encoder::reset() natively: encoder->Drain/Flush, Terminate(), re-Init on the same context (fall back to full context teardown if re-Init fails). Cheaper and more targeted than the ffmpeg path's drop-and-lazily-reopen.
• caps(): supports_rfi: false (AMF has no NVENC-style reference invalidation — intra-refresh is the substitute), intra_refresh: true once Phase 2 lands (this flag is what makes the session glue rate-limit client keyframe requests — see the IDR_WINDOW logic in punktfunk1.rs), supports_hdr_metadata: true once the SEI lands, chroma_444: false permanently (VCN hardware does not encode 4:4:4; probe_can_encode_444 stays false — this is not an FFmpeg limitation).

3.6 Encoder trait contract (do not break)

From encode.rs + the punktfunk1 loop: AUs must come out FIFO in submit order (inflight.pop_front() pairs with poll order); data is Annex-B with in-band headers on IDRs (both a playable ES and self-contained wire AUs); poll returning Ok(None) is legal and watchdog-arbitrated; submit must never block indefinitely; after flush(), poll drains remaining AUs then returns Ok(None); single encode thread owns the encoder (manual unsafe impl Send with the same proof shape as FfmpegWinEncoder).

4. Integration seams (exact)

• encode.rs::open_video_backend, WindowsBackend::Amf arm: try amf::AmfEncoder::open first; on Err, tracing::warn! + fall back to ffmpeg_win (when the amf-qsv feature is built) — the same graceful-degrade shape as zero-copy→system today. Escape hatch: PUNKTFUNK_AMF_FFMPEG=1 skips the native path (field triage). Phase 3 deletes both the fallback arm and the hatch.
• resolved_backend_label / crate::gpu session record: new label "amf" stays (the mgmt API shows the same name; add "amf-ffmpeg" only if the fallback fires, so field logs distinguish the paths).
• probe_can_encode (GameStream codec advertisement) and windows_codec_support: replace the ffmpeg open-probe with a native factory probe (CreateComponent per codec on the selected adapter) once Phase 2 lands; cache shape stays.
• can_encode_444: unchanged (false).
• The encode-stall watchdog and Encoder::reset (punktfunk1.rs): unchanged — it remains the backstop for in-FFI hangs the native path can't self-detect.

5. Validation plan (this box has an AMD iGPU — use it)

Baseline first, on the ffmpeg path (already deployed 2026-07-06 with the watchdog): a long session on the iGPU with PUNKTFUNK_PERF=1, record wait_us_p50/p99, encode_us, client-measured latency, and whether the watchdog ever fires. Then per phase:

1. Open/probe smoke per codec (AVC, HEVC, HEVC-10) on the iGPU. DONE — the gated live tests in amf.rs (amf_encode_live_smoke AVC+HEVC+AV1-probe, amf_hdr_encode_live_smoke, amf_native_probe_live, amf_intra_refresh_property_live) pass on the lab Ryzen iGPU (VCN3/RDNA2): both codecs across a native reset(), HEVC Main10 IDR carrying the mastering(137)+CLL(144) SEIs byte-verified, intra-refresh property accepted, probe honestly h264/h265=true, av1=false.
2. A/B the encode latency: expect encode_us p50 ~2 frame periods → ≤ 1 frame period (see §3.3 for why wait_us is the wrong metric on the ffmpeg side). MEASURED 2026-07-06 by the gated amf_latency_ab_bench (PUNKTFUNK_AMF_BENCH=1, 1080p60 HEVC, 180 paced frames, same D3D11 NV12 input to both encoders, lab iGPU, debug build): native encode_us p50 5.18 ms (0.31 frame periods) / p99 5.81 ms vs libavcodec-AMF p50 16.9 ms (1.01 frame periods) / p99 17.5 ms — 3.3× lower, ~11.7 ms/frame saved, and the native path is decisively sub-frame (the ~2-frame hold that used to live in encode_us is gone). Note the ffmpeg baseline came in at ~1 frame period, not the ~2 this plan projected: the shipping ffmpeg config already sets AMF latency=true (a ~1-frame hold), so the realized win is 3.3× / ~12 ms rather than the ~30 ms projected against an un-tuned 2-frame baseline; direction and sub-frame collapse are exactly as §3.3 described. Release builds should show a lower native number still (debug charges host-side surface-create + copy-submit into the 5.18 ms). Zero-copy baseline for the input side already measured 2026-07-06 on the lab iGPU (1080p120 HDR P010): submit_us p50 2.7–2.9 ms (system readback) → 0.26 ms (zero-copy D3D11), p99 6.6 ms → 0.5 ms.
3. Behavior: IDR on connect; mode switch mid-stream; HDR session (PQ VUI + 0xCE convergence); client keyframe-request recovery; encoder reset() under an injected failure; ≥30 min soak for the freeze class (watchdog log line encode stall detected must NOT appear).
4. Driver matrix beyond the lab box is field data: VCN1 (Raven) through VCN4/5 differ in preset support — the optional-property tolerance in §3.4 is what absorbs this.

6. Risks / open questions

• Vtable mirroring correctness is the concentrated risk: pin one AMF header version in a comment, mirror minimally, and unit-test AMFQueryVersion/AMFInit + a headless CreateComponent probe (skips cleanly on non-AMD boxes, like the NVENC live-gated tests).
• Per-frame CreateSurfaceFromDX11Native allocation churn — if it shows up in profiles, AMF supports pre-created surface pools; start simple.
• AV1 is RDNA3+; probe, never assume (same rule as everywhere in this codebase).
• Hybrid boxes: context must init on the selected adapter's device (the capture device) — inherited for free by taking the capturer's device, but test with the web-console GPU preference pointed at each GPU.
• The AMF runtime ships with the AMD driver, not with us — a missing/ancient amfrt64.dll must produce a clean "install/update the AMD driver" error at open, then fall back (Phase 1) or fail the session with that message (Phase 3).

7. Phasing

Phase	Scope	Exit criterion	Status
1	FFI layer + AVC/HEVC (SDR + 10-bit HDR), bounded poll, native reset(), dispatch with ffmpeg fallback + PUNKTFUNK_AMF_FFMPEG hatch	§5.2–5.3 pass on the lab iGPU	DONE 2026-07-06 (§5.2 measured; §5.3 on-glass in progress)
2	Intra-refresh (caps().intra_refresh), in-band HDR SEI (supports_hdr_metadata), AV1, native codec probe	field-validated on ≥2 VCN generations	CODE DONE 2026-07-06 (lab VCN3 only; AV1/RDNA3 + 2nd VCN gen still owed)
3	Delete the ffmpeg-AMF dispatch arm + hatch; FFmpeg remains QSV-only	one release of field silence on the fallback label	DONE 2026-07-06 — see the gate note below

Phase 3 gate note (honesty): the stated exit criterion (one release of field silence on the fallback label) was NOT met — Phase 3 was cut the same day the native path was written, on explicit direction, alongside a live macOS-client on-glass test. What Phase 3 removed: the WindowsBackend::Amf libavcodec fallback arm, the PUNKTFUNK_AMF_FFMPEG hatch, and the AMF→ffmpeg routes in windows_codec_support / can_encode_444. AMD dispatch, codec advertisement, and the 4:4:4 answer are all native-only now; FFmpeg (ffmpeg_win.rs) is reached only for QSV in production (its WinVendor::Amf machinery is retained solely as the amf_latency_ab_bench comparator, not deleted — excising it would churn the Intel-unvalidated QSV code for no gain). Residual risk this pre-emption carries: with the ffmpeg readback path gone, an AMD box whose capturer can't produce video-processor NV12/P010 (falls back to Bgra/Rgb10a2, or hands DDA CPU frames) now fails the session instead of degrading — the design's answer is the native AMFVideoConverter front-end (§3.2), owed if that fallback is ever seen in the field. Not observed on lab hardware (the VP yields NV12/P010). Reverting Phase 3 is a small, localized diff if field data turns up trouble.