punktfunk/design/native-amf-encoder.md

Native AMF encoder — handoff design

Status: PROPOSED (handoff, 2026-07-06). Design + decision record for replacing the libavcodec *_amf path with a direct AMF SDK encoder on Windows AMD (encode/windows/amf.rs, the AMD analogue of the direct-SDK encode/windows/nvenc.rs). Written for an implementer without prior context; every in-repo seam is named. Companion context: the encode-stall watchdog + Encoder::reset() (punktfunk1.rs / encode.rs, shipped 2026-07-06) and the AMF/QSV backend module docs in encode/windows/ffmpeg_win.rs.

1. Why (measured, not speculative)

Three independent reasons, in order of weight:

1. The libavcodec AMF wrapper's structural ~2-frame output hold. hevc_amf/av1_amf need frame N+2 submitted before they release frame N's AU. Measured on the Ryzen 7000 iGPU (VCN, 720p60): encode→retrieve 36 ms p50, dead-stable, invariant across pipeline depth 1/2, every usage preset, and any spin budget (a 150 ms poll spin provably never produced the owed AU — it pegged at exactly 150 ms). See the poll doc comment in ffmpeg_win.rs. The direct-SDK NVENC path retrieves in ~1–2 ms. At 60 Hz this is ~33 ms of pure pipeline latency no FFmpeg-side knob can remove; at 120 Hz it is two whole frame budgets.
2. Silent driver wedges surface as forever-EAGAIN, not errors. The field failure (AMD/Intel streams freezing after ~3–5 min) was invisible because the wrapper's only "not ready" signal is EAGAIN, indistinguishable from a healthy pipeline warming up. The 2026-07-06 watchdog converts that into a bounded rebuild + IDR, but it is a safety net with a ~2 s detection floor. The AMF runtime itself returns typed AMF_RESULT codes (AMF_INPUT_FULL, device-lost, etc.) — a native path sees the wedge on the frame it happens.
3. Feature gaps libavcodec cannot express. No intra-refresh wave (every FEC-unrecoverable loss is answered with a full IDR — the 20–40× frame-size spike the Linux NVENC intra-refresh mode exists to avoid), no in-band HDR mastering SEI (EncoderCaps::supports_hdr_metadata is NVENC-only today), coarse per-frame control.

2. The decision: drop FFmpeg for AMF, keep it for QSV

Native AMF replaces the libavcodec AMF path (phased, §7). FFmpeg stays for QSV.

• QSV via libavcodec with async_depth=1 + low_power VDEnc is already near the hardware latency floor; a direct libvpl port would buy little for its cost. Revisit only if Intel field data shows a QSV-specific gap (separate doc if so).
• Because QSV stays on FFmpeg, the FFmpeg DLLs keep shipping and the amf-qsv build feature keeps existing. Dropping FFmpeg entirely is therefore not on the table here — "drop" means: the AMF dispatch stops going through it once the native path is validated.
• During bring-up the ffmpeg-AMF path remains as an automatic open-failure fallback and an explicit escape hatch (§7), then its AMF dispatch is deleted in Phase 3. Two permanently maintained AMF paths would double the driver-matrix burden, and the one we'd keep "for safety" is precisely the one with the wedge/latency pathology.

3. Architecture

New module crates/punktfunk-host/src/encode/windows/amf.rs implementing crate::encode::Encoder, compiled unconditionally on Windows (no new build-time dependency and no new cargo feature): the AMF runtime is loaded at runtime from the driver-installed amfrt64.dll, exactly as nvenc.rs loads nvEncodeAPI64.dll (load_api). A box without an AMD driver simply fails the open and the dispatch falls through. In-tree FFI decl module (amf_sys submodule or #[repr(C)] blocks in-file, mirroring the small interface subset we use) — model it on how ffmpeg_win.rs mirrors AVD3D11VADeviceContext rather than pulling a binding crate (none is maintained).

3.1 FFI strategy (the load-bearing detail)

The AMF public headers (GPUOpen AMF/amf/public/include) define C-compatible vtable structs for every interface (AMFFactoryVtbl, AMFContextVtbl, AMFComponentVtbl, AMFSurfaceVtbl, AMFDataVtbl, AMFBufferVtbl, AMFVariantStruct, …) — this is not a guess: FFmpeg's amfenc.c is plain C and drives AMF exclusively through those vtables, so the C ABI is the stable, supported surface. Mirror only what we call:

• Entry points: GetProcAddress("AMFQueryVersion") (gate: runtime ≥ the pinned AMF_FULL_VERSION we mirror headers from) and GetProcAddress("AMFInit") → AMFFactory*.
• factory->CreateContext → context->InitDX11(capturer_device, AMF_DX11_1) — the capturer's own ID3D11Device, same-device requirement as every other backend (the capture textures are not shared-handle; see the ensure_inner_d3d11 rebind logic in ffmpeg_win.rs for the device-change lifecycle to replicate).
• factory->CreateComponent(context, name) with AMFVideoEncoderVCE_AVC / AMFVideoEncoder_HEVC / AMFVideoEncoder_AV1 → encoder->Init(AMF_SURFACE_NV12|P010, w, h).
• Per-frame: context->CreateSurfaceFromDX11Native(texture, &surface, observer) → per-surface properties (pts via SetPts, forced-IDR picture type) → encoder->SubmitInput(surface); retrieve via encoder->QueryOutput(&data) (AMF_REPEAT = not ready), AMFBuffer::GetNative/GetSize → EncodedFrame.
• Every mirrored struct/call carries a // SAFETY: proof — the whole encode module tree is under #![deny(clippy::undocumented_unsafe_blocks)] (unsafe-proof program).

Reference implementations to crib from (read both before writing FFI): FFmpeg amfenc.c (the C vtbl usage, property plumbing, result-code handling) and OBS plugins/obs-ffmpeg/texture-amf.cpp (D3D11 texture submission + low-latency streaming config, C++ but the call sequence is what matters). Verify every property name against the pinned SDK headers — names below are from those references and must not be trusted blind.

3.2 Input path (zero-copy by construction)

Own a small D3D11 texture ring (NV12 or P010, D3D11_BIND_RENDER_TARGET | SHADER_RESOURCE, size = pipeline_depth + 2), CopySubresourceRegion the captured texture into the next slot (GPU-local, same pattern as ZeroCopyInner::submit), wrap the slot with CreateSurfaceFromDX11Native, submit. The copy decouples the encoder from the capturer's rotating IDD ring; do NOT wrap the capturer's texture directly. This makes PUNKTFUNK_ZEROCOPY irrelevant for native AMF — there is no readback path to fall back to. Handle the capturer's video-processor format fallback (Bgra/Rgb10a2 instead of NV12/P010 — see pool_mismatch in ffmpeg_win.rs) by returning an open/submit error in Phase 1 so dispatch falls back to the ffmpeg path; an AMFVideoConverter front-end is a Phase 2 option if that fallback ever fires in the field. FramePayload::Cpu (DDA without video processor): same treatment — ffmpeg fallback in Phase 1.

3.3 Retrieval model

Bounded-blocking poll, the vaapi.rs::poll model: after SubmitInput, spin QueryOutput with ~250 µs sleeps up to a budget of min(3/4 frame interval, 12 ms); on expiry return Ok(None) (the session loop keeps the frame in flight and the watchdog arbitrates wedges). VCN encode at streaming settings is ~1–5 ms, so the AU ships the same tick — this is where the ~2-frame hold dies. Expected observable — measure encode_us (submit→AU, in FrameMsg/the web-console stats), not wait_us: on the ffmpeg path the hold hides in encode_us (~2 frame periods) because its non-blocking poll returns EAGAIN in ~2 µs; on the native bounded poll the ASIC wait becomes visible as a few ms of wait_us while encode_us collapses to ~1 frame period or less.

3.4 Encoder configuration (initial property set)

Mirror the intent of the ffmpeg opts block in open_win_encoder (ffmpeg_win.rs:216-247). AVC names given; HEVC/AV1 have _HEVC_/_AV1_ twins — check headers:

Intent	AMF property (verify!)
usage preset (keep PUNKTFUNK_AMF_USAGE mapping)	AMF_VIDEO_ENCODER_USAGE = ULTRA_LOW_LATENCY (default)
CBR, target==peak	RATE_CONTROL_METHOD=CBR, TARGET_BITRATE, PEAK_BITRATE
1-frame VBV (keep PUNKTFUNK_VBV_FRAMES)	VBV_BUFFER_SIZE
HRD + no filler	ENFORCE_HRD=true, FILLER_DATA_ENABLE=false
latency-first quality	QUALITY_PRESET=SPEED
no B-frames (AVC)	B_PIC_PATTERN=0
infinite GOP	IDR_PERIOD=0 (HEVC: GOP_SIZE/NUM_GOPS_PER_IDR — check)
low-latency submission	LOWLATENCY_MODE=true (newer SDKs)
in-band VPS/SPS/PPS on IDR (wire contract: EncodedFrame doc)	HEVC HEADER_INSERTION_MODE=IDR_ALIGNED; AVC HEADER_INSERTION_SPACING — check
SDR/HDR VUI	FULL_RANGE_COLOR=false + color primaries/transfer/matrix props (BT.709 vs BT.2020-PQ, mirroring open_win_encoder)
10-bit	COLOR_BIT_DEPTH=10 + P010 surfaces
per-frame forced IDR	on the input surface: AMF_VIDEO_ENCODER_FORCE_PICTURE_TYPE=IDR
intra-refresh wave (Phase 2)	AVC INTRA_REFRESH_NUM_MBS_PER_SLOT; HEVC CTB twin — check
HDR mastering SEI (Phase 2)	HEVC INPUT_HDR_METADATA (AMFHDRMetadata buffer)

SetProperty failures on optional properties (LOWLATENCY_MODE, intra-refresh) must be log-and-continue, not fatal — availability varies by VCN generation/driver.

3.5 Error + stall semantics (interplay with the 2026-07-06 watchdog)

• SubmitInput → AMF_INPUT_FULL: return Err from submit — the session loop's submit-failure path runs the bounded in-place reset. Any other non-OK result: Err.
• QueryOutput → AMF_REPEAT: keep spinning within the poll budget, then Ok(None). AMF_EOF: Ok(None) after flush. Anything else: Err (the loop's poll-error path resets).
• Implement Encoder::reset() natively: encoder->Drain/Flush, Terminate(), re-Init on the same context (fall back to full context teardown if re-Init fails). Cheaper and more targeted than the ffmpeg path's drop-and-lazily-reopen.
• caps(): supports_rfi: false (AMF has no NVENC-style reference invalidation — intra-refresh is the substitute), intra_refresh: true once Phase 2 lands (this flag is what makes the session glue rate-limit client keyframe requests — see the IDR_WINDOW logic in punktfunk1.rs), supports_hdr_metadata: true once the SEI lands, chroma_444: false permanently (VCN hardware does not encode 4:4:4; probe_can_encode_444 stays false — this is not an FFmpeg limitation).

3.6 Encoder trait contract (do not break)

From encode.rs + the punktfunk1 loop: AUs must come out FIFO in submit order (inflight.pop_front() pairs with poll order); data is Annex-B with in-band headers on IDRs (both a playable ES and self-contained wire AUs); poll returning Ok(None) is legal and watchdog-arbitrated; submit must never block indefinitely; after flush(), poll drains remaining AUs then returns Ok(None); single encode thread owns the encoder (manual unsafe impl Send with the same proof shape as FfmpegWinEncoder).

4. Integration seams (exact)

• encode.rs::open_video_backend, WindowsBackend::Amf arm: try amf::AmfEncoder::open first; on Err, tracing::warn! + fall back to ffmpeg_win (when the amf-qsv feature is built) — the same graceful-degrade shape as zero-copy→system today. Escape hatch: PUNKTFUNK_AMF_FFMPEG=1 skips the native path (field triage). Phase 3 deletes both the fallback arm and the hatch.
• resolved_backend_label / crate::gpu session record: new label "amf" stays (the mgmt API shows the same name; add "amf-ffmpeg" only if the fallback fires, so field logs distinguish the paths).
• probe_can_encode (GameStream codec advertisement) and windows_codec_support: replace the ffmpeg open-probe with a native factory probe (CreateComponent per codec on the selected adapter) once Phase 2 lands; cache shape stays.
• can_encode_444: unchanged (false).
• The encode-stall watchdog and Encoder::reset (punktfunk1.rs): unchanged — it remains the backstop for in-FFI hangs the native path can't self-detect.

5. Validation plan (this box has an AMD iGPU — use it)

Baseline first, on the ffmpeg path (already deployed 2026-07-06 with the watchdog): a long session on the iGPU with PUNKTFUNK_PERF=1, record wait_us_p50/p99, encode_us, client-measured latency, and whether the watchdog ever fires. Then per phase:

1. Open/probe smoke per codec (AVC, HEVC, HEVC-10) on the iGPU.
2. A/B the encode latency: expect encode_us p50 ~2 frame periods → ≤ 1 frame period (see §3.3 for why wait_us is the wrong metric on the ffmpeg side); client capture→glass p50 drops by ~30 ms at 60 Hz / ~17 ms at 120 Hz. Zero-copy baseline for the input side already measured 2026-07-06 on the lab iGPU (1080p120 HDR P010): submit_us p50 2.7–2.9 ms (system readback) → 0.26 ms (zero-copy D3D11), p99 6.6 ms → 0.5 ms.
3. Behavior: IDR on connect; mode switch mid-stream; HDR session (PQ VUI + 0xCE convergence); client keyframe-request recovery; encoder reset() under an injected failure; ≥30 min soak for the freeze class (watchdog log line encode stall detected must NOT appear).
4. Driver matrix beyond the lab box is field data: VCN1 (Raven) through VCN4/5 differ in preset support — the optional-property tolerance in §3.4 is what absorbs this.

6. Risks / open questions

• Vtable mirroring correctness is the concentrated risk: pin one AMF header version in a comment, mirror minimally, and unit-test AMFQueryVersion/AMFInit + a headless CreateComponent probe (skips cleanly on non-AMD boxes, like the NVENC live-gated tests).
• Per-frame CreateSurfaceFromDX11Native allocation churn — if it shows up in profiles, AMF supports pre-created surface pools; start simple.
• AV1 is RDNA3+; probe, never assume (same rule as everywhere in this codebase).
• Hybrid boxes: context must init on the selected adapter's device (the capture device) — inherited for free by taking the capturer's device, but test with the web-console GPU preference pointed at each GPU.
• The AMF runtime ships with the AMD driver, not with us — a missing/ancient amfrt64.dll must produce a clean "install/update the AMD driver" error at open, then fall back (Phase 1) or fail the session with that message (Phase 3).

7. Phasing

Phase	Scope	Exit criterion
1	FFI layer + AVC/HEVC (SDR + 10-bit HDR), bounded poll, native reset(), dispatch with ffmpeg fallback + PUNKTFUNK_AMF_FFMPEG hatch	§5.2–5.3 pass on the lab iGPU
2	Intra-refresh (caps().intra_refresh), in-band HDR SEI (supports_hdr_metadata), AV1, native codec probe	field-validated on ≥2 VCN generations
3	Delete the ffmpeg-AMF dispatch arm + hatch; FFmpeg remains QSV-only	one release of field silence on the fallback label