punktfunk/design/native-amf-encoder.md

# Native AMF encoder — handoff design

> **Status: PROPOSED (handoff, 2026-07-06).** Design + decision record for replacing the
> libavcodec `*_amf` path with a direct AMF SDK encoder on Windows AMD
> (`encode/windows/amf.rs`, the AMD analogue of the direct-SDK `encode/windows/nvenc.rs`).
> Written for an implementer without prior context; every in-repo seam is named. Companion
> context: the encode-stall watchdog + `Encoder::reset()` (punktfunk1.rs / encode.rs, shipped
> 2026-07-06) and the AMF/QSV backend module docs in `encode/windows/ffmpeg_win.rs`.

## 1. Why (measured, not speculative)

Three independent reasons, in order of weight:

1. **The libavcodec AMF wrapper's structural ~2-frame output hold.** `hevc_amf`/`av1_amf`
   need frame N+2 submitted before they release frame N's AU. Measured on the Ryzen 7000
   iGPU (VCN, 720p60): encode→retrieve **36 ms p50, dead-stable**, invariant across pipeline
   depth 1/2, every `usage` preset, and any spin budget (a 150 ms poll spin provably never
   produced the owed AU — it pegged at exactly 150 ms). See the `poll` doc comment in
   `ffmpeg_win.rs`. The direct-SDK NVENC path retrieves in ~1–2 ms. At 60 Hz this is ~33 ms
   of pure pipeline latency no FFmpeg-side knob can remove; at 120 Hz it is two whole frame
   budgets.
2. **Silent driver wedges surface as forever-EAGAIN, not errors.** The field failure
   (AMD/Intel streams freezing after ~3–5 min) was invisible because the wrapper's only
   "not ready" signal is EAGAIN, indistinguishable from a healthy pipeline warming up. The
   2026-07-06 watchdog converts that into a bounded rebuild + IDR, but it is a safety net
   with a ~2 s detection floor. The AMF runtime itself returns typed `AMF_RESULT` codes
   (`AMF_INPUT_FULL`, device-lost, etc.) — a native path sees the wedge on the frame it
   happens.
3. **Feature gaps libavcodec cannot express.** No intra-refresh wave (every
   FEC-unrecoverable loss is answered with a full IDR — the 20–40× frame-size spike the
   Linux NVENC intra-refresh mode exists to avoid), no in-band HDR mastering SEI
   (`EncoderCaps::supports_hdr_metadata` is NVENC-only today), coarse per-frame control.

## 2. The decision: drop FFmpeg for AMF, keep it for QSV

**Native AMF replaces the libavcodec AMF path (phased, §7). FFmpeg stays for QSV.**

- QSV via libavcodec with `async_depth=1` + `low_power` VDEnc is already near the hardware
  latency floor; a direct libvpl port would buy little for its cost. Revisit only if Intel
  field data shows a QSV-specific gap (separate doc if so).
- Because QSV stays on FFmpeg, the FFmpeg DLLs keep shipping and the `amf-qsv` build
  feature keeps existing. Dropping FFmpeg *entirely* is therefore not on the table here —
  "drop" means: the AMF dispatch stops going through it once the native path is validated.
- During bring-up the ffmpeg-AMF path remains as an automatic open-failure fallback and an
  explicit escape hatch (§7), then its AMF dispatch is deleted in Phase 3. Two permanently
  maintained AMF paths would double the driver-matrix burden, and the one we'd keep "for
  safety" is precisely the one with the wedge/latency pathology.

## 3. Architecture

New module `crates/punktfunk-host/src/encode/windows/amf.rs` implementing
`crate::encode::Encoder`, compiled unconditionally on Windows (**no new build-time
dependency and no new cargo feature**): the AMF runtime is loaded at runtime from the
driver-installed `amfrt64.dll`, exactly as `nvenc.rs` loads `nvEncodeAPI64.dll`
(`load_api`). A box without an AMD driver simply fails the open and the dispatch falls
through. In-tree FFI decl module (`amf_sys` submodule or `#[repr(C)]` blocks in-file,
mirroring the small interface subset we use) — model it on how `ffmpeg_win.rs` mirrors
`AVD3D11VADeviceContext` rather than pulling a binding crate (none is maintained).

### 3.1 FFI strategy (the load-bearing detail)

The AMF public headers (GPUOpen `AMF/amf/public/include`) define **C-compatible vtable
structs** for every interface (`AMFFactoryVtbl`, `AMFContextVtbl`, `AMFComponentVtbl`,
`AMFSurfaceVtbl`, `AMFDataVtbl`, `AMFBufferVtbl`, `AMFVariantStruct`, …) — this is not a
guess: FFmpeg's `amfenc.c` is plain C and drives AMF exclusively through those vtables, so
the C ABI is the stable, supported surface. Mirror only what we call:

- Entry points: `GetProcAddress("AMFQueryVersion")` (gate: runtime ≥ the pinned
  `AMF_FULL_VERSION` we mirror headers from) and `GetProcAddress("AMFInit")` →
  `AMFFactory*`.
- `factory->CreateContext` → `context->InitDX11(capturer_device, AMF_DX11_1)` — **the
  capturer's own `ID3D11Device`**, same-device requirement as every other backend (the
  capture textures are not shared-handle; see the `ensure_inner_d3d11` rebind logic in
  `ffmpeg_win.rs` for the device-change lifecycle to replicate).
- `factory->CreateComponent(context, name)` with `AMFVideoEncoderVCE_AVC` /
  `AMFVideoEncoder_HEVC` / `AMFVideoEncoder_AV1` → `encoder->Init(AMF_SURFACE_NV12|P010,
  w, h)`.
- Per-frame: `context->CreateSurfaceFromDX11Native(texture, &surface, observer)` →
  per-surface properties (pts via `SetPts`, forced-IDR picture type) →
  `encoder->SubmitInput(surface)`; retrieve via `encoder->QueryOutput(&data)`
  (`AMF_REPEAT` = not ready), `AMFBuffer::GetNative/GetSize` → `EncodedFrame`.
- Every mirrored struct/call carries a `// SAFETY:` proof — the whole encode module tree is
  under `#![deny(clippy::undocumented_unsafe_blocks)]` (unsafe-proof program).

Reference implementations to crib from (read both before writing FFI): FFmpeg `amfenc.c`
(the C vtbl usage, property plumbing, result-code handling) and OBS
`plugins/obs-ffmpeg/texture-amf.cpp` (D3D11 texture submission + low-latency streaming
config, C++ but the call sequence is what matters). **Verify every property name against
the pinned SDK headers** — names below are from those references and must not be trusted
blind.

### 3.2 Input path (zero-copy by construction)

Own a small D3D11 texture ring (NV12 or P010, `D3D11_BIND_RENDER_TARGET |
SHADER_RESOURCE`, size = `pipeline_depth + 2`), `CopySubresourceRegion` the captured
texture into the next slot (GPU-local, same pattern as `ZeroCopyInner::submit`), wrap the
slot with `CreateSurfaceFromDX11Native`, submit. The copy decouples the encoder from the
capturer's rotating IDD ring; do NOT wrap the capturer's texture directly. This makes
`PUNKTFUNK_ZEROCOPY` irrelevant for native AMF — there is no readback path to fall back
to. Handle the capturer's video-processor format fallback (`Bgra`/`Rgb10a2` instead of
NV12/P010 — see `pool_mismatch` in `ffmpeg_win.rs`) by returning an open/submit error in
Phase 1 so dispatch falls back to the ffmpeg path; an AMFVideoConverter front-end is a
Phase 2 option if that fallback ever fires in the field. `FramePayload::Cpu` (DDA without
video processor): same treatment — ffmpeg fallback in Phase 1.

### 3.3 Retrieval model

Bounded-blocking poll, the `vaapi.rs::poll` model: after `SubmitInput`, spin
`QueryOutput` with ~250 µs sleeps up to a budget of `min(3/4 frame interval, 12 ms)`; on
expiry return `Ok(None)` (the session loop keeps the frame in flight and the watchdog
arbitrates wedges). VCN encode at streaming settings is ~1–5 ms, so the AU ships the same
tick — this is where the ~2-frame hold dies. Expected observable — measure **`encode_us`
(submit→AU, in `FrameMsg`/the web-console stats), not `wait_us`**: on the ffmpeg path the
hold hides in `encode_us` (~2 frame periods) because its non-blocking poll returns EAGAIN
in ~2 µs; on the native bounded poll the ASIC wait becomes visible as a few ms of
`wait_us` while `encode_us` collapses to ~1 frame period or less.

### 3.4 Encoder configuration (initial property set)

Mirror the intent of the ffmpeg opts block in `open_win_encoder` (`ffmpeg_win.rs:216-247`).
AVC names given; HEVC/AV1 have `_HEVC_`/`_AV1_` twins — check headers:

| Intent | AMF property (verify!) |
| --- | --- |
| usage preset (keep `PUNKTFUNK_AMF_USAGE` mapping) | `AMF_VIDEO_ENCODER_USAGE` = `ULTRA_LOW_LATENCY` (default) |
| CBR, target==peak | `RATE_CONTROL_METHOD=CBR`, `TARGET_BITRATE`, `PEAK_BITRATE` |
| 1-frame VBV (keep `PUNKTFUNK_VBV_FRAMES`) | `VBV_BUFFER_SIZE` |
| HRD + no filler | `ENFORCE_HRD=true`, `FILLER_DATA_ENABLE=false` |
| latency-first quality | `QUALITY_PRESET=SPEED` |
| no B-frames (AVC) | `B_PIC_PATTERN=0` |
| infinite GOP | `IDR_PERIOD=0` (HEVC: `GOP_SIZE`/`NUM_GOPS_PER_IDR` — check) |
| low-latency submission | `LOWLATENCY_MODE=true` (newer SDKs) |
| in-band VPS/SPS/PPS on IDR (wire contract: `EncodedFrame` doc) | HEVC `HEADER_INSERTION_MODE=IDR_ALIGNED`; AVC `HEADER_INSERTION_SPACING` — check |
| SDR/HDR VUI | `FULL_RANGE_COLOR=false` + color primaries/transfer/matrix props (BT.709 vs BT.2020-PQ, mirroring `open_win_encoder`) |
| 10-bit | `COLOR_BIT_DEPTH=10` + P010 surfaces |
| per-frame forced IDR | on the input surface: `AMF_VIDEO_ENCODER_FORCE_PICTURE_TYPE=IDR` |
| intra-refresh wave (Phase 2) | AVC `INTRA_REFRESH_NUM_MBS_PER_SLOT`; HEVC CTB twin — check |
| HDR mastering SEI (Phase 2) | HEVC `INPUT_HDR_METADATA` (`AMFHDRMetadata` buffer) |

`SetProperty` failures on *optional* properties (LOWLATENCY_MODE, intra-refresh) must be
log-and-continue, not fatal — availability varies by VCN generation/driver.

### 3.5 Error + stall semantics (interplay with the 2026-07-06 watchdog)

- `SubmitInput` → `AMF_INPUT_FULL`: return `Err` from `submit` — the session loop's
  submit-failure path runs the bounded in-place reset. Any other non-OK result: `Err`.
- `QueryOutput` → `AMF_REPEAT`: keep spinning within the poll budget, then `Ok(None)`.
  `AMF_EOF`: `Ok(None)` after flush. Anything else: `Err` (the loop's poll-error path
  resets).
- Implement `Encoder::reset()` natively: `encoder->Drain/Flush`, `Terminate()`, re-`Init`
  on the same context (fall back to full context teardown if re-Init fails). Cheaper and
  more targeted than the ffmpeg path's drop-and-lazily-reopen.
- `caps()`: `supports_rfi: false` (AMF has no NVENC-style reference invalidation —
  intra-refresh is the substitute), `intra_refresh: true` once Phase 2 lands (this flag is
  what makes the session glue rate-limit client keyframe requests — see the `IDR_WINDOW`
  logic in punktfunk1.rs), `supports_hdr_metadata: true` once the SEI lands,
  `chroma_444: false` **permanently** (VCN hardware does not encode 4:4:4;
  `probe_can_encode_444` stays false — this is not an FFmpeg limitation).

### 3.6 Encoder trait contract (do not break)

From `encode.rs` + the punktfunk1 loop: AUs must come out FIFO in submit order
(`inflight.pop_front()` pairs with poll order); `data` is Annex-B with in-band headers on
IDRs (both a playable ES and self-contained wire AUs); `poll` returning `Ok(None)` is
legal and watchdog-arbitrated; `submit` must never block indefinitely; after `flush()`,
`poll` drains remaining AUs then returns `Ok(None)`; single encode thread owns the
encoder (manual `unsafe impl Send` with the same proof shape as `FfmpegWinEncoder`).

## 4. Integration seams (exact)

- `encode.rs::open_video_backend`, `WindowsBackend::Amf` arm: try `amf::AmfEncoder::open`
  first; on `Err`, `tracing::warn!` + fall back to `ffmpeg_win` (when the `amf-qsv`
  feature is built) — the same graceful-degrade shape as zero-copy→system today. Escape
  hatch: `PUNKTFUNK_AMF_FFMPEG=1` skips the native path (field triage). Phase 3 deletes
  both the fallback arm and the hatch.
- `resolved_backend_label` / `crate::gpu` session record: new label `"amf"` stays (the
  mgmt API shows the same name; add `"amf-ffmpeg"` only if the fallback fires, so field
  logs distinguish the paths).
- `probe_can_encode` (GameStream codec advertisement) and `windows_codec_support`: replace
  the ffmpeg open-probe with a native factory probe (`CreateComponent` per codec on the
  selected adapter) once Phase 2 lands; cache shape stays.
- `can_encode_444`: unchanged (`false`).
- The encode-stall watchdog and `Encoder::reset` (punktfunk1.rs): unchanged — it remains
  the backstop for in-FFI hangs the native path can't self-detect.

## 5. Validation plan (this box has an AMD iGPU — use it)

Baseline first, on the ffmpeg path (already deployed 2026-07-06 with the watchdog): a
long session on the iGPU with `PUNKTFUNK_PERF=1`, record `wait_us_p50/p99`,
`encode_us`, client-measured latency, and whether the watchdog ever fires. Then per phase:

1. Open/probe smoke per codec (AVC, HEVC, HEVC-10) on the iGPU.
2. A/B the encode latency: expect `encode_us` p50 ~2 frame periods → ≤ 1 frame period
   (see §3.3 for why `wait_us` is the wrong metric on the ffmpeg side); client
   capture→glass p50 drops by ~30 ms at 60 Hz / ~17 ms at 120 Hz. Zero-copy baseline for
   the input side already measured 2026-07-06 on the lab iGPU (1080p120 HDR P010):
   `submit_us` p50 2.7–2.9 ms (system readback) → **0.26 ms** (zero-copy D3D11), p99
   6.6 ms → 0.5 ms.
3. Behavior: IDR on connect; mode switch mid-stream; HDR session (PQ VUI + 0xCE
   convergence); client keyframe-request recovery; encoder `reset()` under an injected
   failure; ≥30 min soak for the freeze class (watchdog log line
   `encode stall detected` must NOT appear).
4. Driver matrix beyond the lab box is field data: VCN1 (Raven) through VCN4/5 differ in
   preset support — the optional-property tolerance in §3.4 is what absorbs this.

## 6. Risks / open questions

- **Vtable mirroring correctness** is the concentrated risk: pin one AMF header version in
  a comment, mirror minimally, and unit-test `AMFQueryVersion`/`AMFInit` + a headless
  `CreateComponent` probe (skips cleanly on non-AMD boxes, like the NVENC live-gated
  tests).
- Per-frame `CreateSurfaceFromDX11Native` allocation churn — if it shows up in profiles,
  AMF supports pre-created surface pools; start simple.
- AV1 is RDNA3+; probe, never assume (same rule as everywhere in this codebase).
- Hybrid boxes: context must init on the *selected* adapter's device (the capture
  device) — inherited for free by taking the capturer's device, but test with the
  web-console GPU preference pointed at each GPU.
- The AMF runtime ships with the AMD driver, not with us — a missing/ancient `amfrt64.dll`
  must produce a clean "install/update the AMD driver" error at open, then fall back
  (Phase 1) or fail the session with that message (Phase 3).

## 7. Phasing

| Phase | Scope | Exit criterion |
| --- | --- | --- |
| 1 | FFI layer + AVC/HEVC (SDR + 10-bit HDR), bounded poll, native `reset()`, dispatch with ffmpeg fallback + `PUNKTFUNK_AMF_FFMPEG` hatch | §5.2–5.3 pass on the lab iGPU |
| 2 | Intra-refresh (`caps().intra_refresh`), in-band HDR SEI (`supports_hdr_metadata`), AV1, native codec probe | field-validated on ≥2 VCN generations |
| 3 | Delete the ffmpeg-AMF dispatch arm + hatch; FFmpeg remains QSV-only | one release of field silence on the fallback label |