unom/punktfunk
9
0
Fork 2
Code Issues 15 Pull Requests 1 Actions 17 Packages Projects 1 Releases 57 Wiki Activity

macOS host — compile fix (Tier A), then a desktop host on ScreenCaptureKit + VideoToolbox (Tier B) #299

New Issue
Open
opened 2026-08-18 13:26:52 +00:00 by enricobuehler · 0 comments
enricobuehler commented 2026-08-18 13:26:52 +00:00
Owner
Copy Link
Copy Source

Make punktfunk-host build on macOS, then ship a macOS desktop host.

Design: design/macos-host-feasibility.md in punktfunk-planning (39130c5, on main) — investigation only, no code anywhere.

The finding that reframes this: the macOS port is four backends and a packaging story, not a host rewrite. The W6 crate extraction already left a real third-platform seam — cfg(not(any(linux, windows))) arms exist across the pf-* crates and pf-inject even ships a Backend::Unsupported placeholder. Everything portable — orchestrator, session plan, GameStream plane, punktfunk/1 native plane, pairing, mDNS, mgmt API + OpenAPI, web console, plugin runner, game library, update-check — already compiles on macOS today.

For scale: the existing per-platform backend volume is ~41k LOC Linux / ~34k LOC Windows. Tier B is estimated ~3–5k.

Tier A — make it compile (<1 day, unblocked, do this regardless)

Worth doing on its own merits even if Tiers B/C never happen, because it retires the openapi tax: cargo run -p punktfunk-host -- openapi can only run on Linux/Windows today, so api/openapi.json has to be hand-patched or regenerated on a Linux box, and mgmt::tests::openapi_document_is_complete_and_checked_in is the only signal that a mgmt-API change left the spec stale. It also makes the Mac a first-class dev box for all the portable host code above. Must not be gated on the Tier B/C decision.

Re-measured on 4ce7a3c1 (2026-08-18, arm64): cargo check -p punktfunk-host = 10 errors in 6 files; pf-capture, pf-encode, pf-inject, pf-clipboard, pf-vdisplay all check clean, unmodified.

  • audio/mic_pump.rs:172 — unresolved opus (two sites on the line). Gate the call sites to match the manifest gating.
  • hooks.rs:457 — unresolved libc. Same.
  • session_plan.rs:306-307 — crate::zerocopy, encode::linux_zero_copy_is_vaapi, encode::cursor_blend_capable. Give the three helpers non-Linux siblings.
  • gamestream/stream.rs:808 + native/compositor.rs:106 — vdisplay::cancel_pending_tv_restore; native/compositor.rs:63 — vdisplay::try_recover_session. Add no-op non-Linux siblings next to the Linux ones.
  • gamelease.rs:1062 — procscan::Scanner. New since the design doc was written (the doc measured 9 errors in 5 files on 2d3f9f86, 2026-08-01).
  • Add a macOS cargo check -p punktfunk-host CI leg. The hole count grew by one in 17 days with nobody trying — without a guard this rots straight back.

Tier B — the desktop host (~3–5k LOC, public APIs only)

No entitlement requests, no private symbols. Ordered; each step independently useful.

  • pf-capture macOS backend — a src/macos/ sibling to linux//windows/ implementing Capturer (pf-capture/src/lib.rs:21). ScreenCaptureKit is the best capture platform of the three: SCStream hands back IOSurface-backed CVPixelBuffers that go straight into VideoToolbox, so the project's zero-copy invariant is the default path rather than something to fight for — no EGL/dmabuf/CUDA import worker, no IDD driver, no compositor matrix. supports_arrival_wait() (:65) = true — SCStream's delegate callback is the frame-arrival wakeup the latency work wants, so the frame-driven encode trigger comes free. macOS arms for capturer_supports_hdr (:421-430) and capturer_supports_444 (:400/498/511).
  • pf-encode macOS backend — Encoder (enc/codec.rs:319) over VTCompressionSession + EnableLowLatencyRateControl, wired into open_video (pf-encode/src/lib.rs:163). submit/poll map naturally onto VT's async completion callback; the frame-index prediction contract in the trait docs holds unchanged. One vendor, so no backend matrix — ~1–2k vs 16.3k (Linux) / 11.5k (Windows).
  • Host audio — AudioCapturer from the same SCStream (system audio arrives alongside video); macOS arm on open_audio_capture (punktfunk-host/src/audio.rs:222/227/237). Mic uplink via AVAudioEngine, which ships a real platform AEC — directly relevant to mic-latency-and-echo.md.
  • pf-inject macOS backend — Backend::CgEvent replacing the Unsupported placeholder (pf-inject/src/lib.rs:76, dispatch at :241); InputInjector over CGEventPost (+ Accessibility grant). Absolute mapping is straightforward. Keyboard/mouse only — pads are Tier C.
  • pf-clipboard — NSPasteboard behind HostClipboard; flip the available = false fallback (pf-clipboard/src/lib.rs:98-110). No delayed-rendering complication to match Win32's.
  • Packaging — LaunchAgent plist, Developer ID signing + notarization + hardened runtime, first-run TCC onboarding (Screen Recording + Accessibility + Microphone), a packaging/macos/. The Dev ID / notarization machinery already exists for the Mac client, so this is reuse.

Deliberately not in the list: pf-vdisplay (Tier C), pf-zerocopy (nothing to do — SCK → VT is already zero-copy), pf-gpu (one vendor), the compositor matrix (no analogue).

Known ceilings: VideoToolbox has no 4:4:4; HEVC is the workhorse. AV1 encode on Apple silicon is unverified — the only sourcing found was content-farm pages claiming M4 Ultra / M5 Pro / M5 Max have hardware AV1 encoders. Confirm against a real VideoToolbox capability query before it reaches any capability matrix or release note.

Tier C — the game host: NOT recommended, and must not gate A or B

The asymmetry: on Linux and Windows the virtual display and emulated pads are the headline features; on macOS they are exactly the two things the platform withholds — and the Mac game library is the weakest of the three, so Tier C's value is lowest precisely where its cost is highest. Two blockers of different kinds:

  1. Virtual display = private API. There is no public way to create a display on macOS. BetterDisplay, DeskPad, FreeDisplay, OpenDisplay and Sidecar all use the private CoreGraphics CGVirtualDisplay. It works and is widely shipped, and Apple can remove it in any point release. Whether that is acceptable under a headline feature is a product decision, not a technical one.
  2. Gamepads = OS policy, not difficulty. macOS 15+ has a public virtual-HID API (CoreHID.HIDVirtualDevice + com.apple.developer.hid.virtual.device), so DriverKit/Karabiner is no longer needed — but the Game Controller framework deliberately ignores virtual HID devices to prevent input loopback, and macOS has no virtual-controller framework. Every GCController game is blind to an emulated pad however correct the descriptor. The ~8.4k-LOC pf-inject pad stack has no macOS analogue.

Do not re-propose a DriverKit system extension for pads — HIDVirtualDevice supersedes its plumbing and the GameController filter defeats both routes, so the heavier path buys nothing.

One-day spike before any Tier C planning: can SDL / Steam Input see a HIDVirtualDevice pad? They read IOHIDDevice directly, bypassing GameController. If yes, Tier C is partially recoverable for the Steam library specifically.

Structural constraint worth knowing up front

No headless operation, and no service equivalent. Screen Recording and Accessibility are TCC grants that only exist in a logged-in GUI session ⇒ LaunchAgent only, no LocalSystem-service analogue, no pre-login streaming, and "wake and stream" is unreachable (Wake-on-LAN is limited on a Mac host). TCC is per-signed-binary, so the updater must preserve the signing identity or every update revokes capture — that needs a real update-and-verify run before a macOS host could ship an updater at all.

Remaining spikes

  • HDR end to end through ScreenCaptureKit → VideoToolbox — believed available on recent macOS, not verified on glass; hdr-pipeline-plan.md would need a macOS column.
  • TCC persistence across host updates (above).
  • AV1 encode capability query (above).
  • SDL/Steam Input vs HIDVirtualDevice (Tier C gate, above).
Make `punktfunk-host` build on macOS, then ship a macOS **desktop** host. Design: `design/macos-host-feasibility.md` in punktfunk-planning (`39130c5`, on `main`) — investigation only, no code anywhere. **The finding that reframes this: the macOS port is four backends and a packaging story, not a host rewrite.** The W6 crate extraction already left a real third-platform seam — `cfg(not(any(linux, windows)))` arms exist across the `pf-*` crates and `pf-inject` even ships a `Backend::Unsupported` placeholder. Everything portable — orchestrator, session plan, GameStream plane, `punktfunk/1` native plane, pairing, mDNS, mgmt API + OpenAPI, web console, plugin runner, game library, update-check — **already compiles on macOS today**. For scale: the existing per-platform backend volume is ~41k LOC Linux / ~34k LOC Windows. Tier B is estimated ~3–5k. ## Tier A — make it compile (<1 day, unblocked, do this regardless) **Worth doing on its own merits even if Tiers B/C never happen**, because it retires the openapi tax: `cargo run -p punktfunk-host -- openapi` can only run on Linux/Windows today, so `api/openapi.json` has to be hand-patched or regenerated on a Linux box, and `mgmt::tests::openapi_document_is_complete_and_checked_in` is the only signal that a mgmt-API change left the spec stale. It also makes the Mac a first-class dev box for all the portable host code above. **Must not be gated on the Tier B/C decision.** Re-measured on `4ce7a3c1` (2026-08-18, arm64): `cargo check -p punktfunk-host` = **10 errors in 6 files**; `pf-capture`, `pf-encode`, `pf-inject`, `pf-clipboard`, `pf-vdisplay` all check **clean, unmodified**. - [ ] `audio/mic_pump.rs:172` — unresolved `opus` (two sites on the line). Gate the call sites to match the manifest gating. - [ ] `hooks.rs:457` — unresolved `libc`. Same. - [ ] `session_plan.rs:306-307` — `crate::zerocopy`, `encode::linux_zero_copy_is_vaapi`, `encode::cursor_blend_capable`. Give the three helpers non-Linux siblings. - [ ] `gamestream/stream.rs:808` + `native/compositor.rs:106` — `vdisplay::cancel_pending_tv_restore`; `native/compositor.rs:63` — `vdisplay::try_recover_session`. Add no-op non-Linux siblings next to the Linux ones. - [ ] `gamelease.rs:1062` — `procscan::Scanner`. **New since the design doc was written** (the doc measured 9 errors in 5 files on `2d3f9f86`, 2026-08-01). - [ ] **Add a macOS `cargo check -p punktfunk-host` CI leg.** The hole count grew by one in 17 days with nobody trying — without a guard this rots straight back. ## Tier B — the desktop host (~3–5k LOC, public APIs only) No entitlement requests, no private symbols. Ordered; each step independently useful. - [ ] **`pf-capture` macOS backend** — a `src/macos/` sibling to `linux/`/`windows/` implementing `Capturer` (`pf-capture/src/lib.rs:21`). ScreenCaptureKit is **the best capture platform of the three**: `SCStream` hands back IOSurface-backed `CVPixelBuffer`s that go straight into VideoToolbox, so the project's zero-copy invariant is the *default* path rather than something to fight for — no EGL/dmabuf/CUDA import worker, no IDD driver, no compositor matrix. `supports_arrival_wait()` (`:65`) = `true` — `SCStream`'s delegate callback **is** the frame-arrival wakeup the latency work wants, so the frame-driven encode trigger comes free. macOS arms for `capturer_supports_hdr` (`:421-430`) and `capturer_supports_444` (`:400/498/511`). - [ ] **`pf-encode` macOS backend** — `Encoder` (`enc/codec.rs:319`) over `VTCompressionSession` + `EnableLowLatencyRateControl`, wired into `open_video` (`pf-encode/src/lib.rs:163`). `submit`/`poll` map naturally onto VT's async completion callback; the frame-index prediction contract in the trait docs holds unchanged. One vendor, so no backend matrix — ~1–2k vs 16.3k (Linux) / 11.5k (Windows). - [ ] **Host audio** — `AudioCapturer` from the same `SCStream` (system audio arrives alongside video); macOS arm on `open_audio_capture` (`punktfunk-host/src/audio.rs:222/227/237`). Mic uplink via AVAudioEngine, which ships a real **platform AEC** — directly relevant to `mic-latency-and-echo.md`. - [ ] **`pf-inject` macOS backend** — `Backend::CgEvent` replacing the `Unsupported` placeholder (`pf-inject/src/lib.rs:76`, dispatch at `:241`); `InputInjector` over `CGEventPost` (+ Accessibility grant). Absolute mapping is straightforward. **Keyboard/mouse only** — pads are Tier C. - [ ] **`pf-clipboard`** — `NSPasteboard` behind `HostClipboard`; flip the `available = false` fallback (`pf-clipboard/src/lib.rs:98-110`). No delayed-rendering complication to match Win32's. - [ ] **Packaging** — LaunchAgent plist, Developer ID signing + notarization + hardened runtime, first-run TCC onboarding (Screen Recording + Accessibility + Microphone), a `packaging/macos/`. The Dev ID / notarization machinery already exists for the Mac client, so this is reuse. Deliberately **not** in the list: `pf-vdisplay` (Tier C), `pf-zerocopy` (nothing to do — SCK → VT is already zero-copy), `pf-gpu` (one vendor), the compositor matrix (no analogue). **Known ceilings:** VideoToolbox has **no 4:4:4**; HEVC is the workhorse. AV1 encode on Apple silicon is **unverified** — the only sourcing found was content-farm pages claiming M4 Ultra / M5 Pro / M5 Max have hardware AV1 encoders. Confirm against a real VideoToolbox capability query before it reaches any capability matrix or release note. ## Tier C — the game host: NOT recommended, and must not gate A or B The asymmetry: on Linux and Windows the virtual display and emulated pads are the headline features; on macOS they are exactly the two things the platform withholds — and the Mac game library is the weakest of the three, so Tier C's *value* is lowest precisely where its *cost* is highest. Two blockers of different kinds: 1. **Virtual display = private API.** There is no public way to create a display on macOS. BetterDisplay, DeskPad, FreeDisplay, OpenDisplay and Sidecar all use the private CoreGraphics `CGVirtualDisplay`. It works and is widely shipped, and Apple can remove it in any point release. Whether that is acceptable under a headline feature is a **product decision**, not a technical one. 2. **Gamepads = OS policy, not difficulty.** macOS 15+ has a *public* virtual-HID API (`CoreHID.HIDVirtualDevice` + `com.apple.developer.hid.virtual.device`), so DriverKit/Karabiner is no longer needed — **but the Game Controller framework deliberately ignores virtual HID devices** to prevent input loopback, and macOS has no virtual-controller framework. Every `GCController` game is blind to an emulated pad however correct the descriptor. The ~8.4k-LOC `pf-inject` pad stack has no macOS analogue. **Do not re-propose a DriverKit system extension for pads** — `HIDVirtualDevice` supersedes its plumbing and the GameController filter defeats both routes, so the heavier path buys nothing. **One-day spike before *any* Tier C planning:** can SDL / Steam Input see a `HIDVirtualDevice` pad? They read `IOHIDDevice` directly, bypassing GameController. If yes, Tier C is partially recoverable for the Steam library specifically. ## Structural constraint worth knowing up front **No headless operation, and no service equivalent.** Screen Recording and Accessibility are TCC grants that only exist in a logged-in GUI session ⇒ **LaunchAgent only**, no `LocalSystem`-service analogue, no pre-login streaming, and "wake and stream" is unreachable (Wake-on-LAN is limited on a Mac host). TCC is **per-signed-binary**, so the updater must preserve the signing identity or every update revokes capture — that needs a real update-and-verify run before a macOS host could ship an updater at all. ## Remaining spikes - HDR end to end through ScreenCaptureKit → VideoToolbox — believed available on recent macOS, not verified on glass; `hdr-pipeline-plan.md` would need a macOS column. - TCC persistence across host updates (above). - AV1 encode capability query (above). - SDL/Steam Input vs `HIDVirtualDevice` (Tier C gate, above).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
FernandoMX FlipperDan HitFrostbite bluelightspecial enricobuehler (Enrico Bühler) kszlim luxus nannku neoney
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unom/punktfunk#299