perf(android): low-latency decode overhaul — vendor keys, async loop, system tuning
android / android (push) Has been cancelled
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
arch / build-publish (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 9m21s
android / android (push) Has been cancelled
apple / swift (push) Has been cancelled
apple / screenshots (push) Has been cancelled
arch / build-publish (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / web (push) Has been cancelled
ci / docs-site (push) Has been cancelled
ci / bench (push) Has been cancelled
rpm / build-publish (44, fedora-44, punktfunk-fedora44-rpm) (push) Has been cancelled
deb / build-publish (push) Has been cancelled
decky / build-publish (push) Has been cancelled
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Has been cancelled
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Has been cancelled
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Has been cancelled
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Has been cancelled
docker / deploy-docs (push) Has been cancelled
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Has been cancelled
rpm / build-publish (43, bazzite, punktfunk-fedora-rpm) (push) Successful in 9m21s
Close the latency gap on the Android client with per-SoC decoder tuning, an event-driven decode loop, and full system integration. - Decoder selection: rank MediaCodecList decoders in Kotlin (hardware/vendor preferred, software avoided, FEATURE_LowLatency probed) and create the chosen one by name. Per-SoC low-latency keys gated on the codec-name prefix: Qualcomm picture-order + low-latency, Exynos (also Google Tensor), Amlogic, HiSilicon; MediaTek vdec-lowlatency set unconditionally. operating-rate = MAX (Qualcomm) vs priority = 0 (else) are mutually exclusive. NVIDIA/Rockchip/Realtek have no vendor key — covered by ranking + the standard low-latency key. - Async decode loop: AMediaCodec async-notify replaces the poll loop, presenting a decoded frame the instant it is ready instead of waiting out a poll interval. Behind USE_ASYNC_DECODE with the synchronous loop kept for A/B during bring-up. - System integration: Wi-Fi FULL_LOW_LATENCY lock and HDMI ALLM (setPreferMinimalPostProcessing) for the stream's lifetime; game_mode_config.xml opting out of OEM downscaling / FPS overrides. - Pipeline: boost the data-plane pump + audio thread priorities, AAudio usage=Game, DSCP marking on by default on Android, ADPF setPreferPowerEfficiency(false), and setFrameRateWithChangeStrategy(ALWAYS) to force the HDMI mode switch on TV. - lowLatencyMode master toggle (default on) as the escape hatch; the stats HUD now shows the resolved decoder name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -104,14 +104,40 @@ object NativeBridge {
|
||||
external fun nativeWakeOnLan(macsCsv: String, lastIp: String): Boolean
|
||||
|
||||
/**
|
||||
* Start the HEVC decode thread rendering onto [surface] (a SurfaceView's surface). Decode runs
|
||||
* entirely in Rust (NDK AMediaCodec → ANativeWindow) — no per-frame JNI. No-op if already started.
|
||||
* The MediaCodec MIME the host resolved for this session (`"video/hevc"` / `"video/avc"` /
|
||||
* `"video/av01"`), or `""` on a `0` handle. Kotlin ranks `MediaCodecList` decoders for this
|
||||
* MIME (see [io.unom.punktfunk.kit.VideoDecoders]) before [nativeStartVideo]. Cheap; UI-safe.
|
||||
*/
|
||||
external fun nativeStartVideo(handle: Long, surface: android.view.Surface)
|
||||
external fun nativeVideoMime(handle: Long): String
|
||||
|
||||
/**
|
||||
* Start the decode thread rendering onto [surface] (a SurfaceView's surface). Decode runs
|
||||
* entirely in Rust (NDK AMediaCodec → ANativeWindow) — no per-frame JNI. [decoderName] is the
|
||||
* decoder Kotlin ranked from `MediaCodecList` (`""` = let the platform resolve the default for
|
||||
* the MIME); [lowLatencyMode] is the user's master toggle (default on → aggressive per-SoC
|
||||
* tuning; off → conservative); [lowLatencyFeature] is whether [decoderName] advertised
|
||||
* `FEATURE_LowLatency` (HUD label only). [isTv] drives an active HDMI mode switch to the stream
|
||||
* refresh on TV boxes (vs. the softer seamless hint on phones). No-op if already started.
|
||||
*/
|
||||
external fun nativeStartVideo(
|
||||
handle: Long,
|
||||
surface: android.view.Surface,
|
||||
decoderName: String,
|
||||
lowLatencyMode: Boolean,
|
||||
lowLatencyFeature: Boolean,
|
||||
isTv: Boolean,
|
||||
)
|
||||
|
||||
/** Stop + join the decode thread without closing the session. No-op on `0`. */
|
||||
external fun nativeStopVideo(handle: Long)
|
||||
|
||||
/**
|
||||
* The resolved decoder identity for the HUD, e.g. `c2.qti.avc.decoder · low-latency`, or `""`
|
||||
* before the decode thread has resolved one. One-shot (fixed for the session); poll once after
|
||||
* the HUD appears.
|
||||
*/
|
||||
external fun nativeVideoDecoderLabel(handle: Long): String
|
||||
|
||||
/**
|
||||
* Drain ~1 s of live decode stats for the on-stream HUD, or `null` when no decode thread runs.
|
||||
* Returns 18 doubles (unified stats spec, `design/stats-unification.md`):
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
package io.unom.punktfunk.kit
|
||||
|
||||
import android.media.MediaCodecInfo.CodecCapabilities
|
||||
import android.media.MediaCodecList
|
||||
import android.os.Build
|
||||
|
||||
/** The decoder Kotlin ranked for a MIME, handed to [NativeBridge.nativeStartVideo]. */
|
||||
data class DecoderChoice(val name: String, val lowLatencyFeature: Boolean)
|
||||
|
||||
/**
|
||||
* Rank the platform's `MediaCodecList` decoders for a video MIME and pick the best one for
|
||||
* low-latency streaming, the way Moonlight-Android does. There is no NDK `MediaCodecList`, so this
|
||||
* enumeration must live on the Kotlin (framework) side; Rust then creates the chosen decoder by
|
||||
* name (`AMediaCodec_createCodecByName`) and derives the per-SoC vendor low-latency keys from it.
|
||||
*
|
||||
* Ranking (best first): hardware over software; a real SoC-vendor decoder (Qualcomm/Amlogic/…) over
|
||||
* the generic AOSP software fallback; a decoder advertising `FEATURE_LowLatency` over one that
|
||||
* doesn't. Known-bad software decoders (`omx.google.*`, `c2.android.*`, Qualcomm/Samsung SW HEVC)
|
||||
* are dropped outright — matching Moonlight's blacklist.
|
||||
*/
|
||||
object VideoDecoders {
|
||||
/** Decoder-name prefixes/names we never want, mirroring Moonlight's blacklist. */
|
||||
private val BLOCKED_PREFIXES = listOf("omx.google.", "c2.android.", "avcdecoder", "omx.ffmpeg.")
|
||||
private val BLOCKED_EXACT = listOf("omx.qcom.video.decoder.hevcswvdec", "omx.sec.hevc.sw.dec")
|
||||
|
||||
/**
|
||||
* Real SoC-vendor decoder prefixes we prefer over the generic AOSP fallback, covering the common
|
||||
* targets: Qualcomm Snapdragon and MediaTek (most phones + many TV boxes), Samsung Exynos (+
|
||||
* Google Tensor, whose decoder is `c2.exynos.*`), NVIDIA Tegra (Shield TV), Amlogic / Rockchip /
|
||||
* Realtek (TV boxes & smart TVs), and HiSilicon Kirin (older Huawei).
|
||||
*/
|
||||
private val VENDOR_PREFIXES = listOf(
|
||||
"omx.qcom", "c2.qti",
|
||||
"omx.mtk", "c2.mtk",
|
||||
"omx.exynos", "c2.exynos",
|
||||
"omx.nvidia", "c2.nvidia",
|
||||
"omx.amlogic", "c2.amlogic",
|
||||
"omx.rk", "c2.rk",
|
||||
"omx.realtek", "c2.realtek",
|
||||
"omx.hisi", "c2.hisi",
|
||||
)
|
||||
|
||||
/**
|
||||
* Pick the best decoder for [mime] (`"video/hevc"` / `"video/avc"` / `"video/av01"`), or `null`
|
||||
* to let the platform resolve its default. Enumerates once — call at stream start.
|
||||
*/
|
||||
fun pickDecoder(mime: String): DecoderChoice? {
|
||||
if (mime.isEmpty()) return null
|
||||
val infos = runCatching { MediaCodecList(MediaCodecList.REGULAR_CODECS).codecInfos }
|
||||
.getOrNull() ?: return null
|
||||
|
||||
var bestName: String? = null
|
||||
var bestLowLatency = false
|
||||
var bestScore = Int.MIN_VALUE
|
||||
for (info in infos) {
|
||||
if (info.isEncoder) continue
|
||||
val name = info.name
|
||||
val lower = name.lowercase()
|
||||
if (BLOCKED_PREFIXES.any { lower.startsWith(it) } || lower in BLOCKED_EXACT) continue
|
||||
val caps = runCatching { info.getCapabilitiesForType(mime) }.getOrNull() ?: continue
|
||||
|
||||
val hardware = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
|
||||
info.isHardwareAccelerated
|
||||
} else {
|
||||
// Pre-Q heuristic: the software decoders are the ones we can name (already blocked
|
||||
// above), so anything surviving the blacklist is treated as hardware.
|
||||
true
|
||||
}
|
||||
val lowLatency = Build.VERSION.SDK_INT >= Build.VERSION_CODES.R &&
|
||||
runCatching { caps.isFeatureSupported(CodecCapabilities.FEATURE_LowLatency) }
|
||||
.getOrDefault(false)
|
||||
val vendor = VENDOR_PREFIXES.any { lower.startsWith(it) }
|
||||
|
||||
val score = (if (hardware) 100 else 0) +
|
||||
(if (vendor) 40 else 0) +
|
||||
(if (lowLatency) 20 else 0)
|
||||
if (score > bestScore) {
|
||||
bestScore = score
|
||||
bestName = name
|
||||
bestLowLatency = lowLatency
|
||||
}
|
||||
}
|
||||
return bestName?.let { DecoderChoice(it, bestLowLatency) }
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user