feat(clients): codec preference on Windows/Apple/Android clients (Phase 2b)
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s
apple / swift (push) Failing after 52s
windows-drivers / probe-and-proto (push) Successful in 50s
windows-drivers / driver-build (push) Successful in 1m20s
android / android (push) Failing after 2m55s
ci / web (push) Successful in 1m5s
release / apple (push) Successful in 3m38s
apple / screenshots (push) Has been skipped
ci / rust (push) Successful in 4m47s
ci / docs-site (push) Successful in 59s
deb / build-publish (push) Successful in 2m49s
decky / build-publish (push) Successful in 21s
windows-host / package (push) Successful in 7m35s
ci / bench (push) Successful in 5m10s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 34s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 2m41s
windows-msix / package (arm64, C:\Users\Public\ffmpeg-arm64, aarch64-pc-windows-msvc, C:\t-a64) (push) Successful in 1m17s
windows-msix / package (x64, C:\Users\Public\ffmpeg, x86_64-pc-windows-msvc, C:\t) (push) Successful in 1m11s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 2m22s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 2m59s
windows / build (aarch64-pc-windows-msvc) (push) Successful in 1m0s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 52s
windows / build (x86_64-pc-windows-msvc) (push) Successful in 1m7s
flatpak / build-publish (push) Successful in 4m20s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 8m58s
docker / deploy-docs (push) Successful in 23s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 8m45s
Rounds out codec negotiation across the last three clients — each advertises what it can decode, builds its decoder from the resolved Welcome.codec, and exposes a "Video codec" preference picker. **Windows** (Rust, mirrors Linux): `decodable_codecs()` + `ffmpeg_codec_id()`; the D3D11VA and software FFmpeg decoders (and the mid-session D3D11VA→software demotion) open the negotiated codec instead of hardcoding HEVC; settings gain a `codec` field + reactor ComboBox; `--codec` CLI flag. **Apple** (Swift/C-ABI): AnnexB is now codec-aware — a `VideoCodec` enum drives H.264 vs HEVC NAL parsing / parameter-set extraction (`CMVideoFormatDescriptionCreateFromH264ParameterSets` for H.264, no VPS) and AVCC repacking; `PunktfunkConnection` advertises H264|HEVC via `punktfunk_connect_ex7`, reads `resolvedCodec` (`punktfunk_connection_codec`), and threads `videoCodec` into the stage-1/2 pipelines + `VideoDecoder`; SettingsView "Video codec" Picker (auto/HEVC/H.264). AV1 is left out (hosts don't emit it on the native path, and it's not an AnnexB codec). Test call sites updated. **Android** (Kotlin + Rust JNI): the JNI `nativeConnect` gains `preferredCodec`; the native decode loop picks the AMediaCodec MIME (`video/hevc`|`video/avc`) from `connector.codec` and advertises H264|HEVC; Settings `codec` field + Compose dropdown. Core/host/probe/Linux clippy + tests green (unchanged from 2a). Windows/Apple/Android compile on their platform CI (this Linux box can't build them — Windows toolchain / Xcode / the Android NDK's opus-cmake toolchain). All follow the Linux client's validated pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -187,7 +187,8 @@ fun ConnectScreen(settings: Settings, onConnected: (Long) -> Unit) {
|
||||
targetHost, targetPort, w, h, hz,
|
||||
id.certPem, id.privateKeyPem, pinHex ?: "",
|
||||
settings.bitrateKbps, settings.compositor, gamepadPref,
|
||||
hdrEnabled, settings.audioChannels, CONNECT_TIMEOUT_MS,
|
||||
hdrEnabled, settings.audioChannels, settings.preferredCodec(),
|
||||
CONNECT_TIMEOUT_MS,
|
||||
)
|
||||
}
|
||||
connecting = false
|
||||
|
||||
@@ -26,6 +26,9 @@ data class Settings(
|
||||
/** Requested audio channel count: 2 (stereo), 6 (5.1) or 8 (7.1). The host clamps to what it
|
||||
* can capture; the resolved count drives the decoder + AAudio layout. */
|
||||
val audioChannels: Int = 2,
|
||||
/** Preferred video codec: `"auto"` (host decides), `"hevc"`, or `"h264"`. A soft preference — the
|
||||
* host emits it when it can, else falls back. AMediaCodec decodes whichever the host resolves. */
|
||||
val codec: String = "auto",
|
||||
val micEnabled: Boolean = false,
|
||||
/** Show the live stats overlay (FPS / throughput / latency) during a stream. */
|
||||
val statsHudEnabled: Boolean = true,
|
||||
@@ -51,6 +54,7 @@ class SettingsStore(context: Context) {
|
||||
compositor = prefs.getInt(K_COMPOSITOR, 0),
|
||||
gamepad = prefs.getInt(K_GAMEPAD, 0),
|
||||
audioChannels = prefs.getInt(K_AUDIO_CH, 2),
|
||||
codec = prefs.getString(K_CODEC, "auto") ?: "auto",
|
||||
micEnabled = prefs.getBoolean(K_MIC, false),
|
||||
statsHudEnabled = prefs.getBoolean(K_HUD, true),
|
||||
trackpadMode = prefs.getBoolean(K_TRACKPAD, true),
|
||||
@@ -66,6 +70,7 @@ class SettingsStore(context: Context) {
|
||||
.putInt(K_COMPOSITOR, s.compositor)
|
||||
.putInt(K_GAMEPAD, s.gamepad)
|
||||
.putInt(K_AUDIO_CH, s.audioChannels)
|
||||
.putString(K_CODEC, s.codec)
|
||||
.putBoolean(K_MIC, s.micEnabled)
|
||||
.putBoolean(K_HUD, s.statsHudEnabled)
|
||||
.putBoolean(K_TRACKPAD, s.trackpadMode)
|
||||
@@ -81,6 +86,7 @@ class SettingsStore(context: Context) {
|
||||
const val K_COMPOSITOR = "compositor"
|
||||
const val K_GAMEPAD = "gamepad"
|
||||
const val K_AUDIO_CH = "audio_channels"
|
||||
const val K_CODEC = "codec"
|
||||
const val K_MIC = "mic_enabled"
|
||||
const val K_HUD = "stats_hud_enabled"
|
||||
const val K_TRACKPAD = "trackpad_mode"
|
||||
@@ -156,6 +162,21 @@ val AUDIO_CHANNEL_OPTIONS = listOf(
|
||||
8 to "7.1 Surround",
|
||||
)
|
||||
|
||||
/** (stored value, label) for the preferred video codec. `"auto"` = host decides. */
|
||||
val CODEC_OPTIONS = listOf(
|
||||
"auto" to "Automatic",
|
||||
"hevc" to "HEVC (H.265)",
|
||||
"h264" to "H.264 (AVC)",
|
||||
)
|
||||
|
||||
/** The [Settings.codec] string as a `quic::CODEC_*` preference byte (`0` = auto). H264=1, HEVC=2. */
|
||||
fun Settings.preferredCodec(): Int = when (codec) {
|
||||
"h264" -> 1
|
||||
"hevc" -> 2
|
||||
"av1" -> 4
|
||||
else -> 0
|
||||
}
|
||||
|
||||
/** (kbps, label). `0` = host default. */
|
||||
val BITRATE_OPTIONS = listOf(
|
||||
0 to "Automatic",
|
||||
|
||||
@@ -95,6 +95,12 @@ fun SettingsScreen(initial: Settings, onChange: (Settings) -> Unit, onBack: () -
|
||||
selected = s.bitrateKbps,
|
||||
) { kbps -> update(s.copy(bitrateKbps = kbps)) }
|
||||
|
||||
SettingDropdown(
|
||||
label = "Video codec",
|
||||
options = CODEC_OPTIONS,
|
||||
selected = s.codec,
|
||||
) { c -> update(s.copy(codec = c)) }
|
||||
|
||||
// HDR is only meaningful on a panel that can present HDR10; on an SDR display the toggle
|
||||
// is disabled (and HDR is never advertised regardless) so the host doesn't send PQ the
|
||||
// panel would mis-tone-map. The capability is fixed for the device, so read it once.
|
||||
|
||||
@@ -48,6 +48,8 @@ object NativeBridge {
|
||||
gamepadPref: Int,
|
||||
hdrEnabled: Boolean,
|
||||
audioChannels: Int,
|
||||
/** Preferred video codec as a `quic::CODEC_*` bit (`0` = auto). Soft — the host falls back. */
|
||||
preferredCodec: Int,
|
||||
timeoutMs: Int,
|
||||
): Long
|
||||
|
||||
|
||||
@@ -27,16 +27,23 @@ pub fn run(
|
||||
) {
|
||||
boost_thread_priority();
|
||||
let mode = client.mode();
|
||||
let codec = match MediaCodec::from_decoder_type("video/hevc") {
|
||||
// The MediaCodec MIME for the codec the host resolved (`Welcome.codec`): HEVC or H.264. AMediaCodec
|
||||
// needs no out-of-band extradata — the in-band VPS/SPS/PPS on every IDR configure it either way.
|
||||
let mime = match client.codec {
|
||||
punktfunk_core::quic::CODEC_H264 => "video/avc",
|
||||
_ => "video/hevc",
|
||||
};
|
||||
let codec = match MediaCodec::from_decoder_type(mime) {
|
||||
Some(c) => c,
|
||||
None => {
|
||||
log::error!("decode: no HEVC decoder on this device");
|
||||
log::error!("decode: no {mime} decoder on this device");
|
||||
return;
|
||||
}
|
||||
};
|
||||
log::info!("decode: codec mime = {mime}");
|
||||
|
||||
let mut format = MediaFormat::new();
|
||||
format.set_str("mime", "video/hevc");
|
||||
format.set_str("mime", mime);
|
||||
format.set_i32("width", mode.width as i32);
|
||||
format.set_i32("height", mode.height as i32);
|
||||
// Generous input buffer so a large keyframe AU is never truncated.
|
||||
|
||||
@@ -167,6 +167,7 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
|
||||
gamepad_pref: jint,
|
||||
hdr_enabled: jboolean,
|
||||
audio_channels: jint,
|
||||
preferred_codec: jint,
|
||||
timeout_ms: jint,
|
||||
) -> jlong {
|
||||
let host: String = match env.get_string(&host) {
|
||||
@@ -224,6 +225,11 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
|
||||
// decoder + AAudio layout (read in `crate::audio::AudioPlayback::start`). Anything else
|
||||
// normalizes to stereo here.
|
||||
punktfunk_core::audio::normalize_channels(audio_channels.clamp(0, u8::MAX as jint) as u8),
|
||||
// Codecs this device can decode — AMediaCodec decodes both HEVC and H.264 (AV1 isn't wired;
|
||||
// hosts don't emit it on the native path yet). The host resolves the emitted codec from these
|
||||
// + the soft `preferred_codec` and echoes it in `connector.codec`, which drives the mime below.
|
||||
punktfunk_core::quic::CODEC_H264 | punktfunk_core::quic::CODEC_HEVC,
|
||||
preferred_codec.clamp(0, u8::MAX as jint) as u8,
|
||||
None, // launch: default app
|
||||
pin, // Some → Crypto on host-fp mismatch
|
||||
identity, // owned (cert, key) PEM, or None (anonymous)
|
||||
|
||||
Reference in New Issue
Block a user