feat(android): mic uplink + connect-screen redesign
ci / web (push) Successful in 29s
android / android (push) Successful in 1m50s
ci / bench (push) Successful in 1m42s
apple / swift (push) Successful in 53s
ci / rust (push) Successful in 1m4s
ci / docs-site (push) Successful in 31s
decky / build-publish (push) Successful in 13s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
deb / build-publish (push) Successful in 3m21s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 5m15s
docker / deploy-docs (push) Successful in 17s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 5m1s
ci / web (push) Successful in 29s
android / android (push) Successful in 1m50s
ci / bench (push) Successful in 1m42s
apple / swift (push) Successful in 53s
ci / rust (push) Successful in 1m4s
ci / docs-site (push) Successful in 31s
decky / build-publish (push) Successful in 13s
docker / build-push (--build-arg FEDORA_VERSION=44, ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora44-rpm) (push) Successful in 5s
docker / build-push (., web/Dockerfile, punktfunk-web) (push) Successful in 4s
docker / build-push (ci, ci/fedora-rpm.Dockerfile, punktfunk-fedora-rpm) (push) Successful in 3s
docker / build-push (ci, ci/rust-ci.Dockerfile, punktfunk-rust-ci) (push) Successful in 4s
docker / build-push (docs-site, docs-site/Dockerfile, punktfunk-docs) (push) Successful in 4s
deb / build-publish (push) Successful in 3m21s
rpm / build-publish (bazzite, punktfunk-fedora-rpm) (push) Successful in 5m15s
docker / deploy-docs (push) Successful in 17s
rpm / build-publish (fedora-44, punktfunk-fedora44-rpm) (push) Successful in 5m1s
Microphone uplink (client → host's virtual mic, 0xCB) and a cleaner connect screen.
Mic (Rust-heavy, mirrors the audio playback path in reverse):
- crates/punktfunk-android/src/mic.rs: AAudio LowLatency **input** → realtime callback hands
captured f32 to a channel → a worker thread Opus-encodes 20 ms stereo frames (48 kHz, VOIP,
64 kbps) and calls NativeClient::send_mic. MicCapture owns the stream + encode thread (RAII stop).
- session.rs: SessionHandle gains a `mic` slot; nativeStartMic/nativeStopMic JNI (mirror of audio);
stopped in Drop. NativeBridge: the two externs.
- Settings: a `micEnabled` flag + a Microphone toggle in SettingsScreen that requests RECORD_AUDIO
(denied → stays off). StreamScreen starts the mic only if enabled AND the permission is held.
Connect-screen redesign:
- One scrollable Column (was a fixed centered layout that could clip with the new tab bar);
host rows render via forEach (no nested LazyColumn). Colored section labels ("Saved hosts",
"Discovered on the network", "Connect manually"), full-width host cards / fields / Connect button,
a header + subtitle, and a muted footer.
Verified live (emulator pf_phone -> home-worker-2): toggling mic requests RECORD_AUDIO; with it
granted, a session sends mic frames (client "mic: sent=250 … peak=0.439" — real audio) and the host
logs "client datagram stream ended … mic=276". Redesigned screen confirmed via screenshots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -26,6 +26,8 @@ mod audio;
|
||||
#[cfg(target_os = "android")]
|
||||
mod decode;
|
||||
mod feedback;
|
||||
#[cfg(target_os = "android")]
|
||||
mod mic;
|
||||
mod session;
|
||||
|
||||
/// Initialize `android_logger` once when the JVM loads the library. Logs land in logcat under the
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
//! Android microphone uplink (android-only): capture mic PCM via AAudio (LowLatency **input**),
|
||||
//! Opus-encode 20 ms stereo frames, and push them to the host over the connector's mic plane
|
||||
//! (`send_mic` → 0xCB datagram). The mirror of [`crate::audio`] in reverse: AAudio's realtime input
|
||||
//! callback hands captured interleaved f32 to a channel; a worker thread we own does the Opus encode
|
||||
//! + send (encoding is too heavy for the realtime callback, exactly as decode is on the playback
|
||||
//! side). Format matches the host decoder + the Linux client: 48 kHz **stereo**, 20 ms, Opus VOIP.
|
||||
|
||||
use ndk::audio::{
|
||||
AudioCallbackResult, AudioDirection, AudioFormat, AudioPerformanceMode, AudioSharingMode,
|
||||
AudioStream, AudioStreamBuilder,
|
||||
};
|
||||
use punktfunk_core::client::NativeClient;
|
||||
use std::collections::VecDeque;
|
||||
use std::ffi::c_void;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use std::sync::mpsc::{sync_channel, Receiver, RecvTimeoutError, TrySendError};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
const CHANNELS: usize = 2;
|
||||
const SAMPLE_RATE: i32 = 48_000;
|
||||
/// 20 ms per channel @ 48 kHz — the Linux client's frame; the host accepts ≤ 120 ms.
|
||||
const FRAME_SAMPLES: usize = 960;
|
||||
/// Captured-chunk hand-off depth (each ~ one burst); drops on overflow (best-effort uplink).
|
||||
const RING_CHUNKS: usize = 64;
|
||||
/// Opus VOIP target bitrate (speech; tunable).
|
||||
const MIC_BITRATE: i32 = 64_000;
|
||||
|
||||
/// Owned by [`crate::session::SessionHandle`]: the live AAudio input stream + the encode thread.
|
||||
pub struct MicCapture {
|
||||
_stream: AudioStream, // dropping it stops + closes the AAudio input stream
|
||||
shutdown: Arc<AtomicBool>,
|
||||
join: Option<std::thread::JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl MicCapture {
|
||||
/// Open AAudio (LowLatency, 48 kHz/stereo/f32) for **input** with a realtime callback that
|
||||
/// forwards captured PCM to a channel, then spawn the Opus encode + uplink thread. `None` on
|
||||
/// failure (the caller leaves the rest of the session streaming).
|
||||
pub fn start(client: Arc<NativeClient>) -> Option<MicCapture> {
|
||||
let (tx, rx) = sync_channel::<Vec<f32>>(RING_CHUNKS);
|
||||
let captured = Arc::new(AtomicU64::new(0));
|
||||
let cb_captured = captured.clone();
|
||||
|
||||
let callback = move |_s: &AudioStream, data: *mut c_void, num_frames: i32| {
|
||||
let n = num_frames as usize * CHANNELS;
|
||||
// SAFETY: for an input stream AAudio provides `num_frames * channel_count` captured F32
|
||||
// samples at `data` (read-only for us).
|
||||
let inp = unsafe { std::slice::from_raw_parts(data as *const f32, n) };
|
||||
match tx.try_send(inp.to_vec()) {
|
||||
Ok(()) | Err(TrySendError::Full(_)) => {} // drop-newest if the encoder lags
|
||||
Err(TrySendError::Disconnected(_)) => return AudioCallbackResult::Stop,
|
||||
}
|
||||
cb_captured.fetch_add(num_frames as u64, Ordering::Relaxed);
|
||||
AudioCallbackResult::Continue
|
||||
};
|
||||
|
||||
let stream = AudioStreamBuilder::new()
|
||||
.map_err(|e| log::error!("mic: AudioStreamBuilder::new: {e}"))
|
||||
.ok()?
|
||||
.direction(AudioDirection::Input)
|
||||
.sample_rate(SAMPLE_RATE)
|
||||
.channel_count(CHANNELS as i32)
|
||||
.format(AudioFormat::PCM_Float)
|
||||
.performance_mode(AudioPerformanceMode::LowLatency)
|
||||
.sharing_mode(AudioSharingMode::Shared)
|
||||
.data_callback(Box::new(callback))
|
||||
.error_callback(Box::new(|_s, e| {
|
||||
log::warn!("mic: AAudio error (device reroute/disconnect?): {e:?}");
|
||||
}))
|
||||
.open_stream()
|
||||
.map_err(|e| log::error!("mic: open_stream (RECORD_AUDIO granted?): {e}"))
|
||||
.ok()?;
|
||||
|
||||
if let Err(e) = stream.request_start() {
|
||||
log::error!("mic: request_start: {e}");
|
||||
return None;
|
||||
}
|
||||
log::info!(
|
||||
"mic: AAudio input started rate={} ch={} fmt={:?}",
|
||||
stream.sample_rate(),
|
||||
stream.channel_count(),
|
||||
stream.format(),
|
||||
);
|
||||
|
||||
let shutdown = Arc::new(AtomicBool::new(false));
|
||||
let sd = shutdown.clone();
|
||||
let join = std::thread::Builder::new()
|
||||
.name("pf-mic".into())
|
||||
.spawn(move || encode_loop(client, rx, sd, captured))
|
||||
.ok();
|
||||
|
||||
Some(MicCapture {
|
||||
_stream: stream,
|
||||
shutdown,
|
||||
join,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for MicCapture {
|
||||
fn drop(&mut self) {
|
||||
self.shutdown.store(true, Ordering::SeqCst);
|
||||
if let Some(j) = self.join.take() {
|
||||
let _ = j.join();
|
||||
}
|
||||
// `_stream` drops here → AAudio request_stop + close.
|
||||
}
|
||||
}
|
||||
|
||||
/// Consumer: drain captured f32 → accumulate → Opus `encode_float` 20 ms stereo frames → `send_mic`.
|
||||
fn encode_loop(
|
||||
client: Arc<NativeClient>,
|
||||
rx: Receiver<Vec<f32>>,
|
||||
shutdown: Arc<AtomicBool>,
|
||||
captured: Arc<AtomicU64>,
|
||||
) {
|
||||
let mut enc = match opus::Encoder::new(
|
||||
SAMPLE_RATE as u32,
|
||||
opus::Channels::Stereo,
|
||||
opus::Application::Voip,
|
||||
) {
|
||||
Ok(e) => e,
|
||||
Err(e) => {
|
||||
log::error!("mic: opus encoder init: {e} — mic disabled");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let _ = enc.set_bitrate(opus::Bitrate::Bits(MIC_BITRATE));
|
||||
|
||||
let frame = FRAME_SAMPLES * CHANNELS;
|
||||
let mut ring: VecDeque<f32> = VecDeque::with_capacity(frame * 4);
|
||||
let mut out = vec![0u8; 4000]; // max Opus packet for a 20 ms frame fits easily
|
||||
let mut seq: u32 = 0;
|
||||
let mut sent: u64 = 0;
|
||||
let mut peak = 0f32; // loudest |sample| since the last log — tells speech from silence
|
||||
|
||||
while !shutdown.load(Ordering::Relaxed) {
|
||||
match rx.recv_timeout(Duration::from_millis(100)) {
|
||||
Ok(chunk) => ring.extend(chunk),
|
||||
Err(RecvTimeoutError::Timeout) => continue, // wake to re-check shutdown
|
||||
Err(RecvTimeoutError::Disconnected) => break,
|
||||
}
|
||||
while ring.len() >= frame {
|
||||
let pcm: Vec<f32> = ring.drain(..frame).collect();
|
||||
for &s in &pcm {
|
||||
peak = peak.max(s.abs());
|
||||
}
|
||||
match enc.encode_float(&pcm, &mut out) {
|
||||
Ok(len) => {
|
||||
let pts = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos() as u64)
|
||||
.unwrap_or(0);
|
||||
let _ = client.send_mic(seq, pts, out[..len].to_vec());
|
||||
seq = seq.wrapping_add(1);
|
||||
sent += 1;
|
||||
if sent % 250 == 0 {
|
||||
log::info!(
|
||||
"mic: sent={sent} captured_frames={} peak={peak:.3}",
|
||||
captured.load(Ordering::Relaxed),
|
||||
);
|
||||
peak = 0.0;
|
||||
}
|
||||
}
|
||||
Err(e) => log::debug!("mic: opus encode: {e}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
log::info!(
|
||||
"mic: stopped (sent={sent} captured_frames={})",
|
||||
captured.load(Ordering::Relaxed),
|
||||
);
|
||||
}
|
||||
@@ -33,6 +33,8 @@ pub(crate) struct SessionHandle {
|
||||
video: Mutex<Option<VideoThread>>,
|
||||
#[cfg(target_os = "android")]
|
||||
audio: Mutex<Option<crate::audio::AudioPlayback>>,
|
||||
#[cfg(target_os = "android")]
|
||||
mic: Mutex<Option<crate::mic::MicCapture>>,
|
||||
}
|
||||
|
||||
struct VideoThread {
|
||||
@@ -57,6 +59,13 @@ impl SessionHandle {
|
||||
fn stop_audio(&self) {
|
||||
let _ = self.audio.lock().unwrap().take();
|
||||
}
|
||||
|
||||
/// Stop mic uplink. Dropping the [`crate::mic::MicCapture`] joins its encode thread and closes
|
||||
/// the AAudio input stream. Idempotent.
|
||||
#[cfg(target_os = "android")]
|
||||
fn stop_mic(&self) {
|
||||
let _ = self.mic.lock().unwrap().take();
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for SessionHandle {
|
||||
@@ -64,6 +73,8 @@ impl Drop for SessionHandle {
|
||||
self.stop_video();
|
||||
#[cfg(target_os = "android")]
|
||||
self.stop_audio();
|
||||
#[cfg(target_os = "android")]
|
||||
self.stop_mic();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -182,6 +193,8 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeConnect<'lo
|
||||
video: Mutex::new(None),
|
||||
#[cfg(target_os = "android")]
|
||||
audio: Mutex::new(None),
|
||||
#[cfg(target_os = "android")]
|
||||
mic: Mutex::new(None),
|
||||
};
|
||||
Box::into_raw(Box::new(handle)) as jlong
|
||||
}
|
||||
@@ -381,6 +394,47 @@ pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopAudio(
|
||||
}
|
||||
}
|
||||
|
||||
/// `NativeBridge.nativeStartMic(handle)` — start mic capture (AAudio input → Opus → host `send_mic`).
|
||||
/// No-op if already running or on a `0` handle. Caller MUST hold RECORD_AUDIO; a failure (e.g. no
|
||||
/// permission) leaves the rest of the session streaming.
|
||||
#[cfg(target_os = "android")]
|
||||
#[no_mangle]
|
||||
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStartMic(
|
||||
_env: JNIEnv,
|
||||
_this: JObject,
|
||||
handle: jlong,
|
||||
) {
|
||||
if handle == 0 {
|
||||
return;
|
||||
}
|
||||
// SAFETY: live handle per the nativeConnect/nativeClose contract.
|
||||
let h = unsafe { &*(handle as *const SessionHandle) };
|
||||
let mut guard = h.mic.lock().unwrap();
|
||||
if guard.is_some() {
|
||||
return; // already capturing
|
||||
}
|
||||
match crate::mic::MicCapture::start(h.client.clone()) {
|
||||
Some(m) => *guard = Some(m),
|
||||
None => log::error!("nativeStartMic: mic init failed (RECORD_AUDIO? — session unaffected)"),
|
||||
}
|
||||
}
|
||||
|
||||
/// `NativeBridge.nativeStopMic(handle)` — stop + join the mic thread and close the AAudio input
|
||||
/// stream (without closing the session). No-op on `0`.
|
||||
#[cfg(target_os = "android")]
|
||||
#[no_mangle]
|
||||
pub extern "system" fn Java_io_unom_punktfunk_kit_NativeBridge_nativeStopMic(
|
||||
_env: JNIEnv,
|
||||
_this: JObject,
|
||||
handle: jlong,
|
||||
) {
|
||||
if handle != 0 {
|
||||
// SAFETY: live handle per the contract.
|
||||
let h = unsafe { &*(handle as *const SessionHandle) };
|
||||
h.stop_mic();
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Input plane: Kotlin capture → NativeClient::send_input ----------------------------------
|
||||
// All four are `&self` on the `Sync` connector (send_input is a non-blocking datagram push), safe
|
||||
// from the Kotlin UI thread. NOT android-gated — send_input exists on the host build too, so these
|
||||
|
||||
Reference in New Issue
Block a user