Realtime audio

beam_audio runs one voice turn: speech-to-text, the agent, then text-to-speech. It also gives you the session state machine and the message types for a WebSocket protocol. The WebSocket server itself is yours: the crate does not depend on a web framework.

A turn

use beam_audio::{pcm_f32_to_wav_bytes, run_audio_turn, MockStt, MockTts};

let stt = MockStt::new("hello there"); // returns this transcript
let tts = MockTts;                      // returns silence

let audio = pcm_f32_to_wav_bytes(&samples, 16_000);
run_audio_turn(&audio, 1, &session_id, &stt, &tts, &invoker, &sink).await?;

run_audio_turn(raw_audio, seq, session_id, stt, tts, invoker, sink) decodes the audio and sends these messages to the sink, in order:

Message	Content
text, phase stt	"Transcribing…"
text, phase user_transcript	The transcript (final: false)
text, phase llm	"Thinking…"
text, phase assistant, final	The agent's reply
audio_start	seq, format: "audio/wav"
binary	The synthesized WAV
audio_end	seq

When the transcript is empty, the turn stops after a final user_transcript of "(No speech detected)". When decoding fails, it sends a text message with phase error and returns the error.

Backends

You provide four implementations:

Trait	Method	Shipped implementations
SttBackend	transcribe(&self, pcm: &[f32]) -> RiverbaseResult<String>	MockStt, HttpStt (feature audio-http-stt)
TtsBackend	synthesize(&self, text: &str) -> RiverbaseResult<Vec<u8>>	MockTts (returns silence)
AudioInvoker	invoke(&self, session_id: &str, transcript: &str) -> RiverbaseResult<String>	none; call your RiverBeam or domain here
AudioSink	send_text(ServerMessage), send_bytes(Vec<u8>)	none; write to your WebSocket here

HttpStt::new(base_url, api_key) posts the audio to {base_url}/audio/transcriptions as multipart, with the model whisper-1, and reads the text field of the response. The audio-whisper feature is an alias for audio-http-stt.

An AudioInvoker that calls the pipeline:

use std::sync::Arc;

use async_trait::async_trait;
use beam_audio::AudioInvoker;
use beam_core::RiverBeam;
use beam_types::{ContentPart, InvokeRequest};
use riverbase_core::RiverbaseResult;
use uuid::Uuid;

struct BeamInvoker {
    beam: Arc<RiverBeam>,
    agent_name: String,
}

#[async_trait]
impl AudioInvoker for BeamInvoker {
    async fn invoke(&self, session_id: &str, transcript: &str) -> RiverbaseResult<String> {
        let request = InvokeRequest {
            agent_name: self.agent_name.clone(),
            content: vec![ContentPart::text(transcript)],
            session_id: Uuid::parse_str(session_id).unwrap_or_else(|_| Uuid::new_v4()),
            model_override: None,
            metadata: serde_json::Value::Null,
            interaction_id: None,
            request_id: None,
            stream: false,
            actions: None,
            auth_context: None,
        };
        let result = self.beam.invoke_result(request).await?;
        Ok(result.content.iter().filter_map(|p| p.text.as_deref()).collect::<Vec<_>>().join(" "))
    }
}

Audio formats

Turns accept WAV. parse_wav_pcm_f32 reads it, resample_pcm converts the sample rate, and pcm_f32_to_wav_bytes writes it. Other containers become silence inside run_audio_turn. decode_to_pcm_16k returns BEM-352 for WebM (with feature audio-decode) or BEM-353, since no WebM or MP4 decoder ships.

WebSocket protocol

The types for a WebSocket session:

Client → server

• Binary frames: audio bytes, appended to the session buffer.
• Text frames: a ControlMessage, {"type": "start" | "end_of_speech" | "interrupt" | "stop" | "pong"}.

Server → client

• Text frames: a ServerMessage, tagged by type: state, ping, text (text, seq, final, phase), audio_start, audio_end, error.
• Binary frames: WAV audio.

AudioSession tracks the state and the buffer:

A typical handler sends state: LISTENING on connect, appends binary frames with append_audio, and on end_of_speech takes the buffer with take_buffer and runs a turn. interrupt should bump tts_seq, send audio_end, and return to LISTENING; stop clears the buffer and goes IDLE.

Limit	Value	When exceeded
MAX_AUDIO_CHUNK_BYTES	256 KiB per frame	append_audio returns ChunkTooLarge
MAX_AUDIO_BUFFER_BYTES	8 MiB per utterance	BufferOverflow; the buffer is cleared
MIN_UTTERANCE_BYTES	4 KiB	Reject shorter utterances in your handler
MAX_MSG_PER_SEC	100	record_message(now) returns true (the limit is exceeded)

HEARTBEAT_INTERVAL_SEC (15) and HEARTBEAT_TIMEOUT_SEC (45) are provided for your ping loop.

Current limitations

• No production text-to-speech backend ships; MockTts returns silence.
• Only WAV input is decoded.
• set_state refuses a transition to the current state and returns false.