Audio and Turn Detection

Choose an audio format that matches your source, then pick how the server decides when the user has finished speaking. The defaults work for most browser and desktop apps.

Audio formats

Format	Sample rate	Use for
`audio/pcm`	24 kHz, 16-bit mono (default)	Browsers, apps, Web Audio
`audio/pcmu`	8 kHz, μ-law	Telephony (US and most carriers)
`audio/pcma`	8 kHz, A-law	Telephony (EU and international)

Set the format on input and output in session:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": { "format": { "type": "audio/pcm", "rate": 24000 } },
      "output": { "format": { "type": "audio/pcm", "rate": 24000 }, "voice": "YOUR_VOICE_ID" }
    }
  }
}

Send audio as base64 in input_audio_buffer.append. Send continuous frames, including silence, so the server can detect speech. Frames of about 100 ms work well.

For audio/pcmu and audio/pcma, send 8 kHz G.711 audio encoded as base64. The server decodes it.

Turn detection

audio.input.turn_detection controls when a user turn ends and the reply starts.

Mode	Config	Behavior
Server VAD	`{ "type": "server_vad", "silence_duration_ms": 1000 }`	Ends the turn after silence. `silence_duration_ms` has a minimum of 1000
Semantic VAD	`{ "type": "semantic_vad" }`	Waits longer when the sentence sounds unfinished
Manual	`null`	You control the turn with `input_audio_buffer.commit` and `response.create`

Server VAD (default)

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": {
        "turn_detection": { "type": "server_vad", "silence_duration_ms": 1000 }
      }
    }
  }
}

Manual turns

Use manual mode for push-to-talk or when your app decides when to respond.

{ "type": "session.update", "session": { "type": "realtime", "audio": { "input": { "turn_detection": null } } } }

{ "type": "input_audio_buffer.commit" }
{ "type": "response.create" }

Send commit when the user releases the button, then response.create to get the reply.

Barge-in

The user can talk over the reply.

The server confirms the interruption from the transcript. Backchannels such as “ok” or “haan” do not cut off the reply.
When a real interruption is confirmed, the server emits input_audio_buffer.speech_started. Stop local playback at that point.
Tell the server how much audio the user heard with conversation.item.truncate so the next turn’s context matches what was played.

Transcription languages

The server transcribes user speech and replies in the language the user speaks, in its native script. Set audio.input.transcription.language to one code or an array of codes. Arrays are a 60db extension.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": { "transcription": { "language": ["en", "hi"] } }
    }
  }
}

When you omit the field, the default is English and Hindi.

Get Started

Guides

Reference

Audio and Turn Detection

Audio formats

Turn detection

Server VAD (default)

Manual turns

Barge-in

Transcription languages

​Audio formats

​Turn detection

​Server VAD (default)

​Manual turns

​Barge-in

​Transcription languages

​Related

Audio formats

Turn detection

Server VAD (default)

Manual turns

Barge-in

Transcription languages

Related