> ## Documentation Index
> Fetch the complete documentation index at: https://docs.60db.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# STT WebSocket
> Speech-to-Text WebSocket endpoint for real-time transcription
# STT WebSocket
Real-time Speech-to-Text transcription via WebSocket streaming with support for 39 languages (including code-switched Indic+English) and telephony integration. Powered by 60db STT v01 (a non-hallucinating, multi-backend speech recognition stack).
## Endpoint
`ws://api.60db.ai/ws/stt` or `wss://api.60db.ai/ws/stt`
## Authentication
Query parameter authentication:
Your API key for authentication
JWT token (alternative to API key)
Workspace ID for billing. Required when using JWT auth. API keys are automatically pinned to their workspace.
Examples:
```
ws://api.60db.ai/ws/stt?apiKey=sk_live_your_api_key
ws://api.60db.ai/ws/stt?token=eyJ...&workspace_id=24
```
The WebSocket connection checks **workspace wallet balance** before starting a session. If the workspace has insufficient credits, the connection is closed with a `1008` status code and an `INSUFFICIENT_CREDITS` error.
## Connection Details
| Property | Value |
| -------------- | ---------------------------------------------------- |
| Protocol | WebSocket (RFC 6455) |
| Frame types | Binary (telephony) or Text/JSON (browser) |
| Ping/keepalive | Server sends WebSocket pings every 30s (timeout 10s) |
The same port accepts plain HTTP GET requests and responds `200 ok` — safe for load-balancer health checks.
## Session Lifecycle
```
Client Server
│ │
│──── TCP/TLS connect ───────────────────►│
│◄─── {"type":"connecting", ...} ─────────│ authenticating...
│◄─── {"type":"connected", server_info} ──│ proxy wired to upstream
│◄─── {"type":"connection_established"} ──│ ready — safe to send `start`
│ │
│──── {"type":"start", ...} ─────────────►│
│◄─── {"type":"session_started", ...} ────│ safe to send audio
│ │
│──── audio frames / JSON audio ─────────►│
│◄─── {"type":"speech_started"} ──────────│ VAD detected voice
│◄─── {"type":"transcription", ...} ──────│ is_final=true, speech_final=false (first emit, context only)
│◄─── {"type":"transcription", ...} ──────│ is_final=true, speech_final=true (canonical answer)
│ │
│──── {"type":"stop"} ────────────────────►│
│◄─── {"type":"session_stopped"} ─────────│
```
**Two-phase finals (context-gated LLM refinement).** When you supply a `context` object on `start`, every utterance produces **two** `transcription` events sharing a `sentence_id`:
1. **First emit** — `is_final: true, speech_final: false` — fast dict-corrected text. Use for low-latency UI paint and barge-in.
2. **Canonical** — `is_final: true, speech_final: true` — definitive LLM-refined answer. Always arrives.
When `context` is **omitted**, every utterance produces a single `transcription` event with `is_final: true, speech_final: true` (no first emit). Simple consumers can gate exclusively on `speech_final: true` regardless of whether refinement is on.
**Do not send the `start` message until `connection_established` is received.** The backend proxy attaches its client-message listener only after authenticating and opening the upstream connection. Messages sent earlier will be silently dropped. Likewise, **do not send `audio` messages until `session_started` is received** — the upstream returns `unknown message type: audio` if it arrives before the session is ready.
## Client → Server Messages
### `start` — Begin session
Sent once after connection is established. Must be sent before any audio.
```json Request theme={null}
{
"type": "start",
"languages": ["en", "hi"],
"context": {
"general": [
{ "key": "domain", "value": "Healthcare" },
{ "key": "doctor", "value": "Dr. Martha Smith" }
],
"text": "Routine diabetes follow-up consultation.",
"terms": ["Celebrex", "Zyrtec", "Metformin", "HbA1c"]
},
"config": {
"encoding": "linear",
"sample_rate": 48000,
"utterance_end_ms": 500,
"continuous_mode": true,
"interim_results_frequency": 300,
"audio_enhancement": "adaptive",
"diarize": false,
"remove_fillers": false
}
}
```
**Parameters:**
Array of ISO 639-1 language codes from the supported set (see `GET /stt/languages`), e.g. `["en", "hi"]`.
* `["en"]` — fast path, lowest latency
* `["en","hi"]` — shared-backend multi-language (the Indic+English pipeline handles both natively), no LID overhead
* `null` or omit — auto-detect across all 39 v1 languages
* Arabic dialect tags (`ar-eg`, `ar-lv`, …) are rejected; pass `ar` for best-effort MSA
* Max 5 languages per session
**Unsupported:** `ur`, `ja`, `ko`, `zh`, `th`, `vi`, `id`, `tl`, `sw`, `tr`, `fa`, `he` — these return an `unsupported_language` error.
**Never send `languages: "auto"` or `languages: ["auto"]`.** The auto-detect entry in `GET /stt/languages` is a convenience for the REST `/stt` form-upload flow only. On WebSocket, the server only accepts real ISO codes and uses `null` as the auto-detect signal. Sending `"auto"` returns `language 'auto' is not in the v1 supported`. The 60db WebSocket proxy (`/ws/stt`) strips the string `"auto"` from incoming `start` and `config` messages as a safety net, but your client should send `null` directly.
Optional hint object `{general, text, terms}` that opens the server-side LLM refinement gate. When supplied, each utterance emits **two** `transcription` events sharing a `sentence_id` — a fast first emit (`speech_final: false`) with dict-corrected text, followed \~300–700 ms later by a canonical emit (`speech_final: true`) with LLM-refined text (proper nouns corrected, fillers removed, punctuation added, script consistency enforced). See [Canonical-answer semantics](#canonical-answer-semantics-speech-final) below.
* `general` — array of `{key, value}` pairs. Free-form metadata (`domain`, `topic`, speaker names) surfaced to the LLM verbatim as hint lines.
* `text` — background paragraph describing the session. Useful for narrative context.
* `terms` — array of proper nouns, acronyms, and domain-specific jargon to preserve in the transcript.
All three fields are optional; at least one should be populated. Omit `context` entirely to disable refinement for the session — finals then arrive once as plain `transcription` events.
Audio encoding format. Use `"linear"` for browser Int16 PCM, `"mulaw"` for G.711 telephony. Options: `"linear"`, `"mulaw"`. The first raw binary frame auto-selects `mulaw`.
Actual sample rate of the audio being sent. Server resamples to 16 kHz internally with stateful `audioop.ratecv`. Options: `8000`, `16000`, `24000`, `44100`, `48000`. Must match the real capture rate — do not hardcode 16000 on 48 kHz browser input.
Silence duration (ms) after last speech chunk before finalizing the utterance. **Values below 300 ms are silently clamped to 300 ms.** Recommended 500–1000 ms for voicebots, 300 ms for barge-in heavy flows.
Keep session alive between utterances. Required for voicebot / phone call use cases. `false` = single-shot STT that stops after the first final.
How often (ms) to emit interim partial results during speech. **Values below 300 ms are silently clamped to 300 ms.** Use 300 for barge-in, 500 otherwise, omit to disable.
Run pyannote speaker diarization on each finalized utterance and attach a `speakers` array. Requires `HF_TOKEN` on the server. Adds \~50–150 ms latency per final. Check `session_started.diarize` to confirm the request was accepted.
Lower bound on diarization speaker count. `0` and negative values are silently clamped to `null`. Only read when `diarize=true`.
Upper bound on diarization speaker count. Only read when `diarize=true`.
Real-time audio enhancement to improve transcription quality on noisy input. Options:
| Value | Description |
| ------------ | -------------------------------------------------------------------------------------------------- |
| `"off"` | No audio processing (default) |
| `"light"` | Noise reduction only — best for mildly noisy environments |
| `"adaptive"` | Automatic noise reduction + gain control based on input levels — best for variable/telephony audio |
Use `"adaptive"` for telephony or noisy environments, `"light"` when you only need mild cleanup, and `"off"` when the input is already clean.
Ask the LLM refinement pass to strip filler words (`um`, `uh`, `like`, `you know`, …) from the canonical transcript. Only takes effect when `context` is set — refinement is gated on context, and the raw first-emit still contains the fillers. Non-boolean values are coerced to `false` by the 60db proxy.
Reserved for legacy client compatibility. **Ignored by 60db STT** — the non-hallucinating backend never emits a `no_speech_prob`. Safe to pass for migration convenience.
### `audio` — JSON audio chunk (browser mode)
```json Request theme={null}
{
"type": "audio",
"audio": "",
"encoding": "linear",
"sample_rate": 48000,
"timestamp": 1700000000000
}
```
**Fields:**
Must be `"audio"`
Base64-encoded audio bytes (Int16 PCM or μ-law)
`"linear"` or `"mulaw"`
Actual sample rate of the audio
Unix ms timestamp — useful for latency measurement
### Binary frame — raw μ-law audio (telephony mode)
Send a raw WebSocket binary frame with μ-law bytes, no JSON wrapper.
The server auto-detects this as telephony mode on the first binary frame.
```
Recommended chunk size: 480 bytes = 60ms at 8kHz
Twilio default: 160 bytes = 20ms — batch 3 chunks into 60ms before sending
```
### `config` — Change language mid-session
```json Request theme={null}
{
"type": "config",
"languages": ["hi"],
"continuous_mode": true
}
```
Both `languages` and `continuous_mode` are optional; include only fields you want to change.
Send `"languages": null` to revert to auto-detect.
### `stop` — End session
```json Request theme={null}
{
"type": "stop"
}
```
Server processes any remaining audio buffer, sends `session_stopped`, then closes.
### `test` — Ping / latency check
```json Request theme={null}
{
"type": "test",
"message": "ping",
"timestamp": 1700000000000
}
```
Server echoes `test_response` with the same `timestamp` for round-trip measurement.
## Server → Client Messages
### `connecting` — Authentication in progress
```json Response theme={null}
{
"connecting": true,
"message": "Authenticating...",
"timestamp": 1775465918269
}
```
### `connection_established` — Authentication successful
```json Response theme={null}
{
"connection_established": {
"service": "stt",
"user_id": 43,
"credit_balance": 9.97,
"workspace": "default"
}
}
```
**Fields:**
Service name: `"stt"`
Your user ID
Available credits
Workspace name
### `connected` — Proxy wired to upstream STT server
Sent by the backend proxy after it has opened the upstream connection and attached its client-message listener. **After this point, client messages are no longer dropped.**
```json Response theme={null}
{
"type": "connected",
"server_info": {
"server_type": "60db STT",
"ready": true,
"total_languages": 40,
"features": {
"vad_segmentation": true,
"multi_language_per_session": true,
"max_languages_per_session": 5,
"telephony_mulaw": true,
"browser_pcm": true,
"continuous_mode": true,
"interim_results": true,
"speaker_diarization": true,
"code_switching_indic_english": true,
"non_hallucinating": true
},
"timing": {
"min_utterance_end_ms": 300,
"default_utterance_end_ms": 500,
"max_utterance_seconds": 30
}
}
}
```
### `session_started` — `start` message processed, audio is now accepted
Sent by the upstream STT server after a `start` message is received and validated. It is safe to begin sending `audio` frames immediately after this event.
```json Response theme={null}
{
"type": "session_started",
"session_id": "sess_8c3d1a9f4b7e2c51",
"language": "Multi-language: EN, HI",
"languages": ["en", "hi"],
"device": "cuda:0",
"model": "60db-stt-v01",
"processing_mode": "sentence_based_continuous",
"continuous_mode": true,
"interim_frequency": 300,
"diarize": false
}
```
### `speech_started` — VAD detected voice activity
```json Response theme={null}
{
"type": "speech_started",
"timestamp": 1700000000.123
}
```
Use this for barge-in: interrupt TTS playback when this arrives.
Fired after 2 consecutive VAD-positive chunks (\~64ms of confirmed speech).
### `transcription` — Transcription result
All results (interim and final) share the same `transcription` type — differentiate with flags.
**Final result** (`is_final=true`, `speech_final=true`):
```json Response theme={null}
{
"type": "transcription",
"text": "Hello, how are you?",
"confidence": 0.87,
"language": "en",
"language_name": "EN",
"is_final": true,
"speech_final": true,
"is_partial": false,
"sentence_id": 3,
"duration": 1.82,
"latency": 0.43,
"timestamp": 1700000000.456,
"words": [
{ "word": "Hello", "start": 0.0, "end": 0.32, "confidence": 0.94 },
{ "word": "how", "start": 0.35, "end": 0.52, "confidence": 0.92 }
],
"speakers": [
{ "speaker": "SPEAKER_00", "start": 0.0, "end": 1.82 }
]
}
```
**Empty speech\_final signal** (`text=""`, `is_final=true`, `speech_final=true`):
Sent when audio was detected but transcription was rejected (silence, hallucination, low confidence, wrong language).
Client should reset its state on this message and not treat it as an error.
```json Response theme={null}
{
"type": "transcription",
"text": "",
"confidence": 0.0,
"is_final": true,
"speech_final": true,
"processing_mode": "speech_end_no_result",
"timestamp": 1700000000.789
}
```
**Interim result** (`is_final=false`, `speech_final=false`) — only sent when `interim_results_frequency` is set:
```json Response theme={null}
{
"type": "transcription",
"text": "Hello how",
"confidence": 0.72,
"language": "en",
"is_final": false,
"speech_final": false,
"is_partial": true
}
```
Use interims only for barge-in word-count checks. Never send interim text to the LLM — a final with `is_final=true, speech_final=true` will follow.
**Response Fields:**
Transcribed text. Empty string = speech-end-no-result signal.
0.0–1.0. Telephony typically 0.35–0.75; browser 0.55–0.95.
Detected language code e.g. `"en"`.
Uppercase language code (e.g. `"EN"`) in the WS shape. Note: REST `/stt` returns the full English name (`"English"`) here — WS preserves the legacy uppercase-code shape for client compatibility.
`true` = end of speech reached. May still be followed by a canonical upgrade if LLM refinement is active.
`true` = canonical answer, will not be revised. When LLM refinement is on, one `is_final: true, speech_final: false` event is followed by one `is_final: true, speech_final: true`. When refinement is off, every final is `speech_final: true`. See [Canonical-answer semantics](#canonical-answer-semantics-speech-final).
`true` for interim results only.
Monotonically increasing counter per session.
Duration (seconds) of the audio segment transcribed.
Seconds from processing start to result ready (excludes queue time).
Word-level timestamps `[{word, start, end, confidence, boosted?, original?}]`. Note: the field name is **`confidence`**, not `probability` (60db STT convention — different from legacy Whisper docs). Present on finals; empty on interims.
When the keyword/context-terms boost replaced a word, the entry includes
`boosted: true` and `original` (the pre-boost word):
```json theme={null}
{ "word": "Acme", "start": 1.5, "end": 2.0, "confidence": 0.85,
"boosted": true, "original": "akmie" }
```
The segment-level `text` is already rebuilt from boosted words upstream —
no client-side stitching required. Recommended UI: subtle underline on
`boosted: true` words, with `original` shown on hover.
List of `[{speaker, start, end}]` diarization turns when `config.diarize=true`. Omitted or `null` otherwise. Raw speaker IDs look like `SPEAKER_00`, `SPEAKER_01`; clients typically re-label these as "Speaker 1", "Speaker 2" in order of first appearance.
Marker for utterances that should be skipped by the consumer (no useful
text, **never billed**). Omitted on ordinary finals.
| Value | Meaning |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `speech_end_no_result` | Speech ended but the recognizer produced no text. |
| `speech_end_too_short` | Utterance below the minimum duration to recognize. |
| `hallucination_rejected` | Word-rate guard rejected output as likely hallucination. |
| `low_snr_dropped` | Audio dropped before LID/ASR; SNR below floor. **No credits charged.** Treat as "skip this utterance — no useful text". |
Audio signal-to-noise ratio (dB) for this utterance, when measured.
Optional. Surface as a "good / fair / poor" badge: `>= 15` good, `0–15`
fair, `< 0` poor.
### Canonical-answer semantics: `speech_final`
`is_final` and `speech_final` are **NOT identical** when LLM refinement is active — they split into two distinct meanings:
| `is_final` | `speech_final` | Meaning |
| ---------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `false` | `false` | Interim partial — text may still change as more audio arrives. |
| `true` | `false` | End-of-speech reached, dict-corrected text. **The LLM is still processing.** A follow-up event with `speech_final: true` will arrive shortly with the canonical text. Only emitted when refinement is active for this utterance. |
| `true` | `true` | **Canonical answer.** Definitive, will not be revised. LLM-refined text when context was supplied, otherwise the original ASR text. |
The same `sentence_id` is echoed across both phases so clients can reconcile.
**Canonical event example (after LLM refinement):**
```json Response theme={null}
{
"type": "transcription",
"sentence_id": 3,
"text": "डॉक्टर साहब, मेरा sugar level बहुत high है। Metformin की dose बढ़ाओ।",
"confidence": 0.87,
"language": "hi",
"language_name": "HI",
"is_final": true,
"speech_final": true,
"is_partial": false,
"duration": 1.82,
"words": [
{ "word": "डॉक्टर", "start": 0.0, "end": 0.32, "confidence": 0.94 }
],
"speakers": null,
"timestamp": 1700000000.789
}
```
**Guarantees:**
* **Exactly one canonical event per utterance.** When refinement is on, you get two `transcription` events per utterance (first emit + canonical). When refinement is off, you get one (`speech_final: true`). Never zero, never three.
* **Same `sentence_id`** across both phases. Reconcile on that key.
* **The canonical always arrives.** Consumers waiting on `speech_final: true` never hang.
* **`sentence_id` ordering is preserved per session**, but canonicals are NOT guaranteed to arrive in `sentence_id` order when LLM is on — two utterances finalizing close in time may complete refinement out of order. Key on `sentence_id`, not arrival order.
* **`words[]` corresponds to the original ASR output** on both phases — the LLM does not realign tokens. Use `words[]` for word-level timing, `text` for display.
**Recommended client patterns:**
*Simplest — don't care about the first-emit optimization:*
```js theme={null}
function onMessage(msg) {
if (msg.type === 'transcription' && msg.speech_final) {
// Canonical — render and forget
render(msg);
}
// Ignore is_final && !speech_final (intermediate, will be replaced)
// Ignore is_partial (interim, handle separately if needed)
}
```
*With fast first-emit (UX-aware):*
```js theme={null}
const livePainted = new Map(); // sentence_id → line slot
function onMessage(msg) {
if (msg.type !== 'transcription') return;
const sid = msg.sentence_id;
if (msg.is_partial) { renderPartial(msg); return; }
let entry = livePainted.get(sid);
if (!entry) {
entry = createLine();
livePainted.set(sid, entry);
}
entry.text = msg.text;
entry.pending = msg.is_final && !msg.speech_final; // dim while LLM runs
render(entry);
if (msg.speech_final) {
finalize(entry);
livePainted.delete(sid);
}
}
```
**For voicebot NLU routing**: feed the first-emit text (`speech_final: false`) to NLU immediately for fast intent dispatch — don't wait for canonical. If your NLU benefits from proper-noun accuracy (name-spelling slots, drug-name lookup), run a second-pass call on the canonical (`speech_final: true`) text and reconcile on `sentence_id`.
**Legacy `refined` event.** Earlier builds emitted a separate `refined` event \~400 ms after the final instead of a second `transcription`. The 60db `/ws/stt` proxy transparently handles both shapes — if you're still seeing `refined` events in the wire trace, upstream workers haven't been restarted onto the two-phase build yet. New client code should target the two-phase flow only; `refined` is accepted but deprecated.
### `language_changed` — After `config` message changes language
```json Response theme={null}
{
"type": "language_changed",
"language": "Multi-language: HI",
"language_code": ["hi"]
}
```
### `mode_changed` — After `config` message changes `continuous_mode`
```json Response theme={null}
{
"type": "mode_changed",
"continuous_mode": true,
"mode_name": "continuous",
"silence_threshold": 0.5
}
```
### `session_stopped` — After `stop` is processed
```json Response theme={null}
{
"type": "session_stopped",
"billing_summary": {
"total_duration_seconds": 12.40,
"total_cost": 0.000620,
"characters_transcribed": 188,
"client_estimated_seconds": 12.42
}
}
```
**`billing_summary` fields:**
| Field | Type | Notes |
| -------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `total_duration_seconds` | number | The audio actually billed — sum of canonical finals that passed the billable predicate. **Sessions that previously double-billed** when the client sent `stop` are now billed correctly; expect \~50% lower numbers for affected sessions vs historical values. |
| `total_cost` | number | USD cost for the session. |
| `characters_transcribed` | number | Total characters across canonical transcriptions. |
| `client_estimated_seconds` | number | **Diagnostic only** — rough estimate of audio the client sent. Useful for debugging duration drift; never display as a billed number. |
### `error` — Processing error
```json Response theme={null}
{
"type": "error",
"error": "Audio processing error: ...",
"timestamp": 1700000000.0
}
```
#### Concurrency-limit error frame
When a user has reached their per-user STT session cap (counted across REST + WS combined), the server sends an error frame and closes with code `1008`:
```json Response theme={null}
{
"type": "error",
"error": "Too many concurrent STT sessions for this user",
"error_code": "STT_CONCURRENCY_LIMIT",
"details": { "limit": 8 }
}
```
Existing sessions are unaffected; the cap releases when an in-flight session ends. **Do not auto-reconnect on `1008`** — the limit only frees when an in-flight session completes. The frontend should distinguish concurrency-limit closes from auth/network failures so the UI message is correct.
### `test_response` — Reply to `test` ping
```json Response theme={null}
{
"type": "test_response",
"message": "pong - Sentence-based STT ready",
"timestamp": 1700000000000,
"processing_mode": "sentence_based"
}
```
## Complete Example
```javascript theme={null}
const WebSocket = require('ws');
const API_KEY = 'sk_live_your_key';
const ws = new WebSocket(`ws://api.60db.ai/ws/stt?apiKey=${API_KEY}`);
ws.on('open', () => {
console.log('✓ Connected');
});
ws.on('message', (data) => {
const msg = JSON.parse(data);
console.log('←', msg.type || Object.keys(msg)[0]);
if (msg.connection_established) {
console.log(' User ID:', msg.connection_established.user_id);
console.log(' Credits:', msg.connection_established.credit_balance);
// Start session with config
ws.send(JSON.stringify({
type: 'start',
languages: ['en', 'hi'],
config: {
encoding: 'mulaw',
sample_rate: 8000,
continuous_mode: true,
utterance_end_ms: 500,
interim_results_frequency: 300
}
}));
} else if (msg.type === 'connected') {
console.log('✓ Session started! Send audio now...');
// Send audio chunks (480 bytes = ~60ms at 8kHz)
const audioInterval = setInterval(() => {
const audioChunk = getAudioChunk(); // Your audio capture function
ws.send(audioChunk);
}, 60);
// Stop after 5 seconds
setTimeout(() => {
clearInterval(audioInterval);
ws.send(JSON.stringify({ type: 'stop' }));
}, 5000);
} else if (msg.type === 'speech_started') {
console.log('🎤 Speech detected - barge-in opportunity');
} else if (msg.type === 'transcription') {
if (msg.is_final) {
console.log('✓ Final:', msg.text, `(confidence: ${msg.confidence})`);
} else {
console.log(' Partial:', msg.text);
}
} else if (msg.type === 'session_stopped') {
console.log('✓ Session stopped');
console.log(' Duration:', msg.billing_summary.total_duration_seconds, 's');
console.log(' Cost: $', msg.billing_summary.total_cost);
ws.close();
}
});
ws.on('error', (error) => {
console.error('Error:', error);
});
ws.on('close', () => {
console.log('Connection closed');
});
```
```python theme={null}
import asyncio
import json
import websockets
API_KEY = "sk_live_your_key"
url = f"ws://api.60db.ai/ws/stt?apiKey={API_KEY}"
async def stt_websocket():
async with websockets.connect(url) as ws:
# Wait for connection
msg = json.loads(await ws.recv())
if msg.get('connection_established'):
print(f"✓ Connected (User: {msg['connection_established']['user_id']})")
print(f" Credits: ${msg['connection_established']['credit_balance']}")
# Start session
await ws.send(json.dumps({
"type": "start",
"languages": ["en", "hi"],
"config": {
"encoding": "mulaw",
"sample_rate": 8000,
"continuous_mode": True,
"utterance_end_ms": 500,
"interim_results_frequency": 300
}
}))
# Wait for connected
msg = json.loads(await ws.recv())
assert msg["type"] == "connected"
print("✓ Session started!")
# Send audio for 5 seconds
for _ in range(83): # ~5000ms / 60ms
audio_chunk = get_audio_chunk() # Your audio capture function
await ws.send(audio_chunk) # Send as binary
await asyncio.sleep(0.06)
# Stop session
await ws.send(json.dumps({"type": "stop"}))
# Process remaining messages
while True:
msg = json.loads(await ws.recv())
msg_type = msg.get("type")
if msg_type == "speech_started":
print("🎤 Speech detected")
elif msg_type == "transcription":
if msg.get("is_final"):
print(f"✓ {msg['text']} (confidence: {msg['confidence']})")
else:
print(f" {msg['text']}...")
elif msg_type == "session_stopped":
print(f"✓ Session stopped")
print(f" Duration: {msg['billing_summary']['total_duration_seconds']}s")
print(f" Cost: ${msg['billing_summary']['total_cost']}")
break
asyncio.run(stt_websocket())
```
```javascript theme={null}
const ws = new WebSocket('ws://api.60db.ai/ws/stt?apiKey=sk_live_your_key');
let mediaRecorder;
let audioContext;
ws.onopen = () => {
console.log('✓ Connected');
// Start session
ws.send(JSON.stringify({
type: 'start',
languages: ['en'],
config: {
encoding: 'linear',
sample_rate: 48000,
continuous_mode: true,
interim_results_frequency: 300
}
}));
};
ws.onmessage = (event) => {
const msg = JSON.parse(event.data);
if (msg.type === 'connected') {
console.log('✓ Session started! Start speaking...');
startAudioCapture();
} else if (msg.type === 'speech_started') {
console.log('🎤 Speech detected');
} else if (msg.type === 'transcription') {
if (msg.is_final) {
console.log('✓', msg.text);
updateTranscriptDisplay(msg.text);
} else {
console.log('...', msg.text);
updateInterimDisplay(msg.text);
}
} else if (msg.type === 'session_stopped') {
console.log('✓ Session stopped');
stopAudioCapture();
ws.close();
}
};
async function startAudioCapture() {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
audioContext = new AudioContext({ sampleRate: 48000 });
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(4096, 1, 1);
processor.onaudioprocess = (e) => {
const inputData = e.inputBuffer.getChannelData(0);
const pcm16 = new Int16Array(inputData.length);
for (let i = 0; i < inputData.length; i++) {
pcm16[i] = Math.max(-32768, Math.min(32767, inputData[i] * 32768));
}
ws.send(JSON.stringify({
type: 'audio',
audio: btoa(String.fromCharCode(...new Uint8Array(pcm16.buffer))),
encoding: 'linear',
sample_rate: 48000
}));
};
source.connect(processor);
processor.connect(audioContext.destination);
mediaRecorder = { stream, processor, source };
}
function stopAudioCapture() {
if (mediaRecorder) {
mediaRecorder.stream.getTracks().forEach(track => track.stop());
mediaRecorder.processor.disconnect();
audioContext.close();
}
}
```
## Audio Requirements
| Property | Telephony (μ-law) | Browser (PCM) |
| ----------- | ----------------- | ----------------------------- |
| Encoding | `mulaw` (8-bit) | `linear` (16-bit) |
| Sample Rate | 8000 Hz | 16000, 24000, 44100, 48000 Hz |
| Chunk Size | 480 bytes (60ms) | 960-1920 bytes (60-120ms) |
| Channels | Mono (1 channel) | Mono (1 channel) |
## Supported Languages
39 transcription languages total (25 European, 13 Indic with Hinglish code-switching, and Arabic MSA). Fetch the full catalog from `GET /stt/languages`.
| Code | Language | Code | Language |
| ---- | ------------ | ---- | --------- |
| `en` | English | `hi` | Hindi |
| `es` | Spanish | `bn` | Bengali |
| `fr` | French | `mr` | Marathi |
| `de` | German | `pa` | Punjabi |
| `it` | Italian | `gu` | Gujarati |
| `pt` | Portuguese | `ta` | Tamil |
| `nl` | Dutch | `te` | Telugu |
| `pl` | Polish | `kn` | Kannada |
| `ru` | Russian | `ml` | Malayalam |
| `uk` | Ukrainian | `or` | Odia |
| `cs` | Czech | `as` | Assamese |
| `sv` | Swedish | `ne` | Nepali |
| `ar` | Arabic (MSA) | `sa` | Sanskrit |
**Code-switching (Indic + English):** `hi+en`, `bn+en`, `mr+en`, `pa+en`, `gu+en`, `or+en`, `as+en`, `ne+en`, `te+en`, `kn+en`, `ta+en`, `ml+en` — collapses to the fast path when both languages share the same pipeline.
**Not supported** (explicit rejection): `ur`, `ja`, `ko`, `zh`, `th`, `vi`, `id`, `tl`, `sw`, `tr`, `fa`, `he`. These return an `unsupported_language` error — there is **no silent aliasing**. Arabic dialect tags (`ar-eg`, `ar-lv`, `ar-gu`, `ar-ma`) return `dialect_not_supported` — pass `ar` for best-effort MSA transcription of dialectal audio.
## Limitations and Best Practices
**Handshake ordering (most common mistake)**
Do **not** send `start` in `ws.onopen`. Wait for the proxy's `connection_established` message first — it marks the point at which the proxy has attached its client-message listener. Likewise, wait for `session_started` before sending `audio` frames, otherwise the upstream server returns `unknown message type: audio`.
**`utterance_end_ms` clamp**
Values below 300 ms are silently clamped to 300 ms. Sub-300 ms would fragment utterances and cause the non-hallucinating backends to drop short segments.
**Language count**
Max 5 languages per session. Cross-backend multi-language (e.g. `["en","ar"]`) runs per-utterance LID which adds \~20–50 ms per utterance. Same-backend multi-language (e.g. `["en","hi"]`) collapses to a single backend's fast path with zero LID overhead — use this whenever possible.
**Telephony confidence**
8 kHz μ-law → 16 kHz resampling reduces backend confidence by \~0.10–0.15 compared to native wideband input. Use a client-side threshold of 0.35 for telephony vs 0.55 for browser.
**Buffer limits**
* Pre-speech ring buffer: 1.0 s (captures first word before VAD fires)
* Minimum coalesce before VAD: 160 ms (μ-law 1280 bytes, linear `sample_rate × 2 × 160 / 1000` bytes)
* Maximum utterance duration: 30 s (anything longer force-finalizes)
**VAD**
* Speech start: Silero probability > `STT_VAD_THRESHOLD` (default 0.5, server-configurable)
* Silence → utterance end: `utterance_end_ms` of consecutive sub-threshold audio
* No separate continuation threshold
**Non-hallucinating architecture**
The 60db STT backend uses non-hallucinating architectures that do not output `no_speech_prob` — the CTC/RNN-T topology emits blank tokens on non-speech. Hallucination guard is a word-rate sanity check (> 5 words/second → rejected).
**Diarization**
`diarize=true` requires `HF_TOKEN` on the server plus gated model approval for `pyannote/speaker-diarization-3.1`. Without both, the request silently falls back to a deterministic mock. Check `session_started.diarize` to confirm the request was accepted.
## Pricing
* **Rate**: \$0.00000833 per second
* **Minimum**: \$0.01 per session
* **Billing**: Per second of audio processed
## Error Codes
| Close code | `error_code` (in preceding error frame) | Description |
| ---------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| 1008 | `UNAUTHENTICATED` | Authentication failed |
| 1008 | `INSUFFICIENT_CREDITS` | Workspace wallet has no funds |
| 1008 | `STT_CONCURRENCY_LIMIT` | Per-user concurrency cap reached. `details.limit` carries the active cap. Do not auto-reconnect; cap releases when an in-flight session ends. |
| 1011 | — | Internal server error |
| 1006 | — | Connection lost |
## Testing
```bash theme={null}
# Install wscat
npm install -g wscat
# Test STT
wscat -c "ws://api.60db.ai/ws/stt?apiKey=sk_live_your_key"
```
Then send:
```json theme={null}
{"type":"start","languages":["en"],"config":{"encoding":"mulaw","sample_rate":8000,"continuous_mode":true}}
```
## Related
* [TTS WebSocket](/api-reference/websocket/tts) - Text-to-Speech endpoint
* [WebSocket API Reference](/websocket-api) - Complete documentation
* [Voices API](/api-reference/voices/get-voices) - Get available voices