STT WebSocket API

Real-time Speech-to-Text transcription via WebSocket streaming with support for 39 languages (including code-switched Indic+English) and telephony integration. Powered by 60db STT v01 (a non-hallucinating, multi-backend speech recognition stack).

🚀 Quick Start (Copy & Paste)

const WebSocket = require('ws');

// 1. Your API key
const API_KEY = 'sk_live_your_api_key';

// 2. Connect
const ws = new WebSocket(`wss://api.60db.ai/ws/stt?apiKey=${API_KEY}`);

// 3. Handle messages
ws.on('message', (data) => {
  const msg = JSON.parse(data);

  // Authenticated? Start session!
  // NOTE: on STT the fields are top-level and the frame is tagged by `type`.
  if (msg.type === 'connection_established') {
    console.log('✅ Authenticated');
    ws.send(JSON.stringify({
      type: 'start',
      languages: ['en'],
      config: {
        encoding: 'mulaw',
        sample_rate: 8000,
        audio_enhancement: 'on',   // recommended for phone audio
        early_final: false,        // keep whole sentences in one final
        turn_detector: true,       // wait until the caller finished the thought
        interim_results_frequency: 500
      }
    }));
  }

  // Session ready? Send audio!
  // Gate on `session_started`, not `connected` — `connected` fires before the
  // session exists and arrives twice (proxy frame, then upstream frame).
  if (msg.type === 'session_started') {
    console.log('✅ Ready! Send audio now');

    // Send dummy audio (480 bytes every 60ms)
    let count = 0;
    const interval = setInterval(() => {
      ws.send(Buffer.alloc(480, 0xff));
      if (++count >= 83) {  // 5 seconds
        clearInterval(interval);
        ws.send(JSON.stringify({ type: 'stop' }));
      }
    }, 60);
  }

  // Got text!
  if (msg.type === 'transcription' && msg.is_final) {
    console.log('📝', msg.text);
  }

  // Done!
  if (msg.type === 'session_stopped') {
    console.log('✅ Complete! Cost:', msg.billing_summary.total_cost);
    ws.close();
  }
});

That’s it! You’ll see:

✅ Authenticated
✅ Ready! Send audio now
📝 Hello world (transcribed text)
✅ Complete! Cost: $0.000043

📖 How It Works (5 Simple Steps)

Connect with your API key
Send { type: "start", ... } to begin session
Stream audio data (binary chunks)
Receive text transcriptions in real-time
Stop with { type: "stop" } when done

Endpoint

Authentication

Query parameter authentication: Example:

ws://api.60db.ai/ws/stt?apiKey=sk_live_your_api_key

Connection Details

Property	Value
Protocol	WebSocket (RFC 6455)
Frame types	Binary (telephony) or Text/JSON (browser)
Ping/keepalive	The server answers client-initiated WebSocket pings with a pong; it does not ping first. Send a ping every 20–30 s so intermediaries do not idle the socket out.

Session Lifecycle

Client                          Server
  │                               │
  │──── TCP/TLS connect ─────────►│
  │◄─── {"type":"connecting"} ────│  authenticating...
  │◄─── {"type":"connected"} ─────│  proxy wired to upstream
  │◄─── connection_established ──│  authenticated — safe to send `start`
  │                               │
  │──── {"type":"start", ...} ───►│
  │◄─── {"type":"session_started"}│  safe to send audio
  │                               │
  │──── audio frames / messages ─►│
  │◄─── {"type":"speech_started"} │  VAD detected voice
  │◄─── {"type":"transcription"}  │  is_final=false (interim, optional, repeats)
  │◄─── {"type":"turn_complete"}  │  only with turn_detector=true
  │◄─── {"type":"transcription"}  │  is_final=true, speech_final=true (committed final)
  │                or             │
  │◄─── {"type":"no_speech"}      │  terminal=true — the turn had no words
  │                               │
  │──── {"type":"stop"} ─────────►│
  │◄─── {"type":"session_stopped"}│

One committed final per stretch of audio. Every result is a transcription with is_final: true, and speech_final mirrors is_final (there is no separate “canonical” phase and no LLM rewriting — see §Text is never rewritten). A single caller turn can still produce more than one final:

a short pause can split a sentence into two finals (with early_final: true);
a monologue longer than max_utterance_sec emits one or more segment_final: true chunks, then a normal final.

Each final covers a non-overlapping slice of audio, so append them in arrival order. De-duplicate on final_id. A turn that contained no words ends with a terminal no_speech instead.

Text is never rewritten

After recognition, the server never rewrites the script or re-guesses the language, and there is no LLM refinement pass. The only post-recognition change is strict keyword/term spelling correction from keywords / context.terms (force tuner → Fortuner). keywords correct near-miss spellings in place; word timings are preserved. Everything else in text is exactly what the recognizer produced.

Client → Server Messages

`start` — Begin session

Sent once after connection is established. Must be sent before any audio.

{
  "type": "start",
  "languages": ["en", "hi"],
  "keywords": ["Metformin", "HbA1c", "Celebrex"],
  "context": {
    "general": [
      { "key": "domain",  "value": "Healthcare" },
      { "key": "doctor",  "value": "Dr. Martha Smith" }
    ],
    "text":  "Routine diabetes follow-up consultation.",
    "terms": ["Celebrex", "Zyrtec", "Metformin", "HbA1c"]
  },
  "config": {
    "encoding": "mulaw",
    "sample_rate": 8000,
    "utterance_end_ms": 1000,
    "early_final": false,
    "turn_detector": true,
    "interim_results_frequency": 500,
    "audio_enhancement": "on",
    "diarize": false
  }
}

Parameters:

Turn-ending profiles

early_final and turn_detector together decide how fast a turn closes and whether a sentence with a mid-pause stays in one final. Pick one profile:

Profile	`turn_detector`	`early_final`	Time to final	Sentences
A. Whole sentences (recommended)	`true`	`false`	~2 s	Kept whole
B. Balanced	`false`	`false`	~1.5 s	A pause over 1 s splits
C. Fastest (server default)	`false`	`true`	~1.0 s	A pause over ~300 ms can split

Profile A is best for voicebots where a cut-off sentence causes a wrong answer. Profiles B and C are faster; join split finals in your middle layer if you use them. No words are lost with any profile — only where turns are cut changes. Confirm the effective values in session_started.

`audio` — JSON audio chunk (browser mode)

{
  "type": "audio",
  "audio": "<base64-encoded Int16 PCM or μ-law bytes>",
  "encoding": "linear",
  "sample_rate": 48000,
  "timestamp": 1700000000000
}

Fields:

Binary frame — raw μ-law audio (telephony mode)

Send a raw WebSocket binary frame with μ-law bytes, no JSON wrapper. The server auto-detects this as telephony mode on the first binary frame.

Recommended chunk size: 480 bytes = 60ms at 8kHz
Twilio default: 160 bytes = 20ms — batch 3 chunks into 60ms before sending

`config` — Change settings mid-session

{
  "type": "config",
  "languages": ["hi"],
  "turn_detector": true,
  "utterance_end_ms": 1200
}

Send only the fields you want to change. Send "languages": null to revert to auto-detect. Most changes are acknowledged with a *_changed event: language_changed, turn_detector_changed, utterance_end_changed, interim_frequency_changed, interim_delta_streaming_changed, token_streaming_changed, max_utterance_changed, audio_enhancement_changed, audio_format_changed, keywords_changed, context_changed. If any value is invalid you get an error and nothing in that message is applied. diarize, min_speakers and max_speakers can only be set in start. Changing languages mid-sentence first finalizes the current sentence, then switches — prefer switching between turns.

`finalize` — End the current turn now

{
  "type": "finalize"
}

Closes the current turn immediately and returns its transcription (or no_speech), or nothing if no audio is buffered. The session continues. Use it when your application already knows the caller is done — e.g. after a DTMF key press.

`stop` — End session

{
  "type": "stop"
}

Server processes any remaining audio buffer, sends session_stopped, then closes.

`test` — Ping / latency check

{
  "type": "test",
  "message": "ping",
  "timestamp": 1700000000000
}

Server echoes test_response with the same timestamp for round-trip measurement.

Server → Client Messages

`connecting` — Authentication in progress

{
  "type": "connecting",
  "message": "Authenticating...",
  "timestamp": 1775465918269
}

`connection_established` — Authentication successful

{
  "type": "connection_established",
  "service": "stt",
  "user_id": 43,
  "credit_balance": 9.97,
  "workspace": "default"
}

On STT these fields are top-level and the frame is tagged by type. Check msg.type === "connection_established" — msg.connection_established is the TTS frame’s shape, and a client written against it will never start its STT session.

Fields:

string

Service name: "stt"

integer

Your user ID

number

Available credits

string

Workspace name

`connected` — After `start` message is processed

{
  "type": "connected",
  "server_info": {
    "server_type": "60db STT",
    "device": "cuda",
    "model": "60db-stt-v01",
    "processing_mode": "sentence_based_modular",
    "supported_languages": { "en": "English", "hi": "Hindi" },
    "total_languages": 40,
    "features": {
      "sentence_based_processing": true,
      "real_time_streaming": true,
      "telephony_support": true,
      "unicode_support": true,
      "mixed_language_support": true
    }
  }
}

Expect two connected frames: the proxy’s own handshake (session_id plus a small server_info), then the upstream capability frame shown above. Neither means “audio is accepted” — wait for session_started. Keep any connected handler idempotent so audio capture does not start twice.

`session_started` — `start` accepted, audio is now allowed

{
  "type": "session_started",
  "session_id": "sess_8c3d1a9f4b7e2c51",
  "languages": ["en", "hi"],
  "model": "60db-stt-v01",
  "encoding": "mulaw",
  "sample_rate": 8000,
  "audio_enhancement": "on",
  "utterance_end_ms": 1000,
  "early_final": false,
  "turn_detector": true,
  "interim_frequency": 500,
  "interim_min_voiced_ms": null,
  "interim_delta_streaming": false,
  "token_streaming": false,
  "max_utterance_seconds": 30.0,
  "diarize": false,
  "ignored_config_keys": []
}

Trust this event over your request. It echoes the effective settings after normalization — check early_final/turn_detector here to confirm which profile the session is running. interim_frequency is the effective interim_results_frequency (null = off); interim_min_voiced_ms: null means the default (600). ignored_config_keys lists every key the server did not recognise (e.g. a browser echoCancellation) — unknown keys never fail a session. Send audio only after this event.

`speech_started` — VAD detected voice activity

{
  "type": "speech_started",
  "timestamp": 1700000000.123
}

Use this for barge-in: interrupt TTS playback when this arrives. Fired after 2 consecutive VAD-positive chunks (~64ms of confirmed speech).

`transcription` — Transcription result

All results (interim and final) share the same transcription type — differentiate with flags. Final result (is_final=true, speech_final=true):

{
  "type": "transcription",
  "text": "Hello, how are you?",
  "confidence": 0.87,
  "language": "en",
  "language_name": "EN",
  "is_final": true,
  "speech_final": true,
  "segment_final": false,
  "is_partial": false,
  "utterance_id": 3,
  "sentence_id": 3,
  "final_id": "aa653fdf0a5c4372aee5d3af4dac2164:3",
  "processing_mode": "sentence_complete",
  "endpoint_reason": "vad_silence",
  "duration": 1.82,
  "latency": 0.43,
  "timestamp": 1700000000.456,
  "words": [
    { "word": "Hello", "start": 0.0, "end": 0.32, "probability": 0.94 },
    { "word": "how",   "start": 0.35, "end": 0.52, "probability": 0.92 }
  ],
  "utterance_end_ms": 1820
}

Segment final (segment_final=true) — max_utterance_sec cut a long monologue; the same turn continues, more finals follow:

{
  "type": "transcription",
  "text": "…",
  "is_final": true,
  "speech_final": true,
  "segment_final": true,
  "processing_mode": "max_utterance_segment",
  "endpoint_reason": "max_utterance"
}

Withheld final (text="", processing_mode="low_confidence_transcript"): Sent only when the text was too short and too unsure to trust — confidence below 0.55, at most 4 words, at most 14 characters. The event still arrives so you can clear the interim it replaces; it is not billed. Longer or confident text is never withheld, and a continuation after a short-pause split is never withheld.

{
  "type": "transcription",
  "text": "",
  "is_final": true,
  "speech_final": true,
  "segment_final": false,
  "processing_mode": "low_confidence_transcript",
  "words": []
}

Interim result (is_final=false, speech_final=false) — only sent when interim_results_frequency is set:

{
  "type": "transcription",
  "text": "Hello how",
  "confidence": 0.72,
  "language": "en",
  "is_final": false,
  "speech_final": false,
  "is_partial": true
}

Use interims only for barge-in word-count checks. Never send interim text to the LLM — a final with is_final=true, speech_final=true will follow. Response Fields:

string

Transcribed text. Empty string = speech-end-no-result signal.

number

0.0–1.0. Telephony typically 0.35–0.75; browser 0.55–0.95.

string

Detected language code e.g. "en".

string

Uppercase language code e.g. "EN".

boolean

false = interim · true = committed final for this stretch of audio.

boolean

Same as is_final. The two flags always agree — there is no separate “canonical” phase.

boolean

true = this final was cut by the max_utterance_sec length cap and more of the same turn follows. Check this before speech_final so a segment isn’t treated as the end of the turn.

boolean

true for interim results only.

integer

Turn number. Interims carry the id their final will have. sentence_id is the same value.

string

Unique id of a committed result. Use it to de-duplicate — if a network retry delivers one twice, drop the duplicate.

string

What ended the turn: vad_silence, client_finalize, client_stop, max_utterance, language_change, or response_timeout. interim on interims.

number

Seconds of audio covered by this final. (utterance_end_ms is the same value in milliseconds.)

number

Server processing time in seconds.

array

Finals only: [{ word, start, end, probability }], seconds from turn start. The score field is probability. Empty on interims. For Indian languages the times are evenly spread estimates and probability repeats confidence.

integer

duration in milliseconds.

string

How this result was produced. On ordinary results: streaming_interim (interim), sentence_complete (final), max_utterance_segment (segment final). On a withheld final: low_confidence_transcript (empty text; see Withheld final above). Treat unknown values as “withheld”.

string | null

Only with diarize: true — dominant speaker label for the final (also speaker_label, speakers).

`language_changed` — After `config` message changes language

{
  "type": "language_changed",
  "language": "Multi-language: HI",
  "language_code": ["hi"]
}

Other config changes are acknowledged with their own *_changed frame (turn_detector_changed, utterance_end_changed, interim_frequency_changed, audio_enhancement_changed, audio_format_changed, keywords_changed, context_changed, …).

`no_speech` — The turn had no words

{
  "type": "no_speech",
  "terminal": true,
  "outcome_type": "no_speech",
  "final_id": "aa653fdf0a5c4372aee5d3af4dac2164:6",
  "is_final": true,
  "speech_final": true,
  "segment_final": false,
  "processing_mode": "vad_silence",
  "endpoint_reason": "vad_silence",
  "duration": 3.7,
  "utterance_id": 6,
  "timestamp": 1700000000.5
}

terminal: true — the turn is over with no words. Clear any interim text; if you collected segment finals for this turn, dispatch them now. Not billed.
terminal: false — an interim check found no words; the turn is still open. Ignore it (or clear the interim display). There is no text field.

`turn_complete` — End-of-thought confirmed (only with `turn_detector: true`)

Sent right before the final it confirms.

{
  "type": "turn_complete",
  "complete": true,
  "forced": false,
  "probability": 0.983,
  "threshold": 0.5,
  "silence_ms": 1003
}

forced: true means the turn was ended by the safety limit (3 × utterance_end_ms) even though the check judged it unfinished.

`interim_delta` — Delta for an interim (only with `interim_delta_streaming: true`)

{
  "type": "interim_delta",
  "text": " है",
  "full_text": "मेरा नाम कपिल है",
  "operation": "append",
  "append_only": true,
  "utterance_id": 1,
  "language": "hi",
  "timestamp": 1700000000.1
}

operation: "append" → add text to the line. operation: "replace" → the guess changed; replace the line with full_text.

`transcription_tokens` — Word tokens (only with `token_streaming: true`)

Word tokens with stable ids for UIs that animate words. transcription stays the source of truth. event is append for interims, finalize for the final.

{
  "type": "transcription_tokens",
  "event": "append",
  "source": "interim",
  "utterance_id": 7,
  "tokens": [
    { "id": 1, "index": 0, "text": "Hello", "is_final": false },
    { "id": 2, "index": 1, "text": " world", "is_final": false }
  ],
  "language": "en",
  "is_partial": true,
  "timestamp": 1700000000.345
}

`warning` — Non-fatal notice

{
  "type": "warning",
  "code": "audio_format_assumed",
  "message": "Binary frames declare no audio format; assuming 8000 Hz mulaw.",
  "encoding": "mulaw",
  "sample_rate": 8000,
  "timestamp": 1700000000.0
}

Sent once when binary audio arrives without an encoding/sample_rate declared in start (it’s then treated as 8 kHz μ-law).

`session_stopped` — After `stop` is processed

{
  "type": "session_stopped",
  "billing_summary": {
    "total_duration_seconds": 5.2,
    "total_cost": 0.000043,
    "characters_transcribed": 42,
    "client_estimated_seconds": 5.24
  }
}

Every committed final (is_final: true, non-empty text) is charged by its duration. Because a turn’s several finals cover non-overlapping audio, billing each one sums to the real spoken time with no double-charge. Withheld/empty finals (low_confidence_transcript) and no_speech cost nothing. client_estimated_seconds is a diagnostic estimate of what the client sent; never show it as a billed figure.

`error` — Rejected message or failed final

Two kinds, told apart by terminal:

Non-terminal (no terminal field) — the server rejected a client message (invalid start/config, unknown type, audio-format mismatch). The turn is unaffected; fix the client and resend.
Terminal (terminal: true) — a turn’s recognition failed (failure_type such as transcription_timeout, transcription_busy, backend_unavailable, language_identification_uncertain). That turn is closed without text but the session stays open. Handle it like a terminal no_speech: clear the interim, dispatch any finals already collected, keep the session.

{
  "type": "error",
  "terminal": true,
  "outcome_type": "error",
  "failure_type": "transcription_timeout",
  "error": "Transcription timed out.",
  "final_id": "aa653fdf0a5c4372aee5d3af4dac2164:7",
  "endpoint_reason": "vad_silence",
  "timestamp": 1700000000.1
}

`test_response` — Reply to `test` ping

{
  "type": "test_response",
  "message": "pong - qlabs-stt-proxy ready",
  "timestamp": 1700000000000,
  "processing_mode": "proxy"
}

Answered by the 60db proxy itself — it never reaches the STT server, so a reply confirms the proxy hop only, and the echoed timestamp measures round-trip to the proxy.

Complete Example

Audio Requirements

Property	Telephony (μ-law)	Browser (PCM)
Encoding	`mulaw` (8-bit)	`linear` (16-bit)
Sample Rate	8000 Hz	16000, 24000, 44100, 48000 Hz
Chunk Size	480 bytes (60ms)	960-1920 bytes (60-120ms)
Channels	Mono (1 channel)	Mono (1 channel)

Supported Languages

45 languages — 12 Indian (as, bn, gu, hi, kn, ml, mr, ne, or, pa, ta, te) plus 33 others including English, Spanish, French, German, Arabic, Mandarin (zh), Japanese (ja), Korean (ko), Turkish (tr), Vietnamese (vi), and Hebrew (he). Fetch the authoritative catalog (with each language’s quality_tier) from GET /v1/languages (or GET /stt/languages via the proxy). Send a small explicit languages list — each extra language is another chance of a wrong pick. Mixed speech such as Hinglish is handled within a turn (Hindi in Devanagari, English words in Latin); en + hi is the most-tested pair. Arabic dialect tags (ar-eg, …) are rejected — pass ar for best-effort MSA. Common languages:

Code	Language	Code	Language
`en`	English	`hi`	Hindi
`bn`	Bengali	`es`	Spanish
`fr`	French	`de`	German
`gu`	Gujarati	`ta`	Tamil
`te`	Telugu	`kn`	Kannada
`ml`	Malayalam	`mr`	Marathi
`pa`	Punjabi	`ar`	Arabic

Pricing

Rate: $0.00000833 per second
Minimum: $0.01 per session
Billing: Per second of audio processed

TTS WebSocket - Text-to-Speech endpoint
WebSocket Quick Start - Get started guide
WebSocket Playground - Test in browser

​STT WebSocket API

​🚀 Quick Start (Copy & Paste)

​📖 How It Works (5 Simple Steps)

​Endpoint

​Authentication

​Connection Details

​Session Lifecycle

​Text is never rewritten

​Client → Server Messages

​start — Begin session

​Turn-ending profiles

​audio — JSON audio chunk (browser mode)

​Binary frame — raw μ-law audio (telephony mode)

​config — Change settings mid-session

​finalize — End the current turn now

​stop — End session

​test — Ping / latency check

​Server → Client Messages

​connecting — Authentication in progress

​connection_established — Authentication successful

​connected — After start message is processed

​session_started — start accepted, audio is now allowed

​speech_started — VAD detected voice activity

​transcription — Transcription result

​language_changed — After config message changes language

​no_speech — The turn had no words

​turn_complete — End-of-thought confirmed (only with turn_detector: true)

​interim_delta — Delta for an interim (only with interim_delta_streaming: true)

​transcription_tokens — Word tokens (only with token_streaming: true)

​warning — Non-fatal notice

​session_stopped — After stop is processed

​error — Rejected message or failed final

​test_response — Reply to test ping

​Complete Example

​Audio Requirements

​Supported Languages

​Pricing

​Related

STT WebSocket API

🚀 Quick Start (Copy & Paste)

📖 How It Works (5 Simple Steps)

Endpoint

Authentication

Connection Details

Session Lifecycle

Text is never rewritten

Client → Server Messages

`start` — Begin session

Turn-ending profiles

`audio` — JSON audio chunk (browser mode)

Binary frame — raw μ-law audio (telephony mode)

`config` — Change settings mid-session

`finalize` — End the current turn now

`stop` — End session

`test` — Ping / latency check

Server → Client Messages

`connecting` — Authentication in progress

`connection_established` — Authentication successful

`connected` — After `start` message is processed

`session_started` — `start` accepted, audio is now allowed

`speech_started` — VAD detected voice activity

`transcription` — Transcription result

`language_changed` — After `config` message changes language

`no_speech` — The turn had no words

`turn_complete` — End-of-thought confirmed (only with `turn_detector: true`)

`interim_delta` — Delta for an interim (only with `interim_delta_streaming: true`)

`transcription_tokens` — Word tokens (only with `token_streaming: true`)

`warning` — Non-fatal notice

`session_stopped` — After `stop` is processed

`error` — Rejected message or failed final

`test_response` — Reply to `test` ping

Complete Example

Audio Requirements

Supported Languages

Pricing

Related