STT WebSocket API

Real-time Speech-to-Text transcription via WebSocket streaming with support for 39 languages (including code-switched Indic+English) and telephony integration. Powered by 60db STT v01 (a non-hallucinating, multi-backend speech recognition stack).

🚀 Quick Start (Copy & Paste)

const WebSocket = require('ws');

// 1. Your API key
const API_KEY = 'sk_live_your_api_key';

// 2. Connect
const ws = new WebSocket(`wss://api.60db.ai/ws/stt?apiKey=${API_KEY}`);

// 3. Handle messages
ws.on('message', (data) => {
  const msg = JSON.parse(data);

  // Authenticated? Start session!
  // NOTE: on STT the fields are top-level and the frame is tagged by `type`.
  if (msg.type === 'connection_established') {
    console.log('✅ Authenticated');
    ws.send(JSON.stringify({
      type: 'start',
      languages: ['en'],
      config: { encoding: 'mulaw', sample_rate: 8000, continuous_mode: true }
    }));
  }

  // Session ready? Send audio!
  // Gate on `session_started`, not `connected` — `connected` fires before the
  // session exists and arrives twice (proxy frame, then upstream frame).
  if (msg.type === 'session_started') {
    console.log('✅ Ready! Send audio now');

    // Send dummy audio (480 bytes every 60ms)
    let count = 0;
    const interval = setInterval(() => {
      ws.send(Buffer.alloc(480, 0xff));
      if (++count >= 83) {  // 5 seconds
        clearInterval(interval);
        ws.send(JSON.stringify({ type: 'stop' }));
      }
    }, 60);
  }

  // Got text!
  if (msg.type === 'transcription' && msg.is_final) {
    console.log('📝', msg.text);
  }

  // Done!
  if (msg.type === 'session_stopped') {
    console.log('✅ Complete! Cost:', msg.billing_summary.total_cost);
    ws.close();
  }
});

That’s it! You’ll see:

✅ Authenticated
✅ Ready! Send audio now
📝 Hello world (transcribed text)
✅ Complete! Cost: $0.000043

📖 How It Works (5 Simple Steps)

Connect with your API key
Send { type: "start", ... } to begin session
Stream audio data (binary chunks)
Receive text transcriptions in real-time
Stop with { type: "stop" } when done

Endpoint

Authentication

Query parameter authentication: Example:

ws://api.60db.ai/ws/stt?apiKey=sk_live_your_api_key

Connection Details

Property	Value
Protocol	WebSocket (RFC 6455)
Frame types	Binary (telephony) or Text/JSON (browser)
Ping/keepalive	The server answers client-initiated WebSocket pings with a pong; it does not ping first. Send a ping every 20–30 s so intermediaries do not idle the socket out.

Session Lifecycle

Client                          Server
  │                               │
  │──── TCP/TLS connect ─────────►│
  │◄─── {"type":"connecting"} ────│  authenticating...
  │◄─── {"type":"connected"} ─────│  proxy wired to upstream
  │◄─── connection_established ──│  authenticated — safe to send `start`
  │                               │
  │──── {"type":"start", ...} ───►│
  │◄─── {"type":"session_started"}│  safe to send audio
  │                               │
  │──── audio frames / messages ─►│
  │◄─── {"type":"speech_started"} │  VAD detected voice
  │◄─── {"type":"transcription"}  │  is_final=true, speech_final=false (first emit, context only)
  │◄─── {"type":"transcription"}  │  is_final=true, speech_final=true  (canonical answer)
  │                               │
  │──── {"type":"stop"} ─────────►│
  │◄─── {"type":"session_stopped"}│

Two-phase finals (context-gated LLM refinement). When you supply a context object on start, every utterance produces two transcription events sharing a sentence_id:

First emit — is_final: true, speech_final: false — fast dict-corrected text. Use for low-latency UI paint and barge-in.
Canonical — is_final: true, speech_final: true — definitive LLM-refined answer. Always arrives.

When context is omitted, every utterance produces a single transcription event with is_final: true, speech_final: true (no first emit). Simple consumers can gate exclusively on speech_final: true and ignore the rest — that gives them exactly one canonical event per utterance regardless of whether refinement is on.

Client → Server Messages

`start` — Begin session

Sent once after connection is established. Must be sent before any audio.

{
  "type": "start",
  "languages": ["en", "hi"],
  "context": {
    "general": [
      { "key": "domain",  "value": "Healthcare" },
      { "key": "doctor",  "value": "Dr. Martha Smith" }
    ],
    "text":  "Routine diabetes follow-up consultation.",
    "terms": ["Celebrex", "Zyrtec", "Metformin", "HbA1c"]
  },
  "config": {
    "encoding": "mulaw",
    "sample_rate": 8000,
    "utterance_end_ms": 1000,
    "continuous_mode": true,
    "interim_results_frequency": 300,
    "audio_enhancement": "adaptive",
    "diarize": false,
    "remove_fillers": false
  }
}

Parameters:

`audio` — JSON audio chunk (browser mode)

{
  "type": "audio",
  "audio": "<base64-encoded Int16 PCM or μ-law bytes>",
  "encoding": "linear",
  "sample_rate": 48000,
  "timestamp": 1700000000000
}

Fields:

Binary frame — raw μ-law audio (telephony mode)

Send a raw WebSocket binary frame with μ-law bytes, no JSON wrapper. The server auto-detects this as telephony mode on the first binary frame.

Recommended chunk size: 480 bytes = 60ms at 8kHz
Twilio default: 160 bytes = 20ms — batch 3 chunks into 60ms before sending

`config` — Change language mid-session

{
  "type": "config",
  "languages": ["hi"],
  "continuous_mode": true
}

Both languages and continuous_mode are optional; include only fields you want to change. Send "languages": null to revert to auto-detect.

`stop` — End session

{
  "type": "stop"
}

Server processes any remaining audio buffer, sends session_stopped, then closes.

`test` — Ping / latency check

{
  "type": "test",
  "message": "ping",
  "timestamp": 1700000000000
}

Server echoes test_response with the same timestamp for round-trip measurement.

Server → Client Messages

`connecting` — Authentication in progress

{
  "type": "connecting",
  "message": "Authenticating...",
  "timestamp": 1775465918269
}

`connection_established` — Authentication successful

{
  "type": "connection_established",
  "service": "stt",
  "user_id": 43,
  "credit_balance": 9.97,
  "workspace": "default"
}

On STT these fields are top-level and the frame is tagged by type. Check msg.type === "connection_established" — msg.connection_established is the TTS frame’s shape, and a client written against it will never start its STT session.

Fields:

string

Service name: "stt"

integer

Your user ID

number

Available credits

string

Workspace name

`connected` — After `start` message is processed

{
  "type": "connected",
  "server_info": {
    "server_type": "60db STT",
    "device": "cuda",
    "model": "60db-stt-v01",
    "processing_mode": "sentence_based_modular",
    "supported_languages": { "en": "English", "hi": "Hindi" },
    "total_languages": 40,
    "features": {
      "sentence_based_processing": true,
      "real_time_streaming": true,
      "telephony_support": true,
      "unicode_support": true,
      "mixed_language_support": true
    }
  }
}

Expect two connected frames: the proxy’s own handshake (session_id plus a small server_info), then the upstream capability frame shown above. Neither means “audio is accepted” — wait for session_started. Keep any connected handler idempotent so audio capture does not start twice.

`session_started` — `start` accepted, audio is now allowed

{
  "type": "session_started",
  "session_id": "sess_8c3d1a9f4b7e2c51",
  "language": "Multi-language: EN, HI",
  "languages": ["en", "hi"],
  "model": "60db-stt-v01",
  "processing_mode": "sentence_based_continuous",
  "continuous_mode": true,
  "interim_frequency": 300,
  "diarize": false,
  "llm_refinement": true
}

Send audio only after this event — earlier frames are answered with unknown message type: audio. llm_refinement: true confirms your context opened the refinement gate, so every utterance will arrive as a two-phase pair; false means one canonical event per utterance. languages is the resolved candidate list after normalization.

`speech_started` — VAD detected voice activity

{
  "type": "speech_started",
  "timestamp": 1700000000.123
}

Use this for barge-in: interrupt TTS playback when this arrives. Fired after 2 consecutive VAD-positive chunks (~64ms of confirmed speech).

`transcription` — Transcription result

All results (interim and final) share the same transcription type — differentiate with flags. Final result (is_final=true, speech_final=true):

{
  "type": "transcription",
  "text": "Hello, how are you?",
  "confidence": 0.87,
  "language": "en",
  "language_name": "EN",
  "is_final": true,
  "speech_final": true,
  "is_partial": false,
  "sentence_id": 3,
  "processing_mode": "sentence_complete",
  "duration": 1.82,
  "latency": 0.43,
  "timestamp": 1700000000.456,
  "words": [
    { "word": "Hello", "start": 0.0, "end": 0.32, "confidence": 0.94 },
    { "word": "how",   "start": 0.35, "end": 0.52, "confidence": 0.92 }
  ],
  "utterance_end_ms": 1820
}

Empty speech_final signal (text="", is_final=true, speech_final=true): Sent when audio was detected but transcription was rejected (silence, hallucination, low confidence, wrong language). Client should reset its state on this message and not treat it as an error.

{
  "type": "transcription",
  "text": "",
  "confidence": 0.0,
  "is_final": true,
  "speech_final": true,
  "processing_mode": "speech_end_no_result",
  "timestamp": 1700000000.789
}

Interim result (is_final=false, speech_final=false) — only sent when interim_results_frequency is set:

{
  "type": "transcription",
  "text": "Hello how",
  "confidence": 0.72,
  "language": "en",
  "is_final": false,
  "speech_final": false,
  "is_partial": true
}

Use interims only for barge-in word-count checks. Never send interim text to the LLM — a final with is_final=true, speech_final=true will follow. Response Fields:

string

Transcribed text. Empty string = speech-end-no-result signal.

number

0.0–1.0. Telephony typically 0.35–0.75; browser 0.55–0.95.

string

Detected language code e.g. "en".

string

Uppercase language code e.g. "EN".

boolean

true = end of speech reached. May still be followed by a canonical upgrade if LLM refinement is active.

boolean

true = canonical answer, will not be revised. When LLM refinement is on, one is_final: true, speech_final: false event is followed by one is_final: true, speech_final: true. When refinement is off, every final is speech_final: true. See Canonical-answer semantics.

boolean

true for interim results only.

integer

Monotonically increasing counter per session.

number

Duration (seconds) of the audio segment transcribed.

number

Seconds from processing start to result ready (excludes queue time).

array

Word-level timestamps [{word, start, end, confidence}]. Note: the field is confidence, not probability. Present on finals; empty on interims.

integer

Timestamp (ms) of last word in the utterance.

string

Marker for utterances the consumer should skip. Omitted on ordinary finals. None of these are billed.

Value	Meaning
`speech_end_no_result`	Speech ended but the recognizer produced no text.
`speech_end_too_short`	Utterance below the minimum duration to recognize.
`hallucination_rejected`	Word-rate guard rejected the output as a likely hallucination.
`low_snr_dropped`	Audio dropped before language ID / ASR — signal-to-noise below the floor.

boolean

On canonical emits where refinement ran: true = text is the LLM-refined version, false = the LLM was skipped or failed and the fast first-emit text was promoted unchanged. llm_reason carries the why, llm_latency_ms the round-trip time. The canonical always arrives either way.

boolean

Added by the 60db proxy when the upstream rejected the utterance as a suspected hallucination but interim text was available — the proxy sends that text instead of an empty final, with tentative_reason: "hallucination_rejected". Safe to route on; flag it for review.

Canonical-answer semantics: `speech_final`

is_final and speech_final are NOT identical when LLM refinement is active — they split into two distinct meanings:

`is_final`	`speech_final`	Meaning
`false`	`false`	Interim partial — text may still change as more audio arrives.
`true`	`false`	End-of-speech reached, dict-corrected text. The LLM is still processing. A follow-up event with `speech_final: true` will arrive shortly with the canonical text. Only emitted when refinement is active for this utterance.
`true`	`true`	Canonical answer. Definitive, will not be revised. LLM-refined text when context was supplied, otherwise the original ASR text.

The same sentence_id is echoed across both phases so clients can reconcile. Canonical event example (after LLM refinement):

{
  "type": "transcription",
  "sentence_id": 3,
  "text": "डॉक्टर साहब, मेरा sugar level बहुत high है। Metformin की dose बढ़ाओ।",
  "confidence": 0.87,
  "language": "hi",
  "language_name": "HI",
  "is_final": true,
  "speech_final": true,
  "is_partial": false,
  "duration": 1.82,
  "words": [
    { "word": "डॉक्टर", "start": 0.0, "end": 0.32, "confidence": 0.94 }
  ],
  "speakers": null,
  "timestamp": 1700000000.789
}

Guarantees:

Exactly one canonical event per utterance. When refinement is on, you get two transcription events per utterance (first emit + canonical). When refinement is off, you get one (speech_final: true). Never zero, never three.
Same sentence_id across both phases. Reconcile on that key.
The canonical always arrives. Consumers waiting on speech_final: true never hang.
sentence_id ordering is preserved per session, but canonicals are NOT guaranteed to arrive in sentence_id order when LLM is on — two utterances finalizing close in time may complete refinement out of order. Key on sentence_id, not arrival order.
words[] corresponds to the original ASR output on both phases — the LLM does not realign tokens. Use words[] for word-level timing, text for display.

Recommended client patterns: Simplest — don’t care about the first-emit optimization:

function onMessage(msg) {
  if (msg.type === 'transcription' && msg.speech_final) {
    // Canonical — render and forget
    render(msg);
  }
  // Ignore is_final && !speech_final  (intermediate, will be replaced)
  // Ignore is_partial                 (interim, handle separately if needed)
}

With fast first-emit (UX-aware):

const livePainted = new Map();   // sentence_id → line slot

function onMessage(msg) {
  if (msg.type !== 'transcription') return;
  const sid = msg.sentence_id;

  if (msg.is_partial) { renderPartial(msg); return; }

  let entry = livePainted.get(sid);
  if (!entry) {
    entry = createLine();
    livePainted.set(sid, entry);
  }
  entry.text    = msg.text;
  entry.pending = msg.is_final && !msg.speech_final;   // dim while LLM runs
  render(entry);

  if (msg.speech_final) {
    finalize(entry);
    livePainted.delete(sid);
  }
}

For voicebot NLU routing: feed the first-emit text (speech_final: false) to NLU immediately for fast intent dispatch — don’t wait for canonical. If your NLU benefits from proper-noun accuracy (name-spelling slots, drug-name lookup), run a second-pass call on the canonical (speech_final: true) text and reconcile on sentence_id.

Legacy refined event. Earlier builds emitted a separate refined event ~400 ms after the final instead of a second transcription. The 60db /ws/stt proxy transparently handles both shapes — if you’re still seeing refined events in the wire trace, upstream workers haven’t been restarted onto the two-phase build yet. New client code should target the two-phase flow only; refined is accepted but deprecated.

`language_changed` — After `config` message changes language

{
  "type": "language_changed",
  "language": "Multi-language: HI",
  "language_code": ["hi"]
}

`mode_changed` — After `config` message changes `continuous_mode`

{
  "type": "mode_changed",
  "continuous_mode": true,
  "mode_name": "continuous",
  "silence_threshold": 0.5
}

`session_stopped` — After `stop` is processed

{
  "type": "session_stopped",
  "billing_summary": {
    "total_duration_seconds": 5.2,
    "total_cost": 0.000043,
    "characters_transcribed": 42,
    "client_estimated_seconds": 5.24
  }
}

Only canonical finals (is_final: true and speech_final: true) with a duration are charged — first emits are previews, so turning on LLM refinement does not double your bill, and the four skip modes above cost nothing. client_estimated_seconds is a diagnostic estimate of what the client sent; never show it as a billed figure.

`error` — Processing error

{
  "type": "error",
  "error": "Audio processing error: ...",
  "timestamp": 1700000000.0
}

`test_response` — Reply to `test` ping

{
  "type": "test_response",
  "message": "pong - qlabs-stt-proxy ready",
  "timestamp": 1700000000000,
  "processing_mode": "proxy"
}

Answered by the 60db proxy itself — it never reaches the STT server, so a reply confirms the proxy hop only, and the echoed timestamp measures round-trip to the proxy.

Complete Example

Audio Requirements

Property	Telephony (μ-law)	Browser (PCM)
Encoding	`mulaw` (8-bit)	`linear` (16-bit)
Sample Rate	8000 Hz	16000, 24000, 44100, 48000 Hz
Chunk Size	480 bytes (60ms)	960-1920 bytes (60-120ms)
Channels	Mono (1 channel)	Mono (1 channel)

Supported Languages

39 languages total, backed by Parakeet-TDT (25 European), Vaani-FastConformer (13 Indic + Hinglish), and FC-Arabic (MSA). Fetch the full catalog from GET /stt/languages. Not supported (explicit rejection, no silent aliasing): ur, ja, ko, zh, th, vi, id, tl, sw, tr, fa, he. Arabic dialect tags (ar-eg, ar-lv, …) return dialect_not_supported — pass ar for best-effort MSA. Common languages:

Code	Language	Code	Language
`en`	English	`hi`	Hindi
`bn`	Bengali	`es`	Spanish
`fr`	French	`de`	German
`gu`	Gujarati	`ta`	Tamil
`te`	Telugu	`kn`	Kannada
`ml`	Malayalam	`mr`	Marathi
`pa`	Punjabi	`ar`	Arabic

Pricing

Rate: $0.00000833 per second
Minimum: $0.01 per session
Billing: Per second of audio processed

TTS WebSocket - Text-to-Speech endpoint
WebSocket Quick Start - Get started guide
WebSocket Playground - Test in browser

​STT WebSocket API

​🚀 Quick Start (Copy & Paste)

​📖 How It Works (5 Simple Steps)

​Endpoint

​Authentication

​Connection Details

​Session Lifecycle

​Client → Server Messages

​start — Begin session

​audio — JSON audio chunk (browser mode)

​Binary frame — raw μ-law audio (telephony mode)

​config — Change language mid-session

​stop — End session

​test — Ping / latency check

​Server → Client Messages

​connecting — Authentication in progress

​connection_established — Authentication successful

​connected — After start message is processed

​session_started — start accepted, audio is now allowed

​speech_started — VAD detected voice activity

​transcription — Transcription result

​Canonical-answer semantics: speech_final

​language_changed — After config message changes language

​mode_changed — After config message changes continuous_mode

​session_stopped — After stop is processed

​error — Processing error

​test_response — Reply to test ping

​Complete Example

​Audio Requirements

​Supported Languages

​Pricing

​Related

STT WebSocket API

🚀 Quick Start (Copy & Paste)

📖 How It Works (5 Simple Steps)

Endpoint

Authentication

Connection Details

Session Lifecycle

Client → Server Messages

`start` — Begin session

`audio` — JSON audio chunk (browser mode)

Binary frame — raw μ-law audio (telephony mode)

`config` — Change language mid-session

`stop` — End session

`test` — Ping / latency check

Server → Client Messages

`connecting` — Authentication in progress

`connection_established` — Authentication successful

`connected` — After `start` message is processed

`session_started` — `start` accepted, audio is now allowed

`speech_started` — VAD detected voice activity

`transcription` — Transcription result

Canonical-answer semantics: `speech_final`

`language_changed` — After `config` message changes language

`mode_changed` — After `config` message changes `continuous_mode`

`session_stopped` — After `stop` is processed

`error` — Processing error

`test_response` — Reply to `test` ping

Complete Example

Audio Requirements

Supported Languages

Pricing

Related