Voice limits by plan, more professional voices, marketplace KYC
Surfaces: Dashboard, REST, MCP, Docs
Changed
- Instant voice limit — Free, Starter and Flow workspaces can keep 1 instant (cloned) voice; Creator and Pro keep 10. The limit counts live voices, so deleting one frees a slot. Over the limit,
POST /voicesreturns403 VOICE_LIMIT_REACHED(was429) - Professional voices — available on Creator (2 voices) and Pro (5 voices); other plans get
403 PVC_NOT_AVAILABLE_ON_PLAN, and a full plan gets403 PVC_LIMIT_REACHED. You can now create more than one professional voice, and the page lists them all - Marketplace listing — needs a Creator or Pro plan, a submitted KYC tax form and bank details. Submitting works while they are under review; the voice goes live only after KYC is verified and bank details are active. Otherwise submitting returns
403 MARKETPLACE_REQUIREMENTS_NOT_MET - Downgrades — existing voices are kept; creating and listing new ones is blocked while over the limit
Added
- Get Voice Limits (
GET /voices/quota), List Professional Voices (GET /voices/professional/jobs) and Get Marketplace Eligibility (GET /voices/professional/marketplace-eligibility) - MCP — read-only tools
sixtydb_get_voice_quota,sixtydb_list_professional_voicesandsixtydb_get_marketplace_eligibility
WebSocket STT and TTS: auto-detect fix and refreshed protocol
Surfaces: WebSocket, Proxy, Docs
Fixed
- STT auto-detect — omitting
languages, or sendingnull,[],"auto"or["auto"], now expands to the full language list the STT server advertises. Before, the session started but real speech failed withNo compatible route exists for this candidate set.session_started.language_modeiscandidate_autofor two or more languages andselectedfor one - TTS upstream — the
/ws/ttsproxy connects to the TTS server through its hostname instead of a fixed IP address
Changed
- TTS
context_creatednow includessession_idandbinary_audio(alwaysfalse; audio is base64 inaudio_chunk) - TTS
warningandcontext_abortedare documented.warningwith codesynthesis_interruptedmeans new text arrived mid-synthesis; sendflush_contextto speak it - Realtime TTS runs without audio enhancement — the proxy sets
audio_enhancetofalseon everycreate_context, whatever the client sends
POST /tts-synthesize returns the documented response
Surfaces: REST, Docs
Fixed
- Documented response fields —
POST /tts-synthesize(withoutstream) now returnssuccess,message,audio_base64,sample_rate,duration_seconds,encodingandoutput_format, as documented. Previously it returned the engine’s rawaudioContent(headerless PCM) and an internalconditioningobject - Playable audio — for
LINEAR16,audio_base64is a complete WAV file (16-bit mono PCM with header), so it can be saved or played directly - Backward compatible —
audioContentis still returned unchanged for existing integrations. Streaming ("stream": true) is unchanged: NDJSON lines withresult.audioContent - No charge without audio — a response that carries no audio now returns an error and is not billed. See Text to Speech
Studio: audio projects, smart regeneration, emotion tags, MP3 download
Surfaces: Dashboard, REST
Changed
- Audio projects — create a project, write your text and generate narration. Import and URL import were removed
- Smart regeneration — the first generation converts every line; after that only new or edited lines are generated (and charged). Lines that need generating are marked, and “Generate all” skips finished blocks and respects your plan’s concurrency limit
- Download — the whole project downloads as one MP3 (
GET /studio/projects/:projectId/download). Missing lines are generated first; the endpoint returns409 AUDIO_INCOMPLETEwith the missing lines otherwise - Emotion tags — insert
[happy],[laugh],[pause 1s]and the rest from the Emotion picker, or use Enhance / Remove tags, the same as Text to Speech - Billing — Studio generation is always charged to the workspace you are signed in to, at the TTS rate
Professional voice clones: upload only, marketplace listing optional
Surfaces: Dashboard, REST, Docs
Changed
- Upload only — the professional clone wizard is now Intro → Upload → Verify → Review. The teleprompter recording method was removed
- Marketplace listing is optional — choose it on the Review step (off by default). Unlisted voices are ready to use privately right after training, with no review.
POST /voices/professional/jobs/:id/finalizeacceptslist_in_marketplace(boolean); jobs returnlist_in_marketplaceand the new statusprivate - Submit later —
POST /voices/professional/jobs/:id/submit-to-marketplacesends a private voice to marketplace review (private→awaiting_approval) without retraining. See Professional Voice Clones
Plan-based concurrency limits for TTS and STT
Surfaces: REST, WebSocket, Dashboard, Docs
Changed
- Concurrency limits now depend on your plan and apply per workspace (REST + WebSocket combined, shared by members and API keys). TTS: Free 1, Starter 5, Creator 10, Pro 20. STT: Free 2, Starter 10, Creator 20, Pro 50. Previously a fixed per-user cap (TTS 5, STT 8)
- 429 / 1008 errors —
TTS_CONCURRENCY_LIMIT/STT_CONCURRENCY_LIMITnow name your plan and its limit;detailsaddsplan - API —
GET /plansaddsconcurrency { tts, stt }per plan;GET /billing/current-planaddsconcurrencyfor the workspace. See Concurrency limits
Annual billing — Starter, Creator and Pro can be billed yearly at 2 months free
Surfaces: REST, Docs
Added
Annual billing
- Yearly prices — Starter $60/yr ($5/mo), Creator $220/yr ($18.33/mo), Pro $990/yr ($82.50/mo). Yearly is monthly × 10, so you get 2 months free. Credits stay a monthly allotment (120K / 400K / 2M per month) and reset every month on your subscription start day
- Subscribe on a cycle —
POST /billing/subscribetakesbilling_cycle: "monthly" | "yearly"(defaultmonthly) and returnsaction(subscribe,upgradeorswitch_to_yearly),billing_cycleandamount. See Subscribe to a Plan - Plan prices —
GET /plansaddspricing.yearly,yearly_monthly_equivalentandyearly_savings_pct. See List Plans - Current plan —
GET /billing/current-planaddsdata.plan.billing_cycle,next_credit_reset_atandnext_payment_at. For yearly plansnext_payment_usdis the yearly price. See Current Plan - Plan changes — monthly to yearly and upgrades take effect immediately, and unused plan credits carry over. Yearly to a higher yearly plan gets a prorated discount at checkout. Downgrades and cancellations take effect at the end of the billing cycle
- Errors —
403 YEARLY_TO_MONTHLY_NOT_ALLOWED(cancel the yearly plan, then choose monthly),400 DOWNGRADE_KEEPS_CYCLE,400 CYCLE_NOT_AVAILABLE,400 INVALID_BILLING_CYCLE
Updated
- Plans & credits guide — new Annual billing section with the price table, plan-change rules and refund policy. Top-ups on yearly plans use the monthly plan rate
Pay-as-you-go credit top-ups for Creator & Pro — buy credits at your plan rate
Surfaces: REST, Dashboard, Docs
Added
Pay-as-you-go top-ups (Creator and Pro)
- Buy top-ups — Creator and Pro workspaces can buy $10–$1,000 of credits (presets $10, $25, $50, $100) with
POST /billing/add-funds(purpose: "credits"). Top-up credits never expire and stay usable after a downgrade - Plan rate — credits =
floor(usd × plan monthly credits / plan price). Creator buys about 18,181.8 credits per $1 ($10 → 181,818), Pro about 20,202 per $1 ($10 → 202,020). The credits are quoted at checkout and fixed, even if you change plan before the payment completes. Coupons lower the price, not the credits - Quote and balance —
GET /billing/topupreturns eligibility, your plan rate, preset quotes and the plan / top-up credit balance.GET /billing/topup/quote?amount=previews a custom amount - Dashboard — Subscription → Platform has a new Top-up credits card with a Pay-as-you-go button. When credits run out, the You’re out of credits dialog offers Pay-as-you-go (Creator, Pro) or Upgrade to Creator (Free, Starter) instead of an error toast
- Ledger —
GET /billing/usage-logslists top-ups asTOP_UProws, andSA_TOP_UPfor offline payments recorded by 60db support.AUTO_TOP_UPis reserved for automatic top-up - Errors —
403 TOPUP_NOT_AVAILABLE_ON_PLANon Free and Starter, and400 TOPUP_AMOUNT_OUT_OF_RANGEoutside $10–$1,000
Updated
- Out-of-credits
402— the message now points to a top-up (Creator, Pro) or an upgrade (Free, Starter), and the body includesdetails.topup_availableanddetails.auto_topup_enabled - Plans & credits guide — new Plans, Credits & Top-ups page. Usage draws plan credits first, then top-up credits
- Pricing pages — Memory and Judge pricing no longer describe a pay-as-you-go wallet only
Coming soon
- Automatic top-up — save a card once and credits reload automatically when your balance falls below a threshold you choose, up to a monthly limit you set. Not available yet
Emotion tags and Enhance for Text-to-Speech
Surfaces: REST, Docs
Added
- Emotion tags — put
[happy],[calm],[apologetic]and 17 other emotion tags inline intextto change how the following sentences sound, blend two with[sad, calm], and add sound tags such as[laugh],[sigh],[gasp]and[pause 500ms]. Unknown tags are dropped, never spoken. Works onPOST /tts-synthesizeand in the dashboard. See Emotion Tags - Enhance —
POST /tts/enhanceadds emotion and sound tags to your text (up to 5,000 characters) and changes nothing else: your words, spacing and language come back exactly as sent. No credits are charged. See Enhance Text - Dashboard — the Text to Speech page highlights tags as colored chips and adds Add emotion, Enhance, Undo enhance and Remove tags
- Audio enhancement — generated audio is now polished by default. Send
audio_enhance: falseonPOST /tts-synthesizeto skip it (the olderenhancestill works). It is never applied to the real-time WebSocket TTS used by voice agents, so live calls keep their low latency. The dashboard has a new Audio enhancement switch. See Text to Speech
Click-to-Call — two-leg agent-then-customer calls, signed outcome webhooks, per-call recordings
Surfaces: REST, Docs
Added
Click-to-Call (Production)
- Place a call —
POST /dialer/click2callrings the agent on your own DID, then the customer, then bridges them. No SIP client, no WebRTC, no microphone. Returns 201 when a call was dialled and 200 when thereference_idwas already bound to a live call and nothing new was dialled (data.placed/data.replayed) - Read a call —
GET /dialer/click2call/{id}bycall_idor your ownreference_id;?refresh=1polls the platform for the authoritative state and repairs a call whose terminal webhook never arrived - History —
GET /dialer/click2callwithlimit/offset,mine=0for the whole workspace, andapi_key_idfor a per-integration statement - Signed webhooks — pass
callback_url+callback_secret(16–256 chars, yours to choose) per call; every verified event is relayed to you withx-qcall-signature: v1=<hmac-sha256>overtimestamp + "." + raw_body. Up to 6 attempts, ≥2 min apart, abandoned after 24 h - Delivery inspection —
GET /dialer/click2call/{id}/eventsshows every event, whether it was applied, and the status your own endpoint returned for it - Recordings —
record: trueon placement, thenPOST /dialer/click2call/{id}/recordingfor a short-lived playback URL (audio uploads after the call ends — checkrecording_available) - Readiness —
GET /dialer/click2call/configreports whether the server can dial, whether it can verify results, and the numeric limits it enforces - Idempotency —
reference_idis optional and minted per call when omitted; it is permanent per trunk, and a failed call still consumes it. Refusals that placed no call (429,503) are safe to retry with the same one - Billing — ₹0.30/min ($0.003/min) on bridged talk time only, prorated per second. Unanswered calls are free.
cost_usdandbilling_statusare on the call itself - Limits — 20 calls/min per caller (
C2C_RATE_LIMITED), ring timeouts 5–60 s (defaults 30 agent / 45 customer),max_duration30–7200 s (default 3600),metadata≤ 2 KiB - See the Click-to-Call guide and API reference
Updated
- Dialer feature guide — now distinguishes the two ways to place a call:
POST /dialer/calls(your SIP client dials) vsPOST /dialer/click2call(60db dials both legs) - Introduction — Click-to-Call feature card
Music (beta) & Dialer — song generation, voice library, call management, recordings, webhooks
Surfaces: REST, SDK, MCP, Docs
Added
Music API (Beta, Free)
- Song generation —
POST /songs(simple mode: one prompt, we write lyrics; advanced: full control) - Song polling —
GET /songs/{id}(async generation, ~1.5–2.5 min) - Voice management — catalog voices or save custom voices from audio samples
- Download —
GET /songs/{id}/download(MP3) - SDK methods —
client.music.create(),list(),get(),update(),delete(),download(),waitForCompletion(),listVoices(),createVoice() - MCP tools —
sixtydb_music_*tools for Claude Desktop - See Music feature guide and API reference
Dialer API (Production)
- Number management — rent Indian numbers (E.164,
+919876543210), set caller ID, release - Call reservation —
POST /dialer/calls(SIP client places the actual call; API reserves) - Call history —
GET /dialer/logs(CDR with billsec, disposition) - Recordings & transcripts — auto-record, fetch playback URL (~5 min), get transcript
- Webhooks — subscribe to call/recording events (create, test, rotate signing key, delivery history)
- Usage/billing — track charges per-minute and monthly subscriptions
- Setup prerequisites — trunk + KYC done in dashboard (not API; gates are 409/403 on missing setup)
- Pricing — ₹600/month ($6) per number; ₹0.30/min ($0.003) per call (inbound/outbound, prorated/sec)
- SDK methods —
client.dialer.status(),numbers.search()/buy()/list()/release()/setCallerId(),calls.list()/get()/reserve(),recordings.list()/playbackUrl()/transcript()/delete()/setEnabled(),billing.subscriptions()/usage(),webhooks.list()/create()/update()/delete()/rotateSecret()/test()/deliveries() - MCP tools —
sixtydb_dialer_*tools;confirm: truerequired for operations that charge money or release resources - See Dialer feature guide and API reference
Updated
- SDKs — Music and Dialer sections added to JavaScript and Python docs
- Introduction — feature cards for Music and Dialer
- MCP Server — new music and dialer tool groups
STT WebSocket — single-phase finals, turn-shaping config (early_final / turn_detector), audio_enhancement on|off
Surfaces: WebSocket, Proxy, SDK, DocsThe streaming STT protocol was simplified to a single-phase model. The server no longer rewrites text after recognition (no LLM refinement), so the two-phase
speech_final flow and its accompanying fields are gone. This is the behaviour every integration (proxy, web app, teleprompter) now targets.Changed
speech_finalnow always equalsis_final. Every committed result is a singletranscriptionwithis_final: true(andspeech_final: true); interims carry bothfalse. There is no second “canonical” emit. Simple consumers append each final and clear the interim.config.audio_enhancementvalues are now"on"|"off"(defaultoff). The old"light"/"adaptive"(and booleantrue) are accepted for compatibility and map to"on". Use"on"for noisy / telephony audio.- Billing now charges every committed final by its
duration. Withheld finals (emptytext) andno_speechare never charged. A single turn can produce several finals (short-pause splits,segment_finalon long monologues) — their durations are non-overlapping, so there is no double-charge.
Added
- Turn-shaping config:
early_final(bool, defaulttrue; sendfalseto keep whole sentences in one final) andturn_detector(bool, defaultfalse; sendtrueto wait until the caller finished the thought — emitsturn_complete). See the three latency/accuracy profiles in the STT reference. - More config:
interim_min_voiced_ms(default 600),interim_delta_streaming,token_streaming,max_utterance_sec(3–30, default 30). finalizeclient message ({ "type": "finalize" }) — closes the current turn now and returns its transcription (orno_speech); the session continues.- New server events:
no_speech(turn had no words;terminaltrue/false),turn_complete(only withturn_detector: true),interim_delta(withinterim_delta_streaming),transcription_tokens(withtoken_streaming), andwarning(e.g.audio_format_assumed). - New transcription fields:
segment_final(length-cap split — check beforespeech_final),final_id(unique per committed result — de-duplicate on it), andendpoint_reason(vad_silence|client_finalize|client_stop|max_utterance|language_change|response_timeout|interim). - Language coverage is now 45 (adds
ja,ko,zh,vi,tr,he, and more).GET /v1/languagesis authoritative.
Removed
config.continuous_mode— streaming is always continuous; the field is ignored (a session no longer stops after the first utterance).config.remove_fillersandconfig.no_speech_threshold— no longer used (filler stripping was an LLM-refinement feature, which is gone).- The two-phase /
refinedprotocol and its fields: the separaterefinedevent,is_final: true, speech_final: falsefirst emits,llm_applied/llm_reason/llm_latency_ms,tentative/tentative_reason, andsession_started.llm_refinement. - The
mode_changedevent (it acknowledgedcontinuous_mode). - The old skip-mode markers (
speech_end_no_result,speech_end_too_short,hallucination_rejected) — withheld low-confidence finals now arrive astext: ""withprocessing_mode: "low_confidence_transcript".
Migration
- Drop
continuous_mode/remove_fillers/no_speech_thresholdfromstart.config; addearly_final: falseandturn_detector: truefor whole-sentence finals (profile A). - Change
audio_enhancement: "adaptive"/"light"→"on". - Gate on any
is_final: true(equivalentlyspeech_final: true); stop waiting for a second canonical emit and removerefined-event handling. words[]timings useprobability, notconfidence.
STT WebSocket — utterance_end_ms floor raised to 1000 ms, handshake shape corrected
Surfaces: WebSocket, Proxy, Docs
Changed
config.utterance_end_ms minimum and default are now 1000 ms (were 300 ms minimum / 500 ms default).- The upstream STT server rejects a
startcarrying a lower value instead of clamping it, which killed the session outright. - The 60db
/ws/sttproxy now clamps any value below 1000 ms up to 1000 ms before forwarding — onstartand on mid-sessionconfig— so existing clients sending500keep working and simply get 1000 ms behaviour. - Need faster turn-taking? Drive barge-in from
speech_startedand interim results rather than lowering this value.
Fixed (docs)
STT handshake frames were documented with the TTS shape./ws/stt sends {"type":"connecting"} and {"type":"connection_established","service":"stt","user_id":…} — tagged by type, fields top-level. The examples previously branched on msg.connection_established, the nested /ws/tts shape, which never matches on STT. Examples now also wait for session_started before streaming audio, since connected arrives before the session exists and is emitted twice (proxy frame, then upstream capability frame).Documented
session_started.llm_refinement— whether yourcontextactually opened the refinement gate.llm_applied,llm_reason,llm_latency_mson canonical transcription emits.tentative/tentative_reason— proxy-substituted text when upstream rejects an utterance as a hallucination.- The proxy’s word-preservation guardrail: a refinement that keeps under 40% of the first emit’s words is rolled back to the original text.
connection_closed, theAUTH_FAILEDerror code, and the nested mid-sessionINSUFFICIENT_CREDITSerror shape.- Billing: only canonical finals are charged;
speech_end_no_result,speech_end_too_short,hallucination_rejectedandlow_snr_droppednever are.
TTS REST — Inworld payload alignment (POST /tts-synthesize)
Changed
POST /tts-synthesize now accepts the full Inworld schema.audio_config: { audio_encoding, sample_rate_hertz }— nested block matching the upstream schema.timestamp_type(default"WORD") — enables per-word timestamps in the final NDJSON chunk.- Legacy flat keys (
sample_rate,audio_encoding) still work; new clients should useaudio_config.
sample_rate_hertz moved from 48000 → 24000 to match the upstream default and reduce bandwidth. Callers that need 48 kHz must send it explicitly.See Text to Speech reference.STT WebSocket — remove_fillers config key
Added
config.remove_fillers on WebSocket start (STT).- New boolean field inside
start.config. Defaultfalse. - When
trueandcontextis set (LLM refinement gate is open), the canonicaltranscriptionemit (speech_final: true) drops filler words —um,uh,like,you know, … - No effect when
contextis omitted (refinement stays off). - The 60db
/ws/sttproxy coerces non-boolean values tofalserather than rejecting the session.
Onboarding, email delivery (Cloudflare), and platform fixes
Surfaces: REST, Email
Added
Onboarding fields (POST /user/onboarding).- Now accepts
default_voice_id(string),marketing_consent(boolean),consent_timestamp, andconsent_source— all optional and backward compatible. - Marketing consent is server-stamped:
consent_timestamp/consent_sourceare set on the server; any client-sent values are ignored. - Consent is persisted best-effort after the onboarding commit, so a consent-write failure can never fail the onboarding save.
/unsubscribe).- Public, HMAC-token-verified endpoint for lifecycle email. Sends a
List-Unsubscribe(one-click) header for bulk-sender compliance.
Changed
Email delivery now uses Cloudflare Email Service.- Transactional and lifecycle mail send via Cloudflare (
smtp.mx.cloudflare.net, implicit TLS on 465) authenticated with a Cloudflare API token; DKIM/ARC are applied automatically. ConfigureCLOUDFLARE_API_TOKEN(used as the SMTP password) andSMTP_EMAIL(a sender on a domain onboarded for Email Sending). - Signup verification email is now sent after the account is committed, so a transient email/transport failure can no longer roll back account creation.
Fixed
GET /workspaces duplicate rows.- A workspace with more than one active membership row no longer appears multiple times in the response; the list is de-duplicated per workspace.
- Submitting onboarding no longer returns
500when the marketing-consent columns are absent from the database (consent persistence is best-effort — apply the consent migration to enable it).
- SMTP / Cloudflare credentials are scrubbed from any thrown or logged email error.
Security
- Cerbos authorization checks added across additional routes (STT, TTS, SLM, Studio, webhooks, sixtydb).
Migrations
- New environments must apply the consent columns on
users(marketing_consent,consent_timestamp,consent_source); plus the events / lifecycle-email tables if the lifecycle email program is enabled.
STT editor: transcribe API + audio playback
Coupons for wallet top-ups
Surfaces: REST, Billing
Added
- Coupon system for wallet top-ups — percentage / fixed discounts and deposit multipliers (e.g. 2× / 3×). Redemptions are recorded for audit (
coupon_redemptions, referencing the top-up’spayment_transaction_id).
STT/TTS reliability + new features
Added
Longer request timeout (RESTPOST /stt).- Server-side request timeout raised from 120 s → 600 s; clients should match. The 25 MB / ~1-hour file-size cap remains.
- New
keywordsform field onPOST /stt. CSV with optional per-term weights, e.g.Acme:5,XYZ Pharma:8. - The same boost runs on the WebSocket path via the existing
context.termsfield — no client change required to benefit; existingtermspayloads now produceboosted/originalmarkers automatically. - Words replaced by the boost appear in the response with
boosted: trueandoriginal: "<pre-boost word>".
return_timestamps=wordadds per-wordstart/endto the REST response.include_confidence=trueadds per-wordconfidence(0-1).
min_speakers/max_speakersform fields onPOST /stt.
script_correction=truefor code-mixed Devanagari / Latin audio.
snr_dbis now a top-level field on every STT response and a per-message field on WStranscriptionevents. Surface as a “good / fair / poor” badge (>= 15good,0–15fair,< 0poor).
Limits
Per-user concurrency cap. Counted across REST + WS combined.Excess returns:
- REST —
429witherror_code: STT_CONCURRENCY_LIMIT/TTS_CONCURRENCY_LIMITanddetails.limit. - WebSocket — error frame followed by close code
1008. The cap releases when an in-flight request completes; no server-side queueing.
New error codes
499 is non-standard but used (nginx convention) to signal the client closed the connection before the response was sent.New warning codes
The same
low_snr_dropped value appears as a processing_mode on WS transcription events — treat as “skip this utterance, no useful text”.Billing changes
- WS double-bill fix. Sessions that previously double-billed when the client sent
stopare now billed correctly.billing_summary.total_duration_secondswill show as ~50 % of historical values for affected sessions. billing_summary.client_estimated_seconds(new, diagnostic) — rough estimate of audio the client sent. Useful only for debugging duration drift; never display as a billed number.- Low-SNR refunds. REST sessions where upstream dropped audio for low SNR no longer charge the user. WS already handled this.
- Cancellation propagation. Client disconnects mid-request now correctly abort the upstream call. No charge for cancelled requests (
499 STT_CLIENT_CANCELLED). - Diarization surcharge documented. When
diarize=true, requests incur a documented +30 % surcharge on top of the base STT rate to cover GPU diarization (pyannote).
Plan-tier gate
context (LLM refinement) is now a paid feature. On the Free plan the field is silently stripped server-side and the response includes warning_codes: ["llm_refinement_not_in_plan"]. Surface an upgrade prompt or hide / disable the input on Free plans for better UX.Surface summary
- REST —
POST /sttaccepts new form fields (keywords,languages,min_speakers,max_speakers,return_timestamps,include_confidence,script_correction,min_split_sec); returns new fields (snr_db,language_source: "long_audio_chunked", per-wordboosted/original/confidence, per-segmentchunk_idx). - REST
POST /tts,/tts-synthesize,/tts-stream— may now return429 TTS_CONCURRENCY_LIMIT. - WebSocket
/ws/stt— addslow_snr_droppedtoprocessing_mode; per-wordboosted/original; message-levelsnr_db; concurrency-limit error frame + close1008; correctedbilling_summarywith newclient_estimated_secondsfield. - WebSocket
/ws/tts— concurrency-limit error frame (legacy{error: {message, code, details}}shape) + close1008. - Web UI — axios timeout raised to 600 s, error-code-aware toasts,
warning_codessurfaced, WS1008distinguished from auth/network failures. The 25 MB upload cap is unchanged.
LLM context refinement for STT
Added — Context-gated LLM refinement
Speech-to-Text now accepts an optional context hint that opens a server-side LLM refinement gate. When supplied, the response transcript is polished for proper-noun accuracy, filler removal, punctuation, and script consistency in mixed-language audio.The shape differs per transport:REST — context: string
Free-form paragraph describing the session (domain, speakers, jargon). Serialized as a multipart form field.cURL
JavaScript
Python
CLI
MCP
WebSocket — structured {general, text, terms}
Start message
Changed — WebSocket two-phase canonical flow
Whencontext is supplied, each utterance now produces two transcription events sharing a sentence_id:- First emit —
is_final: true, speech_final: false— fast dict-corrected text. Use for low-latency UI paint and voicebot barge-in / NLU. - Canonical —
is_final: true, speech_final: true— definitive answer. Either LLM-refined (whenllm_applied: true) or the original text re-emitted (when LLM was skipped or failed).
Simple consumers can gate exclusively on
speech_final: true and ignore the rest — one canonical event per utterance regardless of whether refinement is on.See the WebSocket STT Reference for the full table and reconciliation patterns.Added — Word-preservation guardrails (proxy + UI)
To defend against over-aggressive LLM refinement and fast-speech hallucination rejection:- Refined word-retention guardrail. If the canonical text drops more than ~60% of the first-emit’s token count, the proxy rolls back to the first-emit text and marks
llm_applied: false, llm_reason: "dropped_too_many_words". Tuned so legitimate polish (filler removal, 10–40% compression) passes through untouched. - Hallucination-rejected fallback. When the upstream server’s word-rate guard emits an empty final (
processing_mode: "hallucination_rejected", common on fast-paced Indic/English mixed audio), the proxy upgrades it into a tentative canonical using the cached first-emit / last-interim text plustentative: trueandtentative_reason: "hallucination_rejected"so words never silently disappear.
/ws/stt proxy, so voicebots and web clients inherit them automatically.Added — Backward-compat refined event handling
Pre-migration upstream builds emit LLM refinement as a separate refined event rather than a second transcription. The proxy and web client transparently accept both shapes, so existing integrations keep working through the rollout.If you see
refined events in the wire trace, upstream workers haven’t been
restarted onto the two-phase build yet — it’s still fully functional; the
refined event is deprecated but accepted.Surface summary
- REST —
POST /stt,POST /v1/transcribe—context: stringform field. - WebSocket —
/v1/stream,/ws/stt—context: {general, text, terms}onstart; two-phase canonical flow;llm_applied/llm_latency_ms/llm_reasonon canonical. - JavaScript SDK —
client.speechToText(audio, { context }). - Python SDK —
client.speech_to_text(audio_file, context=...). - CLI —
60db stt:transcribe --context "...". - MCP Server —
sixtydb_stt_transcribeacceptscontext. - Proxy — word-retention guardrail, hallucination fallback, first-emit caching, legacy
refinedshim. - Web UI — context input in Speech-to-Text page and realtime demo; segments rendered with
tentativemarker when flagged.