Surfaces: REST, Docs
Changed
- The analyzer’s
languagenow limits the languages used to transcribe each speaker turn. Set it to a list of codes such ashi,enfor Hindi and English calls. Before this, the setting was stored but not used for transcription, so short turns could come back in the wrong language. Leave it empty for automatic detection. See Create analyzer
Surfaces: REST, Docs
Added
require_zero_retentionon agents (boolean, defaultfalse). See Zero retention and Create agent503 ZERO_RETENTION_NOT_CONFIRMED— when the option is on, every call start with the agent is refused before anything is dialed or charged. No provider can confirm zero retention yet, so turn the option off to make calls
Notes
- Inbound calls are routed by the voice runtime and are not refused
Surfaces: REST, Dashboard, Docs
Added
- Self healing — an analyzer that watches AI agents can draft a change to the agent’s instructions when the same problem repeats in 3 calls, and optionally apply it, measure the next 10 calls and roll it back if it got worse. Modes:
off,suggest,auto. See Self healing self_heal_modeon Set Sources andself_healin Get Sources- Healing endpoints — List, Approve, Dismiss and Undo healing changes. For signed-in users, not API keys
Notes
- Drafts are free. At most one open change per agent, a 24 hour cool-down and one draft per agent per day
- Dialer healing is not included yet
Surfaces: REST, WebSocket, Dashboard, Docs
Added
- Outbound campaigns — call a CSV list of numbers with one agent. Set calling windows, retries, concurrency and the do-not-call list. See Outbound campaigns and List campaigns
- Do-not-call list —
GET,POSTandDELETE/agents/do-not-call. See Add do-not-call number - Integrations — connect HubSpot, Zendesk, Slack and other providers, or your own MCP server, and give their tools to an agent. See Integrations and custom webhooks and List integration catalog
- Custom webhooks with several actions — one connection holds up to 20 actions. Each action has its own URL, method, description, parameters and fixed values, and is offered to an agent as its own tool. The shared headers and one
whsec_signing secret apply to all of them. Choice lists (enum) refuse values outside the list. See Custom webhooks - Connect integrations to agents —
PUT /agents/integrations/{id}/agentssets which agents hold a connection’s tools in one request. Connections list the agents that hold their tools inused_by. See Connect integrations to agents and Assign integration agents - Direct WebSocket per agent — one permanent URL,
media_websocket_urlon each agent, for a client that dials a WebSocket with a shared secret. Each connection places one call. See Agent Media WebSocket integration_toolson agents. See Create agent
Notes
- A campaign runs 3 calls at once on a 60db Dialer line. On your own line it runs 1 to 50
- Creating, checking and resuming a campaign needs an active plan
Surfaces: REST, MCP, Docs
Added
- Call analysis API — create an analyzer, add recordings by file upload, URL or from your AI agent and Dialer calls, run the analysis, then read problems, analytics and the analysis of each call. Works with a normal API key at
https://api.60db.ai/deep-analytics. See Call Analysis API and Analyze your first calls - Fix your AI agent — suggest, apply and undo better agent instructions from the problems found in a call. Needs the key’s
agentspermission - New
call_analysisAPI-key permission — the analyzers API now requires it. New keys have it by default. Keys created before this change do not: switch on Call analysis in Developers → API keys, orPUT /developer/api/{id}withpermissions.call_analysis = true. Without it every analyzer route returns403 DA_API_KEY_SCOPE_MISSING - MCP — 14
sixtydb_call_analysis_*tools. See Call Analysis Tools
Notes
- Analysis uses speech-to-text and model credits. You confirm the recording count before each run
- API keys are limited to 300 requests per minute on these routes. Other analytics routes are not available to API keys
Surfaces: REST, Dashboard, Docs
Added
- Inbound calls — an agent answers calls to a number you own. Phone lines of kind
byotakeinbound_number,numbers,allowed_addresses,allowed_numbers,normalize_numbers,default_country_codeandbridge_transfers. See Bring your own SIP - REST —
GET /agents/phone-lines/{lineId}/setup(Get phone line setup) andPOST /agents/phone-lines/{lineId}/check(Check phone line), a read-only guide and a dry-run for inbound calls - Knowledge on your server —
knowledge_webhook_urlon agents, signed with the agent’swhsec_...secret. See Knowledge for agents - Guides — Move from LiveKit or Pipecat and Names: HelloKit and 60db
Changed
usernameandpasswordare now optional on abyophone line (both or neither). Leave them out for a carrier that trusts your IP- A number now answers in exactly one place: another phone line, a Dialer number or a LiveKit route. Creating a clash returns
409 NUMBER_IN_USEor409 NUMBER_OWNED_ELSEWHERE - Setting both
knowledge_collection_idandknowledge_webhook_urlreturns422 KNOWLEDGE_SOURCE_CONFLICT
Removed
- The
INBOUND_NOT_AVAILABLEerror. It was never returned by the API; inbound calls are available
Notes
- Only the 60db carrier has been tested end to end for inbound calls. Other carriers and PBX recipes are marked untested in the guide
- SIP is UDP only with plain RTP (PCMU and PCMA). There is no TLS or SRTP
Surfaces: REST, SDK, Dashboard, MCP, Docs
Added
- Agents — build phone voice agents with a script, a 60db voice, languages and optional tools. Place outbound calls from a 60db Dialer number or your own SIP trunk, and follow each call to completion. See Agents
- REST —
/agentsendpoints for agents, phone lines, calls, transcripts, recordings, the tool-call log and webhook secret rotation. See Agents API reference - Tools and webhooks — agents call your endpoint during a call with JSON Schema tools. Call and recording events carry
X-60db-Signatureand are retried up to five times. See Tools and webhooks - SDKs —
client.agentsin JavaScript (60db-service1.2.0) and Python (sixtydb2.1.0), withverifyWebhookSignatureandverify_webhook_signature - MCP — ten
sixtydb_agents_*tools. Placing a call and deleting an agent requireconfirm: true. See Agents tools - Dashboard — Agents pages with templates, phone lines, test calls, call history and the tool log
Notes
- There is no agent runtime fee ($0). Line minutes on 60db numbers bill from your Dialer balance at ₹0.30/min
- Inbound calls on your own SIP trunk arrived on 2026-11-06, see the entry above
Surfaces: REST, WebSocket, Dashboard, MCP, Docs
Added
- Automatic prompt caching on Chat — shared prompt prefixes are cached and cached input tokens are billed at a discounted cached-input rate.
usage.prompt_tokens_details.cached_tokensreports how many tokens were served from cache. Optionalcache_controlandprompt_cache_keyhints are accepted. See Prompt caching - STT re-upload cache —
POST /sttacceptscacheandcache_control. Byte-identical audio with the same options from the same workspace within 30 days returns the stored transcript, and the response includescached - Analytics and usage logs —
llm_cached_tokensin Get usage,metadata.cached_tokensin usage logs,pricing.cached_input_rate_per_millioninGET /v1/usage
Fixed
- Corrected example costs in the usage docs to the $0.50 per 1M token LLM rate and the $0.0005 per minute STT rate
Surfaces: REST, WebSocket, Dashboard, Docs
Changed
- Pro plan concurrency is now unlimited for TTS and STT (REST + WebSocket). Pro workspaces no longer get
429 TTS_CONCURRENCY_LIMIT/STT_CONCURRENCY_LIMITor a1008close for concurrency. Free, Starter and Creator limits are unchanged - API —
concurrency.tts/concurrency.sttinGET /plansandGET /billing/current-planarenullwhen unlimited. See Concurrency limits
Surfaces: Dashboard, REST, MCP, Docs
Changed
- Instant voice limit — Free, Starter and Flow workspaces can keep 1 instant (cloned) voice; Creator and Pro keep 10. The limit counts live voices, so deleting one frees a slot. Over the limit,
POST /voicesreturns403 VOICE_LIMIT_REACHED(was429) - Professional voices — available on Creator (2 voices) and Pro (5 voices); other plans get
403 PVC_NOT_AVAILABLE_ON_PLAN, and a full plan gets403 PVC_LIMIT_REACHED. You can now create more than one professional voice, and the page lists them all - Marketplace listing — needs a Creator or Pro plan, a submitted KYC tax form and bank details. Submitting works while they are under review; the voice goes live only after KYC is verified and bank details are active. Otherwise submitting returns
403 MARKETPLACE_REQUIREMENTS_NOT_MET - Downgrades — existing voices are kept; creating and listing new ones is blocked while over the limit
Added
- Get Voice Limits (
GET /voices/quota), List Professional Voices (GET /voices/professional/jobs) and Get Marketplace Eligibility (GET /voices/professional/marketplace-eligibility) - MCP — read-only tools
sixtydb_get_voice_quota,sixtydb_list_professional_voicesandsixtydb_get_marketplace_eligibility
Surfaces: WebSocket, Proxy, Docs
Fixed
- STT auto-detect — omitting
languages, or sendingnull,[],"auto"or["auto"], now expands to the full language list the STT server advertises. Before, the session started but real speech failed withNo compatible route exists for this candidate set.session_started.language_modeiscandidate_autofor two or more languages andselectedfor one - TTS upstream — the
/ws/ttsproxy connects to the TTS server through its hostname instead of a fixed IP address
Changed
- TTS
context_creatednow includessession_idandbinary_audio(alwaysfalse; audio is base64 inaudio_chunk) - TTS
warningandcontext_abortedare documented.warningwith codesynthesis_interruptedmeans new text arrived mid-synthesis; sendflush_contextto speak it
Surfaces: REST, Docs
Fixed
- Documented response fields —
POST /tts-synthesize(withoutstream) now returnssuccess,message,audio_base64,sample_rate,duration_seconds,encodingandoutput_format, as documented. Previously it returned the engine’s rawaudioContent(headerless PCM) and an internalconditioningobject - Playable audio — for
LINEAR16,audio_base64is a complete WAV file (16-bit mono PCM with header), so it can be saved or played directly - Backward compatible —
audioContentis still returned unchanged for existing integrations. Streaming ("stream": true) is unchanged: NDJSON lines withresult.audioContent - No charge without audio — a response that carries no audio now returns an error and is not billed. See Text to Speech
Surfaces: Dashboard, REST
Changed
- Audio projects — create a project, write your text and generate narration. Import and URL import were removed
- Smart regeneration — the first generation converts every line; after that only new or edited lines are generated (and charged). Lines that need generating are marked, and “Generate all” skips finished blocks and respects your plan’s concurrency limit
- Download — the whole project downloads as one MP3 (
GET /studio/projects/:projectId/download). Missing lines are generated first; the endpoint returns409 AUDIO_INCOMPLETEwith the missing lines otherwise - Emotion tags — insert
[happy],[laugh],[pause 1s]and the rest from the Emotion picker, or use Enhance / Remove tags, the same as Text to Speech - Billing — Studio generation is always charged to the workspace you are signed in to, at the TTS rate
Surfaces: Dashboard, REST, Docs
Changed
- Upload only — the professional clone wizard is now Intro → Upload → Verify → Review. The teleprompter recording method was removed
- Marketplace listing is optional — choose it on the Review step (off by default). Unlisted voices are ready to use privately right after training, with no review.
POST /voices/professional/jobs/:id/finalizeacceptslist_in_marketplace(boolean); jobs returnlist_in_marketplaceand the new statusprivate - Submit later —
POST /voices/professional/jobs/:id/submit-to-marketplacesends a private voice to marketplace review (private→awaiting_approval) without retraining. See Professional Voice Clones
Surfaces: REST, WebSocket, Dashboard, Docs
Changed
- Concurrency limits now depend on your plan and apply per workspace (REST + WebSocket combined, shared by members and API keys). TTS: Free 1, Starter 5, Creator 10, Pro 20. STT: Free 2, Starter 10, Creator 20, Pro 50. Previously a fixed per-user cap (TTS 5, STT 8)
- 429 / 1008 errors —
TTS_CONCURRENCY_LIMIT/STT_CONCURRENCY_LIMITnow name your plan and its limit;detailsaddsplan - API —
GET /plansaddsconcurrency { tts, stt }per plan;GET /billing/current-planaddsconcurrencyfor the workspace. See Concurrency limits
Surfaces: REST, Docs
Added
Annual billing
- Yearly prices — Starter $60/yr ($5/mo), Creator $220/yr ($18.33/mo), Pro $990/yr ($82.50/mo). Yearly is monthly × 10, so you get 2 months free. Credits stay a monthly allotment (120K / 400K / 2M per month) and reset every month on your subscription start day
- Subscribe on a cycle —
POST /billing/subscribetakesbilling_cycle: "monthly" | "yearly"(defaultmonthly) and returnsaction(subscribe,upgradeorswitch_to_yearly),billing_cycleandamount. See Subscribe to a Plan - Plan prices —
GET /plansaddspricing.yearly,yearly_monthly_equivalentandyearly_savings_pct. See List Plans - Current plan —
GET /billing/current-planaddsdata.plan.billing_cycle,next_credit_reset_atandnext_payment_at. For yearly plansnext_payment_usdis the yearly price. See Current Plan - Plan changes — monthly to yearly and upgrades take effect immediately, and unused plan credits carry over. Yearly to a higher yearly plan gets a prorated discount at checkout. Downgrades and cancellations take effect at the end of the billing cycle
- Errors —
403 YEARLY_TO_MONTHLY_NOT_ALLOWED(cancel the yearly plan, then choose monthly),400 DOWNGRADE_KEEPS_CYCLE,400 CYCLE_NOT_AVAILABLE,400 INVALID_BILLING_CYCLE
Updated
- Plans & credits guide — new Annual billing section with the price table, plan-change rules and refund policy. Top-ups on yearly plans use the monthly plan rate
Surfaces: REST, Dashboard, Docs
Added
Pay-as-you-go top-ups (Creator and Pro)
- Buy top-ups — Creator and Pro workspaces can buy $10–$1,000 of credits (presets $10, $25, $50, $100) with
POST /billing/add-funds(purpose: "credits"). Top-up credits never expire and stay usable after a downgrade - Plan rate — credits =
floor(usd × plan monthly credits / plan price). Creator buys about 18,181.8 credits per $1 ($10 → 181,818), Pro about 20,202 per $1 ($10 → 202,020). The credits are quoted at checkout and fixed, even if you change plan before the payment completes. Coupons lower the price, not the credits - Quote and balance —
GET /billing/topupreturns eligibility, your plan rate, preset quotes and the plan / top-up credit balance.GET /billing/topup/quote?amount=previews a custom amount - Dashboard — Subscription → Platform has a new Top-up credits card with a Pay-as-you-go button. When credits run out, the You’re out of credits dialog offers Pay-as-you-go (Creator, Pro) or Upgrade to Creator (Free, Starter) instead of an error toast
- Ledger —
GET /billing/usage-logslists top-ups asTOP_UProws, andSA_TOP_UPfor offline payments recorded by 60db support.AUTO_TOP_UPis reserved for automatic top-up - Errors —
403 TOPUP_NOT_AVAILABLE_ON_PLANon Free and Starter, and400 TOPUP_AMOUNT_OUT_OF_RANGEoutside $10–$1,000
Updated
- Out-of-credits
402— the message now points to a top-up (Creator, Pro) or an upgrade (Free, Starter), and the body includesdetails.topup_availableanddetails.auto_topup_enabled - Plans & credits guide — new Plans, Credits & Top-ups page. Usage draws plan credits first, then top-up credits
- Pricing pages — Memory and Judge pricing no longer describe a pay-as-you-go wallet only
Coming soon
- Automatic top-up — save a card once and credits reload automatically when your balance falls below a threshold you choose, up to a monthly limit you set. Not available yet
Surfaces: REST, Docs
Added
- Emotion tags — put
[happy],[calm],[apologetic]and 17 other emotion tags inline intextto change how the following sentences sound, blend two with[sad, calm], and add sound tags such as[laugh],[sigh],[gasp]and[pause 500ms]. Unknown tags are dropped, never spoken. Works onPOST /tts-synthesizeand in the dashboard. See Emotion Tags - Enhance —
POST /tts/enhanceadds emotion and sound tags to your text (up to 5,000 characters) and changes nothing else: your words, spacing and language come back exactly as sent. No credits are charged. See Enhance Text - Dashboard — the Text to Speech page highlights tags as colored chips and adds Add emotion, Enhance, Undo enhance and Remove tags
- Audio enhancement — generated audio is now polished by default. Send
audio_enhance: falseonPOST /tts-synthesizeto skip it (the olderenhancestill works). It is never applied to the real-time WebSocket TTS used by voice agents, so live calls keep their low latency. The dashboard has a new Audio enhancement switch. See Text to Speech
Surfaces: REST, Docs
Added
Click-to-Call (Production)
- Place a call —
POST /dialer/click2callrings the agent on your own DID, then the customer, then bridges them. No SIP client, no WebRTC, no microphone. Returns 201 when a call was dialled and 200 when thereference_idwas already bound to a live call and nothing new was dialled (data.placed/data.replayed) - Read a call —
GET /dialer/click2call/{id}bycall_idor your ownreference_id;?refresh=1polls the platform for the authoritative state and repairs a call whose terminal webhook never arrived - History —
GET /dialer/click2callwithlimit/offset,mine=0for the whole workspace, andapi_key_idfor a per-integration statement - Signed webhooks — pass
callback_url+callback_secret(16–256 chars, yours to choose) per call; every verified event is relayed to you withx-qcall-signature: v1=<hmac-sha256>overtimestamp + "." + raw_body. Up to 6 attempts, ≥2 min apart, abandoned after 24 h - Delivery inspection —
GET /dialer/click2call/{id}/eventsshows every event, whether it was applied, and the status your own endpoint returned for it - Recordings —
record: trueon placement, thenPOST /dialer/click2call/{id}/recordingfor a short-lived playback URL (audio uploads after the call ends — checkrecording_available) - Readiness —
GET /dialer/click2call/configreports whether the server can dial, whether it can verify results, and the numeric limits it enforces - Idempotency —
reference_idis optional and minted per call when omitted; it is permanent per trunk, and a failed call still consumes it. Refusals that placed no call (429,503) are safe to retry with the same one - Billing — ₹0.30/min ($0.003/min) on bridged talk time only, prorated per second. Unanswered calls are free.
cost_usdandbilling_statusare on the call itself - Limits — 20 calls/min per caller (
C2C_RATE_LIMITED), ring timeouts 5–60 s (defaults 30 agent / 45 customer),max_duration30–7200 s (default 3600),metadata≤ 2 KiB - See the Click-to-Call guide and API reference
Updated
- Dialer feature guide — now distinguishes the two ways to place a call:
POST /dialer/calls(your SIP client dials) vsPOST /dialer/click2call(60db dials both legs) - Introduction — Click-to-Call feature card
Surfaces: REST, SDK, MCP, Docs
Added
Music API (Beta, Free)
- Song generation —
POST /songs(simple mode: one prompt, we write lyrics; advanced: full control) - Song polling —
GET /songs/{id}(async generation, ~1.5–2.5 min) - Voice management — catalog voices or save custom voices from audio samples
- Download —
GET /songs/{id}/download(MP3) - SDK methods —
client.music.create(),list(),get(),update(),delete(),download(),waitForCompletion(),listVoices(),createVoice() - MCP tools —
sixtydb_music_*tools for Claude Desktop - See Music feature guide and API reference
Dialer API (Production)
- Number management — rent Indian numbers (E.164,
+919876543210), set caller ID, release - Call reservation —
POST /dialer/calls(SIP client places the actual call; API reserves) - Call history —
GET /dialer/logs(CDR with billsec, disposition) - Recordings & transcripts — auto-record, fetch playback URL (~5 min), get transcript
- Webhooks — subscribe to call/recording events (create, test, rotate signing key, delivery history)
- Usage/billing — track charges per-minute and monthly subscriptions
- Setup prerequisites — trunk + KYC done in dashboard (not API; gates are 409/403 on missing setup)
- Pricing — ₹600/month ($6) per number; ₹0.30/min ($0.003) per call (inbound/outbound, prorated/sec)
- SDK methods —
client.dialer.status(),numbers.search()/buy()/list()/release()/setCallerId(),calls.list()/get()/reserve(),recordings.list()/playbackUrl()/transcript()/delete()/setEnabled(),billing.subscriptions()/usage(),webhooks.list()/create()/update()/delete()/rotateSecret()/test()/deliveries() - MCP tools —
sixtydb_dialer_*tools;confirm: truerequired for operations that charge money or release resources - See Dialer feature guide and API reference
Updated
- SDKs — Music and Dialer sections added to JavaScript and Python docs
- Introduction — feature cards for Music and Dialer
- MCP Server — new music and dialer tool groups
STT WebSocket — single-phase finals, turn-shaping config (early_final / turn_detector), audio_enhancement on|off
Surfaces: WebSocket, Proxy, SDK, DocsThe streaming STT protocol was simplified to a single-phase model. The server no longer rewrites text after recognition (no LLM refinement), so the two-phase
speech_final flow and its accompanying fields are gone. This is the behaviour every integration (proxy, web app, teleprompter) now targets.Changed
speech_finalnow always equalsis_final. Every committed result is a singletranscriptionwithis_final: true(andspeech_final: true); interims carry bothfalse. There is no second “canonical” emit. Simple consumers append each final and clear the interim.config.audio_enhancementvalues are now"on"|"off"(defaultoff). The old"light"/"adaptive"(and booleantrue) are accepted for compatibility and map to"on". Use"on"for noisy / telephony audio.- Billing now charges every committed final by its
duration. Withheld finals (emptytext) andno_speechare never charged. A single turn can produce several finals (short-pause splits,segment_finalon long monologues) — their durations are non-overlapping, so there is no double-charge.
Added
- Turn-shaping config:
early_final(bool, defaulttrue; sendfalseto keep whole sentences in one final) andturn_detector(bool, defaultfalse; sendtrueto wait until the caller finished the thought — emitsturn_complete). See the three latency/accuracy profiles in the STT reference. - More config:
interim_min_voiced_ms(default 600),interim_delta_streaming,token_streaming,max_utterance_sec(3–30, default 30). finalizeclient message ({ "type": "finalize" }) — closes the current turn now and returns its transcription (orno_speech); the session continues.- New server events:
no_speech(turn had no words;terminaltrue/false),turn_complete(only withturn_detector: true),interim_delta(withinterim_delta_streaming),transcription_tokens(withtoken_streaming), andwarning(e.g.audio_format_assumed). - New transcription fields:
segment_final(length-cap split — check beforespeech_final),final_id(unique per committed result — de-duplicate on it), andendpoint_reason(vad_silence|client_finalize|client_stop|max_utterance|language_change|response_timeout|interim). - Language coverage is now 45 (adds
ja,ko,zh,vi,tr,he, and more).GET /v1/languagesis authoritative.
Removed
config.continuous_mode— streaming is always continuous; the field is ignored (a session no longer stops after the first utterance).config.remove_fillersandconfig.no_speech_threshold— no longer used (filler stripping was an LLM-refinement feature, which is gone).- The two-phase /
refinedprotocol and its fields: the separaterefinedevent,is_final: true, speech_final: falsefirst emits,llm_applied/llm_reason/llm_latency_ms,tentative/tentative_reason, andsession_started.llm_refinement. - The
mode_changedevent (it acknowledgedcontinuous_mode). - The old skip-mode markers (
speech_end_no_result,speech_end_too_short,hallucination_rejected) — withheld low-confidence finals now arrive astext: ""withprocessing_mode: "low_confidence_transcript".
Migration
- Drop
continuous_mode/remove_fillers/no_speech_thresholdfromstart.config; addearly_final: falseandturn_detector: truefor whole-sentence finals (profile A). - Change
audio_enhancement: "adaptive"/"light"→"on". - Gate on any
is_final: true(equivalentlyspeech_final: true); stop waiting for a second canonical emit and removerefined-event handling. words[]timings useprobability, notconfidence.
Surfaces: WebSocket, Proxy, Docs
Changed
config.utterance_end_ms minimum and default are now 1000 ms (were 300 ms minimum / 500 ms default).- The upstream STT server rejects a
startcarrying a lower value instead of clamping it, which killed the session outright. - The 60db
/ws/sttproxy now clamps any value below 1000 ms up to 1000 ms before forwarding — onstartand on mid-sessionconfig— so existing clients sending500keep working and simply get 1000 ms behaviour. - Need faster turn-taking? Drive barge-in from
speech_startedand interim results rather than lowering this value.
Fixed (docs)
STT handshake frames were documented with the TTS shape./ws/stt sends {"type":"connecting"} and {"type":"connection_established","service":"stt","user_id":…} — tagged by type, fields top-level. The examples previously branched on msg.connection_established, the nested /ws/tts shape, which never matches on STT. Examples now also wait for session_started before streaming audio, since connected arrives before the session exists and is emitted twice (proxy frame, then upstream capability frame).Documented
session_started.llm_refinement— whether yourcontextactually opened the refinement gate.llm_applied,llm_reason,llm_latency_mson canonical transcription emits.tentative/tentative_reason— proxy-substituted text when upstream rejects an utterance as a hallucination.- The proxy’s word-preservation guardrail: a refinement that keeps under 40% of the first emit’s words is rolled back to the original text.
connection_closed, theAUTH_FAILEDerror code, and the nested mid-sessionINSUFFICIENT_CREDITSerror shape.- Billing: only canonical finals are charged;
speech_end_no_result,speech_end_too_short,hallucination_rejectedandlow_snr_droppednever are.
Changed
POST /tts-synthesize now accepts the full Inworld schema.audio_config: { audio_encoding, sample_rate_hertz }— nested block matching the upstream schema.timestamp_type(default"WORD") — enables per-word timestamps in the final NDJSON chunk.- Legacy flat keys (
sample_rate,audio_encoding) still work; new clients should useaudio_config.
sample_rate_hertz moved from 48000 → 24000 to match the upstream default and reduce bandwidth. Callers that need 48 kHz must send it explicitly.See Text to Speech reference.Added
config.remove_fillers on WebSocket start (STT).- New boolean field inside
start.config. Defaultfalse. - When
trueandcontextis set (LLM refinement gate is open), the canonicaltranscriptionemit (speech_final: true) drops filler words —um,uh,like,you know, … - No effect when
contextis omitted (refinement stays off). - The 60db
/ws/sttproxy coerces non-boolean values tofalserather than rejecting the session.
Surfaces: REST, Email
Added
Onboarding fields (POST /user/onboarding).- Now accepts
default_voice_id(string),marketing_consent(boolean),consent_timestamp, andconsent_source— all optional and backward compatible. - Marketing consent is server-stamped:
consent_timestamp/consent_sourceare set on the server; any client-sent values are ignored. - Consent is persisted best-effort after the onboarding commit, so a consent-write failure can never fail the onboarding save.
/unsubscribe).- Public, HMAC-token-verified endpoint for lifecycle email. Sends a
List-Unsubscribe(one-click) header for bulk-sender compliance.
Changed
Email delivery now uses Cloudflare Email Service.- Transactional and lifecycle mail send via Cloudflare (
smtp.mx.cloudflare.net, implicit TLS on 465) authenticated with a Cloudflare API token; DKIM/ARC are applied automatically. ConfigureCLOUDFLARE_API_TOKEN(used as the SMTP password) andSMTP_EMAIL(a sender on a domain onboarded for Email Sending). - Signup verification email is now sent after the account is committed, so a transient email/transport failure can no longer roll back account creation.
Fixed
GET /workspaces duplicate rows.- A workspace with more than one active membership row no longer appears multiple times in the response; the list is de-duplicated per workspace.
- Submitting onboarding no longer returns
500when the marketing-consent columns are absent from the database (consent persistence is best-effort — apply the consent migration to enable it).
- SMTP / Cloudflare credentials are scrubbed from any thrown or logged email error.
Security
- Cerbos authorization checks added across additional routes (STT, TTS, SLM, Studio, webhooks, sixtydb).
Migrations
- New environments must apply the consent columns on
users(marketing_consent,consent_timestamp,consent_source); plus the events / lifecycle-email tables if the lifecycle email program is enabled.
Added
Longer request timeout (RESTPOST /stt).- Server-side request timeout raised from 120 s → 600 s; clients should match. The 25 MB / ~1-hour file-size cap remains.
- New
keywordsform field onPOST /stt. CSV with optional per-term weights, e.g.Acme:5,XYZ Pharma:8. - The same boost runs on the WebSocket path via the existing
context.termsfield — no client change required to benefit; existingtermspayloads now produceboosted/originalmarkers automatically. - Words replaced by the boost appear in the response with
boosted: trueandoriginal: "<pre-boost word>".
return_timestamps=wordadds per-wordstart/endto the REST response.include_confidence=trueadds per-wordconfidence(0-1).
min_speakers/max_speakersform fields onPOST /stt.
script_correction=truefor code-mixed Devanagari / Latin audio.
snr_dbis now a top-level field on every STT response and a per-message field on WStranscriptionevents. Surface as a “good / fair / poor” badge (>= 15good,0–15fair,< 0poor).
Limits
Per-user concurrency cap. Counted across REST + WS combined.Excess returns:
- REST —
429witherror_code: STT_CONCURRENCY_LIMIT/TTS_CONCURRENCY_LIMITanddetails.limit. - WebSocket — error frame followed by close code
1008. The cap releases when an in-flight request completes; no server-side queueing.
New error codes
499 is non-standard but used (nginx convention) to signal the client closed the connection before the response was sent.New warning codes
The same
low_snr_dropped value appears as a processing_mode on WS transcription events — treat as “skip this utterance, no useful text”.Billing changes
- WS double-bill fix. Sessions that previously double-billed when the client sent
stopare now billed correctly.billing_summary.total_duration_secondswill show as ~50 % of historical values for affected sessions. billing_summary.client_estimated_seconds(new, diagnostic) — rough estimate of audio the client sent. Useful only for debugging duration drift; never display as a billed number.- Low-SNR refunds. REST sessions where upstream dropped audio for low SNR no longer charge the user. WS already handled this.
- Cancellation propagation. Client disconnects mid-request now correctly abort the upstream call. No charge for cancelled requests (
499 STT_CLIENT_CANCELLED). - Diarization surcharge documented. When
diarize=true, requests incur a documented +30 % surcharge on top of the base STT rate to cover GPU diarization (pyannote).
Plan-tier gate
context (LLM refinement) is now a paid feature. On the Free plan the field is silently stripped server-side and the response includes warning_codes: ["llm_refinement_not_in_plan"]. Surface an upgrade prompt or hide / disable the input on Free plans for better UX.Surface summary
- REST —
POST /sttaccepts new form fields (keywords,languages,min_speakers,max_speakers,return_timestamps,include_confidence,script_correction,min_split_sec); returns new fields (snr_db,language_source: "long_audio_chunked", per-wordboosted/original/confidence, per-segmentchunk_idx). - REST
POST /tts,/tts-synthesize,/tts-stream— may now return429 TTS_CONCURRENCY_LIMIT. - WebSocket
/ws/stt— addslow_snr_droppedtoprocessing_mode; per-wordboosted/original; message-levelsnr_db; concurrency-limit error frame + close1008; correctedbilling_summarywith newclient_estimated_secondsfield. - WebSocket
/ws/tts— concurrency-limit error frame (legacy{error: {message, code, details}}shape) + close1008. - Web UI — axios timeout raised to 600 s, error-code-aware toasts,
warning_codessurfaced, WS1008distinguished from auth/network failures. The 25 MB upload cap is unchanged.
Added — Context-gated LLM refinement
Speech-to-Text now accepts an optional context hint that opens a server-side LLM refinement gate. When supplied, the response transcript is polished for proper-noun accuracy, filler removal, punctuation, and script consistency in mixed-language audio.The shape differs per transport:REST — context: string
Free-form paragraph describing the session (domain, speakers, jargon). Serialized as a multipart form field.cURL
JavaScript
Python
CLI
MCP
WebSocket — structured {general, text, terms}
Start message
Changed — WebSocket two-phase canonical flow
Whencontext is supplied, each utterance now produces two transcription events sharing a sentence_id:- First emit —
is_final: true, speech_final: false— fast dict-corrected text. Use for low-latency UI paint and voicebot barge-in / NLU. - Canonical —
is_final: true, speech_final: true— definitive answer. Either LLM-refined (whenllm_applied: true) or the original text re-emitted (when LLM was skipped or failed).
Simple consumers can gate exclusively on
speech_final: true and ignore the rest — one canonical event per utterance regardless of whether refinement is on.See the WebSocket STT Reference for the full table and reconciliation patterns.Added — Word-preservation guardrails (proxy + UI)
To defend against over-aggressive LLM refinement and fast-speech hallucination rejection:- Refined word-retention guardrail. If the canonical text drops more than ~60% of the first-emit’s token count, the proxy rolls back to the first-emit text and marks
llm_applied: false, llm_reason: "dropped_too_many_words". Tuned so legitimate polish (filler removal, 10–40% compression) passes through untouched. - Hallucination-rejected fallback. When the upstream server’s word-rate guard emits an empty final (
processing_mode: "hallucination_rejected", common on fast-paced Indic/English mixed audio), the proxy upgrades it into a tentative canonical using the cached first-emit / last-interim text plustentative: trueandtentative_reason: "hallucination_rejected"so words never silently disappear.
/ws/stt proxy, so voicebots and web clients inherit them automatically.Added — Backward-compat refined event handling
Pre-migration upstream builds emit LLM refinement as a separate refined event rather than a second transcription. The proxy and web client transparently accept both shapes, so existing integrations keep working through the rollout.If you see
refined events in the wire trace, upstream workers haven’t been
restarted onto the two-phase build yet — it’s still fully functional; the
refined event is deprecated but accepted.Surface summary
- REST —
POST /stt,POST /v1/transcribe—context: stringform field. - WebSocket —
/v1/stream,/ws/stt—context: {general, text, terms}onstart; two-phase canonical flow;llm_applied/llm_latency_ms/llm_reasonon canonical. - JavaScript SDK —
client.speechToText(audio, { context }). - Python SDK —
client.speech_to_text(audio_file, context=...). - CLI —
60db stt:transcribe --context "...". - MCP Server —
sixtydb_stt_transcribeacceptscontext. - Proxy — word-retention guardrail, hallucination fallback, first-emit caching, legacy
refinedshim. - Web UI — context input in Speech-to-Text page and realtime demo; segments rendered with
tentativemarker when flagged.