Overview
Reliability tools show you how well an agent works, and find problems before real callers do.- Latency per turn. See where each turn’s time went: speech to text, pause hold, model, tools, voice and queue.
- Accuracy and safety results. See the results of each call, and the pooled rates for one agent version.
- Simulation. AI callers phone your agent. Each call is judged. You see which stories fail and why. Needs a Pro or Business plan.
- Draft and re-run. Make an unpublished draft from a run, edit it, run the same stories again, and compare.
- Judge quality. Check how closely the Judge agrees with answers that people gave.
Numbers you can trust
- One source at a time. Choose
production(live calls),simulationortest(text tests). The numbers are never mixed. - Times are whole milliseconds. Rates are between 0 and 1.
- A missing value is
null. The dashboard shows “Not enough data”. It never shows 0 for a missing value. - Only
passandfailcount for accuracy. A call withinfra_errororsimulator_erroris not the agent’s fault. If more than 5% of the finished calls haveinfra_error, the run isincompleteand its pass rate cannot be trusted. - Significance needs 30 samples on each side. With fewer,
significantisnull.
Latency per turn
On a call, the Latency tab draws one bar for each turn. The parts of a bar add up to the turn’s total time:
Early parallel work is shown beside the bar. It is not on the path, so it is never inside the bar. A turn that never finished shows where it stopped, and it counts at the timeout. The API gives the same data in Get call result.
Evaluations on a call
The Latency tab of a call also lists the evaluations stored for it. Each one shows a result (pass, fail or needs_review) and, when the Judge gives one, how sure it was. needs_review means the Judge’s confidence was below 0.85. It is counted apart and never counts as a pass. A result the dashboard cannot read is shown as “Result not readable”. It is never shown as a pass. If no evaluations were stored, the tab says so.
Percentiles and stages
The Reliability tab shows p50, p90, p95 and p99 for one agent version. A percentile needs 30 samples. Below that it isnull, and the tab shows “Not enough data” with the sample count.
A table below the tiles shows the same percentiles for each of nine stages: speech to text, pause hold, model first word, model rest, voice, queue, knowledge search, tools and other. A stage that a turn did not measure adds no sample, so stages can have different sample counts. The rule for each percentile is the value at rank ceil(n × q) of the sorted turns, with no interpolation. A simulated call from an overloaded worker is left out of every latency number but still counts for accuracy.
Run a simulation## Run a simulation
- Open the agent, choose Reliability, then Start a simulation. Or call Start simulation.
- Choose Generate stories (the agent’s model writes 9 to 50 stories) or Write a story.
- Set repeats, the estimated credits per call and the credit ceiling. The dialog asks for a free estimate and shows the number of calls, the credits, and how many calls your ceilings let start, before you confirm.
- Watch the run. Read it every 3 to 5 seconds until its state is
finished,cancelledorstopped_budget. You can close the browser. The run goes on, and you can read the result later.
Limits
A start is refused if the balance cannot cover the lower of the credit ceiling and the estimate × the number of calls. A run that reaches its ceiling ends in the state
stopped_budget with a partial report.
Write a story
A story is one JSON object:say, now, quiet, cut_in, burst, breath, beep, ai, listen and hang_up. Checks are contains, not_contains, tool_called, tool_not_called, entity and task_success. Each check item is { "check": { "check": "<type>", "arg": … }, "must_pass": true|false }. The story must answer every tool the agent has in tool_answers, or the start answers 400 with field: "tool_answers".
60db passes steps and checks on as they are. The voice service checks them. If a step or a check is wrong, you get 400 with the field name. A check with must_pass: true fails the call on one miss.
Read a run
Get simulation gives:- progress:
total,done,queued,claimed - the
pass_rate(nulluntil a call is judged) andincomplete - a pass rate for each scenario, with a flaky mark when results change between repeats
- failure groups, for example
failed_check:containsorstalled_at:tool - regressions against a baseline run
- one entry for each call, with its label and summary
Fix and re-run
- Create a draft from a finished run. The draft is unpublished.
- Edit the draft script. The limit is 32,000 characters.
- Re-run the same stories against the draft. Set a new credit ceiling. The new run is compared with the first run.
- If the draft is better, save the agent yourself. 60db never publishes an agent from a simulation.
404.
Compare versions
Compare agent versions gives the rate for each version, the change, andsignificant. It also gives p50, p90, p95 and p99 for both versions and the change (b minus a). significant is true when the two 95% intervals do not overlap, false when they do, and null when either side has fewer than 30 judged calls. Read the sample counts next to each rate.
Judge quality
Choose the eval to check from the agent’s list of evals. It shows which evals score calls and how each golden check stands. An eval marked unreliable has too many false passes. You can also type a name. Run golden eval measures how closely a Judge eval agrees with answers that people gave. It makes one paid Judge request for each example. With fewer than 10 examples it says nothing.unreliable: true means the Judge disagrees too often to trust its scores.
Permissions
AI Simulation is a Pro feature: on Free, Starter or Creator, the simulation endpoints return
403 POLICY_DENY. An API key needs the agents scope.