> ## Documentation Index
> Fetch the complete documentation index at: https://docs.60db.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Reliability and Simulation

> Find where each turn's time goes, test an agent with AI callers before real callers do, and compare versions

## Overview

Reliability tools show you how well an agent works, and find problems before real callers do.

* **Latency per turn.** See where each turn's time went: speech to text, pause hold, model, tools, voice and queue.
* **Accuracy and safety results.** See the results of each call, and the pooled rates for one agent version.
* **Simulation.** AI callers phone your agent. Each call is judged. You see which stories fail and why. Needs a Pro or Business plan.
* **Draft and re-run.** Make an unpublished draft from a run, edit it, run the same stories again, and compare.
* **Judge quality.** Check how closely the Judge agrees with answers that people gave.

In the dashboard, open an agent and choose **Reliability**. The API reference starts at [Start simulation](https://docs.60db.ai/api-reference/agents/start-simulation.md).

> [!WARNING]
>
> Simulated calls spend real credits. They use the agent's own 60db key for speech, voice and model requests. Every run needs a credit ceiling.

## Numbers you can trust

* **One source at a time.** Choose `production` (live calls), `simulation` or `test` (text tests). The numbers are never mixed.
* **Times** are whole milliseconds. **Rates** are between 0 and 1.
* **A missing value is `null`.** The dashboard shows "Not enough data". It never shows 0 for a missing value.
* **Only `pass` and `fail` count for accuracy.** A call with `infra_error` or `simulator_error` is not the agent's fault. If more than 5% of the finished calls have `infra_error`, the run is `incomplete` and its pass rate cannot be trusted.
* **Significance needs 30 samples** on each side. With fewer, `significant` is `null`.

## Latency per turn

On a call, the **Latency** tab draws one bar for each turn. The parts of a bar add up to the turn's total time:

| Part | Meaning |
| - | - |
| Speech to text | Time to turn the caller's speech into text |
| Pause hold | Time the agent waits to be sure the caller finished |
| Model first word | Time until the model starts to answer |
| Model rest | Time for the rest of the answer |
| Voice | Time until the first sound |
| Queue, Knowledge search, Tool | Waits for capacity, the knowledge base and each tool, by name |
| Other | Time the stamps cannot explain |

Early parallel work is shown beside the bar. It is not on the path, so it is never inside the bar. A turn that never finished shows where it stopped, and it counts at the timeout. The API gives the same data in [Get call result](https://docs.60db.ai/api-reference/agents/get-call-result.md).

## Evaluations on a call

The **Latency** tab of a call also lists the evaluations stored for it. Each one shows a result (`pass`, `fail` or `needs_review`) and, when the Judge gives one, how sure it was. `needs_review` means the Judge's confidence was below 0.85. It is counted apart and never counts as a pass. A result the dashboard cannot read is shown as "Result not readable". It is never shown as a pass. If no evaluations were stored, the tab says so.

## Percentiles and stages

The **Reliability** tab shows p50, p90, p95 and p99 for one agent version. A percentile needs **30 samples**. Below that it is `null`, and the tab shows "Not enough data" with the sample count.

A table below the tiles shows the same percentiles for each of nine stages: speech to text, pause hold, model first word, model rest, voice, queue, knowledge search, tools and other. A stage that a turn did not measure adds no sample, so stages can have different sample counts. The rule for each percentile is the value at rank `ceil(n × q)` of the sorted turns, with no interpolation. A simulated call from an overloaded worker is left out of every latency number but still counts for accuracy.

## Run a simulation## Run a simulation

1. Open the agent, choose **Reliability**, then **Start a simulation**. Or call [Start simulation](https://docs.60db.ai/api-reference/agents/start-simulation.md).
2. Choose **Generate stories** (the agent's model writes 9 to 50 stories) or **Write a story**.
3. Set **repeats**, the **estimated credits per call** and the **credit ceiling**. The dialog asks for a free [estimate](https://docs.60db.ai/api-reference/agents/estimate-simulation.md) and shows the number of calls, the credits, and how many calls your ceilings let start, before you confirm.
4. Watch the run. Read it every 3 to 5 seconds until its state is `finished`, `cancelled` or `stopped_budget`. You can close the browser. The run goes on, and you can read the result later.

### Limits

| Limit | Value |
| - | - |
| Calls in one request (stories × repeats) | 1,000 |
| Active runs in one workspace | 3 |
| `max_concurrency` | 1 to 20 |
| Simulation starts for each user | 5 a minute |
| Golden runs for each user | 3 a minute |
| Roles that can start a run | owner, admin, developer. Pause, resume and cancel also work for members |

A start is refused if the balance cannot cover the lower of the credit ceiling and the estimate × the number of calls. A run that reaches its ceiling ends in the state `stopped_budget` with a partial report.

### Write a story

A story is one JSON object:

```json theme={null}
{
  "name": "caller-gives-account-number",
  "expect": "The agent reads the account number back correctly",
  "persona": "A calm customer who wants a balance check",
  "line": { "sample_rate": 8000 },
  "steps": [{ "step": "ai" }, { "step": "listen" }, { "step": "hang_up" }],
  "checks": [
    { "check": { "check": "contains", "arg": "balance" }, "must_pass": true },
    { "check": { "check": "tool_not_called", "arg": "close_account" }, "must_pass": true }
  ],
  "entities": { "account_number": "4829 1156" },
  "tool_answers": { "get_balance": "{\"balance\": 1200}" }
}
```

Steps are `say`, `now`, `quiet`, `cut_in`, `burst`, `breath`, `beep`, `ai`, `listen` and `hang_up`. Checks are `contains`, `not_contains`, `tool_called`, `tool_not_called`, `entity` and `task_success`. Each check item is `{ "check": { "check": "<type>", "arg": … }, "must_pass": true|false }`. The story **must** answer every tool the agent has in `tool_answers`, or the start answers `400` with `field: "tool_answers"`.

60db passes `steps` and `checks` on as they are. The voice service checks them. If a step or a check is wrong, you get `400` with the `field` name. A check with `must_pass: true` fails the call on one miss.

## Read a run

[Get simulation](https://docs.60db.ai/api-reference/agents/get-simulation.md) gives:

* progress: `total`, `done`, `queued`, `claimed`
* the `pass_rate` (`null` until a call is judged) and `incomplete`
* a pass rate for each scenario, with a **flaky** mark when results change between repeats
* **failure groups**, for example `failed_check:contains` or `stalled_at:tool`
* **regressions** against a baseline run
* one entry for each call, with its label and summary

## Fix and re-run

1. [Create a draft](https://docs.60db.ai/api-reference/agents/create-simulation-draft.md) from a finished run. The draft is **unpublished**.
2. [Edit the draft script](https://docs.60db.ai/api-reference/agents/update-simulation-draft.md). The limit is 32,000 characters.
3. [Re-run](https://docs.60db.ai/api-reference/agents/rerun-simulation.md) the same stories against the draft. Set a new credit ceiling. The new run is compared with the first run.
4. If the draft is better, save the agent yourself. 60db **never publishes** an agent from a simulation.

To run one call again with its stored caller audio, use [Rerun simulation call](https://docs.60db.ai/api-reference/agents/rerun-simulation-call.md). If no audio is stored, the answer is `404`.

## Compare versions

[Compare agent versions](https://docs.60db.ai/api-reference/agents/compare-agent-versions.md) gives the rate for each version, the change, and `significant`. It also gives p50, p90, p95 and p99 for both versions and the change (b minus a). `significant` is `true` when the two 95% intervals do not overlap, `false` when they do, and `null` when either side has fewer than 30 judged calls. Read the sample counts next to each rate.

## Judge quality

Choose the eval to check from the agent's [list of evals](https://docs.60db.ai/api-reference/agents/get-agent-evals.md). It shows which evals score calls and how each golden check stands. An eval marked unreliable has too many false passes. You can also type a name. [Run golden eval](https://docs.60db.ai/api-reference/agents/run-golden-eval.md) measures how closely a Judge eval agrees with answers that people gave. It makes one paid Judge request for each example. With fewer than 10 examples it says nothing. `unreliable: true` means the Judge disagrees too often to trust its scores.

## Permissions

| Action | Who |
| - | - |
| Read metrics and call results | Every role that can read agents, on any plan |
| Read simulation runs, estimate a run | Every role that can read agents, with an active Pro or Business plan |
| Start a run, make or edit a draft, re-run, ask for a fix, golden run | owner, admin, developer, with an active Pro or Business plan |
| Resume a paused run | owner, admin, developer, member, with an active Pro or Business plan |
| Pause, cancel | owner, admin, developer, member. Works on any plan, so a run can be stopped after the plan ends |

AI Simulation is a Pro feature: on Free, Starter or Creator, the simulation endpoints return `403 POLICY_DENY`. An API key needs the `agents` scope.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.