Judge

Overview

60db’s Judge answers questions you define about a piece of content, and returns a probability distribution for every answer. It never writes prose. That is the whole point. Ask a language model “is this caller angry?” and you get a paragraph you then have to parse. Ask Judge and you get 0.86 on dismissive — a number you can sort by, threshold on, and compare across a hundred thousand runs.

Your Labels, Not Ours

You supply the options and the descriptions; the model only picks between them

Comparable Output

Every answer is a distribution over a fixed answer space, so runs sort and diff

Up to 32 Questions

Each asked independently against the same content, in one request

Saved Rubrics

Store a question set once, then run it by rubric_id forever

Confidence & Review Queue

Below 0.85, route it to a person — filter history on needs_review

Turn Extraction

Intent, operation and entity spans from a single conversational turn

Your existing 60db API key already works. Judge wraps the 60db Jev model behind the same platform credential as TTS, STT and Memory — there is no separate Judge key to issue. Keys created from now on carry a judge scope; keys issued earlier are accepted on their slm scope.

Core concepts

Questions, not prompts

A Judge request has two halves: the state (the content — a transcript, a ticket, a document, any JSON structure), and the questions asked about it. Each question is keyed by a name of your choosing, which is how you read the answer back. The key is never sent to the model. Questions are asked independently. Every one sees the same state and its own criteria, and nothing else — they are not a conversation, and question 3 cannot see what question 1 answered.

The descriptions are what the model reads — they are the thing to tune, not a prompt. A label with a vague description produces a vague answer.

The three question types

Type	Question it answers	You supply	You get back
`choice`	Which one of these?	Map of option name → description, max 255	`choice`, `probabilities` over every option, `confidence`
`score`	Where on this ladder?	Ordered array, lowest first, max 10 levels	`score` (weighted position, e.g. `1.87`), `legend`, `probabilities`, `confidence`
`noul`	True or false?	Optional `{ "true": "...", "false": "..." }`	`noul` — a raw probability of true

Always include an unknown option on a choice. Without an escape hatch the model must pick one of your real labels even on off-topic content.

Two results are easy to misread:

score is a weighted position, not an index. 1.87 on a four-rung ladder means the mass sits between rungs 1 and 2 — it is not “rung 1” rounded.
noul has deliberately no confidence field. The number is the probability of true; it is not a calibrated correctness estimate.

Confidence and escalation

Every run reports min_confidence — the lowest confidence across its answers (null when every question was noul). Below 0.85, a human should look. That is the service’s own escalation threshold, and it is the filter the review queue is built on. escalated_questions tells you how many answers the service already refined with its second-stage model because confidence was low.

Evaluating content

JavaScript
Python
cURL
CLI

import { SixtyDBClient } from '60db-service';

const client = new SixtyDBClient('your-api-key');

const run = await client.judge.evaluate({
  state: callTranscript,
  questions: {
    tone: {
      type: 'choice',
      instructions: 'How did the agent come across?',
      criteria: {
        professional: 'Calm, courteous, takes ownership',
        dismissive: 'Brushes the caller off',
        unknown: 'Not enough of the call to tell',   // always add an escape hatch
      },
    },
    satisfaction: { type: 'score', criteria: ['Angry', 'Neutral', 'Happy'] },
    resolved: { type: 'noul', instructions: 'Was it solved?' },
  },
  label: 'support-call-qa',   // groups runs in history
});

run.answers.tone.choice           // 'dismissive'
run.answers.tone.probabilities    // { dismissive: 0.86, professional: 0.11, unknown: 0.03 }
run.answers.tone.confidence       // 0.79
run.answers.satisfaction.score    // 1.87 — weighted position, not an index
run.answers.resolved.noul         // 0.28 — probability of true, NOT a confidence
run.min_confidence                // under 0.85 → send it to a person
run.credits_charged               // 0.00000161

from sixtydb import SixtyDBClient

client = SixtyDBClient("your-api-key")

run = client.judge.evaluate(
    state=call_transcript,
    questions={
        "tone": {
            "type": "choice",
            "instructions": "How did the agent come across?",
            "criteria": {
                "professional": "Calm, courteous, takes ownership",
                "dismissive": "Brushes the caller off",
                "unknown": "Not enough of the call to tell",   # always add an escape hatch
            },
        },
        "satisfaction": {"type": "score", "criteria": ["Angry", "Neutral", "Happy"]},
        "resolved": {"type": "noul", "instructions": "Was it solved?"},
    },
    label="support-call-qa",
)

run["answers"]["tone"]["choice"]          # 'dismissive'
run["answers"]["tone"]["probabilities"]   # {'dismissive': 0.86, 'professional': 0.11, ...}
run["answers"]["satisfaction"]["score"]   # 1.87 — weighted, not an index
run["answers"]["resolved"]["noul"]        # 0.28 — probability of true, not a confidence
run["min_confidence"]                     # under 0.85 → send to a human
run["credits_charged"]

curl -X POST https://api.60db.ai/judge/evaluate \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Agent: I can'\''t refund that, it'\''s outside the window.\nCaller: This is the third time I'\''ve called about this.",
    "questions": {
      "tone": {
        "type": "choice",
        "instructions": "How did the agent come across?",
        "criteria": {
          "professional": "Calm, courteous, takes ownership",
          "dismissive": "Brushes the caller off",
          "unknown": "Not enough of the call to tell"
        }
      },
      "satisfaction": {
        "type": "score",
        "criteria": ["Angry", "Unhappy", "Neutral", "Satisfied"]
      },
      "resolved": { "type": "noul", "instructions": "Was the problem solved?" }
    },
    "label": "support-call-qa"
  }'

# rubric.json holds the questions map
60db judge:evaluate --rubric-file rubric.json --state-file call.txt

60db judge:models   # authoritative model list, plus a starter rubric to copy

Alongside answers, every run returns a summary — one headline value per question (the chosen option, the score, or the probability). That is the shape to store on your own record when you do not need the full distribution.

Saved rubrics

Save a question set once and run it by id from then on. Runs of a saved rubric file themselves under it, so listing by rubric_id returns the whole series — which is what makes a rubric a trend line rather than a one-off.

JavaScript
Python
CLI

const rubric = await client.judge.rubrics.create({
  name: 'Call QA',                 // unique per workspace
  description: 'Tone, satisfaction and outcome on inbound calls',
  questions,
  shared: true,                    // workspace-wide; owner/admin only
});

await client.judge.evaluate({ state: transcript, rubric_id: rubric.id });

await client.judge.rubrics.list();
await client.judge.rubrics.get(id);
await client.judge.rubrics.update(id, { description: 'updated' });
await client.judge.rubrics.delete(id);   // past runs stay readable

rubric = client.judge.rubrics.create(
    name="Call QA",                 # unique per workspace
    description="Tone, satisfaction and outcome on inbound calls",
    questions=questions,
    shared=True,                    # workspace-wide; owner/admin only
)

client.judge.evaluate(state=transcript, rubric_id=rubric["id"])

client.judge.rubrics.list()
client.judge.rubrics.get(rubric_id)
client.judge.rubrics.update(rubric_id, description="updated")
client.judge.rubrics.delete(rubric_id)   # past runs stay readable

60db judge:create-rubric --name "Call QA" --file rubric.json
60db judge:evaluate --rubric 8b0da171-... --state-file call.txt

60db judge:rubrics                    # list
60db judge:delete-rubric --id <id>    # delete (past runs stay readable)

Pass either questions or rubric_id, never both. Saved rubrics are re-validated on every run — a rubric is not trusted just because it was valid when stored.

Extracting from a turn

extract is the understanding layer between speech-to-text and a reply: what did the speaker just ask for, and what values did they give me. It labels one turn with an intent and an operation, and pulls entity spans out with their exact positions.

JavaScript
Python
CLI

const turn = await client.judge.extract({
  text: 'move my 10am appointment to friday',
  schema: {
    intents:    { booking: 'Wants to arrange or change a booking' },
    operations: { reschedule: 'Move an existing booking' },
    entities:   { date: 'A calendar date', time: 'A clock time' },
    // responsePaths: { confirm: 'Read the change back' }   ← selects the v2 model
  },
});

turn.intent.label       // 'booking'
turn.operation.label    // 'reschedule'
turn.entities           // [{ label: 'date', text: 'friday', start: 28, end: 34, confidence: 0.8 }]

turn = client.judge.extract(
    text="move my 10am appointment to friday",
    schema={
        "intents": {"booking": "Wants to arrange or change a booking"},
        "operations": {"reschedule": "Move an existing booking"},
        "entities": {"date": "A calendar date", "time": "A clock time"},
        # "responsePaths": {"confirm": "Read the change back"}   ← selects the v2 model
    },
)

turn["intent"]["label"]      # 'booking'
turn["entities"]             # [{'label': 'date', 'text': 'friday', 'start': 28, 'end': 34, ...}]

60db judge:extract --text "move my 10am appointment to friday" --schema-file schema.json

start/end are Unicode code point offsets, not UTF-16 indices. Slice with Array.from(text).slice(start, end).join('') in JavaScript, or "".join(list(text)[start:end]) in Python — the naive text.slice() / text[start:end] lands in the wrong place once the turn contains an emoji.

Supplying schema.responsePaths selects the v2 model, which also returns a response_path and tracks conversation state. text is capped at 2,048 Unicode code points; the required unknown label is added for you if you leave it out of a classification map.

The review queue

Rather than re-reading everything the model was already sure about, list only what it wasn’t:

JavaScript
Python
CLI

const { runs, pagination } = await client.judge.runs.list({
  needs_review: true,     // only runs whose min_confidence fell below 0.85
  limit: 20,
});

await client.judge.runs.get(id);      // one run, with what was sent and returned
await client.judge.runs.delete(id);

page = client.judge.runs.list(needs_review=True, limit=20)
page["runs"]         # only runs the judge was under 85% sure about
page["pagination"]   # {'total': ..., 'limit': ..., 'has_more': ...}

client.judge.runs.get(run_id)      # one run, with what was sent and returned
client.judge.runs.delete(run_id)

60db judge:runs --needs-review
60db judge:run --id <id>          # one run in full, with what was sent and returned
60db judge:delete-run --id <id>   # remove it from history

History also filters by kind (evaluate or extract), by rubric_id for one rubric’s whole series, and by your own confidence_below threshold. Pass save: false on a run to bill it but keep it out of history entirely.

Models

Fetch the model list; don’t hardcode it. jev-latest is an alias that resolves server-side to a versioned id, and that versioned id is what the evaluate response reports back as model. Responses are cached for 5 minutes, so polling to populate a picker is free.

await client.judge.models();                // authoritative list — don't hardcode
await client.judge.usage('current_month');  // spend and run counts
await client.judge.health();                // upstream readiness (owner/admin)

Pricing

Judge is pay-as-you-go on input tokens only —

0.010 per 1M. Output is structurally `

0`, because a classifier emits no text. It bills to the same workspace wallet as every other 60db service. The one thing that will surprise you:

input_tokens ≈ questions × content + every question's own text

The service builds one row per question, and every row carries the shared state again. Ten questions about a 20,000-character document encode that document ten times — so keep rubrics tight when the content is long. Context is 8K tokens per question, not per request; over-long content is rejected with a 400 before anything is charged.

Scenario	Tokens	Per 1,000 runs
Ticket triage — 2 questions, short ticket	~70	$0.0007
Call QA — 3 questions, 3-minute call	~1,500	$0.0150
Call QA — 3 questions, 15-minute call	~7,500	$0.0750
Deep audit — 10 questions, long document	~50,000	$0.5000
Extract — one conversational turn	~36	$0.0004

A failed run costs nothing. The wallet is debited before the call and refunded automatically if anything goes wrong — a busy queue (429), an unavailable service (503), a rejected rubric (400). Both entries appear in the ledger, netting to zero. See Judge pricing for the full reconciliation story.

Scoring every call rather than a 2% sample is the point of pricing like this. Ten thousand three-minute calls a day costs roughly $4.50/month.

Errors

Status	Meaning
`400`	Invalid rubric — the message names the offending question. Also returned over 32 KiB or the 8K-token context.
`402`	Insufficient credits; `details.shortfall` says how much is missing.
`403`	Your role cannot run the judge, an API key lacks the `judge` scope, or `shared: true` needs owner/admin.
`404`	Unknown `rubric_id`, or it belongs to another user.
`409`	A rubric with that name already exists in this workspace.
`429`	The inference queue is full — back off and retry.
`502` / `503`	The judge service is misconfigured or unavailable.

API Reference

Evaluate

Score content against a rubric

Extract

Intent, operation and entity spans

Rubrics

Save, list, update and delete

Runs

History and the review queue

Models

Model names this deployment serves

Usage

Spend and run counts

Pricing

Rates, token maths and refunds

CLI Commands

60db judge:* from the terminal

MCP Tools

Judge from Claude and other agents

Get Started

SDKs

Core Features

Overview

Your Labels, Not Ours

Comparable Output

Up to 32 Questions

Saved Rubrics

Confidence & Review Queue

Turn Extraction

Core concepts

Questions, not prompts

The three question types

Confidence and escalation

Evaluating content

Saved rubrics

Extracting from a turn

The review queue

Models

Pricing

Errors

API Reference

Evaluate

Extract

Rubrics

Runs

Models

Usage

Pricing

CLI Commands

MCP Tools

​Overview

Your Labels, Not Ours

Comparable Output

Up to 32 Questions

Saved Rubrics

Confidence & Review Queue

Turn Extraction

​Core concepts

​Questions, not prompts

​The three question types

​Confidence and escalation

​Evaluating content

​Saved rubrics

​Extracting from a turn

​The review queue

​Models

​Pricing

​Errors

​API Reference

Evaluate

Extract

Rubrics

Runs

Models

Usage

Pricing

CLI Commands

MCP Tools

Overview

Core concepts

Questions, not prompts

The three question types

Confidence and escalation

Evaluating content

Saved rubrics

Extracting from a turn

The review queue

Models

Pricing

Errors

API Reference