Schemas

Evaluation Models

Data structures for agent success evaluation results.

AgentSuccessEvaluationResultmodel

Represents an agent performance evaluation result.

Prop	Type
`result_id`	integer
`user_id`	string
`agent_version`required	string
`session_id`required	string
`is_success`required	boolean
`failure_type`	string
`failure_reason`	string
`evaluation_name`	string
`created_at`	integer
`regular_vs_shadow`	RegularVsShadow
`number_of_correction_per_session`	integer
`user_turns_to_resolution`	integer
`is_escalated`	boolean
`tags`	array[string] \| null

GetAgentSuccessEvaluationResultsRequestrequest model

Request model for getting agent success evaluation results.

Prop	Type
`agent_version`	string
`limit`	integer
`start_time`	datetime
`end_time`	datetime

GetAgentSuccessEvaluationResultsResponseresponse model

Response model for getting agent success evaluation results.

Prop	Type
`success`required	boolean
`agent_success_evaluation_results`required	array[AgentSuccessEvaluationResult]
`msg`	string

RetrievedLearningEvaluationResultmodel

The latest per-learning relevance/impact verdict for one target interaction. One row per (user_id, session_id, interaction_id, kind, learning_id); the stored set is the most recent successfully persisted evaluation for the session, not an append-only history. The same learning used on multiple interactions is evaluated separately against each response. Produced automatically when a session that published interactions with retrieved_learnings goes through group evaluation.

Prop	Type
`result_id`	integer
`user_id`required	string
`session_id`required	string
`agent_version`	string
`interaction_id`	integer \| null
`interaction_created_at`	integer \| null
`kind`required	"profile" \| "user_playbook" \| "agent_playbook"
`learning_id`required	string
`is_relevant`	boolean \| null
`relevance_reason`	string
`impact`	"positive" \| "negative" \| "neutral" \| null
`impact_reason`	string
`diagnosis`	PlaybookDiagnosis \| null
`evaluated_playbook_digest`	string \| null
`diagnosis_evidence_complete`	boolean
`created_at`	integer

PlaybookDiagnosismodel

Prop	Type	Description
`category`required	"content_defect" \| "application_failure" \| "external_failure" \| "no_issue" \| "unknown"
`reason`required	string	1–4,000 characters
`evidence_interaction_ids`	list[int]	At most 20 entries. Defaults to [].

Diagnosis explains the observed problem; it is not replay evidence or proof of improvement.

Missing or unknown diagnosis does not disqualify otherwise eligible historical evidence.

Out-of-session citations are discarded; only verified, supported diagnoses can influence tuning. See diagnosis and tuning.

GetRetrievedLearningEvaluationResultsRequestrequest model

Request model for POST /api/get_retrieved_learning_evaluation_results.

Prop	Type
`user_id`	string
`session_id`	string
`start_time`	datetime
`end_time`	datetime
`limit`	integer

GetRetrievedLearningEvaluationResultsResponseresponse model

Response model for POST /api/get_retrieved_learning_evaluation_results.

Prop	Type
`success`required	boolean
`results`required	array[RetrievedLearningEvaluationResult]
`msg`	string

EvaluationSourceSetRequestmodel

One labeled request-source cohort for evaluation comparison. Sources match Request.source exactly, including the empty string for requests published without a source.

Prop	Type
`label`required	string
`sources`required	array[string]

SourceSetEvaluationMetricsmodel

Metrics for one source set in SourceSetComparison.

Prop	Type
`label`required	string
`sources`required	array[string]
`session_count`required	integer
`session_ids`	array[string]
`success_rate_pp`required	float
`buckets`required	array[HeroBucket]
`context_tiles`required	ContextTile
`score_distribution`required	ScoreDistribution
`rule_attribution`required	array[RuleAttributionRow]
`braintrust_tiles`	array[BraintrustTileRow]

SourceSetComparisonmodel

Request-source comparison payload. Each evaluation result is assigned by joining to the session's first request and reading that request's source.

Prop	Type
`available_sources`	array[string]
`source_sessions`	array[{ source: string, session_ids: array[string], sessions: array[{ user_id: string, session_id: string }] }]
`sets`	array[SourceSetEvaluationMetrics]
`unmatched_session_count`	integer

GetEvaluationOverviewRequestrequest model

Input for the Evaluation Overview endpoint (POST /api/get_evaluation_overview).

Prop	Type
`from_ts`required	integer
`to_ts`required	integer
`bucket`	string
`include_shadow`	boolean
`source_sets`	array[EvaluationSourceSetRequest]

GetEvaluationOverviewResponseresponse model

Response model for the Evaluation Overview endpoint. The global fields populate the all-source hero, context, and rule-attribution sections. Source-set metrics are returned separately in source_set_comparison; the web portal uses those source-set metrics to scope the page when a user selects a single source.

Prop	Type
`hero`required	HeroBlock
`context_tiles`required	ContextTile
`rule_attribution`required	array[RuleAttributionRow]
`score_distribution`required	ScoreDistribution
`braintrust_tiles`	array[BraintrustTileRow]
`shadow_win_rate_trend`	ShadowWinRateTrend
`recent_results`	array[AgentSuccessEvaluationResult]
`source_set_comparison`	SourceSetComparison

ShadowComparisonOutputmodel

LLM judge verdict for a single per-turn Emend-vs-shadow comparison (F1). The position of the two responses shown to the judge is randomized per call to mitigate position bias; the mapping is recorded on ShadowComparisonVerdict.emend_is_request_1.

Prop	Type
`better_request`required	string
`is_significantly_better`required	boolean
`comparison_reason`	string

ShadowComparisonVerdictmodel

One per-turn comparison verdict, stored per (interaction_id, judge_prompt_version) in the shadow_comparison_verdicts table.

Prop	Type
`verdict_id`required	integer
`interaction_id`required	string
`session_id`required	string
`agent_version`required	string
`emend_is_request_1`required	boolean
`output`required	ShadowComparisonOutput
`judge_prompt_version`required	string
`created_at`required	datetime

ShadowWinRateTrendPointmodel

One daily bucket of per-turn shadow comparison verdicts (F1). Buckets are UTC-aligned and surfaced in ascending date order.

Prop	Type
`date`required	string
`n`required	integer
`wins`required	integer
`losses`required	integer
`ties`required	integer

ShadowWinRateTrendWindowTotalmodel

Aggregate of all shadow verdicts across the trend window (F1). Used to render the headline win-rate tile on the Evaluation page.

Prop	Type
`n`required	integer
`wins`required	integer
`losses`required	integer
`ties`required	integer
`win_rate`required	float
`net_win`required	float

ShadowWinRateTrendmodel

F1 shadow win-rate trend payload. Carried on GetEvaluationOverviewResponse.shadow_win_rate_trend.

Verdicts produced under a previous rubric epoch are filtered out at storage time by judge_prompt_version, so the dashboard never silently mixes incompatible rubrics into the headline number.

Prop	Type
`daily`	array[ShadowWinRateTrendPoint]
`window_total`	ShadowWinRateTrendWindowTotal
`judge_prompt_version`	string

GetRecentShadowComparisonsResponseresponse model

Returned by GET /api/evaluations/shadow_comparisons/recent.

Prop	Type
`verdicts`	array[ShadowComparisonVerdict]

GET /api/evaluations/shadow_comparisons/recentendpoint

Returns the N most recent per-turn shadow comparison verdicts (F1). Powers two surfaces on the Evaluation page:

The drawer triggered from the per-turn comparison tile — shows the N most recent verdicts so you can spot-check the judge.
The “Top 10 disagreements” widget — fetches a wider pool and the frontend filters to is_significantly_better=True losses to surface actionable rule-correction candidates.

Verdicts are restricted to the org's currently pinned Config.shadow_comparison_judge_prompt_version so verdicts from an older rubric never mix into the drawer. A 30-day lookback is enforced server-side so the storage layer can use an index range scan instead of a full table read.

Query parameters

Prop	Type
`limit`	integer

Response

GetRecentShadowComparisonsResponse.

Errors

503 Storage not configured — the server has no storage backend wired.
Returns verdicts: [] (200) when the storage backend does not implement the shadow_comparison_verdicts feature (e.g. the disk backend), or when no verdicts in the 30-day window match the pinned prompt version.

RegenerateRequestrequest model

Input for POST /api/evaluations/regenerate — kicks off a replay-the-judge job over a closed window.

Prop	Type
`evaluation_name`	string
`from_ts`required	integer
`to_ts`required	integer

RegenerateStartResponseresponse model

Returned by POST /api/evaluations/regenerate.

Prop	Type
`job_id`required	string
`total`required	integer

RegenerateFailuremodel

One failed session in a regen job's failure list.

Prop	Type
`session_id`required	string
`reason`required	string

RegenerateStatusResponseresponse model

Returned by GET /api/evaluations/regenerate/{job_id}. Carries the live lifecycle state of the regen worker plus the F3 sampling and concurrency counters described in Measuring Emend's Impact.

Prop	Type
`job_id`required	string
`status`required	string
`total`required	integer
`completed`required	integer
`failed`required	integer
`failures`required	array[RegenerateFailure]
`started_at`required	float
`finished_at`	float
`total_candidates`	integer
`sampled_count`	integer
`concurrency_limit`	integer

GradeOnDemandRequestrequest model

Input for POST /api/evaluations/grade_on_demand — single-session click-through grading triggered when a dashboard user opens a session that wasn't in the sampled regen set. See Sampling and freshness.

Prop	Type
`session_id`required	string
`agent_version`required	string
`evaluation_name`	string

GradeOnDemandResponseresponse model

Returned by POST /api/evaluations/grade_on_demand. Results are cached for 24 hours via the operation-state mechanism — the second call for the same (session, agent_version) within the cache window will return cached: true with the original result_id.

Prop	Type
`session_id`required	string
`result_id`	integer
`cached`	boolean
`skipped_reason`	string
`retrieved_learning_status`	string

PreviousEnums Next Interaction Models