API Reference

Agent Evaluation

Methods for retrieving, regenerating, and grading agent success evaluation results.

get_agent_success_evaluation_resultsmethod

Get session-level agent success evaluation results to analyze performance. For the end-to-end workflow covering configuration, automatic scheduling, evaluation_only=True, source-set comparison, shadow responses, and explicit regrading, see Evaluating Agent Performance.

response = client.get_agent_success_evaluation_results(    agent_version=None,    limit=100)

Prop	Type
`agent_version`	string
`limit`	integer
`start_time`	datetime
`end_time`	datetime

Returns GetAgentSuccessEvaluationResultsResponse, containing AgentSuccessEvaluationResult objects — see Evaluation Models.

Example

evaluation-results.py

# Get all resultsresponse = client.get_agent_success_evaluation_results(limit=100) # Calculate success ratetotal = len(response.agent_success_evaluation_results)successful = sum(1 for r in response.agent_success_evaluation_results if r.is_success)print(f"Success rate: {successful/total*100:.1f}%") # Review failuresprint("\nFailure Analysis:")for result in response.agent_success_evaluation_results:    if not result.is_success:        print(f"Session: {result.session_id}")        print(f"  Failure Type: {result.failure_type}")        print(f"  Reason: {result.failure_reason}") # Compare agent versionsv1_results = client.get_agent_success_evaluation_results(    agent_version="v1.0.0",    limit=100)v2_results = client.get_agent_success_evaluation_results(    agent_version="v2.0.0",    limit=100) v1_success = sum(1 for r in v1_results.agent_success_evaluation_results if r.is_success)v2_success = sum(1 for r in v2_results.agent_success_evaluation_results if r.is_success) print(f"\nVersion Comparison:")print(f"v1.0.0: {v1_success}/{len(v1_results.agent_success_evaluation_results)} successful")print(f"v2.0.0: {v2_success}/{len(v2_results.agent_success_evaluation_results)} successful") # Group failures by typefrom collections import Counterfailure_types = Counter(    r.failure_type for r in response.agent_success_evaluation_results if not r.is_success)print("\nFailure Types:")for failure_type, count in failure_types.most_common():    print(f"  {failure_type}: {count}")

regenerate_evaluationsmethod

Start an asynchronous replay-the-judge job over a Unix timestamp window. Use this after changing the success rubric, evaluator model, or prompt and needing historical sessions re-scored.

import time from emend import EmendClient client = EmendClient() job = client.regenerate_evaluations(    from_ts=int(time.time()) - 7 * 24 * 60 * 60,    to_ts=int(time.time()),) status = client.get_evaluation_regeneration_status(job.job_id) # Optional: request cancellation while the job is running.# client.cancel_evaluation_regeneration(job.job_id)

Prop	Type
`from_ts`required	integer
`to_ts`required	integer
`evaluation_name`	string

get_evaluation_regeneration_statusmethod

Return the current status and progress for a regeneration job ID.

cancel_evaluation_regenerationmethod

Request cancellation for a running regeneration job. Work already in progress may finish before the job stops.

grade_on_demandmethod

Grade one session synchronously without waiting for the normal session inactivity delay. Results are cached for 24 hours per session and agent version.

from emend import EmendClient client = EmendClient() result = client.grade_on_demand(    session_id="session_001",    agent_version="v2.1.0",) if result.skipped_reason:    print(f"Skipped: {result.skipped_reason}")else:    print(f"Result id: {result.result_id}, cached: {result.cached}")

Prop	Type
`session_id`required	string
`agent_version`required	string
`evaluation_name`	string

get_retrieved_learning_evaluation_resultsmethod

Read the latest per-learning relevance and impact verdicts produced for a graded session.

retrieved-learning-verdicts.py

verdicts = client.get_retrieved_learning_evaluation_results(    user_id="user_001",    session_id="session_001",    start_time=window_start,    end_time=window_end,    limit=100,) for verdict in verdicts.results:    print(        verdict.interaction_id,        verdict.kind,        verdict.learning_id,        verdict.is_relevant,        verdict.impact,    )

Prop	Type
`user_id`	string
`session_id`	string
`start_time`	datetime
`end_time`	datetime
`limit`	integer

Publish the stable identities of every profile or playbook injected into agent turns through retrieved_learnings. This enables the detailed per-learning verdicts returned here and provides attribution data for future retrieved-learning optimization. Call grade_on_demand before this read when an immediate demo result is needed, and inspect its retrieved_learning_status, skipped_reason, and cached fields. This read does not trigger grading. See the complete grade-and-read example in Evaluating Agent Performance for reasons, missing-result diagnostics, and sampling guidance.

How verdicts roll up per response

Each result remains a judge verdict for one learning, but it is attributed to the specific response interaction that used that learning. The same learning used on two responses therefore produces two independently targeted verdicts. The Evaluation page groups these rows by interaction so each response counts once in the displayed metrics:

relevance is positive when at least one graded learning is relevant;
impact is positive when at least one learning is positive and none is negative, negative when at least one is negative and none is positive, mixed when both occur, and neutral when every graded impact is neutral;
null verdicts are ignored when another learning on the response was graded, while an all-null response is reported as ungraded.

PreviousRequest Management Next Retrieval Experiments