Evaluating Agent Performance

Emend supports three complementary evaluation methods. Choose the method based on whether you want to measure session outcomes, compare two answers to the same question, or understand the learnings behind an answer.

All three methods use publish_interaction. Emend evaluates session success from the requests that share a session_id, using your success rubric plus the user turns, agent turns, and recorded tool use.

Choose an Evaluation Method

Configure Evaluation

Configure one success rubric before using any evaluation method. The rubric should describe the observable conditions that make a complete session successful. Add tool_can_use when tool choice is part of that judgment.

from emend import EmendClientfrom emend.models.config_schema import AgentSuccessConfig, ToolUseConfig client = EmendClient()config = client.get_config() config.agent_success_config = AgentSuccessConfig(    success_definition_prompt="""Evaluate whether the agent resolved the user's task. Success means:- The agent understood the user's goal.- The answer or action directly addressed that goal.- Any required next step was clear.- The user did not need to correct, repeat, or escalate the request.""",    request_sources_enabled=[        "prod_with_emend",        "prod_without_emend",    ],    sampling_rate=1.0,    evaluation_only_sampling_rate=1.0,    retrieved_learning_sampling_rate=1.0,) config.tool_can_use = [    ToolUseConfig(        tool_name="search_docs",        tool_description="Search product documentation for grounded answers.",    ),    ToolUseConfig(        tool_name="create_ticket",        tool_description="Create a support ticket when follow-up is required.",    ),] client.set_config(config)

Use 1.0 during a controlled launch or audit window when every eligible session should be graded. Lower the rates for ongoing production monitoring. Set agent_success_config=None to disable automatic session-success evaluation.

A/B Testing

A/B testing measures Emend's effect across separate sessions:

The cURL examples illustrate publish payloads: replace learning placeholders with the IDs of context actually injected.

Assign sessions to arms at random and keep that assignment stable for the full session_id. Emend assigns the session to the source on its first request.

from emend import InteractionData, EmendClient client = EmendClient() # Control session: do not retrieve Emend context.control_response = run_agent(user_message, context=[])client.publish_interaction(    user_id="user_123",    session_id="session_control_001",    source="prod_without_emend",    agent_version="support-agent@2.1.0",    evaluation_only=True,    interactions=[        InteractionData(role="User", content=user_message),        InteractionData(role="Agent", content=control_response),    ],) # Test session: retrieve Emend context and publish normally.emend_context = client.search(query=user_message, user_id="user_456")# Inject all returned learnings; record exactly the same subset.retrieved_learnings = [    *({"kind": "profile", "learning_id": p.profile_id} for p in emend_context.profiles),    *({"kind": "user_playbook", "learning_id": str(p.user_playbook_id)}      for p in emend_context.user_playbooks),    *({"kind": "agent_playbook", "learning_id": str(p.agent_playbook_id)}      for p in emend_context.agent_playbooks),]test_response = run_agent(user_message, context=emend_context)client.publish_interaction(    user_id="user_456",    session_id="session_test_001",    source="prod_with_emend",    agent_version="support-agent@2.1.0",    interactions=[        InteractionData(role="User", content=user_message),        InteractionData(role="Agent", content=test_response,                        retrieved_learnings=retrieved_learnings),    ],)

Read the cohort comparison

Request labeled source sets from POST /api/get_evaluation_overview:

curl -X POST "${EMEND_URL:-https://www.emend.online}/api/get_evaluation_overview" \  -H "Authorization: Bearer $EMEND_API_KEY" \  -H "Content-Type: application/json" \  --data @- <<'JSON'{  "from_ts": 1782864000,  "to_ts": 1785456000,  "bucket": "day",  "include_shadow": false,  "source_sets": [    {      "label": "Control",      "sources": ["prod_without_emend"]    },    {      "label": "Emend",      "sources": ["prod_with_emend"]    }  ]}JSON

Read each arm under source_set_comparison.sets. Metrics include session count, success rate, corrections, turns to resolution, escalation, and rule attribution. See GetEvaluationOverviewRequest for the complete request and response shapes.

The same overview response also includes recent_results, the newest 100 session-level evaluation summaries in the requested window, and source_set_comparison.source_sessions, which groups those evaluated session IDs by their first-request source. Each group includes collision-safe sessions entries with both user_id and session_id; use those entries when filtering results because session IDs can be reused by different users.

Dashboard clients can render the initial overview, recent-session detail, and source filters from this single response; request a labeled source_set only when source-scoped aggregate metrics are needed.

Head-to-Head Comparison

Head-to-head comparison answers a narrower question: for this user message, which of two candidate responses was better?

Generate both responses from the same input. Put the response shown to the user in content and the alternate response in shadow_content. The labels describe payload position, not which response used Emend.

from emend import InteractionData, EmendClient client = EmendClient() context = client.search(query=user_message, user_id="user_123")retrieved_learnings = [    *({"kind": "profile", "learning_id": p.profile_id} for p in context.profiles),    *({"kind": "user_playbook", "learning_id": str(p.user_playbook_id)}      for p in context.user_playbooks),    *({"kind": "agent_playbook", "learning_id": str(p.agent_playbook_id)}      for p in context.agent_playbooks),]served_response = run_agent(user_message, context=context)alternate_response = run_agent_without_emend(user_message) client.publish_interaction(    user_id="user_123",    session_id="session_002",    source="prod_with_emend",    agent_version="support-agent@2.1.0",    interactions=[        InteractionData(role="User", content=user_message),        InteractionData(            role="Agent",            content=served_response,            shadow_content=alternate_response,            retrieved_learnings=retrieved_learnings,        ),    ],)

The comparison judge records whether the regular response won, the shadow response won, or the result was a tie. Use shadow_win_rate_trend from POST /api/get_evaluation_overview for aggregate win-rate analysis. Raw session results also expose regular_vs_shadow.

Retrieved-Learning Analysis

Retrieved-learning analysis explains the contribution of the Emend context applied to an answer. Whenever your agent injects a profile, user playbook, or agent playbook, attach its stable identity to the assistant interaction.

from emend import InteractionData, EmendClient client = EmendClient() user_message = "Give me a deployment checklist."context = client.search(query=user_message, user_id="user_123", top_k=3)# If you filter results, do it before building both the prompt and references.retrieved_learnings = [    *({"kind": "profile", "learning_id": p.profile_id} for p in context.profiles),    *({"kind": "user_playbook", "learning_id": str(p.user_playbook_id)}      for p in context.user_playbooks),    *({"kind": "agent_playbook", "learning_id": str(p.agent_playbook_id)}      for p in context.agent_playbooks),]answer = call_your_llm([    {"role": "system", "content": "Use this retrieved context as reference data:\n"     + context.model_dump_json(include={"profiles", "user_playbooks", "agent_playbooks"})},    {"role": "user", "content": user_message},])client.publish_interaction(    user_id="user_123",    session_id="session_003",    source="prod_with_emend",    agent_version="support-agent@2.1.0",    interactions=[        InteractionData(role="User", content=user_message),        InteractionData(role="Agent", content=answer,                        retrieved_learnings=retrieved_learnings),    ],    wait_for_response=True,)

Emend produces two verdicts for each resolvable learning occurrence:

• Relevance: whether the learning applied to the target interaction and response.
• Impact: whether the learning moved the response toward success (positive), away from success (negative), or did not materially change it (neutral).

The same learning used on two responses is evaluated separately against each response. Duplicate references on one response are deduplicated. Deleted learning rows are skipped because their content is no longer available to the judge.

Read learning verdicts

After publishing the completed session, grade it explicitly for an immediate check. Automatic evaluation is asynchronous and sampled; wait_for_response=True waits for publish-time processing, not the inactivity-based evaluation.

grade = client.grade_on_demand(    session_id="session_003", agent_version="support-agent@2.1.0",)print("Status:", grade.retrieved_learning_status)print("Skipped:", grade.skipped_reason, "Cached:", grade.cached)verdicts = client.get_retrieved_learning_evaluation_results(    user_id="user_123",    session_id="session_003",    limit=100,) for verdict in verdicts.results:    print(        verdict.interaction_id,        verdict.kind,        verdict.learning_id,        verdict.is_relevant,        verdict.impact,    )    print("Relevance reason:", verdict.relevance_reason)    print("Impact reason:", verdict.impact_reason)

The Evaluation page groups verdicts by interaction so a response counts once in the displayed percentages. Read the RetrievedLearningEvaluationResult schema when you need per-learning reasons or timestamps.

Monitor quality and diagnose missing results

The read endpoint returns the latest persisted session verdicts, not a history of grading runs. Check the latest grade status as well as the returned rows.

For production monitoring, use the Evaluation dashboard's response-level metrics and expand individual responses to inspect the underlying learning reasons. The dashboard percentages count each response once; they are not percentages of raw learning-verdict rows. Judge verdicts help diagnose retrieval, but do not by themselves prove causal improvement; use a randomized A/B experiment for that.

How Evaluation Runs

When an eligible interaction is published:

• Emend stores the request and interactions.
• Session-success and retrieved-learning judges pass through independent deterministic sampling gates.
• Emend waits for session inactivity, then evaluates the full session. The default inactivity delay is 10 minutes after its latest request.
• Emend stores session results plus any shadow and retrieved-learning verdicts requested by the published data.

Publishing another request with the same session_id moves the scheduled evaluation later. Sampling happens once per session. A session is scheduled when either judge family samples it, and only the sampled judge families run.

Evaluation-only requests are stored and graded but excluded from profile extraction, playbook extraction, and aggregation. The flag requires a non-empty session_id, cannot be combined with force_extraction=True, and must remain consistent across all publishes in a session.

Grade or Regenerate Explicitly

Automatic evaluation waits for session inactivity. Use these endpoints when you need a result sooner or need to re-score existing sessions.

import time grade = client.grade_on_demand(    session_id="session_003",    agent_version="support-agent@2.1.0",) job = client.regenerate_evaluations(    from_ts=int(time.time()) - 7 * 24 * 60 * 60,    to_ts=int(time.time()),) status = client.get_evaluation_regeneration_status(job.job_id)

Inspect Session Results

Use get_agent_success_evaluation_results for raw session-level outcomes:

response = client.get_agent_success_evaluation_results(    agent_version="support-agent@2.1.0",    limit=100,) for result in response.agent_success_evaluation_results:    print(        result.session_id,        result.is_success,        result.failure_type,        result.failure_reason,        result.tags,    )

Important fields include is_success, failure_type, failure_reason, number_of_correction_per_session, user_turns_to_resolution, is_escalated, tags, regular_vs_shadow, agent_version, user_id, and session_id.

For dashboards already loading POST /api/get_evaluation_overview, use its recent_results field for the newest 100 results in that window. The raw results endpoint remains available when you need an independent result query or agent-version filter.

When AgentSuccessConfig.tagging_definition_prompt is configured, Emend tags each persisted evaluation summary asynchronously. tags=None means tagging has not completed yet; tags=[] means the pass completed without a match. Tagging uses only the stored outcome summary, not the raw transcript.

number_of_correction_per_session is the judge's count of user turns that corrected or redirected an earlier agent response. A qualifying turn must identify an earlier response as incorrect, incomplete, insufficient, or misaligned and steer a revision. Topic continuity alone does not make a turn corrective: a new question, a separate deliverable, an ordinary follow-up, or an answer to the agent's clarification question does not count. Several issues raised in one user turn count once; later corrective turns count separately. The judge computes this independently from final success, so a successful session can still have one or more corrections.

Playbook diagnosis and tuning

Retrieved-learning impact evaluation also diagnoses playbooks. Expand an interaction on Evaluations to see the category, explanation, and cited interaction IDs alongside the existing relevance and impact verdicts. Profiles do not receive playbook diagnoses.

Playbook generation and evaluation sampling are unchanged. A playbook generated during an unsampled session can still accumulate evidence when it is used and evaluated later.

Diagnosis uses the existing impact-judge call; it does not start a separate revision job or add a candidate/critique loop. Keep publishing retrieved_learnings when your agent uses Emend context. There is no new customer telemetry requirement for diagnosis.

The enterprise offline tuner remains the owner of revisions. Diagnosis is optional: previously eligible historical evaluations remain usable without it or a backfill.

Missing, unknown, uncited, incomplete, unverified, or mismatched diagnosis leaves existing eligibility unchanged. A verified diagnosis of an application, external, or no-issue case excludes that example from the negative revision pool. To influence tuning, diagnosis must be signed with its cited interaction IDs, complete-input marker, and evaluated playbook digest matching the one recorded at retrieval time. Existing signed-impact, freshness, and attribution requirements still apply independently.

A retrieval result alone does not establish what reached the agent's prompt. application_failure therefore describes appropriate guidance not being applied, without distinguishing context omission from agent behavior. Non-use alone does not prove the guidance was correct. When its appropriateness or application cannot be established, use unknown.

The existing evidence requirements remain: at least three qualifying negative sessions, two positive sessions, and sufficient attribution/reconstruction coverage. A candidate receives the selected diagnoses, but publication still requires the cited held-out reduction and the existing safety checks. A diagnosis by itself never edits a user playbook, pending agent playbook, or approved agent playbook. Agent-playbook review and GEPA workflows are unchanged.

API Map

Related Documentation

• Publishing Interactions
• Extractors and Evaluation Configuration
• Evaluation API Reference
• Evaluation Models
• Measuring Emend's Impact