Measuring Emend's Impact

The Evaluation page reports quality at three levels:

• Session outcomes — whether evaluated conversations succeeded, needed corrections, took more turns, or escalated.
• Retrieved-learning effects — whether the profiles and playbooks supplied to a response were relevant and helped.
• Per-turn comparisons — whether a response using Emend beat an alternate response generated without retrieved context.

Start with the metric reference below to understand the numbers. The second half of this page covers integration, sampling, and the methodology required to make source-set comparisons meaningful.

How Filters and Windows Work

Where the source is read

Emend reads the source of the first request in each session and uses that source for the entire session — sticky assignment. If later requests in the same session use a different source, the comparison still uses the first request’s source.

Metric Reference

Session outcomes

Every session-outcome metric is calculated from stored agent-success evaluation results. The configured success rubric defines what the judge considers success. The session judge also independently counts corrective user turns and determines escalation.

Correction count and final success are independent: a session may require one or more corrections and still end successfully. Rows evaluated before the judge-derived count was introduced retain their earlier values until they are regenerated. Regenerate every historical window used for a comparison or experiment after upgrading; do not mix legacy and regenerated rows in one analysis.

Success-rate and escalation deltas are shown in percentage points (pp). Corrections and turns deltas are absolute differences. The Source-set comparison applies these same formulas after grouping evaluated sessions by their first-request source.

Retrieved-learning effects

This section appears when published responses include retrieved_learnings. Emend judges each attached profile or playbook separately, then groups those verdicts by response so a response with many learnings does not outweigh one with a single learning.

Collective impact uses these response-level categories:

How retrieved-learning evidence is bounded

Within a session, one verdict belongs to (interaction_id, kind, learning_id). Duplicate references on one response are deduplicated; the same learning used on another response is judged again. Null verdicts are ignored when another learning on that response was graded, while an all-null response is excluded from that judge family’s percentage. Legacy rows without interaction attribution are also excluded.

The dashboard reads at most the 1,000 most recent learning-verdict rows in the selected interaction-time range. When that page is full, it omits the final response because the boundary may contain only some of its learning verdicts.

To populate this section, follow the search → inject → publish example below, then grade and read verdicts.

Expand Interaction verdicts on the dashboard to inspect the per-learning relevance, impact, and judge reasons behind these response-level numbers.

Per-turn comparison

These metrics compare each served response that used Emend with a shadow response produced without retrieved context.

Learning attribution and imported scorers

Learning attribution is observational: a positive net does not prove that the learning caused the successful outcome. Use a retrieval experiment when you need a causal comparison.

Retrieval A/B Experiments

Retrieval experiments measure the effect of serving learned context while keeping the rest of the production pipeline the same. Emend deterministically assigns each user to one of two arms:

Start and monitor the experiment from Experiments in the web portal, or use the client lifecycle methods. There can be only one active experiment per organization, experiment IDs cannot be reused, and stopped experiments remain available in history.

The results scorecard also compares average output tokens per published session. Emend stores a canonical token count for each newly published interaction and sums all non-user roles, including assistant, system, and tool content. Sessions containing older non-user interactions without a stored count are excluded from that average, and the scorecard shows measured-session coverage for each arm.

client.start_retrieval_experiment(    experiment_id="support-agent-2026-08",    holdout_percentage=10,)

Search and publish attribution

Pass user_id to every learning-search request. Search does not require a session ID for assignment: the arm is stable for the combination of organization, experiment, and user. When an experiment is active, the response includes experiment_id and arm; a holdout response is successful but has no retrieved profiles or playbooks.

from emend import EmendClientfrom emend.models.api_schema.domain.entities import InteractionData client = EmendClient() search = client.search(query=user_message, user_id=user.id)agent_response = run_agent(    user_message,    profiles=search.profiles,    agent_playbooks=search.agent_playbooks,    user_playbooks=search.user_playbooks,) # Report all three lists passed to run_agent above.retrieved_learnings = [    *({"kind": "profile", "learning_id": p.profile_id} for p in search.profiles),    *({"kind": "user_playbook", "learning_id": str(p.user_playbook_id)}      for p in search.user_playbooks),    *({"kind": "agent_playbook", "learning_id": str(p.agent_playbook_id)}      for p in search.agent_playbooks),] interactions = [    InteractionData(role="user", content=user_message),    InteractionData(role="assistant", content=agent_response,                    retrieved_learnings=retrieved_learnings),] client.publish_interaction(    user_id=user.id,    interactions=interactions,    session_id=session_id,    agent_version="v1",    retrieval_experiment_id=(        search.experiment.experiment_id if search.experiment else None    ),    retrieval_experiment_arm=(search.experiment.arm if search.experiment else None),)

If no experiment is active, experiment is absent and both publish fields should be omitted. If it is present, send both fields together. Emend rejects an unknown experiment, a single missing field, or an arm that does not match the user’s deterministic assignment.

Dashboard inspection searches bypass experiment gating so operators can still inspect learned context. Agent traffic should not send the dashboard caller type.

Recommended: publish retrieved_learnings for every profile or playbook actually injected into treatment responses, including context the answer did not cite. The cURL list is illustrative: replace it with the full injected subset; use [] for holdout responses. Experiment ID/arm attribution and learning IDs serve different purposes; send both when applicable.

Methodology Contract

Assignment is user-level, but outcomes remain session-level. Emend joins each evaluated session to the first attributed request for its user and session, then compares treatment sessions with holdout sessions. This means one user’s sessions stay in one arm even though the search request itself has no session ID.

The results page reports success, corrections, turns to resolution, and escalation for both arms. Treatment lift is the difference in session success rates. Its 95% confidence interval is clustered by user so repeat sessions from one person are not treated as independent random assignments. Evaluation coverage shows what share of evaluated sessions could be joined to experiment attribution; investigate untagged sessions before relying on the lift.

Keep these integration invariants:

• Always use the same stable application user_id for search and publish.
• Echo the search response’s experiment ID and arm on every related publish.
• Publish both arms normally; only the search response differs.
• Keep the agent version, evaluation rubric, traffic mix, and other runtime behavior comparable across arms.
• Stop an experiment before starting another. Do not reuse an experiment ID.

Sampling and freshness

Regen jobs sample at most 200 sessions per (day × group) stratum by default so cost stays predictable as your traffic grows. The dashboard surfaces the sampled n per point so it’s always honest about how many sessions back each number. If you click into a session that wasn’t in the sampled set, Emend grades it on demand via POST /api/evaluations/grade_on_demand and caches the result for 24 hours.

To tune the defaults, set eval_sample_n_per_stratum and eval_concurrency_limit in your Config.

API Reference

See Retrieval Experiments for the lifecycle and reporting methods, Search Models for response assignment metadata, and publish_interaction for publish attribution fields.

Per-turn comparison (F1)

The Evaluation page also renders a per-turn head-to-head win rate comparing your agent’s response with Emend’s retrieved context vs. without. F2 measures session-level outcomes; F1 zooms in to individual turns so you can spot-check the judge and surface specific cases where the Emend-less response was actually better.

To produce this signal, your agent code generates two responses per turn and uploads both on the same Interaction via the existing publish_interaction() API.

Integration

For each agent turn you want graded, generate the regular response WITH Emend rules in context, then re-run your LLM WITHOUT them to produce a shadow response. Publish both on the same agent interaction:

from emend import EmendClient, InteractionData client = EmendClient() # Retrieve Emend rules for this turn (your retrieval call)rules = client.search(query=user_message, user_id=user.id)retrieved_learnings = [    *({"kind": "profile", "learning_id": p.profile_id} for p in rules.profiles),    *({"kind": "user_playbook", "learning_id": str(p.user_playbook_id)}      for p in rules.user_playbooks),    *({"kind": "agent_playbook", "learning_id": str(p.agent_playbook_id)}      for p in rules.agent_playbooks),] # Generate the regular response WITH rulesregular_response = llm.generate(user_message, rules=rules) # Generate the shadow response WITHOUT rulesshadow_response = llm.generate(user_message, rules=[]) # Publish both on the same agent Interactionclient.publish_interaction(    user_id=user.id,    interactions=[        InteractionData(role="User", content=user_message),        InteractionData(            role="Agent",            content=regular_response,        # what was served to the user            shadow_content=shadow_response,  # used only for grading            retrieved_learnings=retrieved_learnings,        ),    ],    session_id=session_id,    source="my-integration",    agent_version="v1",)

Dashboard diagnostics

• A Win rate trend plots daily win and loss rates and shows the underlying wins, losses, ties, and n for each UTC day.
• A View recent comparisons drawer with the most recent judged turns, each rendered side-by-side with the judge’s rationale. Backed by GET /api/evaluations/shadow_comparisons/recent.
• A Top disagreements widget listing turns where the shadow response was significantly better than the Emend-augmented one (output.is_significantly_better=True losses) — actionable cases for updating your Emend rules.

Methodology contract

• The judge compares the two responses against request-local transcript context; no session history outside the current publish request is shown to the judge.
• Position (Request 1 vs Request 2) is randomized per call to mitigate LLM judge position bias. The mapping is recorded on ShadowComparisonVerdict.emend_is_request_1 so wins/losses can be derived deterministically downstream.
• The judge prompt version is pinned per org via Config.shadow_comparison_judge_prompt_version. Verdicts are stored with the version that produced them; the dashboard filters to your currently pinned version so a future rubric bump never silently mixes epochs into the headline number.
• Sampling: regen jobs are stratified per day at Config.eval_sample_n_per_stratum (default 200) so cost stays predictable. Clicking into a non-sampled session triggers on-demand grading via POST /api/evaluations/grade_on_demand with a 24h cache.

Where the data lands in the API

The win-rate trend is carried on GetEvaluationOverviewResponse.shadow_win_rate_trend (see the ShadowWinRateTrend schema). Individual verdicts are returned by the recent-verdicts endpoint above; each verdict is a ShadowComparisonVerdict wrapping a ShadowComparisonOutput.