API Reference

Retrieval Experiments

Start, stop, inspect, and report user-level retrieval holdout experiments.

Retrieval experiments deterministically assign users to treatment or holdout. Treatment searches retrieve learned context normally; holdout searches succeed with empty learning results. Both arms continue to publish, receive session evaluation, and teach Emend.

Info

Only one experiment can be active for an organization. Experiment IDs are unique and stopped experiments remain in history.

list_retrieval_experimentsmethod

experiments = client.list_retrieval_experiments()print(experiments.active_experiment)

Calls GET /api/retrieval_experiments and returns the active record, when one exists, plus all historical records in reverse start order.

start_retrieval_experimentmethod

experiments = client.start_retrieval_experiment(    experiment_id="support-agent-2026-08",    holdout_percentage=10,)

Calls POST /api/retrieval_experiments. holdout_percentage must be greater than 0 and less than 100. Starting while another experiment is active or reusing an ID returns 409.

stop_retrieval_experimentmethod

experiments = client.stop_retrieval_experiment("support-agent-2026-08")

Calls POST /api/retrieval_experiments/stop. Stopping ends new assignments but does not remove stored publish attribution or historical results.

get_retrieval_experiment_resultsmethod

results = client.get_retrieval_experiment_results("support-agent-2026-08")print(results.treatment.success_rate)print(results.holdout.success_rate)print(results.treatment.average_output_tokens)print(results.holdout.average_output_tokens)print(results.success_rate_lift_percentage_points)print(results.confidence_interval_95_percentage_points)

Calls GET /api/retrieval_experiments/{experiment_id}/results. Results group session-level success evaluations by the user-level arm stored on published requests. It also compares stored non-user content-token totals per published session. The confidence interval is clustered by user. Evaluation coverage and the unattributed session count show whether evaluated sessions were published with experiment metadata; output_token_session_count shows how many published sessions have complete persisted token counts.

Response schemas

RetrievalExperimentListResponseresponse model

Prop	Type
`active_experiment`	RetrievalExperimentRecord \| None
`experiments`	list[RetrievalExperimentRecord]

RetrievalExperimentRecordmodel

Prop	Type
`experiment_id`	string
`holdout_percentage`	float
`started_at`	integer
`ended_at`	int \| None

RetrievalExperimentResultsResponseresponse model

Prop	Type
`experiment`	RetrievalExperimentRecord
`treatment`	RetrievalExperimentArmMetrics
`holdout`	RetrievalExperimentArmMetrics
`success_rate_lift_percentage_points`	float \| None
`relative_success_lift`	float \| None
`confidence_interval_95_percentage_points`	tuple[float, float] \| None
`evaluated_session_coverage`	float \| None
`unattributed_evaluated_session_count`	integer

RetrievalExperimentArmMetricsmodel

Prop	Type
`arm`	"treatment" \| "holdout"
`assigned_user_count`	integer
`published_session_count`	integer
`evaluated_user_count`	integer
`evaluated_session_count`	integer
`success_rate`	float \| None
`average_corrections`	float \| None
`average_turns_to_resolution`	float \| None
`escalation_rate`	float \| None
`average_output_tokens`	float \| None
`output_token_session_count`	integer

Use the complete integration pattern in Measuring Emend's Impact.

PreviousAgent Evaluation Next Operations