Build

Extractors & Evaluation

Configure profile extractors, playbook aggregation, and agent success evaluation in Emend.

Extractor settings define how Emend extracts profiles, collects playbook entries, evaluates agent success, and asynchronously tags their persisted results. Prompt customization is optional; Emend ships with defaults so you can publish interactions before tuning extraction behavior.

Profile Extractor Configuration

The profile extractor automatically generates user profiles from interactions. By default, Emend looks for durable user facts, preferences, goals, and constraints. Configure the extractor only when you want to narrow what information to capture or add tags.

from emend.models.config_schema import ProfileExtractorConfig profile_config = ProfileExtractorConfig(    # Optional: override the default profile extraction prompt    extraction_definition_prompt="""Extract the following user information:- Name and contact details- Preferences and interests- Goals and intent""",    # Optional: Context about the interaction type    context_prompt="""This is a conversation between a sales agent and a potential customer.Extract any relevant customer information.""",    # Optional: How to categorize extracted profiles    tagging_definition_prompt="""Categorize extracted profiles as one of:- 'basic_info': Name, contact, demographics- 'preferences': Likes, dislikes, style preferences- 'intent': Goals, purchase intent, timeline""",    # Optional: Only process interactions from these sources    request_sources_enabled=["chat", "email"],    # Optional: Require manual triggering instead of auto extraction    manual_trigger=False)

ProfileExtractorConfig fields

Prop	Type	Description
`extractor_name`	str	Deprecated compatibility field. Accepted in legacy configs but ignored for runtime selection. Default: None.
`extraction_definition_prompt`	str	Optional override describing what user information to extract from interactions. Legacy field name profile_content_definition_prompt is accepted and auto-migrated. Default: built-in profile prompt.
`context_prompt`	str	Provides context about the interaction type to improve extraction accuracy. Default: None.
`tagging_definition_prompt`	str	Defines tags to attach asynchronously to persisted profiles. Default: None.
`should_extract_profile_prompt_override`	str	Custom logic to determine when profile extraction should run. Default: None.
`request_sources_enabled`	list[str]	Limits extraction to specific sources (e.g., "chat", "email"). If not set, processes all sources. Default: None.
`manual_trigger`	bool	If True, skip auto extraction and require manual triggering via rerun_profile_generation. Default: False.
`window_size_override`	int	Override the global window_size for this extractor. Default: None.
`stride_size_override`	int	Override the global stride_size for this extractor. Default: None.

Optional: Configure the Profile Extractor

Configure one profile extractor with the scope you want to capture. Use request_sources_enabled and manual_trigger when you need narrower control over when it runs.

profile_config = ProfileExtractorConfig(    extraction_definition_prompt="Extract product preferences, style choices, budget range",    tagging_definition_prompt="choose from 'style_preference' or 'budget_preference'",    request_sources_enabled=["chat"]) config.profile_extractor_config = profile_config # Later, run the configured extractor manually:client.rerun_profile_generation(    user_id="user_123",    wait_for_response=True)

Info

extractor_names is still accepted by rerun APIs for older clients, but it no longer selects or skips extractors. Emend runs the configured singleton profile extractor when it is enabled.

Agent Playbook Configuration

Playbook configuration defines how Emend learns from user interactions to improve agent behavior. By default, Emend looks for durable agent improvement signals in published interactions. Configure the playbook extractor only when you want a narrower playbook focus, custom tags, or different aggregation thresholds.

The playbook system works in two stages:

User Playbooks - Extracted from each interaction and stored per user/agent version
Agent Playbooks - Consolidated from multiple user playbooks into actionable insights for agent improvement

from emend.models.config_schema import (    UserPlaybookExtractorConfig,    PlaybookAggregatorConfig) playbook_config = UserPlaybookExtractorConfig(    # Optional: override the default playbook extraction prompt    extraction_definition_prompt="""Analyze the interaction and extract playbook entries about:- Was the customer satisfied with the response?- What could have been done better?- Any specific complaints or praise?""",    # Optional: How to categorize the playbook entries    tagging_definition_prompt="""Rate satisfaction: 'positive', 'neutral', 'negative'""",    # Optional: When to aggregate user playbooks    aggregation_config=PlaybookAggregatorConfig(        min_cluster_size=5,        reaggregation_trigger_count=3    ))

Set it on config.user_playbook_extractor_config before saving. (PlaybookConfig remains as a deprecated alias, and the legacy field names playbook_name / playbook_definition_prompt / playbook_aggregator_config are still accepted and auto-migrated.)

UserPlaybookExtractorConfig

Prop	Type	Description
`extractor_name / playbook_name`	str	Deprecated compatibility fields. Accepted in legacy configs but ignored for runtime selection.
`extraction_definition_prompt`	str	Optional override describing what to extract from each interaction. Legacy field playbook_definition_prompt is accepted and auto-migrated.
`context_prompt`	str	Additional context for playbook extraction.
`tagging_definition_prompt`	str	Defines tags to attach asynchronously to persisted playbook entries.
`aggregation_config`	PlaybookAggregatorConfig	Controls when user playbooks are aggregated into agent playbooks. Defaults to an enabled PlaybookAggregatorConfig; missing and legacy null values use these defaults. Legacy name playbook_aggregator_config is auto-migrated.
`deduplication_config`	DeduplicationConfig	Controls deduplication against existing user playbooks.
`request_sources_enabled`	list[str]	Only extract from these request sources. If not set, extracts from all sources.
`window_size_override`	int	Override the global window_size for this extractor.
`stride_size_override`	int	Override the global stride_size for this extractor.

PlaybookAggregatorConfig

Controls when user playbooks are aggregated into consolidated agent playbooks.

Prop	Type	Description
`min_cluster_size`	int	Default: 2. Minimum user playbooks required before first aggregation runs. Set to 1 to disable aggregation while keeping user-playbook extraction enabled.
`reaggregation_trigger_count`	int	Default: 2. Number of new user playbooks that trigger re-aggregation.
`clustering_similarity`	float \| null	Default: model-specific. Cosine similarity threshold for clustering (0.0–1.0). Defaults to 0.30 for MiniLM/other models and 0.85 for Nomic; higher = tighter clusters.
`direction_overlap_threshold`	float	Default: 0.6. Token overlap threshold for grouping playbooks by direction (0.0–1.0).

Aggregation is enabled by default whenever user-playbook extraction is enabled.

Info

Stored configurations where aggregation_config is missing or null are normalized to the defaults above when loaded. To disable only aggregation, set min_cluster_size=1; setting the entire user-playbook extractor to null disables extraction as well.

Example: With min_cluster_size=5 and reaggregation_trigger_count=3:

First aggregation runs after 5 user playbooks are collected
Re-aggregation runs every 3 new user playbooks (at 8, 11, 14, etc.)

Agent Success Configuration

Success configuration defines how Emend evaluates whether the agent achieved its goals in each session. It is enabled by default with 5% deterministic session sampling.

Best Practice

Define success based on user outcomes and goals, not technical implementation details. Focus on what the user accomplished rather than how the agent responded.

For the full publishing and analysis workflow, including evaluation_only=True, source-set comparison, shadow responses, on-demand grading, and regenerate jobs, see Evaluating Agent Performance.

from emend.models.config_schema import (    AgentSuccessConfig,    ToolUseConfig) success_config = AgentSuccessConfig(    # Required: Define what success looks like (focus on user outcomes)    success_definition_prompt="""Evaluate if the agent successfully:1. Understood the customer's booking request2. Provided accurate availability information3. Completed the booking or explained next steps4. Left the customer satisfied A successful interaction ends with either:- A confirmed booking- Clear next steps agreed upon- Customer explicitly stating they're satisfied""",    # Optional: How to categorize outcomes    tagging_definition_prompt="""Classify outcome as:- 'booking_completed': Customer completed a booking- 'booking_pending': Customer will return to complete- 'booking_cancelled': Customer decided not to book- 'information_only': Customer was just browsing""",    # Optional: Evaluate only a portion of sessions (0.5 = 50%; default is 0.05)    sampling_rate=0.5,    # Optional: Give evaluation-only sessions different success-judge coverage.    # None (the default) dynamically inherits sampling_rate.    evaluation_only_sampling_rate=1.0,) # Configure tools the agent can use at the Config level# (shared across success evaluation and playbook extraction)config.tool_can_use = [    ToolUseConfig(        tool_name="check_availability",        tool_description="Check room/service availability for given dates"    ),    ToolUseConfig(        tool_name="create_booking",        tool_description="Create a new booking for the customer"    )]

AgentSuccessConfig

Prop	Type	Description
`evaluation_name`	str	Deprecated compatibility field. Accepted in legacy configs and requests but ignored for runtime selection. Default: None.
`success_definition_prompt`required	str	Describes what constitutes a successful interaction (focus on user outcomes). Default: built-in AI agent rubric when using Config defaults.
`tagging_definition_prompt`	str	Defines tags to attach asynchronously to persisted evaluation summaries. Default: None.
`request_sources_enabled`	list[str]	Only evaluate requests from these sources. If not set, evaluates all sources. Default: None.
`sampling_rate`	float	Fraction of sessions to evaluate (0.0-1.0). Increase for audits or reduce for cost control. Default: 0.05.
`evaluation_only_sampling_rate`	float \| None	Session-success sampling rate for evaluation-only sessions (0.0-1.0). None inherits the current sampling_rate. Default: None.
`retrieved_learning_sampling_rate`	float \| None	Independent retrieved-learning judge rate (0.0-1.0). None inherits sampling_rate. Default: None.
`window_size_override`	int	Override the global window_size for this evaluator. Default: None.
`stride_size_override`	int	Override the global stride_size for this evaluator. Default: None.

ToolUseConfig

Describes tools available to the agent. This provides the evaluator with context about what actions were possible during the interaction.

Prop	Type	Description
`tool_name`	str	Name of the tool
`tool_description`	str	Description of what the tool does

See Evaluating Agent Performance for end-to-end examples that use this configuration.

PreviousConfiguration Guide Next LLM Configuration