Concepts

Shadow Content

The shadow_content field allows you to capture an alternative agent response alongside the production response, enabling A/B testing and comparison of different agent behaviors.

How it works

This separation allows you to:

• Compare agent versions: Test how a new agent version would respond without affecting production
• Evaluate profile effectiveness: See how different user profile configurations affect agent responses
• A/B test playbook integration: Compare responses with and without specific agent playbook entries applied

When to use shadow_content:

• Testing a new agent version's responses against the current production version
• Evaluating how different user profile configurations affect agent behavior
• Comparing responses with different playbook/prompt improvements applied
• Running shadow evaluations without impacting the user experience

Example: Comparing Agent Responses

# Agent v1.0 is in production, testing v2.0 response in shadowclient.publish_interaction(    user_id="user_123",    interactions=[        {            "role": "User",            "content": "I would like to get a refund"        },        {            "role": "Agent",            # Production response (v1.0) - generic, asks for info            "content": "Happy to help you with that. What is your email or phone number?",            # Shadow response (v2.0) - personalized, uses known user data            "shadow_content": "Happy to assist you with that. I can see your phone number is 123-456-7890. Is that the right number to send the refund confirmation to?"        }    ],    session_id="support_session_001",    agent_version="v1.0")

In this example:

• content stores the production response from agent v1.0
• shadow_content captures how agent v2.0 (with access to user profiles) would have responded
• Per-turn evaluation uses shadow_content to assess the v2.0 approach
• You can compare evaluation results to decide whether to promote v2.0 to production

Expert Content

The expert_content field lets you provide a human expert's ideal response alongside the agent's actual response. Emend compares the two and generates actionable playbook entries — SOPs (Situation/Trigger, Instruction, Pitfall) — so your agent learns to match expert behavior.

How it works

When expert_content is present on any interaction in a session, Emend automatically switches to an expert playbook extraction pipeline that:

• Pairs each expert response with the agent's response and the preceding user question
• Analyzes alignment gaps between the agent and expert
• Generates structured SOPs: what triggered the gap, what the agent should do instead, and what pitfalls to avoid

When to use expert_content:

• Training from gold-standard responses: When domain experts review agent conversations and provide ideal answers
• Quality assurance workflows: Compare production agent output against expert-vetted responses
• Bootstrapping new agents: Seed playbook entries from existing expert knowledge before you have enough user interactions
• Compliance or accuracy-critical domains: Ensure agent responses meet expert standards (legal, medical, financial)

Example: Learning from Expert Responses

client.publish_interaction(    user_id="user_123",    interactions=[        {            "role": "User",            "content": "What is your return policy for electronics?"        },        {            "role": "Agent",            # Agent's actual response — incomplete            "content": "You can return electronics within 30 days of purchase.",            # Expert's ideal response — comprehensive            "expert_content": (                "Electronics can be returned within 30 days of purchase with the original receipt. "                "Items must be in original packaging and include all accessories. "                "Opened software and personal care electronics are final sale. "                "For defective items, the return window extends to 90 days. "                "Refunds are processed to the original payment method within 5-7 business days."            )        }    ],    session_id="expert_review_001",    source="expert_review",    agent_version="v1.0")

In this example:

• content stores the agent's actual (incomplete) response
• expert_content provides the expert's comprehensive ideal response
• Emend compares the two and extracts playbook entries like: “Agent should mention packaging requirements, exceptions for opened software, extended window for defective items, and refund processing timeline”
• The extracted entries enter the standard aggregation pipeline, improving the agent for all users