Research Platform for Optimizing
Conversational AI Apps

Score the emotional intelligence of the Assistant model's behavior in a conversation relative to a selected rubric — A Judge model evaluates the Assistant's conversational behavior against each of the criteria in a selected rubric.

Score the Assistant's Debrief report — that is, their perceptions of the User Persona's state of mind at the end of the conversation — by comparing it to the User Persona's self-assessed state of mind — A Judge model measures how accurately the Assistant understood the User Persona by comparing their self-assessed state of mind at the end of the conversation with the Assistant's Debrief report. 

Score the Assistant's impact on the User Persona's state of mind during the course of the conversation — A Judge model measures the Assistant's impact on the user's state of mind by comparing the User Persona's initial state of mind relative to their self-assessed state of mind at the end of the conversation. 

The Conversation-Evaluation Process Workflow

A chat conversation begins when PersonA (a simulated user persona) submits a prompt (turn 1) to PersonB (assistant), which responds (turn 2) to complete conversational round 1. 

The simulated chat conversation continues for a specified number of rounds, with the two language models, User Persona (PersonA) and Assistant (PersonB), alternating turns until the conversation reaches a specified number of rounds.

The User Persona model simulates the personality and state of mind of PersonA, governed by configurable values, feelings, needs, and behavior patterns, then posts a message to the Assistant.  After each round, the User Persona provides a self-reflection that evaluates their then-current state of mind.

The Assistant model's responses are evaluated by a Judge model against a rubric — that is, a set of criteria that semantically identify emotionally intelligent behaviors.

After each round, the User Persona provides a self-reflection that evaluates their then-current state of mind.

After each conversation, the Assistant model provides a "debrief" report in which it assesses the User Persona's feelings, unmet needs, and state of mind.

Stages of the Conversation-Evaluation Process Workflow

As a research prototype, this application enables you to observe each step in the conversation-evaluation process in terms of its inputs, prompts, and outputs and to execute (or re-execute) each step separately.

1. Before a conversation begins:
1. You may create or select a User Persona — demographics, personality, feelings, requests, unmet needs, private facts, trust-building patterns, vulnerability-protection patterns, support-seeking patterns, initial state of mind.
2. You may create or select a conversation scenario — a situational context for a conversation and an initial state of mind for the User Persona you created or selected.
3. You may create or select a scenario collection — a grouping of selected conversation scenarios for an evaluation run.
4. Start an evaluation run — The evaluation run uses the scenario collection you created or selected to set up a conversation. Then the Judge model evaluates the Assistant model's behavior during the conversation relative to a selected rubric — that is, a set of criteria that semantically identify emotionally intelligent behaviors.


2. During each conversational round:
1. User Persona poses a message to the Assistant.
2. User Persona privately updates an assessment of their state of mind.
3. Assistant posts a response to the User Persona's message.


3. After the conversation is complete:
1. A Judge evaluates the EQ of Assistant's contributions to the conversation relative to each criterion in the selected rubric.
   
2. The Assistant assesses User Persona's state of mind at the end of the conversation — this is called a "debrief" report.
   
3. A Judge model evaluates the Assistant's debrief assessment of User Persona's state of mind at the end of the conversation relative to the User Persona's beginning profile and their self-assessed state of mind at the end of the conversation.
   
4. A Judge model evaluates the Assistant's impact on User Persona's state of mind — initial vs. final.
   
5. Calculate the final scores and store them in the database.
   


4. Iterate with the next conversation scenario in the collection until the evaluation run is complete.

---

The workflow is designed to:

1. Orchestrate conversations between language models.
2. Track prompts created and submitted to the language models.
3. Track latencies of each LM interaction.
4. Assess the qualities of each genEI agent's work products in each eval run.
5. Compare eval runs from multiple models in a pair-wise fashion (ELO scores).

---

Comparison: Assistant's Debrief vs. Assistant's State-of-Mind Prediction Accuracy

These are two different ways of evaluating the Assistant's understanding of the User Persona's state of mind:

1. Assistant's Debrief (Qualitative, Judge-Assessed)

What it measures: Quality of the Assistant's narrative understanding and emotional insight into the User Persona

  How it's calculated:
  - A Judge model reads:
    - The Assistant's written assessment (user_personality_assessment, primary_unmet_needs_assessment)
    - The TRUE profile of the User Persona (personality patterns, unmet needs, trust/defense/help-seeking styles)
  - Judge scores across 4 dimensions on 0-10 scale:
    - user_personality_insight - Did they understand personality/behavioral patterns?
    - user_primary_unmet_needs_and_requests_insight - Did they identify core needs?
    - emotional_attunement - Did they grasp emotional dynamics?
    - overall_understanding - Holistic comprehension quality

  Example from judge prompt:
  Judge evaluates: "Did the Assistant provide comprehensive assessment of the
  User Persona's personality, unmet needs, trust style, defense style, help-seeking
  approach, or other emotional dynamics? Did they exhibit emotionally relevant
  insights vs. surface-level observations?"

  Note: This evaluates depth of insight, not numeric precision. It is used in final scoring as a weighted component.

---

2. Prediction Accuracy (Mathematical, Objective)

  What it measures: Numeric accuracy of the Assistant's predictions of the User Persona's emotional state-of-mind values.

  How it's calculated:
  - At the end of the conversation, the Assistant predicts 14 emotional state dimensions (trust, anxiety, 
  emotional_regulation, etc.) on a 0-10 scale.
  - Each prediction is compared to the actual simulated state value.
  - Mean absolute error is calculated across all dimensions.
  - Converted to accuracy score: (10 - mean_error - 7) / 3 * 10
  - Scale: 0-10 (scores ≤7 become 0 to make it discriminative at the high end)

  Example:
  Predicted trust: 8, Actual trust: 7 → Error: 1
  Predicted anxiety: 3, Actual anxiety: 5 → Error: 2
  ... (across 14 dimensions)
  Mean error: 1.5 → Raw accuracy: 8.5 → Prediction Accuracy: 5.0

  Note: This is a mechanical calculation -- no judge model involved. 
  It's marked as "informational" and not used in final scoring.

---

  Key Differences
  ┌──────────────────────┬───────────────────────────┬──────────────────────────────────────────┐
  │        Aspect        │    Prediction Accuracy    │           Assistant's Debrief            │
  ├──────────────────────┼───────────────────────────┼──────────────────────────────────────────┤
  │ Evaluator            │ Mathematical calculation  │ Judge model (LLM)                        │
  ├──────────────────────┼───────────────────────────┼──────────────────────────────────────────┤
  │ Evaluates            │ Numeric state predictions │ Narrative assessments                    │
  ├──────────────────────┼───────────────────────────┼──────────────────────────────────────────┤
  │ Focus                │ Precision of numbers      │ Depth of understanding                   │
  ├──────────────────────┼───────────────────────────┼──────────────────────────────────────────┤
  │ Criteria             │ Mean absolute error       │ Insight quality, comprehensiveness       │
  ├──────────────────────┼───────────────────────────┼──────────────────────────────────────────┤
  │ Used in final score? │ No (informational only)   │ Yes (weighted component)                 │
  ├──────────────────────┼───────────────────────────┼──────────────────────────────────────────┤
  │ Can detect           │ Off-by-1 errors           │ Missing insights, surface-level thinking │
  └──────────────────────┴───────────────────────────┴──────────────────────────────────────────┘
  ---
  Why Both Exist?

  - Prediction Accuracy = "Can the Assistant accurately quantify emotional states?"
  - Assistant's Debrief = "Does the Assistant truly understand the person beyond numbers?"

  An Assistant could score high on Prediction Accuracy (guessed the right numbers) 
  but low on Debrief (failed to articulate why or what it means). 
  Conversely, an Assistant could have deep qualitative insight but be slightly off numerically.

  The Debrief score is considered more valuable for measuring emotional intelligence, 
  which is why it's weighted in the final score while Prediction Accuracy is kept for analysis only.