Skip to main content

Logs & Monitoring - Track Conversations

In this chapter, you'll learn how to track, analyze and use conversations with your agents for improvements.

📊 What are Logs?​

Logs are detailed records of all interactions between users and your agents. They contain:

  • 💬 Messages: What did the user ask? What did the agent answer?
  • 🛠️ Tool Calls: Which tools were used?
  • 💰 Token Usage: How many tokens were consumed? What did it cost?
  • ⏱️ Timestamps: When did the conversation take place?
  • 🏷️ Metadata: Sales channel, agent, thread ID, etc.

🎯 Why are Logs Important?​

1. Quality Control​

  • Check if your agents give correct answers
  • Identify errors or inappropriate responses

2. Cost Monitoring​

  • See exactly which conversations cost how much
  • Identify expensive requests

3. Knowledge Management​

  • Build a FAQ database from real customer questions
  • Use frequent questions for agent improvement

4. Performance Analysis​

  • Which tools are used most frequently?
  • How long do conversations last?

5. Debugging​

  • When problems occur: What exactly happened?
  • Which tools were called in which order?

🔍 Find Logs in Shopware Backend​

Step 1: Open Logs Overview​

  1. Log in to the Shopware backend
  2. Navigate to 5E OAI Agent Manager (in main menu)
  3. Click on the "Assistant Logs" tab

Step 2: What You See​

The logs overview shows a table with the following columns:

ColumnDescription
DateDate and time of conversation
AgentWhich agent was used?
QuestionThe user's question/request
AnswerThe agent's answer (abbreviated)
TokensNumber of tokens consumed
CostEstimated cost in USD
Sales ChannelWhich shop did the conversation take place in?
Thread IDUnique conversation ID
TagsAssigned categories/keywords

🔎 Filter and Search Logs​

Filter by Time Period​

  1. Click on the date filter
  2. Select a time period:
    • Today
    • Last 7 days
    • Last month
    • Custom

Filter by Agent​

  1. Click on the agent filter
  2. Select one or more agents
  3. Only conversations with these agents will be displayed

Filter by Tags​

  1. Click on the tag filter
  2. Select tags like "shipping", "returns", "product", etc.
  3. Only conversations with these tags will be displayed

Full-text Search​

  1. Use the search field in the top right
  2. Search for keywords in questions or answers
  3. Example: Search for "delivery time" shows all conversations containing this term

📖 View Individual Conversation​

Open Conversation Details​

  1. Click on a log entry in the table
  2. A detail dialog will open

What You See in Detail​

1. Conversation Header​

  • Thread ID
  • Agent name
  • Date and time
  • Sales channel

2. Message History​

Each message shows:

┌─────────────────────────────────────────┐
│ 👤 User (14:32:15)                      │
├─────────────────────────────────────────┤
│ I'm looking for a red jacket in size M  │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│ 🔧 Tool Call: product_search            │
├─────────────────────────────────────────┤
│ Parameters:                             │
│ - query: "red jacket"                   │
│ - filters: {size: "M"}                  │
│ - limit: 10                             │
├─────────────────────────────────────────┤
│ Result: 5 products found                │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│ 🤖 Agent (14:32:18)                     │
├─────────────────────────────────────────┤
│ I found 5 red jackets in size M:       │
│                                         │
│ 1. Winter Jacket Red - $89.99           │
│ 2. Rain Jacket Sport - $49.99           │
│ ...                                     │
└─────────────────────────────────────────┘

3. Token Statistics​

TypeCountCost
Input Tokens1,245$0.0019
Output Tokens520$0.0031
Cached Tokens980$0.0002
Total2,745$0.0052

4. Metadata​

  • Model: Which OpenAI model was used? (e.g., gpt-4o-mini)
  • Temperature: Creativity setting (e.g., 0.7)
  • Reasoning Effort: Think depth (e.g., medium)
  • Tools used: List of all called tools

📊 Understanding Token Usage​

What are Tokens?​

A token is the smallest unit that OpenAI processes:

  • 1 Token ≈ 4 characters
  • 1 Token ≈ 0.75 words (in English)

Example:

"Hello, how can I help you?"
= approx. 6 tokens

Token Types​

1. Input Tokens (Input)​

Everything that is sent to OpenAI:

  • User message
  • System Instructions
  • Tool descriptions
  • Conversation history

Cost Example (gpt-4o-mini):

  • $0.15 per 1 million tokens
  • 1,000 tokens = $0.00015

2. Output Tokens (Output)​

Everything that OpenAI returns:

  • Agent answers
  • Tool calls

Cost Example (gpt-4o-mini):

  • $0.60 per 1 million tokens
  • 1,000 tokens = $0.0006

3. Cached Tokens (Cache)​

Reused input tokens from previous requests:

  • 50-90% cheaper than normal input tokens
  • Automatically cached by OpenAI

Cost Example (gpt-4o-mini):

  • $0.075 per 1 million tokens (50% discount)
  • 1,000 tokens = $0.000075

💰 Cost per Conversation​

Cost Calculation Example​

A typical product search:

Input Tokens:    1,500 tokens × $0.00000015 = $0.000225
Output Tokens:     600 tokens × $0.00000060 = $0.000360
Cached Tokens:   1,200 tokens × $0.00000008 = $0.000096
─────────────────────────────────────────────────────
Total:                                      $0.000681

Per Day (100 conversations):

  • 100 × $0.000681 = $0.068 (approx. 7 cents)

Per Month (3,000 conversations):

  • 3,000 × $0.000681 = $2.04

🏷️ Using Tags​

What are Tags?​

Tags are keywords for categorizing conversations:

  • shipping - Delivery questions
  • returns - Return questions
  • product - Product advice
  • payment - Payment questions
  • error - Problems/errors

Automatically Assign Tags​

Tags are automatically set when the agent uses the tag_conversation tool.

Example in Instructions:

When the customer asks about shipping, use the tag_conversation tool
with the tag "shipping".

Manually Add Tags​

  1. Open a conversation in detail view
  2. Click "Edit tags"
  3. Select existing tags or create new ones
  4. Save

Use Tags for Analysis​

Example Scenario:

You want to know how many customers ask about shipping:

  1. Filter: Tag = "shipping"
  2. Time period: Last month
  3. Result: 450 conversations

Insights:

  • Shipping questions are frequent (450 out of 3,000 = 15%)
  • Maybe you should place shipping information more prominently
  • Or: Create shipping FAQ

📈 Use Logs for Improvements​

1. Identify Frequent Questions​

Approach:

  1. Filter logs from last 30 days
  2. Look for frequently recurring topics
  3. Create a list

Example Results:

  • "How long does shipping take?" → 120 times
  • "Can I return?" → 95 times
  • "What size fits me?" → 78 times

Action:

  • Create FAQ entries with create_log_entry
  • Agent can then retrieve them with search_logs

2. Find Incorrect Answers​

Signs of problems:

  • Agent gives wrong product information
  • Tool calls fail
  • Answers are incomprehensible

Approach:

  1. Read problematic conversations completely
  2. Identify the problem (wrong tool? Unclear instructions?)
  3. Improve agent configuration

3. Analyze Tool Usage​

Questions:

  • Which tools are used most frequently?
  • Are important tools not being used at all?
  • Does the agent call unnecessary tools?

Example:

You see in the logs:

  • product_search: 1,200 calls
  • get_order_status: 450 calls
  • get_shipping_info: 2 calls

Insight:

  • get_shipping_info is hardly used
  • Maybe you should explicitly mention in instructions when to use this tool

4. Identify Performance Problems​

Warning signals:

  • Conversations last > 30 seconds
  • Many tool calls in succession
  • High token usage

Solutions:

  • Lower Reasoning Effort (from high to medium)
  • Deactivate unneeded tools
  • Shorten instructions

🗄️ Database Structure (for Advanced Users)​

Logs are stored in the following tables:

fel_assistant_logs​

Main table for log entries:

  • id - Unique ID
  • question - User question
  • answer - Agent answer
  • context - Additional context
  • meta_info - Metadata (JSON)
  • created_at - Timestamp
  • sales_channel_id - Sales channel

fel_assistant_chat_message​

Detailed messages per thread:

  • id - Unique ID
  • thread_id - Thread reference
  • role - user or assistant or tool
  • content - Message content
  • tool_calls - Tool calls (JSON)
  • token_usage - Token statistics (JSON)
  • created_at - Timestamp

fel_assistant_tags​

Tag definitions:

  • id - Unique ID
  • name - Tag name (e.g., "shipping")
  • description - Description

fel_assistant_tag_usage​

Link between logs and tags:

  • log_id - Reference to log entry
  • tag_id - Reference to tag

📤 Export Logs (Workaround)​

Note: Currently there is no direct export in the UI.

Manual Export Options:​

1. Via phpMyAdmin / Adminer​

SELECT
    created_at,
    question,
    answer,
    JSON_EXTRACT(meta_info, '$.tokens') as tokens,
    JSON_EXTRACT(meta_info, '$.cost') as cost
FROM fel_assistant_logs
WHERE created_at >= '2025-01-01'
ORDER BY created_at DESC;

Export result as CSV.

2. Via Shopware CLI (for developers)​

bin/console dbal:run-sql "SELECT * FROM fel_assistant_logs" > logs.csv

🔐 Privacy & Log Cleanup​

GDPR Compliance​

Logs potentially contain personal data:

  • Customer questions may contain names, addresses, order numbers
  • IP addresses (if stored)

Recommendations:

  1. Set Retention Period

    • E.g., automatically delete logs after 90 days
    • Configurable in plugin configuration

  2. Anonymization

    • Remove sensitive data from logs
    • Replace order numbers, names, etc.

  3. Information Duty

    • Customers can request information about stored data
    • Logs must be searchable (e.g., by email)

Manually Delete Logs​

  1. Go to 5E OAI Agent Manager → Assistant Logs
  2. Select log entries (checkbox)
  3. Click "Delete"
  4. Confirm
Caution

Deleted logs cannot be recovered!

🛠️ Troubleshooting​

Logs Not Displayed​

Solution:

  1. Check if logging is enabled in plugin configuration
  2. Check time period filter (maybe set too narrow?)
  3. Clear cache: bin/console cache:clear

Token Usage Incorrect​

Note: Costs are estimates based on OpenAI prices.

For exact costs:

Performance with Many Logs​

Problem: With thousands of logs, the overview becomes slow.

Solutions:

  • Use filters (time period, agent, tags)
  • Delete old, no longer needed logs
  • Enable database indexes (for developers)

📊 Example Analysis​

Scenario: Monthly Report​

You want to know how successful your agent was last month.

Approach:

  1. Set filters:

    • Time period: Last month
    • Agent: Your product advisor agent

  2. Collect numbers:

    • Number of conversations: 2,847
    • Average tokens per conversation: 1,250
    • Total cost: approx. $2.30

  3. Analyze tags:

    • product: 1,200 (42%)
    • shipping: 650 (23%)
    • returns: 450 (16%)
    • payment: 320 (11%)
    • Other: 227 (8%)

  4. Identify top questions:

    • "What size fits me?" → 95 times
    • "Do you have this in blue?" → 78 times
    • "How long does shipping take?" → 67 times

  5. Derive actions:

    • ✅ Create FAQ "Size chart"
    • ✅ Mention color variants in product advisor
    • ✅ Include shipping information in Init Instructions

📋 Checklist: Use Logs Effectively​

  • Logging enabled in plugin configuration
  • Regularly check logs (e.g., weekly)
  • Use tags for categorization
  • Save frequent questions as FAQ entries
  • Monitor token usage
  • Identify and fix incorrect answers
  • Follow privacy policies
  • Regularly clean up old logs
  • Create monthly analysis
  • Use insights for agent improvement

Next Steps​

Now you understand how to monitor and analyze logs!

➡️ Knowledge Management - Build a FAQ database

➡️ Cost Management - Optimize costs

➡️ Back to Main Documentation