Skip to main content
Admin › Evaluation › Tracing
Tracing records every step of AI request processing.
  • What documents were searched, what tools were invoked, what prompt was sent to the LLM — every step is transparently visible.
Tracing search screen

Admin > Evaluation > Tracing — search traces by Chat ID or Message ID

Example — without tracing vs. with tracing
Agent answered “I can’t find that information”
Tracing is a licensed feature. Requires a license with trace feature enabled.

​
Tracing Concepts

Processing a single user message involves multiple steps.
  • Tracing records all these steps in a Trace > Run hierarchy.

​
Tracing Entry Methods

All three paths lead to the same Evaluation › Tracing screen — buttons ① and ② navigate automatically when clicked, while ③ searches directly by ID.
The standard path where an admin picks a specific conversation from the operations screen and enters tracing.
1

Open conversation logs

Go to Monitoring › Conversation Logs.
2

Select a conversation

Click the conversation (request) row you want to investigate to open its detail.
Entering tracing via the Trace button in conversation logs

Expanding a conversation shows a 'Trace' button on the right

Arrive at Evaluation › Tracing

Clicking the Trace button in the detail automatically opens the Evaluation › Tracing screen.

Destination · Evaluation › Tracing

No matter which of the three paths you enter through, the destination is a single screen: Evaluation › Tracing below.
Evaluation › Tracing — list of message cards

Evaluation › Tracing arrival screen — search list (message cards)

Once you arrive, pick a trace from the search list (message cards) and open the detail view.

​
Searching Traces

Searching by Chat ID or Message ID on the Evaluation › Tracing screen produces a list of message cards like the screen above.
  • Each card is a summary of one user message’s trace, and clicking a card opens the detail view.
The search-criteria and card-item tables are collapsed below.
Search retrieves all traces matching the entered ID with no time limit. (There are no separate filters such as date or status.)

​
How to Read the Trace Detail View

This is the heart of tracing.
  • Here you see what steps a message went through and what went into and came out of each step.
(Detailed notations such as badges and symbols are collapsed inside each tab — expand them when you encounter an unfamiliar marker.)
Trace detail modal — Run tree on the left, detail panel on the right

Trace detail — pick a step in the left Run tree and its inputs/outputs appear on the right

The screen is split into the left Run tree (lists processing steps in order and hierarchy, with total latency and total tokens at the top) and the right detail panel (inputs, outputs, and tokens of the selected step). A single message typically goes like this — the agent (CH) repeats LM reasoning and tool calls to gather data, and finally final_answer (LM) writes the answer.
Each line (Run) is shown as colored badge + name + ● status + latency. Knowing just the frequently seen badges is enough to read the tree. (For the full list including colors, expand the legend below.)
  • ● Status — green means OK, red means an error at that step.
  • Adjacent time — that step’s latency. The step with the largest value is the bottleneck.
Auxiliary tasks such as auto-generating the title, tags, and search query are recorded separately from the response in the background_tasks trace.
The detail tree distinguishes step types by colored badges. (On search-result cards, they are grouped under labels like Agent · LLM · Embed.)
The trace’s overall status is Error if any included Run is Error; if there’s no Error but a Running exists, it shows as Running.

​
Reading in Two Phases

Reading the agent’s response in two phases helps you find the cause quickly.

​
Debugging Points

Check active_capabilities (tools/features available at that moment) in the Inputs of Phase 1’s LM reasoning Run.
  • Desired tool not in the list → that feature (Knowledge Base, DB, etc.) isn’t connected to the agent
  • Tool present but not called → the model judged low relevance between the question and the tool. Make the tool/feature description more specific
Check sources_count and source_names in the Outputs of the final_answer (LM) Run. (Search results are contained in this output, not in a separate Run.)
  • sources_count is 0 → zero search results. A KB document is missing, or search settings (Top K, Reranker, etc.) need review
  • Sources exist but the answer is off → check what was passed in the same Run’s Inputs and adjust the answer prompt
Click the TL (Tool) Run marked with a red ● and check both the error content in Outputs and the parameters passed in Inputs.
Compare the latency next to each step in the Run tree. The longest step is the bottleneck.
  • LM slow → consider switching to a faster model
  • TL/EM slow → check tool/search settings or external services
  • GD (guardrail) slow → disable LLM judgment or switch to a faster model

​
Trace Analysis Report

A feature that analyzes trace data with an LLM to automatically identify the root cause of a problem.
1

Start analysis

Click the Trace Analysis (Analyze Trace) button at the top of the trace detail modal.
The analysis model list excludes models with base_model_id (custom models), preset models, and arena models. Only base models are selectable.
2

Review analysis result

The LLM comprehensively analyzes trace data + agent settings + conversation history + KB/DB/guardrail settings to generate a structured report.
Entering a problem description enables analysis focused on that context. Example: “KB found documents but they weren’t reflected in the answer”
3

Save/share report

When a previously analyzed report exists, the “View Report” button shows it directly without re-analysis.

​
Trace Management

​
Permissions

​
Data Cleanup

Old traces are cleaned up via the /api/traces/cleanup developer API.
  • Bulk-delete traces before a specific timestamp (in milliseconds, ms).
Trace deletion is irreversible. Download necessary analysis reports before deleting.

​
Use Cases

  1. Click the View Tracing button on a chat message
  2. Select the final_answer (LM) step in the left Run tree
  3. In Outputs, check sources_count · source_names to see whether search results were reflected
  4. In Inputs, check what was passed to the model
  5. Generate a Trace Analysis Report to auto-identify the root cause
  1. Open the trace of a slow response
  2. Compare the latency next to each step in the Run tree
  3. Identify the longest step (e.g., tool 0.8s, LLM 3.2s)
  4. Optimize that step (adjust search settings, change the model, etc.)
  1. Open the suspected trace
  2. Select the TL (Tool) Run marked with a red ●
  3. Check the error content in Outputs
  4. Verify the passed parameters in Inputs
  1. Check overall trace usage via total tokens at the top of the detail view
  2. Compare per-LM Run Token Usage (Input/Output/Total) in the Run tree
  3. Check the token ratio of Phase 1 (agent run) vs. Phase 2 (final_answer)
  4. Identify unnecessarily large prompts or repeated calls

​
FAQ

Yes — when message tracing is enabled (default: enabled), all AI requests are auto-recorded. No additional setup needed.
Default retention is 30 days. Admins can change settings or manually clean up.
Trace recording is asynchronously processed in the background, so it has nearly no impact on response speed.
Yes — trace analysis is a separate LLM call, tracked in usage as trace_analysis. Analysis only runs when manually triggered.

Guardrail Logs

Dedicated log for guardrail detection events

Auto-Evaluations

Automatic evaluation results for agent response quality

Usage

Token usage and cost analysis