What Otis records
Signals
The judgments Otis makes about individual messages, such as frustration or delight, and how they add up on a task.
A signal is a judgment that Otis makes about one message, such as whether the user who wrote it was frustrated. Otis infers signals from the text. Your app doesn't send them.
There are seven signals. Six are read from what the user wrote, and one is read from what your AI replied. This page explains what each signal means, how a message is classified, and how signals affect a task.
Purpose of signals
Most users never rate a response or leave a comment. What they write still shows how the work is going: a user who repeats a request three times is having a different experience from one who says the result saved them an hour. Signals let Otis read that evidence from every conversation.
Signals are the inferred counterpart to feedback, which is the explicit reaction a user or an agent sends through your app.
The seven signals
| Signal | Read from | What it means |
|---|---|---|
frustrated | User's message | Frustration or annoyance, including repeated attempts to get help. |
confused | User's message | The user is lost, contradicted or overwhelmed. |
delighted | User's message | Satisfaction, happiness or gratitude. |
value_acknowledgment | User's message | The user says the AI saved time, solved a problem or produced something usable. |
escalation_request | User's message | The user asks for a human, or gives up on the AI. |
product_suggestion | User's message | A concrete suggestion or complaint about your product. |
error_or_tool_issue | AI's reply | The reply reports an error, a failed tool call, or that the AI could not do what was asked. |
Each signal is defined narrowly. A plain acknowledgment such as "OK, got it" isn't delight, and a neutral report of a problem isn't frustration.
Otis uses the same seven signals for every project.
Message classification
An AI model reads each message and judges each signal separately, so one message can carry several signals or none. A signal describes that one message, and not the conversation around it.
For each signal the model answers yes, weak or no. It answers weak when the message could be read either way.
Classification runs every few minutes. Otis waits until a session's messages have been classified before it divides the session into tasks, so a task has its signals when it first appears.
Messages Otis skips
Otis doesn't classify a message that is:
- very long
- blank, or made up only of redaction placeholders
- not prose, such as JSON, HTML, a table of values or a stack trace
A skipped message gets no signals.
Signals on a task
Only conversation tasks carry inferred signals. Activity tasks and tool tasks have no messages for Otis to read.
For each task, Otis counts the messages on which each signal fired, and keeps the confident and weak counts apart.
Five of the signals decide the task's outcome:
- Positive.
delightedorvalue_acknowledgmenton at least one message marks the task as a success. - Negative.
frustrated,confusedorescalation_requeston at least one message marks the task as struggled.
A weak answer counts the same as a yes here. If a task has both positive and negative signals, the positive ones decide and the task is a success. Tasks describes where signals sit among the other kinds of evidence.
product_suggestion and error_or_tool_issue never change a task's outcome.
Reading signals
- No signal doesn't mean a neutral message. A skipped message also has no signals.
- Weak is a label. It means the message could be read either way. It doesn't measure how strongly the user felt.
error_or_tool_issuerecords what the AI said. It fires when the reply reports a failure. An operation that failed in your telemetry is a separate record, described under Errors. Either can occur without the other.
Signals in Otis
In the data browser, the span list has a Signals column and a filter for each signal. A filled dot is a yes and an outlined dot is a weak. The task list has Frustration and Delight columns that use the same dots, and a Signal filter that matches only tasks with a yes.