What Otis records
Feedback
The explicit reactions your app sends about an AI response or a tool call, and how Otis turns each one into evidence about a task.
Feedback is an explicit reaction to one specific operation. A user gives a thumbs-up to an AI response, rates it, or corrects it. An agent files a report about one of your tools. Your app sends each reaction through the SDK, and Otis attaches it to the task that contains the operation it is about.
For each piece of feedback, Otis works out whether it is positive or negative and how strong it is. This page explains those rules, so that you know which feedback changes a task's outcome and which is only recorded.
Purpose of feedback
Signals are what Otis infers from a user's messages. Feedback is what the user or the agent states outright. It is rarer than a signal, and it is more direct evidence of how the work went.
Feedback and tasks
Your app sends feedback together with the ID of the span it is about, such as the span for one AI response. Linking feedback to an AI response describes how to capture that ID.
Otis attaches the feedback to each task in the same session that contains that span. The ID must therefore be an Otis span ID. Feedback sent with an ID from your own system, such as a message ID from your database, is stored but doesn't attach to any task.
Otis reads feedback when it records a task, which happens after the session has been idle for 30 minutes. Feedback that arrives later may not be reflected in the task.
Kinds of feedback
Otis sorts each piece of feedback into one of four kinds. It checks them in the order below and uses the first that matches.
| Kind | How Otis recognizes it | Direction | Strength |
|---|---|---|---|
| Tool report | The feedback is named tool, or carries a severity. | Negative | Set by the severity |
| Correction | The type is edit, or the feedback includes the expected output. | Negative | How much of the AI's reply the correction changed |
| Rating | The feedback includes scores. | Positive at a mean of 0.5 or more, negative below it | How far the mean score is from the middle |
| Thumbs | The name is one Otis knows, listed below. | Set by the name | Strong |
Otis matches a name on the part after its last dot, so feedback.thumbs_up and thumbs_up are the same name.
- Positive names:
thumbs_up,up,positive,like,helpful,good,upvote - Negative names:
thumbs_down,down,negative,dislike,unhelpful,bad,confused,hallucination_suspected,report,flag
Feedback that changes the outcome
Feedback changes a task's outcome only when it is strong enough:
- A thumbs-up or thumbs-down always counts.
- A tool report counts at
blockingorfrictionseverity. Anitdoesn't. - A rating counts unless the mean score is exactly 0.5.
- A correction counts when it changed most of the AI's reply. If Otis doesn't have both texts to compare, the correction doesn't count.
Feedback that counts is treated like a signal. Positive feedback marks the task as a success, and negative feedback marks it as struggled. If a task has both, it is a success. Tasks describes where this sits among the other kinds of evidence.
Otis records every piece of feedback on the task, including feedback that is too weak to change the outcome.
Feedback that Otis can't read
- Names Otis doesn't know. Feedback with a name outside the two lists, and with no scores, severity or expected output, has no effect.
acceptandrejectare not on the lists. The exception is feedback sent with the typefeedback, which Otis treats as negative when it can't place the name. - Scores outside 0 to 1. Otis reads a mean of 0.5 or more as positive, so a rating sent as 1 to 5 stars is positive at every star. Convert scores to the range 0 to 1 before you send them.
- Comments. Otis stores the comment text and scans it for personal data. It doesn't read the comment to decide whether the feedback is positive or negative.
- Activity tasks. Feedback attaches to conversation tasks and tool tasks. An activity task doesn't take feedback.
Feedback from agents about your tools
When you instrument an MCP server that has two or more tools, the SDK adds a tool that lets the calling agent report a problem with a tool call. Each report has a category, a severity, a comment and an optional suggested change.
A report is always negative, with the strength set by its severity. A task that carries a report is also marked with the product_suggestion signal, whatever the severity. Agent feedback on MCP tools describes the report in full.
Feedback about Otis
Your team can also react to the insights Otis writes, for example by dismissing one. That reaction is about Otis. It is stored separately from your product's feedback, and it never affects a task.
Feedback in Otis
The effect of feedback shows in each task's outcome. In the task list of the data browser, the Signal filter for product_suggestion finds the tasks on which an agent filed a tool report. You can also ask Otis about feedback in chat, such as which tools agents report most often.
Related
- Feedback signals covers the SDK calls, the fields you can send, and how to carry a span ID to the browser.
- Signals covers the judgments Otis infers from messages, which decide outcomes in the same way.
- Tasks covers how Otis decides a task's outcome.