What Otis computes
Evidence
How Otis picks the specific cases it investigates and quotes, and what it checks before a quote appears in an insight.
Evidence is the set of specific cases behind a finding. It includes the quotes shown in an insight, with the sessions and tasks they came from. It also includes the recurring problems that Otis picks out of your telemetry as worth investigating.
This page explains how Otis chooses those cases and what it checks before showing them, so that you know how much weight a quote or a count can carry.
Purpose of evidence
A rate tells you that something happens and how often. It doesn't tell you what the problem looks like when it happens. A real example lets you check that a finding means what its headline says, and it gives your team something concrete to reproduce.
Recurring problems
Once a day, Otis groups three kinds of recent event and looks for groups that many users share.
| Kind | What is grouped | How Otis groups it |
|---|---|---|
| Errors | Spans that failed | By the type of exception and where in your code it was raised. Recurring errors describes the grouping. |
| Tool reports | Reports that an AI agent filed about one of your tools | By the tool, the category of problem and the severity. |
| Product requests | User messages that ask for something your product doesn't do | By the capability asked for. Otis groups requests that mean the same thing. |
Selection of groups
Otis investigates the groups that affect the most users. It also gives weight to a group that is new, and to one that keeps recurring. A group that only one or two users share isn't investigated.
Otis picks a few groups of each kind each day. A group that has been picked isn't picked again for a while.
Being picked starts an investigation. It doesn't guarantee an insight. Otis publishes only if the investigation confirms that the group describes one real problem and that its numbers hold up.
Quotes in an insight
An insight can include up to five quotes. Each appears under the heading "Evidence".
Quote checks
Every quote must match stored text word for word. Otis compares each quote with the text it cites. A changed word fails the check.
A quote that fails is sent back to be corrected. A quote that still fails is removed, and the insight is published without it.
Parts of a quote
- The text. It links to the span it came from.
- The speaker. A made-up name stands in for the user. Otis never shows a real name, and shows "Anonymous" when the span has no user.
- The session and the task. Each links to its page in the data browser.
- Labels, such as the user's segment, the task's outcome, and the number of the session. Otis takes these from what it has recorded. The AI model that writes the insight doesn't write them.
Personal data in a quote is replaced with placeholders before Otis stores the text. Privacy describes the redaction.
Reading evidence
- A quote shows what a problem looks like. It doesn't show how large the problem is. Take the size from the counts in the insight.
- A quote may not be a user's words. Otis can quote any text on the span it cites. That includes your AI's reply and the output of a tool.
- Check the unit of a count. A recurring problem is reported with its number of users and its number of occurrences. One user can account for many occurrences.
- "At least" marks a lower bound. For a very large group, or a kind of request that Otis is still processing, the true number of users may be higher.
- A group is Otis's own grouping. Two errors with different messages can share a group, and one bug can be split across two. An investigation can conclude that a group mixes unrelated problems.
- Links go stale. Otis keeps spans for 90 days. After that, a quote keeps its text and its link to the span stops working.
Evidence in Otis
On an insight's page, quotes appear in an "Evidence" block inside the text, next to the point they support. In Slack, each quote appears as a quoted line. An insight can also include a session narrative when one session illustrates the finding.
To see the recurring problems Otis has found, ask it in chat for the most common errors, tool reports or product requests.