What Otis computes
Measurements
The named measurements Otis takes of each user, how one becomes active, and what its values mean.
A measurement is a named set of values that Otis computes for each user about one aspect of how they use your product. Examples are whether users accept what your AI produces, how far their work gets through a funnel, and how long they take to reach first value. The Otis app calls a measurement a lens.
This page explains which measurements exist, how one becomes active for your project, and how to read its values. Otis doesn't combine measurements into one overall score. The last section explains what it does in place of one.
Purpose of measurements
Telemetry alone doesn't say which of your events mean that a user accepted an answer, published a document, or invited a teammate. A measurement supplies that link. It names the events it needs, and once those events are identified in your product, Otis can compute the same values for every user.
Measurements are also how Otis compares groups. An insight that says one group of users differs from another is usually comparing a measurement between the two.
The measurements
Measurements that are always on
Otis computes these from the telemetry every project sends. They need no setup.
| Measurement | What it measures |
|---|---|
| Conversational AI Signals | The share of a user's conversation tasks that succeeded, and whether the user showed frustration. |
| User Actions | How often a user saves, exports, shares, copies or invites, and whether they return after a frustrating exchange. |
| Signal Dynamics | Whether a user's messages showed frustration, confusion or delight. |
| Session Quality | How long and how deep a user's sessions are, and how often a session covers more than one task. |
| Retention Health | The shares of your users who are active, declining and gone, for the project as a whole. |
| Cluster Dynamics | The number of intent clusters, how concentrated tasks are among them, and how they are growing, for the project as a whole. |
| Explicit Feedback | Nothing by itself. It is the instruction to send feedback, which Otis reads on each task. |
Measurements you choose
These need events that only your product can supply. Each is active once your project has kept it.
| Measurement | What it measures |
|---|---|
| Output Acceptance | Whether users accept, edit, regenerate or abandon what your AI produces. |
| Autonomous Completion | Whether an AI agent finishes its task without a person stepping in. |
| Downstream Outcome | Whether a user's AI activity is followed by a business result, such as a booking, within a set number of days. |
| Knowledge Corpus | How a knowledge base that your AI draws on grows and is used. |
| Funnel Tracker | Progress through the funnels you declare. |
| Trust Calibration | Whether users review AI output before accepting it, and how often they override it. |
| Team Activation | How widely the product is adopted inside each account. |
| Value Realization | How quickly a user reaches the event that marks first value, and whether they return after it. |
| Autonomy Spectrum | Which level of AI autonomy users choose, and how that changes. |
| Counterfactual Value | How the AI-assisted way of doing a job compares with the manual way. |
| Engagement Patterns | When users are active across days and hours, and how they recover after a gap. |
Otis also computes whether a user returns after their first value event. This needs the first value event from Value Realization.
Activation of a measurement
- Otis recommends it. When a coding agent analyzes your codebase during setup, it scores how well each measurement fits your product.
- You keep or skip it. The chat in the Otis app shows the recommendations in a panel titled "Review lenses" during onboarding. The setup agent asks the same question if you haven't answered it there.
- The setup agent instruments it. It adds the events the measurement needs and records which of your events play which part.
- Otis verifies it. After you deploy, Otis checks that the expected events are arriving and marks the measurement as verified.
A measurement has one of six statuses: recommended, approved, skipped, in progress, instrumented or verified. Agent skills describes the setup steps.
A recommended measurement is already active. If the events it needs are present, Otis computes it before you approve it. Skipping a measurement is what turns it off.
Event matching
Each measurement names the parts it needs, such as "the AI generated something" and "the user accepted it". Otis has to know which of your events plays each part.
- Declared. The setup agent records the match when it instruments your code. A declared match is used at once.
- Inferred. Otis also looks for matches itself in your recent telemetry. It accepts an inferred match only when it is confident, and the event is common enough across your users.
Otis never infers the events that decide an outcome, such as acceptance, completion, or the success and failure stages of a funnel. Those must be declared. A wrong guess there would change what counts as success.
A measurement with no matched events produces no values.
Measurement values
Otis computes measurements about every 12 hours for a project that is sending telemetry. It covers every identified user who was active in the last 30 days.
Each value describes the last 30 days as of the run. Otis stores one value for each user, measurement and day, and keeps them for a year. Values on consecutive days cover almost the same 30 days, so they change slowly and shouldn't be added together.
A user with no data for a measurement has no value for it. A missing value doesn't mean zero.
Reading measurements
- Many values are scores between 0 and 1. Otis scales several quantities so that they can be compared. Time to first value, for example, is stored as a speed score in which 1 means the same day and 0 means a week or more, or never. Ask Otis what a value measures before you compare it with a figure from another tool.
- Some measurements only cover AI conversations. Conversational AI Signals and Signal Dynamics are computed from messages between users and your AI. A user who only clicks through your product has no values for them.
- Project-wide measurements have one value. Retention Health and Cluster Dynamics describe the whole project, so every user carries the same value. They can't be used to compare users.
- A measurement is only as good as its matched events. If an event is matched to the wrong part, every value built on it is wrong. Otis records a change when a measurement's definition is edited. Changes describes this.
No overall score
Otis doesn't compute a single success score or affinity score for a user. It places each user in a segment with two fixed yes-or-no rules, which read your scoring funnel, reported task successes, investment events and regular activity. Segments and cohorts describes the rules.
Measurements in Otis
You meet measurements in three places:
- During onboarding, in the "Review lenses" panel in chat.
- In insights, where Otis compares a measurement between groups of users.
- In answers from Otis, in chat, in Slack, or through the MCP server. A coding agent can also read the list of measurements and their status through the MCP server.
Related
- Agent skills covers the setup steps that recommend, instrument and verify measurements.
- Funnels covers the measurement that most other analysis builds on.
- Segments and cohorts covers how Otis groups users.
- Signals and Feedback cover the evidence behind the measurements that are always on.