How Otis knows your product
Data health and gaps
What Otis accepts and drops when telemetry arrives, how it notices a break, what it waits for in a project's first days, and how long it keeps each kind of record.
Every analysis Otis produces depends on the telemetry behind it. That telemetry has a first day, it can stop or change shape, and it expires. This page explains what Otis does in each case, so that you can tell a real change in your product from a change in the data.
Purpose of data health checks
A fall in a number can mean that fewer users did something. It can also mean that an event stopped arriving, that a deploy renamed it, or that the period reaches back before your project had data. Otis checks for some of these, and for others it relies on you. This page says which.
Ingest limits
Otis applies limits to each request your app sends. Most of them act silently.
| What arrives | What Otis does | How you find out |
|---|---|---|
| A span that started more than 48 hours ago | Drops that span and keeps the rest of the request | You don't. The request succeeds. |
| A span that starts more than an hour in the future | Drops that span | You don't. The request succeeds. |
| A text value over 64 KB | Cuts it and marks it as truncated | You don't |
| A span with more than 128 attributes | Removes the excess | You don't |
| More than 10,000 requests in a minute for one project | Rejects the request | A warning in your app's console |
| A missing, unknown or revoked API key | Rejects the request | A warning in your app's console |
| A request made with a development key | Keeps the spans for one day and doesn't analyze them | The Development row on the Ingestion tab |
A property named like personal data, such as email or name | Removes the property | A verification warning if every property was removed |
Two of these deserve attention:
- Watch your app's console for rejected requests. The Ingestion tab counts what Otis stored. A rejected request shows as a warning from the SDK, with its status code.
- Stay under the rate limit. The SDK retries when Otis is temporarily unavailable. It doesn't retry a request that was refused for the rate limit or for a bad key.
The Ingestion tab
The Ingestion tab in your project's settings shows the spans received for each environment, by hour or by day, with the peak rate. It shows two kinds of error:
- Error spans are your own operations that failed, such as an AI call that returned an error. They are part of your telemetry. Errors describes them.
- Ingestion errors, in red, are spans that Otis received and couldn't process.
Telemetry checks
Otis checks each project's production telemetry about once an hour, and more often in a project's first few days. It looks for three things:
- A drop in volume. The last complete hour has far fewer spans than the same hour yesterday. Otis doesn't flag a drop that looks like ordinary quiet, such as a weekend hour compared with a weekday.
- A spike in failed operations. A much larger share of the last hour's spans failed than in the same hour yesterday.
- Processing errors. Otis failed to process a noticeable share of recent spans.
Each check needs enough spans in the hour it compares with. A very small or new project therefore gets no verdict from these checks.
Otis also folds in the result of verification, which checks that your events carry user and session IDs.
Check results in Otis
- The project's home page card shows "Telemetry degraded" for a problem and "Review telemetry" for a warning.
- The daily briefing adds a line when events failed to process in the last 24 hours.
- A chat with Otis can list the open problems with a suggested fix for each.
If a drop in telemetry would be serious for you, monitor your own export errors as well.
Breaks in telemetry
When Otis detects a drop in volume or a run of processing errors, it records the period as a break in your project's changes. A break means that numbers from before and after it may differ because of the telemetry and not because of your users.
Otis is instructed to check whether the period of a finding spans a break, to say so if it does, and never to offer a break as the cause of a change in behavior. The same applies when you ask it a question in chat.
A rate or a count of returning users that spans a break can be affected by it, so check the dates of an insight against any outage you know of.
A very short break isn't recorded.
The first days of a project
Otis waits for enough data before it produces each kind of result. Until then the result is absent, and Otis doesn't show a placeholder for it.
| Result | Minimum |
|---|---|
| Any analysis | Production telemetry in the last 24 hours |
| A session and its tasks | A session that is large enough to analyze and has been idle for 30 minutes |
| Intent clusters | Enough recent tasks to group |
| A description of a user | Several sessions from that user |
| Built-in segments | A funnel success or a reported task success in the project |
| A user counted as regular | Activity in two of the last four weeks, so not before the second week |
| A churned or inactive account | 28 days without activity |
| Numbers in the briefing | Enough history for the number's period, and its minimum sample |
| Tested findings | Seven days of telemetry, and enough users or accounts on each side of a comparison |
| A test of a deploy or a flag variant | Enough users in each group |
When you ask Otis a question that reaches back further than your data, it is told how many days the answer really covers and is instructed to describe it that way.
Instrumentation changes
- A new event is stored as soon as it arrives. Otis picks up a declared match to a measurement within minutes, and looks for other matches itself every 12 hours.
- A renamed event stops matching its measurement. Verification raises a problem when the old name has gone silent while a similar name on the same service is arriving, and names the new one. Update the match through the setup agent.
- A changed definition of a funnel, a measurement or your investment events is recorded as a change. Figures from before the edit and after it measure different things.
- A new version of your app is recorded as a deploy the first time Otis sees its version.
- A change to how you assign user or session IDs isn't detected. Every user gets a new identity, and Otis reads the old users as gone and the new ones as arrivals. Users, groups and accounts describes how identity is derived.
None of these rewrites what Otis has already stored.
Old telemetry
Otis drops spans that started more than 48 hours before they arrive. You can't upload your history through the normal SDK path.
An import of older telemetry needs Otis to prepare your project first, and it reaches back 90 days at most. Contact Otis before you send it. Telemetry sent before the project is prepared is dropped, and the request still succeeds.
Insights that Otis produces from imported history aren't posted to Slack and don't appear in the briefing.
Sampling
Otis stores every span it accepts. It doesn't sample your telemetry.
Late telemetry
- A span is counted on the Ingestion tab when it arrives, whatever time it says it started.
- A span is filed under the time it started everywhere else.
- A session is processed after 30 minutes of quiet. If more spans arrive for it later, Otis processes the session again.
- Daily counts are rebuilt for the last three days, so a span that arrives up to three days late is still counted.
Retention
Retention is the same for every project.
| Record | Kept for |
|---|---|
| Spans, signals and tasks | 90 days |
| Development spans | 1 day |
| Sessions, with their summaries and narratives | 365 days |
| Measurements, funnel figures and segment membership | 365 days |
| User, account and document properties | 365 days after their last update |
| Lifecycle stages and movements | 2 years |
| Cohort membership | No limit |
| Insights, chats and briefings | No limit |
Some things stop working before the record behind them expires:
- A link from a figure in an insight to the users or tasks behind it works for 30 days.
- A chart in an insight loses its data after 90 days.
- A quote in an insight keeps its text. Its link to the span stops working after 90 days, and its link to the session after 365.
- A session older than 90 days still shows its summary and narrative. Its spans and tasks are gone.
Data deletion and export
To delete a project, to delete one user's data, or to export your data, contact Otis. Privacy covers this alongside redaction and hashing.
Related
- Spans covers what a span holds and how Otis handles it when it arrives.
- Onboarding covers verification and when each result first appears.
- Changes covers the record Otis keeps of breaks, deploys and edited definitions.
- Statistical tests covers the minimum samples for a tested finding.