Changes

A change is a dated record of something that happened to your product, such as a new build going live or a feature flag variant reaching users. Otis keeps these records on a timeline for each project. It writes a record whether or not the change turns out to affect anything.

Otis records five kinds of change today: deploys, feature flag variants, model changes, edits to your measurement definitions, and breaks in your telemetry. This page explains how each kind is detected and what its record holds, so that you know what the timeline covers and what it leaves out.

Purpose of the change record

When a metric moves, the first thing to find out is what changed around that time. A timeline of changes gives Otis and your team dated events to compare behavior before and after.

A change record states that something happened. It makes no claim about the effect. Otis analyzes the effect of a change separately, after the record exists, so a deploy that changed nothing still appears on the timeline. Candidate causes describes that analysis.

Kinds of change

Otis detects deploys, flag variants and model changes from your telemetry. This detection runs once a day, so a change can take up to a day to appear.

Deploys

A deploy record says that a version of a service started running in an environment. Otis writes one the first time it sees spans from a new combination of environment, service and version. The record is dated to the first span Otis saw from that version.

The version comes from the SDK. If you set the release option, the SDK uses that value. Otherwise it reads the commit that your hosting platform provides, on Vercel, Netlify, Railway, Render, Heroku, Fly.io, Cloud Run and GitHub Actions. If your spans carry no version, Otis can't detect deploys from telemetry.

If you connect the GitHub App and your repository creates GitHub deployments, Otis also records each deployment and whether it succeeded.

Feature flag variants

A flag record says that users started receiving a variant of a feature flag. Otis writes one the first time it sees a variant, and the record states how many users had received it. Feature flags describes how your app sends flag assignments.

Otis sees only the variants that users received. It has no record of who changed a flag, what the previous variant was, or what percentage of users the flag targets.

A flag with more than 100 distinct variant values is quarantined. Otis stops tracking that flag and adds a record that says so. This usually means the variant holds a value that differs for each user, such as an ID.

Model changes

A model record says that a service started using an AI model it had not used before. Otis reads the model name from your AI calls, and the record names the model that the new one followed.

Three rules decide what counts as a model change:

• The service must already use a different model. The models in use when you first connect Otis are not recorded as changes. A model added alongside an existing one is recorded, and so is a full replacement.
• Both models need more than a handful of calls. This keeps a single test call from appearing as a change.
• The record is dated to a day. It carries the first day the new model appeared, without a time of day.

Measurement definitions

A definition record says that your team changed how Otis measures your product. Examples are the stages of a funnel, and the events that mark a task as succeeded or failed. Otis writes the record when the definition is edited. Edits to one definition that are made close together are combined into one record.

A metric computed before such an edit and the same metric computed after it use two different definitions. A step in the metric at that date may come from the edit, with no change in how users behaved.

Breaks in telemetry

A break record says that your telemetry stopped being comparable for a period. Otis writes one when telemetry from your project stops arriving, or when Otis fails to process a large share of what arrives. The record gets an end time when the telemetry recovers.

Otis never offers a break as the cause of a metric movement. A break warns that numbers from either side of it may differ because the telemetry changed.

The change record

• Type. One of the five kinds on this page.
• Time. When the change happened, and when Otis detected it. A break also has an end time.
• Scope. The environment, service, version, flag or model that the change applies to.
• Source. Whether Otis detected the change from telemetry, received it from GitHub, or recorded it from an edit your team made.
• Summary. A one-line description, such as the version and the service it was deployed to.
• Details. Fields that depend on the type, such as the variant and user count for a flag, or the earlier and later model for a model change.

Reading the timeline

• Each change is recorded once. A version, a flag variant or a model gets a record the first time Otis sees it. Returning to one that Otis has seen before adds no record, so a rollback to an earlier version doesn't appear as a new deploy.
• The first version and the first flag variants are recorded as changes. When you first connect Otis, the earliest deploy and flag records show when Otis started seeing them. They don't show when you shipped them.
• One deploy can appear twice. A deploy that Otis detects from telemetry and the same deploy reported by GitHub are kept as two records.
• Other events are not on the timeline. Otis records only the five kinds on this page. An incident, a pricing change or a marketing campaign isn't recorded.

Changes in Otis

To see changes, ask Otis what changed, in the Otis app, in Slack, or through the MCP server. Otis lists the changes from the last 14 days unless you ask for a different period.

Otis also keeps a daily summary of recent changes. It reads that summary as context when it answers your questions.

Related

• Feature flags covers how your app sends flag assignments, and how to name variants so that a flag isn't quarantined.
• GitHub App covers connecting a repository.
• Tasks covers the events that mark a task as succeeded or failed, which are one of the definitions that produce a change record when edited.