Sign-ups were down 20% last week. Somebody asks what happened, and the answer turns into an afternoon: scrolling the deploy log, checking Slack, opening three dashboards, and arguing about whether Tuesday's release could really have done that.
This guide is a repeatable method for answering "did that deploy change our conversion rate?" quickly and honestly. It works with any analytics tool. At the end we'll show how Ometra shortens each step.
Step 1: Put your releases where your metrics are
You can't line up a change with a release if the releases live somewhere else. The single most useful habit is to record every production deploy as a marker on your analytics timeline, automatically, from your CI pipeline.
Doing it by hand doesn't last. People forget, and the deploys you forget are exactly the rushed hotfixes you'll later want to see. Automate it as a step after a successful deploy, and include:
- A short label a human can read ("Checkout v2", not just a commit hash)
- The commit SHA so engineers can jump straight to the diff
- The environment, so staging deploys don't clutter production charts
Do the same for changes that don't go through CI: CMS publishes, pricing changes, campaign launches, and outages. A chart annotation is enough. The goal is one timeline that holds everything that could have moved the number.
Step 2: Pick the metric that actually changed
"Conversions are down" can mean three different things:
- Fewer conversions, because fewer people visited
- A lower conversion rate, because the same visitors converted less often
- Both
A release can easily cause the second. It rarely causes the first, unless it broke something like search indexing or page speed badly enough to cost traffic. Before blaming a deploy, check whether visits changed too. If traffic dropped 20% and conversions dropped 20%, the conversion rate is flat, and you're looking at a traffic problem, not a product problem.
Be precise about the denominator, too. A sign-up rate measured against all visitors will move whenever your traffic mix changes. Measured against visitors who reached the sign-up page, it tells you much more about the page itself.
Step 3: Compare like-for-like windows
The most common mistake is comparing "the three days after the deploy" with "the three days before". Traffic and conversion behavior swing by day of the week, so any window that mixes weekdays and weekends differently will show a change that has nothing to do with your release.
Better comparisons:
- Full weeks against full weeks. Seven days after the release against the seven days before, both starting on the same weekday.
- Skip the deploy day itself. It's half old version, half new.
- Same week last year or last month, as a sanity check for seasonality, if you have the history.
If the change only shows up in one narrow window and disappears when you widen it, be suspicious.
Step 4: Rule out the look-alikes
Plenty of things move a conversion rate in the same week as a release. Before you credit or blame the deploy, check for:
- Traffic mix shifts. A newsletter, a social post, or a campaign that brings in lots of low-intent visitors will drag the conversion rate down even if nothing on the site changed. Look at conversion rate by source: if each source converts the same as before but the mix changed, the release isn't your answer.
- Seasonality and holidays. Long weekends and end-of-quarter weeks behave differently.
- Outages. A payment provider or login service having a bad day looks exactly like a broken release.
- Broken tracking. This is the sneaky one. Releases often rename a form, change a button ID, or move a thank-you page, and the measurement breaks while real conversions carry on. If a conversion goal drops to near zero overnight, check that it's still firing before you roll anything back.
Step 5: Segment to find the "where"
A real product regression usually leaves fingerprints. Break the change down by:
- Device. A drop that only appears on mobile points at layout or a mobile-specific bug.
- Page. If only one landing page converts worse, look at what the release changed there.
- Browser. A drop isolated to one browser often means a JavaScript error.
- Source. If only paid traffic converts worse, check whether the landing page those ads point to changed.
Then look at the page itself. A heatmap showing that people no longer reach, or no longer click, the element you moved is strong supporting evidence.
Step 6: Check whether the change is bigger than the noise
Small numbers bounce around. If you get 40 sign-ups in a normal week, getting 32 the next week is well within ordinary variation — a rough rule of thumb is that a count of n naturally wobbles by about two times √n — so roughly ±13 on a base of 40. A drop from 40 to 32 might be nothing at all.
The same drop from 4,000 to 3,200 is a different story: that's far outside normal variation, and it deserves attention today.
You don't need a statistics degree for this. Ask two questions:
- Is the change bigger than the week-to-week swings I normally see?
- Does it persist for more than a few days?
If the answer to both is yes, keep going. If not, keep watching.
Step 7: Confirm the cause
Everything up to now gives you correlation: a change happened around the same time as a release, in the places that release touched. That's a strong lead, but it isn't proof. To confirm cause:
- Roll it back, or turn it off. If you ship behind a feature flag, switching the change off and watching the metric recover is the fastest real evidence you can get.
- Run an A/B test. If the change matters enough to argue about, it matters enough to test against the old version with randomly split traffic.
- Roll out gradually. Shipping to a percentage of users first gives you a built-in comparison group.
Be wary of any tool or person that tells you a deploy caused a change based on timing alone.
A checklist you can reuse
When a conversion metric moves:
- Find the releases and annotations in the window.
- Check whether visits changed, or only the conversion rate.
- Compare full weeks, starting on the same weekday, excluding the deploy day.
- Rule out traffic mix, seasonality, outages, and broken tracking.
- Segment by device, page, browser, and source.
- Compare the size of the change with normal week-to-week swings.
- Confirm with a rollback, a feature flag, or an A/B test.
How Ometra shortens this
We built release tracking into Ometra because we were tired of doing step 1 by hand:
- Deploy markers from any CI. One request after each deploy puts a marker on your traffic chart with its label, commit, and environment. Webflow sites can connect once and have every publish pinned automatically.
- Anomaly alerts with context. When a conversion rate drops or a source vanishes, the alert names the release that shipped around the same time — written as a coincidence to investigate, never as a verdict.
- Heatmaps for the affected page, so you can check step 5 without another tool.
- Annotations for everything that isn't a deploy.
Here's what the deploy step looks like in GitHub Actions:
- name: Notify Ometra of deploy
env:
OMETRA_DEPLOY_SECRET: ${{ secrets.OMETRA_DEPLOY_SECRET }}
run: |
curl -X POST "https://ometra.io/api/sites/YOUR_SITE_ID/deploys" \
-H "Authorization: Bearer $OMETRA_DEPLOY_SECRET" \
-H "Content-Type: application/json" \
-d "{\"environment\":\"production\",\"sha\":\"$GITHUB_SHA\",\"label\":\"Deploy ${GITHUB_SHA:0:7}\"}"
The full setup, including every optional field, is in the deploy tracking docs.
FAQ
How do I know if a deploy hurt conversions? Mark your releases on your analytics timeline, compare full weeks before and after, rule out traffic-mix changes and broken tracking, segment by device and page, and confirm with a rollback or A/B test before concluding the deploy caused it.
What's the difference between a conversion drop and a traffic drop? A traffic drop means fewer visitors; a conversion-rate drop means the same visitors converted less often. Releases more often cause the second. Check visits first.
How long should I wait before judging a release? At least one full week, so you compare complete weekly cycles. Low-traffic sites may need two to four weeks for the signal to rise above normal variation.
Can analytics prove a deploy caused a change? No. Analytics shows what coincided with a release. Proving cause takes a controlled comparison, such as an A/B test, a feature flag toggle, or a gradual rollout.