There is an email running in your Klaviyo account right now that looks healthy and does nothing. We know because we caught one, measured it for eight weeks, and recommended its retirement.
This note walks through that measurement in enough detail that you can judge it for yourself: the design, what the dashboard said, what the controlled read said, and the strongest argument against our conclusion. Client details and figures are altered to preserve confidentiality; the design of the experiment and the direction of every result are exactly as run.
The experiment
A DTC apparel brand, and a pool of high-value one-time buyers selected by 12-month forward CLV before any email was sent. The pool was randomized into three arms:
| Arm | What it received | What it cost |
|---|---|---|
| Control | The brand's normal marketing calendar, nothing extra | Nothing |
| Reminder | The calendar, plus a well-crafted extra email with no offer | Attention only |
| Offer | The same extra touch, carrying a modest second-purchase offer | The discount, when redeemed |
Read dates and stopping rules were registered before launch. The primary endpoint was the change in 12-month forward value per customer versus control, read weekly with a 95% confidence interval. Not opens. Not clicks. Not attributed revenue. The change in what these customers are expected to contribute over the next year.
What the dashboard saw
The reminder performed. Opens were fine. Clicks were fine. Its conversion rate landed within a point of the offer arm, week after week. On any engagement report, this email was doing its job. No one reviewing a Klaviyo dashboard would have flagged it. Most teams would have called it a keeper, and some would have scaled it.
Here is the same eight weeks, viewed the way the trial viewed it:
What the measurement saw
The targeted offer built measurable value: +$5.20 of net forward value per customer by the final read, with a 95% confidence interval of +$1.30 to +$9.10. The interval pulled clear of zero and stayed there.
The reminder built nothing. Not a small something. Nothing. Its lift was statistically zero in every one of the eight weekly reads, the confidence interval straddling zero from the first read to the last. Side by side:
| Reminder | Offer | |
|---|---|---|
| Opens and clicks | Healthy | Healthy |
| Conversion vs. control | Within a point | A couple of points up |
| Net forward value per customer | $0.00, CI spans zero in all 8 reads | +$5.20, CI +$1.30 to +$9.10 |
| Verdict | Retired | Scaled |
Activity is not value. An extra touch only matters if it changes future behavior.
The customers who converted after receiving the reminder were, almost entirely, customers who would have converted anyway. The email was collecting credit for purchases it did not cause. That is what attribution without a control group does: it measures presence, not effect.
The strongest objection
The fairest pushback we can construct: maybe the reminder works on a horizon the trial cannot see. Brand emails compound, the argument goes, and eight weeks is not a year.
Two things blunt it. First, the endpoint already carries the future in it: 12-month forward value moves the moment behavior changes, because the model reprices a customer's expected purchases as soon as their pattern shifts. An email that seeded a later purchase would show up as a repriced customer now, not a converted customer later. Second, the reminder had eight consecutive chances to move that number and never did, in either direction.
What the trial genuinely cannot rule out is an effect too small to detect, somewhere inside the interval's width. But an effect too small to detect over eight weeks on a pool this size is also too small to pay for its place on the calendar. "Possibly nonzero" is not a budget line.
What we did about it
We recommended the client stop sending the reminder and reclaim the budget and attention for the arm that measurably worked. Telling a client to do less email is not a common vendor recommendation; most of the industry is paid in proportion to activity. It is, however, what the data said, and reporting what the data said is the entire job.
Engagement metrics are not wrong, exactly. They answer a different question. Opens and clicks measure whether people interacted with an email. They cannot measure whether the email changed what those people were going to do, because they contain no counterfactual. The only way to see the counterfactual is to hold some customers out and compare. Very few retention programs are run this way, which means most retention calendars contain some emails that create value, some that are neutral, and some that quietly cost money. All of them look roughly the same on the dashboard.
The verdict
A healthy dashboard is not evidence that an email creates value. It is evidence that the email exists. Of the two treatments in this trial, the one the dashboard could not tell apart from its twin was worth +$5.20 per customer; the other was worth exactly nothing, eight reads in a row.
- Pick your longest-running "keeper" flow and ask what evidence, other than engagement, supports it.
- Carve out a holdout the next time it sends. Even a crude one beats none.
- Judge it on a value endpoint, not opens. If it survives, scale it with confidence. If it doesn't, you just found free budget.
The free diagnostic runs this discipline on your own customers, and will tell you which of your emails would survive it.