An email holdout test compares an eligible group that receives a campaign with a comparable group that does not. It asks whether the message changed buying behavior, rather than which orders matched an attribution rule. The concept is simple; a trustworthy comparison needs careful group assignment, consistent measurement, and attention to other messages the same customers receive.
Plan the groups before sending
Start with an eligible audience and assign a comparable subset to receive the promotion while another subset does not receive that particular marketing message. Keep ordinary required service communication intact. Do not withhold essential order information for an experiment.
Use the audience controls available to organize the plan, and document the assignment process. If you cannot create comparable groups or track the outcome consistently, treat the exercise as exploratory rather than a reliable causal test.
Define the treatment precisely
Decide whether you are testing one email, a sequence, or the addition of SMS to an existing email plan. The holdout should differ in that defined marketing treatment while necessary service communication remains intact. If several other things change, the result answers a broader question than the one-email effect.
Randomly assign eligible customers where your process and data allow it, and keep the assignment stable for the observation period. Avoid placing the least engaged people in the holdout and the best customers in the treatment group. That comparison mostly measures who you selected, not what the campaign changed.
Record the assignment method, group sizes, exclusions, intended exposure, and primary outcome before sending. Measure the outcome from a source that can observe purchases in both groups. Message-attributed revenue alone is unsuitable because the group that received no message has no equivalent attribution opportunity.
Keep competing influences visible
Check whether other campaigns, automations, or public offers reach the groups differently. A holdout from one email may still see the same sale elsewhere. That does not make the exercise useless, but it changes what the result can tell you.
Use the same observation period and outcome definition for both groups. Compare order rates or contribution per eligible customer rather than only attributed revenue, since the non-message group has no equivalent message attribution path.
Work through the difference without overstating it
Imagine 2,000 customers in each group. During the same observation period, 60 treatment customers purchase and 50 holdout customers purchase. The observed rates are 3% and 2.5%, a difference of 0.5 percentage points. Applied to 2,000 customers, that is an observed difference of ten purchasers, not evidence that all 60 treatment purchases were created by the message.
The example still needs an uncertainty assessment. Random variation can produce a difference, particularly when purchases are relatively rare. Use an appropriate statistical analysis and consider whether the effect is large enough to matter commercially. A positive point estimate is not the same as a reliable conclusion.
Interpret the difference with restraint
Small groups and rare purchases can produce noisy results. A modest difference may be inconclusive. Record sample size, overlap, timing, and any departures from the plan before making a business claim.
Sendvio campaign reports can contribute to the analysis, but a holdout is a measurement design, not a guarantee built into a dashboard. Use it when the decision is important enough to justify the preparation. The goal is to understand added value, including cases where sending less produces a similar commercial outcome.
Investigate exposure outside the tested message. A holdout customer may see the public sale or receive another permitted automation. That does not automatically invalidate the experiment, but it means you are estimating the additional effect of the specified treatment within that wider environment, not the effect of all marketing versus none.
Compare contribution and negative responses where the decision requires them, using consistent definitions and enough time for relevant returns. Keep departures from the plan visible. If groups were not comparable or exposure was poorly tracked, report the exercise as exploratory. A well-run holdout can reveal that a campaign adds value, adds little, or costs more than it earns; all three outcomes are useful when they change a real decision.