Why use one
Without a comparison, a saved renewal could have renewed anyway. A holdout gives you the comparison: the same kind of accounts, at the same time, without the outreach.
Worked example
40 at-risk accounts get a playbook or outreach and 31 renew (77.5%). 16 similar accounts in the holdout stay on the normal process and 9 renew (56.3%). The difference, about 21 points, is the estimated effect of the outreach. (Example numbers. With groups this small, the result could still be chance.)
Doing it safely
- Keep the holdout small and random.
- Holdout accounts still get your team's normal attention; they just don't get the new outreach.
- Say when the sample is too small to conclude anything.
See holdout measurement in Horizon.
More on this topic: renewals, definitions, guides and tools.
Updated November 3, 2026