Guide

How to run a holdout test for Customer Success outreach

A holdout test proves whether outreach to at-risk accounts changed renewals: you leave a random 10–20% of those accounts on your normal process, give the rest the new outreach, and compare renewal rates. The steps below cover who goes in, how to assign at random in Google Sheets, how big the groups need to be, and how to report the result so finance believes it.

1. Choose the population

Write the rule down before any outreach goes out: for example, every account in the Red or Yellow band with a renewal 90 to 120 days out and no conversation in 45 days. Pull the list once and freeze it with a date. Before assigning, remove the accounts you would never leave out, such as any with a cancellation notice already on record, and apply the same exclusions to both groups. Everyone left is eligible for either group.

2. Assign at random

In Google Sheets, with accounts in column A and ARR in column B, put =RAND() in C2 and this in D2, then fill both down:

=IF(C2<0.2,"Holdout","Outreach")

The cutoff is the holdout share, so 0.2 gives about 20%. For an exact count, rank the random numbers instead; this puts exactly 20 of 100 accounts in the holdout:

=IF(RANK(C2,$C$2:$C$101)<=20,"Holdout","Outreach")

RAND recalculates on every edit, so copy columns C and D and paste values only (Ctrl+Shift+V in Sheets, Paste Special then Values in Excel). Keep that sheet as the record. Don't pick by hand and don't move the large accounts into the outreach group afterwards; the point of the draw is that the two groups differ only in the outreach.

3. How big a holdout

Keep 10–20% of the population back. A smaller holdout means more accounts get the outreach but the result takes longer to read; a larger one reads sooner but withholds the outreach from more accounts. The table shows roughly how many at-risk accounts you need in total for a lift to show clearly, assuming 55% of the holdout renews. (Normal approximation, example figures.)

Lift you hope to seeHoldout 20%Holdout 10%Holdout 50%
20 points (55% to 75%)142 accounts (29 held out)258 (26 held out)84 (42 held out)
15 points (55% to 70%)257 accounts (52 held out)463 (47 held out)157 (79 held out)
10 points (55% to 65%)585 accounts (117 held out)1,048 (105 held out)365 (183 held out)

These are the sizes at which a lift of that size would just clear zero. To be fairly sure of seeing it (80% power, in the statistician's phrase), plan for about twice as many. For most mid-market teams this means one holdout across several campaigns, or a test that runs over two renewal cycles. In Horizon the holdout share is a setting on each campaign, up to 20%.

4. What to measure

5. Reading the result

Lift = renewal rate with outreach − renewal rate in the holdout
Likely range = lift ± 1.96 × √( p₁(1 − p₁) ÷ n₁ + p₂(1 − p₂) ÷ n₂ )

p₁ and n₁ are the renewal rate and size of the outreach group, p₂ and n₂ the holdout's. With 40 outreach accounts of which 31 renewed (77.5%) and 16 holdout accounts of which 9 renewed (56.3%), the lift is 21.3 points and the range runs from −6 to +49 points. It includes zero, so the lift could be chance. The same rates with 160 and 64 accounts give +7 to +35, clear of zero. The retention lift calculator does this arithmetic and adds the ARR at stake.

6. Objections, and how to answer them

7. Reporting the result

One table and one sentence. The table has a row per group with accounts, renewed, renewal rate, ARR due and ARR retained, then the lift, its range, the window, the date of the draw and the exclusions. The sentence is the verdict: "range above zero, the outreach worked" or "range includes zero, keep the test running". It fits slide 4 of the board retention report.

Horizon does this for every outreach campaign. When a campaign starts it sets aside a random share of the matching accounts as the holdout, their account owners keep working them as before, and the report shows goal attainment with outreach and in the holdout, the lift between them, and a plain statement when there are too few accounts to conclude anything.

Updated November 3, 2026

See which of your renewals are at risk.

A 30-minute call on a demo book, or on your own top 25 accounts.

Book a call