---
title: "How to run a holdout test for Customer Success outreach"
url: https://renewalfix.com/guides/how-to-run-a-holdout-test/
description: "How to run a holdout test for Customer Success outreach: choose the at-risk accounts, assign at random in Sheets, size the holdout, and report the lift."
updated: 2026-11-03
---

Guide

# How to run a holdout test for Customer Success outreach

A holdout test proves whether outreach to at-risk accounts changed renewals: you leave a random 10–20% of those accounts on your normal process, give the rest the new outreach, and compare renewal rates. The steps below cover who goes in, how to assign at random in Google Sheets, how big the groups need to be, and how to report the result so finance believes it.

By [John Beckett](https://renewalfix.com/about/#john-beckett) · Updated November 3, 2026

## 1. Choose the population

Write the rule down before any outreach goes out: for example, every account in the Red or Yellow band with a renewal 90 to 120 days out and no conversation in 45 days. Pull the list once and freeze it with a date. Before assigning, remove the accounts you would never leave out, such as any with a cancellation notice already on record, and apply the same exclusions to both groups. Everyone left is eligible for either group.

## 2. Assign at random

In Google Sheets, with accounts in column A and [ARR](https://renewalfix.com/glossary/annual-recurring-revenue/) in column B, put `=RAND()` in C2 and this in D2, then fill both down:

```
=IF(C2<0.2,"Holdout","Outreach")
```

The cutoff is the holdout share, so 0.2 gives about 20%. For an exact count, rank the random numbers instead; this puts exactly 20 of 100 accounts in the holdout:

```
=IF(RANK(C2,$C$2:$C$101)<=20,"Holdout","Outreach")
```

RAND recalculates on every edit, so copy columns C and D and paste values only (Ctrl+Shift+V in Sheets, Paste Special then Values in Excel). Keep that sheet as the record. Don't pick by hand and don't move the large accounts into the outreach group afterwards; the point of the draw is that the two groups differ only in the outreach.

## 3. How big a holdout

Keep 10–20% of the population back. A smaller holdout means more accounts get the outreach but the result takes longer to read; a larger one reads sooner but withholds the outreach from more accounts. The table shows roughly how many at-risk accounts you need in total for a lift to show clearly, assuming 55% of the holdout renews. (Normal approximation, example figures.)

| Lift you hope to see | Holdout 20% | Holdout 10% | Holdout 50% |
| --- | --- | --- | --- |
| 20 points (55% to 75%) | 142 accounts (29 held out) | 258 (26 held out) | 84 (42 held out) |
| 15 points (55% to 70%) | 257 accounts (52 held out) | 463 (47 held out) | 157 (79 held out) |
| 10 points (55% to 65%) | 585 accounts (117 held out) | 1,048 (105 held out) | 365 (183 held out) |

These are the sizes at which a lift of that size would just clear zero. To be fairly sure of seeing it (80% power, in the statistician's phrase), plan for about twice as many. For most mid-market teams this means one holdout across several campaigns, or a test that runs over two renewal cycles. In Horizon the holdout share is a setting on each campaign, up to 20%.

## 4. What to measure

- **Renewal rate:** accounts renewed ÷ accounts due, in each group, over a fixed window, for example renewals due within 120 days of the draw, counted 30 days after each due date.
- **ARR retained:** ARR renewed ÷ ARR due, in each group. Report both; one large account can move the ARR figure on its own.
- **Earlier signs:** replies, meetings booked, or a goal such as "renewal call booked within 30 days", which you can read long before the renewals land.
- **Group membership never changes.** If a holdout account gets the outreach by mistake, it stays in the holdout for the analysis and you note the exception.

## 5. Reading the result

```
Lift = renewal rate with outreach − renewal rate in the holdout
```

```
Likely range = lift ± 1.96 × √( p₁(1 − p₁) ÷ n₁ + p₂(1 − p₂) ÷ n₂ )
```

p₁ and n₁ are the renewal rate and size of the outreach group, p₂ and n₂ the holdout's. With 40 outreach accounts of which 31 renewed (77.5%) and 16 holdout accounts of which 9 renewed (56.3%), the lift is 21.3 points and the range runs from −6 to +49 points. It includes zero, so the lift could be chance. The same rates with 160 and 64 accounts give +7 to +35, clear of zero. The [retention lift calculator](https://renewalfix.com/tools/retention-lift-calculator/) does this arithmetic and adds the ARR at stake.

## 6. Objections, and how to answer them

- **"We're letting accounts churn on purpose."** The holdout keeps the normal process: the same owner, the same renewal call, the same response to a cancellation notice. It only skips the new outreach, and until the test you don't know that outreach helps. The expected cost is small: with 100 at-risk accounts, a 20% holdout and a 15-point lift, about 3 renewals (20 × 0.15), which is the price of knowing whether the other 80 got anything.
- **"Our CSMs will quietly work the holdout harder."** If they do, it pushes the result toward no lift, so a lift you still see is real. Keep the group labels out of the daily view where you can.
- **"We don't have enough accounts."** Pool campaigns under one holdout, run longer, and report the range honestly. A wide range is a finding; skipping the holdout leaves you with a claim.
- **"Finance won't accept a CS experiment."** The holdout is the part finance accepts. Without it, a saved renewal is an account that renewed after someone called it.

## 7. Reporting the result

One table and one sentence. The table has a row per group with accounts, renewed, renewal rate, ARR due and ARR retained, then the lift, its range, the window, the date of the draw and the exclusions. The sentence is the verdict: "range above zero, the outreach worked" or "range includes zero, keep the test running". It fits slide 4 of the [board retention report](https://renewalfix.com/guides/board-retention-report/).

Horizon does this for every outreach campaign. When a campaign starts it sets aside a random share of the matching accounts as the [holdout](https://renewalfix.com/product/holdout-measurement/), their account owners keep working them as before, and the report shows goal attainment with outreach and in the holdout, the lift between them, and a plain statement when there are too few accounts to conclude anything.

More on this topic: [renewals, definitions, guides and tools](https://renewalfix.com/topics/renewals/).

---

Horizon is renewal risk software for Customer Success teams. Book a call: https://calendar.chilipiper.com/book/me/serban-goanta
