A/B test an e-mail campaign
Testing subject lines or templates on part of a list, the settings — duration, test size, winner — how Push picks the winner and sends it to everyone else, and reading the results.
An A/B test sends two to four versions of a campaign to a small part of the list, waits, and sends the version that did best to everyone else. A/B tests are part of the Growth and Scale plans.
Set up a test
- In the campaign's form, switch on Enable A/B Testing.
- Choose the settings:
- Test Duration — how long to wait before choosing: from 1 hour to 48 hours. Four hours or more gives most people time to open.
- Test Percentage — how much of the list gets the test: 10%, 20%, 30% or 50%.
- Winner Criteria — Open Rate or Click Rate.
- Set up the variants — up to four with + Add Variant. Each has a Variant Name, a Weight (%) — its share of the test — and what it changes: Subject Line Override (blank keeps the campaign's subject) and Template (Use campaign template, or another one). The weights must add up to 100 %: until they do, Total Weight: says (should equal 100%).
- Create the campaign and send it as usual.
Test one thing at a time — two subjects with the same template, or two templates with the same subject — so you know what made the difference.
How the winner is chosen
When the test starts, the campaign says A/B test in progress.... After the duration, Push compares the variants:
- with Open Rate, it counts verified opens — the opens Apple Mail and Google's image proxy make on delivery are left out, because they are not people;
- it waits until at least 50 e-mails of the test are delivered, for up to 24 hours more;
- the variant with the higher rate wins, and Push sends it to the rest of the list.
Read the results
The campaign's page shows A/B Test Variants: for each variant Sent, Delivered, Opened, Clicked, Open Rate and Click Rate, with Winner next to the chosen one, plus Winner selected with the time. The usual report follows below it: The campaign report.
If something doesn't work
A/B testing is not available on your current plan.
Your plan is Starter. A/B tests come with Growth and Scale: Change your plan.
The winner was not sent after the test duration
Fewer than 50 test e-mails were delivered. Push keeps waiting for up to 24 hours more, then picks the better variant with what it has. A test on a small list tells you little — use it on lists of a few thousand.
The weights do not add up
Change Weight (%) on the variants until Total Weight: is 100 %.