How to A/B Test Subject Lines Properly

An A/B test on subject lines only tells you one thing: which subject got more people to open the email. It does not tell you which offer converted, which email made money, or whether the winner will work again next month. Run it properly — one variable, a written hypothesis, a sample size you decided in advance, and a winner rule you agreed to before sending — and you get a repeatable answer. Run it casually and you get a false winner that quietly costs you opens for months.

What a subject line test can and cannot tell you

A subject line test measures opens. In Mailerflow, opens are reported as approximate opens in campaign analytics, because image blocking and privacy features in modern inboxes mean open tracking is a signal, not a count. Treat small gaps between variants as noise, not as a result.

Open rate is a proxy for attention. It answers "did the inbox get noticed?" It does not answer "did the reader care about the offer?" A subject line can lift opens and still produce the same clicks, the same revenue, and more unsubscribes. That is why you read opens alongside clicks, complaints and unsubscribes before you declare anything.

Two things make subject line tests noisy. The first is small samples: with a few hundred recipients per variant, a two-point difference is often within normal variation and should be treated as directional. The second is uneven send conditions — different send times, different segments, a news event landing mid-test, or a preview text change you forgot about. Both are fixable, and most of this article is about fixing them.

What you need before you start

A short checklist, nothing more:

• A list large enough to split into two groups that are roughly equivalent.
• One variable you are changing. Only one.
• A hypothesis written down before you send.
• A place to record the result, including the tests that fail.

Mailerflow's A/B testing of campaigns lets you test subject lines and other campaign elements across a split of your audience. The Free plan supports 2,500 contacts and 10,000 emails a month, which is enough room to run small tests without paying for them. If you are on a paid plan, emails are unlimited, so the constraint is your list size, not your send volume.

Step 1: Write a hypothesis, not a hunch

Use this format: If I [change], then [metric] will [direction], because [reason].

Three illustrative examples for a fictional outdoor gear brand, clearly made up for this article:

• If I name the use case in the subject line instead of saying "new arrivals", open rate will rise, because cold subscribers need a reason to care before they click.
• If I lead with the benefit rather than curiosity, open rate will rise, because this segment has bought before and already trusts the brand.
• If I add the first name, open rate will rise, because the list is small and personal and the sender is a known person.

One variable per test. "Test two completely different subject lines and see which wins" produces an unreadable result: if variant B wins, you cannot say whether it was the length, the specificity, the number, or the tone. You have a winner you cannot reuse.

Step 2: Decide your sample size before you send

The logic is simple even without statistics: the smaller the difference you expect, the more recipients you need per variant. As a rule of thumb, detecting a two-point difference needs far more people than detecting a ten-point difference. A ten-point difference on a small list is visible; a two-point difference is not.

A practical rule you can apply yourself: aim for at least a few hundred recipients per variant, and treat anything under that as directional only — a hint for the next test, not a decision. If your list cannot support that, test bigger changes instead of subtle ones, and accept that you are choosing a direction rather than proving a point. Check your own platform's documentation for an exact sample size calculator rather than trusting a number from an article, including this one.

Mailerflow splits a campaign across your audience so each variant gets a share, which means the split percentage you choose directly determines your sample size per variant.

Step 3: Set up the test in your sending tool

Here is the sequence in Mailerflow, with the settings that matter:

1. Open the campaign you want to send and go to the A/B test section.
2. Add variant B. The campaign now has a control (A) and a variant (B).
3. Change only the subject line in variant B. Leave preview text, sender name, content, links and send time identical.
4. Set the split percentage. Check your tool's documentation for what split options it offers — some tools let you hold back a portion of the audience for a later winner send, others send both variants to the full split immediately.
5. Decide the winner rule in your tool. If your tool supports choosing between an automatic winner (based on a metric like open rate) and a manual pick after you review results, choose one now and write it down.
6. If your tool supports sending the winner to the remainder of the audience, set the wait window before that send — for example four hours.
7. Send.

Before you hit send, verify that the only difference between the two variants is the subject line text. If your tool shows a summary screen, check that the subject lines, split and winner rule match what you intended.

If you are short on variant ideas, Mailerflow's AI email builder can generate subject line and preview text options from a description of the email, which is a fast way to get candidates on the table. The hypothesis still comes from you — the AI does not know why your audience opens.

Step 4: Keep everything else identical

The variables that quietly ruin tests:

• Different send times between variants.
• Different segments, even slightly different ones.
• Different preview text. This one is easy to miss and matters most, because preview text sits directly next to the subject line in most inboxes and changes how the subject line reads.
• Different sender name.
• A promo or discount in one version only.
• A holiday, outage or news event landing mid-test.

If two things changed, you cannot attribute the result to either. Write the list of fixed variables down before you send, and check it after.

Step 5: Read the result and pick a winner

Define winner criteria up front, in writing:

• A minimum lift you would actually act on (for example, five points).
• A minimum sample per variant (for example, a few hundred).
• A decision rule for ties: no winner, keep the control.

"Variant B won by 0.4 points on 300 sends" is not a decision. It is noise with a number attached. Record results in a table like this:

Record the hypothesis outcome too, including failed tests. A log of what did not work is more valuable than a folder of winners, because it stops you repeating the same dead end.

A worked example

This is a fictional outdoor gear brand, used to show the reasoning rather than a real result.

Hypothesis: If I name the use case in the subject line instead of saying "new arrivals", open rate will rise, because a cold segment needs a concrete reason to open.

Setup: 4,000 recipients, 50/50 split, same send time, same preview text, winner by open rate after four hours.

• Control A: "New arrivals are here" — 2,000 recipients, 19.0% open rate.
• Variant B: "New arrivals: 3 jackets for wet commutes" — 2,000 recipients, 20.1% open rate.

Honest conclusion: a 1.1-point lift on 2,000 recipients per variant is not enough to act on. The correct decision is "no decision — retest on a larger segment or with a bigger change." Writing that down is not failure; it is the test working as intended.

Second round: the same brand tests a larger change on a bigger segment — control "New arrivals are here" versus variant "Your commute jacket is back in stock". This time the gap is large enough and the sample is large enough to act on, so the variant becomes the new control for the next test. The point is not that specificity always wins. The point is that you only move when the evidence supports moving.

How to check the test actually ran correctly

Verification checklist after the send:

• Both variants show the same recipient count, within a rounding error.
• The send timestamps match.
• The only difference in the campaign record is the subject line.
• Opens are attributed per variant in analytics.
• If your tool supports an automatic winner send to the remainder of the audience, confirm in the campaign report that it went out and to how many recipients.

Mailerflow's per-campaign analytics for delivery, bounces, complaints, clicks and approximate opens is where you confirm variant-level numbers. If a variant shows a suspiciously high complaint or unsubscribe rate alongside a high open rate, that is a signal the subject line overpromised — check it before you promote the variant.

Common mistakes and how to fix them

Stopping the test early because one variant is ahead. Fix: set the wait window before sending and do not look at interim numbers.

Testing on your most engaged segment only, then generalising. Fix: run the test on the segment you will actually send to next, or accept that the result applies only to that segment.

Changing subject line and preview text together. Fix: one variable per test, every time.

Using clickbait that lifts opens and tanks clicks and complaints. Fix: read opens alongside clicks, complaints and unsubscribes before declaring a winner.

Never testing the control again. Fix: when a variant wins, it becomes the new control. Test against it.

Running tests so rarely that nothing compounds. Fix: one test per month, logged, is enough to build a real picture over a year.

Frequently asked questions

How long should I wait before picking a winner? Long enough that the bulk of opens have landed — often a few hours to a day, depending on your list. Set the wait window before you send, and use the same window for both variants so the comparison is fair.

Can I test subject lines in an automation? It depends on your tool. Mailerflow's A/B testing applies to campaigns; for automations, test by running two versions of the email to comparable segments and comparing results manually. Check your own platform's documentation for what it supports.

What if my list is too small to split? Then you probably cannot test reliably at your list size. Test bigger changes instead of subtle ones, and treat results as directional. A test you cannot trust is worse than no test, because it gives you false confidence.

Should I test emoji? Only if it fits your brand and you have a hypothesis about why it would help. Emoji is a variable like any other, and it needs the same sample size and the same one-variable discipline.

Does a subject line test also measure clicks? No. It measures opens. Clicks are a separate metric and a separate question — read them alongside opens, but do not treat an open-rate winner as a click-rate winner without checking.

Next steps

Write your next hypothesis now, before you forget the idea, and put a recurring slot in your calendar for one test a month. Keep a running log of hypotheses and outcomes, including the ones that went nowhere. If you want a faster way to generate variant candidates to test, the Mailerflow AI email builder can produce subject lines and preview text from a short description of the email, and the full feature list shows where A/B testing and analytics live in the product.