Split Tests

A split test, also called an A/B test, sends two or more versions of an email to part of your audience, measures which one does better, and can then send the winner to everyone else. Use it to learn which subject line, offer or design your contacts respond to, and to make sure most of your list gets the best version.

How split tests work

1. Create the versions. Each version is a separate broadcast. Make one, then Duplicate it and change the one thing you want to test.
2. Create the split test under Campaigns → Split Tests: which broadcasts to compare, how to pick the winner, how much of the audience to test on, and how long to wait.
3. Schedule it under Actions → Schedule, with Campaign Type set to Split Test.
4. Mumara ONE sends the test. The test share of your audience is split equally between the versions.
5. The winner is picked after the waiting time, by open rate or unique click-through rate.
6. The rest of the audience gets the winner, if you chose that action. Otherwise the result is just recorded.

Audience: 10,000 contacts, decision percentage 20%

         +-- Version A: 1,000 --+
Test ----+                      +--> wait 6 hours --> winner: B
         +-- Version B: 1,000 --+
                                                  |
Remaining 8,000 contacts  <-----------------------+  (if "send leftover" is chosen)

The Split Tests page

Go to Campaigns → Split Tests. Add New creates a split test, and Bulk Actions → Delete deletes the ones you've ticked.

Column	What it shows
ID	The split test's ID.
Split Test Name	Its name.
Based on	The broadcasts being compared.
Criteria	Find Best if the test only records the winner, or Send Best if it also sends the winner to the rest of the audience.
Created On	When it was created.
Actions	Edit and Delete.

Create a split test

Go to Campaigns → Split Tests and click Add New. The form is headed Split Test Settings.

Split Test Name

A name that says what you're testing, for example March newsletter: subject line or Spring sale: button text.

Broadcasts to compare

Based on broadcast performance is selected: the test compares broadcasts. Under Campaigns, your broadcasts are listed by group. Tick at least two. Tick a group's name to select every broadcast in it.

Each broadcast you tick becomes one version, and each version goes to an equal share of the test audience.

Winning criteria

Option	The winner is the broadcast with
Open Rate	The highest open rate. Best for testing subject lines, pre-headers and sender names.
Unique click-through rate	The highest share of recipients who clicked. Best for testing content, offers, layouts and calls to action.

Decision percentage

The share of your audience that receives the test, split equally between the versions. It starts at 10.

Decision percentage	With 10,000 contacts and two versions
10	500 contacts get each version. The winner can go to the other 9,000.
20	1,000 contacts get each version. The winner can go to the other 8,000.
100	5,000 contacts get each version. Nobody is left over, so the winner is only recorded.

Duration to wait for the results

How long after the test is sent to pick the winner. Enter a number and choose Minutes, Hours, Days or Weeks. It starts at 10 minutes, which is too short for a meaningful result; see How long to wait.

After this time, the winning version is identified and, if you chose to, the rest of the audience receives it.

Action to perform

Option	What happens after the waiting time
Find winning element and update the results	The winner is recorded. Nothing more is sent. Use this with a decision percentage of 100, or when you only want to learn.
Find winning element and send leftover to it	The winner is recorded and sent to everyone who didn't receive the test.

Click Save to create the split test, or Cancel to discard it.

Schedule the split test

Creating a split test doesn't send anything. To send it:

1. Go to Actions → Schedule.
2. On the Campaign Type step, name the campaign and set Campaign Type to Split Test.
3. Choose your test from the Split Tests list that replaces the recipient choice. If it says No split test found., create one first with the Create one here link.
4. On the Broadcasts step, the broadcasts already come from your test, so this step lists your contact lists instead. Tick the lists to send to. Segments aren't offered for split tests.
5. On the Sending Nodes step, tick the pool to send through. In Mumara ONE the entries in this step are your pools. See Dedicated Pools.
6. On the Settings step, choose when to send, keep tracking on (a split test can't pick a winner without open or click tracking), and set the sender details.
7. Click Submit.

See Scheduling Campaigns for every option on the form.

Follow the results

• Campaigns → Scheduled Campaigns shows the campaign's progress. After the test is sent, a countdown shows how long until the winner is picked and sent to the rest of the audience.
• Statistics → Broadcast Stats shows opens and clicks for each version. See Broadcast Stats.

Manage split tests

• Edit a split test from its Actions menu, change the settings and click Save. Changes apply to campaigns that haven't started. Avoid editing a split test while a campaign that uses it is running, because the winner is picked with the settings in force at that moment.
• Delete a split test from its Actions menu, or tick several and choose Bulk Actions → Delete. Don't delete a split test that a scheduled campaign still uses.

What to test

Change one thing between versions, so you know what made the difference.

Test	Examples	Judge by
Subject line	Short or long, a question or a statement, with or without the first name, with or without an emoji	Open Rate
Pre-header	A summary or a teaser	Open Rate
Call to action	Button text, color or position	Unique click-through rate
Content	Long or short copy, image-led or text-led, different offers	Unique click-through rate

How much of your audience to test on

Audience	Suggested decision percentage
Under 1,000 contacts	50 to 100
1,000 to 10,000	20 to 30
10,000 to 100,000	10 to 20
Over 100,000	5 to 10

Smaller audiences need a larger share, or the difference between versions is just chance.

How long to wait

Winning criteria	Suggested wait
Open Rate	4 to 24 hours
Unique click-through rate	24 to 48 hours

Most opens happen in the first few hours, clicks a little later. Think about when your audience reads email: a test sent on Friday evening needs longer than one sent on Tuesday morning.

Troubleshooting

Problem	What to check
The form won't save	Enter a Split Test Name, tick at least two broadcasts and fill in the Decision percentage.
The split test isn't offered on the Schedule page	Check that it was saved under Campaigns → Split Tests and that you set Campaign Type to Split Test.
No winner yet	The waiting time hasn't passed. Check the duration you set and the campaign's status under Scheduled Campaigns.
The results are too close to call	The versions weren't different enough, or too few contacts took part. Test a bigger change, or raise the decision percentage.
The winner was based on too little data	Wait longer, or test on a larger share of the audience.

Next steps

• Broadcasts: create and duplicate the versions
• Scheduling Campaigns: send the test
• Broadcast Stats: compare the results
• Spintax: add variety within a single broadcast