Split Tests
Manage advanced split testing by comparing multiple variations of your email campaigns. This feature ensures optimal performance by automatically analyzing and selecting the best-performing variations for your audience.
Overview
Split tests (A/B testing) allow you to compare different broadcast variations to determine which performs best. The system automatically analyzes results and can select the winning version for the remaining audience.
How Split Tests Work
- Create a Split Test - Define which broadcasts to compare and winning criteria
- Schedule the Campaign - Use Actions → Schedule with Campaign Type "Split Test"
- System sends variations - Broadcasts are sent to equal portions of the audience
- Winner is determined - Based on open rate or click-through rate
- Action is performed - Update results or send winner to remaining audience
Viewing Split Tests
Navigate to Campaigns → Split Tests to view all split tests.
Page Features
- Add New button to create a split test
- Bulk Actions dropdown for operations on selected items
- Our records / User records toggle
- Filter by admin dropdown
- Show entries dropdown
- Search box
Split Test Information
| Column | Description |
|---|---|
| ID | Unique identifier |
| Split Test Name | Name of the split test |
| Based on | Broadcasts being compared (comma-separated) |
| Criteria | Winner determination method (e.g., "Find Best") |
| Created On | Date and time created |
| Actions | Action menu (three dots) |
Creating a Split Test
Navigate to Campaigns → Split Tests and click Add New.
Split Test Settings
Split Test Name
Enter a descriptive name for the split test:
- "Subject Line Test - March Newsletter"
- "CTA Button Color Test"
- "Content Length Comparison"
Based on Broadcast Performance
Split tests compare the performance of different broadcasts to find the best performer.
Campaigns (Broadcasts to Compare)
Select two or more broadcasts to compare:
- Broadcasts are displayed in a hierarchical checkbox list grouped by broadcast group
- Check the broadcasts you want to include in the test
- Each selected broadcast becomes a test variation
- Emails will be distributed equally among selected broadcasts
Example: Select "additional field" and "campaign with click" to compare these two broadcasts.
Winner Configuration
Winning Criteria
Choose how the winner is determined:
| Criteria | Description |
|---|---|
| Open Rate | Broadcast with highest open rate wins |
| Unique click-through rate | Broadcast with highest unique click rate wins |
Decision Percentage Winning Criteria
Enter the percentage of the audience to use for testing (e.g., 100).
Note: Emails will be sent in equal groups to this percentage of the audience.
Examples:
- 100% - All recipients are split equally among variations, winner is determined but no "leftover" audience
- 20% - 20% of recipients are split for testing, 80% can receive the winner
Duration to Wait for the Results
Specify how long to wait before determining the winner:
- Number field - Enter the duration amount (e.g., 3)
- Unit dropdown - Select the time unit
| Unit | Description |
|---|---|
| Minutes | Wait time in minutes |
| Hours | Wait time in hours |
| Days | Wait time in days |
| Weeks | Wait time in weeks |
Note: The winning element will be identified after this duration, and the remaining portion of the list will receive that version.
Action to Perform
Choose what happens after the winner is determined:
| Action | Description |
|---|---|
| Find winning element and update the results | Determine winner and record results (no additional sending) |
| Find winning element and send leftover to it | Determine winner and send to remaining audience |
Save the Split Test
Click Save to create the split test configuration, or Cancel to discard.
Scheduling a Split Test Campaign
After creating a split test, you need to schedule it via the Actions menu.
Accessing Schedule
Navigate to Actions → Schedule to schedule campaigns.
Creating a Split Test Campaign
The scheduling wizard has 4 steps:
Step 1: Campaign Type
| Field | Description |
|---|---|
| Campaign Name | Name for this scheduled campaign |
| Campaign Type | Select Split Test from dropdown |
Campaign Type Options:
- Regular
- Split Test ← Select this
- Evergreen
After selecting Split Test, choose the split test configuration you created.
Below, select the target contact lists (with contact counts shown).
Target audience filters - Toggle to add additional filtering criteria.
Step 2: Broadcasts
Configure broadcast-related settings (varies based on split test configuration).
Step 3: Sending Nodes
Select SMTP servers (sending nodes) for sending the campaign.
Step 4: Settings
Configure delivery settings, tracking, and timing.
Managing Split Tests
Editing a Split Test
- Click the actions menu (three dots) next to a split test
- Select Edit
- Modify settings
- Click Save
Cannot edit split tests that are currently running or have already sent.
Deleting a Split Test
- Click the actions menu next to a split test
- Select Delete
- Confirm deletion
Split Test Strategy
What to Test
Subject Lines
Compare different approaches:
- Short vs. long subjects
- With personalization vs. without
- Question vs. statement
- Urgency vs. curiosity
- With emoji vs. without
Content Variations
Test different content elements:
- Different calls to action
- Image placement
- Content length
- Layout variations
- Offer presentation
Sample Size Guidelines
| Audience Size | Recommended Test % |
|---|---|
| Under 1,000 | 50-100% (limited data) |
| 1,000 - 10,000 | 20-30% |
| 10,000 - 100,000 | 10-20% |
| Over 100,000 | 5-10% |
Duration Guidelines
Allow enough time for meaningful results:
| Winning Criteria | Recommended Duration |
|---|---|
| Open Rate | 4-24 hours |
| Click-through Rate | 24-48 hours |
Best Practices
Test Design
- One variable at a time - Isolate what you're testing for clear results
- Clear hypothesis - Know what you expect and why
- Sufficient sample size - Ensure enough recipients for statistical significance
- Equal distribution - Let the system split audience evenly
Timing
- Allow adequate duration - Don't rush winner determination
- Consider time zones - Account for when recipients open emails
- Weekday vs. weekend - Engagement patterns differ
Analysis
- Document learnings - Record what worked and why
- Apply insights - Use winning approaches in future campaigns
- Continue testing - Always look for optimization opportunities
- Build best practices - Create guidelines based on test results
Troubleshooting
Not Enough Data
Cause: Sample size too small or duration too short.
Solution:
- Increase test percentage
- Extend wait duration
- Use larger contact lists
Inconclusive Results
Cause: Variations performed similarly.
Solution:
- Test more dramatic differences
- Increase sample size
- Consider external factors affecting results
Winner Not Determined
Cause: Duration hasn't elapsed or processing issue.
Solution:
- Verify duration settings
- Check that wait time has passed
- Review campaign status in Scheduled Campaigns
Split Test Not Appearing in Schedule
Cause: Split test not saved or inactive.
Solution:
- Verify split test was saved successfully
- Check split test exists in Campaigns → Split Tests
- Create new split test if needed
Next Steps
- Broadcasts - Create email content for testing
- Dynamic Content Tags - Conditional content and RSS feeds
- Evergreen Campaigns - Recurring automated campaigns
- Drip Campaigns - Automated email sequences
- Scheduling Campaigns - One-time campaign scheduling