A/B testing with Sitecore Email Experience Manager (EXM)

How to do A/B testing with email campaigns in Sitecore?

A/B testing  any email campaign is the best way to ensure that you are giving customers the information they want to see. As you create email campaigns, you are probably placing the content and messages layout based on the assumptions that you feel is appealing to the customer. However, your assumptions might not be accurate, and your recipients might not respond to your campaign in the way you expect them to. A/B testing allows you to create a couple of variations of the similar message so that you can send it to a small group of people first and see how they interact with the message and which one gains more popularity before picking the final message to be sent to your entire email base.

Sitecore Email Experience Manager (EXM) guides through the process of A/B testing but please consider the following factors before you start A/B testing:

  • Decide on the number of recipients to be included in the email campaigns (always include a small pool of recipients in the test to get a reliable result)
  • Confirm on which part of the messages needs to be tested for e.g. either the subject line or any components in the content area, text, links, images etc.
  • Decide on the number of variants to be included in the A/B test. Always consider the number of recipients proportional to the number of variants.
  • Selection of the message variant winner can be either manual or automatic

When you want to run an A/B test, you need to create variants of the original message campaign:

To create a new variant of a message:

  1. Open the message for which the A/B test needs to be performed.
  2. You can add a new variant or duplicate the original variant and all its content also modify it.
    • To add a new variant for the message, go to the message tab, click Add Variant as per the screen-grab:
    • This creates a new message variant based on the same template as the original message but no content will be replicated.
    • If we need to duplicate the original message variant then click the arrow button next to quick test tab and select the "Duplicate this variant" option. This option can be chosen if you want to test minor differences between variants such as subject line, name of a button or image.
    • Message variants are displayed just below the tabs and you can easily toggle between the variants.
    • Please note that you can create as many variants of the message, but when you add more variants you need a larger pool of recipients for the test results to be clear and reliable. Hence it’s recommended to test against maximum of two or three variants.
  3. Make the necessary changes to the new message variant. Once done with the changes to all the variants click Save.

Start an A/B test:

Next go to the Delivery tab of the relevant message, select the variants which you want to be included in the A/B test. For this example here, the variants A and B are selected.

Following options are seen under the Delivery tab:

Size of the test: From the drop down menu, select the percentage of recipients that needs to be included in this test. This set of recipients will be randomly generated from the full list of recipients.

  1. Select Winner: There are two options by which you can specify how the winner can be chosen.
    • Automatic: By choosing this option, the system selects the winning variant based on either best value per visit or highest open rate. Also here you need to specify how long you want the test to run before the winner is selected.
      • If you select “Best value per visit” the winning variant is based on the one that generates the most value on the website per visitor.
      • If you select the “Highest open rate” the winning variant is the one that was opened the message most often.
    • Manual: This option allows you to review the reports of the A/B test and select the winner based on your own criteria.
  2. Schedule Delivery:
    It lists down the following options:
    • Send message now: It starts the A/B test immediately.
    • Schedule message delivery: It schedules the delivery of the A/B test message for a later time. Here you can specify the date and time to start the A/B test.
    • Schedule a recurring delivery: This option allows to send the A/B test message recurrently at a certain interval, date and time.
  3. Notification: Clicking the Notification check box allows you to send a notification as shown below to the specified email address regarding the completion of A/B test.
  4. Multi-language: Clicking the More option, gives the Multi-language option. It allows you to send the message to the recipients’ preferred language.

Once done with selecting the setting under the Delivery tab, click on Start A/B test button at the bottom (Schedule test button in case of scheduled A/B test)

Clicking on Start A/B test button gives the below screen which confirms whether the A/B test needs to be sent to all the included recipients.

Depending on the option chosen as a method for winner selection (Manual/Automatic), the winning variant get selected.

If the winner is to be selected automatically then after the specified time has passed the EXM sends out the winning variant automatically to rest of the recipients in the recipients list.

Please note that wining message will not be sent to any of the recipients that were included in the A/B test.

If the winner needs to be selected manually then review the A/B test reports and select the variant which is best to be sent out. For the manual option each variant is listed with current achieved statistics:

  • Open rate: This displays the percentage of recipients that have opened the message.
  • Click rate: This is percentage of recipients who have clicked the link in the message.
  • Value: It is the engagement value points generated after the recipients have opened the message.

Based on the results for open rate, click rate and value one can decide which variant will be the best and then click ‘Choose as winner’ option. For e.g. here Variant A has been selected as the winner would give pop up dialog box to confirm.

This is will then send out the winning message to all the rest of recipients. Below is screen grab of the sample email which the recipients will receive.

If you need more detailed information on the A/B test results, click the Reports tab which details about the recipient behavior and message performance.