First Look: A/B testing in Dynamics 365 for Marketing

Dynamics 365 for Marketing gets better and better. We just published a bunch of features for you to preview. One of these is the highly awaited A/B testing feature.

You can use email A/B testing to test two slightly different email designs on a small part of your target segment to find out which design is more successful, and then automatically send the winning design to the rest of the segment.

To test this out lets start by creating a segment of 10 contacts.

Next let’s create an email, in which we define an A/B Test with two versions of a subject (version A and version B). Version A will include a dynamic field; the First Name of the recipient:

1: name the test, eg “Testing Personal Subject” (1)
2: click the Version A button (2)
3: type your subject including the dynamic field (3)

Repeat steps 2 and 3, only this time click the Version B button (2) and include the Full Name (3) of the recipient in the subject (3)

Having created a segment and an email with a test, we’ll proceed to create a customer journey and add the two.

When adding the email with the A/B test, you’ll notice the small A and B icons in the upper right corner of the email tile.

They will be greyed out until you flip the A/B testing switch in the Data task pane. After you flip the switch you can set the A/B testing parameters to your liking.

A/B testing parameters:
1: name of test
2: distribution (percentage to receive version A, and percentage to receive version B)
3: metric – opens or clicks (set to Email Opens by default)
4: duration – how long to wait before declaring a winner (set to six hours by default)
5: fall back – what to send if inconclusive test (set to Version A by default)

When the Customer Journey goes live, it starts out as expected by sending version A to 10% of the segment (one recipient, since the segment holds 10 contacts), and version B to another 10%.

Note that the numbers (1) above the email tile reflects the fact that 10 contacts went into to email tile, but eight of them didnt receive an email yet (as expected). They will receive the winning email version once the A/B test concludes.

If we click the View Details link (2) we can understand which two contacts received the test mails.

Another way of checking the results of the A/B test is to scroll to the bottom of the Data task pane of the Customer Journey, where we can click “View Details” to see the results of the test

Opening the first recipients mailbox from above (fig 6) we see that the recipient (Renee Lo) did receive version A (the first name subject), but didn’t open the email.

Same thing can be verified on the Insights tab of the contact Renee Lo.

Opening the second recipients mailbox we see that the recipient (Diane Prescott) did receive version B (the full name subject), and she did in fact open the email.

Same thing can be verified on the Insights tab of the contact Diane Prescott.

Same thing can be verified on the Insights tab of the email it self (two deliveries, one opens). Its one of the key differentiators of Dynamics 365 for Marketing that extensive Insights is readily available across all records in the app helping you to quickly understand the effects of your marketing efforts.

To end this example we will wait the six hours it takes to conclude the A/B test, as defined above (see Fig. 4).

The expected outcome is that the remaining 80% of the segment (eight contacts) gets the version B of the email, since the recipient of the version A email did not open the email, whereas the recipient of version B did open the email.

So fast forward to six hours later (six hours is the duration of the test, ref Fig 4 above).

The customer journey now reflects the fact the all members of the segment did recieve an email (as expected).

One of the eight recipients to receive the winner email is the contact Eric Gruber (2)

And if we open Eric Grubers mailbox, we can verify that he did in fact receive the version B (1).

We can even verify that he received the email in his inbox exactly six hours after the A/B test was initiated – 6:17 PM vs 12:17 AM (2). Nice!

You can now ensure the majority of your segment contacts receives the winning design and be more successful with your email marketing.

Stay tuned for more first looks, including the new multi-column email designer, the new segment designer, and more.

See also

• Early access: Design and run A/B test on your email designs – link
• Dynamics 365 for Marketing August update and early access are rolling out now – link
• My other blog posts on Dynamics 365 for Marketing (can assist you in your journey from beginner to expert) – link