First Look: A/B testing in Dynamics 365 for Marketing

Dynamics 365 for Marketing gets better and better. We just published a bunch of features for you to preview. One of these is the highly awaited A/B testing feature.

You can use email A/B testing to test two slightly different email designs on a small part of your target segment to find out which design is more successful, and then automatically send the winning design to the rest of the segment.

To test this out lets start by creating a segment of 10 contacts.

Fig 1: Segment

Next let’s create an email, in which we define an A/B Test with two versions of a subject (version A and version B). Version A will include a dynamic field; the First Name of the recipient:

1: name the test, eg “Testing Personal Subject” (1)
2: click the Version A button (2)
3: type your subject including the dynamic field (3)

Fig 2: Version A of Subject

Repeat steps 2 and 3, only this time click the Version B button (2) and include the Full Name (3) of the recipient in the subject (3)

Fig 3: Version B of Subject

Having created a segment and an email with a test, we’ll proceed to create a customer journey and add the two.

When adding the email with the A/B test, you’ll notice the small A and B icons in the upper right corner of the email tile.

They will be greyed out until you flip the A/B testing switch in the Data task pane. After you flip the switch you can set the A/B testing parameters to your liking.

A/B testing parameters:
1: name of test
2: distribution (percentage to receive version A, and percentage to receive version B)
3: metric – opens or clicks (set to Email Opens by default)
4: duration – how long to wait before declaring a winner (set to six hours by default)
5: fall back – what to send if inconclusive test (set to Version A by default)

Fig 4: A/B test configuration

When the Customer Journey goes live, it starts out as expected by sending version A to 10% of the segment (one recipient, since the segment holds 10 contacts), and version B to another 10%.

Note that the numbers (1) above the email tile reflects the fact that 10 contacts went into to email tile, but eight of them didnt receive an email yet (as expected). They will receive the winning email version once the A/B test concludes.

If we click the View Details link (2) we can understand which two contacts received the test mails.

Fig 5: Customer Journey sends mail to 20%, 80% awaits conclusion of A/B test
Fig 6: 20% of segment members received mail

Another way of checking the results of the A/B test is to scroll to the bottom of the Data task pane of the Customer Journey, where we can click “View Details” to see the results of the test

Fig 6b: results

Opening the first recipients mailbox from above (fig 6) we see that the recipient (Renee Lo) did receive version A (the first name subject), but didn’t open the email.

Fig 7: First recipient, Renee Lo, received version A and did not open the mail

Same thing can be verified on the Insights tab of the contact Renee Lo.

Fig 8: Insights tab reflects the delivery and that mail wasnt opened

Opening the second recipients mailbox we see that the recipient (Diane Prescott) did receive version B (the full name subject), and she did in fact open the email.

Fig 9: Second recipient, Diane Prescott, received version B and did open the mail

Same thing can be verified on the Insights tab of the contact Diane Prescott.

Fig 10: Insights tab reflects the delivery and that mail was opened

Same thing can be verified on the Insights tab of the email it self (two deliveries, one opens). Its one of the key differentiators of Dynamics 365 for Marketing that extensive Insights is readily available across all records in the app helping you to quickly understand the effects of your marketing efforts.

Fig 11: Insights tab on customer journey reflects count of deliveries and opens

To end this example we will wait the six hours it takes to conclude the A/B test, as defined above (see Fig. 4).

The expected outcome is that the remaining 80% of the segment (eight contacts) gets the version B of the email, since the recipient of the version A email did not open the email, whereas the recipient of version B did open the email.

So fast forward to six hours later (six hours is the duration of the test, ref Fig 4 above).

The customer journey now reflects the fact the all members of the segment did recieve an email (as expected).

Fig 12: Customer Journey sent winner mail the remaining 80%

One of the eight recipients to receive the winner email is the contact Eric Gruber (2)

Fig 13: All remaining members of segment incl Eric Gruber recieved the winner mail

And if we open Eric Grubers mailbox, we can verify that he did in fact receive the version B (1).

We can even verify that he received the email in his inbox exactly six hours after the A/B test was initiated – 6:17 PM vs 12:17 AM (2). Nice!

Fig 14: Eric Gruber recieved the winner mail – version B of Subject

You can now ensure the majority of your segment contacts receives the winning design and be more successful with your email marketing.

Stay tuned for more first looks, including the new multi-column email designer, the new segment designer, and more.

See also

  • Early access: Design and run A/B test on your email designs – link
  • Dynamics 365 for Marketing August update and early access are rolling out now – link
  • My other blog posts on Dynamics 365 for Marketing (can assist you in your journey from beginner to expert) – link

One thought on “First Look: A/B testing in Dynamics 365 for Marketing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s