Skip to main content
A/B testing
Asked a question 2 years ago

I sent my first broadcast email today, and I used A/B testing for my subject line. In my report, it says each one sent to 7 recipients for a total of 14, but at the top under stats it says it was sent to 41 recipients. What happened to the other 27? And it says Option A won even though they both had 7, so I don't get how it was chosen as the winner when it was equal to option B. Can someone explain how none of these numbers make sense?

💡Make the most of your Community Member profile 

Share more about yourself! Click on your gravatar> profile > edit profile 

  • About You tab:
    • Photo: upload a picture of yourself or your brand
    • About You: let other community members know a little about you as a person, a creator, and anything else you'd like to add
    • Banner: add an image of your creator brand, logo, or an image that inspires your creativity
  • Social Media tab:
    • Let other members know where they can find you and your brand across social media platforms. This is a great additional step to connect with other creators in the industry!
Aaron HockleyCommunity Guru
Photographer, Writer, Consultant

Here are all the details58 on how ConvertKit does A/B testing, but let me give you the highlights:

  • ConvertKit will send option A to 15% of the list, and option to 15% of the list.   Note that 7 is about 15% of 41 (with some rounding occurring).
  • Four hours later, ConvertKit will send the message to the other 70% of your list using whichever subject line "won" in the testing.   This is the other 27 folks in your example.
  • Lastly, with the numbers you're sharing, a split test is not going to be statistically significant.  In general you'll need a list of over 1,000 folks to have it show meaningful differences.