A/B Testing

A/B testing is a method of testing that compares the performance of two or more product or service versions A and B, and so on. The most effective A/B testing scenarios are designed to test one main variable (X, Y, count, color, completion rate, etc.). The testing monitors one or two results like level completion, or profitability, etc. For example, if we were to test two to four game versions at a time. We will distribute our game versions amongst our players. Usually, only two to four versions are tested at one given time. In this case, we create three different test cases for our game where we change the number of enemies in the game and see how many more players will complete the level ( in this case, we are testing level 4 of the game). Usually, you always want to test your original version in comparison to the new ones. So, we have an initial version, A, a version B with 10% fewer enemies, and version C with 20% fewer enemies. The next step is to distribute these versions to our players.

We distribute in a way that about a third of our players play each version. This is usually done using percentages. We have 33%, 33%, and 34%, which adds up to a total of 100% distributed to versions A, B, &C. Next, we get our results and see how many players completed each level. For instance, we might notice that only 20% of our Group A players completed level 4, 80% of Group B completed Level 4, and in Group C,70% completed Level 4. These results imply that both options B and C seem like better options than A. We could investigate further why C is not as high as B ( maybe it is too easy, and players get bored playing). Nonetheless, A/B testing results show that version B is our best option (it had the highest completion rate of 80%).




Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Know Everything that you need to About APIs

Incident Postmortem

How to remove items from list in python?

Principle #2 for a Responsive Friendly Design

OpenShift & Use case of OpenShift in BMW

Stack Code Challenge Breakdown

The Riddler Traps Batman in a puzzling knot of wire.

Getting started with Xamarin Forms and Firebase Databases

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Celestin Ntemngwa

Celestin Ntemngwa

More from Medium

One-Way ANOVA Test with RStudio

Tableau for Exploratory Data Analysis(EDA)

Evaluation techniques for interactive systems

Kruskal Algorithm