As product managers, designers and marketers, we love the clarity that comes from A/B testing. We have an idea or design of what we want to test, a theory on why and we let the users behavior determine the outcome from a controlled experiment.
Be aware of A/B testing has limitations and variabilities possibly outside of your control.
- Assumes you have tested and have the right JTBD or Workflow.
- Limit of two options will save time and reduce cost but minimizes insights.
- Temporal effects can be seen from running the test to long or short.
- Benefits may be short lived or cyclical.
- Mode effect from different devices or screen sizes may skew results.
- Interaction effects can amplify results.
- The more tests you run, the higher the chance of error.
- It may not account for all intermittent or unknown variables.
- It’s not multivariate, mixed method or qualitative research.