Is A/B testing a replacement for user feedback? A/B testing is awesome, but it’s great for large companies with dedicated data engineering teams, not for early-stage startups watching their runway. Why?
In this article, let’s compare direct user feedback with A/B tests. Which is better to make a product decision?
A/B testing as it is
So, what are the nuances of A/B testing? Why do they talk about it so much and for which companies it is suitable.
You need large enough testing sample
To do A/B testing, you need to have a large sample. First, you need to have a sufficient user base to mark up sample A and sample B, and both samples must be large enough, otherwise they won’t be statistically significant.
There are even special calculators for calculating sample size, and sales samples can cost more than some startups 🙂
It takes plenty of time
You will need patience and time. First you need to understand exactly what you want to test, then partition users into cohorts, set up testing on the frontend and collection of results on the backend, coordinate teams of engineers and analysts. Large companies even make dedicated platforms for A/B testing.
The next step is to wait until the test is underway. If you’re testing the color of the signup button, those results can be collected relatively quickly, but if you’re testing the pricing, you’ll need to wait a few months for the cycle to complete.
So the entire test can take anywhere from a few weeks to a few months.
It can only be used to test a specific hypothesis
The test object must be specific and version A must differ slightly from version B, otherwise the test will not show the difference in performance and you will not even understand what you are testing.
What tests well through A/B: colors of certain elements of your app (buttons, links), headers, call to actions, etc. The main rule: one change for one test.
So, user feedback or A/B tests?
So A/B tests are useless? Not really. But let’s elaborate on what’s best to use and in what case.
You need an answer to the question: what to implement next?
To figure out which features to implement next, A/B tests won’t help you. You need a customer interview, a questionnaire, or a special platform to collect feedback.
Productroad lets you effortlessly create a platform for your product where users can leave feedback and vote for features.
You need an answer to the question: what is working better?
Suppose you have thousands of customers on the trial and you need to determine how best to convert them into paid subscribers. Asking the customer which button he’d be more willing to click to subscribe to, the red or the blue one, would be strange. In this case, A/B testing is ideal for you.
Let’s summarize
User feedback and A/B testing technologies are not interchangeable. You can’t say that one is better than the other. Each performs its task under certain conditions, each answers a specific question for you.
With this article we wanted to show that there are no Ultima solutions, each technique should be chosen for a specific purpose and for solving a specific problem.