Back to blog

Learn with Threecolts

Small group workshops to help you learn, optimize, and grow.

Learn About Coaching

A guide to A/B Testing on Amazon

Kennedell Amoo-Gottfried
March 22, 2022
September 16, 2022
A guide to A/B Testing on Amazon- Click to read this article on Threecolts Blog today!

Business fundamentals tend to be quite simple, and one of the most important ones is this: the best way to know if something is working is to test it. Repeatedly. 

A fact so obvious it’s hardly worth saying - you must test to see if your website is indeed doing better than how it could be doing. Without it, you're effectively flying blind and potentially missing out on a whole lot of revenue.

But how can you carry out reliable tests while minimizing the variables? Sure, you could do a regular split test, running one version of your site for a couple of weeks and then change it around for the next couple and compare, but that leaves a lot of variables up in the air. 

A difference of two weeks may be imperceptible to you, but to a shopper that time window could represent a whole new set of pressures or motivations that could make them want to buy where they would not have bought the week before. This is especially true near holiday seasons. 

The best way to really test what makes shoppers convert is to test at least two versions of the same thing simultaneously. 

This is where A/B testing comes in, allowing you to show one version of your site to one set of shoppers and another version to another set in real time and in a controlled environment, with all other variables remaining equal.

It used to be that Amazon sellers could only do split test, as simultaneous ones were not available, but the good news is that changed in 2021! Now, Amazon has opened up this opton up through the Manage Your Experiment tool. 

To be eligible to run experiments, you have to be registered as a brand, and any ASINs will need to have passed a traffic threshold. If they have too little traffic, you may need to devote some resources to drive shoppers to your ASIN before you can test. 

How does it work?

To get started, the first thing you do is go to the Manage Your Experiments main screen:

  • Once on the main page, select Create a New Experiment drop-down menu and choose the type of experiment you want to run. 
  • Next, select an ASIN - they will show up if they’re eligible - to run the test on, referred to as a Reference ASIN.
  • When you’ve picked the ASIN, put in the details of your experiment. This includes the name, the hypothesis - which is crucial when looking back at what it was you were trying to test (e.g. “Using a video instead of an image as the main heade will drive conversions”) - and the duration and start dates of the experiment.
  •  Pick the content you want to experiment on. Probably the easiest way to do it is to click the link that says “Start by duplicating Verson A” and then change as you wish to create a Version B.
  • Submit your experiment and check back to see if it was accepted. If it has not, adjust accordingly. 

Experiment content still has to follow the same guidelines as regular content, so keep your product titles to under 200 characters and pick good images. This will go through a validation process before your experiment can go live.

You can edit an experiment while it is scheduled - whether it’s waiting for validation or if it was successfully validated - but once it’s live you can’t alter it, so you would have to create a new experiment if you want to see specific changes.

What makes a good experiment?

First thing’s first - there’s no point doing this kind of experiment if both versions look the same, so make sure that the two versions look very different. You don’t want to be running this kind of experiment just to see the difference a single picture change will make - it’s unlikely to really affect customer behavior and any results will not be that that useful to you. 

What you want to do is play around with entire layout changes, a different approach to copy, text, imagery, color schemes etc., with a view to create a different customer experience that will trigger different reactions. 

Some ideas include: 

  • Changing your product title to include your brand name
  • Simplifying the title to reduce noise, or, conversely, putting more product details in the title
  • Optimizing your main image in different ways
  • Using only lifestyle imagery and appeals, or using only functional imagery
  • Focusing on one set of features over another
  • Using a new set of images and videos compared to what you currently have

Remember - the more data you can collect the better, so ideally you would set the experiment to last around 10 weeks if you want reliable results

What do I make of the results?

Amazon will give you a pretty good list of metrics that show how the versions differ. This includes units, sales, conversion rate, units sold per unique visitor and sample size assigned to that version, as well as a probability score of which version is better.

It will also make a one-year projection of the impact of changing to the better option. This takes the average daily increase in metrics from the winning version and multiplies by 365, but does not take into account the effect of seasonality.

It may turn out that the results are inconclusive, which could be due to a number of reasons, among them the possibility that the versions were too similar or that traffic was too small to yield reliable results. It could turn out that the changes you made were just not something that most shoppers care about very much - you could argue that this is a result in itself. 

Amazon will produce results every couple of weeks or so during your experiment, but be careful to let it run its course and not jump to conclusions based on the peeks you take. The best results will come at the end of the test.

Above all, keep testing. Once you’ve wrapped up one experiment, test it again with a different set of variables, then test again with different ASINs to see if the changes can be applied across your product line. When you’re done with those, just keep on testing. 

There is never too much data. 

Need help?


Schedule a free 30-minute audit call with a specialist today. 

Our Solution Architects are trained to understand your business and present your best options to grow on Amazon. All advice is customized to your needs.

About us


Threecolts acquires, launches, and grows eCommerce software & services, and owns other stellar businesses including Old Street Media, HotShp, SellerBench, Tactical Arbitrage, Bindwise, RefundSniper, ChannelReply, and FeedbackWhiz.

Old Street Media supports businesses with their advertising, inventory management, and other eCommerce services. We collaborate with over 4000 brands and have generated $600M in sales in the past year.

Reach out to HotShp for help with product titles, descriptions, bullet points, social posts, and blog posts.

Reach out to SellerBench for help with shipment reconciliation and FBA fee reimbursements. 99% of FBA sellers are owed money. Get your free audit today.

If you are more interested in the #1 Amazon Arbitrage software, try TacticalArbitrage. With over 6,000 users, TacticalArbitrage will help you discover profitable products to resell on Amazon.

Bindwise will help you to identify costly issues with your Amazon seller accounts. Receive instant Bindwise Alerts about everything related to your store on Amazon. 

RefundSniper is an international Amazon reimbursement service that runs audits on both Vendor Central and Seller Central. Find out how much you are owed by Amazon today.

If you're looking for a way to streamline multi-platform ecommerce support, ChannelReply has you covered. Cut your customer response time in half by having all your customer information in one hub. 

FeedbackWhiz is an Amazon sellers management software that helps merchants scale their business by automating email campaigns, improving seller feedback, getting more product reviews, monitoring listings, and analyzing profit and accounting data.

Learn with Threecolts

Small group workshops to help you learn, optimize, and grow.