How To Hit A Home Run With Your Next A/B Test


One of the most common mistakes I’ve seen enthusiastic A/B testers make is running random tests.

Random tests have very little chance of moving the needle very far.

But how do you choose what to test?  Intutition?  Copying your competitors’ sites?  Finding a list of “3,452 failproof tactics to try before your SaaS turns 50”?

That’s what most people do.  But it’s not what the pros are doing.

The best in the business use a process to gain insight before testing.

The best in the business want answers to these questions: What do your users and visitors care about?  Why did they buy from you instead of your competitors?  Why did they not buy?  Are they getting confused, stuck or turned off somewhere on your site?  What doubts, hesitations and fears do they have?

Once you have answers to these questions, you’ll be able to aim your split tests in the right direction instead of taking shots in the dark.

When I started testing I had no idea how to pursue insight.  So I did what everyone else does.

Now I know better.  I thought I’d share:


1. Watch movies of every single visitor on your site with Inspectlet
One line of javascript and you’ll be able to watch session recordings of all your visitors doing whatever they do on your website.

And they won’t even know they’re being recorded.  No joke.

After watching several of these soundless videos, it often becomes very apparent where people are tripping up in your sales funnel, and where there are user experience difficulties.



2. Usability testing via

Go to and get three free mini usability tests done of your site.  Right now.

Don’t forget to come back to this article and thank me ;).

This service sends a random person to your site; they record their voice and their screen and they narrate what they’re thinking as you watch.

It’s beautiful.

If you have a revenue-generating business it is worth it to pay for longer tests at

You’ll start to notice striking patterns within about 5 tests.



3. On-site surveys with Qualaroo (enterprise) or HotJar (free)
This method is often overlooked — but very powerful.

Put a little popup on parts of the site where you want to know what is leading to (in)action. Ask people on the pricing page what is stopping them from signing up. Ask people who just signed up what they’re hoping to accomplish.


4. Open yourself up to feedback in your “welcome” email
In your welcome email, ask for feedback from people.

Use this as a launching off point for conversations to better understand your customers.


5. Understand why people paid
After people pay you for their first time, you want to know what motivated them to pay you and what nearly stopped them from doing so.

Send a short questionnaire to customers to find out, soon after they pay.

As with most dialogue, it is in your interest to keep it open-ended.


6. Exit surveys via email or Intercom
Churn is a part of life.  But that doesn’t mean you can’t learn from it.

If a user doesn’t sign up for a paid plan after their free trial, shoot off an automated email and find out why. This not only gives you valuable information, but also may give the relationship a second chance.


7. Allow people to give feedback on whatever part of the site they want:
I have to admit I haven’t used this method… yet — but I was delighted to run across this new way of eliciting feedback from customers.

Created by a fellow I know from


8. Walk a mile, or 10,000 in their shoes with a simple trick
All new users, on their way to becoming a paid user, have to see the onboarding.

How often do you go through it?  You probably spend 95% of your time inside your SaaS, and that’s a mistake.  I’ll repeat: ignoring user onboarding is a huge mistake.

If you don’t test, analyze and refine your own onboarding very often, you should probably start.

In the past I’ve hesitated to go through any one signup flow too many times because I only have a few email addresses.

But if you have a gmail address it’s easy to do.

Let’s say your email address is is

I can use an infinite number of gmail addresses by putting “+randomText” after my name and before “”.


All these will, of course, send email to


9. Usability hub

If people don’t understand what your site is about, they won’t buy from you.

Therefore, making sure your site is easily understandable is very important.

There are a couple of options on, but the 5-second test is one that I find the most useful.

All you do is upload an image of your site.  Testers will be shown the image of your site for 5 seconds, and they have to answer a question of your choice, afterwards, like “what do you think this site was about?”.

You’ll quickly be able to separate the unclear designs from the winners.


10. Talk to your angry customers
This one’s from Trajan King of Invincible Startup. Talk to your angry customers. It’ll be uncomfortable, but you’re likely to get real answers and insights from customers who aren’t happy with you.


That’s all for now!

This is a 30,000 foot view of ways to get feedback from users.  Have any questions?  Comments?  I’d love to learn what’s worked for you, below!

Leave a Reply

Your email address will not be published. Required fields are marked *