Some weeks ago we started our first experiment using a low fidelity Minimum Viable Product for one of our projects (Whatson) and the first questions came up before we even started:
How to structure such an experiment? What are the indicators to mark it as a success? How much time should be invested?
The Low Fidelity Minimum Viable Product
There are different kinds of MVPs and as we are just starting this project, we used a low fidelity MVP. This can be something like a paper mock-up or a landing page describing the most important features and problems you want to solve. Stefan Roock provides a nice illustration and classification of MVPs here.
The intention of the Lean Startup way is to test your hypotheses and get customer feedback as soon as possible, even if you do not have a working product yet.
Before we decided to create the landing page, we held customer problem and solution interviews, as suggested by Ash Maurya. Those helped us to improve our understanding qualitatively, but now it was time to present our product idea to a broader audience.
Conversion Funnel
When it comes to web based products or mobile apps potential buyers go through different stages before becoming an actual customer. Steve Blank describes this as the "Get, Keep and Grow" Customers Funnel in his book The Startup Owner's Manual.
As we do not a have a working product yet the phase we concentrate on is the Get Customers phase, which includes two major steps:
Acquisition: That
is when your potential customer visits your website, by clicking on a
paid Google Ad or a Facebook message or whatever channel you
are using.
Activation: That is the point were the visitor becomes a customer, by downloading your product, signing up, etc.
The numbers
So the basic question we actually wanted to answer was, do enough people care about the product and do they find it compelling enough to tell their friends.
Without a working product, however, these questions are not that easy to answer, especially the latter.
As we did not have a usable product we were not quite sure what to define as activation. Another unclear point was the number of acquired and activated customers. How many do we need to mark this experiment as a success?
First we decided that entering your email address can be counted as an activation, as this is commonly used as an indicator.
Steve Blank defines 44% activations of 5000 visitors is a huge success, whereas if you got just 50 sign ups, you should check what went wrong. Furthermore he suggests to start such an experiment as soon as possible.
Furthermore I found this answer on Quora by Joshua Ledgard, Founder - KickoffLabs.com:
"5% is bad.
10% is OK.
20% is ideal.
30% is ahead of the curve."
Furthermore I found this answer on Quora by Joshua Ledgard, Founder - KickoffLabs.com:
"5% is bad.
10% is OK.
20% is ideal.
30% is ahead of the curve."
Draw a line in the sand
Even though we now knew the percentages, we still had to come up with absolute numbers. The book Lean Analytics
confirmed what we actually thought: draw a line in the sand. That
means, at the beginning pick any reasonable number and define a goal. It
is more important to compare the result afterwards and most of the time
you will be somewhere in the middle anyway.
So here is what we came up with. We divided the activation step into two different parts:
Customer giving us their email address and customers liking us on Facebook.
Our goal for the next 2 weeks: 150 Facebook Likes, 20 email addresses and a conversion rate around 30%.
Channels: Google Adwords, Virality through chosen friends, Online forums
We did not acquire anybody through Facebook or Twitter status updates or invite people directly, but included some of our friends. We sent a prepared message to around 30 of our friends and asked them to spread the word.
Result
After optimizing our Google Adwords campaign (using this great article from KISSmetrics) we had a good enough click-through rate and
about 30% of the visitors entered their email address.
Unfortunately we did not have exact numbers on the people that were reached on Facebook, which made it hard to argue that a page-like counts as an activation.
So we decided to make another experiment and verify the engagement of our page-fans. We created an account on Next-Fat-Business and asked our page-fans to give us their vote. At that time we had 115 likes and ended up with around 45 votes.
This was all done within 2 - 3 weeks, which is not that long. During that time we invested some hours per day to promote the product, improve the Adwords campaign, talk to the potential customers and tweak the landing page itself.
This was all done within 2 - 3 weeks, which is not that long. During that time we invested some hours per day to promote the product, improve the Adwords campaign, talk to the potential customers and tweak the landing page itself.
Lessons Learned
Facebook does not provide such a virality as it used to. After having 100 likes it is quite hard to acquire new customers without paid ads. Of course it is quite difficult to create virality without a product or fancy video.
Verify the engagement of your Facebook fans and activate them with a second experiment.
Activation can be anything: entering an email address, a fake sign up or a call for action.
We think that absolute numbers are not that important, but the trend and a little bit of gut feeling.
Use this experiment to evaluate your channels, because acquiring new customers won't be easier even with a working product.
Friends promoting a product to their friends works really well.
After two to three weeks you get a first impression of how appealing your product is. If at any time the conversation rate drops below 10 % stop the experiment and verify why. Furthermore promoting your landing page and product should be something you do consistently from the beginning.
Use this experiment to evaluate your channels, because acquiring new customers won't be easier even with a working product.
Friends promoting a product to their friends works really well.
After two to three weeks you get a first impression of how appealing your product is. If at any time the conversation rate drops below 10 % stop the experiment and verify why. Furthermore promoting your landing page and product should be something you do consistently from the beginning.
Did you have any similar experiences? How did your experiments do?
P.S. I’d love to talk to you on Twitter: here.
** You might also like: How to get Customers' Attention **
P.S. I’d love to talk to you on Twitter: here.
** You might also like: How to get Customers' Attention **