Why (Some) Startups Should Have 25K Beta Users

 

Some people tell me I’m crazy when I tell them I’ve had 25,000 beta users for my startup. “If you have that many users, why wouldn’t you just launch your business and make money off of them?”

Sure, there is logic to these arguments, but we have a different business model than most.

We are not building the most downloaded app or the next startup in the sharing economy or anything in social media aiming at being the next fad. We are building a different type of Engine – an engine that can measure one’s creative thinking ability, something maybe even more advanced than IBM’s Watson or Google’s existing NLP method, something that will set a standard for posterity.

While this approach is uncustomary, it was the right move for us. It is a methodical process that is reaping many benefits already.

But, is it right for your startup and how many beta testers are needed to make your startup succeed?

How We Did It

It took us 1.5 years, 4 full-time employees, 3 consultants, $200K in funding and 25K test cases and early adopters to build our first MVP and find our first real customer. By far, the hardest part was getting those 25K beta test users. This was crucial as we needed these test cases, so we can evaluate, calibrate and modify our engine as needed.

• Started with family, friends and other startup founders: We started early with friends and family first. Every single game we designed, even the crappiest one was tested by our friends and family. This feedback was crucial as we were able to iterate on our product without exposing it to the masses. The one concern we had was getting “true feedback.” This can be a concern for some as your mom or your good friend may reserve comment to avoid hurting your feelings. But, we established up front that we want all feedback – no matter how harsh. Of course, they still kept their reservation. So, then, we approached our friends and founders in other startups. They know how valuable the unfiltered and uncensored feedback is. And we are lucky as we had quite a few of them.
• Worked with larger more general survey-type organizations: We used “Mechanical turk” (Mturk). Most founders are familiar with Mturk as it’s an easy way you can test your product at a fairly large volume. Of course, this service is not free. But it is worth it. We were running in average 200-500 test cases at a time every other week on Mturk for over a year. It was a great way to catch the bugs in the system, gather data, see what responses the end-users are providing and how they are using the system. Furthermore, we were running A/B testing where you select users from certain regions in the world and compare their behavior with respect to each other.
• Partnered with reputable educational institute in a MooC study: Although mturk data was invaluable, we wanted to find users that would push the system to its limit; we also didn’t want them to have monetary incentive to just use the system, provide minimum feedback and forget about us. We really wanted to find users that have a “higher” purpose when they were performing our exercises. What better way is there than partnering with a MooC course on Coursera from a reputable institute? The MooC had over 15,000 registered users and as part of an on-going research, we performed the pre, mid and post assessment for them. I must say that there is nothing like it when all of a sudden you have over 5,000 users log into your system and start playing. You find bugs that you could never even conceive that they would occur. Now, this begs the question how many beta testers did we really need and how many does your company need.

Getting the Most Out of Beta Testers

For early-stage startups, nothing is more important than figuring out what experience their end-users have and how they use it. Don’t be shy to push your beta testers. Don’t be shy to ask questions. And don’t be shy to send reminders to them. As we were running tests and partnering with different companies, not only did we asked them to push themselves and do the best they can, but we asked them to fill out surveys after surveys. Of course, we respected their privacy and made the surveys optional. When we explained to them why and what we are doing and they were more than happy to help us.

How Many Beta Testers Does Your Startup Need?

Few startups are research intensive like ours. The more research-intensive you are, the more # of beta users you need to both validate your product and prove to your first customers that it actually works. The short answer is, as many as you can till you get your first customer.

Note that my startup is in the B2B space. Your first customer is taking a chance on you. So they are more than willing to provide feedback and responses. And remember that success doesn’t happen overnight. Be persistence and you’ll get the results.

Regardless of your startup or how many beta testers you need, here’s what we learned from testing:

• Put Your Product in Testing ASAP. Get feedback, get it now and iterate. If you don’t have anything to test yet, bounce the ideas off of people you know or do a survey to get some quantitative and qualitative insights. Don’t be shy. Ask questions from your participants. You don’t want to build up an idea in your head, only to find your idea has no merit.
• Test Different Markets. While you may think you know what market you’re going after, let the market decide this part. We approached schools first, then businesses, then academia, then HR personnel, then psychologists and consultants. But at any given time, we were ONLY focused on one of them. Do not attack more than one market group at a time. As a startup, chances are you don’t have the resources. We had an idea of who would be receptive to a product like ours, but we never truly knew which markets and which people in those markets would find it most valuable. Our first customer ended up being a management consulting organization in the field in Australia, the 2nd and 3rd customers were corporate training organizations in the U.S.
• Talk to Similar Companies. Don’t recreate the wheel. There may not be any other company like yours, but there are likely companies out there that have done similar or parallel launches that you can learn from. Find those founders and learn from them. Don’t be scared that they might steal your idea. Some of our prospects and partners, are companies in the same space. Usually, there are synergies. You never know; maybe down the line, you might refer a customer to them or vice-versa.