My company Sunstone Games (www.sunstone.co) has recently started a beta test for our first mobile game: Sun Stones.
Shameless plug - if you'd like to play and give feedback, just send mail to firstname.lastname@example.org
This has given me an opportunity to explain to my co-workers what, exactly, an open Beta is for. I thought it might be interesting to the world at large as well, so I've collected my thoughts here.
What a Beta is NOT
1 - A Beta is not "game testing"
Testing is a very important, ongoing need for professional developers. A good test plan puts the whole game through a simple or deep stress test every day or two, looking for broken bits. A professional testing team will hammer on all parts of the game, looking for broken system & features. Testers can find missing or bad textures, geometry problems, spelling errors, etc. But that's not what a Beta is for.
2 - A Beta is not content complete
In console development, at least here in Eugene, "beta" is used internally to mean that all of the content in a game has been created. ("alpha" means all the systems have been created.) But because of the technical limitations of console work, only people on-site can ever see the game before it is released. That's not what a Beta is for.
3 - A Beta is not an early launch
Marketing people sometimes focus way too much on the "launch window" - and want to do everything in their power to ensure that the maximum number of units are moved in that period. For these people a Beta can serve as a way to get units out ahead of the actual window - allowing them to artificially inflate their launch-window sales later. That's not what a Beta is for. (Nor is that what marketing folks should be doing!)
What a Beta IS
A beta is about user reactions - specifically how their reactions validate or invalidate the assumptions you made about your potential customers during the development cycle.
Most of that definition hinges on a particular business-style reading of the word "assumptions," so let's drill down on that word just a bit by giving an example.
Let's say you are sick of buying ugly-looking carrots, so you genetically create a strain of carrots which are bright orange and perfectly formed every time. They cost a bit more to grow, because they can only survive in hydroponic gardens. Your first batch are pretty good, but they don't taste quite as good as normal carrots. You think to yourself "nobody will buy a prettier, more expensive carrot if it doesn't taste great!" So you spend more time, money, and research adjusting your carrots until they taste just like normal carrots.
So you release your carrots, and get them into a few dozen grocery stores to start. But it turns out that your carrots are so bright and beautiful that they look fake - and very few customers are willing to pay more money for a carrot which looks phoney.
The assumption you made, initially, was that people wanted their carrots to be prettier. You based your entire company on this assumption. Since this assumption turned out to be false, all of the other work, effort, and assumptions you made (it should taste good, etc.) were all invalid.
Sometimes people think that a Beta will prove the quality of a product - but quality only matters if your baseline assumptions are valid. A Beta is all about validating your assumptions.
In the case of Sun Stones, our assumptions are something like this:
1 - Players will enjoy the process of discovery.
2 - Players prefer a sense of accomplishment over easy answers.
3 - Players will appreciate a strong unifying theme which is unbroken during play.
4 - By not making assumptions about our players (gender, race, nationality, age, familiarity with games) we will be able to appeal to many non-traditional gamers.