Public Beta – Any Point?May 7, 2007
I’ve never seen any data to back up the following, but I seriously doubt that ‘Public Betas’ serve the purpose that they are intended to do. Having said that, I’ve never read what the stated purpose of a Public Beta is.
I’m sure to some extent they are useless. If a product is good enough to release to the public, then ship it. All a Public Beta will do is arouse a sudden flash of enthusiasm, from the inquisitive who want some ‘bragging rights’ down the pub and possibly alienate your professional Beta testers, though the second is unlikely. Most betas in a Public test will loose interest pretty rapidly, though obviously some will not.
A Public Beta is also likely to generate a fire hose effect of problems that are already fairly well known and that could easily swamp a heretofore unknown bug. This raises some questions in itself.
Do you really have enough staff in place to filter this torrent of information?
Are you going to respond to every report from every tester?
If the answer is no to either of these, then the point of a Public Beta is lost. Especially so in the second case, these people are volunteers and the lack of acknowledgement will quickly dampen their enthusiasm to the point that they are unlikely to respond further. This could easily lead to resentment, which isn’t something software companies want to build in their user base.
Just as effective command and control is needed within the development cycle, it is needed as much in the Beta phase.
If there must be a Public Beta, invite people to register and undertake an obligation to post their findings. Even then don’t expect everyone who applies to actually test something. Some users will use the software in the way they use the existing version, (if it’s an update that’s being tested), then not find anything wrong and post nothing.
Ideally after the registration process, the software company sends the software or at least a URL to where it can be downloaded from to the tester. This way it is possible to establish some control and stream updates to a group of testers at a time. Stagger the releases each week and then there will be some new refreshed testers every time a milestone is reached, thus maintaining interest. The obvious danger here is that the first group of testers do all the testing and the others will have a life of posting duplicates. Staggering the stagger is vital.
Once the product is shipped, send a free copy to every tester who posted some kind of response, good, bad or indifferent. They’ll be there next time they’re needed.