Are you finding the right participants?
Many UX professionals seem to differ when it comes to finding the right participants for usability testing. Can you just grab anyone? Or do the subjects need to be existing users?
Harry Brignul illustrates the dangers of using just anybody as a usability test participant on his blog, with some convincing examples.
On the other hand, Matt Brown from last.fm says he’s gotten some good feedback from his admittedly “loose, informal user testing” at a coffee shop mixing existing users with random passersby.
Who’s right? Obviously, it depends.
Harry’s example uses a specialized application for a certain group of sales people. Matt is testing designs for a music site aimed at the general public.
If you’re targeting the general public, you want to make sure you get a good mix of demographics in your user base. If your application is for a specific audience, not testing with members of that audience is an insanely risky proposition.
Most likely you’ll fall somewhere in between. At NCPC, we serve a lot of law enforcement personnel and folks interested in community improvement, but we also have materials for parents, teenagers, and kids. We need to make sure our offering is flexible enough to handle our disparate audiences, so we have to try and test with all of them.