Archive for October, 2008|Monthly archive page

Are you finding the right participants?

Many UX professionals seem to differ when it comes to finding the right participants for usability testing. Can you just grab anyone? Or do the subjects need to be existing users?

Harry Brignul illustrates the dangers of using just anybody as a usability test participant on his blog, with some convincing examples.

On the other hand, Matt Brown from last.fm says he’s gotten some good feedback from his admittedly “loose, informal user testing” at a coffee shop mixing existing users with random passersby.

Who’s right? Obviously, it depends.

Harry’s example uses a specialized application for a certain group of sales people. Matt is testing designs for a music site aimed at the general public.

If you’re targeting the general public, you want to make sure you get a good mix of demographics in your user base. If your application is for a specific audience, not testing with members of that audience is an insanely risky proposition.

Most likely you’ll fall somewhere in between. At NCPC, we serve a lot of law enforcement personnel and folks interested in community improvement, but we also have materials for parents, teenagers, and kids. We need to make sure our offering is flexible enough to handle our disparate audiences, so we have to try and test with all of them.

Ad Council Partner Conference – Social Media

Today I got the opportunity to go to the Ad Council Partner Conference at the National Association of Broadcasters. The presenters in order of appearance were Ian Schafer (Deep Focus), Michael Becker (iLoop Mobile), John Roland & Tim Conley (Extreme Reach), and Maurice Boissiere (Clear Spring).

Overall the presenters did a very good job. Mr. Schafer said he’d post his slides on his website later today.

It was also pretty cool that one of our PSAs was used by John Roland as an example of a viral video calling attention to a worthwhile cause.

One neat thing that Michael Becker brought up was the ability of 501(c)3 organizations to accept donations via the next generation of sms text messages, though I wish it had been part of his main presentation (which was still the most intriguing in my view).

There were a lot of mentions of Facebook, MySpace, and YouTube though I would’ve liked to see a few more examples of what organizations can do with their own web properties in terms of creating a unified user experience that ties into social media. There were some good examples, especially in relation to mobile devices but there were also a lot of generalities.

Getting Interview Subjects to Talk More

What do you do when you’re interviewing someone, and everything is going swimmingly until you realize that the same answers keep coming out and they’re about 5 words each?

Some people just like to talk more than others – I know I tend to be on the quiet side myself. I’d probably be a terrible interview subject.

So how could someone draw me out without annoying me by asking the same questions over and over?

You’ll need to establish a level of comfort with the subject. This can be very difficult when you’re speaking to them over the phone, especially with all the obligatory things you’ll need to say like “Is it ok if I record this conversation?”

When you first call them up greet them, thank them, and bring up some sort of icebreaker topic. Once you think they’re speaking fluidly you can get started with the real questions.

If they give you a short, blase answer, try asking “Could you tell me a little more about that?”

Elaborate on what they say, paraphrase their answers back to them, and see if that sparks more comments from them.

What suggestions do you have for this problem?

Contextual Interviews vs Regular Interviews

Many user experience practitioners sing the praises of contextual interviews, and with good reason. By traveling to a place where a real user sits and employs your system, and watching them interact with it, you gain insight into their process that you would miss by any other method.

Contextual interviewing can also be expensive with the travel expense of visiting the willing participant and the cost of whatever recording equipment you want to use.

Can the same results be gained by interviewing people over the phone while they use the system? Probably not, but you might come close.

We’re reaching out to some of the users who participated in our surveys and indicated that they’d like to be contacted again to conduct some interviews with us. They’re spread across the country, which we like because our audience is nationwide.

We considered using a remote screen capture tool like TechSmith’s UserVue, and we still might if we feel that we’re missing out on too much data. But for now, we’re going to speak with our participants on the phone while they use our site, and follow along as best we can on our end. Our conference room equipment allows us to record phone conversations, so we’ll be able to keep our interviews for later use.

Note: Always get permission before recording a phone conversation.