Archive for the ‘research’ Tag

Scenario Creation

So now I’ve done some interviews with current users, I’ve got my survey and card sort results, I’ve crafted personas of the major user constituencies, and I’ve shown some wireframes to potential users to make sure the navigation fits their mental model.

What next? Taking a page from Alan Cooper’s “Goal-Directed Design” method I’ve created some high level scenarios that convey how the new site could ideally interact with our users and help them accomplish their goals. Mr. Cooper calls these scenarios “context scenarios” and one of mine looks like this:

  1. Don gets an assignment to perform a training session for parents on identifying gang-related behavior at a nearby school.
  2. Don visits and looks up information about gangs. Since he has been to before, the site remembers that he’s a law enforcement officer and displays content appropriate for him.
  3. In the Gangs section, Don sees emerging trends about gangs, training materials about gangs, NCPC products relevant to gang prevention, and NCPC programs that help prevent gang-related crime.
  4. Don skims the emerging trends and then looks through the training materials. He finds a training session geared toward parents and downloads the PowerPoint slides.
  5. After conducting his training, Don shares his experience and comments on the material at

This scenario is purely textual, it mentions nothing about specific interaction details on the site. It’s used to describe what an ideal interaction might feel like to our user.

Then, I took my context scenarios and matched them up with the relevent wireframes to create task scenarios. Task scenarios are much more detailed flows of how the user would typically progress through the interface to accomplish their tasks (we think). Once the steps and screens are put together, I’ll take my basic prototype and see if a user can accomplish the steps in the scenario, and if they do it in the way I expect them to.

The feedback I get from those tasks will lead to design changes I’ll put into the next, more advanced set of prototypes.

Getting Interview Subjects to Talk More

What do you do when you’re interviewing someone, and everything is going swimmingly until you realize that the same answers keep coming out and they’re about 5 words each?

Some people just like to talk more than others – I know I tend to be on the quiet side myself. I’d probably be a terrible interview subject.

So how could someone draw me out without annoying me by asking the same questions over and over?

You’ll need to establish a level of comfort with the subject. This can be very difficult when you’re speaking to them over the phone, especially with all the obligatory things you’ll need to say like “Is it ok if I record this conversation?”

When you first call them up greet them, thank them, and bring up some sort of icebreaker topic. Once you think they’re speaking fluidly you can get started with the real questions.

If they give you a short, blase answer, try asking “Could you tell me a little more about that?”

Elaborate on what they say, paraphrase their answers back to them, and see if that sparks more comments from them.

What suggestions do you have for this problem?

Contextual Interviews vs Regular Interviews

Many user experience practitioners sing the praises of contextual interviews, and with good reason. By traveling to a place where a real user sits and employs your system, and watching them interact with it, you gain insight into their process that you would miss by any other method.

Contextual interviewing can also be expensive with the travel expense of visiting the willing participant and the cost of whatever recording equipment you want to use.

Can the same results be gained by interviewing people over the phone while they use the system? Probably not, but you might come close.

We’re reaching out to some of the users who participated in our surveys and indicated that they’d like to be contacted again to conduct some interviews with us. They’re spread across the country, which we like because our audience is nationwide.

We considered using a remote screen capture tool like TechSmith’s UserVue, and we still might if we feel that we’re missing out on too much data. But for now, we’re going to speak with our participants on the phone while they use our site, and follow along as best we can on our end. Our conference room equipment allows us to record phone conversations, so we’ll be able to keep our interviews for later use.

Note: Always get permission before recording a phone conversation.

Silverback is pretty cool for the price

For guerrilla usability testing (the only kind of usability testing most non-profits can afford)Clearleft’s Silverback seems like a pretty good solution. We tested it out today in preparation for some prototype testing, and once we had our webcam hooked up it worked very smoothly. The interface is simple and straightforward, and at $50 the cost can’t be beat.

On the downside, you seem to have to export your recordings to mov files before you can play them back. This can take several minutes per recording. It would also be nice if there was a way to organize all your recordings within the App more effectively. Overall, it’s a simple app that accomplishes its goals admirably.

Recruiting Research Participants

Finding enough participants for your research can be tough. Professional recruiting agencies charge upwards of $107 for each usability test participant. On top of that, you need to provide an incentive to the participants themselves. That can get steep even for a lot of large companies.

So how do you recruit participants yourself? It depends on what you’re trying to find out and how you’re going to do it.

For our initial surveys, card sorts, and interviews we wanted to gather information from our existing user base. The goal for this phase was finding out who our actual users are and what they’re interested in. We reached out to our site’s registered users and our recent customers and invited them to participate. (Make sure you follow CAN-SPAM law).

After we have a solid understanding of our audience, we can design our basic prototypes. These are sets of lo-fi drawings or sketches that demonstrate how the navigation will look, where the basic page elements will go, and what the content organization will be like. To test these prototypes, we don’t really need to use participants from our existing or even potential user base. What we’re testing is that the infrastructure of the design itself works well for a general audience. So how do you find those participants? We’ve reached out to our local library and formed a partnership to conduct user testing at the branch down the street. You may have existing relationships with the community that you can leverage for this purpose. If all else fails, this is a stage where you can test on your co-workers with the least danger of introducing organizational bias into the data.

If it’s easy for you to get participants from your target audience, or if your site is aimed exclusively at a very specific group of users then by all means recruit those users for basic prototype testing. But most organizations will want to limit the amount of times they reach out to the same group of existing users and ask them for something.

Once you’ve ironed out all the problems uncovered during basic prototype testing, you can fill in the blanks and make your advanced prototypes. These prototypes are reasonable approximations of the final design, so you’ll want to find participants from your target audience. Try reaching out to other groups and organizations in the same field to recruit.

In general, I think non-profits have an advantage when recruiting participants because you’re appealing to their desire to help a good cause. It increases the likelyhood that people and organizations will assist you and reduces their expectations for material rewards. Make sure not to abuse that goodwill and that the groups that help you get something out of the deal as well.

Starting Your Research Efforts

The starting point for any web redesign effort should be finding out what your users think about your existing site, and where they want improvements. You may be thinking of adding a bunch of tools to integrate with delicious/flickr/youtube etc, and that could be great for your site, but is that what people really want?

Instead of starting with ideas for potential features, follow Jesse James Garrett‘s advice and ask yourself “What do people want to accomplish? How does this activity fit into their lives? How can I deliver on those desires?”

But if you’re here you probably know all about that. So how do you actually start finding out what your users think?

We just finished doing our first phase of research on the user base of, the flagship site of the National Crime Prevention Council, and here’s how we did it:

Fortunately for us, our Research department has a Quask server that we used to create an online survey we could link to or send out via email. If you don’t have access to such a tool, there are online substitutes like Survey Monkey.

A survey can tell us about our users’ age/gender/profession, how often they visit the site and for how long, what topics they’re interested in, and what they use the site for. What it can’t tell us is how they think our content should be organized.

To do that we need to create a card sort exercise. In a card sort, you take a list of items (in this case, pieces of content from your website) and have users put them into groups and name them. This gives us a great deal of insight into how our information hierarchy should be designed. There are a few different web-based card sorting tools, for our project we tried Optimal Sort and were pleased with the results. For more info on card sorting, see this Boxes and Arrows guide.

We linked the card sort to the end of the survey and sent it to our site’s registered users (the ones who’ve agreed to be contacted by us) and wrote a blog post on the home page soliciting participation.

Has your non-profit tried this type of research before? Have you done something completely different?