Archive for the ‘research’ Category

Prototyping with your CMS

We use the open-source content management system Plone to build our site, and have been very pleased with how powerful and user-friendly it is. In the process of redesigning our site, we’ve conducted wireframe tests and settled on our visual design and information architecture. Now we want to build a more advanced prototype, to test how this design will really look and feel in a browser.

To do this, we set up a copy of our Plone site on another server so we’d have one that we could freely modify and change for testing purposes. The final result ends up feeling very much like we think the redesigned site will feel, but we still have a ton of flexibility to move things around, change what doesn’t work, and correct problems where our users get stuck.

Does anyone else use their CMS as an advanced prototyping tool?

Are you finding the right participants?

Many UX professionals seem to differ when it comes to finding the right participants for usability testing. Can you just grab anyone? Or do the subjects need to be existing users?

Harry Brignul illustrates the dangers of using just anybody as a usability test participant on his blog, with some convincing examples.

On the other hand, Matt Brown from says he’s gotten some good feedback from his admittedly “loose, informal user testing” at a coffee shop mixing existing users with random passersby.

Who’s right? Obviously, it depends.

Harry’s example uses a specialized application for a certain group of sales people. Matt is testing designs for a music site aimed at the general public.

If you’re targeting the general public, you want to make sure you get a good mix of demographics in your user base. If your application is for a specific audience, not testing with members of that audience is an insanely risky proposition.

Most likely you’ll fall somewhere in between. At NCPC, we serve a lot of law enforcement personnel and folks interested in community improvement, but we also have materials for parents, teenagers, and kids. We need to make sure our offering is flexible enough to handle our disparate audiences, so we have to try and test with all of them.

Getting Interview Subjects to Talk More

What do you do when you’re interviewing someone, and everything is going swimmingly until you realize that the same answers keep coming out and they’re about 5 words each?

Some people just like to talk more than others – I know I tend to be on the quiet side myself. I’d probably be a terrible interview subject.

So how could someone draw me out without annoying me by asking the same questions over and over?

You’ll need to establish a level of comfort with the subject. This can be very difficult when you’re speaking to them over the phone, especially with all the obligatory things you’ll need to say like “Is it ok if I record this conversation?”

When you first call them up greet them, thank them, and bring up some sort of icebreaker topic. Once you think they’re speaking fluidly you can get started with the real questions.

If they give you a short, blase answer, try asking “Could you tell me a little more about that?”

Elaborate on what they say, paraphrase their answers back to them, and see if that sparks more comments from them.

What suggestions do you have for this problem?

Contextual Interviews vs Regular Interviews

Many user experience practitioners sing the praises of contextual interviews, and with good reason. By traveling to a place where a real user sits and employs your system, and watching them interact with it, you gain insight into their process that you would miss by any other method.

Contextual interviewing can also be expensive with the travel expense of visiting the willing participant and the cost of whatever recording equipment you want to use.

Can the same results be gained by interviewing people over the phone while they use the system? Probably not, but you might come close.

We’re reaching out to some of the users who participated in our surveys and indicated that they’d like to be contacted again to conduct some interviews with us. They’re spread across the country, which we like because our audience is nationwide.

We considered using a remote screen capture tool like TechSmith’s UserVue, and we still might if we feel that we’re missing out on too much data. But for now, we’re going to speak with our participants on the phone while they use our site, and follow along as best we can on our end. Our conference room equipment allows us to record phone conversations, so we’ll be able to keep our interviews for later use.

Note: Always get permission before recording a phone conversation.

Recruiting Research Participants

Finding enough participants for your research can be tough. Professional recruiting agencies charge upwards of $107 for each usability test participant. On top of that, you need to provide an incentive to the participants themselves. That can get steep even for a lot of large companies.

So how do you recruit participants yourself? It depends on what you’re trying to find out and how you’re going to do it.

For our initial surveys, card sorts, and interviews we wanted to gather information from our existing user base. The goal for this phase was finding out who our actual users are and what they’re interested in. We reached out to our site’s registered users and our recent customers and invited them to participate. (Make sure you follow CAN-SPAM law).

After we have a solid understanding of our audience, we can design our basic prototypes. These are sets of lo-fi drawings or sketches that demonstrate how the navigation will look, where the basic page elements will go, and what the content organization will be like. To test these prototypes, we don’t really need to use participants from our existing or even potential user base. What we’re testing is that the infrastructure of the design itself works well for a general audience. So how do you find those participants? We’ve reached out to our local library and formed a partnership to conduct user testing at the branch down the street. You may have existing relationships with the community that you can leverage for this purpose. If all else fails, this is a stage where you can test on your co-workers with the least danger of introducing organizational bias into the data.

If it’s easy for you to get participants from your target audience, or if your site is aimed exclusively at a very specific group of users then by all means recruit those users for basic prototype testing. But most organizations will want to limit the amount of times they reach out to the same group of existing users and ask them for something.

Once you’ve ironed out all the problems uncovered during basic prototype testing, you can fill in the blanks and make your advanced prototypes. These prototypes are reasonable approximations of the final design, so you’ll want to find participants from your target audience. Try reaching out to other groups and organizations in the same field to recruit.

In general, I think non-profits have an advantage when recruiting participants because you’re appealing to their desire to help a good cause. It increases the likelyhood that people and organizations will assist you and reduces their expectations for material rewards. Make sure not to abuse that goodwill and that the groups that help you get something out of the deal as well.

Starting Your Research Efforts

The starting point for any web redesign effort should be finding out what your users think about your existing site, and where they want improvements. You may be thinking of adding a bunch of tools to integrate with delicious/flickr/youtube etc, and that could be great for your site, but is that what people really want?

Instead of starting with ideas for potential features, follow Jesse James Garrett‘s advice and ask yourself “What do people want to accomplish? How does this activity fit into their lives? How can I deliver on those desires?”

But if you’re here you probably know all about that. So how do you actually start finding out what your users think?

We just finished doing our first phase of research on the user base of, the flagship site of the National Crime Prevention Council, and here’s how we did it:

Fortunately for us, our Research department has a Quask server that we used to create an online survey we could link to or send out via email. If you don’t have access to such a tool, there are online substitutes like Survey Monkey.

A survey can tell us about our users’ age/gender/profession, how often they visit the site and for how long, what topics they’re interested in, and what they use the site for. What it can’t tell us is how they think our content should be organized.

To do that we need to create a card sort exercise. In a card sort, you take a list of items (in this case, pieces of content from your website) and have users put them into groups and name them. This gives us a great deal of insight into how our information hierarchy should be designed. There are a few different web-based card sorting tools, for our project we tried Optimal Sort and were pleased with the results. For more info on card sorting, see this Boxes and Arrows guide.

We linked the card sort to the end of the survey and sent it to our site’s registered users (the ones who’ve agreed to be contacted by us) and wrote a blog post on the home page soliciting participation.

Has your non-profit tried this type of research before? Have you done something completely different?