Usability Testing - few thoughts

Usability testing is a technique used to evaluate a product by testing it on users
Usability testing focuses on measuring a human-made product's capacity to meet its intended purpose. Examples of products that commonly benefit from usability testing are web sites or web applications, computer interfaces, documents, or devices.

Usability testing measures the usability, or ease of use, of a specific object or set of objects, whereas general human-computer interaction studies attempt to formulate universal principles.
If you develop web sites, you need to be doing usability testing. It is that simple.
This tutorial will teach you how to conduct a simple usability test on your web site (using the hands-on task based method).


Plan the Usability Test

1) Know the Goals of the Usability Test

The primary point of usability testing is to provide feedback during the design/development process to ensure that the web site will actually be easy and effective to use and provide valuable information to the users. Four primary elements to measure are:

Ease and effectiveness of navigation - Do users find what they need easily. Is there a clear pattern to the navigation that fits easily into the users mental model. Are you links labeled with terms that make sense to your users. (Or, are you speaking in your own private jargon!)
Usefulness of content - What information do your users want/need? Have you organized the content on each page in such a way that it is easy for your users to quickly find it? Or do they have to read all the fine print while standing on their heads?
Effectiveness of presentation - Did the graphic design, fonts and colors highlight the navigation and content, making the site easier to use? Or did the presentation distract or create a barrier between the user and the information?
Task success rate - Were the users able to accomplish the key task they needed/wanted to accomplish. If they were able to complete the task, did they feel satisfied, neutral or angry and frustrated?
2) Determine Usability Testing Timeframe

Meet with the clients who are requesting the usability testing as well as the developers who are designing the site. Ask the client when they hope to have the site live. Ask the developers when they hope to have the system available for usability testing. Request at least 4-8 weeks between the Usability Testing dates and the "Go Live" Date.

Example of an absolute minimum timeframe:

Week 1
Determine usability goals, timeframe, audience, recruiting plan
Review web site with clients/developers, develop usability test instruments
Week 2
Recruit test subjects
Test the test, make adjustments to test or web interface
Week 3
Conduct the tests and gather testing data
Week 4
Compile data and draft a report, review report with all test facilitators for consensus, produce final report
Present final report to clients and developers. Clients/developers decide what recommendations they will address.
3) Determine the Target Audience & Test Subject Recruitment Plan

Ask the client who the primary audience(s) are for this web site. Try and keep the audience focus down to 2-5 audience types. After the primary audience(s) have been named, considering setting a goal of 3-5 representative test subjects for each primary audience type. For example, when testing the UT Austin home page we had 14 test subjects. See Jakob Nielsen's article on why testing 20 users is enough.

3 Faculty/Staff
3 Alumni
3 Prospective Students
3 Students
2 Students w/ Visual Disabilities
Identify possible test subjects, gather their contact information, establish the week(s) that you will be conducting your testing, schedule locations for the testing, double check with your clients, developers and test facilitators that the testing week(s) are realistic.

Consider offering an incentive for people to participate in the usability test. Common incentives are free lunch or gift certificates. For students, we often order pizza and soda. We have also used $20 gift certificates to local bookstores, music stores or UT related stores. Incentives greatly improve attendance. When we have used incentives, we have reliably had 100% attendance in our test subjects.

-----------------------------------------------------------------------------------
Develop the Usability Test Documents

There are four basic documents used for hands-on task based usability testing. These documents are:

Waiver
Entrance Questions
Task Based Questions (the heart of the hands-on usability test)
Exit Questions
If you don't have MS Word you can download a free viewer for Microsoft Word.

Let's look at the purpose of these docments and see a sample of each.

Waiver - Each test subject should sign a waiver or consent form, indicating that they are giving permission for you to take notes (or video/audio tape) them during the testing. Sample Waiver.
Entrance Questions - these documents help you collect demographic information that you can later use when analyzing your results. Questions include name, age, gender, internet experience and target audience group. Sample Entrance Questions.
Task Based Questions - The heart of hands-on usability testing is the Task Based Questionnaire. During the test, the subject is sitting in front of a computer with the appropriate starting page on the browser. The facilitator verbally leaeds the test subject through a series of questions/tasks, encouraging them to think out loud. The facilitator does NOT lead the subject to the answer. Sample Task Based Questions.
Exit Questions - At the end of the testing session, you will want to allow at least 10 minutes for your test subject(s) to give you their opinion of the site. How easy was it to navigate the site? What did you like or dislike? What was confusing? Sample Exit Questions.
Feel free to use the samples above as your template, making changes as needed to adjust to your specific testing needs. The document that will require the most work will be your Task Based Questions. Here are some pointers on how to write your task based questions:

Key Pieces of Info - Think about your site. What key pieces of information will people need to find on your site? Consider writing a task/question for each of your key pieces of information.

Top Ten - Have too many key pieces of information, then test for the "Top Ten" things people need to get from your web site.

Audience Versions - Don't hesitate to write a slightly different version of your Task Based Questions for each of your target audiences. Different target audiences have different needs on your site. When I create different versions, I usally have a core group of questions that work for all my audiences (perhaps 60-70% of the questions) with a few more audience specific questions.

Non-leading Questions - When writing the text of the question, make sure you are NOT leading your test subject to the answer. Use common vocabularly and specifically avoid the vocabulary that you are using in your hot links and buttons. For example, if I wanted to test the ease of finding the "Campus and Parking Maps" and the link text was "Campus and Parking Maps", I might word the question like this, "You are planning on taking your friend to the Texas Memorial Museum this weekend and you need to find out where you can park."

Simple - the task/question should be simple, so the test subject can keep it in their mind without reading it. Try to write the task in the vocabulary of the target audience.

Realistic Scenarios - The Task/Questions should be realistic scenarios that your target audience would really experience. The point of the test is to simulte being "the fly on the wall" while a real person is using your site.
-----------------------------------------------------------------------------------



Conduct the Usability Test

Pre-test preparations

You've developed your Usability Test, scheduled your test subjects and now your are ready to go, right? But before you bring in your first test subject, you have three important quality checks.

1) Test the Usability Test - sit down at your computer with the usability test in hand, read the questions to yourself, and attempt to do all the tasks. Does the test work for you? Or are parts of the web site not ready for prime time yet? Share the Usability Test with your web site developers so they know what sections of the site you are testing. Make sure that the site will be stable and ready during the dates you are conducting the usability test.

The site doesn't need to be perfect, or finished. Just make sure your questions/tasks are actually doable, and not just a dead end.

2) Practice Giving the Test - grab an unsuspecting co-worker or friend and conduct the entire usability test on them. This will help you feel more comfortable when you do the test on your first real test subject. Have your friend/co-worker complete all the paper work (waiver, entrance and exit questions). As you read the task based questions aloud, remember not to lead them to the answer. Encourage them to talk out loud (with some folks you will need to encourage them to do this on every question).

Remind them over and over, that you are not testing them, but you are testing the software. If they have problems completing the tasks, their complaints and frustrations (voiced aloud) will help you convince the developers that things need to be changed.

Tell them you want them to be brutally honest. You had nothing to do with the design of the system and want them to tell you exactly what they are thinking. If they think the site "stinks" or they are ready to pull their hair out, you want them to tell you!

As they give you feedback, both negative and positive, tell them "that is a good point" and write down what they say. Show approval and appreciation for their comments.

3) Supply Checklist - make sure you have everything you need to conduct each test. Your checklist might look something like this:
Waiver
Entrance Questions
Task Based Questions
Exit Questions
Computer with internet connection and all required plug-ins
URL of working web site
Token of appreciation - food or gift certificate
Pen(s)
I like to create a web page that has copies of all the usability testing documents online as well as a link to the site. Then, if I have forgotten anything, I can grab a copy online and print it out wherever I am.

Testing Methodology

Okay, now we are ready to conduct the test.

1) Welcome: Welcome the test subject and thank them for coming. Make them feel at ease. Tell a joke, or talk about the weather. Ask if they have ever been in a usability study before. Assure them that it is fun and easy.

2) Agenda: Outline the main things you will be doing during the usability study. For example, you could say:

First, I will have you sign a waiver that indicates your willingness to participate in this usability study and let's you know that I'll be taking notes of your comments, but will keep all your personal information private.

Second, I will ask you some basic demographic questions.

Third, I will ask you to complete X number of tasks on the new web site. Keep in mind that we aren't testing you, but we are testing the web site. Any problems or frustrations you encounter will help us see where the design needs to be changed.

Fourth, I will ask you for your general feedback on the site. We want to know your opinion.

This process should take about an hour, so let's get started. (Note: It is important to end the test on time. You need to be respectful of the persons time, especially if you want to get honest answers.)

3) Waiver/Entrance Questions: Have the person complete the waiver. Answer any questions they have. Have the person complete the demographic/entrance questions. You can either have them complete the form on paper, or ask them the questions out loud and fill it in for them. Whichever works best for you.

4) Task Based Questions: Have a computer ready with the browser open and sitting on the starting page of your web site. As the test facilitator, you will instruct and observe the subjects performing fairly simple, common tasks. You will verbally lead them through the series of tasks/questions, encouraging them to think out loud and respond to what they are looking at. You will ask questions about their thought processes and their decisions as they work, without being intrusive or leading. You will also take in-depth notes directly on the Task Based Questionnaire.

Things "To Do" and Things "Not to Do" while facilitating the test:

Things to Do Things Not To Do
Listen carefully Fail to Listen
Encourage Criticize
Be Neutral Be Defensive
Speak English Speak Geek
Answer questions with "What would you do?" or "What do you think?" Lead user to the answer
Be Patient Be Impatience, Rush
As you ask each question aloud, try to use the exact words on the test. Do not lead the user to the answer. Do not help the user answer the question. Remind the user we are testing that software, not them. Encourage them to think out loud. (“what words are going thru your mind?”, “what are you looking for?”)

Make note of the click stream (the path the users follows to complete the task). Note any of the users comments and suggestions for making the task easier. Finally, indicate if the person was able to successful complete the task, as well as your opinion of their satisfaction or frustration level.

Remember, someone will have to transcribe these notes. So try to write legible and capture what you are observing.

5) Exit Questions:

When users are finished going through the set of task-based questions, hand them the exit questions and ask them to answer them. Then give them a few minutes to talk informally about their usability experience with you. Ask them what they liked/disliked about the site most, if they have suggestions for improvements, etc. If they’ve done or mentioned anything during the test itself that you want to ask further questions about, now is the time to ask. In my experience, users are only too happy to talk with you when the test is over!

-----------------------------------------------------------------------------------
The Final Report - Usability Findings & Recommendations

Now it is time to compile your usability data, analyze it and write your recommendations. I recommend entering all of your entrance, exit and task information into an excel spreadsheet. If you don't have MS Word and Excel, you can download free viewers for Microsoft Excel and Microsoft Word.

Sample spreadsheet of raw usability data (microsoft excel)

Sample Usability Recommendations Report (microsoft word)

Look for trends in comments and task completion. Document the impact the data shows on effectiveness, efficiency, time on tasks, errors and satisfaction. Group your recommendations in severity/priority order.

1st Priority - must be fixed, brick wall
2nd Priority - would be good to fix, but can wait
3rd Priority - okay as is, could be improved
Don't just critique and point out what is wrong. Suggest a solution or remedy for each problem. Your suggestions could be a change in the design or the content.

If you sense that the application owners/developers will be resistant to suggestions, consider sharing the usability recommendations with them individually and in draft form, then when it comes to the formal meeting where you review the recommendations, there are no surprises.

Another key to helping developers/designers truly understand the usability issues is to actually let them observe the usability testing process. This works best when the develop/designer is behind a one way mirror (so when they scream and cry and state that "any idiot would know what to do", your innocent test subject won't have to be intimidated by them)! Since most offices don't have usability testing rooms with one way mirrors, other methods for letting them see include:

audio recording the sessions and letting them listen
video taping the sessions and letting them watch
live webcast the sessions and let them watch from another room
make sure your test facilitator/notetakers are people that your developers/designers will always believe
in the rarest of cases, allow your develop/designer in the room while testing (but you may have to tie them in a chair and put duct tape over their mouth). I do not recommend this option.
In my experience, I've yet to make use of the audio recording or video taping we've done. I now just pick well trusted facilitators and note takers and pre-review my recommendations with the designers/developers before making formal recommendations.


Additional Usability Resources

Recommended Reading:

Don't Make Me Think by Steve Krug
Good introduction to practical low-cost usability testing. Filled with excellent examples.

Designing Web Usability : The Practice of Simplicity by Jakob Nielsen
Good introduction to usability. More theoretical than "how to".

The Design of Everyday Things by Donald Norman
One of the foundational text of usability. Excellent theory.

Usability Expertise on Campus

TeamWeb - The ITS Web Technology Team ITS offers consulting on many web topics including usability. Request help from TeamWeb.

Define Internals - Jefferson Stewart, Bennett Donovan, Lewis Phillips, Tracy Caillouet, Carey Christian & Glenda Sims conducted an extensive Web Usability Survey of Define Internals. The report was released in June 2005. This extremely comprehensive report is a wonderful text book from which to draw ideas.

Online Usability Resources

Introduction to Usability Testing by Carolee Mitchel
MIT Usability Guidelines
Usability Professionals' Association
Usability Testing Materials
Web Design & Usability Guidelines


http://www.utexas.edu/learn/usability/resources.html