Usability Testing
06 12 09 - 14:22 In order to test the availability and utility of the features in Tradmin, a usability test was designed and performed. As a result of this testing, some changes to the interface will be made as the final refinement.Usability Test Design
Purpose:
Tradmin introduces a number of communication features for the editing-commenting-versioning process of academic papers. The team wants to investigate the utility of these features in the interface.
Hypothesis:
We believe that the commenting, discussion, and private messaging options within Tradmin are all valuable features for the user. Each feature will present a different communication option, however its degree of value will be influenced by the particular user’s existing habits.
Description:
We plan to create a usability test based off upon social science coding. Each user will be asked to “free-edit” a specific portion of the paper, using whatever Tradmin features he/she needs or wants. This activity would entail receiving a section of a paper, ideally one there are already engaged with, and revising it based on their personal style and the capabilities provided by Tradmin. This could provide insight into the types of edits a specific type of user makes, and if there’s a bias towards a specific feature. Following this interaction, each user will be asked to complete an open-ended questionnaire, to capture their thoughts, feelings, and attitudes towards Tradmin as a whole, along with the individual features. These individual responses will be coded to elicit trends and reactions to the tool.
Independent Variables:
• Type of user- technology preferences. For studying the distribution of communication options, it is important to look at users with different initial preferences, such as heavy e-mail user vs. heavy track changes user.
• Text to edit- We will have users examine different content, so they can have more meaningful and compelling interaction with Tradmin.
Dependent Response Variables We Will Observe:
• Coding- The open ended responses will be used to define basic coding parameters in order to quantify the feelings/attitudes towards Tradmin features.
What we anticipate to find:
We expect that there is be a generally positive attitude towards the features of Tradmin. With the software “primed” users will have a general understanding of each feature, and will hopefully see their values.
How many subjects and iterations?
To initially generate the coding variables, we would interview three users. With this foundation we would ideally have 18 users from 3 categories: 6 heavy e-mail users, 6 “live communication” users (e.g. chat, phone), and 6 word-processor tool users (e.g. those who leave comments in Latex or use Track Changes). Then we could have 3 different free-editing text paragraph situations and 2 users per category to examine each situation, randomly assigned. In all, each category would be represented by 6 users, and each situation would be represented by 6 users (2 from each category).
Usability Test
Introduction Speech
Thank you for participating in Team FreshCity’s Usability Study. Your insight is very valuable to us. As you work through the study, please think out loud, for it provides us with more context and information about your interaction with the tool. Remember, your results will be kept anonymous. You may take a break or end the study at any time, and do not hesitate to ask any questions.
Here is a new paper collaboration tool, Tradmin, intended to facilitate the commenting/editing/revision process of academic paper collaboration. Tradmin is a tool that lives inside your common word processor and communicates with other collaborators in the cloud, in the internet. For instance, a local copy of the file is located on your hard drive and mutually shared with your collaborators. In this scenario, you’ve opened your most recent paper [insert topic here], and you have 15 minutes to edit it. This paper has been shared with your collaborators for a few days, so there has already been some activity.
Edit and revise the paper [or more specific section] for 15 minutes. Remember to think out loud.
[User works with paper for 15 minutes. Administrator records any thoughts or comments from the user.]
Now that you’ve interacted with Tradmin, we would like you to answer a few questions about your experience.
User Survey
Date:
Participant:
What is the first thing you do when revising a paper, and why?
What feature did you find most valuable, why?
What do you think the use of the “Comments” feature is?
Describe your thoughts and experience with the “Comments” feature.
What do you think the use of the “Discussion” feature is?
Describe your thoughts and experience with the “Discussion” feature.
What do you think the use of the “Private Messages” feature is?
Describe your thoughts and experience with the “Private Messages” feature.
Rate your overall interaction with the “Comments” feature, on a scale of 1-5 (with 5 being most optimal).
1 2 3 4 5
Rate your overall interaction with the “Discussion” feature.
1 2 3 4 5
Rate your overall interaction with the “Private Messages” feature.
1 2 3 4 5
Screener
This screener was developed in order to gain insight about each user's background in research and technical prefrences. It is administered at the start of each user study.
Usability Test Screener
Thank you for participating in Team Fresh City’s usability test. We greatly appreciate your insight. This purpose of this initial screener is to capture your background, studies, technical preferences, and paper collaboration habits.
1. Name:
2. Gender: M F
3. Years in research: 0-5 6-10 11-15 16-20 21-25 26-30 31+
4. Field of Study:
5. Number of papers written in past year:
6. Word processor preference:
Microsoft Word
Gedit
Mac Pages
Google Docs
WinEdt
Other LaTeX processor
Other ______________________
7. Communication methods for paper collaboration:
Word processor comments
Phone
Text messaging
Video conferencing
Wiki
Instant messaging
In-person meetings
Skype
Other________________________
8. Everyday communication methods, and frequency of uses:
Email [Heavy Medium Light]
Phone [Heavy Medium Light]
Text Messaging [Heavy Medium Light]
Video conferencing [Heavy Medium Light]
Wiki [Heavy Medium Light]
Instant messaging [Heavy Medium Light]
In-person interaction [Heavy Medium Light]
Twitter [Heavy Medium Light]
Facebook [Heavy Medium Light]
LinkedIn [Heavy Medium Light]
Skype [Heavy Medium Light]
Other___________________ [Heavy Medium Light]
Usability Report
Introduction
Tradmin introduces a number of communication features for the editing-commenting-versioning process of academic papers. In order to gain insight on the value, as defined as availability and utility of these features, Fresh City ran a series of usability tests. Although some type of long term user testing would be most valuable in answering this question, it was not feasible for the time frame of this course. The current usability test is based on a social science method of coding. Users are asked to express their feelings about the interface in writing, and researches analyze the responses for trends in word-usage and moods, extracting quantitative data. In this study users were asked to use Tradmin for 10-15 minutes, in a prepared document, and fill out a short survey in response to their experience.
The independent variables in this experimentation were the types of users. It was important to interview users with varied communication habits, such as email users, phone users, or track changes user. Each user was intended to edit different content, content that pertained to their field, in order to have a more meaningful interaction. However, due to time constraints each user was given the same content in this first round of experimentation.
For more detailed information about the usability test, please see Usability Test Design (above).
Participants
Three users were interviewed during this process, chosen to represent different fields and technology preferences:
• User 1: An experienced technical professor who is dependent on email but will use some social networking to collaborate.
• User 2: A non-technical professor who has just started their academic career. User 2 prefers to collaborate via email or through in-person interactions.
• User 3: A technical professor focused on pedagogical research. User 3 uses the most technologically advanced communication with video conferencing, PowerPoint, and desktop sharing.
Reponses
In this usability test, the independent variables are the written responses to the survey questions. In a full usability test, these initial responses would be coded and be used as a key for future tests. Due to the type of data, it is not as easy to classify at this time with specific quantitative values. In this report, each feature of Tradmin will be discussed, along with a list of positively and negatively associated words/phrases from the user responses.
Buddy List
The buddy list was User 3’s favorite feature. Overall, it was a positive experience for users, however some found the on/off toggling confusing at first. The on/off visual could have been clearer.
| Positive |
Negative |
| focus | confusing |
| map to paper |
unclear |
| awesome | took a second |
Comments
Besides the expected computer-coding bugs the team expected to get feedback on, the response to the comment feature was very positive. It was User 2’s favorite feature, and was thought to be an improvement over current systems. There was some confusion with initiating comments, but once explained, the process was very intuitive and liked.
| Positive |
Negative |
| makes sense |
confusing |
| really like |
unclear |
| prefer | took a second |
| easy | can't |
| intuitive |
hard at first |
| very fluid |
|
| clustered | |
| connected |
Discussion
The discussion feature was User 1’s favorite feature, and was one the whole appreciated by all. It had good usability, though may not have the greatest utility for all. One user felt that email would be preferred over this feature, and another thought about using it as a task list.
| Positive |
Negative |
| use it |
might not |
| higher lever |
should |
| useful | |
| easy | |
| fluid |
Private Messages
None of the users were particularly excited about the private messaging feature. There was some thought that any discussion or talk about the paper should be shared by all. One user would not trust the feature to be private. There is some cognitive irony, since this private space is located in a shared space, it seems unsafe. User 1 said they would use email or make a phone call instead of using this feature.
| Positive | Negative |
| easy | would'n't |
| with all |
|
| not needed |
|
| don't know |
|
| be careful |
Interpretation
Though only three users were interviewed during this stage, there are some clear feelings from the responses. The Buddy List is an appreciated and easy to use feature in Tradmin. There were some usability problems, so the interaction aspects of this feature could be revised. Making the buddy list function clearer could be a benefit for the user. There were many positive feelings and comments about the comments feature, programming bugs aside. This feature has good utility and availability, and the users feel a need and want to use it. That being said, there are a few changes, such as incremental deletion and clearer usability that could improve its functionality. The feelings about the discussion feature were mixed, though more positive than negative. This feature is worth keeping in Tradmin, but may not have the most optimal formatting. Changes to the discussion’s utility could be beneficial, tough not completely necessary. The private messages received the most negative feedback, and most users were in accord that it may not be necessary. This feature should probably be minimized in the tool, or given different utility. In comparison to the other features, the private messages feature is definitely least useful and needed.
Intended Changes
In response to this testing, team Fresh City intends to make changes to Tradmin in order to increase its utility and availability for users. These changes range from small tweaks, to larger function based changes, and are as follows:
Clarify on-off toggle buttons in Buddy List. Currently, it is hard to tell whether the buttons are on or off, this needs to be made more apparent. Second, the functionality of the buttons is cognitively backwards. The buttons should be “on” or pressed when the comments are visible, and “off” or unpressed when comments are hidden.
Turn off collaborator menus when user clicks off of menus. The buttons on these menus need to physically clicked to hide the drop down menus. The menus should just disappear when the menus are clicked off of.
Make comment generation more obvious. For a first time user, the act of generating a comment is unclear or unknown. This functionality needs to be more obvious and easy to use. This will be solved by adding a faint set of instructions, “Click here to add comment.”
Focus on comment entries for typing. When a comment box appears, the type field should be activated and ready for typing.
Shrink the size of the Private Messages. The Private Message is going to be made smaller. The Discussion panel seems to be much valuable for users, in comparison to the Private Messages. Because of this difference in value, the two features should not use the same real estate.
Use check boxes for Private Messaging recipients. Currently, the Private Messages uses a fill in, auto-complete field for message recipients. This will be changed to a drop down check box list, to lower the amount of stored knowledge for users, and also the make it feel more secure.
Appendix
Originally, the team proposed this usability test: Original Usability Test.docx. While this was proposed as a short-term test to fit the time-scale of the class, we realized that in order to find meaningful results for this experiment we would need to run a long-term test. In a long term test, the users would have Tradmin to work with on their own papers to adapt to the features of the program and find what is useful and useless for them.