Front Page

Team Fresh City

About

This is the website for Team Fresh City, part of the HFID class in Fall 2009.

Links

Needs Analysis

Introduction
Information Gathering
Process
User Goals
Lexicon
Concept Model
Personas

Inspirational Designs

Inspirational Designs
SVN
Google Docs
OneNote

Design Development

User Script
Design Process
Usability Report

Design Refinement

Cognitive Walkthrough
Prototype
Design Refinement Presentation
Design Refinement

Usability Testing

Usability Testing

Final Refinement

Final Prototype Refinement
Generation 2 Prototype
Usability Testing
Final Prototype (Generation 3)
Final Presentation
Final Report

Division of Labor

Division of Labor

Archives

01 Dec - 31 Dec 2009

Search!

Stuff

Powered by Pivot - 1.40.7: 'Dreadwind' 
XML: RSS Feed 
XML: Atom Feed 

« Design Refinement | Home | Final Prototype Refin… »

Usability Testing

06 12 09 - 14:22 In order to test the availability and utility of the features in Tradmin, a usability test was designed and performed.  As a result of this testing, some changes to the interface will be made as the final refinement.

Usability Test Design

Purpose:

Tradmin introduces a number of communication features for the editing-commenting-versioning process of academic papers. The team wants to investigate the utility of these features in the interface.

Hypothesis:

We believe that the commenting, discussion, and private messaging options within Tradmin are all valuable features for the user. Each feature will present a different communication option, however its degree of value will be influenced by the particular user’s existing habits.
 

Description:

We plan to create a usability test based off upon social science coding. Each user will be asked to “free-edit” a specific portion of the paper, using whatever Tradmin features he/she needs or wants.  This activity would entail receiving a section of a paper, ideally one there are already engaged with, and revising it based on their personal style and the capabilities provided by Tradmin.  This could provide insight into the types of edits a specific type of user makes, and if there’s a bias towards a specific feature.  Following this interaction, each user will be asked to complete an open-ended questionnaire, to capture their thoughts, feelings, and attitudes towards Tradmin as a whole, along with the individual features. These individual responses will be coded to elicit trends and reactions to the tool.

Independent Variables:

•    Type of user- technology preferences. For studying the distribution of communication options, it is important to look at users with different initial preferences, such as heavy e-mail user vs. heavy track changes user.
•    Text to edit- We will have users examine different content, so they can have more meaningful and compelling interaction with Tradmin.

Dependent Response Variables We Will Observe:

•    Coding- The open ended responses will be used to define basic coding parameters in order to quantify the feelings/attitudes towards Tradmin features.
 

What we anticipate to find:

We expect that there is be a generally positive attitude towards the features of Tradmin.  With the software “primed” users will have a general understanding of each feature, and will hopefully see their values.

How many subjects and iterations?

To initially generate the coding variables, we would interview three users.  With this foundation we would ideally have 18 users from 3 categories: 6 heavy e-mail users, 6 “live communication” users (e.g. chat, phone), and 6 word-processor tool users (e.g. those who leave comments in Latex or use Track Changes). Then we could have 3 different free-editing text paragraph situations and 2 users per category to examine each situation, randomly assigned. In all, each category would be represented by 6 users, and each situation would be represented by 6 users (2 from each category).

Usability Test

Introduction Speech

Thank you for participating in Team FreshCity’s Usability Study.  Your insight is very valuable to us.  As you work through the study, please think out loud, for it provides us with more context and information about your interaction with the tool.  Remember, your results will be kept anonymous.  You may take a break or end the study at any time, and do not hesitate to ask any questions.

Here is a new paper collaboration tool, Tradmin, intended to facilitate the commenting/editing/revision process of academic paper collaboration.  Tradmin is a tool that lives inside your common word processor and communicates with other collaborators in the cloud, in the internet.  For instance, a local copy of the file is located on your hard drive and mutually shared with your collaborators.  In this scenario, you’ve opened your most recent paper [insert topic here], and you have 15 minutes to edit it.  This paper has been shared with your collaborators for a few days, so there has already been some activity. 

Edit and revise the paper [or more specific section] for 15 minutes.  Remember to think out loud. 

[User works with paper for 15 minutes.  Administrator records any thoughts or comments from the user.]

Now that you’ve interacted with Tradmin, we would like you to answer a few questions about your experience.

User Survey

Date:
Participant:

What is the first thing you do when revising a paper, and why?


What feature did you find most valuable, why?


What do you think the use of the “Comments” feature is?


Describe your thoughts and experience with the “Comments” feature.


What do you think the use of the “Discussion” feature is?


Describe your thoughts and experience with the “Discussion” feature.


What do you think the use of the “Private Messages” feature is?


Describe your thoughts and experience with the “Private Messages” feature.


Rate your overall interaction with the “Comments” feature, on a scale of 1-5 (with 5 being most optimal).
1        2        3        4        5


Rate your overall interaction with the “Discussion” feature.
1        2        3        4        5


Rate your overall interaction with the “Private Messages” feature.
1        2        3        4        5

Screener

This screener was developed in order to gain insight about each user's background in research and technical prefrences.  It is administered at the start of each user study.

Usability Test Screener

Thank you for participating in Team Fresh City’s usability test.  We greatly appreciate your insight.  This purpose of this initial screener is to capture your background, studies, technical preferences, and paper collaboration habits. 


1.    Name:

2.    Gender:  M  F

3.    Years in research: 0-5    6-10   11-15    16-20    21-25    26-30    31+

4.    Field of Study:

5.    Number of papers written in past year:

6.    Word processor preference:

    Microsoft Word
    Gedit
    Mac Pages
    Google Docs
    WinEdt
    Other LaTeX processor
    Other ______________________


7.    Communication methods for paper collaboration:

    Word processor comments
    Email
    Phone
    Text messaging
    Video conferencing
    Wiki
    Instant messaging
    In-person meetings
    Skype
    Other________________________


8.    Everyday communication methods, and frequency of uses:

    Email                                     [Heavy     Medium     Light]
    Phone                                   [Heavy     Medium     Light]
    Text Messaging                      [Heavy     Medium     Light]
    Video conferencing                 [Heavy     Medium     Light]
    Wiki                                      [Heavy     Medium     Light]
    Instant messaging                   [Heavy     Medium     Light]
    In-person interaction               [Heavy     Medium     Light]
    Twitter                                 [Heavy     Medium     Light]
    Facebook                               [Heavy     Medium     Light]
    LinkedIn                                [Heavy     Medium     Light]
    Skype                                   [Heavy     Medium     Light]
    Other___________________  [Heavy     Medium     Light]

Usability Report

Introduction

Tradmin introduces a number of communication features for the editing-commenting-versioning process of academic papers.   In order to gain insight on the value, as defined as availability and utility of these features, Fresh City ran a series of usability tests.  Although some type of long term user testing would be most valuable in answering this question, it was not feasible for the time frame of this course.  The current usability test is based on a social science method of coding. Users are asked to express their feelings about the interface in writing, and researches analyze the responses for trends in word-usage and moods, extracting quantitative data.  In this study users were asked to use Tradmin for 10-15 minutes, in a prepared document, and fill out a short survey in response to their experience. 

The independent variables in this experimentation were the types of users.  It was important to interview users with varied communication habits, such as email users, phone users, or track changes user.  Each user was intended to edit different content, content that pertained to their field, in order to have a more meaningful interaction.  However, due to time constraints each user was given the same content in this first round of experimentation.

For more detailed information about the usability test, please see Usability Test Design (above).

Participants

Three users were interviewed during this process, chosen to represent different fields and technology preferences:

•    User 1: An experienced technical professor who is dependent on email but will use some social networking to collaborate.
•    User 2: A non-technical professor who has just started their academic career.  User 2 prefers to collaborate via email or through in-person interactions. 
•    User 3: A technical professor focused on pedagogical research.  User 3 uses the most technologically advanced communication with video conferencing, PowerPoint, and desktop sharing. 

Reponses

In this usability test, the independent variables are the written responses to the survey questions.  In a full usability test, these initial responses would be coded and be used as a key for future tests.  Due to the type of data, it is not as easy to classify at this time with specific quantitative values.   In this report, each feature of Tradmin will be discussed, along with a list of positively and negatively associated words/phrases from the user responses.

Buddy List

The buddy list was User 3’s favorite feature. Overall, it was a positive experience for users, however some found the on/off toggling confusing at first.  The on/off visual could have been clearer.

Positive
Negative
 focus  confusing
 map to paper
 unclear
 awesome  took a second

Comments

Besides the expected computer-coding bugs the team expected to get feedback on, the response to the comment feature was very positive.  It was User 2’s favorite feature, and was thought to be an improvement over current systems.  There was some confusion with initiating comments, but once explained, the process was very intuitive and liked.

Positive
Negative
 makes sense
 confusing
 really like
 unclear
 prefer  took a second
 easy  can't
 intuitive
 hard at first
 very fluid
 
 clustered  
 connected  

Discussion

The discussion feature was User 1’s favorite feature, and was one the whole appreciated by all.  It had good usability, though may not have the greatest utility for all.  One user felt that email would be preferred over this feature, and another thought about using it as a task list.

Positive
Negative
 use it
 might not
 higher lever
 should
 useful  
 easy  
 fluid  

Private Messages

None of the users were particularly excited about the private messaging feature.  There was some thought that any discussion or talk about the paper should be shared by all.  One user would not trust the feature to be private.  There is some cognitive irony, since this private space is located in a shared space, it seems unsafe.  User 1 said they would use email or make a phone call instead of using this feature.

 Positive Negative
 easy would'n't
  with all
  not needed
   don't know
   be careful

 

Interpretation

Though only three users were interviewed during this stage, there are some clear feelings from the responses.  The Buddy List is an appreciated and easy to use feature in Tradmin.  There were some usability problems, so the interaction aspects of this feature could be revised.  Making the buddy list function clearer could be a benefit for the user.  There were many positive feelings and comments about the comments feature, programming bugs aside.  This feature has good utility and availability, and the users feel a need and want to use it.  That being said, there are a few changes, such as incremental deletion and clearer usability that could improve its functionality.  The feelings about the discussion feature were mixed, though more positive than negative.  This feature is worth keeping in Tradmin, but may not have the most optimal formatting.  Changes to the discussion’s utility could be beneficial, tough not completely necessary.  The private messages received the most negative feedback, and most users were in accord that it may not be necessary.  This feature should probably be minimized in the tool, or given different utility.  In comparison to the other features, the private messages feature is definitely least useful and needed. 

Intended Changes

In response to this testing, team Fresh City intends to make changes to Tradmin in order to increase its utility and availability for users.  These changes range from small tweaks, to larger function based changes, and are as follows:

Clarify on-off toggle buttons in Buddy List.  Currently, it is hard to tell whether the buttons are on or off, this needs to be made more apparent.  Second, the functionality of the buttons is cognitively backwards.  The buttons should be “on” or pressed when the comments are visible, and “off” or unpressed when comments are hidden.

Turn off collaborator menus when user clicks off of menus.  The buttons on these menus need to physically clicked to hide the drop down menus.  The menus should just disappear when the menus are clicked off of. 

Make comment generation more obvious.  For a first time user, the act of generating a comment is unclear or unknown.  This functionality needs to be more obvious and easy to use.  This will be solved by adding a faint set of instructions, “Click here to add comment.”

Focus on comment entries for typing.  When a comment box appears, the type field should be activated and ready for typing.

Shrink the size of the Private Messages.  The Private Message is going to be made smaller.   The Discussion panel seems to be much valuable for users, in comparison to the Private Messages.  Because of this difference in value, the two features should not use the same real estate. 

Use check boxes for Private Messaging recipients.  Currently, the Private Messages uses a fill in, auto-complete field for message recipients.  This will be changed to a drop down check box list, to lower the amount of stored knowledge for users, and also the make it feel more secure.

Appendix

 Originally, the team proposed this usability test: Original Usability Test.docx. While this was proposed as a short-term test to fit the time-scale of the class, we realized that in order to find meaningful results for this experiment we would need to run a long-term test. In a long term test, the users would have Tradmin to work with on their own papers to adapt to the features of the program and find what is useful and useless for them.