Front Page

Team Fresh City


This is the website for Team Fresh City, part of the HFID class in Fall 2009.


Needs Analysis

Information Gathering
User Goals
Concept Model

Inspirational Designs

Inspirational Designs
Google Docs

Design Development

User Script
Design Process
Usability Report

Design Refinement

Cognitive Walkthrough
Design Refinement Presentation
Design Refinement

Usability Testing

Usability Testing

Final Refinement

Final Prototype Refinement
Generation 2 Prototype
Usability Testing
Final Prototype (Generation 3)
Final Presentation
Final Report

Division of Labor

Division of Labor


01 Dec - 31 Dec 2009



Powered by Pivot - 1.40.7: 'Dreadwind' 
XML: RSS Feed 
XML: Atom Feed 

« Design Development: D… | Home | Cognitive Walkthrough… »

Design Devlopment: Usability Report

19 10 09 - 23:27

The paper prototyping process is a value exercise in interface design, and through this practice we hoped to:

  1. Learn about the process of creating a paperprotoype and working through interviews
  2. Identify challenge areas of our interface
  3. Confirm that our design was adressing the identified needs
  4. Ensure that we fully developed the concept, and that desired functionality was not missing
  5. Revise and improve the concept

Key findings and results:

  1. Paper prototyping is definitley a valuable tool; we identified many problems areas
  2. Shifted focus of Tradmin away from editing and moved towards logging communication
  3. Revision of prototyping between interviews was an effective way to narrow focus and use scarce interview time efficiently  


As we discussed in the design process document, the space of improving the experience of academic paper collaboration is vast and well beyond the scope of this class.  To bring the problem down to a more manageable scale, we decided to focus specifically on what we believe is one of the most important aspects of this process: parsing edits and comments, and actually revising the paper.  One common trait that we’ve found in our users is that a deluge of comments and suggestions leads many of them to be ignored.  This is not just a phenomenon in paper writing either – studies have documented the fact that many people, when faced with a mountain of data to sort through, instead of prioritizing, will choose to sort through none of it.  Thus, the goal of this product (and prototype) is to ease the experience of the user who is collecting and taking into account the comments and edits of their collaborators. 

In this portion of the design process, we developed and tested a paper prototype, of our concept, Tradmin.  Paper prototyping is a variation of co-design, in which we instruct the user to perform actions to achieve goals with our tool.  Through this experimentation we can observe and learn about the interaction with our concept.  The goal of paperprototyping is to ellicit user feedback and revise the tool accordingly.  The goal of Tradmin is facilitate the communication of collaborators, and improve the handling of paper comments.    



We followed the traditional method of paper prototyping, in which we created a protoype with basic office supplies, and generated a corresponding user script.  To conduct this experiment, we went back to the users we originally interviewed.  We chose individuals who represented the demographics of our personas (age, career, values, goals), and those whom were especially interested in helping us with our project.  We ended up interviewing three Olin faculty memebers due to scheduling constraints.  We realize that just interviewing Olin professors greatly limits the scope of user experience we can draw on, since we could not interview a nontechnical professor.  However, since our interviewees supported the breadth of our personas, they served as a strong foundation of feedback and experience.  


To focus our interface, we sketched a flow chart of the collaborative paper writing process:

[Process Flow Chart.  This flow chart was used to break down the process of editing/revising/commenting on a paper, in order to address all of a potential tool's functionality.]

We decided that the area we most wanted to focus on (and the area in which we could get the most leverage) was in viewing and organizing comments, indicated by the red box.  The other tasks indicated in the flow chart, while important to our users, we decided should be implemented as a secondary priority.  Note, that is not to say that our users find those other functionalities to be less important, but from a design perspective, we can do the most good by first focusing on the outlined area and then branching out. The frames shown in the flow chart were developed into paper prototype pieces using office supplies to reflect interface pieces.

Our interviewing process was centered around revisioning. We began this phase by developing a user script which informed our decisions for frames/boxes to include in the first prototype. We conducted our first user interview with a person similar to our "Frank" persona. After receiving user one's feedback we reflected and developed a more refined second prototype, which we then took out to user 2, someone similar to "Stephen". We reflected and revised a final time and conducted our third user interview, someone representing a mix of "Stephen" and "Janice" personalities. By going through an iterative process we were able to capture more information about the usability of the final prototype and develop functionality that meet our user's needs.

Task Scenario

Users were asked to perform certain actions within a scenario during the paper prototyping interview.  Walking the users through our interface would provide us with feedback regarding the usability and functionality of our tool concept.  A specific scenario was created to guide our users through the interactions, in order to provide us with the most effective feedback.  The overall goals of the tasks were to see if the interface was intuitive and easy to learn for a new user, to learn about the value and functionality of our features, and to ensure that we did not leave out any of the prototype's functionality.  The following is a brief overview and justification for each task:

  • Send paper to collaborators:  Adding collaborators to a project is key in using this tool.  This task tells us if this function is intuitive and low energy enough for Tradmin to be used.  
  • View the comments of two specific collaborators:  This highlights the basic functionality of our tool.  We could observe the initial expectations of the user.
  • If you only had 30 minutes, what would you do?:  This prompt was supporting the values of importance, transparency and priority.  We felt that sorting and viewing higher priority comments would be useful when strapped for time.  We could also see what kind of editing information was most important for each user.   
  • Find only important comments:  We wanted to user to explore the sort function.  We could find out what sorting filters were most beneficial and helpful.
  • Revise paper, and dismiss comment:  Interacting with comments/edits is a large portion of the interface.  We wanted to learn what instincts users have learned from current commenting/editing software, and if our method was usable based off of these experiences.
  • Learn about your collaborators general overall comments:  This would allow the user to use the summary/general comment features.  We wanted to learn if connecting these summary comments to the paper in a more visual way was helpful.
  • Ask a collaborator for clarification on a comment:  Tradmin allows the user to send messages and communicate through the tool.  Since this is a new feature for a word processor type of tool, it was important to learn if the user would use the built in tool, or default to their regular email client.  

Overall, our task scenario was successful during the paper prototype interviews.  It was changed slightly after each revision of the paper prototype, in response to the changes to the prototype.

Prototype 1

The prototype we brought to user one was our first iteration. This prototype focused on different ways to sort and filter comments, as well how to notify, add and give permissions to reviewers. This prototype was very complex, with dozens of small working parts for menus, submenus and pop-up windows. Most tasks were performed through a function bar on the top of the screen, which opened smaller dialog boxes with options. To sort comments, the user could press buttons on the bottom of the screen for comment priority. Feedback could also be sorted by selecting or unselecting a reviewer’s name.

[Paper Prototype 1.  This is an image of our first paper prototype.]

Prototype 1 – Results

User one immediately pointed out many faults with our design. He could not find how to add collaborators to his project. When confronted with the permissions window, he wasn’t sure what any of the settings meant. He commented that the interface was too dense – simple actions required multiple ‘clicks,’ making it confusing and frustrating to him. He also explained how our user interface was unrealistic for a paper he would review. Our interface had functions to sort through hundreds of comments easily, but user one said he usually only receives about 10 comments. Our very granular sort options didn’t apply to most papers, so he found them very confusing.

Prototype 1 – Discussion

With user one, we found two major problems in our prototype: first, the interface was far too dense, and second, our assumptions about pain points in paper writing were exagerated. For example, our comments tabs displayed over 100 comments, where our user said this quanity would most likely be less than 15. Perhaps the most important comment from user one was, “I know what you want me to click, but I wouldn’t ever actually do that, I would just read through the paper.” We realized that our interface was focused on the wrong problems, making our interface complicated and overly difficult to use. Specifically, our users were considerably less interested in being able to organize their comments than we originally anticipated.  We decided to cancel our interviews later that day, giving us time to rework our prototype into something more useful to the user.

Prototype 2

After feedback from prototype one, we decided to change some items and implement prototype two. The goal of this new prototype is to give users the ability to work with multiple individuals’ comments at once and focus on what tasks they need to complete. We focused more on the ability to view multiple individuals’ comments at once. We took away some of the hidden features of tabs and placed them onto a single screen. We added a buddy list for easy addition/removal of collaborators on a paper, and we removed the “summary” feature from prototype one, because it was unclear to user one what that meant. In its place, we introduced an “overall comment” box for collaborators to right summary comments on the paper. We introduced a “tasks” pane to give action to items usually described in comments, as a way to provide a list of things to be completed.

[Paper Prototype 2.  An image depicting the first revision of our paper prototype.]

Prototype 2 - Results

User two had never conducted a paper prototype interview before. It took some time to get him warmed up to the idea, but ultimately, he played along. He added collaborators with ease using our buddy list feature and could work with individual comments without difficulty. It was easy for him to respond to a comment if he needed clarification. He also glanced at the “overall comments” to get the gist of what people were saying about the paper, though he also said glancing at the comments would not take very long. He did not interact with the “Tasks” pane at all because he figured everything to address was described in the comments. User two did appreciate being able to email people to remind them about giving feedback and to respond to someone’s comment for clarification.  Overall user two responded very well to the usability of the prototype, but was not completely impressed by its functionality. 

Prototype 2 - Discussion

User one interacted with the buddy list and comments without issue. He did not interact with “Tasks” because he believed that the comments his collaborators were writing were the action items he needed to respond to. Like user one, he mentioned that he trusted his collaborators enough to edit things they disliked and fix wording, and that he’d only receive 10-15 comments on a given paper to address issues. Ultimately, our user groups are low on time and they just need to “get her done” as quickly as possible. User two also felt like the product was very similar to Word track changes because of the focus on handling edits and comments.  We have decided to focus on how users deal with comments and keep track of them between versions of a project as we move forward.

Prototype 3 - Final Version

The third revision of the paper prototype took some emphasis off of the editing-and-commenting space of paper collaboration, and focused much more on the values of communication and real-time communication.  This third concept is centered on an application that lives within your current word processor, but manages a paper that lives in the cloud.  Through this application all collaborators on a particular paper share a common document to edit, comment, and communicate about.  The interface is broken into three major components: the buddy list, the paper comments, and overall comment log.  These three main features allow for transparent collaboration, as well as a way to manage the revisions and cycling of an academic paper.

When the application is open, the buddy list manages all of the paper collaborators.  The buddy list is a collection of tabs, color coded per person.  Within this buddy list, the paper collaborator can add members, and message collaborators.  This message feature is synced with the user’s mail client, and is open to any type of message including a call for edits, reminders, or questions.  The user uses this buddy list to access collaborators’ comments. 

The comment section of the application is a variation of the traditional comment method.  The paper is live, and what is seen by the user is the current state of the paper.  All users are free to make any paper edits, as they see fit.  These edits are not tracked or logged.  Comments are handled with large comment boxes and supporting pointers.  Any collaborator can make a comment or comment on a preexisting comment.  These comments are coded with color, and tracked.  Comments can be used to point out high level thoughts or changes, as well as communicate with collaborators.  These comments are left connected with the paper until deleted.  

The last part of the application is the overall comment log.  This feature is included to get rid of the randomly placed overall comment block at the beginning or end of a paper, or the body of and email that describes the paper on a higher level.  This feature connects comments with the paper, and keeps them together at all times.  This log also tracks changes made by collaborators. When opening the application after a few days it is important for the user to know what has changed without scrolling through a dense paper with comments.  Comments can be posted into this box, along with links to the paper.  Clicking on a link would automatically scroll the paper to the correct location. 

This prototype is a solution to the long thread of email communication, crowded edits, and separated papers and comments. 

[Paper Prototype 3.  This was the third, and final revision of the paper prototype.]

Prototype 3 – Results

This third prototype was tested with one of our users that falls between the Janice and Stephen personas.  Overall, the user was quite happy with the added functionality of the program.  The user felt that it was simple and based on the conventions that most computer users are used to.  The overall comments, large comment boxes, and messaging features were something that they would definitely use.  This user had a hard time understanding the form language of the interface and was not always sure what to click or press for the desired function.  The user also felt that the prototype assumed too much trust.  This user would prefer to assign permissions to different collaborators in order to monitor some of the edits and changes.  It would have also been useful if some collaborators were blocked from seeing all comments, since some could be data sensitive.  The user also felt that an automatic “nagging” feature, though impersonal could help foster relationships.  It gives the administrator or first author the default of blaming the program.  Lastly the user expressed a want for a “download” comment feature.  It would be helpful to review the comments separately, not necessarily connected with the paper.  Reading through the comments provides an understanding of the reviewer and overall implicit paper feelings. 


A lot of valuable things were learned from this prototyping session.  First, on a positive note, the design seems to be moving in a beneficial, valuable direction.  It took many iterations to work out the worthless features and buttons, but it is condensing into a valuable tool.  However, it is not completely there yet.  There is still some missing or ambiguous functionality.  There aren’t completely defined actions for some features (like adding comments versus editing), or the comment log (small details are still missing).  We also learned about small additional features that could add a lot of value to the program, such as permissions or a printable comment log. 

There are definitely a few things we will look to change from this paper prototyping session.  These smaller details, such as commenting and leaving messages need to be worked out.  The language of the interface also needs to be made clearer.  Users must be able to work the software, such as knowing what button to press, with little trouble.  Lastly, some new features could be added to increase value.  However, after talking with one person, it could only be a personal preference.  The team must decide whether or not these features would be beneficial to our users and personas.  This prototype would benefit from more detailed design and interaction work. 

Though a lot was learned from this third paper prototype interaction, there are few things we did not gain insight on.  Because not all of the interface details were designed, we were only able to get very high level comments and interactions with the prototype.  It would have been good to learn more about the series of clicks and actions needed to complete a task.  Though we learned some about these things, there is still some unknown space.  We also did not learn about the exact reactions and interactions with real comments.  Editing a paper is hard to simulate, since it is heavily dependent on the topic, collaborators, and point in the process.  For the paper prototype, the comments were made up, and some were not even worked out.  Because of this, users could not interact with the specific comments, and we did not gain insight about specific paper editing and collaboration interactions, as connected with real comments, collaborators, and papers.

 The final prototype is a good representation of our concept but some simple changes based on results of the last interview will polish and complete the concept. The multiple revisions process proved to be helpful in isolating issues with functionality and usability quickly. Unfortunately, we were unable to interview multiple people between versions. However, by meeting people across the breadth of our personas, we were able to balance functions they desired (highlighted by when user three requested we have the "tracking changes" capability that user two did not care for). We are eager to refine and move forward into interface development. 

Appendix - User Script

Thank you for taking some time out of your busy schedule. As you know, we took our interviews and came up with a product idea involved with academic paper collaboration. We've generated a first-pass paper mock-up of our interface, and with your help, hope to gain your insight into our product, which will be valuable for generating a thoughtful design. As a first draft,this prototype is rough, and any confusion that may occur is due to our current design. We want to use your experience to improve this product. this test will take about 45 minutes.

This is the paper prototype we've developed. Instead of a keyboard and mouse, we'll be using finger interaction, and one of us will be playing "the Computer". The computer, <INSERT NAME>, has no speech recognition or artificial intelligence. However, you can direct questions to <INSERT OTHER TEAM MEMBER'S NAME>, if you are stuck. Please think outloud about what steps you're taking so we better understand the interaction. Do you have any questions or concerns?

Tradmin is an add-on tool for text editors (Word, Latex, etc.) that allow you to solicit and review feedback from peers. You have recently finished a draft of your advanced classical mechanics paper. Now you would like to send it to your four collaborators, Dan, Tom, Rachel, and Donna, using Tradmin. Your collaborators will have a chance to mark up and comment the paper and send it back to you for further revision.

Please send your paper to all of your collaborators for feedback. [open text editor, tradmin, click send paper, and send to selected collaborators]

3 Days later you received 2 email notifications that you received feedback from Dan and Tom. You open up Tradmin to view collaborator feedback. *expect the user to open the paper in whatever editor they choose*

Great. You've received feedback from a couple of your colleagues, Tom and Dan. What do you do next? [View Tom's comments, Dan's comments, maybe simultaneous exploration]

You only have 30 minutes to work on your paper before class. What do you work on? [use the toggle boxes to look at select comments], this is early on in the rewrite process, and you only want to view the really big and important edits you’ll have to make.  How do you find those? [select only high priority edits]

You want to get the overall impression that your colleagues had on the paper. How would you look at it? [user goes to summaries]

You need more clarification on a comment you received from a colleague. What do you do? [reply to comment]

This is a lot of feedback. How would you go about sorting it?

Thank you for your time!