Badlands

Streamlining the US Visa application process through interface design.
Olin College of Engineering

Usability Study

A/B Tests

Experiment One

Hypothesis:

A separate login page will make it simpler to login, and increase engagement in new sign-ups.

Factors:

Our main factor is the placement/location of the login page. Everything will remain constant except for the placement of the login dialogue. Option A places it on the main page, and Option B places it on a separate page.

Dependent Variable:

Time it takes for a user to login. Our hypothesis is supported if a user takes less time to log in with Option B.

Testing Logistics:

We want to test this with six users. Each user will only try one condition, so we will have three users test one condition and three users test the other condition.

Option A:
Option B:

Experiment Two

Hypothesis:

Having the login section at the top of the page, visually separate and lower on the visual hierarchy than the “Create an Account” section, will make it easier for the user to log in.

Factors:

Our main factor is the placement/location of the login panel. In option A, the login panel is of equal visual importance as the “Create an Account” panel. The login panel is on the right side of a split screen, and the “Create an Account” is on the left. In option B, “Create an Account” is higher on the visual hierarchy than the login panel. “Create an Account” is large and in the center of the screen while the login panel is a small header on the upper right hand corner.

Dependent Variable:

Time it takes for a user to log in. Our hypothesis is successful if the user takes less time to log in with Option B.

Testing Logistics:

We want to test this with six users. Each user will only try one condition, so we will have three users test one condition and three users test the other condition.

Option A:
Option B:

Experiment Three

Hypothesis:

Users will read the timeline description if it includes a visual, and is in a split screen layout, but skip over it if it a normal paragraph at the top of the page.

Factors:

The main factor is the placement/location of the “Welcome to Timeline” description. Option A has a split screen - the “Welcome to timeline” is on the left side while the “Create account” is on the right. This option includes a visual of what our app looks like. Option B has “Welcome to timeline” as a paragraph at the top of the page, with “Create an account” under it. There is no split screen or visual in option B.

Dependent Variable:

Time it takes to grasp and understand what our application is. We can measure how long it takes for a user to read the description (and if they read the description in the first place). We can also measure how many questions they ask us about the application. Our hypothesis is successful if the user reads the description more often in option A and has less question about what the application is in option A.

Testing Logistics:

We want to test this with six users. Each user will only try one condition, so we will have three users test one condition and three users test the other condition.

Option A:
Option B:

Testing Feedback

Round One

Reed (Prototype #1):

  • Clicked login button immediately, and expected to see a separate login screen on the next page. This supports keeping the username, password, Google, and Facebook buttons off of the first screen
  • Had a hard time finding the create account screen, because he expected to find it next to the username and password as a little link that said “create account”. Users expect to find a create account next to the username and password fill ins.
  • Didn’t read the description of the app, probably because the login button was at the top of the page (above the description).

Sharon (Prototype #2):

  • Sharon only saw a login button at the top, and a create account screen at the right of the split screen. This division made sense to her. Perhaps because there was no username and password next to the log-in, she understood to look for “Create Account” somewhere else.
  • Sharon wanted to see a preview of the application before she had to enter in her name and info. She skimmed the description of the app on the first page because the image caught her attention, but, a better time for her to see it would have been after she entered her “Coming from” and “Coming to” information, and went to the next page.

Tanya (Prototype #3):

  • Tanya had little problem finding the log in or create account panels. She saw them both as split screens on the initial page.
  • However, she did not read the description. The description image seems to attract more attention, and if it is just a paragraph at the top of the screen, it may be missed.
  • Tanya went through the normal account login process, only to decide to log in with Facebook or Google once she was on the next page. The little Facebook/Google icons don’t seem to make sense to the users

Khloe (Prototype #4):

  • She clicked through the date icons immediately, and was surprised when they didn’t fill in. This is more a product of our InVision app than the testing process. We should have had some sort of autofill appear.
  • She hovered over the username and password after she filled out all the create account info boxes. These seem to take away from the create account panel, and add confusion.
  • She was able to log in with no problem, and preferred the log in with Facebook option, She didn’t choose this on the first screen however, and clicked it on the next screen. This supports that the little Facebook/Google icons don’t make sense to the users, and that the login page should be a separate screen.
  • She read the description before she created an account.

Main Takeaway:

Having username and password inputs with the login on the top is confusing, and people don’t understand what the small Facebook and Google buttons are. Having login on a separate page with big buttons is better. For this reason, we will eliminate prototype /#1 in our second round of testing. We will focus our second round on which prototype makes it the easiest to quickly understand what the app is.

Round Two

Anya (Prototype #2):

Anya was able to tell us what the app was in about 15 seconds by reading the description on the login page. She was also able to quickly login and tell us even more about our app once she reached the Passport page. But, she didn’t seem to understand the “timeline” part of our app. This is perhaps the most confusing part of our description, and we should consider an image or diagram on the second on-boarding screen that better explains this.

Paul (Prototype #3):

Paul took a long time to figure out what our app was, and finished giving a confused description after about 50 seconds. He was confused about what a “timeline” was, and thought the app might be for students looking for a work visa. The description was the first thing he read on the page.

Michelle (Prototype #4):

Michelle was also able to tell us what the app was in about 15 seconds by reading the description on the login page, but she wasn’t sure what an F1 visa was. She read the title first, followed by the description underneath.

Main takeaways:

We should come up with a new title for our app, since F1 Visa Timeline seems to be confusing to people who aren’t familiar with the process. Additionally, crowded pages that provide too much information at once seem to make it more confusing for people to understand what the app is, so we should avoid this.

Other feedback:

We should divide our description of the app between two pages: basic information on the main page, and a diagram/example of the Timeline Application on the second page of the on-boarding process (In our case, this is after a user clicks “Create Account”). This second page could be tailored to the information that was just entered, on where the user is coming from, and going to, and what dates. In other words, it’s a preview of what would happen once they create the account.