Usability Study:
Colorado Unemployment Website
This study was conducted to evaluate the overall usability of the Colorado unemployment website. There was a heavy focus on finding out whether users are able to successfully get to where they need to on the website to accomplish their goals.
Designed and Conducted by Mckenna Brewer & Chance Wills
INTRODUCTION & GOALS
Applying for unemployment is a process that no one wants to go through, but over a million Americans currently receive unemployment benefits every month. Each state has its own unique process of applying for unemployment, and Colorado is no different. Any citizen of Colorado looking to apply for unemployment must use the dedicated unemployment website created by the Colorado Department of Labor and Employment. All issues surrounding unemployment must be solved through the use of this website, as there is no dedicated in-person unemployment office citizens can just walk into. The site takes users through the steps of applying for unemployment benefits, provides users with assistance managing their claims, and offers job placement resources. At first glance, the website seems pretty standard for a government run site; a little confusing in places but generally usable. However, the more you explore the site, the more visual and navigation errors begin to appear.
Specific problems include poor navigation, inconsistent visual design, and a variety of redundancies that lead to an overwhelming amount of information for users.
IMPORTANT CONTEXT
In the United States, the most common person applying for unemployment is:
A high school graduate who may or may not have some college experience or an associates degree
Was last employed in natural resources, construction, or maintenance occupations
Is between the ages of 25 to 54
Colorado has the slowest rate of claim processing and first payments in the entire country.
When considering the background and circumstances of the users of this site, the website's seemingly minor usability issues begin to seem a lot more severe.
GOALS
This usability test aimed to identify the severity of the site's problems as pointed out above and whether or not these problems inhibit usability. The specific goals of the study are listed below:
Assess overall effectiveness of the website in answering user questions
Assess how easily users are able to successfully accomplish their goals
Measure user satisfaction of the design
Identify specific problem areas of the design
1. What is the relationship between technological ability and a user's ability to overcome small usability issues?
2. How successfully do users actually begin the unemployment application process upon their first visit to the website?
3. How well (and easily) can users navigate through the website?
4. How easily can users find the answers to common questions about unemployment?
5. Are users satisfied with the overall experience?
RESEARCH QUESTIONS
Based on the above goals, this study aimed to answer the following research questions:
METHODOLOGY
SESSIONS
This usability study was conducted using a summative testing method in which data was collected by an evaluator and through participant questionnaires. The tests conducted were designed to adhere to a within-subjects experimental design. The testing sessions were designed to last approximately 30 minutes. There were six tests conducted in total, one pilot test and five official tests. The pilot test and one usability test were conducted in person, while the remaining four usability tests were conducted remotely over zoom.
Participants were asked to complete four tasks on the Colorado Unemployment website. Their performance was evaluated using these metrics:
Successful Task Completion
Each task had a prescribed success criteria.
Users were evaluated on a pass/fail basis.
Completion Time
Time was recorded from the start of the task to when the user stated they had completed it
Happy Path
Each task had a “happy path,” an idealized route to success.
User adherence to these happy paths was recorded
Error Rate
Each deviation from the happy path was counted
A deviation was measured as a click or interaction that did not follow the prescribed path
TASKS
Participants completed four tasks centered around key interactions that users might use on the site. Scenarios were created to help guide the participants through the tasks.
TASK 1:
Determine unemployment eligibility (using a fake identity)
TASK 2:
Navigate to and begin an application for unemployment
TASK 3:
Find information on how to resolve an identity verification hold on a claim
TASK 4:
Find out how to file an appeal
DATA COLLECTION
Both attitudinal and behavioral data was collected during the test. Metrics were collected by the test evaluator on an evaluator log sheet, and by pre, during, and post task questionnaires.
Attitudinal
Pre-Test
Participants were asked about their age, occupation, and comfort level with using technology.
Post-Task
After each task participants were asked to assess their performance and satisfaction.
Post-Test
After the testing was complete, participants were asked to rate their overall experience using the website and for any additional feedback on the website or the tests.
Behavioral
Task Success
Task Time
Number of errors
Site Navigation
User Satisfaction
PARTICIPANTS
In the test plan for this study, we initially identified the desire to test users from a range of age, technical ability, and employment experience. Unfortunately, we were unable to recruit such a broad sample due to researcher limitations. One of the researchers had to rush their recruitment and testing timeline in anticipation of Hurricane Debby and possible power/wifi outages. This led to them having to interview from a smaller personal network that would be available before the storm. Due to this adjustment, four out of five participants were within the age range of 20-24. We were able to recruit one participant to represent the 51-65 age range (it is worth noting that our pilot test participant also came from this range). This range biases our findings to primarily users in their early 20s.
RESULTS & FINDINGS
Below is the collected data and results recorded during usability testing. The data has been totalled and averaged.
It is important to note the data for task 2 for one of the participants has been thrown out due to test administrator error, thus the task 2 data is displayed as out of 4, instead of 5
LOG SHEET RESULTS
POST-TASK QUESTIONNAIRE RESULTS
POST-TEST QUESTIONNAIRE RESULTS
The results of this study revealed many issues plaguing the Colorado Unemployment site. Users had to spend a great deal of time navigating the site to find where to go, and when they did choose a direction, it was often the wrong one.
NAVIGATION & NAMING ISSUES
Of the issues plaguing the Colorado Unemployment site, navigation is the most disruptive. Users frequently struggled and became frustrated trying to find the pages and information they needed on the site.
Users struggled to find the happy path for 3 of the 4 tasks
Only 7 of the 19 individual tasks performed followed the happy path
Despite the availability of quick links users overwhelmingly favored the top nav bar, FAQs, and the virtual assistant
One participant (P3) failed every task due to repeatedly using the virtual assistant
Inconsistent language through interactions
(e.g Start a Claim -> Apply for Benefits)
Undefined/uncommon abbreviations/jargon used in critical interactions
UI is used in many critical navigations. Most of the users did not realize that it meant Unemployment Insurance causing them to ignore the interactions.
Users were frequently prompted to log-in to “MyUI+” throughout the process
“I feel like this one is hard [task 3] because I feel like in order to get this information I would need to log into the portal.” - P3
POOR INFORMATION ARCHITECTURE & VISUAL HIERARCHY
Many of the navigation issues participants faced can be attributed to the information architecture of the site. The site consistently fails to present information in a clear and coherent manner that users would expect to see on a site like this.
Quick links and other navigation shortcuts are not immediately visible upon opening the page
Users were faced with a large banner and no continuation to indicate that they could scroll down the page. Consequently, the shortcuts were rarely noticed.
Important interactions were overshadowed by less important sections
Users missed a crucial menu in task 4 due to it being grayed out and to the left of a bolded and central “Contact Us” section.
“It felt that it was the most relevant information to put on that screen… the fact that it wasn’t on the main part of the screen tripped me up a little bit.” - P5
Dense, difficult-to-skim text overwhelmed and confused users
“I think they could’ve dumbed it down a bit more, if that makes sense.” - P5
RECOMENDATIONS
1. Reduce the amount of page content
Improve overall information architecture/visual hierarchy to make skimming/reading easier
2. Top-level navigation needs to change
Explicitly name important sections, change UI naming convention to provide clarity, and create a dedicated section for all types of claim holds
3. Implement a consistent design system
4. Fix website responsiveness issues
5. Cut down on virtual assistant prompts to log in to MyUI+
FINAL THOUGHTS
The process of creating this usability study and conducting it was a valuable learning experience for us both. While we generally feel that by the end of the study there were very few issues to be resolved, there are still improvements to be made. The number one issue we ran into during the tests were participants not knowing how to communicate that they felt they completed their tasks. Oftentimes, there would be an awkward pause while the participant sort of sat there doing nothing, and as evaluators we had to ask if they were finished. Going forward, we would include instructions in the script that let participants know what it means to be done with their tasks and that they need to communicate once they complete said task.
It is of the belief of the team that if the Colorado unemployment website was to undergo changes and follow the recommendations of this study, its users would likely feel a lot less stressed and on edge while using the site. Stress and frustrations will never be entirely eliminated of course, due to the fact that being unemployed as an adult is generally not a positive experience. However, these changes would make it so unemployed citizens could feel some relief after being able to quickly and successfully apply for benefits.