Intranet usability testing project

Through my Human Centered Design and Engineering (HCDE) master’s degree, I worked with a team to conduct usability testing on the intranet for the HCDE department at the University of Washington. The intranet houses information, policies, and resources for anyone currently in the department including, staff, faculty, and students.

Timeline

January 2022 to March 2022

Problem

Through initial interviews with staff in the HCDE department, we learned the intranet was created many years ago, before the current staff was hired. Through the years, the staff added content and made small improvements. However, they didn’t research the site’s users or effectiveness, so they weren’t aware if the intranet was meeting user needs. They heard anecdotally that it was difficult to navigate.

Team structure

4 user researchers

Our objective

Identify barriers for HCDE faculty, staff, and students when locating policies and resources through the intranet.

The process

Initially, we met with HCDE intranet stakeholders to get a better understanding of the project. Specifically, technical limitations, the rationale behind some of the current designs, strategies for recruiting participants that tailor with users’ schedules and work priorities, pain points users have experienced, and the type of data that should be collected for each task to best suit the research purposes. 

Our biggest takeaway was that user needs weren’t being taken into account in the development of the Intranet. Additionally, intranet content continued to be created without a content strategy in place. Since no previous research had been conducted on the Intranet, we took a two-phased approach to learn more about the users. This included problem discovery and usability evaluation.

Problem discovery

Our discovery research questions included:

  • Where do users currently go to find HCDE information and resources?

  • Who uses the Intranet and for what purpose? 

  • What prevents people from using the Intranet?

  • What challenges do people face when trying to find information and resources? 

  • Do users understand the difference between the HCDE Intranet and website? 

We conducted a survey and received responses from 53 participants. The survey included questions about users' roles, time with HCDE, and whether they had ever managed content on the site. This gave us screening data so we could recruit a representative mix of participants for our studies. The survey was sent to staff, faculty, and students via listserv and Slack.

The survey helped us gain a better understanding of the current user experience and develop scenarios for the usability test, based on the most common tasks. We learned that 96% of survey participants didn’t know the intranet existed. From the people who did use the Intranet,  we learned that the site was difficult to navigate and the information was often outdated.

Users most commonly used the Intranet for these tasks:

  • Reserve a room

  • Request a reimbursement 

  • And get NSF grant information

Before conducting the usability studies, we also mapped out the information architecture, to get a better understanding of the optimal paths for task completion.

Usability testing

To better understand how usable the HCDE intranet is, we developed a usability script that also included user interview questions. Our main research questions included:

  • Can new users efficiently find information using the HCDE Intranet?

  • Where do users experience pain points while looking for information or completing tasks?

  • How do users associate different topics between the HCDE Intranet and the main website?

We conducted 11 remote moderated usability evaluations with students, staff, and professors. Participants were a mix of people who are new to HCDE (fewer than 2 years) and those who have been with the department longer (2+ years). They were representative of varying experience levels of the HCDE Intranet including current users and non-users.

The usability sessions were held over Zoom and had participants share their screen. At the end of the session, we verbally asked the post-test questions to get participants’ immediate reactions after each task. We combined the interview with usability testing due to project time constraints, decreased demands on participants, and to allow comparison between users’ self-reported and actual behavior.

We categorized findings by scope (the area of technological impact) and issue type (a description of the problem underlying the finding). Scope categories included: entire Intranet, grants, policies, reimbursement forms, scheduling tools, specifical user goal and website. Issue areas included: cognitive load, comprehensibility, content, discoverability, navigation, predictability, SEO, trustworthiness and bugs.

Card sorting

Since information findability was important to participants, we also conducted a card sorting activity. We conducted unmoderated card sorting to gain insights into how users organized and thought about information. Two card sorting boards were created in Optimal Workshop for these user roles:

  • Faculty/staff

  • Students

We identified the card topics based on existing information in the HCDE Intranet and HCDE website and insights from our research. Results were collected from 10 students and 10 faculty/staff. The results were coded for these recommendations:

  • Information/resource should move to another category

  • Category/information should get relabeled

  • Information should be linked in different categories

Findings and recommendation 

After analyzing and triangulating the results of the survey, usability test, and card sorting survey, we identified the top usability issues and rated the severity of each issue, based on Dumas’ and Redish’s severity scale.

Successes

We found that participants who were introduced to the Intranet found it’s information helpful and interesting. 6 out of 11 participants did not know about the Intranet prior to the interview and usability session but appreciated knowing there were resources available in an information hub.

"I think the Intranet is actually pretty useful, there are a lot of functions here." - S44

Users also appreciated the block-grid layout on the Intranet home page. The icons and descriptions increased users’ confidence as they sought to determine a course of action.

Opportunities for improvement

These were the top usability issues we observed during our sessions:

  • Users are overwhelmed when task workflows do not offer the sequential steps they need to achieve their goal.

  • Users are unsure which information is out-of-date and who to contact for information about a given task or policy.

  • Information and task workflows are not clearly distinguished, users are overwhelmed by large volumes of information while trying to complete tasks.

  • Users experience a high cognitive load and become unsure that they are in a place meant for them when they see information for roles other than theirs.

  • Lack of awareness that the Intranet exists among users who have been with HCDE for fewer than 5 years

Recommendations

To address these issues, we developed these recommendations for the staff working on the HCDE intranet:

  • Examine the flow of top user tasks from start to finish. Remove unnecessary hurdles, and make logical next steps visible and comprehensible. For example, simplify the number of colors and patterns used in the scheduling map.

  • Consider renaming some elements and group information based on users’ expectation and mental model. Change the title name to one that aligns users’ expectations with the internet's purpose and function.

  • Conduct a content audit and remove redundant and outdated files. Consider adding “last updated” date to policies or documents.

  • Categorize information and policies based on relevant roles' use cases. Clearly distinguish what information is relevant to a specific HCDE role (e.g., student, staff, faculty, etc.). This may involve reorganizing the information by role or using clear headings to separate role-specific information.

  • Increase the awareness of the Intranet in the HCDE community, especially for people who are new to UW.

We presented these insights to HCDE intranet staff members who were very receptive to the feedback and opportunities for improvement.

Success measurements

To measure the success of HCDE Intranet updates, I’d recommend following up with additional usability testing. Additionally, HCDE staff could review metrics, such as page hits, time on page, and numbers of intranet visitors.

What I would do differently

Meet with key stakeholders earlier 

Some of the task flows in the intranet were confusing. For example, there were multiple reimbursement forms, so we weren’t sure which one should be used for travel reimbursement. This made it difficult to determine what was considered a success for some of the tasks. After we started writing the usability script, we met with the owner of the reimbursement forms. They provided a lot of great insight into the reimbursement process, which enabled us to clearly define the success criteria for completing a travel reimbursement. In the future, I’d make a list of key stakeholders and meet with them earlier in the process.

Recruit for users who use assistive technology 

Accessibility is one of my biggest passions, so I aimed to include screen reader users in our usability testing. In our survey we asked if the participant used a screen reader. We didn’t recruit anyone who primarily uses a screen reader to get information. In future studies, I’d want to make sure usability testing includes people who use assistive technology to ensure the research is inclusive and accurate.