Online Citizenship Test Tool

Immigration, Refugees and Citizenship Canada (IRCC) - 2021
Usability Testing Project
Context
Online tool to allow Canadian citizenship clients to take the knowledge test, which is one milestone in the citizenship application process. The platform was developed in response to COVID-19 when all in-person testing had come to a pause.

The Usability Testing team was involved early on to inform design decisions and make sure the tool is user-friendly and easy to use. The tool had an officer-facing portal exclusive to officers to review results of the clients' tests. Both the client and the officer portals were usability tested with their respective audience.
Contributions Overview
Usability testing informed the design of the tool along different design sprints. Usability testing uncovered critical errors in the intial rounds, which when left unfixed, clients could have unduly failed their knowledge test. In addition, they could have resulted in more inquiries to the deparment (which cost money!).

Usability testing helped:
• Improve the task flow of the entire process from receiving an invitation, to veryfying the identity, to taking the test,
• Provide clear instructions and language
• Reduce user burden of having to remember which questions were left unanswered or marked for review by providing more salient visual cues
• Appropriately prompt users when a critical action is required

Findings and recommendations were discussed with the design team at each sprint and the decision was to keep testing until it conveyed little to no usability issues.

Disclaimer

(1) The first and last rounds were led by the owner of this portfolie, but round two was led by other researchers in the same team and its results are covered here for completeness but credit for that round's findings and recommendations goes to them.

(2) This page contains no screenshots of the tool of confideniality reasons.

Methods and Results in Depth

Test Objective
Consistent accross all rounds
Client Portal
To explore the usability aspects of the Online Citizenship Test tool; from initial login, accessing the identity confirmation page via email link (email not tested), terms and conditions page, starting and taking the test, and receiving results in the website and by email.

The first round axplored only parts of the above because some of the pages had not been added yet. On the other hand, the last round explored the above in addition to other new features such as the client’s Permanant Resident card identification, client’s photo identification, and the live webcam feed displayed on the test pages.
Officer Portal
To test out the overall language and content of the admin side of the tool.
Participants
Distinct in each iteration
Client Portal
• Four to six participants in each iteration
• All participants were either permenant resident clients who had a citizenship application in process, or newly Canadian citizens
Officer Portal
• Six processing officer participated in the final iteration
Timeline
For the usability testing only
• Rounds of testing were conducted between April and July 2020 and soon after the tool was launched.
Iteration 1 - Findings
Client Portal
The prototype at this stage focused on fucntionality more than user-friendliness, as a result, participants unsuccessfully thought that they had completed the usability tasks successfully while they had not. For example, they missed answering questions that they had skipped because the tool did not nudge them to. They thought that the test was submitted while submission needed one more action that they had missed. Overall, the tool relied on user's memory and attention to complete critical actions that the UI should have been more salient about.
Iteration 1 - Recommendations
                                     
The recommendations were mostly around making visual cues about unanswered/user-skipped questions more salient, and adding more system nudges (i.e. warning or pop-up messages) when a critical user actions is overlooked. In additon, there was a recommendation about reducing the number of action buttons since submitting the test had two buttons, with one unnecessarily misleading users and could result in them failing the test even when all questions had been answered correctly.
Iteration 2 - Findings
Client Portal
Many new pages were added to match the business requirements for verifying clients before taking the test and ensuring they read and agree to privacy notices and terms and conditions. However, the task flow for gaining access to the test and the instructions provided no longer matched the tasks, as some instructions were confusing and sometimes unnecessarily lengthy. This could have lead to confusion, errors, delays, and increases in help-seeking inquiries to the department.
When taking the test, participants were overall able to use the tool to complete the citizenship test, but further changes to the in-test user interface (UI) were further needed to improve the user experience.
Iteration 2 - Recommendations
                                     
The recommendations were mostly around clarifying and simplifying the task-flow of invitation and signing in to take the test, organizing information such that items demanding action come first and having detailed information available as supplementals. In addition, recommendations centered around using UI elements in a consistent way and such that each is serving only one purpose at a time. Furthermore, in-test UI feedback needed to be immediate after user action and to account for different user actions. Finally, key UI elements needed to be more falient and visible on smaller displays.

‍
Final Iteration - Findings & Recommendations
Client Portal
This version of the prototype tested well, as participants found language, instructions, and the in-test UI straightforward and intuitive.

Recommendations were around language clarifications and addition of specific instructions to improve the overall user experience.
Officer Portal - Findings & Recommendations
                                
Officers found the tool user friendly and informative. One major confusion was around the wording of some of activity log events that required their review before confirming that no cheating took place and approving the client's score.

Recommendations were around clarification of the language, adding more information in some sections, and addition of help text to aid officers in making a more informed decision.
Powered by Webflow