01 – MASTER THESIS

Usability evaluation of the Apple Watch Series 4 smartwatch

01 – MASTER THESIS

Usability evaluation of the Apple Watch Series 4 smartwatch

01 – MASTER THESIS

Usability evaluation of the Apple Watch Series 4 smartwatch

01 – MASTER THESIS

Usability evaluation of the Apple Watch Series 4 smartwatch

01 – MASTER THESIS

Usability evaluation of the Apple Watch Series 4 smartwatch

Master Thesis - Cover img

Created as a part of the Master Thesis at Aalborg University, 2020

Created as a part of the Master Thesis at Aalborg University, 2020

Created as a part of the Master Thesis at Aalborg University, 2020

Created as a part of the Master Thesis at Aalborg University, 2020

Created as a part of the Master Thesis at Aalborg University, 2020

Introduction

Since the first smartwatch in 1998, the SW industry has become a highly profitable market, expecting to reach a value of 117,51 million dollars by 2025 at shipment volume.

First Apple Watch was released in 2015, and since then has become one of the market leaders in the SW industry, securing a 47,9% global market share in 3Q 2019.

Furthermore, wearable adoption is increasing rapidly, registering from 13% in 2015 to 33% in 2019, an increase of 20% in just three years.

So far, academic studies on SWs have been more technology rather than audience-driven, so there is a need for understanding of actual user perceptions and intentions on SWs. In addition, there is a need for a shared understanding of how users use these devices in order to create guidelines and principles. Academic research on SWs is still in an early stage, and only a few studies have intensively studied the usability of smartwatches.

With this Master Thesis, my intent was to evaluate the usability of AW Series 4 based on if the person owns an AW or not. Evaluating it from these two perspectives gave richer data and a better understanding of the people who use the AW. Furthermore, investigate what the current state of its usability is and how it can be improved. And lastly, does it and to what extend the previous ownership of AW and iPhone have an impact on the usability of the AW.

To answer these questions, the following problem statement has been formulated:

What is the usability of the Apple Watch at the current state (AW Series 4, 44mm, Software version 6.2.1), and how can it be improved?

Framing the problem

In the collaboration with the senior product designer and the product manager, I created the new onboarding journey to increase the renewal rate for SQL Prompt plugin. 

SQL Prompt is a plugin for SSMS and Visual Studio for writing, formatting, analyzing, and refactoring SQL code. Even though SQL Prompt has many features, 50% of customers use only one feature (suggestions), which sometimes leads to them not renewing the license.

Framing the problem

The problem with the former website was an outdated and non-responsive design. Before the project kickoff, the client conducted small-scale research where they found that the target audience perceives the website as “cold” and “hard to navigate”.

My role was to redesign the website from the ground up, in a collaboration with the stakeholders and developers at Enterwell.

With the new website, the goal was to make the user feel like the Hrabri telephone's website is a safe space where they can ask for help, learn how to deal with different situations and emotions.

Due to the client's budget constraints, no further research was conducted, and instead, I relied on best practices for great user experience and desk research. 

Reviewing the available knowledge

In order not to reinvent the wheel I have conducted a literature review on smartwatches and usability, technology adoption, and models and theories of user acceptance.

The three key things I learned are:

  1. SWs are used in a variety of ways, such as Personal Assistance, Wellness, Healthcare, Sports, and others.
  2. Two main constraints of SW are small screen size and battery life.
  3. Good product usability is critical to the wide adoption of wearable devices.

Identifying the participants

Using snowball and convenient sampling two groups of people were recruited - people who own an Apple Watch, and people who do not own an Apple Watch. Doing this enabled me to evaluate the usability of two different perspectives.

To sum up, the required criteria for the population are:

  • Has or has not used an AW before
  • Owns an iPhone
  • Fluent in English or Croatian

 The plan was to recruit 10 people, 5 for each group, but one person canceled at the last minute and due to the time constraint I did not have time to recruit a new person. Therefore, 9 people were recruited (4 for people who own an Apple Watch and 5 for those who don't).

Evaluation setup

The research was conducted in three stages:

  1. Pre-test Questionnaire
  2. Usability testing
  3. Semi-structured Interview

To increase the reliability all participants were instructed to update their iPhone to the latest software version, i.e. v13.4

1. Pre-test questionnaire 

The self-administered questionnaire was chosen for this study to gather different opinions from participants regarding the AW. The questions were divided into 5 meaningful parts, in a total of 14 questions some of which were only asked if the person owns an Apple Watch. 
Before rolling out the questionnaire I have conducted a pilot test after which some questions were refined. 

The questionnaire was summarized using descriptive statistics and visualized using graphs and charts.

Master Thesis – Questionnaire questions

Questionnaire questions (Q4, Q5, Q7 and Q11 are only for those people who own an Apple Watch)

Questionnaire questions (Q4, Q5, Q7 and Q11 are only for those people who own an Apple Watch)

Questionnaire questions (Q4, Q5, Q7 and Q11 are only for those people who own an Apple Watch)

Questionnaire questions (Q4, Q5, Q7 and Q11 are only for those people who own an Apple Watch)

2. Usability testing

The usability testing consists out of 10 Task groups which utilize main types of interaction on Apple Watch - tap, press, swipe and drag, and target selection. It was analyzed using performance metrics (task success, time on task, efficiency, and learnability), observation, retrospective think-aloud, and jotted notes.

The format used for the usability testing:

  1. Introduction – Introduction to what is the goal of this Usability Testing, reminder that this is confidential. Setting the scene for tasks to be carried out, explaining what will be tested, that after each task group, we will have a short discussion (Retrospective think-aloud), before moving to the next task. Lastly, reminding them that we are here to evaluate the Apple Watch, not them.
  2. Tasks (+Retrospective think-aloud) – Ten task groups, followed by Retrospective think-aloud after each Task group. Each Task group consists of several tasks.
  3. Conclusion – Short conclusion about the Usability testing, and a break before moving to the third phase of the research, the semi-structured interview.
Increasing validity and reliability

In order to increase the validity and reliability of the Usability testing, I have created a set of rules for the Apple Watch settings. They are as follows:

  • Turn Wi-Fi off
  • Turn Do not disturb off
  • Disable the screenshot option
  • Sett App layout to “list view”
  • Set Watch brightness to “Medium”
  • Add “Motion” Watch face
  • Turn on Notification indicator under Notifications
  • Check that Background refresh is on for mail app
  • Check that Dictation is on
  • Check that notifications are enabled, are mirroring iPhone
  • Enable all three options for Ask Siri, enable voice feedback
  • Enable heart rate and fitness tracking under Privacy
Master Thesis - Task Groups

Task groups for the Usability testing

Task groups for the Usability testing

Task groups for the Usability testing

Task groups for the Usability testing

3. Post-test interview

To help creating a better interview guide I have used the three key questions by Kvale - What, Why and How. The interviews were audio-recorded and transcribed. After which they were analyzed using concept-driven coding in NVIVO. Before the interviews I conducted a pilot test after which one question was rewritten to be more understandable. 

What: Getting a deeper understanding of the answers the participant gave in the questionnaire, further discussion on how they perceive the AW, and the usability of it.

Why: In order to have richer data which will help in the analysis of AW.

How: Through a semi-structured interview.

Master Thesis - Interview Guide

Interview questions

Interview questions

Interview questions

Key findings of the usability evaluation

After conducting the three before-mentioned stages, each was separately analyzed, but to not go into too much lenght I only show the analysis results from usability. For full results please refer to this link.

Effectiveness

Analysis based on the rate of completed tasks, percentage of completed tasks, and what the participants said themselves throughout the UT.

  • Overall task completion is positive, with 73,33% for Apple Watch owners, and 68% for non-owners.
  • The biggest problem that influences effectiveness is the Force Touch feature. Besides that, some users were unable to find the power reserve mode, but that is not suprising because like Force Touch it is not very clear.
  • Longer ownership does not necessarily mean that the person will experience a higher effectiveness rate of AW.
Master Thesis – Percent of task completion both groups

Number and percent of task completed for (AW=Yes) and non (AW=No) Apple Watch owners

Number and percent of task completed for (AW=Yes) and non (AW=No) Apple Watch owners

Number and percent of task completed for (AW=Yes) and non (AW=No) Apple Watch owners

Number and percent of task completed for (AW=Yes) and non (AW=No) Apple Watch owners

Master Thesis - AW Ownership vs total completed tasks

Apple Watch ownership compared to total of completed tasks for Apple Watch owners

Apple Watch ownership compared to total of completed tasks for Apple Watch owners

Apple Watch ownership compared to total of completed tasks for Apple Watch owners

Efficiency

Analysis based on a combination of task success and time on task metrics.

  • The efficiency level depends on the task group, because of the Force Touch feature. Because if the task required for that feature to be utilized in order to successfully conduct the task, and if they did not know how to use it they would fail the task.
  • The longer the person owns an iPhone and AW the better efficiency they will experience, although on negligible level
Master Thesis – Efficiency (Both groups)

Efficiency levels for (AW=Yes) and non (AW=No) Apple Watch owners

Efficiency levels for (AW=Yes) and non (AW=No) Apple Watch owners

Efficiency levels for (AW=Yes) and non (AW=No) Apple Watch owners

Master Thesis - Average Time on task vs AW & iPhone Years Owned (Both groups)

Average time spent on each Task group compared to time of Apple Watch and iPhone ownership for (AW=Yes) and non (AW=No) Apple Watch owners

Average time spent on each Task group compared to time of Apple Watch and iPhone ownership for (AW=Yes) and non (AW=No) Apple Watch owners

Average time spent on each Task group compared to time of Apple Watch and iPhone ownership for (AW=Yes) and non (AW=No) Apple Watch owners

Satisfaction

Evaluation based on what the participants said during the Usability testing, and the observation of they carried out the tasks.

  • Overall positive satisfaction of AW among both groups.
  • When asked what comes to their mind when they hear “Apple Watch”, the results from both groups are positive with answers such as “simple, premium, easier workout” etc.
Learnability

Most products require some time to learn how to use, and usually, the learning is not a thing that happens in an instant, but it develops over time as the experience increases. I wanted to evaluate the learnability of AW mainly because of the two participants group to investigate the learnability of each of them.

The analysis was based by looking at the mean task time over time, although I would argue that evaluating the learnability only by looking at the mean task time over some time does not necessarily reflect how is the learnability of the product or service that is being tested, due to the various levels of complexity of Task groups.

  • People who used the AW for the first time quickly displayed an increase in learnability and efficiency.
  • Learnability is improving among both groups based on the results.
Master Thesis – Learnability (Both groups)

Learnability of the Apple Watch for (AW=Yes) and non (AW=No) owners

Learnability of the Apple Watch for (AW=Yes) and non (AW=No) owners

Learnability of the Apple Watch for (AW=Yes) and non (AW=No) owners

Learnability of the Apple Watch for (AW=Yes) and non (AW=No) owners

Learnability of the Apple Watch for (AW=Yes) and non (AW=No) owners

Conclusion

  • The results showed that people use the Apple Watch as a supplementary device to their iPhone to get glanceable information.
  • Both owners and non-owners of Apple Watch experienced average effectiveness, with non-owners being slightly lower than owners.
  • Force Touch had the biggest effect on effectiveness, because of which some Task groups could not have been completed, as using Force Touch was required to complete the specific Task groups.
  • The time of ownership of the Apple Watch does not influence its effectiveness, but it does on efficiency on a negligible level.
  • Both sample groups perceive the Apple Watch as useful and like the Apple Ecosystem.

    This is a summarized version of the project. For the full master thesis please refer to this link.

Impact

In their latest WatchOS, v14 Apple has removed the Force Touch feature and instead uses a button instead.

This Master Thesis was awarded the highest grade.

Conclusion

  • The results showed that people use the Apple Watch as a supplementary device to their iPhone to get glanceable information.
  • Both owners and non-owners of Apple Watch experienced average effectiveness, with non-owners being slightly lower than owners.
  • Force Touch had the biggest effect on effectiveness, because of which some Task groups could not have been completed, as using Force Touch was required to complete the specific Task groups.
  • The time of ownership of the Apple Watch does not influence its effectiveness, but it does on efficiency on a negligible level.
  • Both sample groups perceive the Apple Watch as useful and like the Apple Ecosystem.

    This is a summarized version of the project. For the full master thesis please refer to this link.

Impact

In their latest WatchOS, v14 Apple has removed the Force Touch feature and instead uses a button instead.

This Master Thesis was awarded the highest grade.

© 2020 Denis Maček

© 2020 Denis Maček

© 2020 Denis Maček