UX is everything – Measure it completely and correctly.
Forrester Research calls it the “mobile moment”—that brief instant when users first try your mobile app and decide whether they love it or hate it.
Multiple studies confirm that the success of your mobile app—and potentially your business—depends on launch speed, performance, stability, battery usage, and other aspects of the user experience.
For example:
- Ninety-six percent of users say mobile app performance is important. 49 percent expect mobile apps to respond in two seconds or less, and 84 percent will stop using the app if it performs badly.
- When there are performance or stability issues, 55 percent hold the mobile app responsible, and 53 percent uninstall or stop using the app.
- When a mobile app fails, 31 percent of users will tell others about their poor experience.2 Because 34 percent of users choose apps based on peer feedback3 (more than price), one bad experience has huge impact on the company’s brand.
- Thirty-six percent say app performance and 37 percent say stability issues have caused them to have a lower opinion of the company that created the mobile app.
The clear message for businesses that create and distribute mobile apps is that if you don’t create an outstanding user experience, your mobile app is dead in the water.
Yet existing mobile analytics tools still do not adequately measure the user experience. Some measure only a few variables or have inconsistent measurement processes; others measure variables that provide incomplete or misleading information.
It’s time to overcome the inertia and begin measuring the mobile app user experience—completely, correctly, and in real time—so that developers can create apps that amaze and delight users, and enterprises can maximize both user satisfaction and business results. This paper explores the challenges of mobile app user experience measurement and a more intelligent approach to measuring what truly matters.
Taming the complexities of user experience measurement
It’s easy to understand why companies have been slow to implement comprehensive user experience measurement practices. It can be an extremely complex endeavor.
For starters, the entire development process for mobile apps is different from that of traditional applications, with the focus on fast release cycles and rapid, feedback-driven iterations. The user experience has traditionally been the domain of specialists, but in the mobile arena it is the province of every developer.
Equally important, it can be very difficult for development teams to pinpoint which metrics of the user experience to measure and monitor. How exactly do you measure “performance?” Should you measure the network response time or the UI response time? For which platforms and operating systems? Under what conditions? How do you measure stability and reliability? Should you measure resource usage such as battery draw?
Many mobile analytics and application management solutions have appeared on the market—but the diversity of the solutions and their capabilities only add another layer of complexity. How do you determine which features and functions are critical? How do you evaluate which option is best for your situation?
Equally important, many of the tools currently on the market are limited in their functionality. Some do not measure from the end user’s perspective; for example they measure only the network response time or they capture problems such as crashes without measuring the impact on the user. Others require developers to add tags to their code to measure the user experience.
“In the end, no number of green lights on a performance management dashboard will matter if end-users are dissatisfied with a mobile app’s performance and usability.”
– Aberdeen Group4
A smarter approach: “measure, focus, improve”
HPE AppPulse Mobile brings a unique approach to the measurement of the mobile app user experience, and this approach has proven uniquely effective for our customers. The goal is to measure everything that impacts the user experience, so developers and QA can focus on the user actions users care about most and continuously improve their mobile apps.
The HPE solution captures, names, and measures ALL user actions automatically, without requiring developers to add code or maintain tags, because it’s simply a SaaS wrapper to the mobile app. Other mobile analytics solutions require developers to add code tagging and decide where an operation starts and ends. This process is time-consuming and error-prone—and requires maintenance when code changes.
HPE AppPulse Mobile measures and reports on the user experience and business impact in three broad categories:
- Performance: It quantifies the response time of each user action and the launch time of the mobile app.
- Stability: It tracks user actions that result in crashes, as well as errors that result from user actions.
- Resource usage: Both battery usage and cellular data usage are measured and monitored.
In addition, HPE AppPulse Mobile shows a graphical view of the entire user flow. It follows the funnel of the user’s interaction with your app, screen by screen, and lets you drill down into data about user actions. This enables you to understand the level of engagement with your app, where problems occur that impact the user experience, even when and why users exit your app—so you can design and build better flows and constantly improve the user experience.
Response time measured from the user’s perspective
HPE AppPulse Mobile measures app performance in terms of UI response time, as perceived by the user. For each user action, the HPE solution tracks the async tasks required to process the action that were initiated from this user action, and measures the time from the user taps until the screen is ready with the “primary information” for the user (technically, the last thread/async task that updated the UI).
This approach is in stark contrast to most other measurement solutions, which measure the network request time. The problem with measuring the network request time is that in many cases a network request does not impact the user at all. It doesn’t measure the client processing time, which is more significant in modern applications, especially in the mobile arena where many different operating systems and devices can impact the client processing time.
To illustrate the point, consider this example. Let’s say you own a mobile app for hotel reservations. It’s a competitive market and you need to deliver a user experience that’s truly superior—so you want to maximize the speed of operations such as a hotel search. The “speed” experienced by the user depends on a lot more than the network request time. When a user enters a selection for a hotel search, the app contacts the server and asks for the relevant search results. The server sends the results. Now the app has to process the data, which means it may need to compress it. Then the hotel’s list has to be sorted per user preferences. If hotel images are also displayed, the images must be opened and displayed. This all takes time. And the processing time also varies depending on the size of the device and its screen. When the display is larger (as it is on tablets), your app needs to process more items because more items are visible. Each user experiences the performance differently, even if network time is the same. Simply put, the response time experienced by the user is dependent on much more than the network request time.
Launch time includes start time and resume time
The HPE approach is also unique because it measures the launch time for mobile apps. The launch time—when the app was not running in the background already—usually takes longer than a standard action, so the HPE solution measures both start time and resume time. This way there are accurate and meaningful metrics about the user’s experience, even if the mobile app had already been running in the background.
Traditional approaches sort all “requests/actions” either by slowest actions or by top volume. However, the slowest actions might not be the actions with the highest volume or with the greatest business impact. The “favorites” might perform well, leading to incorrect conclusions about the overall user experience.
That is why the HPE approach draws the app owner’s attention to slow app operations that impact the most users and hence have the highest impact on the business. It does so by measuring the number of unique users that experienced the action as slow, and sorting the slow actions accordingly.
Relevant performance reporting with device/OS breakdowns
The HPE approach to user experience measurement allows development/operations teams to see a breakdown of response times by device or operating system, so they can focus on one particular platform at a time. Other mobile analytics solutions break down the network response time by device and OS, which is not always relevant because what matters most is the actual performance experienced by the device/OS user.
The HPE solution measures the true client performance, which allows the app owner to see and respond to performance issues and assign QA and development resources accordingly.
Crash analysis with business implications
HPE AppPulse Mobile measures crashes—because a crash is a fatal error that can also be fatal to the success of the mobile app. Unlike other solutions, HPE AppPulse Mobile first shows the user actions that caused the crash, then it counts the number of unique users affected by this crash, and groups the crashes according to their impact on the business. This enables mobile product managers to prioritize which are most urgent. For example, a crash that occurs during the “payment” phase of a mobile app transaction is much more severe to the business than a crash on “logout.”
Compared with solutions that track the “exception stack trace” or “error type,” this approach provides a more relevant metric that the mobile product manager can easily understand.
Of course, not all crashes are related to a specific user action, and the HPE solution takes this into account. Crashes that did not result from a specific user action, such as an OS shutdown, are grouped as unclassified crashes.
Error tracking with drill-down capabilities
HPE AppPulse Mobile captures user actions that result in error messages from the network, such as an HTTP error from a specific URL. It does so from the user’s perspective, focusing on whether the user received an error message or not as a result of an action, and whether the user was affected by that error.
Specifically, it tracks the total number of users that received an error message, the user action that resulted in the error message, and the percentage of actions that resulted in at least one error message, along with the app version number. In addition, it tracks the daily average of users with errors on app launch, and the daily average of users with unclassified errors.
AppPulse Mobile also allows you to drill down to get a list of the 20 most recent URL requests that came back with an error message. This makes it possible to see, understand, and prioritize which error message issues to address first.
Battery usage measured from the user’s point of view
Battery life is the single most important feature for new phone buyers.7 That is why AppPulse Mobile makes it easy to measure and monitor the battery consumption of your mobile app in a meaningful way. While other tools might tell you the state of the battery during a crash, AppPulse Mobile shows metrics that are more relevant from the user’s perspective, such as the percentage of battery consumed per minute of use, by device type and by app version. This makes it possible to re-think your app behavior to minimize battery drainage and thereby improve the user experience.
Comprehensive tracking of cellular data consumption
Users won’t continue to use apps that eat up their cellular plan’s data allotment, so it’s important for mobile app developers to measure and monitor data usage very carefully. AppPulse Mobile is unique in its ability to pinpoint which user actions consume the most data. It does so by measuring the kilobytes of data consumed per minute of use. You can see the percentage of data consumed by a specific user action, the total amount of data per cellular hit, a daily average of the number of hits per user, the consumption by app version, and the total number of users.
Equally important, you can see a breakdown of cellular consumption by service, so you can tell how much data is consumed by third-party services, such as LinkedIn or Stock.com, as well as the data consumed by your own app.
The user flow: follow their journey screen by screen
The user flow feature of HPE AppPulse Mobile presents a data-driven, graphical view of user engagement that follows the “funnel” of user interactions with your mobile app, screen by screen.
For each screen, it shows you key metrics such as the total number of sessions and the most popular user actions. You can then drill down into metrics such as user actions that were slow, or actions that resulted in error or crashes.
You can also see the number of users who exited your app—so you can quickly determine how to optimize the user experience and keep users fully engaged. It gives you the option of seeing data over various time periods, for a particular version number, or for all versions.
FunDex: a single score that encapsulates the user experience
HPE AppPulse Mobile provides a concise, intuitive measurement of the user experience, called “FunDex.” The FunDex is a single score, from 0 to 100, that encapsulates all aspects of the user experience measured by HPE AppPulse Mobile—performance, stability, and resource usage—in real time. At any given moment, you can see a graphical view of the “FunDex per user” and the “average FunDex of all users” to quantify the average user experience. The visual representations enable teams to compare metrics over time, compare results between versions, compare with industry standards (future), and much more.
The average user experience score also makes it possible to anticipate the experience of a new user, as well as to correlate usage and user experience. For example, it is possible to compare the FunDex of the users that use the app the most with the FunDex of users that use the app very little, and understand the impact of the FunDex on usage.
Simply put, FunDex provides an intuitive, easy way to see how to improve the user experience of mobile apps, to identify actionable steps, and to prioritize those actions to deliver the highest user satisfaction and maximize business results. For additional details about FunDex, please download the white paper, “What’s Your FunDex?”.
Learn more at: hpe.com/software/apppulse