Mobile UX Marathon: Quantitative and Qualitative Research Methods - Louise Barry

Hello and welcome to the Mobile UX Marathon, a series of weekly webinars on how to improve your user experience and conversion rate on the mobile web. Today's webinar focuses on qualitative and quantitative research methods.

In our last webinar, we introduced our maturity framework. Today, we'll dive deeper into the most popular parts of CRO (Conversion Rate Optimization) and explore how these relate to each stage of that maturity model. Please share your questions with us on the UX Marathon website and later join our live stream to see those questions answered. The website also has more information about upcoming live streams, resources, decks, and so on. The link for that is in the video description below.

Before I begin, I'd like to introduce myself. Hello, my name is Louise. I work in the Mobile UX team based in Dublin, collaborating with our largest customers in the UK and across EMEA. I work across various verticals, including travel, finance, retail, gaming, and more. I have a background in UX analytics and CRO.

Here is an overview of today's agenda. I'll start with a quick refresher on the CRO maturity model and the CRO process. I'll define what qualitative and quantitative research is, to ensure we're all on the same page. Then, I'll go through qualitative and quantitative research methods based on that maturity model.

If you watched my previous webinar, you'll be familiar with this slide: our CRO maturity model, which maps out the factors that determine a business's maturity. Today, I'm going to talk more about one of the factors we examined as part of that model: insight methods. Many people confuse CRO to be solely about these activities, but they are just one part of the CRO process. Ignoring other factors, such as processes and strategy, can lead to getting stuck and struggling to make progress with your CRO efforts. It's like building a house without proper foundations—eventually, you'll start to see the cracks.

When we look at the CRO process, you'll see it's broken into three parts: qualitative, quantitative, and heuristic analysis.

Today, I'm going to talk you through qualitative and quantitative insight methods based on the maturity model. Essentially, I'll discuss the types of activities you should be doing based on your position within that maturity model or framework. By sticking to the activities that align with your maturity level, you are far more likely to achieve the results you're looking for and experience growth.

Here, you can see I've broken down the maturity model by qualitative and quantitative research. I want to emphasise that you can mix and match between these different activities. My list is purely a guide and should not be followed word-for-word. What we're going to do is look at the types of insight methods that companies could implement based on their maturity. We've compiled this list specifically for today's webinar as a reference, acknowledging that activities can vary depending on different factors and that every company is different.

CRO is a repeatable process that provides insight into where your problem is, what the problem is, and an understanding of your customers' minds. The typical outputs from these activities are insights, which you can then apply to your site to improve conversions. As with other factors, there are no shortcuts. You can't perform advanced research methods without first learning the basics.

Before I begin, please remember: do not try all of these activities at once. We recommend starting with one or two activities, preferably the ones you're more familiar with, and then continue working with them to get consistent results—not one-off potential flukes. Demonstrate the consistency in the insights you're receiving and the subsequent impact. Aim for quick wins that help gain buy-in and focus on key problem areas. This is where you should see a big improvement in user behaviour.

As I mentioned earlier, qualitative research is conducted through observation and inquiry, while quantitative research is conducted through data measurement, analysis, and comparison. Essentially, quantitative research will tell you the "what," while qualitative research will tell you the "why." Remember, both sets of activities complement each other and should be integrated as part of your CRO insight methods.

Maturity Model

Let's look at the maturity model, starting with a nascent client and the quantitative research methods they can use. Again, this is just a starting point, so feel free to mix and match these activities based on the knowledge you have in your business. We always encourage you to do both qualitative and quantitative research, as mentioned in the previous slide. Quantitative research will tell you the "what," and qualitative research will tell you the "why." They really do go hand in hand and complement each other.

First off, when it comes to quantitative research, please make sure your data is clean, accurate, and up-to-date. There is nothing worse than making decisions based on unreliable data.

I would invest time in making sure that whatever tool you are using to capture your data is accurate and reliable. Ensure you have a minimum of three months of reliable data before you start actively reviewing it to understand user behaviour. Avoid making quick decisions based on data alone. Every case is different, but in general, avoid reacting immediately to a negative metric unless it is repeated and becomes a trend. You would be surprised by the number of people who react solely based on bounce rates and don’t cross-reference them with other data points such as time on page. They focus on one metric and miss the full picture and overall pattern of behaviour. You want to start collecting data that provides a good overview of user behaviour. Set up alerts to notify you of any sudden spikes or drops in key metrics, but be aware that reacting immediately isn't always the best action—being informed is.

Qualitative Research for Nascent

Qualitative research methods can be done on a more ad-hoc basis at this stage. For example, guerrilla user testing, which is a less official version of user or usability testing, is a fantastic way to gain further insight into users' experiences of your website. User testing, both official and guerrilla, should be part of your overall CRM cycle and not something done only for a new feature or when a page goes live. At every stage, users should be asked their opinions on new designs, features, and prototypes. The insights gained from this should be documented, and the quantitative research should take this insight into account to determine how common this behaviour is within the data.

For guerrilla user testing, rather than recruiting users and interviewing them in labs, you can ask colleagues, friends, or family to test out features and provide feedback. What is important is that they are honest, have the correct devices, and preferably are in the right environment for using that device. You could also start adding pop-ups to specific pages on your site. For example, if a user lands on a page and then goes to leave, you could create a pop-up or an alert to ask the user for feedback or to rate their experience on your website.

As part of our audits, we regularly do user testing to determine whether the pain points we have identified match the users' opinions, and we then compare the results to what we see in the data. When we run design sprints, user testing is done before, during, and after.

Emerging Methods

Quantitative Research

If you identify as emerging, these are some of the minimum quantitative insight methods we recommend:

1. A/B Testing: This is crucial. A/B testing involves showing two variants of the same web page to different segments of website visitors simultaneously and then comparing which variation drives more conversions. After gathering your research and insights, test your ideas on a small demographic to ensure changes align with user preferences and avoid financial loss. Always perform A/B testing before making any changes to your website.
2. Heatmaps: These provide a visual summary of user interactions. Different types of heatmaps include mouse movements, scroll maps, and click maps. While these methods alone may not provide comprehensive insights, combining different data points helps build a clearer picture.
3. User Segmentation: This involves identifying and grouping different user behaviours. It's essential for running A/B tests to know your target audience and their behaviours.
4. Event Tracking: This tracks clickable features on your website and provides valuable insights into user interactions. Event tracking is vital for understanding how users engage with your page.

Qualitative Research

For emerging qualitative research methods, consider the following:

1. User Testing: More formal user testing can be conducted in rented labs to observe users directly. This allows you to be more targeted with user types and understand specific demographics better.
2. Video Recordings: Video recordings can eliminate guesswork by showing real visitor behaviour on your site. Observing clicks, taps, and mouse movements helps identify usability issues on the fly.
3. Surveys: Surveys allow you to ask users directly about their experiences. Target surveys based on user segments to ensure relevant questions.
4. Card Sorting: This technique involves asking users to organise information into logical groups, which helps improve the information architecture of your site.

By implementing these methods, you can gain a deeper understanding of user behaviour and make informed decisions to enhance user experience and conversion rates.

Connected Businesses

Connected businesses should be confident in both qualitative and quantitative insight methods. Here are some recommendations:

Quantitative Research

1. Data and Tracking: Ensure robust data and tracking systems. These businesses should have extensive event tracking in place, with dedicated resources for this purpose.
2. Senior Management Involvement: Senior management should have an overview of various reports that provide a high-level impact of the CRO activities. They should be capable of calculating metrics such as customer lifetime value.

Qualitative Research

1. Eye Tracking: This is a great way to see how users scan a page, checking if they notice key propositions and call-to-action elements.
2. User Interviews: Conduct user interviews regularly, especially before designing new products or features. This helps gather user insights early on. Repeating interviews after a new feature launch can help determine its success and understand user behaviour.
3. Email Surveys: Use email surveys to gather user opinions on specific topics, such as a new feature, page, or the overall website. Ensure you have done prior research to target the right audience and ask relevant questions. Poorly targeted surveys can annoy users, so it's crucial to execute this method correctly.
4. Focus Groups: Similar to user interviews but conducted in a group setting. Depending on the focus group’s purpose, incorporating activities like card sorting can provide additional insights.

Multi-Moment Businesses

For businesses that are advanced in their CRO practices, here are some suggestions:

• Combine user testing and card sorting to gain deeper insights.
• Leverage advanced methods such as eye tracking and user interviews.
• Conduct email surveys and focus groups with precise targeting and clear objectives.
• Ensure robust data tracking and involve senior management in high-level impact analysis.
• Use a mix of qualitative and quantitative methods to build a comprehensive understanding of user behaviour and preferences.

Multi-Moment Businesses

Multi-moment businesses are experts in both qualitative and quantitative research. They are incredibly advanced, having gathered and analysed extensive data. These businesses excel at personalization, tailoring the user experience based on specific criteria.

To summarise, don't try all of these activities at once. Remember, this is purely a guide. If you're in a position to combine activities from different maturity stages, do so. We can't emphasise enough the importance of both qualitative and quantitative insight activities simultaneously. Always use A/B testing to justify implementing your findings. Ensure you don't get too comfortable relying on the same one or two insight methods. Be aware of all the different types and choose the correct one based on the user challenge.

Here are some useful resources, which will be shared on the website alongside the video. Please take a look. The qualitative and quantitative insight methods will also be available to download as a PDF.

During the live stream, we will cover the CRO Mobile checklist, the CRO maturity model framework, Q&A, and the qualitative and quantitative methods of research Q&A. Please submit your questions, and we will answer them live on air. Depending on the questions, we might even bring in guest speakers to help answer them. Please don't be shy and submit your questions. It's important that we customise the live streams as much as possible.

That's it from me. Thank you so much for watching this video. You can find more resources on the website, and you can download this presentation there. The Mobile UX Marathon website has all the information and resources discussed today. Thank you very much!

‍