How to conduct a true intent study

Computer show intercept inviting users to participate in online survey

True intent studies measure the experience users have with a website and provide insights for improving the design by answering the following questions:

  • Who’s visiting?
  • Why are they coming?
  • What do they like or dislike?
  • How well do they succeed?

The answers to these questions are powerful; they help us empathize with users and understand how to address their needs. Enjoy the following tricks and tips for conducting a great True Intent Study.

 


 

Intercepting Users

You need to answer a few questions before intercepting users on your website.

  • Determine where to place the intercept.  This will depend on the questions you want answered. For example, if you’re investigating the support experience, place the intercept on a support page rather than the home page. Targeting pages can help you quickly collect the right data. But make sure you place the intercept at the beginning of a coherent task, not at the very end.
  • Decide how long to run the study. Generally, you want to conduct the study for one week,  so you can capture differences between weekday and weekend visitors. However, you may need to run it longer or shorter (e.g., a month or day) depending on your target audience. The length of the study also depends on how many people you want to participate. I recommend between 300 to 600 participants but you can use more or less.
  • Prevent double-dipping. Visitors may get annoyed if the intercept appears repeatedly, so you should normally disable the invitation if they have already participated.
  • Set the percent of users to invite. Your goal is to collect feedback from enough participants over the time period the study will run. To do this, you need to know how many unique visitors come to your site. For example, the sample rate should be set to 10% if you receive around 100,000 unique visitors over 7 days and want 500 people to complete the survey and you expect 5% to accept the invitation. It’s not always easy to accurately calculate how many people will accept and complete the survey because there are many factors effecting it (e.g., users may be in a hurry to complete a task or eager to receive a reward for participating).

Use the calculator below to estimate the appropriate sample rate for your intercept. To use the calculator, pull web analytics to estimate the number of unique visitor for the time period the study will run, then just enter the number of participants you want in the study.

[CP_CALCULATED_FIELDS id=”6″]

 

 


 

Survey Interaction

True intent studies are designed for participants to complete the task they came to do on the site. Once finished, they provide feedback about their experience. There are three main approaches to facilitate this interaction.

  1. Emulator: Once users agree to participate, they continue using your site through an emulator hosted on a third-party site. The emulator assists users through the survey process.
  2. Overlay: Once users agree to participate, a java script overlay appears on top of the website leading users through the process of using the site and then completing the survey.
  3. Window: Once users agree to participate, the survey pops in a new window allowing users to complete their task and survey in separate browser windows. The first two approaches are designed to force users through the process so they are more likely to complete the survey. On the other hand, the “Window” approach provides users with a more natural interaction.

 

Screen Shot 2015-04-18 at 11.01.00 PM

 

In my opinion, the emulator is the least effective approach because there are typically issues with the performance of your site on the emulator. Therefore, the feedback may be more about the emulator than your actual site. For example, I have A/B tested this approach with the “Window” method and have seen users rate the site significantly poorer when using the site through an emulator. The “Overlay” method may be better than the emulator because there are no performance issues; however, the overlay can interfere with the natural interaction of using the site. One of the biggest drawbacks with the “Emulator” and “Overlay” methods is that they perform poorly on mobile devices, especially smart phones. The “Window” method is my favorite because it provides users with the most natural experience with using the site. Although there is the concern that separating the windows may prevent participants from remembering to complete the survey,  I have not found this to be the case. Through A/B testing, I find that the “Window” method yields 5-10% more respondents than the other methods. The “Window” method also works beautifully on mobile devices.

Because the “Window” approach is so effective, you can easily conduct true intent studies with your own intercept. Click here to learn more about the benefits of owning your own intercept and how to purchase one.

Next, you need to include appropriate wording in the intercept to maximize the number of people who participate in the study.

  • The text should explain up front that they will complete their task on the site before taking the survey. This explanation will help increase the number of people who participate, especially those who have an urgent task to complete.
  • Appeal to users good nature by placing text stating that “I want to help”, rather than “Take Survey” or “Start Evaluation”.

See example intercept below.

online intercept

 

 

 


 

Questions & Data Analysis

The survey should collect both qualitative and quantitative data, each type of data provides value in understanding your audience and providing insights for improving the experience. Your survey should include demographic, subjective rating, and binary response questions.

 

Question Order

Order the survey questions from most to least important because participants may experience burn out, wherein they will not finish the survey. The most important questions placed at the beginning of the survey are “Tell us why you came to the site today?”. It’s important to capture users’ true intent for visiting when it’s fresh in their mind. The next most important question is “Were you successful doing what you came to do?”. Following this logic, the last questions on your survey might me ” You are female or male?”.

 

Quantitative Questions 

Include demographic questions such as age, gender, comfort with technology, etc. to better understand how different characteristics influence performance and over all experience. Choose these questions carefully so you don’t regret not asking later.

Use subjective rating questions to benchmark key attributes of the site against future design improvements. For example, “On a scale from 1 to 5 please indicate how strongly you agree or disagree with the following statements:

  • “This website is easy to use.”
  • “I am able to find what I need quickly on this website.”
  • “I enjoy using the website”
  • “It is easy to navigate within the website”

Note: The questions above are an efficient and effective substitute for the System Usability Scale (SUS)

Use a “Check all that apply” list of reasons for visiting the site. You should ask visitors why they came to the site as a free form question as well, however asking it as a checklist comes in handy when coding the free form responses (e.g., some comments are vague, “I came to check out the site”). In addition to asking visitors why they came, consider the following yes/no questions for tracking performance:

  1. “Were you able to complete what you came to do on the site?”
  2. “Did you experience any problems with the website today? For example, the site may have been slow or difficult to navigate.”

If users report that they experienced a problem, follow up with a “Check all that apply” list of problems. Also, always include “Other” with a free form explanation in your “Check all that apply” lists.

 

Qualitative Questions 

Your survey should include free form response questions so users can voice their thoughts and feelings. There are three key questions to include, but you can always include more to fit your needs. Consider the following:

  1. “Please tell us why you came to the site today? (Be specific as you can.)”
  2. “Please describe the biggest problem you experienced on the site today?”
  3. “What changes to the site would be most helpful to you?”

 

The qualitative data can be the most meaningful in understanding your audience and how to improve your site. Now that you have collected the data, it’s the time to have fun with the analysis!

 

Analysis

The quantitative data can be used to find key trends with your audience. For example, you may find that a specific age group likes or dislikes certain elements of the site. This data can also be used as a baseline to evaluate the implementation of new designs. It’s always nice to see improvements from applying the findings.

The qualitative data is always rich with insights. You may be tempted to do a quick qualitative analysis (e.g., read through and summarize the comments). However, resist and dive into more comprehensive analysis such as creating an affinity diagram or experience map with the comments. I love to see teams working together reading and organizing real comments from the users of their site; this is when the magic of innovation most often occurs.

 

Use these tricks and tips to conduct your next True Intent Study!

 

For additional resources, click here to preview an excellent presentation given at UXPA 2015.