Introducing Randomization: How to Reduce Bias in UX Research

We are happy to announce that Randomization is now live on Useberry! Launching a top-notch randomization feature was our foremost priority and we are happy to say that, the team has delivered one of the most robust and versatile randomization features on the market. This new feature is especially exciting considering the diverse objectives of daily UX studies. Every day, UX researchers run countless studies with many different objectives. Whether the objective might be mapping user journeys, observing user behavior, comparing designs, or evaluating product experiences, every study is fundamentally driven by the quest for finding insights. Great insights are a common, unifying goal among all these studies.

Insights validate designs, point out preferences, reveal problem areas, and lead to better user experiences that we all strive for. The key to unlocking such valuable insights is the results of your studies. This is why, we need to do everything in our power to ensure that the results are valid and reliable. Here is where “Randomization” comes into play. Randomization is a fantastic tool in the hands of a study creator for getting better results.

In this article, we will go over what Randomization is, how it helps achieve better results, and review some use cases. We will also watch a tutorial by Harry Karanatsios, UX Researcher at Useberry, on how to set up randomization on our platform.

What is Randomization?

Randomization is a strategic ally for researchers, enhancing the quality and credibility of their findings. It is a technique used to control for potential biases and confounding variables that might influence the results of a study or experiment. Randomization involves randomly assigning participants to different experimental conditions/products/variations or the random presentation of tasks, questions, sets of questions, and response options. This way, everyone gets a random, and therefore fair, experience.

For example, if we’re testing two new layouts for a website, some people will see Layout A first, and others will see Layout B first, all chosen by chance. This approach helps us make sure our results aren’t accidentally biased because of the order or the way we show the content. It’s like ensuring every flavor of ice cream gets an equal chance to be tasted first at a taste test. So we really know which one people like best, without any unfair advantages.

Why Randomize?

Randomization is a good practice in both qualitative and quantitative studies and helps to prevent patterns, biases, fatigue, or predictability/learning that might affect user behavior or perception and distort your data, thus making the results more reliable, and valid.

Whether it’s a questionnaire, website, or product test, Randomization plays a crucial role. It’s like shuffling a deck of cards before dealing them to ensure the deck stays “random” on each round. This is important because it prevents certain patterns or biases from sneaking into our data. If users start guessing what’s coming next or get tired of repetitive tasks, then their reactions won’t be natural anymore. By randomizing, we mix things up, keeping each participant’s experience fresh and realistic. This way, we get results that reflect what users think and feel, not just what they expect or learn to do during the study. In short, randomization helps us get a clearer, more honest picture of user behavior and preferences.

Benefits of Randomization

It’s crucial to understand the tangible benefits randomization brings to UX Research. Each benefit plays a significant role in enhancing the integrity and effectiveness of our studies. Let’s take a closer look at these benefits, illustrating how they contribute to more effective and reliable UX studies:

• Reducing Order Bias: Mixes up the order of tasks for each participant to prevent the order from influencing their responses.
• Minimizing Learning Effects: Ensures participants don’t use knowledge from previous tasks to influence their performance in subsequent ones.
• Balancing External Factors: Evens out outside influences like time of day or mood across different participants.
• Increasing Generalizability: Makes the results more applicable to a broader audience by avoiding biases specific to a certain task sequence.
• Controlling for Order Effects: Prevents the sequence of tasks from affecting how participants perceive and respond to them.
• Mitigating Participant Fatigue: Distributes the difficulty of tasks evenly to avoid tiring participants, which can affect their responses.
• Improving Research Rigor: Enhances the overall quality and trustworthiness of the study by reducing systematic biases.
• Accounting for Individual Differences: Deals with the varying backgrounds and experiences of participants by exposing them to a diverse set of tasks.
• Enhancing Environmental Validity: Simulates real-world unpredictability, where users encounter tasks in no set order.
• Mitigating Participant Bias: Prevents participants from predicting and preparing for upcoming tasks, which could skew their responses.
• Reducing biases: Diminishes the impact of participants’ expectations or unintentional cues from researchers.
• Enhancing Statistical Validity: Supports the use of statistical methods that assume each task response is independent of others.

How to Randomize with Useberry?

Now that we’ve explored the benefits of randomization in UX research, you might be wondering how to effectively implement this in your studies. This section is designed to guide you through the process! Let’s start with our UX Researcher Harry Karanatsios’ video guide, walking you through the essentials of setting up a UX study with randomization on Useberry.

We hope you liked the video guide. Now that you are more familiar with our Randomization feature, let’s take a look at how things work on the platform and see an example of using the new “Group & Randomize” block:

The “Group & Randomize” block that Useberry introduces in BETA, allows you to create “Stages” and “Sets”, apply Randomization, and define how many “Sets” you would like participants to see.

How to use Group & Randomize block

A Group & Randomize block can have multiple Stages and each Stage can contain multiple Sets.

• To create a new Stage you must have more than one Set in your current Stage
• To create a new Stage click the “+Add Stage” button

A “Set” allows you to bunch (group) Blocks (i.e. single task, tree test, open analytics, etc) or Sets together.

• To create a new “Set” on a Stage, click on the “+ Add Set” button on that Stage
• In Stage 1 you can only create Sets of blocks
• For any other Stage than Stage 1, you can create Sets of the sets you created in the previous Stage.

Randomization allows you to show participants blocks or sets in a random order so that participants do not see the stimuli in the same order.

• On Stage 1, you can randomize the blocks within each Set
• For any other Stage than Stage 1, you can randomize the sets of the previous Stage

Note: Logic Jumps might not work as intended if you apply Randomization (edited)

You can also define the number of sets or blocks presented to participants.

• The “Assign” option appears when you enable randomization for one set
• The “Assign” option allows you to select how many blocks or sets will be shown to participants.
  • In Stage 1 you can select how many blocks within a set will appear to participants
  • In any other Stage than Stage 1, you can select how many sets from the previous Stage will appear to participants

If you would like to go through more examples of setting up a study with Randomization on Useberry, you might want to take a look at our in-depth and technical guide as well.

Randomization Examples

Let’s take a look at some examples of when and where we can use randomization for UX studies:

• Different Website Designs: If you’re testing two designs, randomly assign participants to start with either Design A or Design B.
• Order of Tasks: When users have several tasks, change the order for each participant. This way, everyone doesn’t start and end with the same tasks.
• Types of Content: If your study involves observing texts, images, or videos, mix up which type each user sees first.
• Survey Questions: In a questionnaire, change the order of the questions for each participant filling it out.
• A/B Testing: When testing two versions of any given design, randomly decide who sees which version.

Conclusion

The role of Randomization in UX research is fundamental, and this is precisely why we are thrilled about the launch of our new randomization feature on our platform. It’s the key to ensuring that our studies are without bias and give us reliable results that apply to a wide range of users. By incorporating randomness in our methods, we can be more confident that our insights are reliable and relevant in various contexts. Both experienced and new UX researchers alike should consider this practice essential. With tools like Useberry, adopting randomization becomes more straightforward, thanks to user-friendly guides and easy-to-understand UI. Ultimately, randomizing our research leads us closer to creating digital products that are more effective and enjoyable for everyone and provide a better user experience.

*Randomization by Useberry is available for all Paid Plan Accounts.